At the very least, you have to show him you will not tolerate this. Why should you tolerate this? It's awful.
He's trying to come between you and your children. Decent people don't do that. He's abusing your children. They're in education - how on earth are they going to pay him rent? Does he want them to leave education? What does he want? Answer: to bully them - and bully you, through them. He bullies you, directly. The whole 'This is my house' thing is utterly bullying and extremely destabilising.
At the very least, you have to make it absolutely clear you won't tolerate this. He's pushing, to see how far he can push. That's classic abusive behaviour. He's already pushed you a long way past sensible boundaries - bullying your children is a good long way over normal boundaries, and I doubt this is where he started. You probably didn't even notice when he started - you probably just felt a little uncomfortable. You've only started noticing now, a long way down the road.
You have to be prepared to follow through. Make plans to take yourself and your family back to your home if he doesn't stop this behaviour.
When did you become pregnant? Is your child here yet? Abuse tends to start/get worse in pregnancy - because the woman is vulnerable. Is that when he started pushing your boundaries?
A life with someone who bullies you, and targets your children to bully you through - and bullies dependent children (your children depend on you and your partner) - is no life. It's not joyful, is it? Why put yourself and your children through that?