he is creating that manipulative narrative to keep you where he wants you.
Of COURSE his ex gets on well with you, one; she is shot of him, two: you are picking up all the pieces and three: she is getting more weekends off to herself, working PT and pottering about doing what she wants to do and have everyone else support her to do it.
Your BF is manipulating you into doing his bidding, the way you frame this is to leave the kids out of the conversation entirely, he keeps telling you that you have no right to be the priority in your own life, that if they were your kids you would have to parent them etc etc, but HE IS their parent, his EX is their parent and somehow a lot of this is either falling to you or dragging you into THEIR family set up.
What you say to him is that my needs are not being met in this relationship, this is not the way I want to live, this is not my life. You tell him that he rightly needs to focus on his small children, and spend property time with them to bond etc etc and you have other priorities and responsibilities in life that have been around for longer than he and his kids have and that you are neglecting them. so best for all for him to move back to his parents/wherever so that you can focus on your responsibilities and he can focus on his.
You say in one of your posts that you don't want kids - this could be the case, but it could also be that you don't want them with him. Somoene said that to me when I said it about my husband. He was right, My Ex as not the one for me and this guy is not The One for you.
Let me put it this way; Every relationship we have is a learning experience, you have a lesson to learn and you have learned it. You have learned to love someone, to trust them and to let them into your home and heart.
But this is not the set up you want and it's not the set up you need or that the kids need.
So now you need to let this lesson go, turn the page so that you can move on to the next one, and towards The One you are supposed to be with. I know that comes across a bit woo, but I did a lot a online dating and i could see literally every lesson I was supposed to learn, every skill I was supposed to gain and when I had them, it was clear the time would come to move on. Now I have My One.
I know you think the world of him, but he thinks you are there to make his life easier. You do need to ask him to move back out. Keep your lives more separate - you can still date, but it won't be a relationship that has any future as a happy family as he is not invested in your happiness and never will be. He will also never change. He needs subservience, his ex clearly wasn't happy either - and they are her kids!
HE BLEW this, not you. I'm really cross on your behalf.