but then I can't always be with my kids because we don't always live in the same house. Which is a horrible thing have said to me
You have completely misunderstood this.
I was not being horrible to you. You've made so much of this about your situation when it's just not.
My point grew out of Andrew's line of discussion about how you couldn't be with two sets of kids simultaneously.
If you've got more than one child, yoi don't have to choose one of them to spend time with
Presumably your kids are with you together? Then they go to their dad's together. The fact that they're not always with you in YOUR house is totally irrelevant to my point. My kids aren't always in my house with me either.
You're determined to take offence when there is none.
And Andrew is determined to come up with loads of scenarios where he couldn't possibly be available to his 'first family'
Congratulations- you're right, Andrew. You are indeed unable to be in two places at once. What are you trying to prove? It sucks. And? Does that mean you shouldn't do your best when you can? Or because it's possible to be legitimately unavailable, should you always choose to be unavailable? Because, say, I could be in the theatre you know? 
OP didn't specifically state how far away her ex lives, no. But she did text him more or less expecting that he could come round. So I think we can assume...