I find it extremely odd that he is under your roof before you had got to the stage of being around each others' children. Meeting the other children is part of meeting him and getting to know who he is.
Either he, or their mother may have issues around letting other women around 'their' children, but thats making you unhappy, and you've put up with that all this time. I think its time for it to end and if you tell him the reason why, he has the opportunity to make a change, if he doesn't well you know where you stand. We can't all compartmentalise our lives (we'd never get to see anyone!)
I agree with a father putting his children first, but I can't see how this is doing that. What is he hiding them/you away for?
There is nothing more lonely than being in a relationship alone. When you feel someone wants to be with you (not just 'there' in front of telly, eating your food), you don't worry about the time they are away, but tbh did you ever see him at the weekend? If it started on this basis, thats different, but still at some point it should have moved to everybody meeting up together with respective DC. so that leisure time could be shared and family life shared. Its not the same as someone working away, which is a lifestyle someone can choose to accept, and even then they have the right to change their mind and feel its not for them permanenty.
This isn't a reason I can see for them to never be together at the w'end. To allow him to move in before having got the children used to the idea of them even being together is all arse about face tbh. Its hiding from reality.
It could be lots of things, but its weird on many levels. It seems more needy that you have been in agreement with this happening, when you are not happy with it and are not sure yourself whether you have the right; putting it more in terms of being needy instead. If one person 'wants' more than the other doesn't necessarily mean 'needy', but each has the right to be with someone that does match them in ways that are important to them.
I do know of someone that used to go 'home', not to his mother's but to his wife! having not told them that he had actually left! and I have heard of your situation before, but because it never moved beyond that, that said it all (he's not allowing it to move beyond that) so she declared it over, as it wasn't what she wanted left her feeling she was asking for a relationship that he couldn't give, end of. It is he that can't give it, not you that can't have it (just not with him).
He is not being transparent with you if you do not have sufficient reasons to understand this situation, which denies you the opportunity to make your own decisions (unless you decide that is sufficient reason to base your decision on, which I think it absolutely is). You are not happy with the situation, and doubting yourself as a result.