op....have you really really really discussed it with him ? and i mean honestly calmly stated what you feel and want ?
I know the consensus is the relationship wont work, but have you told him what you have told us ? Maybe he is not as clear about what you want...? if you have ummed and ahhed and avoided stating things maybe he genuinely thinks you are on board with the plans.
Before you end it, do maybe have a clear, unemotional talk with him. Give him a chance , he may be struggling with the idea of another baby, or how his kids are going to feel about a step sibling,maybe he is trying to make it 'ok' by offering them this home which as someone else pointed out they might not care that much about. He may have some guilt to deal with over the fact he so far has not been able to give them a home.
It is interesting his dd has not been to stay for months....he may be finding that very hard to deal with.
How did you agree to move so far away in the first place ? did you or do you find yourself 'steam rollered' by him ? in which case maybe try using a councellor to help facilitate a discussion and possibly this needs to include his children too. They might have different views to their dad's.
If he isnt right for you and what you want just doesnt fit with what he wants then so be it , but you have been together for 4 yrs it is worth a try to sort out.
I would keep it neutral and avoid the feelings about 'your' baby not being as important as 'his' children and try to focus any discussion on what is best for 'your blended family'.
Firstly I would say you need to thrash out the baby issue ....how does he really feel about this ? ok he might not be as excited as you, he has done it before after all, but if he really honestly does not want another child with you or anyone he needs to speak up....and now is the time....if you have a baby is he going to want to parent it ? or is he going along with it but has no intention of it affecting him and his life ?
Presuming he does want a baby...then next i would say that the whole family then need to consider a house 'suitable' for all of you...and that means compromises on both sides...so half way between your family and his kids....a bigger house in not such a great location....or maybe a house needing work which will have to wait so you can buy a bigger house.
I actually think his kids maybe do need their own rooms ...this is their family home with their dad after all, and its hard if they feel like 'guests'.
when my DP and I bought a home together we had 4 girls and we sat them down and talked about bedrooms and though the little ones ( 11 yrs old both) wanted to share we found a house with rooms for all, as we wanted everyone to have their own space...over the years it has paid off...and sometimes they all pile in together and sometimes they all skulk in their own rooms, but everyone knows its their home.