Why should he get to swan in and out when he chooses without any thought to what the rest of the family is doing?
Because he is paying half for this home? Why does he NOT get to use it how he wants?
He isn't swanning in & out, he wants to come home for his lunch. I doubt the popping in for tea would work out.
I doubt the kids, when they are teenagers, will worry about HIS routine & sleep when they decide they want lifts everywhere & are rolling in at all times of the day & night.
It's a home, not a prison or some fortress that no one can be allowed to use, especially as that person is an adult who is funding part of the living that happens in that home.
Going by the info on this thread, op, what does your dh do at the weekends? Or when he has time off work?
Perhaps this is why he felt he needed to go away for a few days, so not to interrupt your routines or disturb/"freak out" his kids. 
She thrives on routine and stability That is going to be out the window when dc4 makes an appearance. How do you expect your dc3 to react when you have dc4? I think you might find that you are thankful for your dh popping home when dealing with a clingly, easily freaked out toddler & a demanding newborn.
Better to get her used to small changes (like daddy using his own home at lunch time) now, than wait til a new born hits the house with full force. I think this could actually be a great way of getting the toddler used some minor changes in order to give her the 'tools' she will most definitely need to have, to cope with the appearance of the little being that will be, by default, more clingy & needy than her.
She NEEDS to have her routine disturbed & changes made so she slips into life as an older sibling a bit easier than she is likely to do so at the moment.
Yes, the baby may wake her but by then she'll be older and not need a nap as much. well she will, won't she, as currently she only sleeps for 45 mins at night.
Your biggest issue here op, which I think you know, but you are putting it all on your dh, is how your dc3 will cope with a younger child in the house. You say you are 10 weeks, so you have about 6mths left before dc4, I don't see your toddler changing that much, in that time. You have to see that your dh coming home at lunch time could be a help at that point, not a hindrance.