Solost I know this has been difficult to read today, but it has clearly had a positive impact, for you to admit that your H hasn't taken any responsibility and the admission that he's still too cowardly to discuss or even acknowledge the relationship that changed his children's lives. Keep it coming Solost!
Now on to your earlier points. The children's care is not your job alone. As for your H not being able to cope with his own 3 kids because they are boisterous, well he'll bloody well have to cope. He's their Father.
As others have explained, you need to think of this as his problem. Your DS has explained why Saturdays would be a pain at Dad's place. Soon Saturdays with either of you exclusively, will be a pain for him. That's normal.
The conversation with H goes like this:
You: The DCs and especially DD are finding it too confusing with you coming to the house. I've been noticing their confusion and upset for a while, but I've been putting some of your needs before theirs, because you live so far away.
You need to see them somewhere else in future and we need to agree a fixed rota of when you'll see them and I want some Saturdays with them on my own. They want to be close to home on Saturdays, both when they are with you or I. Eventually I want to work towards you having them overnight and a shared parenting arrangement, but at the moment, the older ones are saying they don't want to come to your flat. In the interim, what do you suggest?
What will follow is a lot of wheedling and manipulation, but keep saying:
"No, you must stop coming to the house. If you want to see them, some weeks it's going to have to be weekday nights with me having them at weekends. What do you suggest?"
At some point he will mumble about possibly moving nearer home, or you can suggest it. You can then say that if he does that, he'll have to take their wishes at that stage into account about meeting and spending time with the OW. But for now, they don't want to meet her and he must see them on neutral ground. You might have to pretend at this stage that you no longer have any ideological objections to them meeting her, but that at the moment, they are saying that they don't want to.
As he's never spoken to them about her, he cannot call your bluff on that one.
Play the long game here Solost. With any luck, by the time he's moved, she won't be with him and he'll have a place of his own.
You asked too about other people's arrangements. I have several friends who are separated/divorced. I'll give you a range of examples that are most relevant to your situation.
One whose ex-partner also lives 100 miles away and has his son at his place (with OW) on alternate weekends, but now that the lad is 15 he doesn't want to go as much. Hence the Dad has started to take his son to more midweek football games, leaving work early to do this. The boy knows he is still welcome at the weekends, but the Dad understands if the boy wants to go somewhere with his mates and goes somewhere with him on the Sunday instead. He also takes him on holiday each summer hols and allows him to bring a mate. Still wasn't ideal for the female partner he left behind as she had a much bigger workload, but she learnt to develop a social life on her weekends off.
The other involves children who started a shared-parenting arrangement when they were the same age as your DCs. The two parents live close by and for the past 2 years they have lived for 3 days with one parent, 4 days with the other, then it swaps around to 3:4 the following week, ensuring that both parents get at least one weekend free or with the children each calendar month.
None of my friends has their ex-spouse coming to the house and FWIW, I have talked to a couple of them about a "friend" (you!) who is still doing this. They agree that it would never have worked, for either them or the children, plus they wouldn't have wanted to leave the house and have their privacy invaded by their ex.
In the short term, your H will have to negotiate with his parents or other friends to use their house as base camp. In the long term, he's going to have to move nearer to their schools/home.
When you're having this conversation incidentally, either hand him the completed deed of separation (must be ready to go by now as it's over 2 months since you saw the solicitor) or tell him it's coming his way.
Don't be afraid either of explaining that this isn't just for the DCs, it's also for you because you want to move on and keep your home territory sacrosanct and private from now on.