I would look at formalising things now.
Consult a solicitor about divorce proceedings, formalise contact, the where and when, so he gets them overnight at his parents and you get some time off as well.
Try and keep it amicable, tell him the DC need to have some stuff at his place as well so he can wash and clean their things instead of handing back a load do laundry for you to do, and feed them and they have a proper routine and understand what to expect.
you can apply for child maintenance as well as you are separated and you are doing the bulk of the overnights, if you have shared care that can change when it is in action.
Start thinking as if you are splitting up as you are, gather up his things and box them up, ask your solicitor if you can get an occupation order till the divorce is sorted as current arrangements are not really sustainable. But I'd also talk to your STXH, and ask him to respect your privacy and discuss how you want the split to happen and what expectations should be, as you cant walk in and out of his parents home its unfair on you and the DC he does it to your home, which he is no longer treating as his home.
If you can keep things amicable and agree things between you it will be better for you and the DC and will keep costs down in terms of a divorce.
Give some thought on how you want contact with DC to move forward and what you want in terms of the house etc. Then sit down and discuss it with your STBXH.
Right now he is having the best of everything, his pants and socks washed, cooked food handed to him and fun time with the kids along with popping and doing a few chores at yours to make him feel good about himself.
Also if you're not interested in his work and stuff, just say you don't have time to for small talk, he's there to put the kids to bed (or whatever reason) and you'll let him get on with it and then walk away and do something else.