Nope. It won’t get better until he stops coming to your house and you protect your safe space and have a haven from him.
Essentially he’s controlling you by refusing to be a proper parent and provide his DC with somewhere to sleep when it’s his contact time.
He’s also preventing you from moving on. How can you date/have sex with someone else if he doesn’t do over night care and the DC are always at your house? It’s possible, but difficult and he’s ensuring that it’s difficult for you to move forward with your life as he still believes he has ownership over you and your home, despite you buying him out.
You need to tell him this is the last time he comes to yours. It’s up to him to provide a home environment for the kids to stay at when they’re with him. That home needs to have clothes and wash stuff and toys etc so you don’t need to pack a bag or anything when they’re with him, you literally hand them over and off they go.
If he doesn’t like that/says he can’t afford it or whatever, that’s a HIM problem and not one for you to sort out. The family courts would not support him “having” to do contact at your home, you deserve your peace and privacy.
You think you’re doing what is best for the DC by allowing contact in your home, but actually you’re blurring the boundaries and making it seem like it’s still daddy’s house when it’s not (& I bet you have to clear up after their activities when he leaves as well).
The sooner you stop him coming to your home, the sooner you will have the time and space to heal and move forward.