Been there, got the t-shirt. Luckily my ex moved out into rented, we are divorced now but I'm still stuck in the marital home as it won't sell and he uses every single opportunity to use that against me.
Honestly, you need to stop hiding away. Don't let him have the run of the house while you are in the box room. If he calls you on your time asking when you will be back, ignore or just say I'm not sure. You are giving him the power. Decide on your own boundaries and stick to them like glue. If he goes out and stays out then good, that's your time to breathe a sigh of relief. If he's out shagging and enjoying himself then brilliant, hopefully he will meet someone and give him the motivation to get this wrapped up.
As for painting you as the bad parent and bad mouthing you to people. Who cares what he says, the people who matter will know the truth and that's what really matters. I've been threatened with being reported to social services, the school, the police....never heard from any of them. You have to find your inner strength because he will throw everything at you that he can.
I can almost bet my 69.58% of the house I own (yes not 70% that's how petty it got) that mediation will not work, you cannot reason with them. Act decisively and get the court ball rolling, show him you mean business.
In the words of my 16 year old daughter, you are a bad bitch and can do this ❤️