@PithyBeaker I know we all keep bashing on about this and you must be sick of it but please do stop communications with him now and get his stuff gone. Essentially you have 3 adults (and to a certain extent, 3 children) who want him back in your home asap and he is going to do anything he can to ensure he moves back in.
Logically, he's out of luck and time and he knows it. He could buy somewhere bigger himself but he doesn't want to (much easier to live in yours, heavily subsidised and have you there skivvying too), his ex doesn't want him to (if he has less money then probably less to go to her) and his mother won't want him to as (unless she lives a very inconvenient distance away) she will know she's going to end up picking up some of the childcare slack, plus it would take months and he wants back in now.
He isn't stupid. He must know the chances of him finding another woman, single, solvent and with a 4 bedroom house conveniently empty waiting for him and his DC is extremely slim. Then when you factor in her wanting to get together with him and have him and his feral DCs move in at warp speed, the odds narrow further. Then factor in her not having any family or friends to question the speed at which he needs to get his feet under her table, then it narrows further still. Plus he doesn't want to wait and doesn't see why he should. He feels utterly entitled to you and your home and has made that totally clear.
As far as the 3 adults are concerned (and he will have rewritten it and probably believes the following narrative himself) you are an utter bitch who has thrown him and his DC out and they are sadly all cramped in a tiny flat whilst you and your DS are now rattling around in a 4 bed house that is faaaaarrrr to big for you and really what on earth would you be doing with all that space and you can't possibly need it whereas my DC/DGC do need it and the only thing stopping them being in it is you being an fffing selfish bitch. They are almost certainly all good and mad at you (without any right at all to be so) and he is going to do nothing but pressurise you via whatever method he thinks might work to get you to have him move back in. I really don't think it can be doing your mental health any favours to be putting up with that, especially as he will be thoroughly aiming for anything he considers could be a weak spot 😥
Please take care of yourself x