OP, you must tell yourself, and believe it, that you have given this relationship all you could, but for you it is now over.
Your DP has put himself and his children above you and your DC.
He expected you to welcome his DC full time in your home if that's how things played out with his ex.
He doesn't even respect your boundary of taking them back to their mum's on time.
Yes his DC will be sad not to spend time in your house, with you and your son.
Yes it will be hard for them all to stay in a 1 bedroom flat.
But your DP is the person responsible for that, not you.
He is back in the position he was when he met you.
He could have saved money whilst living with you and paying no rent, but he chose to keep paying his ex even though he has their children 50% of the time.
He could have put that into a future housing account. How much would he have now?
As PP have said, his DC will still have 2 homes, and two fully involved parents.
Your DC only has one home and one fully involved parent.
He won't realise it but he needs you to put him first. Choose him. Prioritise him.
Stay strong in your resolve to carry out your decision.
Friday will be very difficult. Is there anyone that could come over if you needed them to, if things get too heated?
I think you must accept that he is highly unlikely to want to continue the relationship with you after you tell him he's got to move out.
And actually I think you might be thinking the same thing now too, after considering all the points made on this thread about his treatment of you, and how things are between you now.
If it gets really difficult, call the police.
They will come, they will ask him if he has anywhere to go (he has his flat) and then get him to go there.
Depending how badly the conversation goes, worst case, you could get the locks changed while he's at his flat, pack up his stuff, and get a friend to take it to him, or at least go with you.
Good luck, OP, I hope it goes as well as it could.
💐💐💐