Hey Mo
First ever post (long term lurker) just to say how amazing you are. Under such terrible circumstances you are an amazing parent for your children.
Like Akire, I think that last night's return home time is literally the only thing he can mention. And even then he's wrong. There is NOTHING he can say against you, as you've done nothing wrong. I'd be tempted to provide him with the number for your local ss referral hub. Mention you are happy to discuss any aspect of your parenting - past and present.
I'd also be tempted to point out the local fostering allowances in your area ( in mine £200 a week for under eleven and £250 a week at eleven and over) as a more accurate picture of what it costs to raise a child, as he's so keen to take the advice of the ss. I'm sure he won't change but it may be worth making the point.
Good luck with telling the school. I suspect they will be very supportive. As others have said, I would leave as soon as you possibly can. I'm concerned that he has plans to remove the children from you/lodge them with his parents/ take them abroad.
Huge good wishes