You have had good advice here re police and SS and I'm so pleased you called them, and yes please follow up tomorrow.
I think, personally, he is furious with you for catching him. I also think he is desperate to make you the baddie. He's already tried blaming your for the affair. Now, given he's been caught red handed how would he do that? Maybe by playing the 'she's stopping me seeing the kids' card. But you (dreadful woman that you are) can't even do that right!! You not only facilitate access you go along so the DC see you getting along.
What's a shithead to do? Oh, yes. Do something that would actually make you STOP being so reasonable. Like dumping a 7&4 year old on the street. After all, he can claim 'misunderstanding' afterwards. Or just lie and say he had to ad you won't open the door or some such nonsense.
Now - what does this achive? Well, let's see - he's now the poor dad who can't see the kids. Yay, not the baddie any more. You, if he does have them, are unlikely to leave the house in case he does it again (stopping you from meeting someone else - cos obviously as that's the first thing he'd do, he expects it of you). And finally if you absolutely so stop them, I lay money he's wander off to his nice newly promoted single life (leaving you as skint as possible if he can) with no guilt, and more important, no akward responsibilities.
Bastard.
So - after you speak to the police - you could consider the following: that as you want your DDs to have a relationship with their father, but he is currently not behaving rationally or responsibly he needs an agreed on responsible adult to pick up and drop off DC at agreed times. Or contact centre. Which he pays for, until you are satisfied he is safe. That you will only discuss DC or divorce via email, anything else is none of his business. if they are more than, for example, 20 mins late without an email or txt to say why, contact will be cancelled. You will not be telling the DC that they are seeing him until they are actually picked up. That if you offer him contact dates he needs to confirm within x days or you will make other plans. Oh and mention that you have informed police & SS and have received advice and been informed that you need to ensure DCs safety.
Keep it all very calm, and to the point. Cold even. Do not get drawn into any 'but you did x'