Wednesday's event is a non sequitur. When he is 'using it against you' no doubt he'll explain that you wouldn't have had to leave work unexpectedly at a particularly inconvenient time, have the stress of getting stuck in traffic, and have no alternative but to ask ds2 to look after ds3 in the back garden for a few minutes until your dm arrived, if he hadn't reneged on his agreement to have the dc from school.
With regard to you 'stopping' him seeing the dc, it's patently obvious that you're doing no such thing. By choosing to live somewhere that allegedly isn't suitable for the dc, he's chosen to put himself in the situation of having to entertain them at other venues - in common with many thousands of other divorced/separated parents.
I must say I find it curious that he regards the home of his df to be unsuitable for the dc. Presumably he was once a child who lived with the very same df and lived to tell the tale? What harm can befall the dc at his df's home and what steps is he taking to remove any possible risk to them?
In respect of his pathetic threat to have you prosecuted for the alleged crime of arranging a loan in your joint names which, in the event, was not required, the words 'out, arse, his, of, talking' come to mind.
Having read a number of your other posts, it appears that during the course of your marriage to an emotionally abusive and financially improvident dickhead spouse, you had no alternative but to assume sole responsibility for organising the family finances and keeping the books, so to speak because the arse couldn't be arsed to keep the wolf from the door
At a time of acute financial difficulty 5 years ago you applied for a loan in your joint names as a married couple in order to prevent the marital home being subject to repossession and, insodoing, you committed no crime.
As it transpired the loan was not required but, had it been necessary to proceed, I have no doubt that you would have informed the self-entitled twat your spouse of the need to secure outside/additional financing to keep the roof over your heads prior to formal acceptance of the offer.
In short, the only 'crime' you've committed is one against yourself allowing this utter tosser to trespass all over you.
He's 'declared war'? Tell him to bring it on - mumsnet's guns are fully loaded and waiting for his next salvo.