Sweetheart, if you can find a way to calm and centre yourself, breathe and tell yourself that this will pass and you will be OK. What form do the panic attacks take? Is it worth taking yourself to the Drs to see if there is anything they can do to help? If you are reluctant to go down that path just yet, perhaps try Bach's Rescue Remedy in the meantime. it really helped me overcome mild agoraphobia.
Upshot of all this is that this man is trying to get you to do what HE wants, after HE is the one that broke this. If you are going to split, he will have to face the prospect that he needs to formalise money and access, and that the property issue will need to be addressed at some point.
It might be appropriate to state that you have no plans as yet to move (assuming that you are not planning this) but that one way or another, you are not prepared to live in house where he feels it appropriate to enter at his will, or withhold the keys. If he hands over the keys and ceases demanding access (and ranting and raving at you in front of the kids) then the situation will be much more manageable. Ultimately there is a possibility that he can make moves to prevent you from moving too far, but moving far away may not be in the best interests for you all anyway.
Try to find Ice Cold Fury, it's more productive than emotional outburst.
If he rants at you again, close the door. Don't let him enter your home again, have him wait on the doorstep. You do not have to have anyone shouting at you and your DC have the right NOT to see that. Remind him that HE is a grownup, HE is the one that caused this and HE is the one that is refusing to see that for every action, there is a consequence.
Remind him that while HE reckons he could just say 'ho hum, never mind eh?' and put it all behind him if YOU cheated on HIM YOU however find the idea of him getting his rocks off in the back of a car with someone you previously considered a friend as repellant and humiliating. You are entitled to that opinion. Just as he would be entitled to his.