Just remember, he wants to distress you. So saying there's somewhere you need to be, is like waving a red flag at a bull.
They do not do kindness, compassion or compromise. Theh want to make your life difficult. So play your cards close to your chest. Don't tell him you need him to do something because he'll go 'good, I'll do the opposite and piss her off'. Do not share any plans with him that do not concern him. Always remember he is in his mind, at war with you. So he is always looking for a weakness. And to crush you using that weakness.
Eg: for you appointment, you should have just not reminded him and instead left for it early without telling him. Never tell them your hopes, dreams, wants or desires.
Timing is so important now. Pursue the divorce ASAP. If the house is owned, sell it (the divorce will help bring the about). You need to get out before he loses that room and moseys on back. Which he will. If its rented, see about finding a new home for you and the kids. You need somewhere he has no access to.
You might be wise to keep the keys in your locks wherever you are asleep too.