Right, if the argument didn’t involve the DC at all, then this is just a punishment for you (and your children by extension), especially your DC with disabilities, and it’s absolute bollocks, OP. I’m so sorry.
Do make notes just in case you ever do end up having to go the official route, so when you’re sitting in front of a mediator, you can say, “STBXH argued on DATE over whether he felt I should be allowed to have coffee with a male friend. The next day, DATE, STBXH announced that his flat was no longer suitable for visits from DC with disabilities. DC had been having overnight visits without a problem and with STBXH’s full approval for X months, from DATE to DATE.”
If you want to really force his hand, email him very formally now, state roughly the above, and then ask him to provide a list of what had made his flat unsuitable between the last visit and now.
If he tries to claim the flat was always unsuitable, you have the choice of informing him that you’ll be reporting him, so he should think really hard about whether he wants to play this game.