This is part of being divorced. I know you didn't choose it, but this is the new world you are now in.
He is an equal parent, and in this new world they stay with him (i.e. his mum) on his days, even if they are sick.
To insist you keep them against his wishes would be overbearing and out of line.
You have nothing to feel guilty about. It is fine to miss them and worry about them, but it is not fine to feel guilty for something that is not your fault, and when you are doing the right thing in very difficult circumstances. Drop the guilt.
Have a good cry, then go out as planned.
Whenever I try to stick up for myself and stand my ground, I’m just met with aggression and verbal abuse.
Don't stand your ground in face-to-face conversations. It is a waste of time and energy. Instead, always send written messages, factually outlining events.
Keep Notes Of Bullshit.
You are doing so brilliantly, in such a short time. Of course there will be times when you get drawn into pointless arguments that you don't want, but overall you are winning this.