You do not have low esteem OP
You might think you do but really you don't.
KH is lashing out because it's not gone his way.
So offer him half the school holidays with the kids. You do the other half and resume your holiday as it was planned.
Ask him which 3 weeks he would prefer. He just wants you not to do anything not that he wants to
KH has just realised that his OW didn't actually desire him either. He was just filling a gap.
Now his home life is broken, her home life is broken it's not quite as much fun is it. Oh diddums.
They were happy to swap flirty messages, send pics and be ever so loved up but without the deceit it's nothing.
You are decent and strong OP.
So let him stomp out his drama.
Get some stuff tonight, put the kids to bed and have a bottle of wine with PIL.
They can see the kids with you.
It will be ok.