OP, I hope I'm wrong about this, but could the reason he's backed out of going on holiday with you and the kids, be that he has another woman in the background, who he's been able to see easily, with you and the kids being out of the way? If so, maybe his reason for wanting a 'solo' holiday, is actually that he's planning on a nice holiday away with her, rather than with you and the children?
Also, I'm a little confused, you refer to the children and being 'mine and my partner's', do you mean that you and he are the parents of all 3 children, or does he have children with someone else, who you've also taken away with you?
Either way, you are not being unreasonable, as a man who refuses to holiday with his partner and children, but then takes a 'solo' holiday while his partner stay's home and takes care of everything, is a selfish arsehole, and in your shoes, I would be showing him the door.