Assuming the choice is still open, I would either do both, if there was any way it could be managed, or I would explain to him that he has given you no choice but to choose between a holiday with him and one with your parents and that you were choosing to go with your parents as you feel he has put you in an impossible situation in forcing you to make that choice.
He may try to say you are forcing him to go with your parents, but you are not as you are willing to go without him, and presumably would let him go somewhere alone, or would accompany him somewhere else, finances allowing. He likely resents you spending money on yourself, but this is something reasonable - a one-off, only-chance-to-do-this event you are asking to do. It’s not setting a precedent of selfishness. It’s also open to discussion whether your child goes or stays with him. Given he doesn’t want to go, I might have been open to leaving the child with him, if that was what he wanted, though given all the red flags, I’m inclined to think he may be using the child to try to prevent you from going.
You mention finances. Do you actually have the choice now, to go with your parents, or has he already spent the money you would have needed to go with them? If he has done that, I’m afraid my inclination would be to refuse to go with him on principle, even if that means not having a holiday. If he then continues spending your shared money on something you have declined, I think you would probably have to consider leaving the relationship.
Or has he paid the deposit and only when he pays the final fee, will he have spent the money that you would need to go with your parents? If that is the case and you have joint finances, I would be inclined to reserve the money you would need to go with your parents, either by putting it in a separate account of your own that he can’t access, or by paying the required amount to my parents so they can book the holiday.
Do you have money of your own, or have you got into a situation where he is the earner? Does he control the finances? No judgement, if so, as I have been in that position myself and didn’t realize, until too late, just how vulnerable I had made myself and my children when he became abusive.
Hope you can work this out, OP. It will take a lot of strength, but what you do now will set an example for your child for future relationships when they grow up. Don’t let him walk over you as, having won once, he will either continue, or worse, escalate..