Three separate issues here:
First, OP - your DH does not want to spend time with your parents, so stop trying to make him. He may be too polite to spell it out that he really doesn't like them. That is okay - he is not obliged to like them or spend time with them.
But he does not get to prevent you from spending time with them. He does not get to complain about being 'left alone' - tough, he is a grown adult who has no right to control or interfere with your relationship with your parents.
Second, DH was controlling and manipulative to book a holiday to somewhere else that he knows you don't want to go (assuming he remembered you don't like it). He has very effectively made the point that you are also trying to make him go somewhere he doesn't want to go - a silly way to go about it though; you need better communication.
If you really, really will hate it, don't go, but beware you are not just being petty or cutting off your nose to spite your face. In other words, if there is a chance it will be okay (you don't have to love it), then go to show you are willing to spend time with him.
Third, the money. It is selfish of you to spend the family holiday budget on a trip with your parents that he does not want to go on.
This year, I would borrow if necessary to stretch to going on both trips - you both go to his trip, and just you to your parents. Also consider asking your parents for financial help for you to see them on this trip this year.
Then next year, plan a proper holiday together that you both want to go on, that you can both afford.
For future, tell your parents that you can't afford an annual holiday with them as the family holiday budget won't stretch to it. Either they will pay for you in future, or they and you will make simpler and cheaper plans to spend time together.