Yes, this isn't really at all clear.
But basically I have surmised your H is pissed off that you're making the cheapy week/weekend (whatever it is) in the sun All About YOUR Birthday, instead of seeing that it's the best time to go vis a vis cost etc.
I mean, how Very Dare You want to do something nice for your birthday, shocker!!
So because you went for afternoon tea the day after your birthday, first day of your trip, with your DD and not him because he didn't want to go last year, and this year your birthday is actually on the first day of your trip, he's already getting the hump because he's assuming you're going to do the same thing again.
He's an arse, frankly. I mean, who gets pissed off about something like that? If he'd WANTED to come with you and you'd not let him, yes - that's pissed-off-worthy! But he didn't want to, so he's sulking because you did it anyway.
You've let slip that he doesn't ever like doing what you do - so in all honesty, I wouldn't bother with this. He doesn't like you, you don't like him much (at the moment at least) so call it a day.