Not read all the comments but I'm sure you've been told it's entirely normal, good, great, correct that you would be last on his priorities etc etc.
However I disagree. The occasional thing once in a blue moon with his ex, fine. To be expected when they have kids together but no I'd not be happy if my husband spent every 'holiday' with his ex and their kids. Especially after 5 years together. You are supposed to be the family together with him and his kids, not just him, his ex and their kids and then you on the side for unimportant times of the year. I just don't think this way of doing things is very conclusive of creating a new family/long term relationship, what happens if you get married? Have children with him? Is he still going to be skipping off to Christmas dinner with his ex and first kids and leaving you alone with yours?
They aren't together anymore, he is with you, you are his family along with his children and I think you should be making your own traditions as that family especially after so long together, if you'd been around for 6 months fair enough but 5 years and he's still sharing special occasions with his ex rather than his partner? No. That type of thing just doesn't scream committed relationship/family to me.
I have step children, their parents (my husband and his ex) do the occasional thing together, a birthday party, parents evenings, school plays whatever but things like Christmas and Easter and whatever else are spent with us, his family (Inc his kids). Granted we have children now of our own so it wouldn't even work him doing this but even so, I'd not be happy about it.
You can't force him to stop though I guess but personally I'd be leaving because I just wouldn't really feel like we were a family, it's like he's got his proper family for the special times and then you for the rest of the times on the side.