Sadly, I don't think he's ever going to step up and be a 50/50 parent and as a PP said, I think you do need to rock the boat, otherwise he'll keep putting you through this nonsense, thinking you won't make a fuss.
Make a fuss.
If you don't do it now you'll end taking this crap for years.
I should be on annual leave this week with my youngest DD. I'm working because my ExH announced three weeks ago that he'd be taking her to see and stay with family this week. I didn't make a fuss. Just communicated that I would cancel my annual leave. 14 years on - I still don't want to rock the boat. It's ridiculous. I made my bed and live with it all the time. Please do better than I did/do.