Wow Pandora, inspired and in awe of your dd.
Mother, Is your dd at school? If so that feels very unfair - even if she wasn't at school it's not acceptable to not have equal weekends as that is typical. He's not likely to get a judge to agree to the current schedule.
I think you will end up in court as he's not being reasonable - please don't fear this.
Can you state that current arrangements aren't working as predictability is important plus weekend time must be shared. Propose every other weekend - Friday to Sunday or Friday to Monday, plus a night during the week. If he can't make that night due to work on a regular basis, you understand and where possible will try to rearrange (if it suits you and your dd)
School holidays to be spilt equally or slightly extended such as he has an extra day over half term. This is the norm for most parents, unless both parents have equally work responsibilities. You can propose that you each agree at the start of the year the holiday schedule BUT with the view it's equal. Christmas etc to be alternated.
However from what you have said it seems it's highly unlikely he will agree so you have to take the lead and you may get flack from him BUT it won't last forever.
How do you feel about this? If it ends up in court - so be it. The benefits are, you will get more fairness that you do today, you will get predictability and a solution that will last until your daughter is older. It seems like a choice between confrontation now to get an agreement you can both live with or continuous low level confrontation until your dd is grown.
The benefit of a court order is both sides have to comply and it can really reduce all the unnecessary arguing that happens over contact so I feel it can be beneficial.
If he fails to agree to your proposal (seems likely) ,invite him to attend mediation, if he fails to agree there (and I know one situation where the dad just sat not willing to say anything so it was a hour of non talking) then go to solicitor and get the court process started.
In the meantime can you insist on equal weekends and shared holidays? He can't come into your house and take your daughter, even if he feels like he can. Most controlling men back down once they realise their previous compliant partner has now put in place a firm boundary...but they will test it many times, they many rant but ultimately he has to go to court to get your proposal over turned.