Hmm, I have a season ticket for a lower league club. It's the first games of the season this weekend after not being able to go to a game for 500 odd days. That's quite a big deal. I'm guessing fixtures weren't known when the holiday was booked?
However, I won't be going because it clashes with an event I'm going to with my sister. Entirely my choice though, not dictated by any arrangement the rest of the family made. It was booked before I knew there'd be a game, I probably wouldn't have agreed to go if I knew, although it was always a possibility. I wouldn't cancel plans with anyone else, except possibly, depending on who it was and how important the event was to them, for the very biggest games. For the big cup games you don't know fixtures in advance.
I have in the past arranged family holidays to be near an important away fixture and left for a few hours to see the game.
I have also joined the family later in the day, by train, after going to a game on the first day of a holiday.
Our second home game, clashes with the first day of our holiday and the ground (an hour from home) is literally on the way. I'm currently trying to negotiate a plan where we all go to the football on the way, but I'm not winning. If I can't persuade them to come, I won't go, but it does frustrate me that, just once, they won't join me in something I enjoy. Would that be a possibility for you OP?
Football is the only thing I have for myself. I am usually completely at their beck and call. I honestly don't think c. 20 afternoons a year is too much to ask. The kinds of games where I "adjust" a family holiday probably come up once every 5 years though, not regularly.