How far away is he travelling? - as that can make a huge amount of difference.
My dad sounds like your OH. I grew up not really seeing my dad much, because he'd work long hours, and then be away at weekends (3-4hrs each way to a game, and as he went by coach, there'd be a good few hours pub-time before the match once the coach had arrived. It was an all-day event). As I grew older, I loved football too, so went to all home games for my team from the age of 10yo, as I had a season ticket.
At uni, I met my OH, and still went to all home games, but my OH supports a different team to me. I made the decision to put my family first, so once OH and I married, I stopped going to as many games, and once I had kids, I went to even fewer, as I now live 4+hrs from my team's ground. (Don't have my season ticket anymore, which is a story in itself).
My dad, still goes to all home games, and since he has retired, he now goes to all away games, and as many European Aways as possible too. He lives 3 hours from me, but we rarely see each other because he's "too busy" and despite my girls being very talented dancers (winning competitions at festivals etc) he has never seen them in a performance/show because the football season doesn't allow it.
Regarding your situation, I think you need to make a decision now. Either there is some compromise that you can come to, you accept that he'll never change and put up with it, or you accept he'll never change and act on it. Obviously, I don't know your OH so I don't know whether he would change or not, but having lived with someone who put football in front of their family, I knew when I started my own family, that it was not a compromise I was willing to make, so chose to put my family first. (Of course, my dad goes on and on about how I'm not a true fan, how I must hate football now, how he doesn't understand it etc. I do miss going to see live footie - it's brilliant - but spending a small fortune on coaches, season tickets, and food/drink when there, not to mention the time-cost of being away from my family for a whole day every weekend, it's not worth it, imo.)