I am like you, OP.
I get 4h sleep most weekdays because of my lifestyle - single mum, high-powered job, exercise, hobbies. Weekends and during leave I tend to sleep a bit more, maybe 6-8h. I am always awake by 6am, workdays a lot earlier than that, and use that time productively. But Friday nights are a write-off; the adrenaline I experience every day due to work wears off that night.
My ex husband used to get annoyed, too, but conversely used to think nothing of falling asleep on the sofa during the day on a weekend and expected to be left to it.
In the end it was just another way in which our lives differed significantly; we worked on different timelines.
It'll be down to both of you to find a way to work with your differences. Trying to keep awake for the sake of him will cause you resentment. Likewise, having a partner who is always asleep when you want to spend time with them is boring.
Having said that, maybe you need to both change how you spend your time together? If it's watching TV, it is no wonder you're falling asleep; it's not exactly mentally stimulating. I used to keep awake a lot more when we used to play board games during the evenings.