What do you do in the evenings if you don't watch TV much? Are you sitting with your DH chatting, do you have lots of guests, are you doing your own thing in different rooms?
At different times in my life, I've not had a TV or had one that sat unused but I was doing different things instead of watching TV depending on my lifestyle at the time. At one time, I was out socialising, working, doing activities outside of the house so my main living areas were mostly used for socialising. At another time, I was at home more but was doing different activities at home, like learning a language, painting, making music, so my needs from a living space were different. And during the early DC years, I was catching up on everything that needed to be done around the house so was never in the sitting room anyway
What does each memeber of the family need space for?
Your DC need a play space. The biggest advantages of a dedicated playroom are being able to leave toys out mid game to come back to the next day and being able to shut away the clutter and mess.
Your DH wants an adult space. I agree with him. If you have the luxury of having the room in your home, having a space where you can switch off from being a parent is great. Ditto, a space that isn't used for work.
The question is, what are you going to use the adult space for most of the time?
It might be better to have the playroom downstairs with sliding doors to separate the space from the more adult sitting area hide the mess, if you have another use for the upstairs room. It really depends on what you do to relax. A quiet reading room, craft room or gym upstairs would work. It doesn't work having a gaming, TV or music room next to the DC's bedrooms.