YANBU but I wouldn't say he is either.
I think all the posters jumping to red flag are abit dramatic. Some people just do relationships differently, doesn't mean either of you are wrong, sadly might just mean you're incompatible.
Some people (mnetters) are together 5+ years and still live apart and see each other 2/3 a week because they like their own space and that's what works for them.
Others see each other every night after a couple months because they take the view that if they don't have any other plans that night it's nicer to be with someone that be alone.
Both are absolutely fine. The key is finding someone who's wants and what works for them are compatible with what you want, and when they don't line up entirely you come to an easy compromise that you're both happy with.
I personally once had a relationship where I often finished work at 8/9 and would always go round to his/him come to mine after because we were both night owls and we were both of the opinion we'd rather watch tv together than alone. Now I understand some people like the space and solitude of watching tv alone, but I'm just not like that, doesn't mean either perspective are wrong.
IF he is controlling there will be other signs, don't ignore those. However I'd say this is just a sign of different expectations/places in a relationship that you need to have a light chat about and fix, it doesn't need to be some big conflict, just work out what works for you both.