Well, you're not being controlling.
What he is doing is a particularly nasty and underhand form of gaslighting. He's aware there's this thing called 'being controlling'. Because he is a selfish git, he has understood this to mean 'a situation where someone doesn't let me be totally selfish'. He is using this rhetoric of 'being controlling' against you.
I'm not saying he's doing it on purpose or that he knows he's doing it (thought those things may be true). But he's still doing it.
Instead of focussing on whether you're being controlling (which you're not!), could you set things out in terms of what everyone in this situation needs.
You need your sleep. You are doing all the night waking, so your sleep is very badly impacted already, and you need your sleep to be the priority.
The baby needs to get into a routine where night is the time for sleep, not the time for noise and light. We know new babies think nighttime is party time, and they need to reset that clock.
And he needs time to relax (everyone does).
At the moment, you are being kept awake until 1am, then waking with the baby.
The baby is getting little sense that night is the time for sleep.
He is getting four hours of relaxation time (and, forgive me asking, but I do wonder whether he's hard at work around the house from when he gets in from work to 9pm?)
I would think it's obvious he could sacrifice a bit of time to make things fairer for you and the baby. Whether he does that by accepting he can watch TV pretty much anywhere or whether he does that by getting up at 5 with the baby to let you have a lie in, so that he's too tired to stay up to 1am, is negotiable. But it's not on for him to ignore everyone else's needs.