When I had my first baby I thought it was outrageous that my DH wasn’t allowed to stay on the ward. I would have appreciated his help, but looking back it wasn’t essential - I managed on my own, even though DS1 was the baby who cried all night and the midwife got annoyed with me even though I only buzzed twice. I’d had a pretty awful birth and episiotomy and the stitches fell out.
When I had DS2 we had moved house so we were in a different hospital where the rules were different. I was induced and DH could stay with me on the antenatal ward. Great! I thought. Except the option for him was an uncomfortable chair where he got barely any sleep anyway. Because all the other women’s partners were also staying it was noisy, the partner of the woman on the bay next to me watched Eastenders on his iPad all night with no headphones. I also went into labour very quickly in the induction ward and my waters broke which was quite embarrassing especially because there were men listening to all that was going on. DS2 was born less than 30 minutes later - thankfully on a delivery suite.
The next night on the postnatal ward, DH had had about 1 hours sleep on aforementioned chair and his back was agony so I told him to go home and get some sleep so he was in a fit state to look after the 3 of us. All of the other partners stayed and I was surprised how vulnerable I felt even though I’d had a pretty good birth. Having to grab maternity towels and dash to the toilets with blood dripping down my legs. One of the partners was a hindrance rather than a help, spending most of the time moaning on the phone to his parents about how they hadn’t been discharged yet and moaning about how useless the staff were. DH came back at 10am and stayed for the day until we left and that was totally fine. Fortunately I’d had an straightforward birth this time so we only had to stay for 24 hours.
So on balance I agree with you. YANBU.