Mishy, for what it's worth, here's my experience, and I don't think you are being unreasonable. The only way both of you could be is if you don't listen to each other and find a solution that ensures you are both happy, and that you both can be flexible enough to change it, if and when the situation demands.
I had a female friend who promised to be with me at the hospital in case dh didn't feel able to - I really wanted a home birth, but wasn't allowed to 'cos I was 41. She flew to Chicago for 6 weeks on the Monday before my due date on the Thursday.
My dh wouldn't have refused to be with me if I had really needed him, as I wouldn't have insisted he were there no matter what, because when you love someone you don't bully them.
At the antenatal class where the fathers were invited when the men went off on their own, he was the only one brave enough to say that he felt there was tremendous pressure on fathers to be there whether or not they felt able to do so, or not. He was openly derided for daring to speak the unspeakable truth. He was 48 at the time.
DD arrived on her due date, after an 8-hour labour and I'd lost 1 1/2 stone during the pregnancy without trying and was never sick. DH was there the whole time, and due to the lack of midwives, it would have been a lot more frightening if I'd been completely on my own.
To be fair, some of his remarks weren't well received - especially the one when he commented on the lipstick round the tube for the gas and air - but I didn't have enough breath to say 'wtf has that got to do with anything'. Like many others, when I got my due date, I drew a great big star on the day on the calendar in the kitchen and lived to regret it. It was on the 9th of the month and for 9 days it haunted me iyswim.
When I woke at 5.30am on 'the day' knowing I was in labour, I got up and sat in the bath until the contractions were about 10 mins apart and rang the hospital who told me to have breakfast. To pretend that it was just like any other day, I had breakfast and put some makeup on at the kitchen table.
However, he did stay 'north' except when the mw told him to look at dd's head popping in and out several times as if she couldn't make up her mind. And he saw her birth - something I had stolen from me as I was given too much pethidine too late and missed the main event. He said that he was really glad he was there and wouldn't have missed it for anything. In my birth plan, I asked for him to be the first to hold dd as I already felt by then that I knew her iyswim.
After all this drivel, mish, all it takes is good communication between you and your dh. Only you know what feels right for you, and only he knows what is right for him. It's only by talking openly about your individual hopes and fears together, that you can come to a decision together that will ensure you both win.