It depends on the hospital, how the birth goes and what your plans are.
With ds1, dh, ds and I were all put in a lovely 'recovery suite' just down the corridor from the delivery suites, because I had noted on my birth plan that I didn't want to be admitted (am terrified of hospital wards) but would rather have the 4 hour discharge, as long as there were no complicating factors.
The room was lovely, big pine double bed and ensuite shower room. Dh had a good sleep, while I gazed at my pfb, then I was checked over by a mw, ds by a paediatrician and we were allowed to go home. Ds was born at 12.30 pm and I was back in my own bed tea time.
This wasn't a posh or private hospital, just a bog standard NHS trust in Lancashire and I didn't have a straightforward birth either (epidural and kiwi ventouse after 3 day back to back labour).
By the time I had ds2 we had moved to the Midlands. Ds was born at 1.30 am and dh wasn't allowed to accompany me up to the ward, we said goodbye in the corridor about an hour after her was born and dh was allowed back when partner's visiting hours started at 8.30 am the next morning.
With dd - same hospital as ds2 - she was born at 3.00 pm, but not on the labour ward (very fast induced labour). She couldn't maintain her temperature, so we were transferred to a labour room where she was put under a heat lamp and I was stitched. We stayed there until 8.00 pm, so I got to have a shower etc in the ensuite bathroom, then again we dh had to leave and I was transferred up to the ward by a porter.
Fwiw, both stays on the ward were nowhere near as scary or bad as I thought they would be (and no-one could have been more scared than me). It was too hot and noisy, yes, but everyone is only interested in their own new baby really and too exhausted to be taking much notice of anyone else. If you keep the curtains round you, you have your own little space and if you're anything like me you will just spend the whole time gazing at your beautiful newborn anyway. If you are really worried about being on a ward, enquire about booking a private room - it doesn't cost all that much.
I second taking a stash of nice foodie treats - I didn't bother with the hospital food either time and after having dd had a large bar of Galaxy stashed away - never has a simple bar of chocolate tasted so good!
I also took some decent mineral water, energy bars (granola honey and nut ones), some of those pre-packed packets of maltloaf and butter and some bananas and apples and grapes. Then when dh came in in the morning, he brought me a fresh sandwich from home and a flask of decent coffee - much nicer than hospital food, imho. 
I had all the same worries as you, its only natural, but it will be fine and when the time comes, all that will matter is your beautiful baby. I've done it three times now, if it had been as bad as I worried it would be, ds1 would still be an only child and/or dd would definitely never have been!