As others have said its all about where you feel comfortable and if the fear of the 'something going wrong' won't go then you won't be comfortable at home.
I've never been in a hospital except for tests (I was born at home too), so I'm pretty uncomfortable there. I knew I would have been more tense in hospital.
A friend due at the same time was a nurse. She tried a home birth but fretted and had to be transferred. Her second birth in the hospital was much better.
In terms of the backup I have a close friend who had her first DD in Guys, her heart rate dropped to 0 and it was 30 minutes before she got a section because it was between nursing shifts. The real problem was poor monitoring throughout labour.
This friend was terrified while I was in labour. But visiting the next day she marvelled at how lucky we were (and not just lucky to avoid the awful trauma she had)
It takes 20 minutes to get a theatre ready. If your midwives call and you live in an urban area you might not even experience delay. The care you get with homebirth really is very good.
Birth (and indeed life) is risky. Being in a safe and comfortable place cared for by competent people you trust is the best mitigation.