We all make our own choices, and none of them are wrong. Some people are happy to leave their kids, some arent. Each choice, every feeling is valid.
For eg. My DD is 9. She has never been left with a babysitter that wasn't family, and even then, over the years I'd say we only did it 6 or 7 times. Last year, when she was 8 we left her overnight with my sister for the first time.
I have had to leave her with her dad for 5 days for work, and it was fine, but then he is a SAHD so used to it. The longest DH has ever left her alone with me is 2 days. Neither of us particularly want to be away from her.
I wouldn't do it for a holiday. Work was different because I had no choice, but given the choice I wouldn't. Neither would I feel guilty, that I missed out, that I needed time away from her or that I was causing her harm not doing it.
You are not unreasonable to not want to do it, and neither must you do it. There is no special benefit to be had from going away, nor from staying with her. You don't have to do anything you arent comfortable with.