I used to feel this way, especially when the dses were little. Like Lobsterquadrille2, I used to lie awake listening to every little noise - I even used to leave a light on downstairs, so my room wouldn't be completely dark.
But like Lobster, I actually like it now. One ds has graduated and lives down south, and for the last two years, the other two dses have been away at university, so when dh has been away overnight - which has happened a lot - I was all on my own. I will be honest, and say that I was not at all happy at the prospect - I didn't think I was going to cope at all - I was going through a bad patch with depression, which didn't help. But I was wrong - I coped fine, and there are plenty of things I liked about it - I could stay up as late as I want, watch whatever I wanted, and if I didn't fancy cooking for supper, I could just have beans on toast - it was wonderful.
This year ds2 graduated, and has moved back home, because he's doing his teacher training nearby, so it makes sense for him to be at home and commute to university - and he needs to be fed every night!! 
In fairness, though, he does do his share of the cooking and clearing up, and he tends to have study to do in the evenings, so I do get to hog the remote and watch whatever trashy TV I choose.