Lots of things depend on the age of the children. But the great thing about children is that they grow up!
I used to have a finance director in my old company who took his kids
hill walking with him and his DW. Their big trip when the kids were about 6 and 8 years old was in the foothills of the Himalayas!
Travelling with children isn't easy but, with planning and a bit of compromise, it's doable. You may not want to go far off the beaten track when you've got children in tow but you can still see bits of the country you're in.
Advancing in your career? Can you take more qualifications? Places like the OU or certain professional bodies can offer distance learning. Even doing an OU course for interest keeps the mind ticking over. My friend did a course on geology with the OU as she had always wanted to do it and could fit it in when the kids were in school or in bed. Didn't help her career but learning isn't always linked to a career outcome - sometimes it's just the sheer joy of learning and it's a great thing for the kids to see that learning carries on throughout life.
Getting me time depends on how much support you get. Getting time for hobbies or friends is important. Or even just having a night away in a city that has a museum you want to see or an area you'd like to visit on your own.
Have you spoken to your family about it? It's fine to try to 'quell' the feelings but it's a lot better if you can say 'this is what I need, and I need you to help me get it'. After all, if you don't say you need it, you won't get it. You need to identify what you need. Saying 'I want to advance my career' is fine. But saying 'I want to do an evening class in x' or 'I've found a great professional course I can do' will set you on the road and help your DH or whoever understand what support you'll need.
I started my own business at the age of 54. Total change in direction. Life's too short to not give things a go.