Its not a dilemma that I can answer because I dont fully home school, just supplement.
Life is so short, and so many people get into a rut - because of the children - and take a step backwards and dont do what they want - because we dont want to interrupt their education - etc. All of a sudden, the kids have grown up, you are in your 40s,50s, and have done nothing with your life.
You mention your DH didnt mix socially at school, but with his now current job, he was obviously very clever and probably focussed on his interests rather than people, and it sounds like he has a really good career.
Kids moving schools regularly cant get very hard on them. It takes time to be accepted, curriculums are presented differently in different schools. Whilst they all cover the same topics, schools dont have to teach them at the same time. So there is risk of lots of duplication on some topics, and other topics that they could never be taught. This will cause huge gaps in their education, and they might (might) miss out on the skills to be able to follow in their fathers footsteps.
Lots of people do home educate - very successfully - but you have to have the commitment to do it properly for their sakes.
You could make the travel experiences as opportunties to learn about different cultures, and make it part of the trip that you do this. explore the local history, lifestyles, social environment, etc. They will have a greater understanding and acceptance of the world for having experienced this.
I would recommend you do follow the curriculum, so that if they do go to school at any time, they wont have problems with interfering bodies. The curriculum is very broad so you can fit in life experiences alongside what is expected that they should be taught.
Dont worry about family comments, but do investigate the whole HE system properly so you know where you stand, where you can source your data, how to prepare lessons, etc. It will be hard, but not as traumatic as pulling your kids in and out of schools and the disruption that that would/could cause.
Do what you feel in your heart suits your family and your lifestyle. Make sure your children have and maintain friendships