So - she crossed the line (and you haven't said what, so I suspect it's worse than you're admitting to yourself), she's single, and you've ended up living near her?
Sorry, but that really stinks. He's using her as his emotional mistress, and possibly physically too.
Whether it's deliberate or just unintentional abuse, he's also isolated you, kept you financially impoverished (that's abuse), and emotionally neglected you.
It's DEFINITELY time to start taking care of yourself. Work out some things YOU want in life - just small stuff - and make them happen. Do you want the kids going to school? Part of a home ed group with social life? Do you want to join a group and have a babysitter once a week?
Unless you desperately want another baby, don't get pregnant. (btw, getting a woman pregnant, vulnerable & attached is a classic abuse behavior) Find yourself some fun in your life. You're not just there to service his feel-good need of being the family man when it suits him.
Do you want to return to the UK? If he travels so much maybe you could set up base in the UK. The oldest child may benefit from school as they get older. You can travel in the summer to keep providing those amazing experiences, and he can come home often - if that's what you want.