Honestly?
He goes back for visits and sees his teenage mates now living the grown-up version of small-town, teenage-mate lives and sadly there's a big slice of him that wants that too. If you give in to it, not only will you end up living in a place you don't want to live, but with a man who regresses to something you probably don't want to live with either.
It's so obvious. In his mind's eye, he can see a lovely, lazy, ambition-free life (in every sense - work, social horizons, experience of the world). He trots off to work, unencumbered by childcare because his parents pick up the slack you don't. You do the rest. And he trots off down to t'pub every night to tell the same stories he told at 18 to his group of mates while you carry the rest of the cans.
I don't think you should be talking about moving, I think you should be talking, hard - and you should be looking at him even harder - about what kind of people you really are and what you want. Because it sounds as if you've got yourself a very boring, quite traditional when-can-I-go-back-to-my-village-and-live-the-tiniest-life-possible little man here, who you accidentally met when he was on his one adventure outside of mummy's skirts, and now he's had enough of that and wants to go home and slip back into being the person he was at 16.
It's not confusing that he references being nearer his parents but doesn't seem to want to actually communicate with them, by the way - it's classic, it's actually one of the strongest pointers that this is what is going on. He's a child, this is a child's response (man-child, that is). He's not interested in his parents as people. He has no notion of engaging with them. They're just The Parents and somewhere in his very, very narrow-minded head, he should be living round the corner, and he should be handing his kids over to them to do the parenting you don't do, while he Goes To Pub. Preferably on a Sunday afternoon, when he intends to roll home in time for the roast you've cooked for everyone, so he can nod hello to Mum and Dad sitting like bookends in the sitting room you hoovered this morning, with the Sunday papers. Sound familiar?
NEVER agree to this. It will kill your marriage. As well as your career, by the sound of it.
I honestly think the first stage in this conversation would be to lay it down that never, ever will you want to just return to your hometown to live the lives your parents did and if that's where his ambition lies, you need some very honest conversations.