I’m concerned that their move may affect me
It's honestly fine for you to have concerns, it's not fine for you use those potential concerns as a reason to stop them off doing what they want to do.
I'm not sure what you would gain from a conversation to be honest, it's not really appropriate for you to tell them not to move, or law down the law as to what your relationship will look like, as they aren't your parents to do that, and from what you've said, your DH wants them to move closer? So I'm not surprised they aren't entertaining a big discussion, you seem to be purely thinking of yourself here, without thinking about what anyone else wants and putting that front and centre of the decision making process.
You can obviously make it clear how much of a relationship you want to/don't want to have once it has all happened, and have boundaries in place, but you seem to be actively trying to get them to change their mind and seem to be taking it extremely personally that they still want to go ahead and that they won't listen to your complaints, to the point that you are refusing to have anything to do with them now, which is incredibly sad for your DH who must be very much stuck in the middle here.
I think the fact you don't have family is actually skewing your viewpoint slightly if you don't mind me saying, if your parents had wanted to move closer to you and your DH dug his heels in and said "no, not happening, think of how this will affect ME, in fact I want nothing to do with them now, as they aren't listening to me, and you can see them alone with the kids from now on", would you have genuinely understood that reaction or would have have thought he was over-reacting somewhat to a situation that may not ever make much difference to your day to day lives without even giving it a chance? I'd be horrified if my DH took this stance.
I do understand though, both my parents have died, I'm a very private person and generally keep people at arms length, so I do get your reluctance, it took a long while for me to have a comfortable relationship with my in laws, but I have never stopped them coming or refused to see them as it was purely my issue and I would consider it very unfair to my DH who loves his parents very much and wants to see them, over time I softened to them, and after we had our daughter I actually fell in love with them as parents because of how much they adore her, their relationship with her is really something else, especially MIL, and knowing I can't ever offer her that sort of relationship from my side of the family means I'm actually really grateful that we are all so close now.