No OP, this is a terrible idea.
It didn't get easier because he can't be arsed parenting even though you clearly needed a break.
He doesn't like your area so he wouldn't move back, does he not think his kids deserved a dad no?
He doesn't drive....is there a medical reason for that or is he just not arsed learning even though it would have made it so much easier to see the kids if he did?
It would be extremely confusing for your kids to have a dad who wasn't bothered with them who suddenly only becomes a dad to them if you took him back.
And what would happen then if it wasn't working out which most likely it won't? Do the kids lose their dad again? I can almost guarantee that if he moved in and it didn't work out that he would use that as an excuse to be in even less involved than he is now.
Tell him if he's serious about you and serious about you all being a family then he needs to prove it first by moving nearby by himself and taking steps that way, a relationship and moving back in could follow if that worked out.
Again, I can almost guarantee that he won't bother.
He says he loves you?? Sounds like the laziest kind of low effort love I've ever heard of in my life.