OP, we moved 200 miles away from a grown up daughter in her 40's and were still punished for not being there she 'needed us', she felt that we'd abandoned her.
So I think you need to ask him WHY he doesn't want to go with you, and tell him that him deciding he doesn't want to go, won't stop YOU from going, then see what reaction you get. He may say 'in that case I'll come with you', or he may agree to renting a place and trying to live on his own, then you're worrying for no reason.
However, from what you've said about him, the fact he has no friends, etc., in your shoes I wouldn't be happy to leave him on his own. I know you need to live your own lives, we felt the same way, and thought that it was our time, as our daughter was an experienced adult with a home of her own, children, etc., that it would be fine. Unfortunately it wasn't, and ended up making us all VERY unhappy, so the last thing you want is to find that he punishes you for leaving him, and clearly these thoughts aren't rational, as is proved by the way my daughter reacted. I have no experience of his difficulties, but I would also point out, that suicide in young men is rife, and if he finds himself alone and struggling, with no one there to give him a hug, and reassure him that everything's fine, he may well struggle. Sorry, I don't like to be a scaremonger, but just felt I should make you aware of where this could lead.