OP, you absolutely have my sympathy.
He may mean well but you feel as you do and that is absolutely your right.
You don't owe him a relationship or a part of your life.
Of course I can understand your value to him and his wish for a relationship, but he doesn't have any right to one.
That level of intrusion into my life like that would make me itch.
However much he wants a relationship with you because of his lack of family, you really are not under any obligation to meet that need.
Haven't read them, but these types of threads will attract the "women suck it up and be kind " posts, but as a post menopausal woman I think nope, you don't have to allow him to trample on your boundaries and constantly insert himself in your lives.
Staying in your home, tagging along on holidays? Nope.
Absolutely not, if that is not what you want.
I think you need to either have a chat or write him a letter.
You can try and do it kindly, but it will likely really upset him as he is so needy and is do intent on being in your life.
You can try and frame it kindly.
You have a really busy life, with a huge family and you simply are not able to, nor wish to give up all the time he requires.
You can say you have no wish to hurt him, that you really wish him well, but that you have X number of siblings and friends, a career, young children and you do not have the time.
Tell him you are happy to exchange cards, but beyond that you really do not have the time and space in your life.
This whole relationship from the beginning has been lopsided and about his needs.
Obviously you were very silly to have entertained it for so long.
Your choices are to suck it up, refuse to reply to his contacting you, or try and send a polite, 'kind as you can' letter, wishing him well but explaining you neither have the time nor the energy/interest in being closely involved.
You need to own that you are going to hurt him, and accept that.
You have indulged him for years when you didn't want to, which is why your other siblings find it so hilarious.
Most people would have put boundaries in years ago.
The alternative is to keep saying no to meeting up and have a conversation that tells him you are too busy and that you have other pressing priorities.
I do feel sorry for him, but as a woman there is always someone that needs our support, time, imput, and it gets very wearing.
Best of luck.