I am so sorry Op, this is exactly like my DS who is 21 now. As he has got older he just is not very happy away from home for more than 1 or 2 nights at the very most . We have cancelled or come home from at least 4 holidays in the last 3 years.
I honestly don't know the answer. He is ALWAYS keen to go when a holiday is first discussed, but when it comes to the time to leave, anything abroad has become almost impossible to do and even if we get there, the holiday is pretty awful generally unless he can do exactly what he wants, which is often stay at home and do all the thing he does here (ie play on PC and PS4 and listen to his regular podcasts, use his trampoline etc)
We planned a driving trip to Brittany one year when he was 17 as we had taken him to France before when he was 10 and said he had fond memories . I was surprised by this as my memory of that hoiday was that he was OK but that he had not really love it and had been very happy to come home but anyway he chose all of the places in Northern France he wanted to visit. Most of them historical and archeological sites, his special interest at the time. We planned a whole tour with accommodation around the sites.
Luckily we knew our son by then and DH booked only places on Booking.com that could be cancelled close to the time. The riots that happened in Paris a few years ago put paid to that. We were due to leave on te Sunday and on Friday when I explained he needed to start thinking about packing and gave him the list of this things to pack etc, he looked shocked and said he didn't want to go to France because of the riots and he felt unsafe . No amount of explaining we wouldn't be near Paris could help and in the end we cancelled the trip, luckily we only lost the deposit (c £200) on the ferry with all the accommodation being fully flexible and cancelled without costs.
We considered going alone but it was a 7 day trip and frankly the holiday was not something I wanted to do for me so we just cancelled and booked a 4 night stay in Devon at the last minute. We had to come home after 3 nights though as DS was so unbearable the last day. The first 2 days were fine though
We don't have any other children so the decision is a lot easier but nowaday I go on long haul holidays with my friend/cousin once a year. Luckily DH is not that bothered about holidays and hates flying etc so he stays home. We plan a few 2-3 night trips in the UK with Just DH and me, (with DS happy at home) or with all three of us but without fail, DS wants to come home after 2 nights away. Our trips with DS have to have very specific things in place and we have to be happy to leave him at home alone in the hotel or cottage to do things if DS does not want to do what we are doing. Once we accepted all that, things got easier.
People will say we let him win but we have been on many many holidays with DS, both abroad and in the UK. The first time we flew to the US for a six month work trip for DH when he was a newborn. He has been to Italy (multiple times as family there), France, Spain, Majorca, the US, Ireland , The Isle on Man (family there) and many many UK breaks with us over the years including camping, hotels, self catering etc
Holidays with him were always hard work but I thought if we kept going he would learn to like them. Unfortunately as he grew older and had more control , he just made life more difficult than he had as a child as it is much harder to get a teenager or adult with ND to do something they don't want to, than it is a child. None of his behaviuor was deliberate, it was just due to anxiety but really a holiday which is more stressful then your nomal life is really not worth it to me any more. Not after almost 20 years of it . I have given up on holidays with him at all really and have much lower expectations when we plan the odd 2-3 night UK break with him.
In your place I would let him stay home if he is not anxious about that. We had a couple of breaks where he was so anxious with my sister staying with him for the odd 1 night break DH and I took (when he was about 14-15) that he was constantly on the phone to us - sigh!
Enjoy your holiday and let him stay home. That is my advice though I know not everyone will agree