As you said, you could have been dxd yourself at 3 and you have gone on to be successful and have a loving family. That's great and your ds may well follow in your footsteps, particularly as he has parents who are able to understand and support him.
Same story for my bil. The school wanted him assessed, fil physically ejected the EP from his house. Bil went on to suffer socially and in other ways through school, but came out of it with a Phd in a scientific subject, is happily married and has 3 lovely dcs. Yes, he did all that and he would still get a dx of AS now if he was assessed (he says this himself). His ds also has AS, but has been massively supported both at home and through school and bil has made sure that his ds own doesn't struggle the way he had to.
I agree, it is perfectly possible for some people to get through life without a dx, live life in their own way and be perfectly fulfilled and happy. The problem is, its so hard to know who will be like that and who will be like my ds, highly anxious and emotionally fragile and in need of a massive amount of support to achieve his potential. We thought ds was doing fine, right up until he crashed and then it seemed to explode out of nowhere and was one of the scariest and most distressing things I have ever been through. I will always feel guilty for not getting him properly supported sooner. We honestly thought he was doing fine, a bit quirky, stubborn and off the wall, but coping in spite of it all. 
I am very hopeful that my ds will go on to achieve whatever he wants to achieve and have a happy and fulfilled life, but in his case I now know that he can't achieve that without getting a lot of support through the next few years and in order to access that support he needs his dx.
It is probably more complicated for your ds because of his vision problems, but as someone said up the thread, you don't have to accept a dx at this point, even if one is suggested. They asked us whether or not we wanted the 'official dx' when ds was assessed.
Also, the thing about the assessment being in a relatively short space of time, in an unfamiliar place etc is sort of valid, but if your area is anything like ours, that is just the final piece in the puzzle rather than the whole assessment. If they have already seen your ds, you have given them a full history and completed the relevant checklists/questionnaires then it will all be taken into consideration, rather than a decision made just based on their observations over a two hour period. My ds's assessment involved 12 weeks working 1:1 with an EP, reports from the OT and ASD Inclusion Team who had been to observe him in school and both checklists/questionnaires and meetings with ourselves in order to gather as much information and history as possible. The actual ADOS assessment was really just the final link in the chain.