Yes, special after mainstream is a bit of a shock, but seeing your child happy really makes up for that.
The first time I went round a special school, I was like okay, but only if it's the transition class (because that looked like a mainstream type classroom) but several months later, when he really wasn't coping at mainstream (same age as your son) and he moved to the special school, I was relieved he was just in a place that understood him - they knew that even following an instruction was difficult for him.
It sounds like your instincts are telling you that you've found the right place - I've looked at SO many schools and I do believe that when you've found the right school, you just know. I think it's fine to have a Plan A, but if something works, it works and shouldn't be ignored.
Your first concern is what's right for your son - even his sister might be better in a separate school, so she can be her own person and not feel protective or embarrassed or worried etc by her brother (my two get on very well and I think separate schools helped that).
EHC Plans now go to age 25 too, so he's got plenty of time to get qualifications and social skills, without being in a stressed environment that he can't cope with.
I know exactly where you are coming from, but unfortunately many people just don't understand why mainstream is so difficult for ASD children, so don't be pressurised by their negative views of special needs schools - take every opportunity to get to know the other mums at the school and amongst them, you will soon feel welcomed and accepted.
I hope he carries on doing so well/x