If it helps my daughter is ASD and last year we were in a similar boat with both gymnastics and swimming. She can swim unassisted now, cartwheels, can spin around on the bar, handstands etc. Don't get me wrong she's not making the Olympic squad any time soon but she's been able to make solid progress.
For both the key was patient and knowledgeable instructors, my daughter isn't non verbal/ limited but at 4 would absolutely not have said a word to either her swimming instructor nor her gym coach but they both were able to go at her pace and she's made huge progress (me not being there watching has helped although I realise that may not be an option for you). My daughter's gym instructor came to tell me how pleased she was that my daughter has actually gone over to speak to her during class one day recently (something she hasn't done in 2 years of going).
I think with the gymnastics as long as he's enjoying and not becoming frustrated just continue gently encouraging him, he's still very young and it's great he's enjoying joining in. My NT daughter was so uncoordinated at 4, she split her lip with her own foot trying to do a forward roll, it was something else.
Maybe he'll get aspects of it and it'll click, maybe he won't but keep an eye out for disability friendly sessions in your local gymnastics club he'll be able to continue enjoying as he gets older even if he makes less progress.
Swimming, a good, trusted instructor was key for my daughter. Understanding that being in the noodle keeping her up was a scary experience for her with the loss of control was huge, also the need for her to have physical contact with her instructor caused issues at times but she got there and she swims confidently now. Your son sounds great and what worked with my daughter might not work for you but it's brilliant that he's enjoying these activities.