DS9 was seriously distressed by being at school - the environment just wasn't possible for him to actually learn in. He had to choose between coping with being crammed in with 30-odd noisy, energetic kids or learning.
It's definitely worth talking to the school about adapting before completely withdrawing. I'll describe the process we went through and if any of it is helpful then great, but above all trust yourself and your relationship with your son.
After he was diagnosed with ASD and ADHD and had a lot of exclusions (illegal ones as we later found out) we gradually worked through:
An EHCP agreeing that the classroom environment wasn't suitable and agreeing 1:1 TA provision during the school day
This effectively meant that DS's social interaction during the day was limited to 2 20 minute breaks and lunchtime. We also found the TA provision hugely variable and very inconsistent. They're not routinely trained in helping kids with neurodivergent diagnoses or anxiety and it's really unfair on both to expect them to work through the curriculum whilst working in a library, small unused office or even a corridor when the child may be so anxious about just going to school that they're in full flight or fight mode from the moment they arrive.
All the way through this school were trying timers, ear defenders, sensory toys, a weighted blanket, special pen grips, everything the SENCO could think of to possible magically unlock his ability to learn in a classroom!
A reduced timetable
When it became clear that DS couldn't cope with the full school day (for him it was such hard work just being there that he couldn't manage to settle back down to learning after the huge stimulation of break times) we requested a reduced timetable. DS would do Maths and English at school, and we would do humanities and science at home. To be honest we usually found that the TA working with DS had managed to coax him through maybe half a maths worksheet and a bit of grammar work during the morning and we did a lot of catching up in the afternoons. He was still tired most of the time just from being in school for half days.
Withdrawing him to home educate
After lockdown, having seen how well he learned at home and knowing how little actual interaction he had with his classmates at school, we unenrolled him. He's absolutely flying through the curriculum now and enjoying it.
His typical week looks like this:
Monday: Maths and English in the morning, Science and German in the afternoon. A run or bike ride at some point through the day.
Tuesday: Maths and English in the morning, German practice, History and Geography in the afternoon. Martial arts class in the evening.
Wednesday: Maths and English in the morning, German practice and a trip to a museum, wildlife park or historical site in the afternoon.
Thursday: Maths and English in the morning, German and Science in the afternoon, Martial arts class in the evening.
Friday: Maths and English in the morning, German practice, History and Geography in the afternoon. Run or bike ride at some point.
We organise playdates with other local home educated kids as often as possible. He's not hugely keen on drawing or sketching but we try to work that in to history, geography and science as much as possible. Once he's properly settled to home ed we'll also talk about music lessons rather than just tinkling about with a ukelele 
If you're in the UK and seriously considering home ed, I can recommend the facebook group 'Home education and your local authority: help dealing with officialdom'. There are some very knowledgeable people on there from all over the country and they can give you an idea of how helpful your local authority may or may not be to home educators.