Take him out. NOW.
I had a very similar experience with my son (now nearly five) last year.
We sent him to a small private pre-prep who ended up sending him home every single day within half an hour of him arriving. He was peeing himself and hurling stuff across the classroom. Due to COVID, we had no idea what was going on and we were desperately confused - he simply did not behave like that at home.
They accused me of being neglectful, emotionally cold and not setting boundaries at home. We tried their techniques for disciplining him and he became hysterical and violent.
He was expelled at Xmas last year. He returned to his forest nursery, where we hadn't had a problem (although they'd encouraged us to have speech therapy) and we went to a paediatrician. Lo and behold, it turns out he is not, in fact, a violent neglected delinquent - he's just autistic and was having meltdowns in the classroom due to being socially overwhelmed.
Since then, he's spent a year back at forest nursery making ladders, cars with wheels and generally having a fine time with 1:1 support. He has a reading age of eight, can add and subtract, write his name, loves meeting new people (although he struggles with what to do), and he's incredibly polite and pleasant with a dry sense of humour and a passion for word play.
The experience of attending the pre-prep left him with PTSD as, for several months, he was obsessed with being sent home from forest nursery. We were traumatised by the idea that he was a sociopath, on the road to prison, unteachable, or that we were profoundly dysfunctional as a family. At one point, we even tried family therapy because we completely lost confidence in ourselves as parents.
He started at a local state school this morning where he will share a support worker with another SEN kid. We talked to the support worker about the accommodations they will be making for his SEN, including putting a schedule on the wall and allowing him to 'time out' when he feels overwhelmed. His class teacher, who met him on a home visit last week, said (entirely sincerely) "I can see I'm going to have lots of fun with you".
DO NOT repeat our mistakes. Take him out of that school. NOW. This is not your fault. Your child is not failing and there's nothing wrong with him, or you. A good state school will be a far better environment for him because they are set up to accommodate SEN.
Also, being slow to speak is not a marker of academic ability. My son only started talking at 2 1/2 and his speech is still quixotic in some ways. He was only toilet trained at 3 1/2 and still wears a nappy at night. It is profoundly weird to have a kid who can count to 150, but can't take himself to the loo, but there we are - it would be strange if we were all the same! :)