Hi OP
What stands out for me is that: a) he apparently behaves well at home and for his clubs and extracurriculars but is messing about at school, and b) he has always struggled with academics. I'm quite struck by the fact that you are saying things like "we had just began to accept that...he was probably never going to properly catch up"---that to me suggests that he was significantly behind in the first few years of school.
I think it's very likely that he is fundamentally struggling with some academic basics (literacy and basic maths) and this is affecting his ability to do his work and take an interest in it. So he is responding by messing about, as a way of having something to do and also to get attention from the class to feel better about himself, because it is pretty soul-crushing when you have to spend hour after hour struggling with work and feeling stupid as a result.
The fact that "well, he made so much progress last year with the other teacher" is neither here nor there. As kids move up the curriculum, lessons get more serious and there is less time for whizzy fun lessons where everything is all-singing and all-dancing. What's more, if kids have not got the fundamentals cracked in terms of phonics, number bond fluency and so on, often they will seem to be OK for a while, and then a bit later, when the curriculum moves up a notch and everything becomes more difficult, it all starts collapsing because suddenly they can't cope any more.
This issue is not going to go away and will only get worse as the work gets harder and as lesson styles get more serious.
I think first and foremost you need to go into school and have a serious talk with the teacher about your son's work and academic level, in very specific terms. Is he a free reader? Is he reading properly, or guessing at/slurring over tricky words? Is his handwriting legible? Is he able to have a stab at spelling unknown words with plausible spellings? Are his number bonds fluent and quick? Is he making progress with his tables and is he able to answer questions quickly on the ones he knows? Does he read comprehension-style questions carefully and does he seem to understand what he is reading?
If he has got weak areas, then you need to get cracking on getting these sorted out now, before he gets older and it actually does become impossible to catch up.
I think it's fine to accept that some kids are less academic than others-any parent with more than one kid knows that kids are not blank slates and that raw talent does count for something. But even kids who are just kinda average in their latent abilities are absolutely capable of improving loads if they put in the hard work at homecertainly, if there is no actual SEN, I don't think that a child being significantly behind their expected year level is something that should just be accepted.
Looking at the timeline of your remarks, it sounds like you make the decision that "we had just began to accept that academic subjects are not his thing and he was probably never going to properly catch up" when he was, what? Six, seven? That's very early to start writing a child off and saying, okay, this kid it's academic so let's ignore the holes in the school work and just put him into loads of sport and drama classes instead.
I think extracurriculars are lovely for confidence-building, but I wonder if you have gone a bit too far with this and missed opportunities for helping your son crack some of his school basics. When ECs take over a child's free time, it can really eat into the time spent on things like reading and being read to, making sure sufficient time and care is put into homework tasks, and doing practice tasks with parents like tables, maths practice, workbook or whatever. If your son is struggling, he may well need extra practice at home. Your son does not have to go to Oxbridge or med school, but regardless of what he does in life he does need to have decent basic skills.