I think you are slightly missing the point with all this tbh.
Maths and English are compulsory because they cover basic skills needed for life. You can improve a few grades by working hard, but you can't transform yourself into a natural mathematician.
PE isn't compulsory because it is only really needed (or useful) if you want to pursue a career related to it.
Some people are naturally good at maths and I would say that plenty couldn't achieve highly in, say, Further Maths, for all the dedication they might throw at it. Music is very similar: some people are tone deaf and, even for non-practical aspects, that can make it harder to discern aspects of style etc. As others have said, some people can' t draw, and some people have a good ear for languages.
People just have different strengths.
PE is added to allow students who have a particular strength in that area to shine. To be honest, few people who are whizzes at further maths or Latin are likely to be taking PE in reality as they sit at opposite ends of an academic spectrum and lead to very different sorts of careers, so I wouldn't worry too much about them.
PE is there to even things up for DC who are sporty rather than academic and will find their path in a different direction. I can't see the problem with that.