My DS hated PE with a passion in Primary school and would try anything to get out of it. When he went to Secondary, although the children weren't 'streamed' he did seem to be put with a group who for the most part were not particularly sporty, so I think that actually helped his self esteem and encouraged him to join in a bit more. (DS will only play games if he thinks he is going to win or is the best, so I think this was a big issue with him at primary)
He still hates football and rugby but at least tolerates them, but is better with the other sports they do such as dodge ball, basket ball, athletics, swimming, badminton. Since going into Yr 10, most of the sporty types have opted for GCSE PE whereas DS has not, so he is now definitely with a non-sporty group.
Sadly, my experience of PE teachers at secondary school is that they are still very competitive bullies like they were in my time at school and they don't seem to want to be bothered with children who are no good at sport, no matter how hard that child tries. So a child who doesn't want to try stands no chance.
We tried to encourage DS to find some activities he likes outside school, so he does swim club, indoor climbing and goes to the gym. This has improved his sporting ability and confidence in PE massively and gives me something positive to say to his PE teacher at parents evening!!!!
If your DS really doesn't want to do PE, can the school not utilise him in some other way in the lesson such helper/referee, like they do in DS school when someone has an injury/illness that prevents them fully participating.
The greatest thing that DS likes about PE in Secondary school is that entering sports day events is voluntary (not compulsory as it was at primary school). Consequently he has not taken part in sports day in the 3 yrs he has been at secondary school.