I can completely understand your worries, but would reassure you a little.
Rugby will probably be streamed pretty fiercely in Y4. You say it's a sporty school, therefore they will be keen to field a strong 1st and 2nd team (depending on the size of the school). Therefore they will not want the boys who are less keen training with the kids who love rugby.
And really, schools are used to dealing with this. The head may not allow you to remove your son, but I would not be completely surprised if there is not a group of boys who do rugby training but will still do tag while the others do contact, and then may do a bit more swimming on rugby days, and who are not forced to do fixtures.
Can you talk to parents who have kids further up the school to find out what happens to the boys who are not keen on rugby. I am willing to bet that they are not forced into it as much as you would think. However, the head will not be keen to excuse your son as it could set a precedent.
Also you say he's in a prep school, so I presume he'll be going to an independent senior school. Do you have an idea of where he will be going at 13? If rugby is compulsory there it would be much better if he has some experience in prep, as I am pretty sure that the head of games at a senior school will be less keen to excuse Y9 boys who don't fancy rugby.
The most important thing is, as PPs have pointed out, who is teaching your child. Are they RFU accredited?
Also remember contact at Y4 is pretty limited. We are not taking about scrums, rucks and mauls. Have a look at the RFU website, there are loads of coaching videos that will show you exactly what is involved.
Moving schools may not be an easy option. You say your school is great in every other way. Do you think that you can find one that is as good, and that will allow you to simply opt out of one of the main boys prep school sports?
Hope you find a way through. I have one rugby mad boy and one who could blow away in the wind and who is not sporty at all. Ironically he received his worst injury to date this weekend playing football. He has been fine in rugby as he tends not to get the ball, and when he does, he is under strict instructions from me to get rid of it as soon as possible!