Is your child attending a non denominational school ? If so , you are quite safe . Schools do not allow children to be proselytised - they just teach about different religions. It's not scary and your child won't catch anything . Don't you want you child to learn about these so they can make an informed choice about what they believe ? Or are you scared they will learn something and disagree with you ?
Ask to see the curriculum if you are stressed about it . It's very low key eg welcoming a new baby in different religions - what seiks , Hindu, Jews , Christiians etc do
So no, don't withdraw your kids from school assemblies, nativity plays etc . You child will hate you, the other parenst will think you are a nutter and the teachers will roll their eyes.
There is one kid who has to be withdrawn from assemblies at my kids school. She has to sit behind a curtain so she doesn't hear the local minister saying such dangerous and controversial things such as
" at harvest time we should be thankful for what we have and think of others less fortunate " -followed by school collection for oxfam
" at Christmas time we should be thankful for the love of our families and think of others who are less fortunate " followed by collection for Charity for the homeless
Meanwhile all the Muslim , Seik , Hindu , Jewish and atheist kids sit there fidgeting just like everyone else .
If your own atheism is so strong, I'm sure it will be able to withstand the ideological onslaught that will happen if you child is allowed to make a Easter card with cotton wool balls and yellow sticky paper beak . Or dress up in a tea towel during the nativity play .