OP yes YABU, how will it make your DS feel that he has to miss out and all his friends will be at the party? How will he feel when they talk about it at school? Obviously you have a germ phobia which you accept, but please do see this from your child's point of view, as parents we have to put our own issues to one side and I think you know it's irrational or you wouldn't have posted. Please do get some help 
My DS does sensory seeking, touching things, putting things in his mouth. We've seen an OT which has helped, and I give him lots of sensory opportunities at home (e.g. tactile toys, trampoline). The OT explained that his muscles need to be used regularly, to give him the input he needs, and said that swimming is a great activity for him because of the water resistance. I don't know if this would be true for your DS too, you would need advice from an OT, so push for that referral, and maybe post on the SEN boards? So although you are worried about germs, you need to look at the bigger picture. If he has possible ASD and/or sensory issues you need to focus on supporting him with that, and not worry about the slight chance of a sick bug. Please take this kindly but I think you are worrying about the wrong things.....
Interestingly, my sensory-seeking DS who puts everything in his mouth is the one least likely to get sick out of our whole family - even when the other children have been vomiting he never seems to get it! I remember one memorable occasion when we came out of a busy, filthy train station, it had been raining and he ran over and licked a bollard
when I asked him why, he 'liked the look of the rain on it and wanted to see what it tasted like'. 
I have to say since I do more sensory stuff at home he has been a little bit better in public.
FWIW, I am a clean and slightly anxious person, but having DS has made me realise that most children are actually pretty robust!