Do whatever makes you feel most comfortable.
For what it’s worth I don’t think you’re being over cautious, you’re just trying to make a call on what’s right for you and your baby.
When my daughter was a similar age we went to a family event with my husbands side of the family. My mother in law offered to hold our baby whilst me and my husband quickly had our dinner. I didn’t realise but she’d taken it upon herself to pass the baby around to anyone who wanted a cuddle like a pass the parcel and there was a moment of panic where I didn’t know where she was.
We had older relatives kissing her and putting their fingers in her mouth (apparently she was teething 🤨). I naively thought everyone knew about washing hands, not kissing babies etc. it never would have crossed my mind to have put my fingers in random baby’s mouth. My point is some families at big events go all mushy over babies and everyone wants to have a hold, and some are the total opposite. It can be really awkward to say ‘no’ to family, especially if you’re not a confrontational person. It drives me mad the way that people judge women and new mothers in this way, you really can’t do right for doing wrong sometimes.
You’ll be best placed to judge what it’s likely to be like at the party, trust your instincts.
As a compromise, there are some great suggestions here regarding wearing a sling and perhaps staying for a couple of hours or so and then either going home or staying somewhere else nearby. I think in hindsight looking back on the experience I had, I would have worn my daughter in a sling and maybe just shown my face for a couple of hours.
You are only 1 month post partum, you’re still recovering yourself and the newborn phase is tough. Worries about bugs aside, I think it’d be perfectly reasonable to say you’re very tired and still not feeling 100% yourself, or that the baby is fussy / colicky etc, if you don’t feel like inviting comments on whether you’re being over cautious or not. It’s your baby, you do what feels right!
Finally, it’s a big event, although I’m sure it’ll be lovely, it’s unlikely you’re going to get lots of one on one time with your grandfather. Missing out one day doesn’t erase all the lovely memories that came before. Don’t feel bad if you decide not to go, or only stay a little while, there’s no reason why you couldn’t come and visit with the baby another day shortly after.