Blimey, OP, your post has really brought the worst side of MN out! I hope you're ignoring the ridiculously rude responses. There's a great lack of empathy in many of these replies. Unfortunately, MN tends to be pro pass the baby, and pro grandparent's wants over the baby's needs. My guess is it's a generational thing as the only people I've met who are like this in real life are older, although I do know very understanding older people too, so maybe it's just some people are... odd. 😅
My thoughts on your situation are, no you are not being unreasonable to say no to people who want to hold your baby.
Your baby is perfectly capable of bonding with people without being held by them. (How some PPs think this isn't possible is staggeringly bizarre.)
It's not true that saying no to people now will make her more clingy.
She will not 'grow up terrified of others'.
And most importantly, her clinginess just now is not because of you. You are not the problem.
With all my DC we were happy with all grandparents holding them as babies. DH's parents live abroad, but we made a point of doing lots of video calls with them so the kids got use to them. Maybe something you could do with the Grandad?
However, we only let other people (friends, great-aunties etc) hold DC once DC were old enough to show that they wanted to go to them (so... once they were about 6/7 months, if I remember correctly). All DC had clingy stages (as most babies do) and they all became more confident in their own time.
My suggestion is that Grandad shows he isn't scary by playing with your daughter however she's happy with in the moment. Smiles, songs, being silly etc. but not over the top if your daughter isn't keen on lots of stimulation. Even just sitting a bit away and interacting a little then continuing to talk to everyone else, so it's not too much interaction at first. If things are going well you could try him holding her, but say to him before that if she gets upset he's to hand her back. Some babies do get distracted by going outside or walking around, so I imagine the time before he was only trying to help her calm down. If, however, he wants to hold her and you don't want him to you can say 'We'd rather you didn't hold her just yet, as we don't think she's ready right now,' then suggest something he can do just now like read her a book. If he, or anyone else pushes you on the subject, you can say 'We're doing what we think is right for our daughter,' and, if need be, leave it at that and go talk to other people at the family event.
You sound like a great Mum, who is really thinking about what's best for your daughter. I hope it goes well for you. 💐😊