I think the whole thing about not wanting ANYONE to hold your baby is frankly quite strange. I only have one so Ds is defacto (according to MN lore anyway) both a PFB and POB. Also ds was what I would consider to be a high needs baby and easily got overstimulated and cried a lot.
I still generally let visitors (any of them) have a quick hold if they asked to.
Then again I find the whole concept of a child (even a baby) 'belonging' to me an odd one too. Ds doesn't belong to me, he is on loan and in my keeping until he gets old enough to take care of himself. My job is to work out what I think is best for him while he can't work it out for himself but he doesn't 'belong' to me.
So if as a newborn he was upset or crying I might have preferred to hold him myself as it was more likely he would settle though not always, (my 80 yr old father was incredibly good at stopping him crying and has always been very good with small babies so he often succeded when I failed at stopping ds crying in the early days)
If however he seemed reasonably happy I'd have no problem with any visitor holding him.
The flip side of this is when people get upset if you DON'T want to hold their babies. Before I had ds, one of my friends asked if I'd like to hold her newly born first child. I said that I didn't really want to (I didn't feel like holding him as he seems so small and fragile). She was actually quite offended.
I'd be very upset if I was the OP and had had been invited to visit, had gone out of my way to travel a long way with a toddler (which has it's own challeges and difficulties) to be met with such a welcome.