Your baby is only 11 weeks old and it's perfectly normal for you and indeed baby to only want and need each other.
Your Baby doesn't need to bond with anyone else except you and you wanting to have that closeness is normal. I certainly feel I will be the same tbh (about to pop)
While it's lovely people want to help, if you're not comfortable, maybe simply say you're OK with baby but can they help in other ways (like some tidying or cleaning or food etc)
Or you can baby wear in one of those wraps, which makes it a bit harder for people to just come over and ask for baby?
But really, if its stressing you out, simply don't offer or take your baby back. I wouldn't ever be offended if a mum wanted to take her baby back if I was holding them.
Depending on the level of relationship you have with them you can even say, you're having a a bit of a protective day with baby so would prefer to keep holding them.
If it were me on the receiving end, I'd offer to do something around the house instead and wouldn't be remotely offended. You are mum and your feelings and decisions trumps any visitor, family or not.
I've been cooking and dropping off meals as I'm batch cooking for myself for when mine arrives, my brother just had his so I've been making extra and taking them a fresh tray and freezing my trays. I wasn't ever expecting to hold baby, but they freely passed me baby and even let me feed him. I kept offering baby back if I felt baby was wanting mum, for your very reasons, but they seem pretty relaxed about others holding him.
I have to admit, I'm not sure I'll be as relaxed as they, I can see myself being quite protective!
However, everyone is different.
The only right way is mothers way :)
Please don't let yourself become stressed and upset. You do you. X