I think you just need to be selfish because all the feelings of not wanting to pass baby around, keep them close, etc are for your baby's benefit.
I don't know any other animal (which is, after all, what we are) which would be happy for a newborn to be passed around, or would want visitors - its entirely natural to want to shut yourself away with your baby for a few weeks and concentrate on getting to know them.
I hated having visitors because they mostly all over-stayed and wanted to hold DS and I just wanted to hold him myself. I was fairly strict about allowing only close family for the first few weeks, and they only got to visit once during this period, but even so we had visitors every day and it left me really drained.
The hospital was the worst though because people seemed to think that visiting time was 'open' and they would just turn up and were all concern when, after 2 hours with me stuck in bed post-cs, I couldn't help but cry in exhausted misery.
I know that by today's standards I probably sound really mean but it was all driven by my instinctive response to my newborn.
If I had the time again I would make it clear to everyone that there would be no visitors during the first week, limited to 15 minutes second week, and no baby being passed around in the first month.
The area I did hold my ground was about nap and feed times - I refused to disturb DS when he was asleep (he napped in his cot upstairs mainly) or feed him in 'public' - not because of embarrassment but because it was a special quiet time for the two of us to share in peace, when I could give him my undivided attention.