Wow, after reading some of these posts, I've never been more grateful that I have so few friends.

If you don't have a bunch of friends, then there aren't a bunch of people to bug you about seeing your baby.
I guess I'm way underprotective or just wasn't presented with the right circumstances to feel this way or something. The NICU where my son was had rules that let his grandparents come at any time to see him with or without me being there. So I didn't have to be there if my mom or MIL wanted to visit. Anyone else had to be brought in with me, but only family and my best friend made an appearance. And most of that visiting happened while I was still in the hospital myself so I wasn't inconvenienced trying to meet someone at a certain time so they could see Aiden.
After we got home it just wasn't an issue. When I did visit with family, I usually took him to their home. I didn't have to entertain many visitors at my place. And when we did visit, I let anyone hold him that wanted to. I did find that a lot of people acted scared because he was so little.
But absolutely set boundaries that you're comfortable with.
ETA - I was blessed that Aiden didn't have any real serious medical issues that kept him in the NICU. He was just little and needed to learn how to eat. I may have been way more protective had he been sick as well as small.