So as a father who is not the residential parent, I can kind of sympathise a little bit. That is to say, with the situation in general rather than the specifics.
When you work out agreements, either through family court or yourselves, the grandparents and non-residential parents siblings are never taken in to account.
So imagine a grandmother, or grandfather who had full access to their grandchild and then based on a situation they weren't even involved in now don't have that freedom, don't feel comfortable and most likely lost a connection with their grandchild during the separation.
I mention the above without meaning any accusations because of course you've (probably) done nothing wrong. It's just difficult to be in the middle of something like this and feel like your love has been dragged away from you. My parents are currently going through this, and my mum wants nothing more than to see my daughter, whereas my father now does want to see her and he asks about her all the time but for a while he said he didn't want to see her because it would only be harder next time. It's a stupid attitude but people are only human and everyone is affected differently.
So moving on to your situation, I would suggest (if it's safe to do so) allowing the grandparents or aunties or whoever to have a day out with the baby, without you there. Maybe with you there, at first, and leading on to unsupervised when you're more comfortable with it. However, it needs to be something they do. You can be helpful and welcoming but you don't need to be a doormat, they need to make the effort and come to you. So you need to let them know clearly that they're welcome, and suggest ideas but make it clear that they need to be active and not passive.
They have no right and he shouldn't be demanding anything BUT think about what's best for your baby. If his family loves your baby, and are good with him then isn't it better in the long run for him to have that relationship? That's what I would focus on.
Just make sure you set boundaries and that you're comfortable that your son is safe, beyond that I think once the initial awkwardness is done with, things will work themselves out.
My response is probably a bit biased, but I saw how hard my family have taken things, and honestly it broke my heart. My sister would never ever contact my ex, and her son will likely never be very close to my daughter, but i've had her on the phone crying to me about the whole situation whilst i've not been able to help or say anything. I can only imagine that if your baby's dad has come to you to say that, then his family must have mentioned it to him.
I hope that helps, and I hope you get more responses to help you decide. It's always a tough situation for everyone but in the end it all comes down to you, so you have to be happy with your decisions.