OP, what he says does sound strange, but I think you need to find some time when the baby is asleep, and ask him exactly what he means by what he said. Make it absolutely clear to him that you don't expect him to spend any less time with his parents than he has done previously, but that you no longer enjoy his Mum's company because she does X, Y or Z, and so when he goes to see them, you'd prefer to stay at home, but he can still take the baby if he wants to, and if they want to visit your house, you'd rather go out and meet up with a friend or something.
My ex, the father of my child, became quite resentful of the baby when it arrived, but when I realised he was jealous of the fact that the minute the baby cried, I went to check what was up, I pointed out to him that the baby was a helpless creature, who couldn't do ANYTHING at all for themself, whereas he was perfectly capable, and so for the time being our child would need my love and attention, more than he did. He sulked for a bit, then thought about it, and realised I was right. After that, he became as involved in the baby's life as he possibly could be, and helped as much as he could. At the end of the day, COMMUNICATION is KEY to a good relationship, and if you can't communicate for whatever reason, you might as well call it a day.