A few years ago, both of my DC had the 'pox. My DS (11) had brought it home from school - because, if I remember rightly, the child who had it originally's parents didn't know it was contagious. At the time, his stepmother and aunt were both pregnant, and there were two small toddlers wandering around whenever my DS went there EOW for a few hours (that's my DC' father's choice, incidentally - only spending 18 hours a month with our DS is, actually, what he wants... he hasn't acknowledged DD [19] now for almost a full decade).
So, knowing the complications that the 'pox can cause to pregnancies, and not wanting the two toddlers (one "belonged" to the stepmother, the other to the aunt) to catch it - particularly my DS' little sister as she was prematurely born and has health complications as a result of that, anyway - I called my ex-MIL and said "DC has the 'pox, he won't be able to visit with you all this coming Saturday because of the potential risk to [stepmother and aunt] and to [younger sister]. Plus he's very clingy and isn't happy to let me out of this sight right now because of how poorly he feels..."
Well, you would have thought I'd refused them access to him on a permanent basis. I got the whole "children get sick, herd immunity, blah, blah, blah" speech - to which I said, very simply that I'm not actually responsible for other people's children, only my own, and that I was trying to be considerate and prevent the unborn sibling/cousin and the gravid mothers from suffering complications as a result. Not to mention that if they'd tried to separate my DS from me at that point in time, he would have been distraught (he was like a little limpet - I couldn't even pee by myself!) which may well have made him worse.
But the lesson I took from that episode is that selfish people make selfish parents/grandparents. And friends. (And that my DC's stepmother and uncle are, fortunately, very reasonable people who saw my point immediately, and agreed with me).
Stick to your gut instinct, OP. If they're that blase about infection when around a 6 week old baby... then who knows what else they're dismissive of!