I know exactly what you mean.
I convinced quickly both times, once first cycle and once second.
And there was a time in my life when it did seem to be something that people were discussing and asking me about a lot, in general 'mum' type conversations. And then getting sniffy, as if I was bragging about it, which was ridiculous, as you've said, I had no control over it.
I suspect I even lost a friend over it, though it's never been spelled out. The worst part is, I've had every other issue going, awful pregnancies, dreadful, traumatic births, issues with feeding, hugely unsettled babies, and I never tried to make people feel bad that I percieved other parts of their parenting journey to be easier than mine.
Thankfully, now my kids much older and most my friends are past the TTC stage, it doesn't come up anymore!