I found myself to be the person asking and it was absolutely mortifying for me too.
I had know this person for a couple of years and I noticed that her stomach had got bigger in recent months. I guessed she was pregnant but wasn't sure.
We ran into each other regularly and I always felt awkward to ignore her assumed pregnancy each time. Then one day, we found each other in an elevator, had the usual 'how are you' etc... and I just thought it would be so rude of me to not ask about the pregnancy, so I did... and she said she wasn't.
It was horrible and couldn't stop apologising. The thing is, it clearly must have had an impact (and I might not have been the only one saying something) because she started to lose a lot of weight afterwards.
A year later, I apologised again and she actually thanked me as she said that this is what it took for her to acknowledge that she'd put on so much weight and prompted her to lose weight, leaving her so much happier.
Ultimately, if you do look pregnant, you have to accept that people will wonder. How would you feel if you looked 20 weeks pregnant and everyone ignored your bump? Probably annoyed. It's a can't win situation.