Not if it's just a neutral statement of fact.
But when it's the type of person who says it smugly, usually in response to you having been to the doctor, as if any illness is just a weakness of your own making, AIBU to want to tell them where to stick it? And the same for people who proclaim in a superior tone that they "don't believe" in painkillers/vaccinations/antibiotics/etc, because they've never bothered and "it's never hurt me."