Cautionary tale of using ChatGPT instead of a clinician

[Chronicworrier123]

I'm posting this because I've seen plenty of posters using ChatGPT as a replacement if many things, including medical professionals.

As a bit of context, I suffer from chronic constipation. I actually posted about this in February, and ended up in AE and was prescribed Resolor in case it ever happened again.

Fast forward and now I'm on HRT, the progesterone has made super uncomfortable and bloated and extremely constipated.

So I ask ChatGPT if taking Resolor is a good idea. Or said yes it was.

Well now I'm in bed with horrendous pain, and have lost at least a litre of fluid (if not more) via liquid stool.

I'm nauseous and feeling like fainting.

It would have been best to just suck it up (@!_ tell my GP about the progesterone intolerance).

That’s not really true though.. I use AI a lot and it regularly doesn’t know the answer to things and will say so. Sometimes it will ask for more information and sometimes it will say that it can’t give a definite answer. For medical stuff it almost always highlights the limitations of AI and that you shouldnt use AI to replace a Dr etc.

It's called hallucinations when AI gives factually incorrect info. I know from personal experience that ChatGPT makes mistakes, sometimes very puzzling ones.

If you are trusting AI to give 100% factual answers then that’s on you! Surely everyone knows to double check things.

Tbf depending on the question, it can be quite hard to double check. Esp if it’s not your area.
The OP’s case is a good example of that.

@MoveOverToTheSea , @YeOldy was ChatGPT not correct in this instance then? OP wanted relief from her constipation and asked ChatGPT if the Resolor that she had would be a good idea. ChatGPT said that Resolor would resolve it, which it did! What is incorrect?

I didn’t say the OP had incorrect advice. I just said that accepting AI replies as 100% factual with no critical thought or fact checking is daft
AI is a brilliant tool but you have to use it carefully.

My question wasn’t about the quality of the advice but the lack of caveat to say “check with a health professional”.

I cannot get any of my agents (paid or free) to answer a medical question with “yes go a head” or to answer a question without the advice to speak to a professional.

ChatGPT like other agents is a useful tool, no more, no less. Its inevitable that in the absence of medical appointments people will ask medical questions (or simply to check opinions) which I assume is the reason comments come with hefty warnings about checking with actual human professionals.

Lots of AI models really struggle with the hugely perplexing question of how many letter Rs are in the word raspberry. They're idiots right now. Useful idiots in some situations but ONLY if you check the output.

Apparently people's conversations on ChatGPT are visible on Google.

Don't know if it's been fixed yet like. But it's not encouraging.

My first experiment with ChatGPT was asking it to rewrite one short paragraph for me with updated references, on a scientific topic related to my work. I had written and referenced the original version several years ago. Sure thing, said ChatGPT and within seconds it generated my work for me. Written beautifully, looked impressive, factually correct. Except I diligently checked the references because I needed them for the document I was creating. None of the six or so it had selected for that one paragraph even existed. None of them in PubMed, none of them in the extensive collection of journals I can access in full online. Not a single one. They were all made up, blended from parts of actual references - a title here, journal there, renowned authors etc so that it looked fully referenced. And done with such confidence that it would have been very easy to trust.

I am sure it will improve, but for the moment, I wouldn't trust a thing that it generated on any kind of medical issue.

🤨

Cite your source please?

When was this?

Is your Google broken? It's all over the place. Pick one.

Just googled. You had to manually click the 'make this chat discoverable' box which said 'allow it to be shown in web searches' in order for this to happen. I appreciate many people probably didn't read it properly but this can't be put down to anything other than unintended user error!

I am sure you think this was clever, except you failed to read the PP, which specified the letter R not total number of letters.
My experience with medical referencing that I noted was about 6 months ago. I redid the work myself.

I wasn't trying to be clever, I was testing the claim. I did misread, but I've just asked it the correct question and got this response.
The models have evolved significantly in the past 6 months.

Can you give an up to date example of this. I use googles Gemini and it does make mistakes but I can never get it to replicate any of the silly examples people give. I think some of them are outdated or urban myth

The Rs thing got presented to me in a meeting within the last 3 months. My QA colleague had used strawberry rather than raspberry. He was very easily been able to convince it that there were only 2 Rs and went back and forth in real time, showing us.

Regardless of how much anyone on the Internet says it has evolved in the last six months, the world I work in, I am not going to be trusting the accuracy any time soon when it REALLY matters. I have to consider that what I am writing is potentially going to end up in the scientific literature as a future source for AI!

I asked chatgpt today and it said "Two Rs" first of all but then corrects itself in the same turn to three Rs.

The AI companies have of course tried to fix the most widely publicised stupid mistakes. But as you say it still makes mistakes. And those mistakes are only the ones you notice because you have the relevant knowledge to recognise them as mistakes.

An article from today highlighting the idiocy of using AI in lieu of proper medical advice.

www.theguardian.com/technology/2025/aug/12/us-man-bromism-salt-diet-chatgpt-openai-health-information?CMP=Share_AndroidApp_Other

Chatgpt can be amazing! It's helped me research a car purchase and colour schemes for redecorating in the past couple of months. I dont think I'd be tempted to go to it for medical advice though.