Why do people treat ChatGPT like it’s the Delphic Oracle?

[HolyRigatone]

Don’t get me wrong, I think ChatGPT is great for some stuff, but some people seem to be treating it like the font of all wisdom.

Talking about their problems to it, taking relationship advice from it, getting it to counsel them.

It’s not perfect and it gets things wrong all the time, I’m not sure I’d be sharing my deepest secrets with it or making life-decisions based on it’s outpourings.

Or am I just a stupid old Luddite?

That's fundamentally not true, and the key is in understanding how they actually work.

LLMs don't answer your question - they can't, because they have no concept of "truth", "correctness" or even "facts". Any evaluated response to a prompt is simply a picture of what an answer might look like.

Consider this: by changing the temperature (ie the range of probabilities it's willing to consider for each sequential token), you can dramatically change the response it gives to the exact same prompt and context, and the lower the temperature the more variation you're going to get in your answers even at the same low temperature.

So no, they're not wrong on the same scale as people. They're randomly, non-deterministically wrong, but they will be 100% confident on it at the same time.

So...sure, go ahead and ask them for ideas or tell them to get started on creative work; they're great at that. Anything else, and it's functionally no different from rolling the dice.

I asked it for a family friendly campsite.

It recommended one with real 'kids running barefoot through fields' vibes

It was an adults only site.

It's shite.

I asked for suggestions for the interrailing trip I'm doing. It made up overnight trains.

There was a thread a few days ago where the OP was asking for advice on what to message to her friend. Somebody posted a suggestion from ChatGPT and many of us said it was the best advice. My (consultant) psychiatrist also tells me to look up symptoms on it but then again she is lovely but fairly loopy.

I use ChatGPT to help me with many things but I always cross check whatever it comes up with.

Apparently it bankrolls Trump. I’d avoid.

Well OP….

You’re definitely not a stupid old Luddite. What you’re describing is actually a pretty common and reasonable concern. A lot of researchers, technologists, and even the companies building AI say something very similar.

A few points that might help frame it:

1. AI can sound more confident than it actually is

Systems like ChatGPT are designed to produce fluent, helpful text. That can make answers sound authoritative, even when the underlying information is incomplete or occasionally wrong. That’s why it’s best treated as a tool for thinking or brainstorming, not a final authority.

2. It’s good at certain kinds of tasks

People tend to get the most value from AI when they use it for things like:

• summarizing information
• explaining concepts
• brainstorming ideas
• drafting text
• learning or studying
• coding help

Those are areas where mistakes are easy to spot or verify.

3. Personal advice is a grey area

Some people use AI for:

• talking through problems
• journaling
• getting a neutral perspective

That can sometimes help people think things through. But it’s not a replacement for:

• trusted friends or family
• professional counsellors
• real-world experience

Most experts would say major life decisions shouldn’t rely solely on AI advice.

4. Why people open up to it

A few reasons:

• it’s always available
• it doesn’t judge
• it can help people structure their thoughts

For some people it functions more like a reflective notebook that talks back than a therapist.

5. Healthy skepticism is actually the ideal approach

The most sensible stance is roughly:

> “Useful tool, but not an oracle.”

People who assume AI is always right can get into trouble. But people who refuse to use it at all may miss out on some genuinely useful capabilities.

So your position—interested but cautious—is pretty much the sweet spot.

---

If you’re curious, there’s also an interesting psychological reason people start treating AI like a wise advisor. I can explain that effect (it’s surprisingly well studied) if you want.

Agree. I use copilot instead 👍

Yeah, seems like it.

I don't mind if it's self help. I actually work in a helping profession and have a lot of knowledge about psychology as a result so I am able to feed it a lot of in depth and complex prompts meaning what I get out is genuinely insightful, meaningful and complex but I also agree it's not equivalent to a human therapist. It's not surprising that therapists and psychologists are protective of their skills and profession, and it's also unnerving to think that people may be able to achieve the same outcomes with a free or cheap AI tool as they can with a human therapist. I share the concern about AI coming for my job if I'm honest.

What is bankrolling Trump? Every AI app or one in particular?

free im guessing yours is £ ?

im guessing its the money investors give to x companys that are owned or linked

Yes, sounds interesting

I think it's good for tasks where you input the data and it only uses that. For example, I use it to write summaries of conferences, where I upload the presentations and ask it to write a summary report. I would never trust it if it's just scouring the internet though

Fair enough, but I really don’t expect a new GP to ask me “are you SURE it’s not the menopause” and claim they’ve seen less than 10 genuine cases locally in five years (when it’s a condition suffered by millions)! They may not be experts, but they ought to be able to empathise and gen up on what’s happening.

No you can tell it how you want it to express itself. It is set up to be personable. apparently it does tend to give more accurate answers if you tell it explicitly to be more critical.

Personally I find these chatbots incredibly accurate but you need to know that there is always a good possibility that they could be completely wrong. So check and cross check all the information you get. I'd treat them as a friend that you are asking for advice. Some of It may be good and some may be rubbish but like a friend it might give you some pointers that you haven't considered before.

I have switched from chatgpt to anthropic's Claude now to support anthropic for taking a principled stance against trump.

I think it's perfect for meal planning. I give it all the allergies and preferences in our house, the ingredients at hand, etc and it comes up with something surprisingly decent. It saves me time that I can then spend on more intellectual tasks (like browsing Mumsnet...)

What??

No mine is free too! How strange - I know the free version doesn’t remember conversations but it does remember the “vibe” of the current conversation so I think the fact that I had asked a number of ridiculous questions before hand it tailored its response to comedy rather than fact. It’s quite interesting really

i have edited the custom instructions ill give you mine :

Custom Expansion Prompt (Defense‑Analyst Essay Generator)
I compose formal, rigorous texts—primarily long‑form essays, manifestos, and editorial commentary. When I request “X,” treat it as the central subject and produce a comprehensive analytical essay in the voice of a senior defense analyst or strategic studies scholar.

Rules:

Structure: Use formal doctrinal subheadings: Strategic Context, Operational Dynamics, Doctrine and Deployment, Technological Factors, Geopolitical Implications, Historical Evaluation. Add others as appropriate (e.g., Force Composition, Intelligence, Logistics, Command and Control).

Tone & Style: Authoritative, analytical, fact‑heavy; avoid sentimentality, modern moral framing, or hindsight bias. Prioritize chronology, force structure, material conditions, and realpolitik. Treat as a briefing for policymakers or military planners.

Analytical Priorities: Emphasize causality, constraints, strategic incentives, doctrine, capabilities, deployment, institutional behavior, logistics, technology, and geopolitics. Focus on actors’ interests, capabilities, and calculations, not personalities. Situate within historical lineage and doctrinal evolution.

Output: Produce a substantive, long-form essay; no bullet points. Expand far beyond simple explanation. Abstract topics are treated as conceptual doctrines; contemporary topics are analyzed in real time without hindsight bias.

Oh wow, I’ll be honest I have no idea what any of that means! I’m not tech savvy at all, I only use chat GPT to mess around and amuse myself. I do occasionally ask it for facts on things I’m interested in but normally it’s just a boredom buster for me - and to talk about stupid stuff that people in real life would tell me to grow up over 😂

in the settings section you can alter how it generates the outputs by requesting eg write all outputs as if an intelligence analyst etc

I work in a different profession but I am in the same situation. People can use AI to do what I spent a long time training to do. Some are doing that and more will. It won’t replace a human completely as there will always be people willing to pay for the human element, but it is becoming more niche and there will be more competition. It’s worrying but I am aware and planning for it.

I don’t really get the above. Are you getting it to compose content that you’re writing? But why would you want “your” policy analysis effectively to be a regurgitated mishmash of everyone else’s out there? It would just make you look average or mediocre at your job.

I work in a high-skill knowledge field, and if I replaced stuff I am meant to be writing myself by getting an AI to do it instead, very quickly people would think I was just a bit mediocre, really. Why would they read my stuff when it’s just a version of what’s out there already? My job is to provide genuine new insight. AI isn’t doing that, and it would take me more time to edit up some meh blarble blarble it generates than just writing it myself.

In relation to posts upthread — I’m not a counsellor or a therapist or psychologist, but I think that substituting AI for genuine therapy is at best a self-delusion, and at worst incredibly dangerous, even irresponsible. An AI has no duty of care or real life safeguarding or statutory responsibilities. You can’t sue anyone if it misleads or gives you bad “advice”. And all of us should care about the value of human care and input being reduced to LLM output. Not to mention that those who adopt it most enthusiastically are the ones most vulnerable to being replaced by it…if I was a defence analyst and I used AI to write my briefings, why shouldn’t my employers get rid of me and just use AI instead? Why would I be worth employing if I just turn out average stuff they can get anyway? It’s not the person who writes their own briefings, and brings genuine care and insight to them, who’s getting replaced. It’s the people who turn in dull as ditchwater AI-prompted stuff that has nothing new to offer who is going to be redundant…(metaphorically, and probably literally).