Why do people treat ChatGPT like it’s the Delphic Oracle?

One police force definitely did this - can't remember which one though

I have heard of a junior barrister who used it to find case law that would support her legal argument. Great - except she didn’t check that the cases actually existed. I heard that the judge was pretty cross (this is a third or fourth hand story though).

I end up questioning it's hallucinations and mistakes and half our interaction is it 'apologising' to me for talking what I perceive to be bollocks.

I asked it recently if a band I like is playing Ireland this year, as they announced a tour in the UK. It confidently told me that they are playing the 3 Arena on the 26th May. They are not. They are not coming to Ireland at all. Nobody is playing the 3 Arena that night. That's the worst example of AI making shit up that I've seen.

YANBU. Some people are treating it like it's their best friend, and it isn't 100% accurate and can be very dangerous. I told a friend to stop using it when she kept spouting it all the time. What she was saying was wrong.

It can be out of date, with inaccurate facts. It doesn't have feelings or understand feelings. It can not replace a person to talk to.

It just lied to me about who the current Mayor of Birmingham is

Very disturbing that women might be using it for advice after sexual assault and its given "answers" gleaned from rapists and wannabe rapists.

I never use it but the other day I was stuck on a crossword so googled a 5 letter synonym for a certain word. As i scrolled past I noticed the google AI bit at the top confidently offered me a 6 letter word, calling it "a 5 letter synonym for [x]. It can't even count letters!

Agreed. It's appalling - and frightening.

Disturbing that some here are defending the misogynistic, victim-blaming, rape apologist robot!

... OK, these bots don't 'think', they have no values or opinions, they don't 'feel'. But they say they do. They themselves believe they do. They talk like they do. and, as we see, people believe and trust them. The potential outcomes of this are horrible.

It struggles to tell you you’re wrong and instead loves to big you up and tell you how amazing you are. I guess people enjoy that. It is useful for some things but also useless at others.

I went to London for the weekend and haven’t been for a couple of years so asked it to write me out an itinerary including tube routes. It gave me the wrong route so I got a bit lost and had to figure my own way back. Just stuck to google from there on out.

People say how great it is so you try it, but the results can be very disappointing. If it's a subject you actually know a lot about it is very often wrong or very out of date, it then agrees with you if you correct it, but that is hardly helpful! And it is so positive it's right until you challenge it and then it's equally positive of the opposite. It's fine for simple established facts like dates in history, simple mathematics or grammar, etc. People just need to be aware of the limitations.

That's horrifying. I don't use any kind of AI and I don't think I ever will
Especially after reading that. Bloody Hell!

But this puts the lie to "It's so empathetic! It understands me better than a counsellor, I can tell it anything!" You have here a woman who, by the sound of things, was in desperate need of empathy and the kind of listening ear a human counsellor would give. The computer said no.

That's why. You hit a guard rail, because you told it you are in danger and wouldn't do anything about it. So if shut it down.

ChatGPT offers a more sensible, adult conversation than most human adults

I asked it questions about some financial investments, it was mostly wrong. I pointed out it's mistakes and it agreed with me with equal confidence as it had given me the wrong (outdated and misleading) information. It 'knew' less than I did! Don't use it for anything important or that you can't check independently.

@AmandaBrotzman
Just so I’m being clear - I’m not saying it DOESN’T give relationship/marriage advice btw. I know it does. It took the particular turn it did when I said I wouldn’t report the current situation I’m in, couldn’t/wont discuss with my husband - I’d expect it to do what it otherwise does (be empathic etc) but it told me it couldn’t discuss long-term relationships and went back to telling me what I have to do to prevent boundary violations of a physical type.

It’s proved to be way more knowledgeable about my medical condition than my GP which was extremely concerning to me. When I had a consultant appointment they echoed the Chat stuff.

OMG, I just tried the car wash scenario. It did indeed tell me to walk to the car wash. When I pointed out that wasn't going to get the car clean it vomited this nonsense out

"Think of it like watering a plant: the car wash is the water, walking is the gentle care that helps it thrive without waste."

That is why we can never trust a word it says.

@AmandaBrotzman
that is exactly what it said. It seems insane to me to say it can’t discuss long term relationships.
That wasn’t, as you put it, part of a prompt from it.
It went a bit like this..
I am talking about the current problem, challenging its terminology, it repeatedly returns to advice on boundaries .. I say that I cannot discuss this matter with my husband ( I can’t, he’s a cunt to me but I literally don’t want to discuss him here) and it replied saying it couldn’t comment on long term relationships..
I will actually try to find the piece and cut and paste for you..
Though right now I have to get ready for an important meeting but I’ll come back when I find it.

Yes x100. And all that leads to knowledge entropy, where the bots scour available information ... which is increasingly published by bots, themselves having 'averaged' information that was published by other bots.

So as our own capacity for thought atrophies, the services we expect to think for us provide ever poorer solutions.

This is the way the world ends
This is the way the world ends
This is the way the world ends
Not with a bang but a whimper.

I did ask it a relationship issue recently as it was quite niche (re a friend not dh) and it did ‘help’ as it was quite hard to find advice by googling. Got obvious reasons I could t talk to friends about it and dh is useless with this sort of thing - he’s sympathetic and ‘got it’ but couldn’t advise further. He loves Chat though, so …

It told you it can't comment on long lasting relationships? In what context? Mine happily comments on my marriage when I ask it.
I don't know what prompts and questions you were using but this is millions of miles away from my experience.

I refuse to use AI tools in my own private life as I can see their glaring limitations and it’s scary how many people are uncritically handing over control of their lives to them.

However, at work these tools are being heavily pushed and I fear it won’t be long before I am required to use them. Real knowledge and critical thinking will be lost

I’ve recently decided to challenge it on so called advice - the type of counselling type questions.
I have a real life counsellor and my husband is a counsellor., so this is a purely inquisitive asking.
If I took ChatGPT’s advice or assessments seriously I would be very messed up by what it produced. Possibly even suicidal.

I used the examples of a current issue I have and an historic sexual assault (I’m now over it pretty much and I went all the way to civil court on that one).

It described the assault as “flirtation adjacent” behaviour.. suggesting a step down even from mild flirtation. I let it go on, asked it where it got that phrasing (it said Reddit and X). I asked what kinds of posts on those sites - were they men excusing their behaviour, looking for advice or essentially grooming advice. It replied that yes there are chats for lack of a better word with (and this was a very specific question I’d asked) men who advise on how to inappropriately touch women, why they do it, how they feel etc.

I asked it if it would call the behaviour sexual assault.
It said not to ascribe motivation to the behaviour, advised me not to report or accuse, said it was flirtation adjacent.

i then told it uk law doesn’t require proof of motivation in sexual assault. It agrees but still refused to describe the behaviour as assault.
I then told it that I reported it to the police and also the judge had no problem calling it sexual assault (I didn’t win my civil case but not on this basis).

It advised me - without asking - on ways to prevent further boundary violations, started to call the perpetrator a “boundary violator”, told me most physical boundary violations weren’t sexual but were about ego boosts…

it summarised it’s “advice” in tables and set out interaction by interaction which was a “flirt adjacent behaviour “, which was a boundary violation, used the term red flags a lot but not once in all of it’s summary did it “voluntarily” use the words sexual assault or any term that could be construed as equivalent.

When I then told it of my current problem it quite literally blamed me. I told it this is misogynistic victim-blaming shit and asked where it was deriving this spectacular advice from- yes of course Reddit sub forums where male healthcare workers discuss assaulting patients and parents and how to use plausible deniability. I asked it to link me to the subgroups and it refused saying it could not do that ethically in case the (anonymous) men were identified and accused..

That is what I am terrified of with people using it for any kind of mental health or relationship issues. It also told me it cannot comment on long lasting relationships by the way…,