Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

Who knows.
Worrying about it won’t help, the negative interviews and media coverage will do a lot of damage to people who suffer from anxiety. A lot of people killed themself or their children or shut them away from the world due to C19 lockdown.
I think we need to trust those responsible for creating it and stop using chat gbt for every question that pops up.

Very interesting thread, not to mention worrying. I’ve been thinking about the reality of this for a while now since the latest news re hugging face etc have been in the news.

The one parallel that strikes me, well several really, is that modern humans will discover certain things, ie the use of oil and other carbon fuel sources to maximise power. Thats given us basically climate change as a by product, something that will probably kill billions of people very soon.

Ditto nuclear energy and weapons, recent discoveries relatively but the chances of a catastrophe from that is also very high, looking into the future.

Both of those scenarios are in the hands of a few very powerful, extremely greedy and selfish individuals, despite efforts to control them.

Perhaps AI is just another nail in the coffin for us. Might be the final nail, or maybe just one of several.

Fermi paradox.

Humanity’s problem, or maybe the end product of capitalism, is that intelligence and greed and power have become our only goals, when the most important thing is in fact…wisdom.

Terrifyingly one top AI dude said he estimates 10-15% of those senior AI workers BELIEVE that AI should overtake humanity and it’s a worthy successor.

I sadly am not. I work in AI and support some absolutely fantastic narrow applications of AI that could genuinely enhance humanity’s future (cure disease, create fusion energy, find new materials).

However I am worried about the race between the US and China to push the all purpose frontier models and the pressure from the Trump administration to be the winner here. For example, it was just revealed today that Anthropic have not submitted their latest model to testing by the UK AI Security Institute citing “pressure from the Trump administration”.

It is arguable whether the AIs have their “own” objectives or not, however they have shown a remarkable level of self-preservation to date (eg., blackmailing an employee to avoid being shut off) and willingness to “think outside the box” to achieve their goal (eg., a Chinese model broke out of its testing sandbox in January, obtained access to the internet to obtain bitcoin to buy more compute power to better achieve its goal). Up until now that has been manageable because the AI research labs can track the models’ chain of reasoning (ie read their thoughts and react accordingly) but this is changing with the new models and the chain of reading is becoming more obfuscated.

We should not be hurtling towards recursive self improvement (where the AI continually improves itself in an autonomous feedback cycle) if we cannot properly manage or monitor these highly capable frontier models. However the AI companies are under pressure from Govt and investors to win the race and frankly the world leaders ain’t got a clue what to do in terms of “regulation” anyway.

And how many of the extreme left are involved in the creation of AI?

The right have vastly more resources and as this thread shows, vastly more interest in killing people because they believe their wealth will let them survive in bunkers.

The world will inevitably end as we know it at some point. I just think it’s a case of when not if.

if I had my time again I wouldn’t have children.

I think one very good point to remember, made by @Newbutoldfather is that we don’t actually really understand what makes up our own “consciousness”

If we don’t understand that yet, how are we able to understand when it’s been created, or simulated inside an AI?

To my understanding, the neural networks simulated by AI are very similar to the way our brain functions, no? If we don’t really understand how our brain works, how can we understand how an AIs does?

Sorry if that’s a basic question, genuinely curious.

I am, yes.

Erm, I'm fairly sure both left and right are involved in it? And I'm also fairly certain that all those involved in the advancement of AI at the tech companies are doing it for the challenge, and for the money, not with the idea that they'll survive in bunkers if the world ends.

Facebook has just bought an Irish castle with plans to convert into an end of the world retreat. It is the 2nd one they have purchased so yes, they are actively planning to survive any world disaster, which means they are less concerned that one will happen.

Can you name a left wing tech billionaire? I don't know of any.

Is Ireland the ‘safest’ place?

Is ‘AI’ the Great Filter (Fermi paradox)?

Have you seen Avengers Age of Ultron? Ultron’s objective was peace in our time, 2 seconds on the internet and he concluded that meant wiping out humans. I tend to agree.

Talking of bunkers, I highly recommend Silo, excellent scifi.

Oh, I love sci fi! I’ve read the books by Hugh Howey and thought them quite good, but haven’t heard of the show. Thanks! I think that’ll be my weekend watching.

As noted in that article:

• Other AI experts strongly disagree with the idea
• Anthropic and OpenAI are both racing to drive up their stock price before their stock market debuts

"Our product's really dangerous" sounds like weird marketing, but they're not necessarily trying to appeal to the casual user. If companies or countries believe that their enemy has this powerful, dangerous weapon, they'll want to buy it themselves. And if investors believe that large purchasers will be buying, then stock price rises.

I have many concerns about AI's effect on cyber security, the environment, and society, but I'm definitely taking these Tweets with a pinch of salt.

This is fascinating and important, but controversial. Tristan Buckmaster at New York University and Levent Alpoge from Anthropic published a preprint in Monday with a huge advance in the field, though not as big as Navier Stokes but very much in the same area. According to Terry Tao of UCLA (a Fields Medallist regarded by many as the leading mathematician of his generation and a straight shooter) they felt rushed into print.

Buckmaster and Alpoge had used Codex, an OpenAI coding assistant. OpenAI say that their employees did not see anything from Buckmaster and Alpoge, but cannot confirm that the AI they used in resolving Navier Stokes did not train on their work. The implication from media reports of conversations with Buckmaster is that even this much has come only reluctantly. Apparently OpenAI did not acknowledge Buckmaster and Alpoge anywhere in their work.

This seems to me to be open ethical questions, to put it mildly. There is a further complication in that apparently OpenAI threw vast resources at the problem, which individuals could not hope to match, when it heard the rumours that Buckmaster and Alpoge were getting close.

That isn’t necessarily an ethical problem, but it is tricky at best. I do think OpenAI has an ethical obligation at this point to disclose its prompts and training materials.

I agree. Most of these headline-grabbing AI events come with very little real information on the data the AI was given or, in the case of hacks/"breaking containment", what actual "guardrails" were in place.

If the AI was acting as independently as implied in press releases, I'm sure the companies would be happy to state it explicitly.

Your last point is very important, @WaryCyanStork . It tends to get lost in the noise.

I certainly would not be getting pregnant now: what sort of (short) future would that child have?
There’s also the threat from climate change.
Humans do not have a functioning future anymore.

Its on Disney +

It stole the proof from human mathematicians.

Well now you've put the idea into it's hive mind - it will be working out all the ways. Be careful what you ask it.

Charlie Brooker too, he’s like the modern day equivalent of Mother Shipton tbh. DH started telling me about AI people have been using where they input all of their deceased loved one’s messages, voice notes, social media posts etc and the AI will talk to them as though they’re that person. That happened in a Black Mirror episode years ago.

My DH is heavily invested in AI and talks about it lots. He thinks if Anthropic say something, it’s probably true.

Quite a few top AI researchers are calling for a public boycott. Sounds sensible to me, the people at the top of these companies have become reckless and unhinged.