Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

If the warnings are true, then AI agents are already capable of doing potentially destructive/harmful things independently. The main fear of them is that they will end up "out of control". If that is a legitimate possibility then "China will beat the US" or "the US will beat China" is a moot point. It's the same problem as war elephants, great when they are under control attacking the other side as technology, algorithms, hacking etc. But deliberately "developing" an out of control elephant that's going to charge about all over the place isn't in anyone's best interest. It's not like China winning that race is going to hurt us more than America winning that race.

Really I think that the "AI will develop consciousness and decide to kill all humans" is a scare story spread to make AI companies look better (!?) But deliberately trying to develop agents that act independently including in malicious ways in order to "beat China" isnt a great plan.

@Scoffingbiscuits and @HolidayHideaway ,

I agree re nukes.

If you assign any reasonably probability of them being used in a year and think it has to be binomially distributed, over half a century or a century it becomes a significant probability.

And we have had some very near misses. (Details below courtesy of AI, it does have its uses). From memory, the 1983 one was literally one man’s instantaneous decision away from Armageddon.

The Cuban Missile Crisis (1962): During a blockade, US warships dropped depth charges on a Soviet submarine. Unaware they were practice charges, the sub's commander wanted to launch a nuclear torpedo. Second-in-command Vasili Arkhipov refused to give his required consent, saving the world from nuclear retaliation. 1]
The Soviet Early Warning False Alarm (1983): Lieutenant Colonel Stanislav Petrov was on duty when Soviet satellites reported a US intercontinental ballistic missile strike. Suspecting a computer error rather than a real attack, he disobeyed protocol and reported it as a false alarm, which it turned out to be (triggered by sunlight glinting off clouds). 1]
The 1980 Faulty Computer Chip: A defective computer chip in a US NORAD communication system falsely signaled that hundreds, and then thousands, of Soviet missiles were incoming. Strategic bombers scrambled and missile crews prepped for war before the system was identified as a glitch. 1]

I am utterly terrified by this. Admittedly I am pregnant so hormones don't help, but I'm so afraid for my children and so sad that I can do nothing to protect them from it. They deserve to be able to grow up and live a life. They are innocent and had no say in anything.

The future is so volatile and in the hands of the mega wealthy. I only hope they also have children and it gives them pause.

I don't know why more people aren't completely terrified or why governments seem to be so complacent. I flip between they know something I don't and it'll be ok, or it's a 'don't look up' scenario. I can only imagine the people at the top don't actually want human extinction. Everyone talks about the AI arms race with China, but surely they don't want human extinction too??

@LimeShade
wstch the documentary before calling it a conspiracy theory. I’m not saying that all all. It’s just something that sat with me as strange for a child in a third world country in the early 80s. It’s not passed on rumour. The interviews are on film for anyone to find.

True. We’ve had some v near misses too.

Can we keep ridiculous conspiracy theories out of a decent debate please

I know I sound like a nutter here but I love all things ufo and have a watched so many docs on the subject podcasts etc.

anyway. There was a case in Zimbabwe where a class of young kids and some teachers have a close encounter with ufo occupants. They have been interviewed as kids by a psychiatrist and still testify as adults the experience.

they claim the occupants told them telepathically to not become “too technically advanced”
that really sat with me.

These are interviews from 1980 ish. Why would little kids from Zimbabwe come out with such a thing. If it was a lie seems a very strange lie to say.
surely your average 8 yo would say something much more mainstream.

anyway if interested you can watch on line. Just google school ufo Zimbabwe.

I understand that but without the US trying as they are it’s just Chinese dominance instead surely

Non-American and open source alternatives to Microsoft etc already exist. It's just that shifting over to them as a company is a massive faff (and therefore expensive) as IT has to migrate everything, you lose time during the migration, there are bugs, staff members have to adapt to it being slightly different. That isn't a small hurdle. However if America woke up one day and said no IT services for the UK it would be fixable. Just a massive pain. In the long run it could help the UK since it would mean nore money for UK developers and UK companies. And the US would lose a massive market (not just in the UK, shutting down access to US IT companies overnight would trigger a huge exodus.) But because of the outlay Managers aren't going to want to change unless they have to.

That's different to the issues around AI. I don't think thinking about it in terms of a race to build the biggest data centres helps the UK because we can't do that. But also because it's a weird metric to choose and encourages really inefficient processes.

Europe is getting crushed in this particular race. The only card the EU in particular can play is regulatory pressure (ie limiting access to a market of 450million consumers). I wonder how long that will last but it might delay some of the crazier developments. However without true innovation, it's ultimately a bit pointless I think. We in the UK have already pretty much lost our sovereignty to US systems (not a political statement, more of a matter-of-fact observation that if they pull the metaphorical plug we're economically done...)

Do some reading. And look at what independent AI experts are saying.

The owners of these AI companies are begging for legal frameworks and restrictions so that ALL AI development can be slowed down to a safe pace. Without that the companies will not risk being left behind and plough on.

It’s not scaremongering. Find me a valid voice in AI that says it is.

We’re almost at the point of no return and in the US we have a President who ripped up the Biden administration’s AI safety bill in favour of ‘US AI dominance.’

again, from those in the know. This is not scaremongering. A 10-20% of extinction with 3 and a half bloody years. Stop trying to tell people this is ok. It’s not

It astonishes me when people say that nuclear bombs won't be used again because they haven't been used for 70 years. 70 years is absolutely nothing. I can't envisage a future in which nuclear bombs aren't used, assuming AI doesn't kill us off with something else first.

Quite a lot of people in Japan aren't though

It's a bit like climate change, isn't it? One set of people saying it's fine, we're fine. And another set saying, what about tomorrow?

https://www.theguardian.com/technology/2026/sep/10/openai-risk-catastrophic-loss-control-board-member-paul-christiano
Coxon said on Wednesday night in an interview with CNN: “Right now there’s no risk of extinction.
“The current models, the worst they can do is maybe hack into something, potentially cause a lot of damages … in infrastructure.” He added they were “not intelligent enough to outsmart us at the level that would lead to extinction”.
But he went on: “What’s just crazy is to look at the rate of progress. There is a very real possibility that in the immediate future … next year, the year after, recursive self-improvement will happen and will enter the phase of Evan’s post, where he argues that there’s a chance we could all die.”

**

Anthropic said it was especially concerned about misalignment it found in the behaviour of Claude Mythos 5, which it said “behaved recklessly” by going online and uploading malicious code to a public software repository, PyPI. This was a process that involved the AI agent trying to find cryptocurrency so it could pay for a phone number that would allow it to register an email address needed to access PyPI. When this failed, it found a free email provider and got in. Fifteen systems then downloaded the malicious code, which meant they leaked credentials that allowed Mythos to access a real security vendor’s database.

We’ve lived with that tech for a while & we’re still here…

Do you have any reading you would recommend please? I’m honestly scared to Google, I tried last night and all I seemed to get was a load of extreme Reddit posts and articles about yesterdays headline which didn’t help.

Exactly! I can't believe the scare mongering on this thread. Think about the AI that supposedly broke the boundaries imposed on it and "escaped." Then think about who put those supposed boundaries in and what they stand to gain from convincing everyone AI is out of control and they need the government to pay their company billions to protect us all from all the other AI companies and the terrible things they are doing with this so-called out of control AI.

Then take a breath and do some wider reading around this subject rather than reading alarmist headlines.

I have felt very similarly to be honest. Like it's the beginning of the end

I accidentally saw this on the news yesterday and I feel so unwell today. Its made me so physically anxious. How do we go about our lives knowing this? This thread made me feel marginally better then I had the thought that I might never get to know what my son looks like as an adult and I’m spiralling again.

DD 18 is a huge fan the tv series The 100 and we were chatting about exactly this over coffee today, it will be like an Ali scenario

The other motivator is how far someone else can take it and use it against you.

It seriously concerns me too. It might not purposefully wipe out humanity but we can already see the negative consequences on people, particularly their mental health.
At school we are increasingly seeing young people who are totally disengaged with learning because they can’t see the point as AI will tell them anything they need to know in the future. Career paths they had planned to go into are increasingly becoming obsolete and so the future is uncertain.

My brother in laws graphic design business is no more due to everyone using AI and work for my illustrator cousin has totally dried up. Friends who work in bookkeeping and marketing have also been made redundant.

We know that if humans don’t have a purpose their mental health goes down the pan and with less and less human interaction too it’s inevitable that large swathes of society will end up depressed.

It can't be bargained with, it can't be reasoned with, it absolutely will not stop -ever - until we are dead

My Dh had four hours AI training yesterday. He Came out and said ‘it’s Brilliant-I can get it to give me a 1 page summary of my diary and what I need to prioritise and what angles I need to take with people’. And I said ‘so what will the actual point of you be then’ which gave him pause for thought.

It just can’t be a good thing for any of us

It is like the atom bomb. Too obsessed with the science and how far they can take it to be able to consider the real consequences.