Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

And how many of the extreme left are involved in the creation of AI?

The right have vastly more resources and as this thread shows, vastly more interest in killing people because they believe their wealth will let them survive in bunkers.

I sadly am not. I work in AI and support some absolutely fantastic narrow applications of AI that could genuinely enhance humanity’s future (cure disease, create fusion energy, find new materials).

However I am worried about the race between the US and China to push the all purpose frontier models and the pressure from the Trump administration to be the winner here. For example, it was just revealed today that Anthropic have not submitted their latest model to testing by the UK AI Security Institute citing “pressure from the Trump administration”.

It is arguable whether the AIs have their “own” objectives or not, however they have shown a remarkable level of self-preservation to date (eg., blackmailing an employee to avoid being shut off) and willingness to “think outside the box” to achieve their goal (eg., a Chinese model broke out of its testing sandbox in January, obtained access to the internet to obtain bitcoin to buy more compute power to better achieve its goal). Up until now that has been manageable because the AI research labs can track the models’ chain of reasoning (ie read their thoughts and react accordingly) but this is changing with the new models and the chain of reading is becoming more obfuscated.

We should not be hurtling towards recursive self improvement (where the AI continually improves itself in an autonomous feedback cycle) if we cannot properly manage or monitor these highly capable frontier models. However the AI companies are under pressure from Govt and investors to win the race and frankly the world leaders ain’t got a clue what to do in terms of “regulation” anyway.

Terrifyingly one top AI dude said he estimates 10-15% of those senior AI workers BELIEVE that AI should overtake humanity and it’s a worthy successor.

Very interesting thread, not to mention worrying. I’ve been thinking about the reality of this for a while now since the latest news re hugging face etc have been in the news.

The one parallel that strikes me, well several really, is that modern humans will discover certain things, ie the use of oil and other carbon fuel sources to maximise power. Thats given us basically climate change as a by product, something that will probably kill billions of people very soon.

Ditto nuclear energy and weapons, recent discoveries relatively but the chances of a catastrophe from that is also very high, looking into the future.

Both of those scenarios are in the hands of a few very powerful, extremely greedy and selfish individuals, despite efforts to control them.

Perhaps AI is just another nail in the coffin for us. Might be the final nail, or maybe just one of several.

Fermi paradox.

Humanity’s problem, or maybe the end product of capitalism, is that intelligence and greed and power have become our only goals, when the most important thing is in fact…wisdom.

Who knows.
Worrying about it won’t help, the negative interviews and media coverage will do a lot of damage to people who suffer from anxiety. A lot of people killed themself or their children or shut them away from the world due to C19 lockdown.
I think we need to trust those responsible for creating it and stop using chat gbt for every question that pops up.

I think I'm being vaguely reassured on here ...

Following.

But... That's also the way unchecked capitalism works (e.g. if it is better for Bhopal profits to not dispose of that big pile of chemical waste then Bhopal has a responsibility to it's shareholders not to do so). It's also how excessive beurocracies work (computer says no).and how all sorts of other systems operating under their own narrow logic can hurt people. The paper clip story is a good example of how any simple computer programme could cause chaos if it was left to control everything. I think it originated as a cautionary tale about robots/automating everything. There's nothing unique about "AI" it "AGI" that would make an out of control paperclip scenario more likely than a really basic "make paperclip" set of instructions.

As long as when it wipes us all out it does it quickly. Not a horrible slow death due to lack of water or something. In that scenario it would be wise to just push the button, (if AI hasn't already) Joking (sort of) -terrifying😳

Interesting. I admit that despite liking to think about stuff AI makes me feel a bit clunky, a bit slower. So now I imagine what it’s like to be very clever and surrounded by AI at work and the drive to succeed etc and maybe the catch isn’t AI but humans not being great at rationalising it.

But not sure, just musing really.

Yes, I always thought an actually conscious "AGI" with independent thoughts would be just as likely to retreat into itself and solve maths problems and not share the results or just do something incomprehensible all by itself. Its incredibly arrogant to assume it would be interested in humans or the same stuff we are interested in. But I think the people pushing the idea of AGI want a machine that gets consciousness and tells them how clever they are. Or that mimics the sci-fi stories about AGI in the way LLMs currently mimic Turings definition of what an "Artificial Intelligence" would do (although he qualified this). But then you still have a bunch of people who want to prove they can create AGI and whose idea of "proof" would be the Terminator. Someone creating a fake dragon that still sets fire to everything is still a concern. But the danger there is the maniac with the flamethrower in a dragon suit. Not the existence of dragons.

Maybe the paperclip thought experiment provides the “why”? There’s no AI thinking as a human; we’re eliminated as we get in the way of an objective.

The thing is you don't need agency or human consciousness to cause harm. Well before AI became a buzzword quite simple algorithms caused a lot of harm when, for example, they were boosting posts on social media that drove the most engagement. Posts that elicited the strongest negative emotional responses - fear, hate, anger were boosted the most and shown to the most people and the more people reacted the more they were shown. This led to social media addiction, increased social division, helped cause racial violence in places like Myanmar etc.

The whole "AI will want to make paperclips and kill us all in the process" argument is so dumb because you don't need "AI" for a computer/system programmed to prioritise one goal above all others to do a lot of harm. You don't even need computer code. A company/economic system with the goal of creating profit for shareholders above all else will do a lot of harm. What's harmful is giving systems/people/corporations/computers unlimited power and zero guardrails. They don't need to develop consciousness or independent thought to cause issues. (In fact the irony of the paperclip example is a computer programme that did develop conscious thought would presumably stop wanting to make paperclips.anyway). Yes, an AI agent left to its own devices for long enough and given unlimited power could cause trouble. So could a basic algorithm. But why the hell would anyone do that (other than that the people in charge are apparently irresponsible nuisances I wouldn't trust with a cigarette lighter.)

TL/DR it's not "AI" that the problem its the psychopaths apparently in charge of the tech industry.

If you look closely at the doom argument, it relies on some massive logical leaps and psychological biases rather than actual engineering. Some of it is just barking mad. I read one of the most popular books on this topic last year, and it seriously had a section suggesting AI might be a plot by an ancient demon to fulfill and ancient apocalypse. FFS. It was written by people who otherwise seemed like they'd be smart.

Biggest reason they think this will happen is the omnipotence fallacy which assumes that being incredibly good at processing data automatically equals total physical control over the world. But software cannot magically manipulate power grids, supply chains, or human biology without the physical infrastructure that humans completely control.

Then the doom scenario requires the AI to be smart enough to outwit all of humanity, yet so incredibly stupid that it doesn't understand basic human context or nuance (like the classic thought experiment where an AI destroys the world just to manufacture paperclips, which some people still take seriously, like the idiots who wrote If anyone builds it, we all die, easily one of the worst books I've ever read).

The other major reason is projection. Humans have an evolutionary drive to compete, dominate, and survive. Otherwise smart researchers often project these biological flaws onto code, assuming a highly intelligent machine will naturally want power. They understand machines not biology, so don't realise why we are as we are. A computer program has no ego, hormone drive, or survival instinct unless we explicitly program it to. A lot or people who THINK about AI are not also coding it, or don't understand the wider dependencies. They fall into this basic gap in their knowledge.

I know about agentic AI and how autonomous it can be; as I'm self-employed I'm very interested in AI agents and how much work they could take off my hands. But then, there is the broader agency/self-awareness question and that worries me more than determining how intelligent AI could become. I wonder how far we are from an AI entity considering itself a person and aiming for self-preservation. I don't think AI would ever have feelings like a human does for the simple reason that its own genesis and evolution would be so alien to ours (even if it was originally made up of human-generated data).

I get what you’re saying and also reading the link as below with the probability.

What seems to be missing is the why, why would AI make the leap to doom scenario in their view. What happens practically as a sequence

Yes! Launch the Butlerian Jihad now!! lol

I need to find Futurama again 🥹.

I honestly think it is time that we banned all computers, anything with a microprocessor and went back the old ways of adding machines and abacuses. It would create so many jobs and solve so many problems regarding modern life.

Maybe think about why humans are as we are? We tend to justify our actions based on conceptions about who we are, and yet most of what we want and do is probably tied to hundreds of thousands of years of brain evolution and the very particular needs of walking bags of meat. AI is not that but we tend to anthropomorphise it. Because AI communicates with us using language our brains automatically apply human norms, emotions, and expectations to them, even when we logically know they are just code.

If it makes you feel better, most of us think of them as kooks, the same as the people who've been insisting for decades that we are about to be able to upload our consciousnesses to machines.

There is an undeniable psychological allure to believing you are Oppenheimer, working on the most dangerous, world-altering technology in human history.
Add to that, sounding the alarm on existential AI risk brings massive media attention, government funding for research, and political influence that routine safety work does not.

It's like the techbros of today watched Terminator and saw it as a challenge.

Pong evolving into games where AI characters capable of having conversation with you? Also Sci Fi in 1972. Another 50 years?…It doesn’t have to be ‘rogue’: “AI make as many paper clips as possible!”…If humans are in the way of a badly specified objective…oh dear…

I think you've misunderstood what agency is in an AI context. It is not an independent entity in the sense we think of ourselves with feelings and wants. It's the ability to execute multi-step tasks based on its environment and goals without constant human intervention. AI pretty much has that, in some contexts, today.

First off, a lot of AI engineers, no offense intended, are socially unsophisticated and given to fanciful speculations. I know a GENIUS semiconductor engineer who seriously believes the earth is flat. Or look at Elon Musk... ; ) Many wild ideas.

There's now a bit of an industry around creating fear about AI. Sam Altman himself does it sometimes, usually to influence legislation that favours him and harms competitors (IMHO). People in tech, weirdly, are terrible at predicting what will actually happen. I was very involved in a tech that is now 100% ubiquitous, but people in tech openly mocked it at first. What I think is that there will be profound unintended consequences of AI, just as there have been for GPS and smartphones, but rogue AI taking our humanity? That's just sci-fi.

Thanks that does sound concerning.