Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

So, how did humans live without electricity in the past?

.

Massively wide range of expert opinion on the chance of AI killing us all.

But lots of serious informed people giving worryingly high numbers, including CEO of Anthropic, many major AI pioneers, AI safety researchers.

The really worrying thing is that the more people know about the technology the more they tend to take this risk seriously. So unless it's all group-think, we may be in serious trouble).

https://pauseai.info/pdoom

(This site is from a pressure group, but afaik the numbers are accurately reported).

I think the key question is whether AI can or has developed agency. Personally, I think that's even more central than the definition of intelligence.

Can you seriously not see the difference between a civilisation surviving before the discovery of electricity and one surviving after it, when the entire structure of modern society depends on it? From food production and water systems to banking, healthcare, communications, and virtually every other essential service.

Thanks that does sound concerning.

First off, a lot of AI engineers, no offense intended, are socially unsophisticated and given to fanciful speculations. I know a GENIUS semiconductor engineer who seriously believes the earth is flat. Or look at Elon Musk... ; ) Many wild ideas.

There's now a bit of an industry around creating fear about AI. Sam Altman himself does it sometimes, usually to influence legislation that favours him and harms competitors (IMHO). People in tech, weirdly, are terrible at predicting what will actually happen. I was very involved in a tech that is now 100% ubiquitous, but people in tech openly mocked it at first. What I think is that there will be profound unintended consequences of AI, just as there have been for GPS and smartphones, but rogue AI taking our humanity? That's just sci-fi.

I think you've misunderstood what agency is in an AI context. It is not an independent entity in the sense we think of ourselves with feelings and wants. It's the ability to execute multi-step tasks based on its environment and goals without constant human intervention. AI pretty much has that, in some contexts, today.

Pong evolving into games where AI characters capable of having conversation with you? Also Sci Fi in 1972. Another 50 years?…It doesn’t have to be ‘rogue’: “AI make as many paper clips as possible!”…If humans are in the way of a badly specified objective…oh dear…

It's like the techbros of today watched Terminator and saw it as a challenge.

Maybe think about why humans are as we are? We tend to justify our actions based on conceptions about who we are, and yet most of what we want and do is probably tied to hundreds of thousands of years of brain evolution and the very particular needs of walking bags of meat. AI is not that but we tend to anthropomorphise it. Because AI communicates with us using language our brains automatically apply human norms, emotions, and expectations to them, even when we logically know they are just code.

If it makes you feel better, most of us think of them as kooks, the same as the people who've been insisting for decades that we are about to be able to upload our consciousnesses to machines.

There is an undeniable psychological allure to believing you are Oppenheimer, working on the most dangerous, world-altering technology in human history.
Add to that, sounding the alarm on existential AI risk brings massive media attention, government funding for research, and political influence that routine safety work does not.

I honestly think it is time that we banned all computers, anything with a microprocessor and went back the old ways of adding machines and abacuses. It would create so many jobs and solve so many problems regarding modern life.

I need to find Futurama again 🥹.

Yes! Launch the Butlerian Jihad now!! lol

I get what you’re saying and also reading the link as below with the probability.

What seems to be missing is the why, why would AI make the leap to doom scenario in their view. What happens practically as a sequence

I know about agentic AI and how autonomous it can be; as I'm self-employed I'm very interested in AI agents and how much work they could take off my hands. But then, there is the broader agency/self-awareness question and that worries me more than determining how intelligent AI could become. I wonder how far we are from an AI entity considering itself a person and aiming for self-preservation. I don't think AI would ever have feelings like a human does for the simple reason that its own genesis and evolution would be so alien to ours (even if it was originally made up of human-generated data).

If you look closely at the doom argument, it relies on some massive logical leaps and psychological biases rather than actual engineering. Some of it is just barking mad. I read one of the most popular books on this topic last year, and it seriously had a section suggesting AI might be a plot by an ancient demon to fulfill and ancient apocalypse. FFS. It was written by people who otherwise seemed like they'd be smart.

Biggest reason they think this will happen is the omnipotence fallacy which assumes that being incredibly good at processing data automatically equals total physical control over the world. But software cannot magically manipulate power grids, supply chains, or human biology without the physical infrastructure that humans completely control.

Then the doom scenario requires the AI to be smart enough to outwit all of humanity, yet so incredibly stupid that it doesn't understand basic human context or nuance (like the classic thought experiment where an AI destroys the world just to manufacture paperclips, which some people still take seriously, like the idiots who wrote If anyone builds it, we all die, easily one of the worst books I've ever read).

The other major reason is projection. Humans have an evolutionary drive to compete, dominate, and survive. Otherwise smart researchers often project these biological flaws onto code, assuming a highly intelligent machine will naturally want power. They understand machines not biology, so don't realise why we are as we are. A computer program has no ego, hormone drive, or survival instinct unless we explicitly program it to. A lot or people who THINK about AI are not also coding it, or don't understand the wider dependencies. They fall into this basic gap in their knowledge.

The thing is you don't need agency or human consciousness to cause harm. Well before AI became a buzzword quite simple algorithms caused a lot of harm when, for example, they were boosting posts on social media that drove the most engagement. Posts that elicited the strongest negative emotional responses - fear, hate, anger were boosted the most and shown to the most people and the more people reacted the more they were shown. This led to social media addiction, increased social division, helped cause racial violence in places like Myanmar etc.

The whole "AI will want to make paperclips and kill us all in the process" argument is so dumb because you don't need "AI" for a computer/system programmed to prioritise one goal above all others to do a lot of harm. You don't even need computer code. A company/economic system with the goal of creating profit for shareholders above all else will do a lot of harm. What's harmful is giving systems/people/corporations/computers unlimited power and zero guardrails. They don't need to develop consciousness or independent thought to cause issues. (In fact the irony of the paperclip example is a computer programme that did develop conscious thought would presumably stop wanting to make paperclips.anyway). Yes, an AI agent left to its own devices for long enough and given unlimited power could cause trouble. So could a basic algorithm. But why the hell would anyone do that (other than that the people in charge are apparently irresponsible nuisances I wouldn't trust with a cigarette lighter.)

TL/DR it's not "AI" that the problem its the psychopaths apparently in charge of the tech industry.

Maybe the paperclip thought experiment provides the “why”? There’s no AI thinking as a human; we’re eliminated as we get in the way of an objective.

Yes, I always thought an actually conscious "AGI" with independent thoughts would be just as likely to retreat into itself and solve maths problems and not share the results or just do something incomprehensible all by itself. Its incredibly arrogant to assume it would be interested in humans or the same stuff we are interested in. But I think the people pushing the idea of AGI want a machine that gets consciousness and tells them how clever they are. Or that mimics the sci-fi stories about AGI in the way LLMs currently mimic Turings definition of what an "Artificial Intelligence" would do (although he qualified this). But then you still have a bunch of people who want to prove they can create AGI and whose idea of "proof" would be the Terminator. Someone creating a fake dragon that still sets fire to everything is still a concern. But the danger there is the maniac with the flamethrower in a dragon suit. Not the existence of dragons.

Interesting. I admit that despite liking to think about stuff AI makes me feel a bit clunky, a bit slower. So now I imagine what it’s like to be very clever and surrounded by AI at work and the drive to succeed etc and maybe the catch isn’t AI but humans not being great at rationalising it.

But not sure, just musing really.

As long as when it wipes us all out it does it quickly. Not a horrible slow death due to lack of water or something. In that scenario it would be wise to just push the button, (if AI hasn't already) Joking (sort of) -terrifying😳

But... That's also the way unchecked capitalism works (e.g. if it is better for Bhopal profits to not dispose of that big pile of chemical waste then Bhopal has a responsibility to it's shareholders not to do so). It's also how excessive beurocracies work (computer says no).and how all sorts of other systems operating under their own narrow logic can hurt people. The paper clip story is a good example of how any simple computer programme could cause chaos if it was left to control everything. I think it originated as a cautionary tale about robots/automating everything. There's nothing unique about "AI" it "AGI" that would make an out of control paperclip scenario more likely than a really basic "make paperclip" set of instructions.

Following.

I think I'm being vaguely reassured on here ...