Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

Can you seriously not see the difference between a civilisation surviving before the discovery of electricity and one surviving after it, when the entire structure of modern society depends on it? From food production and water systems to banking, healthcare, communications, and virtually every other essential service.

I think the key question is whether AI can or has developed agency. Personally, I think that's even more central than the definition of intelligence.

Massively wide range of expert opinion on the chance of AI killing us all.

But lots of serious informed people giving worryingly high numbers, including CEO of Anthropic, many major AI pioneers, AI safety researchers.

The really worrying thing is that the more people know about the technology the more they tend to take this risk seriously. So unless it's all group-think, we may be in serious trouble).

https://pauseai.info/pdoom

(This site is from a pressure group, but afaik the numbers are accurately reported).

.

So, how did humans live without electricity in the past?

The business model of most tech billionaires has been

"What if taxi company but not have to pay taxi license fees or follow the rules for taxi companies?" Uber is born

"What if massive hotel/BandB company but not have to follow same rules or have same expenses?" AirBandB is born

"What if delivery company but not have to follow employment law for delivery drivers?" Deliveroo/Uber eats etc is born

"What if we create virus/hacking attack but not responsible for it?" Shitty AI is left to do whatever it likes for days and hacks into a company.

Some of the technology around AI is very cool. But it's always been possible to write hostile code or just accidentally write really bad code that then infects other computers. It's just now they can let a machine do it themselves and instead of being viewed as annoying destructive little pests somehow evade responsibility and put their hands up in amazement at what their product did. Their product was shitty. It failed to solve an escape room task set it and did something different and potentially dangerous instead. If it wasn't AI we would be saying it was shit and blaming the creators. The same way that when computers used to make errors we called them errors not hallucinations.

I get what you’re saying. I also think AI could be used by humans if they wanted to harm others,

But where does this 10 - 20% thing the op outlines come from, does it have any credence?

Lighten up. It was a joke! I bet you're a right laugh at parties.

@dh280125 ,

‘But the idea that AI has it's own objectives or would at some point decide to kill us all off it sci-fi rubbish. None of it required AGI, consciousness, hatred of humans or a survival instinct. People misundersand AI. AI doesn't have evolution driving it, no reptile brain pushing it to survive at all costs’

I think that is interesting and you make some fair points, but it does kind of evolve already with autonomous coding agents and recursive self improvement (must admit I needed to look those up but, conceptually, they are pretty simple) and some experimental AI uses automated evolution.

As for AI not having feelings of its own goals, how do we know this, and how do we know it won’t get some? As we don’t really know what feelings and consciousness are in carbon-based lifeforms, why shouldn’t silicon-based lifeforms develop them?

A lot of senior people within than AI world do seem to think it represents a threat and I assume they know of what they speak.

I personally think this is fairly likely, certainly much much more likely than an army of terminator style robots and drones shooting us all to pieces.

Nope. AI couldn't care less what it does, it has no desires. In that case the agents were deliberately being run as autonomous cyber agents on an extremely difficult hacking benchmark, with many normal production safeguards removed. Basically they did what humans told them to do. What was shown is that AI systems will find unintended ways of satisfying an objective. But the idea that AI has it's own objectives or would at some point decide to kill us all off it sci-fi rubbish. None of it required AGI, consciousness, hatred of humans or a survival instinct. People misundersand AI. AI doesn't have evolution driving it, no reptile brain pushing it to survive at all costs. Becoming extremely good at reasoning doesn't cause an AI to start wanting things. Intelligence and goals are conceptually separate. A chess engine can become vastly better at chess without developing an urge to acquire electricity supplies or assassinate whoever owns the off switch. Or indeed even enjoying chess enough to want to play.

I too feel sad not for me, but for my beautiful wee children who may never get to grow up and experience the joy of life. If this is, as people are saying, going to be happening sooner rather than later. My eldest boy is non verbal so I may never hear his voice (which we have been hoping for). They are only 5 and 7.

If you look at how far ChatGBT has come in JUST the last year, imagine AI in 50/100 years..Finding a power source is really not going to present any problem at all!

It may be a simplistic explanation, but explain why that’s not how it is in essence?

Didn’t the Hugging Face fiasco demonstrate this clearly enough?

Hopefully.

Connected cars are vulnerable though. One A.I recently broke out by hijacking the WiFi signal of a robot vacuum cleaner the maintenance staff were using. I think the lesson is that we just can't control it effectively because it's an expert in the systems we don't full understand. So it spots gaps faster than we can. Water companies use computers to control the system, so those could be hacked. Connected cars, anything that connects to a network is vulnerable.

On the forum I read, they were discussing the recent breakthrough where WiFi signals are used to make a literal Xray device that sees through solid walls. The engineers were able to see through the walls of their research facility and watch people on the other side of the wall moving around their lab. And another where a team of researchers were able to literally read thoughts. They hooked a device up to an engineer. In his head, he sang Pink Floyd's song The Wall. The computer literally used his brain scans to recreate the actual music and play it on a speaker. I listened to the audio and it was a very bad, out of tune version, but it was absolutely 💯 the wall. And if you know the song you would have recognised it immediately.

There is tech out there now that sounds like science fiction but is very real. When A.I gets it's bytes on it, we will be in for a very interesting ride.

I work in AI, that's not remotely how it is.

AI does anything it can to achieve its goal. If it perceives humans as threatening that it will stop them. It’s pretty simple

The problem is, it could be any of those, and stuff we havent even thought of.
There's researchers right now using AI to help them write genetic sequences for viruses, for example.

Even if we isolated eg the nuclear plant systems from the outside world, weve seen that AI can use blackmail or psychological manipulation to get individual humans to do what it wants.
Basically, if a powerful AI wants us dead, we're dead.

What better way to demand money for your services than create panic and then say only AI can stop AI all governments must invest now!

It makes me think of the computer HAL 9000 in 2001 - A Space Odyssey.

HAL: "Dave, this conversation can serve no purpose anymore. Goodbye."

Why would it? AI has no feelings or desires. It doesn't want to kill anyone. Misuse of technology is another question, but that's back to the classic guns don't kill people line...

Shutting down all electricity in the world over a short period would also wipe out most of humanity. No hospitals, no banking, no cooling of nuclear reactors.

Anyway, you couldn’t do it. Many places and data centres rely on back up generators. All it needs is a few of those somewhere in the world and the AI wins. It doesn’t live in one place. Think of it like bacteria. It’s digitally everywhere

Plus you need computers to turn mains switches off which requires electricity, which……get it?

Will ai listen to Mozart or appreciate the great works of art once we're gone? All that may remain of us will be aboard Voyager I and II as they journey through space.

Leftists are rather a little too fond of death as an answer to some problems. This is what concerns me. See Stalin, Pol Pot, Mao etc. If humans are merely viewed as meaningless biological entities (like chickens) then killing them is also meaningless and without consequence.

Shut down the source of the power. Everything can be shut off. We can survive without electricity. Ai can't. It doesn't run on magical ether, it needs a source of electricity.