Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

Because LLMs are very clever but very specific in what they are and what they do. The idea of "recursive self improvement" in algorithms is a more general one and quite emeshed with the idea of AI. The idea is that this is also happening in the large language models and is the justification for building ever larger data centres and providing as much data as possible. I can accept self improvement is happening although even there there are questions about what they produce if forced only to learn from AI output. But if we accept that there is no reason such self improvement would result in evolving to AGI. I have a very useful toaster. You could probably improve it. It would be cool if my toaster improved itself every day. But it would be highly unlikely my toaster would one day be so good at making toast it would also vacuum the floor. That's not a logical evolution at all. That doesn't mean LLMs couldnt be dangerous or do harm. They arguably already have.

@Persephonia1966 ,

‘But it would be highly unlikely my toaster would one day be so good at making toast it would also vacuum the floor. That's not a logical evolution at all’

But AI is not a toaster! Its main purpose is problem solving in an abstract sense. In that sense, it is much like us. It has been taught to play games, design proteins, solve abstract maths problems etc.

Of course, to become like us, it would probably need to sense like us, but look at how much robots have improved in the last few years, from hardly being able to stand up to having a faster time in the 100m race than Usain Bolt (sub 9 seconds). Apparently it is impossible to separate the ‘thinking’ part of the human brain from the ‘’moving’ part as they have co-evolved.

To me, it doesn’t take a huge leap of the imagination to see AI embedded jn robots (and the military will really want this) evolving into conscious thinking beings.

Of course we can’t know this will happen, as we don’t know how we achieved consciousness but I think it shows a lack of imagination to not see how AI could become a life form.

If you look at good old MrsGren, recursive learning AI embedded in a state of the art robot would qualify as living if you are flexible in your interpretation. And we evolved from single celled organisms in the blink of an eye geologically. Few would have looked at an amoeba and been frightened of thermonuclear weapons.

To be honest, I think complexity will increase, but it won't necessarily lead to consciousness. They remind me more of complex chemical reactions: you set them off, they beget more than you put in... a bit like a slo-mo version of a thermonuclear process.

It doesn't need to be conscious to be dangerous. There are already multiple instances of people attempting to get information for bioweapons. If people can have access to all of our scientific or hacking knowledge at their fingertips it will inevitably lead to dangers.

I think the biggest issues are these and job losses without a plan. Thousands of jobs have gone already, many more thousands to come. There is talk they want to replace 75% of jobs within the next 10 years. If there is no plan in place, things will get very bad. Problem is people won't try and get a plan in place until it's someone they know, or themselves. Governments will get fewer taxation, companies will hoard more profits, AI companies will be essentially subscribed to by every company on the planet and will out perform countries on a massive scale.

On top of that, it isn't blue collar jobs that are going to go first. It's the white collar jobs, the work from home usually higher paid ones. The ones you try and steer your children into for a more comfortable life, which is why so many graduates are struggling at the moment.

The obvious answer is some sort of preemptive taxation and UBI plan, but that isn't materialising fast enough because the pace of this technology is exceeding it. You could easily raise corporation tax to account for the people they have chosen to fire for a machine. Or give tax incentives to companies who keep people. Without these measures we are looking at spiraling debt, chaos, riots, recession, negative equity at the minimum.

Everything about this is rushed which means ordinary people, who have no stake or say in any of it, will pay the price.

People need to ask themselves what they will do if they lose their highly paid job tomorrow with no other jobs to go to. Benefits won't cover your costs, but you can't sell your house because no one is buying. Then who is going to fund your benefits, because we simply won't have the taxation income to pay it? So we either heavily tax corporations across the globe, no tax havens, or people just simply waste away and starve en masse.

The knock on effect of that is the corporations lose customers, so more jobs are shed, leading to more poverty.

Agreed, @Randomusename. Like a combine harvester, once it's set off, it's lethal, and if you get in its path, you're harvested - no conscious intention required.

I think agi is less likely than:

• llm agents escape their boundaries and fuck up something important (I use a lot at work and they are forever ignoring instructions and then saying ‘oops’ when you point it out)
• Companies laying off 1000s of knowledge workers to appease the shareholders and this then destroys their businesses (no one works so no one can afford your product) and the public sector (no higher rate tax payers paying in)
• vastly over valued ai company stock tanks and the trillions of dollars spent go poof overnight which hastens
• general societal collapse (hungry, poor people, no trust in anything anyone says because of ai generated fake news everywhere, no future for the vast majority)

An autonomous ai wiping us out as it is the best thing for the planet is actually a better outcome than us slowly killing ourselves off through our own actions tbh.

There are a lot if people in the business who keep saying AGI is unlikely.

I have posted several times saying that I disagree and explaining why.

No one seems to want to refute the specific points that I have made. I clearly don’t know and am not an expert (as I have said), but I think that my general arguments are sound. I would love a genuine expert to explain to me why I am wrong.

(as well as several high up within AI don’t seem to think it unlikely at all, although people have reasonably questioned their motivations).

Since you complained noone had refuted your points:

1.But AI is not a toaster! Its main purpose is problem solving in an abstract sense
By AI do you mean LLMs? Because AI is a very general term and now covers plenty of things that have been around a while as well as new ideas/development. LLMs are a subset of that and their main purpose is more like roleplay and word play through an incredibly sophisticated probability matrix applied either to words or images.

In that sense, it is much like us debatable. We don't fully agree on what defines consciousness and what makes us human. This is also what makes the "will AI develop sentience" argument so annoying because it distracts everyone with weird poorly thought through philosophical musings. Love philosopjical misings. But it's irrelevant to the "will AI kill us all". Language/communication are a very big part of that, but I would argue it's not the only thing that matters. I also think the definition of "human level intelligence" is being written to fit what is measurable and what computers can do. It's the wrong way round.

It has been taught to play games, design proteins, solve abstract maths problems etc. Yes it is good at solving "language" type puzzles because that is what it is built for. In the same way a toaster is built to make toast. Abstract mathematical problems are effectively just language puzzles at that level. Likewise coding is a language so it can write code. It can also translate between human speech and code which is why it's used for vibe coding. This is genuinely impressive. Its not solving these puzzles in the way humans do and doesn't see language in the way we do but it is so sophisticated someone who doesn't know about Markov chains might think it is. Hence why it doesn't do well at "how many rs in strawberry" questions. They corrected that mistake but then similar mistakes keep getting identified because it can't "learn" what it did wrong and generalise in the way it can with other mistakes (the whole point of machine learning).

Of course, to become like us, it would probably need to sense like us That isn't really the main problem. See the points above. Humanoid robots are cool and all and show great engineering skill. But a model of a thing is not a thing.

Apparently it is impossible to separate the ‘thinking’ part of the human brain from the ‘’moving’ part as they have co-evolved. there we go!

To me, it doesn’t take a huge leap of the imagination to see AI embedded jn robots (and the military will really want this) evolving into conscious thinking beings. Yet but it does require a leap of imagination of some size! If your imagination is highly athletic (a good thing) it is likely not a big leap for it. I do think AI embedded in war machines is concerning. But not because of that.

If you look at good old MrsGren, recursive learning AI embedded in a state of the art robot would qualify as living if you are flexible in your interpretation if you are flexible.in your interpretation of MRSGREN the East India Company was also alive. As was the Roman Empire. Momentum isn't the same thing as life
Plus, If we are as flexible in our interpretation of consciousness then the universe is also sentient. And of course colonies of ants or trees could be both living and sentient. But colonies of ants and trees don't recognise our consciousness in a way that satisfies our ego which in my opinion is a big part of the interest in "AGI" and the various definitions of what it would look like.

Oh BTW by weird poorly thought through philosophical musings I didn't mean people on here. It's more the tech bros themselves that are the worst for it. But I do think it's a distraction

Agree

The annoying thing is that the people owning the companies have become completely obsessed with "winning" the AGI race. And they are very convinced that AGI is potentially dangerous, destructive, like the terminator. Basically it's based on science fiction. LLMs have uses but part of why they are so favoured is, I think, because they are good at the Turing test. I don't think the Turing test means anything when you are deliberately designing something to pass it and Turing himself suggested this. So now we have very sophisticated LLMs that are good at "roleplaying" what the creators want. Every single "super scary" jailbreak/hacking event has been caused by AIs trying to do what they "think" they are supposed to do based on prediction.

The danger here is that if the tech bro types are so convinced that AGIs will destroy the world and the AI is "trying" to meet those expectations then it could end up saying or even doing destructive things because that's what it's prediction software is telling it the "character" whose role it is playing would do.

So we could end up with a computer acting aggressively "for reasons of plot". Which would be super annoying. The way you fix that is by making CEOs liable for the stuff their machines do in the same way a farmer is responsible for his out of control combine harvester. Rather than letting them effectively train their AIs to be nuisances because it makes them feel important.

I'm not an expert - I'm a biologist (and author), and did a 3rd of my degree on philosophy of the life sciences, which gave me an interest in this topic. Hoping a real expert comes along to chat about this soon. It's so fascinating! Before I did that degree I was of your mind, @Newbutoldfather , especially as someone mad on Asimov in my teens. But John Searle's Chinese Room argument really convinced me - until such time as the science of consciousness and AI theory advances, anyway. I know it's been refuted by some, but I don't buy the refutations. (He also had a tin-can thought experiment which I'll try to dig out.)

There might be something crucial going on in what evolution has produced in us (and other animals) over millions of years that can't be replicated without re-running the whole thing under similar conditions. (I wish more AI bods were biologists.) 

That's why I liken current AI to chemical reactions - no matter how complex and whizzy they get, we wouldn't expect a mind to emerge from those. But I'd be intrigued to hear a convincing counter argument. I love this stuff! (And, somewhat conversely, I haven't ruled out panpsychism existing somewhere in the universe - it's a big place  😉)

There might be something crucial going on in what evolution has produced in us (and other animals) over millions of years that can't be replicated without re-running the whole thing under similar conditions. (I wish more AI bods were biologists.) 

That's why I liken current AI to chemical reactions - no matter how complex and whizzy they get, we wouldn't expect a mind to emerge from those. But I'd be intrigued to hear a convincing counter argument. I love this stuff! (And, somewhat conversely, I haven't ruled out panpsychism somewhere in the universe - it's a big place  😉 )

post fail, sorry!

@Persephonia1966 ,

Thank you for taking the time to reply to me at length. I do appreciate your obviously expert perspective.

However, as you might expect, although it gives me more pause for thought, I still disagree with some of it.

‘By AI do you mean LLMs?’

I mean any and all AI. It only takes one to evolve consciousness for AI to have evolved consciousness. I am far less expert on the detail than you here, but I am not sure that affects the generality.

‘I also think the definition of "human level intelligence" is being written to fit what is measurable and what computers can do. It's the wrong way round.’

In a sense, until we know what consciousness is (as you alluded to), it’s all we have got. The fact that the Turing test seems not to have worked is maybe just a problem with that test, and not the methodology. I guess we could ask the question the other way around. How do we know we’re not some kind of chemical AI? We couldn’t prove it beyond doubt.

i also think that is where having some kind of physical body comes into play. We have developed a whole endocrine system to avoid destructive behaviours. So, when we ‘feel’ frightened of dying, it is our way of avoiding a behaviour which stops us propagating our genes. In the same way, some AI has shown to be averse to its own destruction. I know there are more prosaic explanations of why, but you could see it as very early form of this.

‘That isn't really the main problem. See the points above. Humanoid robots are cool and all and show great engineering skill. But a model of a thing is not a thing.’

I get your point and I know a human-seeming robot isn’t a human being! I was more thinking in terms of evolution (recursive learning). Our evolution was very much based on our prehensile thumb and what we could do with it. And I am thinking that movement and sense is very useful to acquire thinking, and certainly feeling. However, maybe AI doesn’t need that stage.

’if you are flexible.inyour interpretation of MRSGREN the East India Company was also alive’

I think that is a bit fatuous! I guess it fails on growth, but I can see the others without too many mental gymnastics.

I guess I am ultimately in the camp of finding the idea of AGI more interesting than some terrorist getting hold of AI and making a bio weapon or an atomic bomb, although of course those are possible.

'a model of a thing is not a thing'
exactly @Persephonia1966 👏

I think you might be the expert we've been waiting for (crossed posts earlier)

I'm not an expert in AI! I don't want to present myself as such. I do know about computers though and probability maths which is the backbone of the LLMs really. And knowing about AI is important to my job but my job is not developing AI if that makes sense. So I don't want to pretend that I know everything about AI but I think I know enough to know what it isn't if that makes sense?

Not looking promising 😳

https://apple.news/Art1x0PtpTzCOAA2O-VVFKA

I'm hoping it's bluster before his talks with China on Thursday. It is in both of their interests to control AI. It's only a matter of time before someone uses it to disrupt either country in a big way. They've given every dissenter a team of specialists, essentially, outside of all the other risks to the human race it is a massive national security risk for governments.

Problem is he has done significant amount of insider trading with his sons in AI so greed is fueling him. But their money means nothing if people have the hacking power to breach banks. People have already hacked state institutions with it now, it's just they stopped short of doing harm.

On top of that 22 UN nations have called for regulation so there is pressure there. I just worry it will take a catastrophe for the world to do something, like with Nukes. We have a chance to step in before it happens here.

We can only hope the midterms are heavily against him. Sadly he now has his own state propaganda TV so the MAGA crowd won't source news elsewhere. Fortunately lots of his voter base are near data centres so their water is being disrupted, their energy bills are sky high, and now fuel is astronomical. Hopefully that will help them see sense.

It's scary, only thing we can do is hope the world and the AI companies don't let a greedy corrupt president destroy everything.

I think AI regulation is a bit of a non-starter. I guess they have to try and talk about it, because the alternative isn’t worth thinking about.

How will the regulation be enforced? Is there going to be a UN body which can enter everyone’s AI labs and see what is happening.

And how will they recruit people to do this given what AI professionals are paid in the private sector?

A start would be that when someone uses AI to hack into another company etc that person is held legally responsible. Rather than being allowed to throw up their hands in suprise and talk about how the AI agent "shocked" them all through it's actions.
I know that then some people will ask how the owners of those AI agents will "control" them but if you read in detail the events surrounding the "AI breakouts" it was very easy to avoid and the companies were extremely negligant in allowing them to happen.
Another law that already exists that could actually be used would be the laws against creating and sharing images of child sexual exploitation. It is now being reported that at least some AI companies (Musk cough) used massive amounts of images of CSE including the worst types in the data sets they used to train their AI. Not only was there no need for that but it is also very much illegal to have that material (you can't just argue it was for "research" not gratification it's still illegal). But for some reason it isn't being followed up on.

Of course, it is extremely unlikely that the Trump government will pursue the American based owners for this. But there are very obvious measures thate could be taken if people wanted to. The whole "AI is so powerful and independent" thing seems partly calculated to discourage regulation/accountability for a group of very rich human men.

Honestly I’m a bit suspicious of all these “look how powerful and scary AI is” stories, I wonder how much of it is planted, bigging up the AI companies to make them sound more advanced and sophisticated than they are to build on their brand and appeal. Not saying we don’t need regulation, of course we do, but I’m taking in these stories in a measured way.

I’m more concerned about the environmental and social factors tbh, and these stories conveniently detract from those.

@Persephonia1966 ,

I think that internal regulation is totally possible. And everything you suggested sounds good.

I am more sceptical of international regulation. How do we know what is going on in a secret lab in China? How could we make sure that they abided by whatever was agreed to (and vice versa).

Everything around CSE with AI disgusts me and terrifies me. Am I right in thinking it was recently ruled in the US that AI CSE is legal? Could be wrong on that, I don't want to Google it because it makes me want to physically vomit.

AI can be regulated. We did it with Nukes, and on a country level we have prisons and fines. These are headed by companies. Make the CEOs accountable for the crimes their product facilitates.

The issue is legislation takes time to catch up and this is moving too quickly.

https://www.dailymail.com/news/article-16153463/Andy-Burnham-speech-chamber-New-York-world-unite-AI-risks.html#

@Ladyoftheapple ,

That, in a microcosm, is the UK’s problem.

We expect (rather than working for) a leadership role on the world stage whilst not bothering to build alliances or develop at home.

I actually think Burnham has some good ideas if he can resist the lure of being a world statesman. We need to be at the forefront of technology at home first, rejoin the EU and work to form meaningful alliances which could demand a global voice.