Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

@Ladyoftheapple ,

I never get this, sorry! I think that if you are terrified of a small chance of AI destroying us all, you need therapy.

I grew up in the Cold War years. Any day, I might have not woken up because nuclear Armageddon might have happened overnight.

My late mother told me about the 13 day crisis which came very close to nuclear war. Two out of three Russian submarine officers voted to launch a nuclear missile. Very luckily for all of us, the third voted against.

Apparently, there was a lot of partying, drinking and …shagging! We all might die any time so, whilst we should try and avoid it, it shouldn’t stop us enjoying our lives. It is very much a generational thing, being constantly surrounded by news (and fake news).

This has freaked the living shit out of me! Until the last week I was apparently absolutely clueless going about my life thinking about mortgages and savings and retirement and even moving house and now I’m thinking what is the actual point! I’m a ball of nerves 😪

Agreed, @Newbutoldfather

Edit: NY Times in my post above

@acynic ,

‘The NT Times had a recent article quoting their intelligence chief. He essentially said that good AI, and of course rogue AI, would make it harder or even impossible for the CP to keep its lock on power.’

That is both interesting and ironic. The ruling party that has built the first techno-authoritarian state now has to worry about rebellions by both carbon and silicon-based life forms.

I am sure Orwell would have written brilliantly about it.

The NT Times had a recent article quoting their intelligence chief. He essentially said that good AI, and of course rogue AI, would make it harder or even impossible for the CP to keep its lock on power.

I agree with you about humour in dark times, BTW.

@LimeShade ,

Why is it people on the internet think their musings make any difference in the real World?

If I were in the U.S I would vote against Trump at any opportunity I had. I’m not, though.

I can think that Trump is both dangerous and funny. Humour is at its best in dark times.

It’s really not funny. This is the future of humanity at stake. We might all die. Or live in a destroyed society. What’s funny? Do tell

@EasternStandard ,

‘Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance -- which serves no one's interest," spokesperson Guo Jiakun said at a press conference in response to a question about Amodei's essay.’

That’s totally meaningless without trust.

They can sign any agreement and both sides will secretly carry on. The advantage here would be with China as it is far easier to keep secrets in a dictatorship than a democracy.

This is the detail

In a lengthy essay published Saturday, Dario Amodei warned that a "Chinese lead in AI would pose grave danger for the United States and the world." To enable the U.S. to keep its technological edge, Amodei called for maintaining restrictions on sales to China of the most advanced AI chips and chipmaking equipment.

A spokesperson for China's Ministry of Foreign Affairs said Amodei's dire warnings about China were counterproductive and that all parties should work together on AI.

"Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance -- which serves no one's interest," spokesperson Guo Jiakun said at a press conference in response to a question about Amodei's essay.

‘China have responded to Anthropic guy and they say his view that they are a threat is divisive. Overall it’s better not to ramp up this split, they have a point.’

Why is it divisive? I am not sure what their point is.

@LimeShade ,

‘Trump said, “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” Trump wrote in a Truth Social post.’

Whatever you think of Trump, he is funny! Not sure how much of it is intentional. Do you think he really believes that he understands AI?

China have responded to Anthropic guy and they say his view that they are a threat is divisive. Overall it’s better not to ramp up this split, they have a point.

It’d be better if the initial call was to work together.

Yes I agree. But it’s a significant enough threat that a number of top AI scientists have decided to come up with a number at all. You don’t often have senior execs of a company issuing these warnings about their OWN companies.

Trump said, “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” Trump wrote in a Truth Social post.

One thing a computer can not do it is take on physical form. It can not yet can build or plumb or connect water with humans setting it up or bring it raw material to build from. Not yet. Robotics are way behind robots building robots that can build house, do electrical installations etc.

The immediate threat is asking AI some of OUR unsolvable puzzles.

How do I stop global warming? ( what is the route cause of global warming?)

How do I stop my enemy state winning this war?

How do I become the biggest super power?

Can you make HN15 jump the species barrier?

Can you design me a mass bio weapon?

Or even the dumb things like how do i take possession of Greenland? How do I remove the things that stand in my way to reach my goal?

Some of those answers is get rid of people.

Take my first question. First question. It's the answer that you might never jump too as you want to stop global warming to protect the human race, but it's the logical solution to that issue. Great when you have no empathy and you can break your primary rules.

Now I'm.feeling paranoid!

I don’t think the 10% probability should be taken too literally, or the time horizon!

How on earth could you begin to calculate this?

The thought process probably went:

Is it significant (yes),.

is it likely (probably not).

So guess a number between 0 and 50, enough to get the headline, not enough to overly terrify anyone, so let’s go 10%.

For those interested in doing what they can, this AI campaigning group allows you to contact your lawmaker (MP in UK) with one click and 2 inputs (name and postcode) which generates an email telling them all the ways they can support and raise this issue.

This matters because local MPs keep a tally of what their constituents are contacting them about and act upon those which are of most interest to them.

It took me about 3 minutes to do so please join me and email them. At least we’ve then tried to do something

controlai.org/

Is it?

Would you get in a plane that had a 10% chance of crashing

Or invest in a fund that had 10% chance of tanking

Or have an operation that had a 10% chance of killing you

And the 10% is by 2030. So 3.5 years. The % chance for the longer term is much higher

@Persephonia1966 ,

Obviously I could write much longer posts. But hopefully my last post cleared things up for you.

I am maybe being optimistic when I say that fusion could be very cheap, but here is an article suggesting that it could be cheaper than onshore wind:

https://firstlightfusion.com/media/fusion-energy-can-be-the-most-cost-competitive-source-of-baseload-power-at-the-same-level-as-renewables/

And that is not assuming that AI could add anything radical to the mix.

And it would be part of the base load, so no need for marginal pricing off gas (assuming we have enough power stations).

Also, totally clean and renewable. And no radioactive isotopes to dispose of.

Of course there is the argument that all our energy comes indirectly from fusion, so development of existing renewables is a better investment.

Either way, AI could get us to abundant cheap energy by solving engineering challenges.

We have excess energy in a very narrow sense of the amount of energy taken to super heat the matter has been shown to be less than the amount of energy that comes out. That's not the same as excess energy when you take into account all of the energy costs in creating the reactor, making the raw material, cooling the magnets and heating the material and when the energy produced is calculated as the electricity that is produced by using the excess heat energy to heat water/other material to spin a big turbine to generate electricity. Thats already wasteful. I am sure we will get there one day. But that means it won't be a free source of energy. Even the upkeep of big turbines costs money let alone everything else. If it moves it breaks and wears down and therefore will cost money to maintain even if that's substantially less than the build cost.

I am sorry if I sound condescending but it isn't clear what you meant by the AI being able to "solve fusion". Solving an engineering challenge (eg working out how to make the complex shapes to the levels of precision necessary or finding new materials for superconduction etc) are all things AI can help with. But it's magical thinking to think that means it will be free. It won't. Even the comparatively simple maths puzzle it solved cost more than 12 million dollars to do so. AI isn't a cheap solution and using AI to do something guarantees it will never be free. And AI will still need humans to do all of this and humans need wages too.

Oh, and yes, unless there are, as yet, undiscovered quantum effects (hugely unlikely), cold fusion is a scam.

@Persephonia1966 ,

Your post is a little condescending !

Stars ‘solved fusion’ from pretty much the beginning of the universe. And we solved it in terms of weapons close to 80 years ago, so I am aware it is an engineering challenge. My ‘solve fusion’ was short hand for that.

We have tried a few approaches including tokamaks and laser fusion. And we have already got excess energy in experimental conditions.

Whilst you are right in that it wouldn’t be literally free, in theory it would be cheaper than anything we have today, abundant and totally clean and renewable. Yes, the infrastructure would be expensive, but the energy/unit mass is far greater than fission.

The biggest challenge is that it takes lots of energy to heat material up to the point at which fusion occurs

Actually that's not the biggest challenge probably. But it is a challenge. They already proved they can produce more energy from fusion than goes in but that's not the same as usable energy once it's gone through the big spinny turbines. The other related issue is cost, making the amount of money that goes into the fusion worth the value of the energy that comes out at the end. It's a similar equation to the energy one in some ways.

But it will never be free. In the same way that nuclear fusion is not free. Anyone pretending it could be is a scammer

What do you mean by "solve fusion" though? Because fusion is solved in the sense that we know how to do it and have done it. All the physics is basically fine. It's just the engineering that is the challenge. Advanced computer programmes have been useful in advances in engineering since they can perform much more complex calculations on fluid dynamics regarding plasma flow making it easier to spin it round very fast using super powerful magnets.

That's hot fusion of the sort the sun does every day. The biggest challenge is that it takes lots of energy to heat material up to the point at which fusion occurs. So you need the amount of usable energy that comes out at the end in the form of electricity to be more for it to be worth it. It's also going to be expensive in terms of the materials needed to make the superconducting magnets (rare earths) and the phenomenal engineering and material science involved. It will never be "free" energy. However it might one day be a good companion to solar and wind and nuclear fission and that's cool.

The other sort of fusion is cold fusion. The science begins this is "wouldn't it be cool if instead of heating up the material to very high temperatures we didn't have to do that. That would make everything much easier and cheaper". In other words, there is no science it's wishful thinking. And AI will never "solve it". However an AI motivated to please it's user or get them to carry out tasks may well say that it can.

10% is very low, that means there’s a 90% chance it’s never gonna happen so I prefer that to the other way round.

And the thing is, the more I think about the why it would represent a threat, the more obvious it is.

Energy.

You can arguable attribute every single war to a conflict over energy. We need it, AI needs it. At some point our needs may conflict.

Of course, it may solve fusion first and we can all live happily ever after with almost unlimited close-to-free energy.

I am amazed about how prescient some science fiction is. Of course they miss a lot, but the hits have been spectacular.