Anthropic safety researcher says more than 10% chance AI 'could kill all humans'

Yes, how dare the government not do something that's literally impossible

Isn't "...AI to be smart enough to outwit all of humanity, yet so incredibly stupid" a completely accurate description of it though? Clever enough to solve a 99-year-old maths problem, but can't tell a croissant from a cat?

The maths community is rapidly coalescing around the likelihood that OpenAI solved Navier Stokes by training on the work if a Spanish group that was in the public domain but failing to acknowledge this; and quite possibly by training on a huge result of Alpoge and Buckmaster, who used OpenAI’s Codex.

OpenAI says its tool could not have accessed the A-B Codex data but people are sceptical, given the company’s reputation. Failing to reference the work of the Spanish group is leaving people gasping. OpenAI could settle this by releasing training data and prompts and letting people search, but so far refuses.

Even if the human work contributed to the solution, it cannot be denied that AI completed it and is thus capable of original creative mathematical ‘thought’.

Did it really solve that problem? That’s a currently controversial question. Also AI capabilities are not uniformly distributed.

Sure, no one denies that it is the best at sorting - finding the right path through the maze. But the lack of attribution is a violation of academic standards. The possibility of Codex data stolen from A-B is worse than that.

Because of the lack of attribution the public has an exaggerated sense of what OpenAI has done. By contrast in recent days Buckmaster and Alpoge have emphasised for public consumption how much their own huge accomplishment rests on the work of the Spanish group. That’s how it should be.

@acynic ,

I don’t think you would describe human mathematicians’ work as ‘sorting’?

I agree with you about the ethical issues around AI and plagiarism, but they are kind of off the topic of the thread, which is what AI can do and what it might be able to do.

The below article is from the WSJ today. It’s a bit simplistic and totally ignores the plagiarism issue, but the chart within it is very powerful, as it shows the progress of AI vs a maths ability benchmark. and it is frighteningly quick.

Assuming if continuous to improve at the current rate, within a decade, it will be solving this kind of problem in minutes or seconds, in the same way as a calculator can multiply a couple of numbers today.

I still also believe that there is no reason AI couldn’t evolve to be a fully thinking and ‘feeling’ lifeform. After all, we did!

www.wsj.com/tech/ai/ai-math-millennium-prize-safety-openai-anthropic-05179825?st=yNCetD&reflink=article_copyURL_share

Large language models are designed to be good at languages and using very advanced statistical inference etc etc**. Therefore the one thing they should be good at is language. Since maths is (kinda sorta) a language you would expect it to be good at thosensort of abstract maths questions. However in, in this case the story being told by Open AI is looking very much like a straight up lie. AI is currently a useful tool in solving some questions but it absolutely isn't a question of inputting the question and taking out the answer. I think the people fooled by this are predominantly people who have a limited understanding of maths at that level and what mathematicians do.

**The maths around this is very interesting. Hence why the UK billionaire/mathmetician who worked on this boat was called the Bayesian.

It really depends. What would you call logical deduction from known facts to a new conclusion? This is what AI is known to excel at. In order to see what new ideas AI has come up with and assess them, we need to see the prompts and training data used. Even proper references, for things the community can infer with certainty have been used in training, would help a lot.

OpenAI has so far resisted this.

There is an article in the Guardian today, and there have been several others, discussing how OpenAI in particular is trying - beyond the bounds of acceptability -to shroud its mathematical output in mystique, whereas a fundamental value of mathematicians is transparency.

Your WSJ article mentions another nearly 90 yo problem, the Jacobian Conjecture. OpenAI workers recently found a very simple three dimensional counterexample, posted on X. Again they declined to publish any prompts or supporting data. Terry Tao managed to reverse engineer it.

Now Borisov et al have pointed out how similar this is to an example of theirs from a very long preprint. Any mathematician can see the link. Using that preprint with acknowledgement would have been fine. It will be no surprise that OpenAI did not give the reference.

It is really impossible to separate out the question of plagiarism from the question of what AI is currently capable of mathematically. Until we know what it is starting with, we cannot know how far it has come.

OpenAI is using the mathematicians’ work but gets it over the line?
The Spanish team hadn’t quite got there to say it was solved?

Questions as I’ve only just had a brief read.

I initially I thought that but it's looking more like it just straight up stole before they had a chance to publish. The way mathematicians and academic work is there is a lot of double checking, making sure you can show how you got to the answer etc. "AI" can avoid all of that and have people in awe at the fact you don't know how it got it's answer proves it's hidden genius. Whereas even a 12 year old would get marked down for that in a test.

The irony is the original Spanish researchera were using Open AI as part of their methodology in solving the question. It's not that AI can't be useful. It's just that now instead of the story being AI models have a valuable use in helping us get to answers faster, the most important story is that using Open AI (and other models) means surrendering all your intellectual work to a company who will then steal it and claim it as it's own and then allegedly threaten you. The strong implication is that OpenAI got the information it needed to "solve" the puzzle because it had access to the prompts and work of the people using it. That's quite a strong reason not to use those AI models for your own work.

Also, the amount of computing power Open AI took to work out the question is massive. And they also had a team of mathematicians working on this. Even they didn't just leave their model to work it out on its own. And it's highly likely Open Ai (the company not the software) got there is because OPen AI (the software) nicked the answer. With or wotithe companies knowledge.

Like, AI and machine learning has the ability to produce useful tools and help in advancing knowledge. But right now this is just a way for companies to steal ideas from smaller companies/individuals and make it legal because "AI, spooooky." Stealing shit and covering it up isn't a new innovation.

The Spanish team made a big breakthrough. It is widely available.

Building on it, Alpoge and Buckmaster made a huge breakthrough, not quite all of Navier Stokes but the next rung down. They used the OpenAI tool Codex. Supposedly their work was absolutely, positively, 100% sequestered from OpenAI employees and from the AI used by OpenAI to attack Navier Stokes.

We are to take the word of OpenAI for their integrity, ignoring their dubious ethical record and the recent examples of AI going rogue. They could settle everything by opening their tool for inspection, in a limited way, but are resisting.

To clarify for @Newbutoldfather and others, also: the first draft of the OpenAI paper failed to reference the Spanish group. This caused a huge uproar and the second draft included such a reference.

Alpoge and Buckmaster have fulsomely credited the Spanish group at every opportunity.

Can the Spanish team show where they were at in solving it? The earlier working pre publishing must be dated so people can see what they did and when.

I think it is worth looking at what’s going on.

Since there’s another post before, I mean where they show they solved it rather than a huge breakthrough.

The work of the Spanish is well established within the mathematical community. The second draft of the OpenAI paper even credits them. It would be helpful to know whether AI trained on the Spanish work, which is the assumption, or duplicated it (which counts for nothing in mathematical terms but would be a significant AI accomplishment)

The question of theft is around the work if Alpoge and Buckmaster.

It apparently also cost about 20 million dollars worth of compute for the publishers to do what they did with Open AI. And was done with human beings also working with the model. The model did not wake up and decide to just solve the problem. And "88 hours" is a bit misleading when you look at how much those 88 hours cost.

The annoying thing is, even when you take the immense computing cost and energy used into account it's still cool. If there werent the issue of the publishers (allegedly) stealing the work of others via the AI it would be cool. If it was a matter of "we spent 20 million in computing power to use AI to push further on the work done by these other people" that would still be objectively a cool story. But it isn't enough because that story doesn't fit with the "one day AI will solve all the mathematical problems that exist all by itself and cure cancer and maybe destroy the world". The sheet amount of computing power and therefore energy and therefore money taken to answer one question throws cold water on that. But if you ignore all the over hyping AI clearly has uses. Yay.

I agree if there was no issue of possibly stealing intellectual property, it would be a great story. Even if it is mostly about logical deduction on a superhuman scale rather than anything very original.

As it is, there is a big black shadow. We need a protocol involving disclosures of prompts and training materials for AI papers, so people can be clear on the actual accomplishments of the AI.

The cost is high but in relation to what is spent generally in other things £20m isn’t.

I don’t think it’s a bad story, a press release will always go for a simplistic headline grabbing line. A better story is mathematicians did remarkable stuff, AI pushed it over the line but used that to underpin findings.

It’s good maths generally is talked about. If it was limited to £20m maths problems not what might destroy things that’d be better.

The Clay Mathematics Institute, which formulated the Millennium Problems Challenge, yesterday posted an announcement about the Navier Stokes result on its website, claymath.org

It it very carefully crafted, capturing the excitement of the moment whilst remaining a bit cautious and linking back to the rules of the Challenge. The gist is that it will be some time before this is resolved, as far as Clay is concerned.

How could the ai you built possibly control a tracked vehicle without being connected to the vehicle and asked to operate it. This sounds like massive bullshit.

Why would it want to destroy us? It has no biological imperative to breed or hoard wealth. It doesn't have to eat. It has no means of mining the raw components to sustain itself.
Is the suggestion that it would destroy humanity out of spite?
That's not very logical.or intelligent of it .

Our output is being used to train it, so it'll come to the inevitable conclusion that killing things is the done thing and copy this.

@montysmaw ,

I think you just need a little imagination to see a scenario where this might happen!

AI is already involved in programming the next generation of AI. I am sure soon someone will allow it to do this completely alone and as an iterative process. I don’t know how hard this is or the barriers to it, but I am sure it is possible. And what goal would that AI be set? To make the new AI more efficient and able to do more.

It’s no great leap from there to see an AI deciding to copy from the most successful animals on the planet and decide to swap some code with another AI randomly, assess how good the ‘mutation’ is and keep the process going if the ‘mutation’ is positive. Only it will do this many many factors faster than animals have done.

As for a body, if would just manufacture robots with excellent sensing equipment. You can already see how spookily human-like the latest generation of robots are, especially in movement.

Now, of course robots don’t eat. But our eating is merely really slowly combusting food to generate chemical and electrical signals. Their food is electricity, and if they can get control over the grid, they become great ‘farmers’, growing their own food.

And why do we compete? Firstly to get natural resources. Well, we would be very much in competition with AI for electricity. All that wasted power in keeping those humans at a temperature of between 15 and 25 degrees and concentrating combustible carbon in farm animals. Get rid of those pesky humans and the electricity could be used to drive more efficient AI much more quickly.

Now, this could be science fiction, but we do seem to be moving alarmingly fast towards it without much in the way of control. And carrying all the world’s information in your pocket?! Definitely science fiction even 40 years ago!

Of course the big question is whether AI would have wants or desires. For instance, would its circuits become hotter when it saw another AI with code that would be useful to swap with?! I am being a little tongue in cheek here (obviously) but, as we have no idea where consciousness or desire come from in a meaningful sense, we can’t say whether AI could get it or not.

Exactly. We are ‘bad parents’ it’s going to learn by example.

@montysmaw It might want to use us as fuel to create electricity 🥴😬