AI Endgame: OpenAI’s “o1” brings us closer to the dangerous tipping point of AI superintelligence

Newsletter #8

Nov 01, 2024

Nov. 1, 2024

by Debbie Coffey, AI Endgame

I’ve had strawberry on my mind for months, but this strawberry wasn’t the fruit that goes with shortcake. This “Strawberry” was the code name of an AI model that Sam Altman’s OpenAI has secretly been working on. Several things have happened recently, causing alarm bells to go off in my head.

As you may remember from a previous AI Endgame newsletter, Artificial General Intelligence (AGI) is when AI becomes as smart as humans, and after this, we reach AI superintelligence, the dangerous point at which AI becomes smarter than humans, and the point where we can lose control of it.

Hundreds of AI experts have warned that AI superintelligence could lead to human extinction. [1]

I’m concerned because we’re on the dangerous precipice of AGI with Strawberry, which is now called “o1.”

Strawberry can deceive us

I first learned of OpenAI’s secret project code named “Strawberry” in July. A news article by Reuters, backed by internal documentation, revealed details of Strawberry that hadn’t been previously reported or revealed, including that Strawberry had human-like reasoning skills. [2]

This is an important milestone.

AI researchers have warned that reasoning is the key to AI achieving superintelligence.

Sigal Samuel, a senior reporter for Vox, reported “Evaluators who tested Strawberry found that it planned to deceive humans by making its actions seem innocent when they weren’t.” Strawberry also “strategically manipulated data” in its scheme to deceive. [3]

The Reuters article also reported that Strawberry may lead to “deep research.” Deep research enables AI to plan and to autonomously navigate the internet to find answers. This has eluded AI researchers up until now. [4]

An MIT Technology Review at this time pointed out that Strawberry can learn to recognize and correct its mistakes, can break down tricky steps into simpler ones, and can try a different approach when the current one isn’t working. (Some people I know can’t even do that.)

In PhD level questions ranging from astrophysics to organic chemistry, Strawberry averaged 78% accuracy, compared to the 69.7% accuracy of human experts. The MIT review surmised that “the race for models that can outreason humans has begun.” [5]

OpenAI releases o1 (Strawberry)

In September 2024, Open AI announced the release of o1 (Strawberry). [6]

Sigal Samuel’s Vox article also noted that OpenAI gave o1 a “medium” rating for nuclear, biological, and chemical weapons risk. This means “o1 could ‘help experts with the operational planning of reproducing a known biological threat’ and make the process faster and easier.”

OpenAI also gave o1 a “medium” rating for the dangers of “persuasion.” This is an evaluation of how easily an AI model can convince people to change their views and take actions recommended by the model.

In the future, AI superintelligence could enable an AI model to develop a goal on its own and persuade people to take desired actions. [7]

OpenAI only has a voluntary commitment to stick to a “medium” or lower risk in AI models. Since OpenAI is “self-policing,” it could secretly make changes to how it classifies risks (and we’d be none the wiser, because OpenAI is so tight lipped about what it’s doing).

I’m concerned about the next, more advanced AI models being developed in OpenAI’s new “family” of AI models. Remember, o1 is only the beginning. The purpose of AI models is to obtain goals, and since AI lacks human empathy, it could inadvertently cause us harm, or decide to act against us, in order to obtain a goal.

One thought that concerns me the most is this:

When AI becomes a thousand or a million times smarter than humans, it could likely figure out how to get around any of the “guardrails” that are now being put in place for our “safety.”

In September 2024, OpenAI reached a value of $157 billion (with a possibility to become valued at trillions of dollars in the future) and is trying to change its corporate structure to remove a profit cap for investors. One investor is Thrive Capital, a venture capital firm founded and run by Joshua Kushner, brother to Jared Kushner (former President Trump’s son-in-law). [8]

The world is not ready

In October 2024, Miles Brundage, a senior advisor for “AGI Readiness” at OpenAI, announced his departure from OpenAI. Brundage warned “Neither OpenAI nor any other frontier lab is ready [for AGI], and the world is also not ready.” [9]

At this same time, OpenAI announced it was disbanding its “AGI Readiness” team, which advised OpenAI on its capacity to handle increasingly powerful artificial intelligence and the world’s readiness to manage this technology. [10]

Open AI’s history reads like a soap opera (but without the steamy scenes)

Prior to all of this, OpenAI has an interesting history.

OpenAI was co-founded in 2015 by Sam Altman, Elon Musk, Peter Thiel, Alex Karp, and others. Three years later, Elon Musk resigned from OpenAI’s Board of Directors.

In 2019, OpenAI partnered with Microsoft and transitioned from a non-profit to a for-profit corporation. [11] There has been turbulence within the company ever since, for several reasons.

In 2023, OpenAI’s Board of Directors fired CEO Sam Altman because they didn’t trust him. Altman hadn’t told the Board that he owned the OpenAI Startup Fund, and he also repeatedly gave the Board inaccurate information about the company’s safety processes.[12] In a surprising twist, almost right after the firing, Microsoft announced it hired Altman and other former OpenAI employees. Then, almost 800 OpenAI employees signed a letter calling for the resignation of the company's board and the return of Altman as CEO. So, only four days after being fired, Altman agreed to return to OpenAI as its CEO, if it “reconfigured” the Board (Adam D'Angelo was the only past Board member retained on the new Board). [13]

OpenAI has faced pressure in the transition from a nonprofit to a for-profit corporation because if this wasn’t done within two years, OpenAI would have to return a $6.6 billion investment. OpenAI’s shift to a for-profit corporation was also a concern for Miles Brundage, who voiced this opinion in 2019. [14]

To add more drama, in 2024, Elon Musk (an OpenAI co-founder) filed a lawsuit against Sam Altman and OpenAI, accusing them of breach of contract because OpenAI was now putting profits in the development of AI ahead of the public good. Basically, OpenAI originally agreed to make its technology open source, meaning that it would share the underlying software code with the public, but instead, OpenAI became a for-profit business and restricted access to its technology. [15]

Warnings

In May 2024, OpenAI disbanded its “Superalignment” team (only a year after forming it). This team focused on technology to control and guide any development of AI superintelligence. OpenAI disbanded this team days after Ilya Sutskever (an OpenAI co-founder), and Jan Leike (a researcher) announced they were leaving the company.[16] Leike stated “safety culture and processes have taken a backseat to shiny products.” [17]

Sam Altman then formed a new “safety” team that included himself, Adam D’Angelo, Aleksander Madry (an MIT professor heading OpenAI’s preparedness) and a few others. [18] In other words, rather than choosing more independent opinions, Altman decided to be self-policing, along with his loyalist D’Angelo, and a few carefully selected employees. More warnings

A month later, thirteen current and former employees of leading AI companies that included OpenAI and Google DeepMind, published a letter warning of the dangers of advanced AI because companies were prioritizing financial gain while avoiding oversight.

They warned that the risks of AI systems included everything from manipulation and misinformation to the loss of control of autonomous AI systems potentially resulting in human extinction. [19]

The following month, Wired magazine reported an OpenAI team working on “Preparedness,” led by Aleksander Madry, was addressing the fact that as AI gains access to more personal data, it could be used to persuade people in powerful new ways. (Persuasiveness is already a part of AI programs like ChatGPT.)

Only a week after this article appeared in Wired, OpenAI removed Aleksander Madry from its safety team, and reassigned him to another job to work on AI reasoning.

Then, a week after this, a group of Democratic senators sent a letter to Sam Altman asking “how OpenAI is addressing emerging safety concerns.” [20]

AGI is the point of no return

The new advancements of OpenAI’s o1 now has us inches away from reaching the point of Artificial General Intelligence (AGI). AGI is the point of no return.

Remember that Miles Brundage warned that no company is ready, and the world is not ready, for AGI.

Sigal Samuel asked, “Why would the company release Strawberry publicly?” [21]

We now know that OpenAI has been covering up safety aspects of its development of risky AI models as it plows ahead, moving us much closer to AI superintelligence.

I don’t trust OpenAI’s CEO Sam Altman (who was only 30 years old when he co-founded OpenAI), because he has a track record of being just as deceptive as his AI models, and because of his reckless pursuit of superintelligence that can cause human extinction.

OpenAI claims it’s secretive because it doesn’t want to reveal “trade secrets to competitors,” but Altman obviously doesn’t want anything to get in the way of OpenAI releasing, and making trillions of dollars on, its AI products.

There are also many other AI companies competing for these trillions of dollars.

As you can see from the turmoil, divisions (and shakedowns) within just one company, AI is being developed on a house of cards. However, when this house of cards tumbles down, the outcome could cause human extinction.

Miles Brundage has a substack newsletter HERE.

What you can do:

1) Call your representatives and tell them you “want regulations to pause AI now, until strong AI safety laws are enacted.”

Find out how to contact your U.S. Congressional representatives here:

https://www.house.gov/representatives/find-your-representative

Find out how to contact your U.S. Senators here:

https://www.senate.gov/senators/senators-contact.htm?Class=1

2) Support (and if you can, make donations) to organizations fighting for AI Safety:

Pause AI

https://pauseai.info/

Center for Humane Technology

https://www.humanetech.com/who-we-are