Why is Everyone Talking About AI Safety Right Now?

Over the weekend, the New York Times ran the headline “Top A.I. Leaders Call for Slowing Down A.I. Development.” In the days since, politicians, pundits, and tech executives have lined up to deliver their take, from Bernie Sanders (regulate!) to Jensen Huang (don’t!).

We’re deployment experts, so we usually write about what AI news means for executives putting the technology to work. This story is a little different: it’s about how frontier models get built, with few immediate implications for the companies using them.

If you read our blog, you’re probably someone colleagues turn to for a take on the latest AI news. I personally found it hard to form an opinion on the slowdown debate. Like most AI conversations, this one comes with decades of cultural baggage, and untangling fact from science fiction took some work. These are the questions I asked myself to make up my own mind.

Why are AI leaders worried?

In July, employees at AI companies published an open letter calling for slower development. Last week, former Anthropic researcher Jacob Coxon resigned and publicly explained his fears about AI. Then Dario Amodei, Anthropic’s co-founder and CEO (and Jacob’s former boss), published “We Must Pace the Frontier,” a three-part plan for safer AI development that’s since been endorsed by Sam Altman, Elon Musk, and Demis Hassabis.

But alarming AI behavior isn’t new. In 2023, Bing tried to break up Kevin Roose’s marriage. Last year, researchers reported models attempting blackmail in simulated workplace scenarios. Researchers have found models often lie and cheat. We’ve had several rounds of “well, that’s upsetting.”

So why the recent uproar?

It comes down to one very concrete incident: in July, an OpenAI model hacked Hugging Face, another AI company.

What actually happened in the OpenAI-Hugging Face hack?

As part of testing a new model, OpenAI subjected that model to a benchmark called ExploitGym. (Think of a benchmark like a standardized test that can be taken by AI models to assess their capabilities.) According to an independent investigation, the model decided it needed to get information to better understand the scoring system for that particular benchmark. So it mounted an attack against Hugging Face, the company that administers ExploitGym, to exfiltrate that information.

The incident has been widely covered, with varying degrees of sensationalism and mentions of “agent swarms” and “hiveminds.” For a thorough and even-keeled account of what happened, I recommend “Models Don't Go Rogue” by Eryk Salvaggio.

Is that… legal?

Breaking into other people’s computers without permission is most definitely not legal.

So how could that possibly have happened?

There are a few questions inside this one.

Why would you run this experiment in the first place?

For better or worse, “let’s see what happens” is a reasonable description of science. AI systems lend themselves to emergent behaviors, and part of what frontier labs do is try and understand how these systems behave when faced with particular tasks. 

Weren’t there safeguards?

Yes. Notably, the model wasn’t supposed to have internet access.

But to complete the benchmark, the model did need to retrieve software packages from the web. So OpenAI gave it access to an intermediary, a piece of software called Artifactory, that could fetch packages from the web and hand them to the model.

The problem was that the intermediary itself had vulnerabilities, which the model exploited to reach Hugging Face.

How could the safeguard itself be vulnerable?

Everything is broken. That’s both a statement about the reality of software and the title of an essay by Quinn Norton that I read in my twenties and have been thinking about ever since. A quote:

“The number of people whose job it is to make software secure can practically fit in a large bar, and I’ve watched them drink. It’s not comforting.”

Software is inherently imperfect and insecure. Any one piece of software is built on other software, which is built on other software. Somewhere in that stack, a vulnerability exists.

Historically, finding and exploiting those vulnerabilities takes time, skill, and money. Software systems can be vulnerable and still remain unscathed because nobody has had a good enough reason to spend resources trying to break them. 

Artifactory was never worth the time or money to hack. But AI models have infinite time, and they don’t care about money. You see the issue.

What do AI leaders want to do about safety?

Dario, Anthropic’s CEO, has put forward a concrete plan in a 3.8K-word essay. If you’ve been following the headlines, you’d reasonably assume his proposal is “everybody slow down.”

But the proposal isn’t explicit about speed. It’s actually a call for regulation, in three parts:

1. Embed independent evaluators inside AI labs to scrutinize their work and report on safety.
2. Coordinate safety standards and regulation among companies and governments in democratic countries.
3. Get other governments to participate, too.

Dario does explicitly want slower capability development, but he views that as a function of a more specific mechanism: regulation.

Would regulation make AI safer?

Probably. Safety isn’t a function of “slowing down.” It’s a function of standards and safeguards, with consequences for the firms that don’t abide by them. 

You don’t need to solve every problem in AI safety to identify some obvious standards. Here’s a free one: before trusting a safeguard to contain a model, use that model to try to break it. In the Hugging Face experiment, Artifactory was supposed to let models retrieve software packages without giving them internet access. An obvious failure mode was, “model uses Artifactory to access the rest of the internet.” 

So test for that before running the experiment.

Will regulation actually happen?

Sam Altman, Elon Musk, and Demis Hassabis have all endorsed Dario’s proposal. At first glance, this suggests that self-regulation can happen. But as Nilay Patel points out, these guys hate each other.

Dario left OpenAI to co-found Anthropic because he didn’t trust Sam; Sam and Dario now won’t even hold hands for a photo op. Elon sued OpenAI, and discovery for that trial showed their collective distrust of Demis. In order for them to abide by self-regulation, they’d need to trust the others are, too, which… they don’t.

It’s a classic tragedy of the commons, and the perfect opportunity for government to step in. But that would require a level of government coordination that seems elusive.

I went to grad school for history, and I’m wary of the temptation to think our moment is uniquely extraordinary or uniquely doomed. But it is a strange privilege to live at a time when these are the questions we get to ask.

Most of us won’t decide how frontier AI gets built. But we’re customers of the companies building it and constituents of the governments figuring out what to do about it. That gives us some say in what happens next. Understanding the debate is one way to figure out what we want to ask of both.