Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.

The good version of unilateral pausing is good

Here is the good version of unilateral pausing:

1. Capability pause: Some frontier lab (let’s call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research.
   
2. Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the existing structure of work and teams within the specific company, reducing disruption to other processes. They have said they would like to work with other companies and countries to adapt and strengthen this definition over time, in particular to make it gradually more compatible with actions by other actors. Importantly, we accept that it may be easiest for a company to pause according to its own, company-specific definition, and handle any asymmetry in their agreements via multiple rounds of negotiation with other actors.
   
3. Safety + security + product: In place of capability research, their efforts to alignment and other safety research, information security, and product. The product part is important! They will still be a trillion dollar company shipping extremely useful tools, and this also provides an escape valve for people within the company who care about products and don’t want to do safety research.
   
4. Restart if necessary: The pause is temporary, and will restart after some period of time if progress is not made to broaden the pause. Potentially one wants separate periods based on different kinds of progress. Other US labs are likely to pause sooner than labs in China, since international diplomacy takes time.

There are multiple places in the post where Scott makes it clear he wants the reader to be thinking of much worse plans than this. Emphasis added:

I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.

As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%.

Again, the correct version of a unilateral pause is a company pivoting to safety + security + product, not “disbanding their entire company immediately”.

Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.

This is again arguing against a strawman: OpenAI would not be “out of the industry” if they decided to unilaterally pause. Sam Altman loves to build products! He would very much still be in the industry.

It is possible that Scott will correctly point out that some people do advocate for the extreme version here, but I don’t think he should get credit for this. The title of the post is “Against Stop Unilaterally and Raise Awareness”. I think it is perfectly reasonable for someone to say that SURA-the-phrase applies to my better version. His post is very strongly being interpreted as arguing against unilateral stopping in general. It is being interpreted that way because of his title and framing.

Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism

Scott appears to agree that the good version of unilateral pausing is good, but attempts to claim that only one group is advocating for this good version, the AI Futures Project.

(A virtuous exception)

I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.

The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.

Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the