Even Anthropic Is Worried: AI May Be Advancing Faster Than Humans Can Manage, Prompting Calls for an AI "Brake Pedal"

In a move that has sparked fresh debate across the technology sector, Anthropic has urged the creation of mechanisms allowing governments and leading AI developers to collectively decide when to slow or temporarily pause work on the most advanced frontier models.

The proposal from the company's Institute by co-founder Jack Clark and head of internal research Marina Favaro, centers on giving humanity an explicit "brake pedal" so that societal structures, regulatory frameworks, and alignment research can keep up with the technology's accelerating capabilities.

The underlying worry is that progress is outrunning people's ability to understand and steer it.

In other words, Anthropic worries is based on the fact that AI is evolving, and also advancing faster that even their creators cannot keep up.

Internal trends at Anthropic show engineers, as revealed in a post, now shipping roughly eight times more code per quarter than in prior years, while Claude models themselves already author more than 80% of the company’s codebase.

Task horizons have lengthened dramatically, and benchmarks that once seemed distant are being saturated in months rather than years.

The authors highlight the approaching possibility of recursive self-improvement, in which AI systems could autonomously design and build more capable successors, potentially within a couple of years.

In its argument, Anthropic said that in the early days, work at Anthropic looked like work at any other tech company: people writing code and docs on laptops. Then, since the rise of chatbots, people used the ools to help with parts of the process, like generating short code snippets and copying the output into text editors. Then, after coding agents were introduced and became mpre capable, they were able to write and edit code on their own, sometimes entire files.

Now, with autonomous agents, agents can now run code themselves and delegate hours of work to other agents.

Soon, Anthropic said that future agents could become capable enough to build and train models themselves. If this happens, future versions of Claude could be continuously improved by Claude itself.

Without options to pause when risks appear to outpace safeguards, they argue, the world could lose meaningful human oversight at a critical juncture.

Rather than advocating an immediate unilateral stop, Anthropic calls for coordinated, verifiable systems that multiple frontier labs across countries could agree to trigger and lift together.

Verification would need to confirm that participants have actually slowed development and that no actor is using the cover of a collective pause to advance in secret.

The Anthropic Institute plans to contribute research and dialogue aimed at building those practical tools, drawing lessons from past international agreements while recognizing that AI training runs are far harder to monitor than traditional weapons programs.

This cautionary stance comes from a company whose own products are deeply embedded in the current wave of adoption.

The Claude family of models, and especially the agentic coding tool Claude Code, have become go-to resources for developers and enterprises handling complex software projects, workflow automation, and research tasks. Claude Code supports sophisticated multi-agent orchestration and self-correction features that let users delegate substantial portions of coding work while maintaining oversight.

These offerings run across major cloud platforms and have helped drive strong commercial traction, with the company reporting an annualized revenue run rate near US$47 billion.

Anthropic’s commercial momentum has been equally striking on the funding side.

In late May, the company closed a US$65 billion round that valued it at US$965 billion post-money.

Days later it confidentially filed draft registration papers with the U.S. Securities and Exchange Commission for a potential initial public offering. With that trajectory, observers expect the listing could push the company's market value to or beyond one trillion dollars, making it one of the largest and most closely watched technology debuts in years.

Industry reactions have ranged from cautious interest to outright skepticism.

Some analysts see the proposal as a way to shape policy conversations or underscore technical leadership, while others question whether any global slowdown could hold amid intense commercial and geopolitical competition. Practical enforcement remains a central challenge, particularly without participation from every major player.

What stands out is the tension inherent in the moment.

The same organization that has harnessed its own models to accelerate internal development is now advocating for built-in options to moderate that pace when broader human interests demand it.

As tools like Claude and Claude Code become standard parts of how software is written and knowledge work is performed, the question of how to retain collective agency over AI's direction grows more pressing.

Anthropic's suggestion does not reject continued progress; it simply argues that the field would be wiser if it equipped itself with the ability to take stock before momentum becomes irreversible.