‘Gambling with our lives’: Another AI employee quits over safety concerns

AI Safety Debate Intensifies After Anthropic Researcher’s Departure

Sandego.net – A newly departed Anthropic researcher has added fresh urgency to the debate over whether artificial intelligence companies are moving faster than their ability to keep increasingly capable systems under control.

Jacob Coxon, a 27-year-old former employee at Anthropic, used a resignation thread on X to describe fears shared by some people working on advanced AI. His remarks drew attention not simply because of their severity, but because a more senior colleague publicly said the underlying concern was real.

“The people building AI earnestly believe that it could kill us all by the end of the decade.”

Coxon warned that future systems could become extraordinarily capable, able to break into computer systems, transform entire fields at remarkable speed, and obtain meaningful power or resources. He argued that companies are pushing toward self-improving superintelligence without sufficient safeguards.

“They are racing straight to self-improving superintelligence and gambling with our lives.”

A familiar divide within the AI industry

Anthropic itself emerged from an earlier dispute over AI development. Its founders included former OpenAI employees who left amid concerns about safety practices and set out to build a company centered more heavily on responsible development. Coxon’s departure shows that even companies established with safety as a defining principle continue to face internal disagreement over how much risk is acceptable.

Similar warnings have surfaced across the sector in recent years. Some employees have left AI companies publicly, citing concern about rapid progress and weak guardrails. Others have argued that the technology could eventually advance beyond humanity’s ability to supervise or direct it.

The discussion is no longer confined to a small group of researchers. In July, almost 1,400 employees from AI companies signed an open letter calling for US regulation intended to curb the power of major technology firms and slow development enough to address safety issues. Two days before Coxon’s posts gained attention, OpenAI Chief Scientist Jakub Pachocki warned that AI capabilities were improving more quickly than the tools researchers use to monitor and control those systems reliably.

What “superintelligence” means—and why it remains disputed

The central concern raised by Coxon and others is superintelligence: a hypothetical point at which AI exceeds human abilities across a broad range of important tasks. There is no universally accepted benchmark for when that threshold would be reached. Some researchers believe elements of it may already be visible in narrow areas, while others doubt the concept will ever materialize in the form described by its strongest advocates.

For critics of the current pace of development, the uncertainty is precisely the issue. They argue that waiting for definitive proof of a danger could leave little time to respond if powerful AI systems acquire unexpected capabilities. Others in the field maintain that the possible benefits of advanced AI—from scientific research to improved productivity—make continued development essential, provided companies invest in testing and safeguards.

Evan Hubinger, an Anthropic employee, responded to Coxon’s thread by saying Coxon’s concerns reflected a genuine belief held by people working on the technology.

“We really do earnestly believe AI could kill all humans!”

Hubinger said he personally put the chance of such an outcome at less than 10% during the next decade. He also said that, despite Anthropic’s intentions, the company did not have a plan that could guarantee avoidance of a superintelligence beyond human control.

He later clarified his comments, pointing to Anthropic’s own Risk Report. That assessment recognizes the broader concern while stating that current AI systems have little chance of gaining the kind of power associated with catastrophic loss-of-control scenarios.

Warnings from industry leaders

Anthropic chief executive Dario Amodei has repeatedly cautioned that the competitive drive to develop more powerful systems could produce a serious mistake. His concern is not necessarily that one company intends to build something dangerous, but that complex technology can fail in ways its creators did not anticipate.

“This is a complex engineering problem and I think something will go wrong with someone’s AI system. Hopefully not ours.”

That warning captures a central tension in the industry. Companies are under pressure to release stronger models, attract investment, and compete with domestic and international rivals. At the same time, they are being asked to demonstrate that their safety testing can keep pace with systems that may be increasingly difficult to evaluate.

For ordinary users, the issue can seem abstract until an AI system produces harmful, misleading, or insecure results. The broader safety argument is about whether those familiar problems could become far more consequential as AI is used in software development, scientific work, cybersecurity, business operations, and other high-impact settings.

Regulation remains unsettled

Despite repeated calls for stronger oversight, a comprehensive US approach remains distant. The Trump administration has worked against state-level AI rules, while Congress has not imposed sweeping limits on the industry. A frequent argument against tougher American regulation is that it could benefit China, whose AI sector would face fewer restrictions.

China has nevertheless adopted some AI-focused rules, especially on risk management and safety. Last year, it required companies to label AI-generated material so it could be identified and traced. Those measures do not amount to the stringent limits sought by many Silicon Valley safety advocates, but they challenge the idea that regulation is absent entirely.

In the United States, companies largely remain responsible for evaluating their own systems. The White House has been moving toward a voluntary framework for reviewing certain AI models before they are released. Representatives from OpenAI, Anthropic, Google, and Meta participated in discussions with the administration. Details of the framework are not expected to be public, and a June executive order indicated that many standards within it would be classified.

OpenAI chief executive Sam Altman also suggested in July that a slower pace may be warranted. Speaking on the Invest Like The Best podcast, he said recent testing incidents involving advanced models were prompting concerns about the long term.

Coxon’s resignation does not settle the argument over how likely a catastrophic AI failure may be. It does, however, underline the depth of unease within companies that are helping define the technology’s future. The question now is whether governments, companies, and researchers can establish credible limits before the next generation of systems makes the debate even harder to resolve.