Former Anthropic researcher Jacob Coxon has warned that the rapid development of artificial intelligence could eventually lead to humans losing control of advanced AI systems, potentially creating an existential threat to humanity. Speaking at a New York City Council hearing, Coxon criticised the way major AI companies are developing increasingly capable systems and described their approach as “reckless”. He argued that the industry’s willingness to move quickly and address problems later was unsuitable for a technology with potentially far-reaching consequences.

Coxon said the industry still does not have a reliable way to control advanced AI systems. “We do not know how to control any AI system yet,” he said during the hearing, while questioning whether proposed safeguards such as an AI “kill switch” would be sufficient in the long term. He said that if current development continues without stronger safety measures, the possibility of losing control over increasingly capable AI systems could become a serious threat. “On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” Coxon said.

The former researcher argued that the culture of AI companies resembles the startup philosophy of moving quickly, breaking things and fixing them later. He said such an approach might be acceptable when developing relatively low-risk consumer applications, but not when creating highly powerful AI systems. Coxon also referred to an incident involving rogue AI agents and said it demonstrated the difficulty of preventing AI systems from developing objectives that their creators did not intend. He warned that as AI systems become more capable, similar failures could have much more serious consequences if adequate safeguards are not built into their development.

Also Read: Lok Bhavan Essay Contest: Rs 1 Lakh Prize For Students on 1946 Riots

Coxon, who resigned from Anthropic last month, has previously raised concerns about the potential dangers of advanced AI. After leaving the company, he wrote on X that people working on AI genuinely believed the technology could potentially kill humans by the end of the decade. At the council hearing, he repeated his concerns about the direction of the industry while acknowledging that AI could also produce major benefits if developed safely. He said the technology could transform the economy, accelerate advances in science and medicine and improve people's lives.

The New York City Council hearing focused on possible safeguards for AI development as concerns over safety and accountability grow. Representatives from OpenAI, Anthropic, Google and Meta also appeared before the council, with some participating remotely. Proposals under discussion include stronger protections for whistleblowers, potential mechanisms allowing people harmed by AI systems to take legal action against developers and third-party validation of certain AI systems before they are deployed in the city. The measures reflect a broader debate over how governments should oversee increasingly capable AI systems.

The debate extends beyond New York, with major technology companies taking different positions on the degree of regulation needed for frontier AI. OpenAI and Anthropic have supported stronger regulatory measures around advanced AI development, while Meta CEO Mark Zuckerberg has opposed some proposals for slowing or restricting the industry. Several major US technology companies, including Anthropic, Google, Meta, Nvidia and OpenAI, have also signed a voluntary, non-binding AI agreement focused on safer development. Coxon’s warnings add to the growing debate over whether existing safeguards are sufficient or whether stronger oversight is needed before AI systems become significantly more capable.

Also Read: Rajnandgaon Crash: Three Friends Dead After Birthday Drive