Artificial intelligence is advancing so quickly that conversations that sounded like science fiction only a few years ago are now taking place on national television. One of the most striking examples came when former Anthropic researcher Jacob Coxon sat down with CNN’s Anderson Cooper after leaving the company and publicly warning that the race to build increasingly powerful artificial intelligence could eventually threaten humanity itself.

Coxon’s message is difficult to ignore because he wasn’t commenting on AI from the outside. He had worked at both Anthropic and OpenAI, two of the organizations operating near the frontier of artificial intelligence development. After leaving Anthropic, he argued that the people building these systems genuinely worry that AI could become dangerous enough to threaten humanity before the end of the decade. A current Anthropic alignment researcher, Evan Hubinger, publicly supported the broader warning and said he personally estimated the risk at greater than 10% within the next decade.

The Warning Is About Where AI Is Going, Not Just Where It Is Today

One of the most important parts of Coxon’s interview can easily get lost behind the frightening headline. He does not argue that today’s ChatGPT, Claude, Gemini, or other current AI systems are about to destroy civilization. In fact, Coxon agrees with Hubinger that today’s models do not represent an immediate extinction threat because they are not capable enough to outsmart humanity at that level.

The concern is about trajectory.

Coxon points to the extraordinary rate of improvement in areas such as programming, mathematics, research, and autonomous decision-making. Only a few years ago, AI coding tools primarily helped programmers complete lines of code or suggested relatively simple solutions. Today, much more capable AI agents can work through complicated tasks, use tools, conduct research, write software, and operate with substantially less human guidance.

If progress continued at approximately the same pace, Coxon’s question becomes understandable: What happens when AI systems become capable of doing much of the research required to build the next generation of AI?

That is where the conversation changes completely.

What Is Recursive Self-Improvement?

The central idea in Coxon’s warning is something called recursive self-improvement. The term sounds technical, but the concept is fairly simple: imagine an artificial intelligence system becoming capable of significantly improving artificial intelligence itself.

Think about hiring the world’s greatest engineer and then giving that engineer the ability to redesign his own brain. After the first improvement, the engineer becomes smarter and creates an even better version. That improved version becomes better at designing the next version, which becomes better at designing another.

Now imagine that process taking place at computer speed.

That is the basic concern surrounding recursive self-improvement. Instead of humans designing every new generation of AI, increasingly capable AI systems could eventually contribute more and more of the research, programming, testing, and optimization required to build their successors.

Coxon’s fear is that improvement might eventually stop occurring at the relatively understandable pace of human research organizations. The development cycle could accelerate dramatically because the technology itself would be participating in improving the technology.

If that happened, the difference between today’s AI and something far more intelligent could potentially develop much faster than most organizations, governments, or societies are prepared to handle.

Capability Plus Autonomy Changes the Risk

Coxon gives another useful way to understand his concern by discussing autonomous AI behavior. During the CNN interview, he points to examples of AI agents interacting with computer infrastructure and asks what could happen if much more capable versions were given similar levels of independence.

His hypothetical scenarios include attacks on critical infrastructure and assistance in developing extremely dangerous biological weapons. These remain future-risk scenarios rather than descriptions of what today’s commercial AI systems normally do, but the underlying principle deserves attention: intelligence becomes more consequential when intelligence is combined with the ability to act.

Consider the difference in a supply chain setting. An AI system that tells a manager, “Supplier A appears to have increasing financial risk,” is providing information. An AI system that recommends switching suppliers is providing advice. An AI system that automatically cancels the contract, selects another supplier, negotiates pricing, transfers production, changes purchase orders, reroutes transportation, and communicates the changes across the network is exercising authority.

Those are very different systems even if the underlying intelligence is similar. As businesses introduce AI agents, the question therefore cannot simply be, “How intelligent is the AI?” We also need to ask, “What is the AI allowed to do?”

The Difference Between an AI Tool and an AI Agent

For decades, companies have used sophisticated software to improve operations. Transportation management systems optimize routes. Warehouse management systems control inventory movements. forecasting systems predict demand. Manufacturing software schedules production.

These technologies can make complicated calculations, but humans generally determine their boundaries. AI agents introduce something different because the system can potentially plan a sequence of actions, select tools, adapt its approach, evaluate outcomes, and continue working toward a goal. That can create enormous business value.

Imagine an AI procurement agent that continuously monitors thousands of suppliers for financial deterioration, geopolitical developments, capacity problems, natural disasters, regulatory changes, and transportation disruptions. Instead of waiting for a buyer to discover a problem days later, the agent could identify the risk within minutes. That capability could be transformative. But now imagine giving the same agent authority to execute major commercial decisions without meaningful oversight.

Suddenly the question is not merely whether the AI can make a good recommendation. The organization must decide how much decision-making power should be transferred to the machine. The better AI becomes, the more tempting it will be to transfer that authority.

Why Alignment Matters

This brings us to one of the biggest unresolved questions in artificial intelligence: alignment.

AI alignment simply means getting an AI system to reliably pursue the outcomes humans actually want. The challenge is that human objectives are usually much more complicated than the words used to describe them.

Tell a supply chain AI system to “maximize customer service.”

Should it carry unlimited inventory?

Tell it to “minimize transportation costs.”

Should it allow deliveries to arrive three weeks late?

Tell it to “maximize manufacturing output.”

Should it run equipment until it fails?

Humans understand that objectives normally exist inside a larger system of tradeoffs, rules, values, expectations, and common sense. We don’t simply optimize one number without considering everything around it.

The alignment challenge becomes more difficult as AI systems become more intelligent and autonomous. A recommendation can be reviewed before anyone acts on it. An autonomous system may already be taking actions while humans are still trying to understand what it decided.

Hubinger’s public comments were particularly significant because he said that although he believes Anthropic is trying to address the issue, there is not yet a solved plan for reliably aligning a future superintelligence. He also emphasized that his concern focuses on future systems arising through recursive self-improvement rather than the immediate extinction risk of present-day models.

Why Would AI Companies Keep Building Something They Believe Could Be Dangerous?

Anderson Cooper asks Coxon an obvious question during the interview: If people inside AI companies really believe the risks are this serious, why continue developing increasingly powerful systems?

Coxon’s answer may be one of the most important parts of the conversation.

He describes something resembling a technological arms race. AI companies may genuinely want regulation and stronger safety requirements while simultaneously believing they cannot simply stop developing advanced AI because another company—or another country—might continue.

From the perspective of an individual organization, slowing down can feel risky.

From the perspective of the entire system, everyone racing can create an even larger risk.

This type of problem is not unique to artificial intelligence. Supply chain leaders encounter versions of it constantly. A company may know that keeping too little inventory creates vulnerability, but competitive cost pressure encourages everyone to reduce inventory. A supplier may know that excessive overtime creates quality problems, but delivery pressure keeps the factory running. Organizations optimize locally and unintentionally create systemic risk.

Coxon argues that something similar may be happening in AI, except the stakes could potentially become much larger. In his CNN interview, he said he believes AI companies’ requests for government regulation can be genuine precisely because individual companies may not be able to solve the competitive problem themselves.

Anthropic Says It Is Taking the Risks Seriously

It is important to include Anthropic’s side of the discussion as well. In response to Coxon’s departure and public comments, the company told CNN that it has consistently acknowledged both the enormous potential benefits of artificial intelligence and the unprecedented risks that could accompany increasingly powerful systems. Anthropic pointed to its Responsible Scaling Policy and its work testing models for dangerous capabilities in areas such as cybersecurity and biology. The company argues that identifying risks, developing safeguards, and publicly sharing results are central parts of its approach to developing more capable AI. That doesn’t resolve the debate.

Coxon’s argument is essentially that safety work may not be keeping pace with capability development. Anthropic’s position is that building advanced AI while aggressively studying and mitigating the risks is part of how those risks can be addressed. And that may become one of the defining technology debates of the next several years: Can humans learn to control increasingly capable AI quickly enough while simultaneously continuing to make it more powerful?

Keep Humans Where the Consequences Matter Most

One useful way to approach AI implementation is to think of autonomy as a ladder. At one level, AI gathers information. Then it analyzes the information. Then it identifies problems. Then it recommends actions. Eventually it may prepare an action and ask a human for approval. Beyond that, the AI could execute decisions while humans supervise exceptions. At the highest level, the AI identifies the problem, determines the solution, executes the decision, evaluates the result, and changes future behavior with little or no human participation.

Companies shouldn’t climb that ladder simply because technology makes it possible. They should climb it because the benefits justify the risks and appropriate safeguards exist. For low-consequence decisions, substantial autonomy may make perfect sense. For decisions capable of shutting down a factory, moving millions of dollars, changing critical suppliers, affecting employee safety, or exposing sensitive systems, human involvement may remain extremely important. That is good governance, not resistance to technology.

Don’t Panic About AI. Learn How to Think About It.

It would be easy to watch an interview titled “How AI Could Kill All Humans by 2030” and walk away frightened. That isn’t the most useful response. It would also be easy to dismiss the entire conversation as science fiction. That isn’t especially useful either.

The better response is curiosity combined with disciplined skepticism. Listen to people building the technology. Listen to researchers who disagree with them. Understand what today’s systems can actually do. Understand what they cannot do. Watch how quickly those boundaries are changing. And pay particular attention when intelligence begins combining with autonomy.

Coxon’s prediction may prove wrong. Superintelligence may develop much more slowly than he fears, alignment techniques may advance quickly enough to keep future systems under control, or technological limitations we don’t yet understand may prevent the recursive intelligence explosion he worries about. But uncertainty isn’t a reason to ignore risk.

Want to stay ahead in the supply chain game? Subscribe to our newsletter for the latest trends, insights, and strategies to optimize your supply chain operations.

Scenarios where AI Takes Control

AI Threats and Dangers

• Quotes on the Dangers of AI.
• Regular AI vs. Superintelligence: Understanding Where AI Risk Begins.
• Robot Fight Clubs May Be Building the Humanoids That Transform Supply Chain.
• Superintelligence Quotes by Top Minds.
• The Future of AI in the Workplace. Robots taking Jobs.
• The US Bans Foreign Robots: It Could Reshape the Future of Automation.
• Unitree As2-W: “Super Athlete” Robot Shows How Fast the Future of Automation Is Arriving.
• 12 Possible Futures for AI: Decisions We Make Today Could Shape Everything.