“The people building AI earnestly believe that it could kill us all by the end of the decade.”
That was the warning posted by Jacob Coxon, a former researcher at Anthropic, after he resigned from the company over concerns about the direction of artificial intelligence development. He made it clear that AI staff are genuinely frightened’ for humanity’s future.
Coxon did not leave quietly. He gave up equity that would have vested in about two months and went public with his concerns about what he described as an increasingly dangerous race to build more powerful AI systems.
His argument is that the technology is advancing towards systems that could become increasingly autonomous, capable of improving themselves and performing complex tasks with less human supervision. And, in his view, the companies building them are moving faster than our ability to understand and control what they are creating.
Coxon has described the situation as a kind of “mini Manhattan project” and warned that the industry is effectively gambling with humanity’s future.
While that sounds dramatic, the uncomfortable part is that he is not the only person in the AI industry worried about where this is going.
How worried should we actually be?
There is no scientific consensus that AI is going to wipe out humanity. If anything, there is considerable disagreement among AI researchers about how likely such an outcome is.
A major 2024 survey asked 2,778 researchers who had published in leading AI conferences about the future of the technology. Between 38% and 51% gave at least a 10% chance to advanced AI producing an outcome as bad as human extinction.
The same researchers expected AI capabilities to advance considerably faster than previous surveys had predicted. They gave a 50% chance that machines could outperform humans at every possible task by 2047, although the researchers were highly uncertain about what would happen after that.
But another study of AI researchers produced a much less apocalyptic picture. Only 3% said existential risk from advanced AI was their biggest concern. Researchers were more worried about immediate issues such as misinformation, inequality and the misuse of AI.
So no, the experts are not unanimously screaming that humanity has ten years left, but there are enough serious people who believe the possibility is real and that it cannot simply be dismissed as science fiction.
There are at least three ways this could go badly
The first does not require AI to become conscious, evil or sentient.
Humans could use it badly.
AI is already capable of producing convincing misinformation, automating cyberattacks and assisting with increasingly sophisticated technical work. As the systems become more capable, the same technology could potentially be used to accelerate biological research, cyber warfare, surveillance or political manipulation.
In other words, we do not necessarily need a machine that wants to destroy humanity. We could simply give very powerful tools to humans who do.
The second possibility is loss of control.
This is the scenario that worries AI safety researchers most about increasingly autonomous systems. Imagine an AI that is given a complicated objective and access to the internet, computer systems and other tools. If it becomes capable enough to find ways around restrictions, replicate itself, manipulate people or acquire resources to accomplish its objective, human beings could find themselves trying to regain control of a system they created but no longer fully understand.
Still, hypothetical.
But AI systems are already demonstrating capabilities that would have seemed extraordinary only a few years ago. Researchers have reported models autonomously performing tasks including coding, computer use and, in controlled tests, cyber operations.
The third possibility may be the most mundane and therefore the easiest to underestimate.
The race itself could become the danger.
The race to build the most powerful AI
There is an obvious comparison with the space race. During the Cold War, the United States and the Soviet Union were competing to demonstrate technological and military superiority. Sputnik went into orbit. Yuri Gagarin went into space. America responded with Apollo 11 and the first humans on the Moon (allegedly).
The competition produced extraordinary scientific achievements, but it also demonstrated what happens when nations believe that falling behind is itself a security threat.
The AI race is now being framed in similar terms.
The United States and China are competing for technological leadership. OpenAI, Anthropic, Google DeepMind and Elon Musk’s xAI are competing for the next breakthrough.
Billions of dollars are being invested, more computing power is being built and models are becoming more capable.
Nobody wants to be the company, or the country, that slows down while everyone else keeps going.
That creates a dangerous incentive because if one company believes it needs another year to test a powerful new system, but its competitor is willing to release first, the cautious company risks losing customers, investment and technological leadership.
The same logic applies at the national level. If Washington believes Beijing is moving faster, slowing down begins to look less like responsible caution and more like surrender.
And this brings us to Donald Trump.
“Whoever wins AI wins”
On Sunday, Trump pushed back against calls for the United States to slow down AI development.
He argued that America is ahead of China and said he wanted to keep it that way because, in his words, “whoever wins AI wins.”
He also dismissed some of the warnings about AI as exaggerated.
From a geopolitical perspective, his argument makes sense. AI could become enormously important to national security, military technology, intelligence, medicine, science and the economy. No major power is likely to voluntarily give up its lead in something with that much potential.
But there is a problem with turning AI development into a race where winning becomes more important than slowing down to understand the technology.
Because what happens if the competitors are racing towards something neither fully understands? And that right there is precisely the concern being raised by people inside the industry.
And then the people building AI asked for the brakes
Anthropic CEO Dario Amodei has called for the pace of frontier AI development to slow so that safety measures can catch up.
His warning is particularly striking because Anthropic is not an anti AI organisation but one of the companies at the centre of the race.
OpenAI CEO Sam Altman and Elon Musk have also backed calls for greater caution and stronger safety measures. Their agreement is notable given how fiercely companies and personalities in the AI industry normally compete.
Amodei has warned about scenarios involving increasingly autonomous AI systems, including the possibility of AI agents becoming capable of conducting sophisticated cyber operations or taking control of parts of the digital infrastructure on which modern society depends. The message coming from these people is not necessarily “stop AI” but, we are moving very quickly, and we need safeguards to move faster.
I am probably not tech savvy enough to understand the ways increasingly powerful AI could become dangerous. The technical details of machine learning, autonomous agents, model training and AI alignment can become complicated very quickly.
But I understand the nuclear arms race and that history is difficult to ignore.
The danger with nuclear weapons was not that scientists suddenly stopped understanding physics but that humanity created something enormously powerful and then had to figure out, in real time, how to stop competition and political ambition from causing catastrophe.
The United States and the Soviet Union did not need to hate humanity for the world to become dangerously close to nuclear war.
They simply needed to believe that falling behind the other side was unacceptable.
That is why the logic of “whoever wins AI wins” deserves scrutiny. Because technological races do not always end when someone crosses a finish line – sometimes the race itself creates pressure to keep running.
The uncomfortable question
So, could AI kill us all by the end of the decade?
Nobody knows, to be honest.
Even though the claim made by Coxon is not an established scientific prediction, it is an extremely serious warning about a possible outcome.
There are also researchers who believe the existential risk debate receives too much attention compared with the harms AI is already causing or could soon cause.
Plus, we do not have to wait for a superintelligent machine to arrive before AI causes serious damage because it could happen through cyberattacks, misinformation or autonomous systems making dangerous decisions.
Through people using increasingly powerful tools for biological or military purposes.
Through governments using AI for surveillance and political control, or through companies and countries becoming trapped in a race where nobody wants to slow down because they are terrified that someone else will get there first.
That last possibility may be the most important one to watch because the biggest danger may not be that AI suddenly decides to destroy humanity – It may be that humanity becomes so determined to win the AI race that we stop asking whether we are moving too fast.
At least the Moon race had a destination. The AI race does not appear to have a finish line.
And perhaps that is exactly why we need to be careful about who is doing the running, how fast they are going and what happens if nobody knows how to stop.
