Worried about an AI apocalypse? Here’s how to think about the risk of doom

# The Impossible Math of Predicting AI Doom

Experts disagree wildly on the probability that artificial intelligence will destroy humanity. Some assign it nearly zero. Others claim it exceeds 95 percent. Jacob Aron, a columnist for New Scientist, argues that neither group deserves your trust.

The range itself reveals the problem. When estimates span from virtually impossible to near-certain, something has gone wrong with the underlying analysis. Aron points out that these predictions rely on guesswork dressed up as mathematics.

The challenge stems from fundamental uncertainty. Nobody has built an artificial general intelligence yet, let alone a superintelligence capable of existential harm. Researchers cannot run experiments on doomsday scenarios. They cannot gather historical data on AI-driven extinction events. Every calculation rests on assumptions that cannot be tested or verified.

This hasn't stopped analysts from publishing specific probability estimates. Some prominent AI safety researchers assign extinction risk figures with decimal points. The precision creates an illusion of rigor. It suggests the math beneath those numbers is solid. In reality, the underlying reasoning often breaks down into philosophical speculation, thought experiments, and intuition about how future systems might behave.

Aron's analysis cuts through the confidence. He explains that people are fundamentally bad at estimating probabilities for unprecedented events. The human brain struggles when asked to quantify something with no historical precedent. Add the complexity of advanced AI systems, and any single number becomes nearly meaningless.

The problem compounds when different researchers adopt different assumptions about AI development. Some assume we will create superintelligence within decades. Others think centuries might pass. Some believe alignment problems, which concern how to make AI systems follow human intentions, are solvable with sufficient effort. Others consider them potentially impossible. Change a handful of assumptions, and your doom probability jumps or plummets.

This doesn't mean the risk is unimportant. The stakes are legitimately high. Even if the probability sits at just 5 or 10 percent, the potential harm justifies serious research into AI safety and alignment. The issue isn't whether to worry about AI risks. The issue is how to worry productively without false certainty.

Aron suggests focusing on concrete questions instead of overall doom percentages. What specific failure modes concern us most? Which systems are hardest to align? What early warning signs might indicate we're heading toward trouble? These questions lead to actionable research. They avoid the trap of defending indefensible numbers.

Some researchers have started moving in this direction. Rather than publishing point estimates for extinction probability, they work on specific technical problems. They study how to make AI systems more interpretable, more controllable, and more honest. They examine economic and social factors that might slow dangerous development.

The takeaway isn't that AI extinction risk is imaginary. Nor is it that we should ignore potential dangers. Rather, we should distrust anyone claiming to know the exact probability. The honest answer is that we face substantial uncertainty. That uncertainty demands serious research and thoughtful policy. It just doesn't require false precision.