Published on November 9, 2025 2:20 PM GMT

The downside of our brain’s neuroplasticity is that we take things for granted. As soon as the present is experienced, it becomes our one reality. Our new normal.

When predicting the future, our priors are anchored in present states, at least intuitively. “How different” a hypothetical situation is from this frame of reference is used as a proxy for “how unlikely” it is. Yet intuitions often feed our probabilistic reasoning.

We could call this the implausibility heuristic: when we treat the perceived difference from our present world model as evidence of improbability. It’s a broader instance of the normalcy bias — our tendency to underestimate the probability and impact of disaster because we struggle to imagine what we haven’t experienced before

Think of what a “sci-fi world” usually means. We typically use it to describe both a world that is vastly different from ours, and one that feels highly implausible. Generally, it makes sense to use “different” ≈ “unlikely”. A highly different scenario is usually conditional on either a large number of changes or a few very radial ones, both making it overall highly improbable.

But there are cases where this proxy breaks down.

(1) Things remaining exactly as they are is unlikely. We expect each day to resemble the next, but over longer time scales we knowingly anticipate variation and decay.

Yet our intuitions still struggle to imagine something different from the present. When we’re young, we can’t imagine being old. When it’s winter, we can’t fully contemplate the heat of summer. Our felt sense that “now it’s freezing cold and I can’t imagine it being boiling hot” doesn’t hold up against what our rational minds know to be true.

The “different” ≈ “unlikely” proxy also fails when (2) a highly different (or “sci-fi”) world depends on only a few events that are themselves quite likely.

A world with flying machines seemed like science fiction in October 1903, when an editorial in the NY Times predicted that it would take “from one million to ten million years” to develop airplanes.

69 days later, the Wright brothers achieved the first heavier-than-air flight.

A world in which the atom was split daily seemed like science fiction in 1933, when Ernest Rutherford dismissed the idea of harnessing atomic energy as “moonshine”. In 1934, Albert Einstein likewise claimed that “there is not the slightest indication that nuclear power will ever be obtainable”.

By 1942, Enrico Fermi had achieved the first controlled nuclear chain reaction, giving rise to both nuclear power and weapons.

Those sci-fi worlds had rested on only a few discoveries or inventions that followed naturally from concentrated effort, once the necessary precursors existed. Such is the nature of breakthroughs: obvious ex post, unknowable ex ante.

These examples are, of course, cherry-picked. For every Wright brothers, there are thousands of failed moonshots. And technological progress in physical engineering isn’t necessarily analogous to AGI.

Still, AI is a domain where previously unthinkable advances have been occurring on a regular basis. My suggestion is that using the base rate of advances within AI could yield better predictions than our intuitive sense of difficulty.

The authors of “A Definition of AGI” evaluated state-of-the-art AI models along the spectrum of human cognitive abilities. They found that current systems exhibit a very jagged profile: excellent where they can leverage large amounts of data (general knowledge, reading and writing ability, maths), but lacking in areas like long-term memory or perception.

What’s left to bridge these deficits?

Adam Khoja (one of the authors) distinguishes AI advances between “business-as-usual” research and engineering (when capabilities improve smoothly with scale), “standard breakthroughs” (like OpenAI’s reasoning models in 2024), and the more radical “paradigm shifts” (such as the invention of Transformers). I’d be very interested in exploring the empirical frequency of each type, and the historical markers separating them.

Per Khoja, based on recent developments, human-level visual reasoning and world modelling, or fixing hallucinations, might only require business as usual. For example, evaluating spatial reasoning on a subset of the SPACE benchmark, GPT-4o (May 2024) scored 43.8% whereas tests from the Center for AI Safety showed that GPT-5 (August 2025) scored 70.8% already. Humans average 88.9%.

It’s worth noting that over-optimisation to the benchmarks, rather than the underlying capability, can occur, and diminishing returns may appear. 43.8% → 70.8% → 88.9% need not imply linear proximity to human reasoning. However, it’s feasible that in areas like these, further research and engineering could close the gap without requiring major conceptual advances.

What about continual learning?

Humans can adapt their behaviour from experience. Current AI models can’t, at least nowhere near as much — they remain “frozen” after training. To learn over time as humans do, a genuine breakthrough may be required.

The question is: how feasible is this breakthrough? Is it a “standard” breakthrough, or does it require a paradigm shift to overcome a fundamental limitation of RL-trained models?

We don’t know — such is the nature of breakthroughs.

If it’s just a standard breakthrough attainable with sustained efforts, we might be able to extrapolate from recent trends in AI progress. And indeed, major efforts seem to be directed toward improving continual learning.

Sam Altman insinuated this in August: “People want memory. People want product features that require us to be able to understand them”.

And this is Dario Amodei, in an interview also released in August:

The pattern that I’ve seen in AI on the research and technical side is that what we’ve seen over and over again is that there’s what looks like a wall. It looks like, “AI models can’t do this,” right? It was like, “AI models can’t reason,” and recently, there’s this, “AI models can’t make new discoveries.” A few years ago, it was like, “AI models can’t write globally-coherent text,” which of course now they obviously can.

Models learn within the context…Maybe we’ll train the model in such a way that it is specialized for learning over the context. You could, even during the context, update the model’s weights

So, there are lots of ideas that are very close to the ideas we have now that could perhaps do this. I think people are very attached to the idea that they want to believe there’s some fundamental wall, that there’s something different, something that can’t be done.

These are perspectives from insiders, which might be biased. Still, they at least suggest that efforts to close these capabilities gaps will continue or intensify.

In the end, it all boils down to how likely such a breakthrough is.

But if it’s as tractable as recent ones, I believe that the base rate of breakthroughs in AI^[1] is a reasonable starting point for our estimates. This base rate could decay, and it’s important to consider arguments for why it might, and potential indicators of a slowdown. History warns us that trends rarely continue indefinitely, and the same could hold for AI. Diminishing data quality, compute limits, or a lack of new conceptual insights could all flatten the curve.

However, absent such evidence or compelling reasons, it seems more rational to anchor our forecasts on observed empirical progress rather than on the intuitive tendency to treat unprecedented as implausible. We’re already living in what once looked like science fiction.

1. ^{^}

   Since “breakthrough” is a fuzzy concept, I’m not aware of empirical base rates of breakthroughs in AI existing, and they probably don’t exist yet. It would be valuable to define them more rigorously and study their trends over time, and across variables like compute and investment.