Sean's Blog

Popping the AGI Bubble

July 26, 2025
Edit on GitHub

It is becoming clearer and clearer that scaling up LLM's will not solve their fundamental limits like hallucinations (making up bullshit) and context rot (decreasing performance with growing noisy input context). While LLM companies are still slashing benchmarks (often by training on test data directly or guessing the test data) - users of LLM's know the truth: LLM's are a useful tool but nowhere near a robust AGI (that can reliably solve complex tasks without human supervision) and there is nothing in sight suggesting this will change soon.

Fundamental Limits

Lacking a World Model

LLM's are not designed with a world model (allowing to predict 3D physical scenarios over time). LLM only have a token (text) model which only entails a brittle world model that does not generalize well (since it is not grounded directly in 4D reality). Read this post on necessary modules for AGI identified by Jeff Hawkins.

Auto-Regressive Architectures

Current LLM's face a core architectural constraint: they generate text sequentially, one token at a time. As Yann LeCun argues, this creates exponential error accumulation. Each prediction depends on all previous tokens, so early mistakes cascade through long sequences, causing models to derail from coherent long-formed reasoning (an issue well known among LLM users - known as context rot).

This sequential generation prevents LLM's from forming thoughts holistically. Unlike human cognition, which manipulates abstract concepts as complete structures, current models must linearize everything into word sequences. They cannot think about problems in abstract space before committing to specific text.

The result is a fundamental bottleneck. Tasks requiring sustained logic or complex reasoning remain difficult regardless of scale improvements (making autonomous agents doing long complex tasks a real challenge). Simply adding more parameters or training data cannot solve this architectural limitation. Transformer based LLM's will thus never overcome the hallucination problem for long-context tasks (context rot).

True progress requires a paradigm shift toward architectures that form and manipulate complete conceptual structures rather than sequential tokens. Until then, LLM's remain sophisticated stochastic pattern matchers, staying far away from reliable reasoning.

Model Collapse

LLM's trained on their own outputs suffer from model collapse - their performance degrades. Thus the trend of more and more LLM generated content (slop) being published on the web, will degrade their performance in the long run.

Fractured Embeddings

The knowledge representation (weight matrices) in neural networks is fractured / entangled leading to issues like adversarial examples and hallucinations.

Problem Classes

Current AI systems can solve low complexity tasks with lots of training data available (text, code, video, audio) and high-complexity tasks which are easy to verify (so big training data with high-signal can be generated) like chess, programming and math - by interpolating on existing data. It is easy to verify who won a chess or go game and thus possible to generate high-signal training data. The same is true for certain classes of programming problems (does it compile? does it run without errors? does it pass tests? does it produce same output as an existing program? does it better on benchmark X?).

It is much harder to generate meaningful rewards for more abstract programming tasks though (is the UI design polished? is the user experience sensible?) - this data can only be inferred from expensive human usage.

So we have two types of problems: easy to verify problems and thus also easy to produce big high-signal training data and hard to verify problems (needs humans in the loop) and thus hard to produce big high-signal training data. For the former we are in Alpha Zero territory (innovative super-intelligence using RL) and the latter we are in Alpha Go territory (interpolate and synthesize on human generated data).

Real Value

LLM + RL solveable examples:

LLM solveable examples:

  • coding in popular programming languages
  • anything coding / math related that is verifiable:
    • translate code base from prog. lang A to B (verify input -> output)
    • optimize slow code (f.e. PDF extraction) -> measure speed and verify output
    • counterexamples, proofs with machine-checkable steps
  • summarizing complicated technical topics (with lots of training data available)
  • generating (interpolating) texts, audio and video

And anything which has tolerance for errors (hallucinations).

Tasks out of Reach

High complexity (very long context / completely novel) tasks with minimal training data available (often limited by needing humans to produce data).

Any critical task that has no tolerance for high error rates (and is not easily verifiable).

LLM (+ RL) unsolveable examples:

  • creating completely new (not merely interpolating) ideas (mathematical theorems / proofs, physics etc.) in unexplored domains (think outside the box)
    • solving old problems by interpolating existing knowledge is possible though
  • novel creative writing
  • coming up with novel funny jokes

Discovery isn't one thing. It's three. You can induce — generalize from data, which lands you at Newton plus some epicycles to explain Mercury's weird orbit. You can deduce — derive rigorously from axioms you already have, which never gives you new axioms. Or you can jump — invent the frame itself, decide that spacetime curves. That third move is the one that matters, and it's exactly the one induction and deduction can't reach.

Good ref: LLM's can't jump

Signals for AGI

  • no benchmark (problem) we can design where humans beat AI
  • extreme and robust (human like) generalisation ability
    • LLM's possess clearly no human like intelligence as they make obvious logical and factual mistakes (very brittle; context dependent problem solving ability) and hallucinate without being capable to self-correct through self-inspection -> LLM's output will always have to be validated using human supervision for critical tasks (just like for self-driving cars)
    • when humans make little slips (errors) while speaking / writing a thought out - the whole sentence (thought) stays consistent (if the thought was consistent in the first place) -> hinting that thoughts form in an abstract space before being verbalized. OTOH LLM's "thoughts" are directly influenced by each previous token (word) and thus a few small slips can quickly throw them off completely (context rot) -> leading to more hallucinations in longer texts.
  • self-awareness (introspection) and ability to self-modify (continuous learning -> update facts / beliefs)
  • reliable printers

Road to AGI

World models (grounded in perception and real interaction) are the correct approach to produce more robust and general AI systems: current LLMs are locked into pure text perception (vision is just bolted on) and possess no ability to generalize knowledge into unknown domains that are not present in training data (they learn more by mimicking than generalizing rules).

My prediction: true general AI systems will only be possible if they can manipulate an abstract 3D scene and then predict how it will interact over time (world model). For this to work the AI needs a good real world simulator which can be learned from watching lots of video footage from real interactions and grounding it via real-world interactions. Thus companies which mass deploy interactive robots and gather a massive real-world action dataset, might have the ultimate AGI moat (another promising angle: just mine all published YT videos for human real-world interactions to pre-train a general real-world-interaction model).

Coming AI Evolution

Since inherent limitations of the technology behind the coming AI evolution (deep learning aka artificial neural networks) remain unsolved (like hallucinations, adversarial exmaples), it is likely that it will unfold like it has for self-driving cars (one amazing jump and then long stagnation).

Level 5 autonomous driving (drive anywhere, anytime without human supervision) is still unsolved. It only works in fully mapped out small regions like in certain cities (where a lot of training data is available).

Daily driving still needs human supervision because the technology is still unreliable. The same arc will likely happen for all coming AI applications (media, robots, etc). It will be useful (autonomous) in controlled environments but in novel situations humans will need to supervise the AI (unless endless training data can cheaply be generated like for chess etc).

Just like vibe-coding enables software-engineers to output much more work (by automating the boring stuff), the same will happen to most industries. Workers will become more productive (as AI agent managers) and only monitor and supervise their AI workers (digital agents or physical robots) when they get stuck.

References

#AI