LLMs built on transformers are about to be eclipsed by a new stack of architectures that rethink how models represent time, context, and “thought” itself. Over the next 18 months, five key breakthroughs—Diffusion LLMs, Power Attention, latent-space thinking, Nested Learning, and Continuous Thought Machines—are likely to redefine what “an AI model” even means.[1]
Intro: The End of “Classic” LLMs
For most people, “AI” still means a giant transformer model predicting the next token in a sequence. But that paradigm is already cracking, and the replacement wave is not a single new architecture, but a stack of ideas that tackle the biggest weaknesses of current LLMs: brittle context, poor long‑term learning, and opaque reasoning.[1]
Over roughly the next year and a half, five technologies described in Pourya Kordi’s breakdown look poised to shift the center of gravity away from pure transformers toward more dynamic, temporally aware, and continually learning systems.
1. Diffusion Language Models
Diffusion isn’t just for images anymore. Diffusion Language Models (DLMs) apply the same idea—iterative denoising—to text or discrete tokens, replacing the one‑shot “next token” prediction style of transformers with a multi‑step refinement process.
Key ideas and implications:
- Instead of emitting the next token directly, a diffusion LLM starts from noise and gradually “denoises” toward a coherent sentence or sequence, analogous to image diffusion models but in a discrete or latent text space.
- This iterative refinement can make outputs more globally consistent, because the model is not locked into greedy or locally optimal token-by-token choices.
- Architectures such as Mercury from Inception Labs explore how diffusion can be adapted to language modeling, hinting at better controllability, robustness, and perhaps more stable training for very large models.
Practically, diffusion LLMs suggest a world where generation feels more like sculpting: the model converges toward meaning over multiple passes instead of committing linearly to each next word.
2. Power Attention and Massive Context
Transformers are built around self‑attention, but their standard attention mechanism scales poorly in both compute and memory as context windows grow. Power Attention is part of a family of advances aiming to break those bottlenecks and make million‑token or even effectively unbounded context windows usable in practice.
What makes Power Attention noteworthy:
- It modifies how attention scores are computed and aggregated so that the model can focus on relevant information in extremely long sequences without exploding cost.
- By restructuring attention math, it enables huge context lengths while retaining strong signal on the parts of the sequence that matter most for the current query or step.
- This makes it realistic to feed in large codebases, multi‑year conversations, or extensive research corpora as “live context” instead of juggling external retrieval hacks.
If classic attention was the engine that made transformers viable, Power Attention and similar methods may be what makes them scale to truly human‑scale information streams.
3. Hidden/Latent‑Space Thinking and Private Chains of Thought
One of the criticism of today’s LLMs is that reasoning is both shallow and exposed: the model “thinks out loud” in the same space it uses to communicate with the user. Emerging research is pushing toward latent‑space thinking, where the model’s internal chain of thought happens in private representational spaces, distinct from the final natural language output.
Why this matters:
- Internal reasoning can occur in compact latent vectors or hidden states, making it cheaper and less noisy than long, verbose textual chains of thought.
- It enables private chains of thought, where the model can explore, revise, or even contradict intermediate hypotheses without exposing that entire process, improving safety and reducing the risk of leaking sensitive intermediate reasoning.
- This opens the door to architectures where you can explicitly separate “thinking time” from “speaking time,” allocating more compute to hard problems while keeping the interface concise.
Conceptually, this pushes models a step closer to how human cognition feels: lots of internal processing, with only a compressed summary surfacing as speech.
4. Nested Learning and True Continual Learning
Traditional LLMs are mostly static: once trained, they only “learn” through expensive, infrequent fine‑tunes. Google’s Nested Learning introduces a new paradigm for continual learning that organizes training into nested levels, allowing models to incorporate new data and tasks more gracefully over time.
Core aspects of Nested Learning:
- Model knowledge is structured hierarchically, so new skills or domains can be integrated into higher “layers” without catastrophically overwriting earlier capabilities.
- The system can adapt continuously to streaming data, more like a living organism updating its understanding of the world instead of a frozen snapshot.
- This creates a path toward AI systems that stay up‑to‑date, learn from long‑term interaction, and avoid the brittle “versioned release” pattern of today’s foundation models.
Nested Learning does not just bolt continual learning onto transformers—it reframes how training itself is staged, potentially becoming a standard for lifelong learning systems.
5. Continuous Thought Machines: Beyond Transformers
The most radical proposal in the stack is the Continuous Thought Machine (CTM), which explicitly steps away from the transformer’s abstraction and returns to richer neuron‑level dynamics. Instead of treating neurons as simple static units, CTMs give each neuron its own temporal processing and then use neural synchronization as a central representational tool.[1]
Highlights from the CTM work:[1]
- Each neuron processes incoming histories over time, with unique parameters capturing temporal structure; timing becomes a first‑class part of the computation.[1]
- Patterns of synchronization between neurons form a latent representation of “thought,” encoding sequences and dependencies in a more biologically inspired way.[1]
- CTMs have been shown to handle tasks as varied as 2D mazes, ImageNet‑1K classification, and parity computation, and they support adaptive compute, stopping early on easy cases and running longer on harder ones.[1]
The stated goal is not immediate state‑of‑the‑art benchmarks, but to open a new design space where neural dynamics and temporal structure are core to the model, potentially providing a more natural path toward complex sequential reasoning.[1]
What This Stack Means for 2026 and Beyond
Taken together, these five ideas sketch a post‑transformer ecosystem where “an AI model” is:[1]
- Iterative and global in generation (diffusion LLMs)
- Massively contextual without collapsing under cost (Power Attention)
- Internally reflective via latent‑space chains of thought
- Continually learning through structures like Nested Learning
- Temporally dynamic and adaptive via architectures like Continuous Thought Machines
2026 is likely to be less about one single “GPT‑like” leap and more about these components recombining into new hybrids, from diffusion‑powered assistants with million‑token context to CTM‑style systems that think continuously and adapt as they go. For builders, the message is clear: treating today’s transformer‑LLM as the final form of AI will age very badly over the next 18 months.[1]
Sources
[1] They Just Retired LLMs: The Next 18 Months Are Wild! https://www.youtube.com/watch?v=h-z71uspNHw