Imagine that artificial intelligence is a school. The Transformer redesigned the classroom so every student could instantly notice what every other student was saying. Mixture of Experts expanded that classroom into a university full of specialist departments, while sending each question only to the most relevant experts. Finally, reasoning models gave students extra examination time, scratch paper, calculators and the ability to check their answers before submitting them.

Those three developments tell much of the story of modern AI. They also explain why the industry keeps demanding more computing power even as individual algorithms become more efficient. The Transformer made AI scalable. Mixture of Experts, usually shortened to MoE, made extremely large models more economical to run. Reasoning models made intelligence partly expandable at the moment a question is asked. Each breakthrough solved a major bottleneck—and then moved pressure to a different part of the hardware system.

1. Before Transformers: Reading Through a Narrow Tunnel

Before 2017, many leading language systems relied on recurrent neural networks, or RNNs, and improved versions called long short-term memory networks, or LSTMs. These models processed text in order. They read the first word, updated an internal state, read the second word, updated the state again, and continued until the sentence ended.

This sounds natural because humans also read sentences from left to right. For a computer, however, it created two problems. First, training was difficult to parallelise. If step 100 depended on step 99, the computer could not easily calculate every step at the same time. Second, information from far earlier in a passage could fade as it travelled through many sequential updates.

2. The Transformer: The Architecture That Changed AI

In 2017, a group of Google researchers published a paper with the unusually confident title Attention Is All You Need. The paper proposed the Transformer, an architecture built around attention mechanisms rather than recurrence or convolution. The original task was machine translation, not a universal chatbot. Yet the design became the foundation for most of today’s major language models. The paper’s central contribution was not that attention itself had never existed, but that a model could be organised primarily around attention and trained highly efficiently in parallel.

What is attention?

Consider the sentence: “The robot placed the glass on the table because it was stable.” What does “it” refer to—the glass or the table? A model must examine relationships among several words to decide. Self-attention lets each token assign different importance to other tokens in the sequence. A token is a small unit of text, often a word, part of a word or punctuation mark.

The model turns each token into numerical representations and calculates which other tokens matter for understanding it. Different attention heads can focus on different kinds of relationships: grammar, position, reference, style or semantic meaning. These heads are not little human minds, but mathematical channels that learn useful patterns.

The original Transformer contained an encoder, which built an understanding of the input, and a decoder, which generated the output. Later model families adopted different portions of the design. BERT, introduced in 2018, became a famous encoder-focused model that learned by examining context in both directions. GPT models used decoder-style Transformers to predict the next token. T5 retained an encoder-decoder structure and converted many language tasks into a text-to-text format.

From prediction machine to general-purpose model

The Transformer alone did not create modern chatbots. Several additional ideas had to come together: large-scale pretraining, better datasets, more computing power, improved optimisation, instruction tuning and human-feedback techniques. Still, the architecture made these advances practical.

GPT-3, described in 2020, was a major demonstration of what happens when a decoder-only Transformer is scaled to 175 billion parameters. A parameter is a learned numerical value inside the model. Parameters do not store facts in a neat encyclopaedia; together, they shape how the model transforms inputs into predictions. GPT-3 showed strong “few-shot” behaviour: it could often infer a task from a few examples placed in the prompt rather than requiring a separate training procedure for every task.

This encouraged a powerful belief: if models, data and training compute continued to grow, capability would often improve in predictable ways. Research on scaling laws found broad relationships among performance, model size, data and compute. The result was an industrial race to build larger training clusters and feed them more high-quality tokens.

Why the Transformer changed hardware

Transformers rely heavily on matrix multiplication—large grids of numbers multiplied and added together. GPUs were designed to perform many similar calculations in parallel, so they became natural engines for training Transformers. Google developed its own Tensor Processing Units, while other companies built custom AI accelerators. But raw arithmetic was only part of the story.

During training, model weights, activations, gradients and optimiser states all compete for memory. During inference—the stage when a trained model answers users—the system must repeatedly move model weights and stored context through the accelerator. This is why high-bandwidth memory, or HBM, became so valuable. HBM stacks multiple DRAM dies close to the processor and provides much more bandwidth than ordinary memory channels.

The hardware impact spread beyond the GPU. Large models required fast interconnects among accelerators, advanced packaging to place processors and HBM close together, powerful networking across servers, vast storage for datasets and checkpoints, and more electricity and cooling. The Transformer did not merely create a new kind of software. It reorganised the economics of the data centre.

3. Mixture of Experts: A University of Specialists

Scaling dense Transformers created a simple but painful problem: a dense model generally uses nearly all of its layers and parameters for every token. If the model becomes ten times larger, the computation required per token can also become dramatically larger. Researchers therefore revisited an older idea called Mixture of Experts.

The roots of MoE go back decades. A 1991 paper on “adaptive mixtures of local experts” described a system in which different networks specialised and a gating mechanism determined which expert should handle each example. The modern breakthrough was applying sparse expert selection inside enormous Transformer models and distributing those experts across many accelerators.

How MoE works

Imagine a hospital containing cardiologists, neurologists, radiologists, surgeons and many other specialists. A patient arriving with an eye problem does not need to be examined by every doctor in the building. A receptionist directs the patient to the most relevant specialists.

In an MoE model, the specialist neural networks are called experts. A small routing system—often called a router or gating network—examines each token and selects a limited number of experts. In a top-two routing design, for example, each token may be processed by two experts even though the model contains eight, dozens or hundreds of them.

This is called sparse activation. The model may contain a very large total number of parameters, but only a fraction are active for each token. Importantly, experts do not always correspond to clean human subjects such as “biology expert” or “Korean expert.” Some may learn statistical patterns that are useful but difficult to name.

Which models made MoE important?

Google researchers and others explored large sparse models well before they became famous among consumers. GShard, published in 2020, demonstrated how MoE could be scaled through automatic sharding across large computing systems. Switch Transformer, published in 2021, simplified routing so that each token was sent to one expert. Its authors reported models reaching up to a trillion parameters while addressing instability and communication costs. Switch Transformer did not invent MoE, but it was a landmark demonstration that sparse Transformers could reach extraordinary scale.

MoE entered broader public discussion in late 2023 when Mistral AI released Mixtral 8x7B. Mistral described it as a sparse decoder-only MoE in which each layer selected two of eight expert groups for every token. Mixtral offered strong performance relative to the amount of computation used, and because it was openly available, developers could experiment with the architecture directly.

DeepSeek later pushed MoE efficiency further with systems that combined routed experts, shared experts and sophisticated load-balancing techniques. DeepSeek-V3, announced in late 2024, disclosed 671 billion total parameters but roughly 37 billion activated for each token. That gap between total and active parameters captured the central attraction of MoE: large capacity without paying the full dense-model compute cost on every step.

What breakthrough did MoE create?

MoE changed the meaning of model size. In a dense model, total parameters and active parameters are similar. In an MoE model, they can be very different. This allowed developers to increase the model’s representational capacity while keeping the arithmetic required for each token closer to that of a smaller model.

MoE therefore improved the economics of scaling. It helped open-weight models challenge larger closed systems, encouraged more competition and made it possible to offer stronger performance at lower inference cost than an equally large dense model might require.

But MoE did not provide free intelligence. Routers can overload popular experts. Systems need load balancing so that tokens are distributed efficiently. If experts sit on different GPUs, data must travel rapidly among devices. This is called expert parallelism, and it creates substantial communication complexity.

Why MoE saves compute but still needs memory

This is the apparent contradiction at the heart of MoE. The model activates only a few experts per token, so arithmetic work can fall. Yet all those experts still have weights that must be stored somewhere. A 600-billion-parameter sparse model does not magically fit into the memory footprint of a 30-billion-parameter model merely because only 30 billion parameters are active at one moment.

For hardware, this means MoE strengthens the importance of HBM capacity, memory bandwidth and interconnects. More capacity lets each accelerator hold more expert weights. More bandwidth moves those weights rapidly. Faster links allow tokens to reach experts located on other GPUs. Advanced packaging, NVLink-class connections, Ethernet or InfiniBand networks, optical components and system software all become part of model performance.

MoE shifts part of the bottleneck from pure compute toward memory placement and communication. It may reduce the number of calculations per token while increasing the sophistication—and sometimes the cost—of the entire system.

4. Reasoning Models: Giving AI More Time to Solve Problems

The next major development changed not only the model architecture, but also when computation is spent. Traditional scaling focused heavily on training: build a larger model, expose it to more data and invest more compute before deployment. Reasoning models add another lever: spend more computation after the user asks a difficult question.

This is called inference-time compute or test-time compute. Inference is the process of using a trained model. A simple request may require a short response. A difficult mathematical proof, programming task or research problem may benefit from generating intermediate steps, trying alternative approaches, checking results and revising errors.

The research path to reasoning models

Reasoning did not appear in one moment. In 2022, researchers showed that chain-of-thought prompting could improve performance by encouraging large language models to generate intermediate reasoning steps. Another technique, self-consistency, sampled multiple reasoning paths and selected the answer supported most consistently. Tool-using models learned to call calculators, code interpreters or search systems. Process-supervision research explored rewarding correct intermediate steps rather than judging only the final answer.

Commercial reasoning models brought these ideas together with specialised post-training and reinforcement learning. OpenAI’s o1, introduced in 2024, popularised the idea of a model that allocates extra internal computation before answering challenging questions. Because model developers do not disclose every architectural and training detail, it is better to view o1 as a major commercialisation of inference-time reasoning rather than the invention of reasoning itself.

DeepSeek-R1, released in 2025, became another landmark because its technical report described how large-scale reinforcement learning could produce sophisticated reasoning behaviour. DeepSeek-R1-Zero was trained with reinforcement learning without supervised fine-tuning as a preliminary step, although it suffered from readability and language-mixing problems. DeepSeek-R1 then used a multi-stage process with cold-start data and reinforcement learning. The project also released distilled smaller models, helping reasoning methods spread through the open-model community.

What breakthrough did reasoning create?

Reasoning models changed capability from something determined only during training into something partly adjustable during use. Consider two students with the same knowledge. One must answer in five seconds. The other receives thirty minutes, scratch paper, a calculator and permission to check the answer. The second student’s brain has not become physically larger, yet performance may improve substantially.

This approach has been especially powerful in mathematics, coding and other tasks where answers can be verified. Reinforcement learning with verifiable rewards can check whether code passes tests or whether a numerical answer is correct. A model can generate several solutions, use a verifier to rank them, execute code and revise failures.

Longer reasoning is not always better. A model can circle around the same idea, follow a wrong path with confidence or waste computation. Some research has found a “sweet spot” beyond which additional thought may hurt performance. Reasoning also adds latency. Users may wait longer, and service providers may pay for many more generated tokens or model passes.

Reasoning and the KV cache

To understand the hardware impact, we need one more term: the KV cache. In a Transformer, attention uses key and value representations derived from earlier tokens. During generation, the model stores these representations so it does not recalculate the entire history for every new token.

Inference usually has two phases. Prefill processes the prompt and can use parallel computing efficiently. Decode generates new tokens one after another. Decode often depends heavily on memory bandwidth because the system repeatedly reads model weights and the KV cache while producing each token.

Reasoning models may generate far more tokens per request, maintain longer contexts, evaluate several candidate solutions and call external tools. This increases the duration for which accelerators remain occupied. It also raises the value of HBM capacity, because larger memories can hold more model weights and KV-cache data, and of HBM bandwidth, because those data must move rapidly.

Hardware designers are responding with more memory and faster links. NVIDIA’s H200, for example, offers 141 GB of HBM3e and 4.8 TB per second of memory bandwidth. Blackwell Ultra systems have moved to still larger HBM capacity and bandwidth, explicitly targeting large models, long contexts and high-concurrency inference. These product changes illustrate how AI hardware is evolving from a narrow focus on training arithmetic toward a broader focus on serving reasoning systems efficiently.

5. How the Three Breakthroughs Connect

The three developments are not competing alternatives. Most MoE language models are still Transformers. A reasoning model may also be an MoE model. The progression is therefore cumulative.

The Transformer created a scalable engine for learning relationships. MoE expanded that engine with conditional computation. Reasoning models then ran the engine for longer, sometimes along several paths, when a problem justified the extra effort.

AI progress often resembles squeezing a balloon. Reduce pressure in one place and another part expands. Parallel training reduced the limits of sequential computation but created huge demand for accelerator clusters. MoE reduced arithmetic per active token but increased the importance of memory placement and communication. Reasoning improved difficult-task performance but increased inference duration, KV-cache use and latency.

6. What This Means for GPUs, HBM and Memory Chips

GPUs and AI accelerators

The Transformer made parallel matrix multiplication the central workload. MoE added routing and communication among experts. Reasoning increased the number and duration of inference steps. Together, these trends support demand not just for faster chips but for complete systems that combine compute, memory and networking.

Training remains extremely demanding, but inference is becoming a much larger strategic market. A training run is a giant project that eventually ends. Inference continues every time a customer asks a question, an employee uses a coding agent or a company runs an automated workflow.

HBM: the fast working memory beside the processor

HBM is especially important because accelerators need enormous quantities of data delivered quickly. Transformer training needs bandwidth for weights, activations and optimisation states. MoE needs capacity for expert weights and bandwidth to access selected experts. Reasoning needs capacity for long contexts and KV caches, plus bandwidth during sequential token generation.

This does not mean more HBM automatically creates better AI. Software efficiency, model design, networking and processor utilisation remain crucial. But insufficient HBM capacity can prevent a model from fitting efficiently, while insufficient bandwidth can leave expensive compute units waiting for data.

Conventional DRAM and NAND

Ordinary server DRAM also matters. It supports CPUs, databases, retrieval systems, user sessions and data staging outside the accelerator. Agentic AI may search corporate documents, query databases and coordinate several applications, creating memory demand across the entire server rather than only inside the GPU package.

NAND flash is slower than DRAM but far cheaper per bit. It stores training datasets, model checkpoints, vector databases, images, videos, logs and agent histories. Multimodal AI expands these storage requirements because video and audio consume much more space than text. However, NAND’s benefit is indirect and can vary widely by workload.

Networking and packaging

When one model spans many accelerators, networking becomes part of the computer. MoE can intensify this dependence because tokens may be routed to experts on different devices. Reasoning and agentic systems may also coordinate multiple models and tools. High-speed copper links, optical interconnects, switches and network software become essential.

Advanced packaging is equally important. HBM stacks need to sit close to accelerators, connected through complex packaging technologies. As compute dies and memory stacks grow, manufacturing yield, heat removal and package design can become limiting factors.

The rebound effect

Efficiency does not guarantee lower total hardware demand. Suppose MoE or quantisation halves the cost of an AI query. Companies may then deploy AI to ten times as many tasks. Faster and cheaper models can unlock coding agents, customer-service systems, scientific assistants and personalised education. This is similar to a Jevons-style rebound effect: lower cost per unit encourages so much additional use that total consumption rises.

7. A Student’s Science Project Inside the Data Centre

Imagine a high-school student asks an AI system: “Design an experiment to test how light colour affects plant growth. Analyse my spreadsheet, identify mistakes and prepare a five-minute presentation.”

The Transformer first connects the pieces of the request. It recognises that “my spreadsheet” refers to supplied data, that the experiment must control variables, and that the final output should fit five minutes. If images of the plants are included, a multimodal Transformer may process visual tokens alongside text.

If the system uses MoE, the router selects expert networks for different tokens. Data-analysis patterns may activate one set of experts, scientific writing another and presentation structure another. The experts are not necessarily labelled that neatly, but sparse routing allows the model to draw on a large pool of parameters without activating all of them.

The reasoning system then plans the work. It checks whether the sample size is sufficient, examines the spreadsheet for missing values, calculates averages, notices that watering levels were inconsistent, revises the proposed conclusion and creates a slide outline. It may call a calculator or code tool to verify the statistics.

Inside the data centre, the prompt enters the prefill phase. Model weights and input representations move through HBM. During each Transformer layer, the MoE router may send tokens to selected experts. If experts sit on different GPUs, high-speed links carry the token representations between them. As the model generates its plan and analysis, the KV cache grows. Every additional reasoning step uses accelerator time, memory capacity and bandwidth.

The final answer may appear as a simple school presentation, but behind it lies a coordinated system of algorithms, memory chips, packaging, networking, storage, power supplies and cooling equipment. Modern AI is not one giant brain. It is an industrial machine for moving and transforming information.

8. What Comes Next?

The Transformer may remain foundational for years, but it is unlikely to stand still. Researchers are testing hybrid systems that combine attention with state-space models, which can process long sequences differently. Diffusion language models aim to generate or revise multiple tokens in parallel rather than always proceeding strictly left to right. Better memory systems may help agents retain useful information without placing every past detail into the prompt.

MoE systems may become more flexible, selecting not only which expert to use but how much of that expert to activate. Reasoning systems may learn when to stop, avoiding unnecessary token generation. Smaller specialised models may handle routine tasks while frontier models supervise difficult cases. On-device AI could keep more personal data close to the user, while data-centre models perform demanding work.

Hardware may also become more memory-centred. Processing-in-memory concepts attempt to move some computation closer to stored data. Optical connections may reduce the energy and latency of communication. Custom inference chips may separate prompt processing from token generation, matching each phase with the most suitable architecture.

No single development is guaranteed to replace the Transformer, MoE or reasoning. The more likely future is a layered system combining them with retrieval, tools, multimodality and specialised hardware.

Conclusion: Intelligence Is Also a Data-Movement Problem

Return to our imaginary school. The Transformer rebuilt the classroom so every student could notice relevant information across the room. MoE expanded the school into a university of specialists and created a routing system that chose which departments should answer each question. Reasoning models gave the students additional time, scratch paper, verification tools and permission to revise their work.

Each advance made AI more capable, but none removed physical constraints. Transformers required huge parallel computers. MoE saved arithmetic while increasing the importance of memory capacity and communication. Reasoning improved complex problem-solving while consuming more inference time, KV-cache space and memory bandwidth.

That is why the AI story cannot be understood through software alone. The quality and economics of a model depend on GPUs and custom accelerators, but also on HBM, conventional DRAM, storage, advanced packaging, switches, optical links, electricity and cooling. A brilliant algorithm that cannot move its data efficiently will leave expensive processors waiting.

The future of AI will therefore be decided not only by how many calculations a chip can perform, but also by how quickly the system can move, store, retrieve and reuse information.

Selected Sources

1. Vaswani et al., “Attention Is All You Need”, 2017.
2. Brown et al., “Language Models are Few-Shot Learners”, 2020.
3. Kaplan et al., “Scaling Laws for Neural Language Models”, 2020.
4. Fedus, Zoph and Shazeer, “Switch Transformers”, 2021.
5. Mistral AI, “Mixtral of Experts”, 2023.
6. DeepSeek-AI, “DeepSeek-V3 Technical Report”, 2024.
7. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, 2022.
8. DeepSeek-AI, “DeepSeek-R1”, 2025.
9. NVIDIA, H200 GPU specifications.
10. NVIDIA Developer, Blackwell Ultra architecture and memory, 2025.

Editorial note: “First” claims in AI are often contested. This article distinguishes early academic concepts, large-scale demonstrations and models that popularised a technology. Hardware effects are directional analysis and vary by architecture, deployment design and workload.