For most of the past decade, the fastest way to make an artificial intelligence model better was to make the chip underneath it bigger. That assumption is now being tested by physics, and the constraint is not the arithmetic units that perform multiplication and addition. It is the movement of data between memory and processor, a bottleneck that engineers have started calling the memory wall.
The numbers explain why. A modern accelerator can execute trillions of operations per second, but moving a value from off-chip memory into the compute core costs orders of magnitude more energy than the operation that follows. Training large models therefore spends much of its time waiting, not calculating. NIST and academic groups tracking hardware efficiency have described this imbalance as the central engineering problem for the next generation of systems.
Why bandwidth, not flops, sets the ceiling
Marketing for AI hardware still leans on floating point operations per second, a figure that describes how fast a chip can calculate once data has arrived. Inference and training both depend far more on how quickly data can be delivered. High bandwidth memory solved part of the problem by stacking memory dies next to the processor, shortening the distance that signals travel. That approach is now close to its practical limits in cost and packaging complexity.
When bandwidth runs short, the workaround is to compress, cache or recompute. Each option trades one resource for another. Compression reduces precision, which can degrade accuracy. Caching requires memory that is itself scarce. Recomputing throws away arithmetic capacity that was the whole point of the chip.
What the research community is trying
Several directions are competing. The first is closer integration, where memory and logic sit in the same package so that data never leaves the silicon. The second is in-memory computing, which performs simple operations directly inside the memory array. The third is a return to specialised architectures that avoid moving data at all when the pattern of the computation allows it.
Progress is uneven. In-memory computing works well for small, regular operations and struggles with the irregular access patterns of attention mechanisms used in language models. Package-level integration helps bandwidth but raises manufacturing cost and reduces the number of chips that can be produced from a single wafer. The IEEE has published a steady stream of work on these trade-offs, and none of it points to a single winner.
The software layer is being rewritten to compensate
Because hardware improvements are slow, much of the recent gain has come from software. Kernel fusion, better scheduling, and quantisation schemes that keep accuracy while shrinking the footprint of each number all reduce pressure on memory. Compilers now decide which values to keep resident and which to stream, a job that used to be left to the programmer.
This shift matters for anyone buying infrastructure. A cluster that looked underpowered on paper can outperform a larger one if the software is better at hiding latency. Benchmarks that measure only peak throughput tend to miss this entirely, which is why independent evaluation groups have argued for workload-specific testing rather than a single headline number.
Cost, power and the limits of scaling
Energy is the second wall. Data centres already consume a growing share of electricity in several countries, and the International Energy Agency has warned that the pace of new demand is outpacing the build-out of generation and transmission. A chip that is fast but inefficient multiplies that pressure.
Cooling adds another constraint. Dense packages concentrate heat in a small area, and removing it requires liquid systems that many existing facilities were never designed to host. Operators are retrofitting, but retrofits take years and capital that competes with building new capacity.
What this means for model developers
Teams that assumed compute would keep getting cheaper should plan for a period where the binding constraint is memory bandwidth and power. Practical responses include training smaller models on better data, using retrieval to avoid storing everything in parameters, and designing inference systems that keep working when a node is slow rather than assuming uniform speed.
The most useful habit is measurement. A team that knows where its time and energy actually go can often find more headroom than a team that simply buys more accelerators. Hardware roadmaps will keep improving, but the era in which raw compute alone decided outcomes is closing.
The measurement gap in hardware marketing
Vendors quote peak bandwidth and peak throughput, figures measured under conditions that rarely occur in production. A realistic picture requires measuring a specific model with a specific batch size on specific hardware, and publishing the configuration alongside the result.
Independent benchmark groups have pushed for exactly this discipline, and their reports are more useful than specification sheets for planning purposes. The gap between the two numbers is often large enough to change a procurement decision.
Where the next gains will come from
The likely winners are combinations rather than single breakthroughs: tighter packaging plus smarter compilers, plus models that are designed with memory in mind from the start. None of these is glamorous. Together they determine whether the current rate of progress can be sustained without an unsustainable increase in cost and power.
For now, the memory wall is a reminder that artificial intelligence is an industrial technology as much as a mathematical one. The limits are physical, the fixes are engineering, and the timeline is measured in years rather than months.
Image: Appaloosa · CC BY-SA 3.0 · via Wikimedia Commons.