Adding more arithmetic units to a chip is relatively straightforward. Feeding them is the hard part. In modern AI hardware, a large share of the design effort goes into moving data rather than computing on it, and for many real workloads the arithmetic units spend a good portion of their time waiting. This article explains why, gives you a simple mental model for diagnosing it, and covers the techniques chip designers and software engineers use in response.

The Basic Imbalance

Arithmetic on a chip has become cheap relative to the cost of fetching the operands. Reading a value from off-chip memory takes far more time and energy than performing a multiplication on it. Over many hardware generations, compute capability has tended to grow faster than memory bandwidth, a gap often called the memory wall.

For AI, this matters because models are large. The weights of a modern network can fill the memory of a device, and generating each output may require reading a large part of them.

Arithmetic Intensity: The Concept That Explains Most of It

Arithmetic intensity is the number of operations a program performs for each byte it moves from memory. It is the single most useful idea for reasoning about this problem.

• A workload with high arithmetic intensity reuses each loaded value many times. Large matrix-matrix multiplications are like this. Such a workload is compute-bound: the arithmetic units are the limit.
• A workload with low arithmetic intensity uses each loaded value only a few times. Generating text one token at a time for a single user is like this, because each weight is read once to produce one small result. Such a workload is memory-bound: bandwidth is the limit.

The same chip can be compute-bound on one task and memory-bound on another. Batching many requests together raises intensity, because one read of the weights serves many inputs. That is why batch size has such a strong effect on how efficiently a chip is used.

The Roofline Idea in Plain Terms

The roofline model plots achievable performance against arithmetic intensity. Performance climbs with intensity while bandwidth is the limit, then flattens at the chip’s compute ceiling. You do not need exact numbers to use the idea. If your workload sits on the rising slope, buying more compute does nothing; you need more bandwidth, better data reuse or less data to move.

The Memory Hierarchy on an AI Chip

Designers respond with layers of storage, each faster and smaller than the one below.

• Registers and small local buffers sit next to the arithmetic units and hold values being used right now.
• On-chip SRAM acts as a fast working area. It is expensive in silicon area, so capacity is limited.
• High-bandwidth off-chip memory, stacked and placed very close to the processor, supplies much higher bandwidth than conventional memory for the same board space.
• Host or system memory is large but far slower to reach.

Good performance comes from keeping data in the faster layers as long as possible and reusing it before it is evicted.

Techniques That Reduce Memory Pressure

Technique	What it does	Trade-off
Batching	Reuses each weight read across many inputs	Can increase latency for individual requests
Operator fusion	Combines steps so intermediate results stay on chip	Needs compiler or kernel support
Lower-precision weights	Shrinks the bytes that must be moved	Requires validating accuracy
Tiling	Processes data in blocks that fit in fast memory	Adds software complexity
Caching intermediate state	Avoids recomputing or rereading data	Consumes memory capacity
Model parallelism	Splits weights across devices	Introduces communication cost

None of these is free, and several interact. Fusing operators, for instance, helps most when the compiler can see the whole graph.

How to Tell Which Limit You Are Hitting

You can reason about it without special tools. Ask these questions about a given workload:

• Does throughput improve noticeably as you increase batch size? If so, you were likely memory-bound at the smaller size.
• Does lowering numeric precision speed things up considerably? That suggests data movement was the cost.
• Do profiler views show arithmetic units idle while memory transfers dominate? That is direct evidence.
• Does throughput barely change when you reduce compute work but keep the data volume constant? Again, memory is the likely limit.

Vendor profiling tools can show utilisation of compute and memory separately. Check the vendor’s current documentation for what your platform exposes.

A Short Hypothetical

Consider a team serving a language model to a handful of users at once. Their accelerator has plenty of arithmetic capacity, yet utilisation of the compute units looks low. Profiling shows long stretches spent reading weights. They try two changes: grouping incoming requests into slightly larger batches, and storing weights in a lower-precision format after confirming that output quality holds up on their evaluation set. Throughput improves because each weight read now serves more requests and the total bytes moved drop. The chip did not change; the data movement did.

What This Means for Choosing Hardware

When you compare accelerators for a memory-bound task, memory capacity and bandwidth deserve at least equal weight with compute figures. For training, which usually has higher intensity, compute and inter-chip communication tend to matter more. Know which regime your workload lives in before reading the datasheet.

Frequently Asked Questions

What is the memory wall? It is the growing gap between how fast processors can compute and how fast memory can supply them with data. It means that data movement, not arithmetic, often sets the practical speed limit.

Why does batching improve efficiency? Loading the weights once and applying them to many inputs raises arithmetic intensity, so the same memory traffic supports more useful work.

Is on-chip memory just a bigger cache? Not exactly. Many AI chips expose on-chip memory as a software-managed scratchpad, so the compiler decides what to keep there rather than relying on automatic caching.

Does lower precision always help? It reduces the bytes moved, which usually helps, but accuracy can suffer. Validate on your own data and follow the vendor’s current guidance on supported formats.

Is bandwidth the only memory metric that matters? No. Capacity decides whether the model fits, and latency matters for some access patterns. Bandwidth is simply the one most often underestimated.

Next Step

Take a model you run and measure throughput at several batch sizes. If throughput rises steeply with batch size, your workload is memory-bound, and your next efforts belong in data reuse, precision and memory-aware hardware choices rather than more raw compute.