When people talk about memory in AI hardware, they usually mean the large external memory attached to the chip. The small memories inside the chip get less attention, and they deserve more, because most of the efficiency tricks in accelerator design happen there.

Why There Is a Hierarchy at All

Fast memory is physically large and expensive per bit. Slow memory is cheap and dense but takes longer to reach and costs more energy per access. No design can afford a huge pool of the fast kind, so chips layer memories: a little very fast storage near the compute units, more moderately fast storage on the chip, and a much larger external pool.

The performance consequence is simple. Moving data costs time and energy, often more than the arithmetic performed on it. A well-designed accelerator keeps data close to the compute units for as long as it is useful and touches external memory as rarely as it can.

The Main Players

Registers and local buffers sit right next to the arithmetic units. They hold the operands being used right now. Their capacity is tiny, and the dataflow design of the chip largely decides what lives here.

On-chip SRAM is the working space shared across a group of compute units. SRAM is fast and needs no refresh, but it takes a lot of silicon area, which is why capacities are modest compared with external memory. Check the vendor’s current documentation for actual sizes, since they vary widely by product.

Caches are SRAM managed automatically by hardware. The chip decides what to keep based on recent access patterns. This is convenient for programmers and works well when access patterns are irregular.

Scratchpads are SRAM managed explicitly by software, meaning the compiler or the programmer decides what to load and when. They avoid some hardware overhead and allow precise planning, but they push the burden onto the toolchain.

External memory holds weights, activations that do not fit on chip, and everything else. Its bandwidth and capacity set hard limits that on-chip tricks can only soften.

Cache or Scratchpad: What the Choice Means for You

Neural network workloads are unusually predictable. The sequence of operations is known in advance, so a compiler can plan data movement ahead of time. That predictability is why many AI accelerators lean toward scratchpads and explicit data movement, while general-purpose processors lean on caches.

For you as a user, the practical effect is that scratchpad-based chips depend heavily on compiler quality. A good compiler will schedule transfers so that compute rarely waits. A weaker one leaves the hardware idle. Cache-based designs are more forgiving of unusual code but may show less predictable performance.

Tiling: The Technique Behind It All

Large tensors rarely fit in on-chip memory, so the work is divided into tiles small enough to fit. The chip loads a tile, does as much work on it as possible, writes results, and moves on. The art is choosing tile shapes that maximise reuse of the data already loaded.

If a tile of weights can be used against many inputs before being replaced, each trip to external memory pays off many times. If tiles are poorly shaped, the same data is fetched repeatedly. This is why identical arithmetic can run at very different speeds depending on how it is scheduled.

A Hypothetical Example

Consider a layer whose weights are too large for the chip’s on-chip memory in one piece. A naive schedule loads a slice of weights, processes one input, then discards the slice and loads the next, so every weight crosses the external memory interface once per input.

A better schedule keeps a slice of weights resident and streams a whole batch of inputs through it before moving on. Now each weight crosses the interface once for the entire batch. The arithmetic is unchanged; the external traffic drops sharply. This is also a clear illustration of why batching often improves efficiency, and why latency-sensitive single-request workloads can look worse on the same chip.

Memory Approaches Compared

Approach	Who manages it	Strength	Weakness
Cache	Hardware, automatically	Easy to program, handles irregular access	Less predictable, uses area for control logic
Scratchpad	Compiler or programmer	Predictable, efficient when well planned	Depends on tooling quality
Registers and local buffers	Dataflow design	Lowest energy per access	Very small capacity
External memory	Memory controller	Large capacity	Highest cost per access

What You Can Do With This Knowledge

You rarely control the memory hierarchy, but you influence how well your workload uses it.

• Prefer operator fusion where your toolchain supports it. Fusing adjacent operations keeps intermediate results on chip instead of writing them out and reading them back.
• Watch for oversized intermediates. Large activation tensors that cannot stay on chip create external traffic you may be able to reduce by restructuring or choosing different batch sizes.
• Try several batch sizes. The best setting often reflects where tiles fit comfortably, not where the arithmetic units are busiest in theory.
• Use the profiler. Look for memory-bound regions rather than assuming compute is the issue.

Frequently Asked Questions

Why not just add more SRAM to the chip? SRAM takes substantial die area and that area competes with compute units. More SRAM means fewer arithmetic units or a larger, costlier chip. Designers balance the two based on the workloads they target.

Is a scratchpad always better than a cache for AI? No. Scratchpads suit predictable workloads and reward strong compilers. Caches suit varied access patterns. Many chips use a mix.

What does it mean when a workload is “memory-bound”? The arithmetic units spend significant time waiting for data. Making the arithmetic faster would not help; moving less data or moving it more efficiently would.

Does a larger on-chip memory always mean better performance? Only if your workload can use it. If your tensors already fit, extra capacity does nothing. If they do not, it can reduce external traffic noticeably.

Where can I see how my model uses on-chip memory? Vendor profiling tools typically show utilisation and data movement. Their interface varies, so check the current documentation for your platform.

Next Step

Run your model under the vendor’s profiler and identify the three layers with the most external memory traffic. Then try one change on each, such as a different batch size or enabling fusion, and compare the measured results.