Open the technical documentation for several AI chips and you will meet the same few ideas in different clothes. Systolic arrays, SIMD and SIMT lanes, tensor units and spatial dataflow fabrics all aim at the same goal: do a huge number of multiply-adds per cycle without drowning in data movement. Understanding the differences helps you read architecture papers, anticipate programming constraints and judge marketing claims. This article compares the main styles on the dimensions that matter in practice.

The Shared Goal

Every one of these designs exploits the fact that neural network layers are dominated by matrix multiplication. They differ in how they organise the arithmetic units, how data flows between them and how much control logic they carry. Those choices set the balance between raw efficiency and flexibility.

SIMD and SIMT: Many Lanes, One Instruction

In a SIMD design, a single instruction operates on a vector of values at once. In the SIMT variant used by GPUs, many lightweight threads execute the same instruction in groups, while the hardware tracks them individually so they can diverge when needed.

The strengths are generality and a mature programming model. The same hardware can run graphics, scientific code and many kinds of AI operator. The cost is that each operation typically reads operands from registers and writes results back, which spends energy on data movement that more specialised designs avoid. Modern GPUs add dedicated matrix units alongside the general lanes to reclaim that efficiency for the dominant operation.

Systolic Arrays: Data Pulses Through a Grid

A systolic array is a grid of simple multiply-accumulate cells. Inputs enter at the edges and move rhythmically from cell to neighbouring cell, with each cell combining the arriving value with its own and passing data onward. Weights may stay fixed in the cells while activations flow past them, or the reverse.

The appeal is data reuse. A value fetched once is used by many cells as it travels through the grid, so far fewer trips to memory are needed per operation. The grid is also simple and regular, which makes it area and power efficient for large matrix multiplications.

The limits are rigidity. A systolic array shines when the matrix dimensions fill the grid well. Small, oddly shaped or irregular operations can leave cells idle. Operations that are not matrix multiplication, such as certain activations or normalisation steps, usually need separate supporting units.

Dataflow and Spatial Architectures: Map the Graph onto the Chip

Dataflow designs take a different view. Instead of fetching instructions and streaming data through a fixed array, they lay out the computation graph across a fabric of compute and memory tiles, and values move directly from producer to consumer as soon as they are ready. Intermediate results can stay on chip rather than round-tripping to external memory.

The potential benefit is high utilisation for models that map well onto the fabric. The challenge is the compiler. Placing and routing a large model across the chip is a hard optimisation problem, and the quality of the compiler largely determines the real-world result. Behaviour on models the compiler has not seen before can be less predictable than on a general-purpose design.

Comparison Table

Style	Core idea	Typical strength	Typical limitation
SIMD / SIMT	Many lanes execute shared instructions	Flexible, mature software ecosystem	More register traffic per operation
Systolic array	Data pulses through a grid of MAC cells	Efficient data reuse on large matrices	Less efficient on small or irregular shapes
Dataflow / spatial	Graph mapped across compute and memory tiles	Keeps intermediates on chip, high utilisation potential	Heavily dependent on compiler quality
Hybrid designs	General lanes plus dedicated matrix units	Balance of flexibility and efficiency	Complexity in scheduling across unit types

Real products rarely fit a single row. Most combine several ideas, so treat the table as a vocabulary rather than a classification of any specific chip.

What This Means for Programmers

The architecture shows up in how you write and tune code.

• On flexible SIMT hardware, you can often write custom kernels and expect reasonable behaviour, though tuning for memory access patterns still matters.
• On systolic-style hardware, performance depends on shaping your tensors to suit the array, such as padding dimensions to match tile sizes. Frameworks and compilers frequently do this for you.
• On dataflow hardware, you typically hand the whole model to a vendor compiler and rely on it. Custom operators may be more constrained.

In every case, check the vendor’s current documentation for supported operators and recommended shapes.

Why Flexibility Is Worth Real Weight

Model architectures change faster than chips do. A design tuned tightly for today’s dominant layer type may be less comfortable with a new attention variant or a new kind of sparsity. Flexibility acts as insurance. Specialisation buys efficiency now; generality buys resilience later. Neither is universally right, and the best answer depends on how stable your workload is.

A Short Hypothetical

Imagine a team with a fixed, well-understood vision model that will run unchanged for a long time at large scale. A highly specialised matrix-oriented accelerator may suit them well, because efficiency on one stable workload is the priority. Now imagine a research group exploring new model structures every few weeks. They would likely value a flexible programming model that handles unfamiliar operators without waiting on compiler support. The same technical comparison leads to different choices because the workloads differ.

Frequently Asked Questions

Are tensor cores a systolic array? They are matrix-multiply units inside a larger flexible processor. Implementation details vary by vendor and generation, and not all are published, so avoid assuming a specific internal structure.

Why is data reuse so important? Moving data costs more time and energy than arithmetic. Designs that reuse each loaded value many times spend less on movement and get closer to their compute ceiling.

Can a specialised chip run any model? Usually many, but not all equally well. Unsupported operators may fall back to slower paths or require changes to the model.

Does a better architecture always mean better performance? No. Software, memory, manufacturing process and system integration all affect the result. Measure your own workload.

Where do CPUs fit in this picture? CPUs use SIMD extensions and some include matrix instructions, but they prioritise general-purpose performance and generally trail dedicated designs on large matrix workloads.

Next Step

Pick one architecture paper or vendor whitepaper and classify each compute block it describes using the vocabulary above. Doing this once makes the next architecture description far easier to read critically.