AI chip datasheets are written to impress. They lead with the largest number the hardware can reach under the most favourable conditions, and they leave the rest for the footnotes. That does not make them dishonest, but it does mean you need a method for reading them. This guide gives you a working checklist, explains the terms that cause the most confusion, and shows how to turn a spec sheet into questions you can test.

Start With the Number That Is Easiest to Misread

The most prominent figure is usually peak arithmetic throughput, quoted as FLOPS for floating-point operations or TOPS for integer operations. Three caveats apply every time.

First, the number is tied to a specific numeric format. A figure quoted for a low-precision format is not comparable to one quoted for a higher-precision format, even if both use the same unit.

Second, it is a theoretical ceiling reached only when every arithmetic unit is busy every cycle. Real models rarely achieve that, because data must arrive in time.

Third, some figures assume special conditions, such as a particular sparsity pattern in the weights. If the footnote says the number depends on such a feature, ask whether your model can use it.

Memory Often Matters More Than Compute

For many workloads, especially large language model inference, the limiting factor is how fast weights can be read from memory, not how fast they can be multiplied. When you read a spec sheet, give memory at least as much attention as compute.

Look for two things: capacity, which decides whether your model fits on the device at all, and bandwidth, which decides how quickly it can be streamed through the arithmetic units. A chip with enormous compute and modest bandwidth will leave much of that compute idle on memory-bound tasks.

Power, Thermals and Form Factor

Quoted power figures may describe typical draw, a configurable limit or a maximum. These are different things. Power determines the cooling a server needs and the running cost over time, so find out which figure is being quoted and under what load.

Form factor matters too. An add-in card, a module for dense servers and an embedded board have different power and cooling assumptions, even when they share a name.

Interconnect and Scaling

If your model or workload will span more than one chip, the links between devices become part of the performance story. Check what the chip offers for device-to-device communication, how many links there are and how the vendor describes multi-node scaling. Scaling is rarely perfectly linear, so treat claims of ideal scaling with care.

A Reading Checklist

What to find	Why it matters	Question to ask
Numeric format behind each throughput figure	Figures are only comparable at equal precision	Which format will my model actually use?
Memory capacity	Decides whether the model fits	Does my model plus working memory fit with headroom?
Memory bandwidth	Often the real limit for inference	Is my workload compute-bound or memory-bound?
Power figure type	Typical, limit and maximum differ	Which one is quoted, and at what load?
Interconnect details	Affects multi-chip scaling	How does communication cost grow as I add devices?
Software support	Determines whether hardware is usable	Are my framework and operators supported?
Footnote conditions	Hide assumptions behind headline numbers	Does my model meet the stated conditions?

Check the Software Section Before the Silicon Section

A spec sheet that says little about software is a warning sign. Look for supported frameworks, supported operators, compiler or runtime tooling and how new model architectures get supported. Check the vendor’s current documentation rather than a third-party summary, because software support changes quickly.

If your model uses an unusual operator that the stack does not support, it may fall back to slow paths or fail to compile entirely. Finding that out on paper is cheaper than finding it out after purchase.

Turn the Sheet Into a Test Plan

The goal of reading a datasheet is not to decide, but to decide what to measure. Once you have worked through the checklist, build a short benchmark around your own model and data:

• Run the real model at the batch sizes and sequence lengths you expect in production.
• Record throughput, latency at the percentile you care about, memory use and power draw.
• Repeat on a second candidate under identical settings.
• Note how much tuning each required, since engineering time is a cost.

Synthetic benchmarks can inform you, but only your workload tells you what you will actually get.

A Short Hypothetical

Suppose an engineer compares two accelerators. One advertises a much higher peak throughput figure. The other advertises less compute but more memory bandwidth. The engineer’s workload is serving a language model with long contexts, which tends to be memory-bound. After reading the footnotes, she notices the higher figure assumes a low-precision format her model has not been validated against. She tests both on her actual model. The lower-compute, higher-bandwidth part serves requests at a better latency under her accuracy constraints. The headline number would have pointed her the wrong way.

Frequently Asked Questions

What is the difference between FLOPS and TOPS? FLOPS counts floating-point operations per second and TOPS counts integer operations per second. Both are only meaningful alongside the numeric format and the conditions under which they were measured.

Is higher peak throughput always better? No. Peak throughput is a ceiling, and real performance depends on memory, software and how well the workload maps onto the hardware.

Why do vendors quote several power numbers? Chips can be configured to run at different power limits, and typical consumption differs from the maximum. Always confirm which figure applies to the performance you are being shown.

Should I trust published benchmark results? Treat them as a starting point. Check what model, precision, batch size and software version were used, and whether they resemble your setup.

How important is software maturity for a new accelerator? Very. Even excellent hardware can be impractical if your framework, operators or deployment tools are not well supported. Check the vendor’s current documentation and test early.

Next Step

Pick one chip you are evaluating and fill in the checklist table above using its datasheet. Wherever a row stays blank or the answer depends on a footnote, turn that gap into a specific test on your own model before taking any claim at face value.