Almost every AI chip spec lists several throughput numbers, one for each numeric format. Understanding what those formats are explains a large part of why modern accelerators are fast, and why a headline figure can be misleading if you do not know which format it refers to.

What a Number Format Actually Is

A floating-point number is stored as three parts: a sign, an exponent and a mantissa. The exponent sets the range, meaning how large or small a value can be. The mantissa sets the precision, meaning how finely values within that range are distinguished.

Different formats split a fixed number of bits between those parts differently. Fewer total bits means less storage, less data to move and smaller arithmetic circuits, which is the whole reason lower precision is attractive.

The Formats You Will See

FP32 uses 32 bits and has generous range and precision. It was the long-time default for training and remains a safe reference point.

FP16 uses 16 bits with a smaller exponent range. It halves memory use, but values can overflow or underflow, which is why training in FP16 often needs loss scaling to keep small gradients from vanishing.

BF16 also uses 16 bits but keeps the same exponent range as FP32 and gives up mantissa precision instead. That tradeoff suits neural networks well, because range problems tend to hurt training more than coarse rounding does. This is why BF16 became popular for training.

FP8 variants use 8 bits, and there are typically two layouts: one favouring range and one favouring precision. Support and conventions differ across hardware and software, so check the vendor’s current documentation for exactly which variants are available and how they are used.

INT8 is an 8-bit integer. It has no exponent at all, so values are mapped onto a fixed grid through a scale factor. It is common for inference, particularly on edge devices.

Lower formats, such as 4-bit, appear mostly in inference for weights, often combined with careful grouping and scaling. They are more sensitive to model and method.

Why Lower Precision Is Faster

There are three separate effects, and it helps to keep them apart.

1. Less data to move. Half as many bits means half the memory traffic for the same number of values. For memory-bound workloads, this alone can help substantially.
2. More arithmetic per unit of silicon. Smaller multipliers take less area, so a chip can include more of them.
3. Lower energy per operation. Smaller circuits and less data movement both cut energy use.

Notice that the first effect applies even if compute is not your limit. Many large-model inference workloads gain mainly from reduced memory traffic.

Mixed Precision: Not Everything Needs the Same Treatment

Rarely is a whole model run in one format. Mixed-precision schemes keep sensitive parts in higher precision while using lower precision for the bulk of the work. A common pattern is to do matrix multiplications in a 16-bit or 8-bit format while accumulating results in 32 bits, and to keep a master copy of weights in higher precision during training.

The reason is that accumulating many small products in low precision loses information quickly. Wide accumulators protect against this at little cost.

Choosing a Precision

Situation	Typical starting point	Main risk
Training from scratch	BF16 with FP32 accumulation	Instability if sensitive ops are downcast
Experimenting with lower-precision training	FP8 where hardware and framework support it	Needs careful scaling and validation
Server inference	FP16 or BF16, then test lower	Accuracy drift when pushing lower
Edge inference	INT8 after calibration	Accuracy loss on outlier-heavy layers
Very memory-limited deployment	Lower-bit weights with grouped scaling	Sensitivity to model and method

Treat this as a starting hypothesis. The right answer depends on your model, and you should confirm it with a measurement.

A Hypothetical Validation Routine

Say a team has a model working well in 16-bit and wants to try INT8 for deployment. A sensible routine looks like this.

First, record the 16-bit model’s results on a held-out evaluation set. This is the baseline. Second, convert to INT8 using a calibration set that resembles production data, not a random sample of unrelated inputs. Third, evaluate again and compare against the baseline using the metric the product actually cares about, not only a generic accuracy score.

If a specific layer causes most of the loss, keep that layer in higher precision and quantize the rest. Many toolchains support per-layer overrides. If the loss is spread across the model, calibration data is the first thing to revisit.

The key discipline is deciding in advance what accuracy drop is acceptable. Without a threshold, any result can be rationalised.

Common Pitfalls

• Comparing speed claims at different formats. A chip’s FP8 number and another chip’s BF16 number are not interchangeable.
• Assuming a format is supported end to end. Hardware may support it while your framework version or kernel library does not.
• Skipping calibration data quality. Quantization scales built from unrepresentative data produce poor results that look like model problems.
• Testing only average accuracy. Rare but important cases can degrade even when the average barely moves.

Frequently Asked Questions

Why did BF16 replace FP16 for much training work? BF16 keeps the wide exponent range of FP32, which avoids many overflow and underflow problems, so it usually needs less special handling than FP16.

Does lower precision always make my model faster? Not always. The speedup depends on whether the workload is limited by compute or memory, and on whether your software stack has well-optimised kernels for that format.

Will lower precision hurt my model’s accuracy? It might, and the only reliable way to know is to evaluate on your own data. Many models tolerate 8-bit inference well; some layers or tasks do not.

What is the difference between quantization and mixed precision? Quantization maps values to a lower-precision representation, often integers. Mixed precision combines several formats within one model or computation. They are often used together.

Is FP4 or INT4 ready for general use? It depends on the model, the method and the hardware. Treat it as something to validate carefully rather than assume, and check current vendor guidance.

Next Step

Pick one model you run today, record its accuracy and speed at its current precision, and test the next step down on a representative evaluation set. Write down the acceptable accuracy drop before you look at the result.