A model trained in a data centre usually stores its numbers in 32-bit or 16-bit floating point. Many edge chips, by contrast, are fastest and most efficient when working with 8-bit integers or other low-precision formats. Quantization is the process of converting a model to those formats. Done carefully, it reduces model size, memory traffic and energy use with a modest cost in accuracy. Done carelessly, it quietly breaks the model. This article is a workflow you can follow, with the checks that catch problems early.

What Quantization Actually Does

Quantization maps a range of floating-point values onto a smaller set of integer levels, using a scale factor, and sometimes an offset, to translate between them. Weights, and often the intermediate values called activations, are represented with fewer bits.

The benefits follow directly:

• Smaller model files, because each number needs fewer bits.
• Less memory traffic, which matters because moving data is often costlier than computing on it.
• Faster, more efficient arithmetic on hardware that has integer units optimised for low precision.

The cost is rounding error. How much it hurts depends on the model and on how well the mapping fits the actual distribution of values.

Step One: Establish a Floating-Point Baseline

Before changing anything, measure the original model on a held-out evaluation set that reflects real deployment inputs. Record the accuracy metrics that matter for the product, plus latency and memory on the reference setup. Without a baseline you cannot tell how much quantization cost you.

Step Two: Check What Your Target Supports

Different edge platforms support different formats and different operators in quantized form. Using the vendor’s current documentation, confirm:

• Which numeric formats the accelerator handles natively.
• Whether it expects per-tensor or per-channel scaling.
• Which operators have quantized implementations and which will fall back to the CPU.
• What conversion tool and model format the vendor recommends.

Designing around the target’s capabilities from the start is much easier than retrofitting.

Step Three: Try Post-Training Quantization First

Post-training quantization converts an already trained model without retraining. It is the quickest route and works well for many models.

The usual procedure is:

1. Choose a calibration dataset, a modest set of representative inputs.
2. Run the model on that data to observe the range of values at each layer.
3. Use those ranges to pick scale factors.
4. Convert the model and evaluate it on your held-out set.

The calibration data matters. If it does not resemble real inputs, the chosen ranges will be wrong and accuracy will suffer. Include the variety you expect in the field, such as different lighting for vision or different noise for audio.

Step Four: Diagnose Accuracy Loss

If accuracy drops too far, find where. Layers differ in sensitivity: some tolerate low precision easily, while others, often the first and last layers or those with wide value ranges, are fragile.

Practical diagnostics include comparing layer outputs between the original and quantized model to locate the layer where error grows, inspecting activation ranges for outliers that stretch the scale, and trying per-channel instead of per-tensor scaling. Some tools allow keeping the most sensitive layers at higher precision while quantizing the rest, a mixed-precision approach, though your hardware must support it efficiently.

Step Five: Move to Quantization-Aware Training if Needed

When post-training methods leave too much accuracy on the table, quantization-aware training simulates the rounding effects during training so the model learns to compensate. It requires access to the training pipeline and data and takes more effort, but often recovers accuracy for difficult models.

Step Six: Validate on the Real Target

Simulated results on a workstation are not the same as running on the device. Deploy the converted model to the actual hardware and measure accuracy, latency, memory use, power draw and temperature under sustained operation. Confirm that the full model runs on the accelerator and has not partially fallen back to the CPU.

Workflow Checklist

Step	Goal	Check before moving on
Baseline	Measure original model	Accuracy, latency and memory recorded
Target review	Learn supported formats and operators	Unsupported operators identified
Post-training quantization	Quick conversion with calibration	Calibration data resembles field inputs
Diagnosis	Find sensitive layers	Cause of any accuracy loss identified
Quantization-aware training	Recover accuracy if necessary	Improvement confirmed on held-out data
On-device validation	Confirm real behaviour	Accuracy, latency and power meet requirements

Pitfalls to Watch For

• Unrepresentative calibration data. The most common cause of unexpected accuracy loss.
• Evaluating only on average accuracy. Check that rare but important classes or conditions are not disproportionately harmed.
• Ignoring pre- and post-processing. Input scaling and output decoding must match what the quantized model expects.
• Assuming fewer bits is always faster. If the hardware lacks efficient support for a format, the benefit may not appear.
• Skipping re-validation after updates. New model versions need fresh calibration and checks.

A Short Hypothetical

A team has a floating-point model that detects defects on a production line. They convert it with post-training quantization using a calibration set drawn mostly from daytime images. On the device, accuracy falls noticeably during night shifts. Inspecting the activations, they find that value ranges in dim conditions differ from the calibration data. They rebuild the calibration set to include images from all shifts, reconvert and re-evaluate. Accuracy returns to an acceptable level without any retraining. The model was never the problem; the calibration data was.

Frequently Asked Questions

Will quantization always reduce accuracy? Usually by a small amount, sometimes by a negligible one, and occasionally by a lot. Measuring on your own data is the only reliable way to know.

How much calibration data do I need? Enough to cover the variety of real inputs. Representativeness matters more than raw quantity; check your tool’s documentation for guidance.

Is 8-bit the only option? No. Some platforms support lower bit widths or mixed precision, but support and benefits depend on the hardware.

Do I need to retrain to quantize? Not always. Post-training quantization needs no retraining, while quantization-aware training does.

Does quantization help with speed on every device? Only where the hardware has efficient low-precision support. Verify on the target.

Next Step

Build a small, representative calibration and evaluation set from real field data, record your floating-point baseline, and run a first post-training quantization pass on your target toolchain. Let the measured accuracy gap decide whether you need anything more elaborate.