Datasheets tell you what a board could do. A benchmark tells you what your model does on it, and the two can be far apart. Running your own tests before committing to a design is cheap compared with discovering the gap after boards are ordered.

Decide What You Are Trying to Learn

Before touching hardware, write down the questions the benchmark must answer. Typical ones:

• Does the model run on this hardware at all, with every operator supported?
• Does it meet the latency requirement, measured end to end?
• Does accuracy hold after conversion for this device?
• How much power does it draw while running, and does it stay cool enough?
• How much memory does it use?

A benchmark without questions turns into a pile of numbers nobody trusts. A benchmark with a pass or fail threshold per question turns into a decision.

Measure the Whole Pipeline, Not Only the Model

A common trap is to time only the model’s inference call. In a real product, a request includes capturing or receiving input, preprocessing, running the model, postprocessing and delivering the result. On a small device, preprocessing such as resizing or format conversion can be a substantial portion of the total, particularly if it runs on a slow CPU.

Time each stage separately, then the whole. If the model is fast and the pipeline is slow, the fix is in the surrounding code, not in choosing different silicon.

Get the Measurement Mechanics Right

Several details quietly distort edge benchmarks.

Warm-up. The first few runs are often slower because of loading, memory allocation or compilation. Discard them and measure steady-state behaviour, but separately record start-up time if cold start matters to your product.

Enough iterations. A single run is not a measurement. Collect many and look at the distribution.

Percentiles, not just averages. Averages hide spikes. If your application must respond within a deadline, the slow tail is what users and downstream systems experience. Record median and high percentiles.

Realistic inputs. Use data that resembles production, in the same format and size. A tiny test image or a constant tensor can behave differently from real input, especially with data-dependent models.

Batch size. Edge systems frequently process one input at a time. Benchmark at the batch size you will actually use.

Thermal state. Test both cold and after sustained operation, since throttling can change results, as covered in guidance on power and thermal budgeting.

Power settings. Record which power mode the board was in. Different modes can change results substantially, and the default is not always what you will ship.

A Hypothetical Test Plan

Suppose a team is choosing between two candidate edge modules for a voice-command model. They define success as a response within a fixed time budget, a small accuracy drop versus the full-precision reference, and a power ceiling.

For each module they follow the same steps: convert the model with the vendor’s toolchain; check that every operator maps to the accelerator and note any that fall back to the CPU; run a recorded set of real audio clips through the full pipeline; log latency distribution, accuracy against labels, power at the supply and temperature over a long run; and repeat in the final enclosure if available.

Module one is faster on the model alone, but two operators fall back to the CPU, and the median is fine while the slow tail misses the budget. Module two is slower on average but consistent, and stays within the budget at the tail. The team picks module two, a result they would not have reached from average-latency numbers or datasheet peak figures.

What to Record

Item	How to capture	Why it matters
Operator coverage	Toolchain conversion report	Fallbacks to CPU can dominate latency
Latency distribution	Many timed runs, median and tail	Deadlines depend on the tail
Pipeline stage times	Timers around each stage	Finds non-model bottlenecks
Accuracy after conversion	Evaluation on held-out data	Speed means little if quality drops
Power draw	Measurement at the supply	Battery life and thermal margin
Temperature and clocks	Logging during long run	Reveals throttling
Memory use	Runtime or system tooling	Determines headroom for the application

Common Failure Modes

• Unsupported operators. The model converts but silently runs parts on a slower processor. Always inspect the conversion report.
• Accuracy regression after quantization. Verify on realistic data, not on the calibration set alone.
• Benchmark-only software configuration. Test with the same software stack and settings you will ship.
• Comparing boards under different conditions. Keep the model, inputs, power mode and ambient conditions identical across candidates.
• Stopping at the first good number. One good run on one day proves little. Repeat, and check variance.

Knowing When to Stop

Testing can expand forever. Stop when each question you wrote down has an answer with margin, or when one candidate fails a hard requirement. The goal is a decision, not a perfect characterisation. If two candidates both pass with comfortable margin, choose on secondary factors like supply, tooling familiarity and cost, and move on.

Frequently Asked Questions

Should I trust the vendor’s published benchmarks? Use them to shortlist, not to decide. They typically use standard models and ideal conditions, which may not match yours.

How many runs are enough? Enough that the median and tail percentiles stop changing noticeably. The exact count depends on variability, so check by comparing two separate sets.

Do I need the final enclosure to benchmark? For performance sensitive to heat, yes. An open board in a cool room can overstate sustained performance.

What if my model does not convert cleanly? Check which operators fail, whether alternatives exist in the model, and the vendor’s current documentation on supported operations. Sometimes small model changes fix it.

Is a development board representative of the final product? Often close, but not always. Memory configuration, cooling and power delivery can differ, so confirm on production-representative hardware.

Next Step

Write a one-page test plan with your five pass or fail questions and thresholds, then run the same plan on your top two candidates with identical inputs. Make the decision from those results.