Running a model on a device at the edge, rather than in a data centre, changes the rules. Power is limited, memory is small, cooling may be nonexistent and the hardware must survive the environment it is deployed in. A chip that looks excellent in a lab can be wrong for a battery-powered sensor or a vehicle. This guide takes you through the decision in the order that avoids the usual regrets: constraints first, hardware families second, validation last.

Start With the Constraints, Not the Chip

Write down the non-negotiables for the product.

• Power budget. Is the device battery-powered, solar-powered or mains-powered? How much power can the AI portion use, sustained and at peak?
• Thermal environment. Can it be cooled actively, or must it run passively in an enclosed case, possibly in hot or cold conditions?
• Latency. How quickly must a result be produced after an input arrives? Real-time control loops have far stricter needs than occasional classification.
• Model size and type. What does the model need in memory and in computation?
• Connectivity. Is the device always online, intermittently online or isolated? This decides how much must run locally.
• Cost and volume. The unit cost target and expected production quantity.
• Lifetime and support. How long must the part remain available and receive updates?

These constraints will eliminate many candidates before you look at a single performance number.

The Main Hardware Families

Edge AI platforms generally fall into a few groups, each with a characteristic balance.

Microcontrollers are tiny, low-power and inexpensive. They suit very small models, such as keyword detection or simple sensor classification, running on severely limited memory. Frameworks for small-footprint inference exist for them.

CPUs in application processors offer flexibility and ease of development. They can run modest models acceptably and often serve as the fallback for operations a specialised block does not support.

NPUs and dedicated AI blocks are accelerators for neural network operations, frequently integrated into a system-on-chip. They deliver much better efficiency on supported operations and quantised models, but they depend on a vendor toolchain.

Embedded GPUs handle larger and more varied models and vision pipelines, at higher power and cost.

FPGAs and custom silicon give maximum control and can be efficient for fixed, high-volume tasks, with greater design effort.

Compare Families Against Your Needs

Option	Typical fit	Main strength	Main caution
Microcontroller	Tiny always-on sensing	Very low power and cost	Strict model and memory limits
CPU only	Light models, prototypes	Simple development, flexible	Limited efficiency on larger models
NPU / AI block	Vision, audio on battery or fanless devices	High efficiency on supported operators	Toolchain and operator coverage
Embedded GPU	Heavier vision and multi-model pipelines	Programmable, higher throughput	Higher power and thermal demand
FPGA / custom	Fixed high-volume workloads	Tailored efficiency	Greater engineering effort

Software Support Decides Many Outcomes

At the edge, toolchains are often less uniform than in the data centre. Before committing, confirm using the vendor’s current documentation:

• Which model formats and frameworks can be converted for the target.
• Which operators are supported by the accelerator and which fall back to the CPU.
• What quantisation options are available and how accuracy is validated.
• How profiling, debugging and over-the-air updating work.

An unsupported operator in the middle of a model can force a slow transfer between accelerator and CPU, damaging both latency and power.

Do Not Forget the Rest of the System

The AI chip is one component. Sensors, image signal processing, memory, storage, radios and the operating system all draw power and affect latency. Sometimes the bottleneck is moving camera data into memory, not the neural network. Plan the whole data path.

Think about security too: secure boot, protected storage for model files and update mechanisms. Requirements differ by industry and jurisdiction, so involve qualified specialists where regulations apply.

Validate on Real Hardware Early

Specifications are a poor predictor of final behaviour, so obtain a development kit as soon as you have a shortlist.

• Run your actual model, converted and quantised as it would ship.
• Measure latency, accuracy after conversion, power draw and temperature under sustained load.
• Test in conditions resembling deployment, such as an enclosed case in a warm room.
• Evaluate the effort of the toolchain as a real cost.

A Short Hypothetical

A team designs a battery-powered camera that detects whether a doorway is occupied. Their power budget is tight and the enclosure is sealed, so they rule out hardware that needs active cooling. The model is a small convolutional network. They shortlist a low-power application processor with an integrated NPU and a microcontroller-class option. On a development kit, the NPU version runs their quantised model with comfortable latency, but one operator falls back to the CPU and consumes more power than expected. They revise the model to use only supported operators, re-measure and then decide. The early test saved them from discovering the issue after a board spin.

Frequently Asked Questions

What is an NPU? A neural processing unit is a block designed to accelerate neural network operations efficiently, commonly found inside system-on-chips for phones, cameras and embedded devices.

Is a bigger TOPS number better? Not by itself. It is a peak figure tied to a numeric format and ideal conditions. Operator support, memory and software determine actual results.

Do I need to quantise my model? Often yes, since many edge accelerators perform best with low-precision integer formats. Validate accuracy afterwards.

Should processing happen on the device or in the cloud? It depends on latency, connectivity, privacy and cost. Many products split the work, doing quick decisions locally and heavier analysis remotely.

How long will my chosen part stay available? Check the vendor’s longevity commitments and product lifecycle documentation before designing it into a long-lived product.

Next Step

Write your constraint list, with power, thermal, latency, memory and cost targets, on a single page, then order a development kit for your top candidate and measure your real model on it before any further design work.