In data centres the competition between accelerators is largely about throughput, measured in operations per second and constrained by power and cooling. At the edge, in cameras, sensors and handheld devices, the competition is different: the winner is usually the chip that does the job within a power budget measured in single digit watts.
This changes the design objectives. A chip that is ten times faster but draws ten times the power is useless in a battery powered device. Engineers optimise for performance per watt, and increasingly for performance per dollar, because edge deployments often involve thousands of units.
Architectures built for the constraint
Several approaches compete. General purpose graphics processors are flexible and widely supported, but they carry overhead. Digital signal processors are efficient at the arithmetic common in vision, and field programmable gate arrays allow a design to be tailored, at the cost of complexity.
Application specific accelerators give the best efficiency for a fixed set of operations, and the worst flexibility when the model changes. Because models do change, vendors now ship reconfigurable blocks and rely on compilers to map new networks onto existing silicon.
The compiler is the product
Hardware specifications attract attention, and the software toolchain decides whether a chip is usable. A compiler must take a trained model and produce efficient code for the target, handling quantisation, memory layout and operator fusion.
Vendors that invest in this see adoption, because developers choose the platform where a model runs without weeks of manual tuning. Support for common frameworks matters more than peak numbers for most buyers.
Quantisation and its trade offs
Reducing numerical precision shrinks memory use and speeds computation, at some cost in accuracy. Eight bit integers are now routine, and lower precision formats are appearing for models that tolerate them. The trick is selecting per layer precision so that accuracy is preserved where it matters.
Measurement is essential here. A model that loses two points of accuracy may be fine for photo tagging and unacceptable for medical triage, and the acceptable threshold depends entirely on the application.
Memory bandwidth at the edge
Edge devices rarely have dedicated high bandwidth memory, so inference is often limited by how quickly weights can be read. Techniques such as weight compression and caching help, and architectures that keep data on chip reduce trips to external memory.
This is why small models remain attractive even when larger ones would be more accurate. A model that fits in on chip memory runs faster and cooler than one that must be streamed from external storage.
Where the market is heading
Standards efforts and open model formats reduce lock in, and the availability of public benchmarks such as MLCommons inference results allows comparison across vendors. The benchmarks are imperfect, but they are better than marketing claims.
Expect continued fragmentation, because the applications differ so much. A camera that detects motion has different needs from a hearing aid that isolates speech, and no single architecture serves both well.
Security and the update path at the edge
An edge device is physically accessible, which changes the threat model. Anyone with the hardware can attempt to extract weights, probe memory or modify the software. Secure boot, encrypted storage and hardware key stores raise the cost of these attacks, and none of them is universal.
Updating a fleet is the second challenge. Devices may be offline, behind restrictive networks or simply forgotten, so a vendor needs a mechanism that works without a reliable connection and a policy for how long support will last.
This is where the efficiency comparison becomes more complicated. A chip that is slightly less efficient but comes with a long support commitment and a working update mechanism is often the better choice for a deployment expected to run for a decade.
Regulation is beginning to formalise this expectation. Requirements for security updates and vulnerability handling now appear in legislation covering connected products in several markets, which turns the update path from a vendor preference into an obligation with a defined minimum period. Buyers should therefore ask for the support timeline in writing and check how previous generations of the same vendor’s silicon fared after their successors launched.
How to evaluate a platform
Start from the workload. Measure latency and power on the actual model, not on a reference network. Check the compiler’s support for the operations the model uses, and ask what happens when the model is updated. Confirm the memory footprint and the thermal envelope under sustained load.
Then consider the supply chain and the software support lifetime, because an edge device may remain in service for a decade. A marginally slower chip with a long support commitment is often the better choice than a faster one that will be abandoned. Ask also how the vendor handles a critical vulnerability discovered after the product ships, because that scenario reveals more about a platform than any benchmark result.
Image: Onorai · CC BY-SA 3.0 · via Wikimedia Commons.