On-Device Inference Is Remaking the Sensor Stack

Key Takeaways

• With on-device inference, a sensor is judged more by what it hands to a nearby model than by how cleanly it streams. 
• Memory bandwidth, not peak TOPS, limits most edge designs, and 8-bit quantization is the lever for fitting a model into on-chip memory.
• In always-on designs, power is won with tiered wake-up: push each decision to the cheapest stage that can make it, because moving data costs more than the math.

Inference Moves Onto the Device

Edge AI is crossing from pilots into mainstream products in 2026. Of the roughly 21 billion connected IoT devices in service, most still send their data off-device for processing. But purpose-built silicon is moving that work onboard: the first STM32 microcontrollers with a dedicated neural processing unit (NPU) reached volume production at the end of 2024, and value-tier edge-AI modules have fallen to price points that make local inference an economical default.

Sensors Now Feed a Model

Once inference runs locally, a sensor is evaluated less by how cleanly it digitizes a signal and more by how useful its output is to the model beside it. This has three consequences:

• Cleaner, wider-range data. Models punish noise and clipping, so front ends are gaining headroom. Dual-range inertial measurement units (IMUs) capture subtle vibration and hard shock simultaneously, with no data lost between ranges. Microphone vendors now raise acoustic overload points along with signal-to-noise ratios, so a loud event does not saturate before the model sees it.
• Compute inside the sensor. Machine-learning cores and in-sensor processing units in IMUs, AI engines inside MEMS microphones, and on-die vision let the part emit a decision or a feature instead of a raw stream, so the host can stay asleep.
• Sparse, event-driven output. Event-based vision sensors and wake-on-sound microphones both use bandwidth and compute only when something changes, instead of streaming around the clock.

These developments also help protect privacy. Time-of-flight and radar report distance and motion, not images, and in-sensor vision keeps raw pixels on the die, so a presence sensor can do its job without a camera feed ever leaving the part.

Bandwidth, Not TOPS, Sets the Ceiling

The headline figure for an edge AI part is usually its TOPS rating (trillions of operations per second), and it’s the wrong thing to size a design around. Two accelerators rated at the same TOPS can post very different real throughput because performance depends on memory bandwidth, the operators the silicon supports, and how well the vendor’s compiler maps your network onto it. 

On most edge designs, moving tensors is the limiter, so the processor stalls waiting on memory well before it runs short of math. Energy per inference, not peak TOPS, is the number that tracks what a battery sees. It helps to think in two regimes, split by where the model lives: on-chip SRAM or external DRAM.

The Microcontroller Regime

On a microcontroller, the ceiling is on-chip SRAM: often just 256 KB to 512 KB. That budget has to hold the model’s activations and working buffers (the runtime tensors), so the model must be small and quantized to fit, with its weights kept in flash. Quantization is the lever: converting weights from 32-bit floating point to 8-bit integers shrinks a model by roughly three to four times with little accuracy loss, enough that a quantized MobileNet-class network lands in about 1 MB of flash where the float version would not.

The Edge SoC Regime

On a larger edge SoC running vision pipelines or a small language model, the constraint moves to external LPDDR, where capacity counts for less than how fast weights and activations can be streamed. Memory is a sourcing question too. The 2026 reallocation of DRAM capacity toward AI data centers has lifted LPDDR prices and tightened supply. Vendors fight the same battle on the bandwidth side: in May, Synaptics called memory bandwidth the primary bottleneck for on-device language-model inference and pointed to aggressive weight compression as the way to claw back throughput.

Wake the Big Chip Last

Staying always-on without draining the battery comes down to stacking detectors by cost: the expensive stages stay dark until a cheaper one earns the right to wake them. Whatever runs continuously sets the system’s idle draw, so the job is to make that bottom stage as cheap as possible and let it gate everything above it.

The Microphone Tier

A smart MEMS microphone runs a tiny neural network for voice-activity detection at tens of microamps, then escalates to a deeper model for keyword spotting only when it hears speech, and wakes the host only after that.

The Motion Tier

An IMU’s machine-learning core classifies activity on-sensor and raises an interrupt. The host processor and any accelerator stay in deep sleep until a class of interest appears.

The Presence Tier

A 60 GHz radar, such as Infineon’s BGT60LTR11AIP, or a multi-zone time-of-flight sensor handles presence detection with on-chip processing, waking a camera or a full vision pipeline only once it confirms a person is there.

The savings show up in how these products spend most of their time: idle. For a rough sense of scale, a system that keeps a host awake to listen draws in the milliamps, while a tiered front end idles at tens of microamps and draws milliamps only for a fraction of a second when something is happening. On a coin cell, the difference between always-on and tiered is the difference between days and years of life.

Put Intelligence at the Cheapest Stage

Across sensing, memory, and power, the design logic boils down to two principles that should drive your part selection:

• Resolve each decision as low in the stack as possible. Every classification an IMU, microphone, or presence sensor can settle on its own is one the host never has to wake for.
• Moving data costs more than the math. Bytes pushed across a bus or up to the cloud burn more energy than the computation itself, so prefer parts that report results rather than raw samples. 

Each function you push into the sensor is one that the host processor, the memory, and the interface no longer carry. The trade-off is flexibility. Push intelligence into the sensor when the task is stable and well understood. Keep it on the host when the model will change often or needs retraining in the field, where reflashing firmware beats respinning a fixed engine. 

For guidance on where in the stack each piece of intelligence belongs, see The Edge-Sensing BOM: What to Spec, from IMUs to Interfaces, which works through the BOM line by line: IMUs, time-of-flight, radar, microphones, sensor hubs, memory, and interfaces.