Edge AI's Real Bottleneck Is Bandwidth, Not Compute, Engineers Say

RAM prices, not raw compute, are reshaping how engineers build AI for edge devices. Bandwidth constraints have made memory the expensive commodity, and that single fact has redirected research toward shrinking models rather than enlarging processors.

"People are trying to work out how they can cram an entire container of liquid into a very small cup, and that has been driven by the fact that compute resources are not the main problem at the moment," said Ed Plowman, CTO at Imagination Technologies. "It's bandwidth, and that's why RAM has gotten so expensive."

The AI industry now faces two separate problems. Hyperscale data centers chase efficient processing for high-parameter large language models. Edge devices — earbuds, phones, home appliances, industrial IoT, robotics, automotive systems — lack the area, power, and memory for such models. Oversized models slow devices down or overheat them. Undersized ones leave the product not smart enough.

What does right-sizing actually require?

The first step is defining the model's purpose. Designers then apply a mix of compression techniques rather than one method, because each trades size against accuracy differently.

• Distillation trains a smaller student model to mimic a larger teacher, transferring learned behavior at the cost of some inherent loss.
• Pruning removes low-impact weights, neurons, heads, or layers, in structured or unstructured form.
• Quantization uses fewer bits for weights and activations while retaining network structure.
• Self-compression gradually reduces precision during training, potentially down to zero bits for unused pathways.

"You have to start from the point of view of what you want the model to do, but then there are a million and one things we can throw at it to make it smaller," Plowman said. "Distillation works better on transformers than on traditional convolutional neural networks."

Magnus Gedda, CTO and chief product officer at Imagimob, an Infineon Technologies company, put the systems problem bluntly. "The model usually shares memory and compute with the rest of the application, so the available resources are smaller than the spec sheet suggests," he said. "Getting those three things to line up — problem fit, size, accuracy after optimization — is where most of the effort goes."

Steve Woo, distinguished inventor and fellow at Rambus, described the shift as one from "largest model wins" to "experience per watt." Compression, pruning, and lower-precision arithmetic shrink models to fit local resources, but memory capacity and bandwidth remain the limiting factors at deployment.

Why is deployment harder than it looks?

Once optimized, deployment itself is mechanical: the model converts to C code, compiles for the target, and flashes to the device. The difficulty lies in using varied hardware — MCUs, accelerators, NPUs, DSPs — to its full potential, without adequate standards or tooling. The result, Gedda noted, is either a generic inefficient deployment or heavy manual work per target.

Sathish Balasubramanian, head of product for EDA AI at Siemens EDA, argued the key is categorizing functionality. "A 4-billion [parameter] model on a sensor is huge, even though we're now talking 1 trillion [in the cloud]," he said. "We need to define the highest-priority, must-have function, and train it well in an SLM. It doesn't need to be an LLM."

His framework sorts features into three tiers: must-have, critical, and nice-to-have. Must-haves belong close to the sensor, in a self-contained loop with local compute, local memory, and sometimes a local backup environment. Devices also need an hour or two of power envelope in reserve so they do not slow down or shut down.

Mick Posner, senior product marketing group director for chiplets at Cadence, made the same point for physical AI. "Intelligent physical systems do not have a one-size-fits-all compute profile," he wrote in a recent blog. Some workloads demand always-on, low-power inference; others require multimodal processing across cameras, radar, lidar, audio, and environmental sensors.

How do the compression techniques differ?

Plowman offered a concrete comparison for quantization. Full-width FP32 or FP16 values travel to their endpoint much as the crow flies. Quantized values navigate city blocks — up one, over one — and still arrive at the same place, provided the small angular steps mean the same thing as direct navigation.

Self-compression is gentler. During training, certain pathways become popular while others serve only as temporary scaffolding. "We look for the sensitivity to those pathways, then quantize and quantize and quantize them down to zero bits," Plowman explained. "If nothing changes in the model and you still get the same outcomes, there was likely no contribution from those pathways."

The complexity lies in managing loss. Engineers must keep good data coverage in the training set for the quantization flow. In automotive, that means preserving long-tail edge cases — rare samples that let an autonomous system handle unusual road conditions.

The automotive sector, notably, still runs CNNs from three or four years ago and has not fully moved to transformers, Plowman said. Carmakers also train their own models and guard them closely. "We get a lot of proxies about what they look like, but it's very rare that we get an entire model to play with," he said of IP providers' visibility.

What comes next?

Licinio Sousa, senior director of product management at Synopsys, framed right-sizing as "a systems problem, not a single-point optimization." Quantization helps the model fit, but real-time latency, power, cost, and functional safety must balance together — and because AI models evolve faster than silicon development cycles, systems need flexibility for the models that come next.

The direction of travel is clear. Successful edge AI deployment starts with defining the workload and choosing an appropriately sized architecture, not with shrinking the largest available model. As Plowman observed, the gold rush for ever-bigger GPUs is becoming less prevalent, and intelligent models running on truly embedded platforms are becoming more realistic.