Neural Video Codecs Cost 10x the Energy of VVC Silicon, Analysis Finds

End-to-end neural video decoders consume roughly an order of magnitude more energy per pixel than mature VVC silicon at the same process node, according to a hardware-cost analysis by Dhruv Aggarwal, a Senior RTL Design Engineer at NVIDIA. His conclusion: traditional VVC hardware will not be displaced from consumer silicon in the next decade.

The analysis, published in a detailed technical article, grounds the argument in measured silicon data rather than benchmark hype. The only published end-to-end neural decoder ASIC, NVCA, synthesized in TSMC 28 nm with 5.01 million gates, achieves 1080p25 decoding at 760 mW. That works out to roughly 14.6 nJ per pixel, excluding weight ROMs stored off-chip.

By contrast, the closest published anchor for a conventional decoder, the Tikekar et al. HEVC decoder ASIC at TSMC 40 nm, measured 1.19 nJ per pixel for 4K30. Scaling that figure forward from 40 nm to 28 nm using the historical 30–50% per-node power reduction, and applying the roughly 1.6–1.8× HEVC-to-VVC complexity uplift, Aggarwal estimates a 4K60 VVC decoder at 28 nm plausibly lands around 1–1.5 nJ per pixel within about 1 million gates. Production VVC IP figures are proprietary and not directly published.

Why is JVET exploring neural tools at all?

Each codec generation, from H.264 to HEVC to VVC, has delivered roughly 30–50% bit-rate reduction over its predecessor, but sustaining that trajectory is getting expensive. In all-intra encoding, the VVC reference encoder VTM is already about 31× more complex than HEVC's HM, measured as reference-software runtime. The driver is the growing number of prediction modes: angular intra modes alone went from 8 in H.264 to 33 in HEVC to 65 in VVC.

Hand-crafted coding tools are yielding diminishing returns, which is where learned tools enter. The Joint Video Experts Team (JVET) is running formal exploration experiments on neural tools for a potential future standard. Its Neural Network-based Video Coding (NNVC) exploration has adopted two neural tools into reference software, though not yet into a normative standard: a fully connected intra prediction mode and a convolutional in-loop filter. Combined Y BD-rate gains run around 10–12% across configurations.

Do end-to-end neural codecs actually beat VVC?

On paper, sometimes. End-to-end neural codecs such as DCVC-FM and NVRC match or exceed VVC under specific test configurations, including single-I-frame or random-access tests on the UVG dataset. NVRC outperforms VVC by 24% BD-rate on UVG, and DCVC-family codecs produce subjectively smoother low-bitrate images than VVC under several quality metrics.

The problem arises when those results must fit inside consumer devices, Aggarwal writes, citing three constrained environments:

• A phone, with thermal and power budgets plus multitasking against other AI workloads.
• A TV, facing standardization challenges, cost sensitivity, and Energy Star ratings.
• Automotive infotainment, where functional-safety verification cost and NPU contention with safety-critical operations dominate.

Neural-based video coding, he argues, "may be winning on the algorithmic end but losing on the silicon floor."

What does the hybrid NNVC path cost in hardware?

The hybrid approach, JVET's neural intra prediction layered onto a VVC core, carries lower but still meaningful hardware cost. NNVC specifies 7 fully connected networks, one per supported block shape from 4×4 to 16×16, of roughly 1,216 neurons per layer, invoked on perhaps 30–40% of intra blocks. The NN intra prediction mode is signaled per block via an nnFlag and competes with the existing 67 intra modes plus MIP, ISP, MRL, and CCLM through rate-distortion optimization rather than replacing them.

If the standard ratifies NNVC, Aggarwal sketches the practical hardware path: a single shared MAC array that time-multiplexes the seven block-shape-specific weight sets, loading the appropriate set at the start of each block-level inference and holding it stationary during compute. Weights would live in on-chip ROM, analogous to VVC's MIP and LFNST coefficients.

The added area, power, and verification cost are non-trivial. They are defensible on premium SoCs once the tool is standardized, but hard to justify on cost-sensitive parts where the 10–12% BD-rate gain does not pay back the silicon. NN-based decoding also takes substantially more cycles than traditional algorithms, stressing the decoder pipeline's tight latency cycle budgets.

Why standardization decides the outcome

A ratified NNVC decoder's bit-exact fixed-point inference must match across devices. That constraint forecloses many standard NPU efficiency techniques, including reduced-precision inference and fused operators. VVC, finalized in 2020 as ITU-T H.266, has established licensing. NNVC remains in JVET's exploration phase, and end-to-end neural codecs sit further back still, with no ratified standard, no cross-vendor interoperability, and no royalty framework to build a product around. For consumer hardware, where compliance is mandatory, that gap is decisive.

What should chip architects do?

Aggarwal prescribes tier-aware deployment rather than wholesale replacement:

• Keep a traditional VVC or AV1 decoder block as the primary path for the next five to seven years. VVC content will be decoded for two decades after the standard's 2020 finalization, and skipping it for neural codecs is misguided.
• Add NNVC tools as auxiliary blocks when standardization and budget allow, coupling a small NN inference block to the VVC decoder with ROM for normative weight sets, SRAM for working storage, bit-exact SADL matching, and a non-trivial verification budget.
• Keep general-purpose end-to-end neural decoder blocks out of consumer SoCs for the next few generations. They may find limited use in specialty scenarios such as VR/AR, machine vision, and edge encoding co-designed with the hardware.

The guidance for a tape-out is concrete: do not skip VVC hardware support to chase neural codecs, and do not over-provision NPU capacity for adoption that will not arrive on schedule. Aggarwal, previously a codec hardware designer at Qualcomm for five years and a co-inventor on two video-coding patents, frames the choice plainly: let the tape-out reflect the world that exists, because the dramatic replacement narrative, even if ultimately feasible, will take decades to materialize.