Global data center electricity consumption is projected to reach 945 terawatt-hours by 2030, roughly double today's levels, with AI workloads as the primary driver, according to the International Energy Agency. That projection reframes the central constraint in hyperscale AI: the ceiling is no longer set by how many accelerators can be deployed, but by how much energy a facility can deliver, distribute, and convert into useful computation.

The consequence for silicon architects is direct. System design is now shaped less by theoretical peak compute and more by how efficiently available power translates into delivered performance. Every watt spent on inefficient data movement is a watt unavailable for useful compute. As AI clusters scale, moving more data per unit of energy has become a first-order determinant of effective system throughput.

The bottleneck moves on-chip

Much of the industry discussion around AI scaling concentrates on off-chip bandwidth — high-bandwidth memory (HBM), scale-up fabrics, and memory pooling built on CXL and PCIe. Those remain critical for training workloads, where memory capacity and external bandwidth are hard constraints. But in modern accelerators, bottlenecks are increasingly distributed across both on-chip and off-chip data movement, demanding coordinated optimization across the full memory and interconnect hierarchy.

The problem grows with accelerator complexity. Today's devices integrate multi-core NPUs, systolic arrays, and vector units that generate and consume data simultaneously, producing an explosion of internal traffic. On-chip fabrics must now carry high-bandwidth streaming flows, latency-sensitive control traffic, and coherency-driven memory access patterns at the same time. Cache hierarchies compound the challenge: local caches deliver substantially higher effective bandwidth than external memory when data reuse is high, but they introduce additional traffic patterns that the fabric must coordinate.

The implication is stark. Even the fastest external memory cannot compensate for an internal network-on-chip that cannot keep up. AI performance increasingly depends on how well data is moved, reused, and prioritized within the system — not merely on how quickly it can be fetched from outside.

Heterogeneous traffic demands intelligent arbitration

Modern AI systems mix compute elements with fundamentally different traffic behavior. GPUs and NPUs drive high-throughput streaming workloads operating on wide, multi-threaded data streams. CPUs introduce bursty, latency-sensitive traffic. Some accelerators require tightly synchronized, coherency-aware communication. Supporting this mix is not simply a matter of adding bandwidth; it requires interconnects that manage multiple traffic classes simultaneously, enforce prioritization, and maintain predictable latency under load.

Quality of service, traffic isolation, and coherence domain management have moved from optional features to core requirements. The split between coherent and non-coherent traffic adds another layer: maintaining data consistency across shared-memory domains introduces synchronization overhead, and without intelligent arbitration and scheduling, contention quickly erodes the benefit of additional compute.

Chiplets redistribute the problem

The industry's shift to chiplet-based architectures — partitioning large SoCs into multiple dies to improve yield, cut cost, and scale beyond reticle limits — does not eliminate the data movement challenge. It redistributes it. Within a chiplet, shorter wires allow higher bandwidth and lower latency, and memory can sit closer to compute. Between chiplets, the die-to-die interface becomes the new constraint, typically worse than monolithic on-chip interconnect in bandwidth density, latency, and power efficiency.

Under AI workloads that demand continuous, high-volume data exchange, die-to-die links can become bottlenecks quickly. Chiplet architectures therefore raise demand for cross-die QoS and prioritization, efficient scheduling of inter-die traffic, and workload partitioning that minimizes unnecessary data movement. Chiplets simultaneously solve scaling problems and make system-level data movement architecture more critical.

From handcrafted fabrics to automation

Handcrafted interconnects still work in tightly scoped designs, but they do not scale to systems with hundreds of compute elements, multiple memory hierarchies, and complex traffic interactions. The harder problem starts before implementation, at the level of architectural intent: manually specifying how data should move through a system is time-consuming, error-prone, and difficult to optimize.

That is driving a shift toward system-level modeling and exploration, automated interconnect generation, and more abstract, programmable descriptions of architecture. The goal is not merely faster interconnect builds but preserving architectural intent from concept through implementation, so that performance, power, and scalability targets are met predictably — essential in a market where design cycles are compressed and performance margins are tight.

Historically treated as a secondary concern addressed after compute and memory decisions, the interconnect is now a primary determinant of system performance, efficiency, and scalability. Performance in hyperscale AI is no longer defined by the number of accelerators in a rack but by how consistently those accelerators can be kept busy, which requires fabrics that prioritize, balance, and deliver data predictably under load. As power budgets harden toward the IEA's 945 TWh outlook, the next generation of AI infrastructure will be defined by systems designed from the outset to move data as efficiently as they process it.