Micron Claims 2.8x Inference Gain by Breaking KV Cache Memory Wall

Micron has measured 2.8 times higher inference throughput and 2.9 times faster time to first token by moving KV cache to a shared CXL memory pool instead of an equal-sized DRAM allocation. The result comes from a GPU cluster the company is building for the US Department of Energy's Pacific Northwest National Laboratory (PNNL), and every figure should be read with one caveat: they are vendor measurements under a 90 percent shared-prefix workload, not independent benchmarks.

The cluster is called Great Abaco. Dr. Skyler Windh and his team at Micron designed it for large-scale scientific, AI, HPC and data-intensive workloads, with Liqid acting as the memory fabric and orchestration partner. A single Abaco rack presents up to 160TB of DDR5 as one addressable pool. Hosts inside the rack can claim, share or release memory without a reboot and without re-cabling, and applications reach the pool through a standard file system interface, so most code runs unmodified.

Why does the KV cache bottleneck matter?

As large language model inference scales across multi-GPU deployments, the key-value cache becomes a major performance bottleneck. Each GPU typically holds its own isolated cache, which can spill into host DRAM. When different GPUs process the same prompt prefix, each repeats the same prefill computation. That is duplicated work on duplicated data.

CXL 2.0 fabric-attached memory attacks the problem directly. It gives multiple GPUs a shared, byte-addressable memory pool, so one GPU can compute a KV cache entry and the others reuse it. Windh's team built cross-host KV sharing into NVIDIA Dynamo and ran it on the Abaco cluster. The hardware configuration is specific: H200 NVL GPUs sit in a Liqid PCIe Gen5 chassis, and four Liqid CXL memory chassis form a single shared pool.

What did the numbers show?

The team tested Llama3-70B with 90 percent shared prefixes. The results, as reported:

• A 32GB cache per server delivered 2.8 times the throughput and 2.9 times faster time to first token than DRAM of the same size.
• Hosts reusing each other's KV blocks achieved 3 times the throughput with half the memory.
• CXL memory ran within 99 percent of host DRAM performance.

That last figure matters most. CXL fabric-attached memory performs on par with a host DRAM baseline, at least in this configuration. What Micron has not published is latency overhead per individual access, or results under low prefix-sharing workloads, where the reuse advantage shrinks and the fabric cost remains.

The integration rests on four components inside an open-source multi-GPU inference framework:

• A block-level cache manager that allocates in CXL memory.
• Sequence-hash-based cache discovery across GPUs.
• An asynchronous transfer engine that moves blocks between CXL and GPU memory without stalling the inference scheduler.
• A CUDA host-registration workflow enabling direct memory access between GPUs and CXL memory.

What does Liqid claim for the commercial case?

Liqid SVP for Solution Architecture Martin Cooper said the Micron work shows that shared CXL memory can lift GPU inference throughput 2.8 times over DRAM. His framing is commercial as much as technical. Liqid places GPUs and CXL memory on one PCIe fabric, so memory scales separately from compute. The serving stack stays as it is: no vLLM patches and no changes to the GPU servers. A shared pool lets operators right-size each server and buy the memory once.

The claim deserves scrutiny. Reuse gains depend heavily on workload prefix overlap, and the 90 percent figure in the test is unusually favorable. Buyers should ask what throughput looks like at 50 percent or 20 percent shared prefixes before pricing a 160TB pool against per-server DRAM.

Micron documented the work in a 60-minute webinar with a 30-slide deck, and offers an Abaco technical brief. One naming note: GFAM in the slides refers to Global Fabric Memory.

If the 99 percent DRAM parity holds across broader workloads, disaggregated CXL memory shifts from a promising architecture to a purchasing decision for inference operators watching memory bills climb faster than GPU counts.