SlipStream Prefetcher Tackles 20x Flash Latency Penalty in GPU Memory
Georgia Tech, NVIDIA Research and Stanford show a 40 MB prefetcher buffer holding High-Bandwidth Flash inference within 8% of ideal HBM-like performance.
- By
- Tom Whitfield
- Filed
- Channel
- Memory & Storage
- Read
- 3 min read
By Tom Whitfield
High-Bandwidth Flash can offer up to 16 times the memory capacity of HBM at comparable read bandwidth, but on-demand reads from it can slow large language model inference by as much as 12.8x. That trade-off is the problem a joint research team from Georgia Tech, NVIDIA Research and Stanford University set out to solve. Their paper, "SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads," appeared as arXiv preprint 2611.14776 in November 2026.
Why is flash-backed memory hard for GPUs?
Large language model inference keeps demanding more memory capacity for larger models and larger batch sizes. High-Bandwidth Flash (HBF) answers that demand by replacing the DRAM dies of an HBM stack with NAND flash.
The capacity gain comes at a price. The researchers measured HBF read latency of nearly 20 microseconds, roughly 20 times that of HBM. Conventional cache-line prefetchers cannot paper over this gap cheaply: covering such latency requires about 20 times more in-flight tracking state. Tensor-streaming designs such as H3 take a different route, following virtual-address order rather than kernel-demand order, but the researchers note they need hundreds of megabytes of buffering.
How does SlipStream work?
SlipStream is a software-hardware co-designed prefetcher that reshapes prefetching into a demand-paced producer-consumer pipeline.
On the software side, SlipStream profiles the repeatable weight-page demand order of data-independent LLM kernels. It then inserts a staging warp that replays this order using non-blocking STAGE instructions.
On the hardware side, the staging warp acts as the producer and the work warps act as the consumers. The key insight is one rule about when to free tracking resources: each STAGE tracking entry retires only when demand first accesses a page, not when the page returns from HBF. Demand therefore paces staging and bounds how far staging leads consumption.
Two further design choices keep the hardware cost down:
- SlipStream tracks 4 KB pages instead of individual cache lines.
- Buffer slots are allocated only when pages return, which decouples buffer size from outstanding traffic.
The takeaway: staging no longer runs ahead of what the GPU actually consumes, so the memory needed to cover flash latency stops scaling with that latency.
What performance did the researchers measure?
Across seven dense models spanning 16 GB to 800 GB, SlipStream with a 40 MB staging buffer reached 92.4% of the performance of an ideal HBF with HBM-like latency. Under the same comparison, H3 reached only 43.1%, even with a 320 MB buffer.
Buffer sizing proved largely independent of both HBF latency and workload. A 16 MB configuration, 20 times smaller than H3's buffer, performed within 4% of the 40 MB version.
SlipStream also supports Mixture-of-Experts models. With a 40 MB buffer, it reached 91.2% of ideal performance on that class of workload.
What does this mean for memory design?
The numbers frame the commercial stakes precisely. If a prefetcher can hold flash-backed memory within about 8% of ideal HBM-like performance while tracking 4 KB pages from tens of megabytes of on-chip storage, NAND capacity becomes usable for weight storage in inference systems, not just for checkpointing on the host.
The authors — Anish Saxena and Hritvik Taneja of Georgia Tech, with Christos Kozyrakis of Stanford and Aamer Jaleel of NVIDIA — position the work against both prefetcher-based and tensor-streaming alternatives, and their measurements suggest the producer-consumer structure closes most of the gap that latency opens.
If follow-on work scales the approach across wider model families and vendor memory stacks, flash-backed capacity could shift from a research curiosity toward a practical tier in the GPU memory hierarchy.
Original: arxiv.org
More from Tom Whitfield
Show full bio
Staff writer covering companies, deals and strategy in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.
999 articles
Related articles
goldman-names-three-stocks-to-ride-high-performance-nand-s-ai-surge-c9c7c8ca
Goldman names three stocks to ride high-performance NAND's AI surge
kioxia-and-sandisk-split-on-high-bandwidth-flash-strategy
Kioxia and Sandisk Split on High-Bandwidth Flash Strategy
hbm-turns-ai-demand-into-a-broader-dram-supply-squeeze
HBM turns AI demand into a broader DRAM supply squeeze
fractile-ceo-ai-bottleneck-is-memory-bandwidth-not-compute-d458a481
Fractile CEO: AI Bottleneck Is Memory Bandwidth, Not Compute


