Lightbits Labs reported a 380x reduction in time-to-first-token on a 10-million-token Llama-4-Scout context window using its Inferra software. The benchmark collapsed a 1.5-hour wait into 13 seconds.

The result anchors a tiered key-value cache scheme designed for AI inferencing. Inferra virtualizes a GPU's high-bandwidth memory (HBM) and extends KV cache storage across an embedded x86 server's DRAM, local NVMe SSDs, and external network-linked NVMe drives. The software pre-fetches KV blocks before GPUs request them.

Avigdor Willenz, Lightbits co-founder and chairman, framed the market opportunity. "The AI inference market will top $117 billion this year," he said. "Inference is a completely different paradigm — retrofitting legacy training and storage systems to solve the KV cache bottleneck simply doesn't work. It was built for a different era."

He added: "We engineered Inferra from the ground up specifically to solve the GPU efficiency problem, and it has already proven successful in customer beta programs."

How does the tiered cache work?

Inferra runs in a GPU server's embedded x86 system and monitors KV cache data block activity. A Sub-Linear Sparse Attention Prefetch engine identifies blocks most likely to be needed next, using locality-sensitive hashing combined with statistical reuse patterns to score candidates. Inferra fetches those blocks from DRAM or external SSDs and pre-loads them into HBM over RDMA links.

Lightbits published three public time-to-first-token benchmarks. Inferra cut Qwen 2.5-7B at 410,000 tokens from 72.6 seconds to 711 milliseconds, a 102x improvement. DeepSeek-R1-70B at 141,000 tokens fell from 70.8 seconds to 465 milliseconds, a 152x gain. Llama-4-Scout at 10 million tokens dropped from 1.5 hours to 13 seconds, a 380x speed-up.

OVH validated the approach. Yaniv Fdida, chief product and technology officer at OVH, said: "With Inferra's intelligent KV cache tiering, we were able to demonstrate substantial GPU utilization gains, paving the way to providing our customers a more scalable and cost-effective foundation for their AI agent and RAG workloads."

What does Inferra claim over competitors?

Lightbits positions Inferra as fully hardware-agnostic, running on any storage and GPU environment with dynamo, vLLM, and LMCache plugins. The company asserts it is the only vendor with algorithmic sub-linear scaling of time-to-first-token against context length; DDN, VAST Data, WEKA, and others scale linearly or stream reactively, according to Lightbits.

Lightbits demonstrated a 10-million-token context on commodity L40S GPUs. No other vendor has published an equivalent benchmark, the company said.

Ramesh Chettuvetty, senior vice president of AI product and business at Lightbits, summarized the commercial pitch. "Inferra delivers instant payback and generates net positive savings from day one," he said. "It eliminates the stalls that traditionally limit long-context performance, so Neoclouds and enterprises can serve larger models and longer conversations at lower infrastructure cost."

Earlier work with ScaleFlux produced 100x to 280x speed-ups of KV cache workloads using Inferra reading data off ScaleFlux computational storage SSDs.

The Pod Efficiency Analyzer is open to neoclouds and enterprises wanting to model inferencing cost savings before deployment. Adoption metrics from customer beta programs and competing vendor benchmarks will determine whether the claimed economics hold against DDN, VAST Data, and WEKA at production scale.