Scality says its AI Inference Factory restores a 14K-token context from storage 14 times faster than recomputing it, with warm time-to-first-token of 166 milliseconds — only 83 milliseconds behind GPU HBM. The claim anchors the company's entry into shared KV cache pooling, a market Cloudian and MinIO entered first, and adds a feature neither rival offers: separated prefill and decode GPU pools.

The product is available now as a software license or a fully managed service. It packages vLLM, NVIDIA's Dynamo serving layer, a Scality AI Connector and Scality ADI object storage behind one endpoint, targeting enterprises, government agencies and neo-cloud providers that want an alternative to per-token cloud inference pricing.

What does the CEO say about sovereignty?

Co-founder and CEO Jérôme Lecat framed the launch around control and reliability. "With AI moving into mission-critical production environments, organisations need greater control over where inference runs, how their models are managed and what happens to their data," Lecat said. "AI Inference Factory brings that experience to on-premises AI, giving organisations the reliability and sovereignty they need to run critical AI workloads on their own terms."

The company argues cloud inference costs rise with usefulness under per-token pricing, and that providers can change model versions and quantisation at their discretion, breaking customer workflows.

How does the architecture work?

The stack has four layers:

• Validated open-weight models maintained within the supported stack, including Mistral, Gemma, gpt-oss, Qwen, Kimi, GLM and DeepSeek.
• Disaggregated inference serving that separates prefill from decode so each scales independently.
• A control plane that authenticates, meters, routes requests to GPUs holding the relevant context and schedules against SLA targets.
• Scality ADI, the object-storage layer for models, enterprise data and inference state.

Prefill computes the KV cache for a new prompt; decode generates tokens one at a time. When both run on the same GPUs, each new prefill interrupts running decodes. Split into two pools sharing one cache, each pool runs at full load and scales on its own.

Co-founder and CTO Giorgio Regni said: "The KV cache on Scality ADI is fast enough to sit in the serving path. Restoring a context from ADI is the same order of magnitude as GPU memory, and 14 times faster than recomputing it, with the GPUs staying busy the whole time. Storage is no longer the reason to keep the KV cache inside the GPU server."

He added that any decode GPU can pick up any context from the shared pool, with essentially no size limit.

What did Scality's own testing show?

• 1.9-second load time for Gemma-3 27B, nearly 10x faster than local NVMe, streamed in parallel across the cluster over RDMA.
• 166 ms warm time-to-first-token on a 14K-token context restore from ADI, 83 ms behind HBM.
• 14x faster KV cache retrieval than recomputation on a 14K-token context, up to 72x faster on a 439K-token context.
• A KV cache more than 80x larger than a single GPU's memory, keeping up to 1,000 concurrent sessions resumable without recomputation.
• 97 percent of network line rate for GPU-storage transfers.
• No measurable impact on token generation, as context restores before the first token.

Scality also cites external work: DistServe (OSDI 2024) measured up to 7.4x more requests served within the same latency when moving the cache directly between GPUs, and Moonshot AI's Mooncake reported 75 percent more requests on Kimi's production traffic from a shared KV cache pool.

ADI, launched in May, tiers data across TLC NVMe, HDD and optionally tape in a single namespace, deploying at multi-petabyte scale with RDMA access to disk. The stack runs on standard Dell, HPE, Lenovo and Supermicro servers and integrates with OpenCode, Hermes, Goose, LangGraph and Pydantic AI. All components are open code.

How crowded is this market?

Nearly every filesystem vendor now supports NVIDIA's STX KV caching scheme, which requires BlueField-4 DPUs, Spectrum-X networking and dedicated CMX flash tiers — DDN, HPE, Hitachi Vantara, IBM, NetApp, Nutanix, VAST Data and WEKA among them. Cloudian's HyperScale AI Data Platform claims up to 60 percent cost savings versus public cloud inference, egress and token fees. MinIO's petabyte-scale MemKV serves a common context pool to an entire GPU cluster at microsecond latencies.

Both offer the same on-premises message as Scality, but without prefill-decode separation — a distinction Scality will now defend against rivals whose cached architectures, by the company's own admission, work on the same principles.