Meta Ditches File Storage for Object Storage in Gigawatt AI Cluster

By Sophie Lindqvist

Meta cut average AI training job ingestion time from 150 minutes to 10 minutes by abandoning file-based storage for cached object storage at its Prometheus supercluster in New Albany, Ohio. The company disclosed the figure itself. No third party has verified it.

Prometheus currently runs hundreds of thousands of GPUs across multiple interconnected buildings. Meta plans to scale it to roughly 1 million GPUs operating in lockstep for frontier models from Meta Supercluster Labs. That target is a roadmap, not a deployed system.

The cluster trains Meta's Muse family of foundational models. A second gigawatt-scale supercluster, Hyperion, is under construction in Richmond Parish, Louisiana. Meta did not state Hyperion's GPU count or timeline.

Which Nvidia Platforms Power Prometheus?

Prometheus runs two Nvidia platforms today. Catalina uses the Ariel variant of the GB200 NVL72, Blackwell generation, in a two-rack configuration. Thirty-six GPUs per rack connect through NVLink into one 72-GPU domain.

Clemente uses the GB300 NVL72, Blackwell Ultra. Meta says GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and lifts dense FP4 throughput by a similar factor. Attention acceleration roughly doubles. A full 72-GPU domain fits in a single rack.

A third platform, San Miguel, based on Vera Rubin NVL72, is due later. Meta calls it a generational leap in per-GPU performance. It will integrate into the existing network fabric. No ship date was given.

Cluster buildings connect through a multi-layer Backend Aggregation network spanning 2 km to over 100 km. Arista 7800-series chassis run deep-buffered Broadcom Jericho3 silicon. Meta claims multiple Pbps of bandwidth per region pair.

Why Move From Files to Objects?

The storage stack previously exposed an NFS-like FileSystem interface on top of Meta's Tectonic fabric. Meta used that design for Llama training and still runs it widely inside the company. Prometheus has now moved to BLOB-based storage.

Meta said the shift "is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance" to maximize GPU utilization. In plain terms: idle GPUs cost money, and file interfaces stalled them.

The new stack has four hardware tiers:

• L1/L2: memory and local flash on GPU hosts
• L3: regional flash-backed BLOB storage, colocated with GPUs
• L4: durable HDD-backed BLOB/Tectonic storage, the source of truth

A fat-client SDK streams data directly from Tectonic servers, with no dataplane proxy. ZippyDB provides O(1) path-to-address lookups. Two caches do the heavy lifting: a distributed data cache built on spare GPU-host memory, reusing components of Meta's Owl peer-to-peer distribution system, and a memcache-like store for read-plan metadata. Prefetching, hedged reads, and TTL or LRU eviction policies round out the design.

The hard number: maximum very large job ingestion time fell from 89 hours to 182 minutes. Meta attributes this to the dataloading scheme alone. Hardware changes occurred in parallel, so the attribution carries uncertainty.

What Hardware Carries the Data?

Flash tiers run Olympic and Yosemite V3.5 Crater Lake platforms. Crater Lake uses single-socket 4th Gen Intel Xeon Sapphire Rapids processors in a 1OU design. Yosemite V3.5 packs up to a dozen 10 RU servers in a 40 RU rack and uses OCP-aligned E1.S and M.2 datacenter NVMe SSDs, not consumer drives.

Bulk storage runs on Grand Canyon, a 4OU chassis holding up to 72 3.5-inch HDDs. Head-node plus JBOD configurations scale to 216 drives or more. All three designs are OCP contributions.

Checkpointing follows the same tiering: in-memory staging, then flash, then durable HDD storage, with delta checkpointing to protect throughput. Meta reports goodput above 85 percent on pre-training runs.

What Can Others Take From This?

The design suits hyperscale operators. Smaller shops can build on Cloudian, MinIO, or Scality object storage instead. Meta did not say whether Prometheus will serve inference or agentic workloads, where ingestion times and GPU lockstep matter less. If inference demand keeps growing, the object-storage bet may pay off twice.

In 2024, Meta trained Llama 3 on 49,152 H100 GPUs with Hammerspace orchestration. The move to BLOB storage supplants that file-based stack within two years, a strikingly fast rewrite for infrastructure at this scale.