More than 11 percent of the GPUs at Meta’s Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week. The massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI training runs on track. Prometheus is among Meta's largest compute deployments to date. Powered by Nvidia's Grace Blackwell and (soon) Vera Rubin rack systems, Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators. But as anyone who has worked on a large scale compute cluster understands, whether for LLM training or traditional HPC workloads, more chips inevitably means more problems. In an eight-GPU node, the chance of having a GPU fail is relatively low. In a 100,000-GPU cluster, failures are unavoidable. Meta is building something that's 10x larger and requires distributed training runs across multiple datacenters. Keeping the entire thing humming long enough to reach the next checkpoint becomes a major headache. In a blog post published this week, Meta offered a unique look at the work it's done to keep failures from derailing the development of models like Muse Spark or its most recent release Muse Wonder. Idle GPUs are the cost of continuity When Meta began the first phase of its Prometheus deployment in New Albany, the company says, it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs. Interruptions don't mean GPU failure. They simply mean something crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint. Meta claims in the year since it's been able to cut down the number of interruptions by 99 percent, to about 11 per day per 8,000 GPUs. One of the ways it's done this is by keeping some of Prometheus' GPUs in reserve. The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain. However, only 64 of them are ever in use at any given moment. The other eight are essentially hot spares. Meta did this because a single failed GPU would take out an entire rack and interrupt the training run. Now, when a GPU fails, one of the spares can slot in and take its place. While this saves time — Meta doesn't have to drop the entire rack from the cluster or wait for someone to swap the faulty component to start training again — a failed GPU would still result in an interruption, albeit a shorter one. Parallel resiliency To mitigate this, Meta took advantage of the fact that large scale training jobs usually use data parallelism, where multiple copies of the model are resident within the cluster. "When a fault can be isolated to a data-parallel model replica, we reduce the impact of a hardware failure by allowing a single faulty replica to be taken offline, while the remaining cluster repeats that step and continues productive training," Meta explained. "The compromised slice simply restarts on available spare hosts and resynchronizes, converting a potential total system halt into a minor, temporary reduction in aggregate throughput." In other words, Meta's training runs only get interrupted if the fault exceeds the data parallel domain. When this happens, the company has implemented a comprehensive checkpointing scheme, which you can think of a bit like auto-save or a revision history. That way if something does fail, Meta doesn't have to start over from scratch. "Our distributed checkpointing architecture asynchronously saves the model state — including gradients and optimizer data — to multi-tier storage," the company wrote. NVLink growing pains Meta says one of the most difficult challenges was making the NVLink fabrics used to connect 72 GPUs behave like one great big accelerator. Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain, which meant the links were far shorter and less numerous. But this functionality was expanded nine-fold with Nvidia's Blackwell generation. "With over 600,000 NVLink connections, even marginal signal degradation triggers frequent NVLink recovery cycles," the company wrote. To overcome this, Meta had to implement a continuous telemetry system and conduct L11 diagnostic burns to isolate weak links before the systems were put into production. Power was another challenge. Ramping up and down tens of thousands of GPUs for a training run can introduce massive spikes in power consumption called transients, which can overwhelm power supply equipment. According to Meta, by focusing on the stability and reliability of its cluster, it was able to achieve more than 85 percent "goodput" — a measure of useful throughput — for pretraining and more than 90 percent for reinforcement learning on its latest model Muse Wonder. The company's ambitions don't stop here. As it works its way toward a million GPUs, Meta hopes to get its interruption rate down to just two per day for every 8,000 GPUs, or about 100,000 GPU-hours mean time between interrupts. ®
