TL;DR
- A GB200 NVL72 joins 72 GPUs into one NVLink domain, so a model you serve across the rack stops on all 72 when any one GPU fails. The other 71 are healthy, and they sit idle until recovery finishes.
- The faults are the ordinary ones: overheating, memory errors, CUDA errors and driver errors, each an independent event on one card. What rack-scale hardware changed is how many GPUs one of them idles, and for how long.
- Recovery today rebuilds rather than restores. The weights are reloaded and resharded across the survivors, and the key-value cache is recomputed by re-running prefill for every request that was in flight. On a single node of 8 B200 GPUs, native starts of four frontier models took 9.4 minutes to 34.2 minutes.
- Restoring the saved state of an already initialized instance took 57 to 70 seconds on that same node. Extending either figure across 72 GPUs is arithmetic, not a measurement.
- In this piece we walk through why one GPU failure idles a whole NVLink domain, what recovery has to rebuild, and how ordinary the underlying faults are. Then we cover what the recovery window costs across 72 GPUs, what the surviving replicas pay, and what changes when recovery copies the state back.
A model served across the rack runs as one computation
A GB200 NVL72 puts 72 GPUs in one rack and joins them into a single NVLink domain. NVIDIA's product page describes it as a rack-scale, liquid-cooled design connecting 36 Grace CPUs and 72 Blackwell GPUs, and those GPUs form the largest NVLink domain the company has offered. NVIDIA calls the result "a 72-GPU NVIDIA NVLink domain that acts as a single, massive GPU".
A scale-up domain is the set of GPUs joined closely enough that one model can be spread across all of them as if they were one device. On a server, it is the 8 GPUs in the box. In an NVL72, it is all 72. Spreading one model across that many GPUs is what couples them.
Tensor parallelism splits the weights and the key-value (KV) cache, the memory holding the working state of every request in flight, across every GPU in the domain. Each layer of the model then waits for every slice before the next layer can run, so a slice that is missing stops the layer, and the layer stops the request.
One failed GPU therefore takes the whole domain out of service. FailSafe, a paper on resilient serving by Xu, Xie, Gandhi, and Kozyrakis, reports that a single GPU failure can leave the entire tensor-parallel execution unavailable across all the GPUs within the affected scale-up domain. The paper names deployments of up to 72 GPUs as current, but its own experiments ran on an 8-GPU node, and nothing in the mechanism changes at 72.
The healthy GPUs do not share the fault. They stop because the next step of the computation needs the slice that died with the failed one. The same shape shows up on a single node, where one worker's NCCL failure stalls the whole distributed job.
Recovery reloads the weights and rebuilds the cache from nothing
The failed GPU held a slice of the model weights and a slice of the KV cache, and both went with it. The weights can be read back, from CPU memory or from persistent storage, but the survivors then have to reshard and rebalance them among themselves, and FailSafe says that work results in substantial data movement and traffic over PCIe.
The KV cache has no copy anywhere. FailSafe describes it as the dynamic per-request state that has to be recomputed from scratch, and recomputing it means re-running the entire prefill phase for every affected request that was in flight. Prefill is the pass that reads a request's whole prompt and builds its cache before the first output token, so re-running it adds directly to time to first token.
For as long as that work takes, all 72 GPUs deliver zero tokens. The failed one delivers nothing because it failed, and the other 71 belong to an instance that has nothing to run until the weights are back in place and the cache is rebuilt.
The faults are the same ones you had on an 8-GPU server
Nothing about the failures themselves is new. FailSafe lists overheating, memory errors, CUDA errors, and driver errors that terminate the software running on the card as the common sources of a hard GPU failure, and it treats each one as an independent event on one GPU.
Those faults stay on the card that has them. Cui and colleagues, studying NCSA's production Delta cluster of A100 and H100 GPUs, found that uncontained memory errors accounted for 61% of the GPU errors not caused by user jobs, and memory management unit errors for another 30%. Delta runs mostly single-GPU jobs, so the study tells you what breaks and nothing about what one break does inside an NVL72.
What the rack changes is the blast radius, the number of GPUs one failure takes out of service. A paper on nonuniform tensor parallelism by Arfeen and colleagues at Carnegie Mellon and NVIDIA uses the term for exactly this. Writing about the move from 8 GPUs to 72 GPUs connected by NVLink, they report that larger scale-up domains increase the blast radius of failures, with one GPU failure potentially affecting tensor-parallel execution on the full scale-up domain.
That paper studies training, but the coupling it describes is the one FailSafe documents for inference. It also records what operators do about the risk. In a GB200 NVL72, Arfeen and colleagues write, "it is often simpler to discard an entire node with any GPU failures rather than try to operate with partially failed nodes." What rack-scale hardware changed is how far each fault reaches, and for how long.
One failure idles the whole rack for the length of the window
SemiAnalysis puts the all-in cost of a GB200 NVL72 at about $3.9M per rack. Every GPU in it waits out the same recovery, so the GPU-time one incident costs you is 72 times the length of the window.
Recovery today is a rebuild, and the first published figure for how long one takes comes from Cedana's benchmark, which timed native cold starts for four frontier models on a single node with 8 B200 GPUs. Each model was served with SGLang using its official cookbook recipe, unmodified, and the clock ran from engine launch until the model was ready to serve, including weight loading and full engine initialization. The four models came up in 9.4 minutes to 34.2 minutes.
Alibaba Cloud's deployment documentation gives an independent figure for the weight load alone, putting a full-version DeepSeek-R1 at 20 to 30 minutes, with the cache recompute still to come. Neither figure comes from a 72-GPU domain, because one is an 8-GPU measurement and the other comes from a deployment guide. Applied to the larger domain as arithmetic, both give the GPU-hours in the table below.
| Recovery window | Where the figure comes from | GPU-hours across 72 GPUs delivering zero tokens |
|---|---|---|
| 9.4 minutes | Fastest native start of the four models on our 8x B200 benchmark | 11.3 |
| 20 minutes | Low end of Alibaba Cloud's 20 to 30 minute load for DeepSeek-R1, weights only | 24 |
| 34.2 minutes | Slowest native start of the four models on the same benchmark | 41 |
SemiAnalysis read the design the same way in June 2024, before the racks shipped, saying that the status quo meant an entire GB200 NVL72 would go down with just one GPU failure or one optical failure. It noted that NVIDIA had added a dedicated engine for reliability, availability, and serviceability in response. That engine predicts and flags a failing part, so you hear about it earlier. It does not change what recovery has to rebuild once the failure lands.
The surviving replicas pay for it too
The replicas that are still up slow down as well. LUMEN, a paper on coordinated failure recovery for distributed LLM serving, describes what the failure does to the workers that are still up: the cluster loses the failed worker's GPU-resident key-value caches and its serving capacity at the same moment, so the surviving workers absorb the redirected traffic while re-running the interrupted requests from scratch.
So the interrupted requests re-run their prefill wherever they are picked up, and the traffic the failed instance was carrying lands on replicas that were sized without it. Requests that never touched the failed GPU wait longer, and they keep waiting until the instance is back. If time to first token spikes across your cluster and nothing in the application changed, look for a GPU failure in the minutes before the spike.
The window is shorter when recovery copies the state back
The recovery window is the time it takes to bring the state back, and a rebuild and a restore take very different amounts of it. A rebuild reloads the weights, initializes the engine, and recomputes the cache by re-running prefill. A restore copies the saved state of an already initialized instance back into memory, so none of that work happens again.
Our benchmark also timed restores of those models on the same node: 57 to 70 seconds from checkpoints held in tmpfs. A tmpfs file system lives in the node's own memory, so those times include no read from a disk or across a network. A restore in your own fleet will depend on the size of the checkpoint and the storage it is read from, so measure it there.
That is an 8-GPU result, not a 72-GPU one. Extended across the domain as arithmetic, a 57 to 70 second window is 1.1 to 1.4 idle GPU-hours, and the same arithmetic on a 20-minute rebuild gives 24. The extension assumes the domain resumes as one unit in the single-node time, but no published measurement shows that, so read the 72-GPU figures as a hypothesis built on an 8-GPU result rather than a measurement.
An NVL72 holds 36 Grace CPUs alongside its 72 GPUs, so a model served across the whole domain spans many machines rather than sitting inside one. In Cedana's terms that is a multi-node workload, and a tier in design partnership is one we are building with those partners rather than shipping, so resuming a whole NVL72 domain as one unit is not something we do today. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
For an operator counting racks, the unit of failure is now the rack. Of the two numbers that decide what an incident costs, the size of the domain is fixed by the hardware, but the length of the window is not. A failure costs the resume time, not the state.
Shortening that window means copying the state back rather than building it again. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. A workload we have checkpointed comes back from saved state rather than being built again from nothing, with its work restored as of the last checkpoint. On the 8-GPU node in the benchmark above that was 57 to 70 seconds instead of 9.4 to 34.2 minutes, and the nodes in your fleet that serve a model inside one box are where that difference is open to you today.
Related:
- What happens to a training job when a GPU fails
- One node failed and the whole training job died
- What torchrun, torchft and Ray Train restart from when a node dies
- What to do when vLLM or SGLang stops responding and nothing crashed
- GPU failure frequency scales with the fleet, not the on-call rotation
Common questions
Why does one GPU failing idle every other GPU in the same rack?
A model served across a scale-up domain runs as one computation, because tensor parallelism splits the weights and the key-value cache across every GPU in the domain and each layer waits for every slice before the next layer can run. The healthy GPUs do not share the fault. They stop because the next step needs the slice that died with the failed one, and they stay stopped until the weights are back in place and the cache is rebuilt.
What kinds of GPU failure cause this, and are they anything new?
No. FailSafe lists overheating, memory errors, CUDA errors, and driver errors that terminate the software running on the card as the common sources of a hard GPU failure, and treats each one as an independent event on one GPU. Nothing about the failures themselves is new at rack scale. What changed is how far each fault reaches, and for how long.
Can I recover just the failed GPU, or does the whole node have to come out?
At rack scale it is often simpler to discard the whole node. Writing about the move to 72-GPU scale-up domains, researchers note that "it is often simpler to discard an entire node with any GPU failures rather than try to operate with partially failed nodes."
