TL;DR
- Your job stopped because the set of ranks is fixed when it starts. When one worker crashes, hangs or gets evicted, every other rank waits for a contribution that never arrives, and the whole job hangs or aborts.
- What you get to read is an NCCL error with nothing specific in it, a watchdog timeout naming a collective that did not finish, or on some hardware no error code at all. Each of those says where progress stopped rather than why.
- Every standard answer rebuilds the group and restarts from the last checkpoint your training code happened to write. So the GPUs that were healthy throw away work that was never the problem. torchft is the exception, and it keeps the group running only for a job you have built as replicas.
- What changes is capturing every GPU of the job at the same step boundary, with the NCCL communicator state included. The last group checkpoint then comes back on healthy GPUs and the run continues from that boundary. That ships today for the GPUs of one job on one node.
- In this piece we walk through why one node's failure stops every GPU, what the NCCL error and the watchdog timeout mean, and what people do today. Then we cover why a saved checkpoint can still start over, and what a coordinated checkpoint does and does not cover.
Why does one node's failure stop every GPU?
The set of GPUs is fixed when the job starts, and every exchange of data between them needs every member to take part. The GPUs talk to each other through NCCL, the NVIDIA Collective Communications Library, and three of its terms carry the rest of this page.
| Term | What it means |
|---|---|
| Rank | One participant in the job, usually one GPU and the process driving it. |
| Communicator | The object every rank creates at startup. It names the full set of ranks and holds the connections among them. Every collective runs inside one. |
| Collective | An operation every rank takes part in, such as an all-reduce. It cannot complete until every rank has contributed. |
An NCCL failure happens when a collective cannot complete, because a rank crashed, a network path failed, or a rank stopped making progress. Every other rank waits for a contribution that never arrives, so the job hangs or aborts.
NVIDIA's remedy is to give up on the group and build a new one. The NCCL user guide for version 2.31.2 says an operation that hits an asynchronous error will usually never complete, and that "To recover, the application needs to call ncclCommAbort on the communicator and re-create it." Every rank has to do that, not only the one that failed.
So on a 16-GPU job, losing one GPU forces all 16 to stop, and the survivors cannot carry on with one fewer participant. NCCL does have an operation for dropping a failed rank: with the abort flag, ncclCommShrink creates a new communicator without the failed ones. But that changes who is in the group without restoring what the failed rank was holding, so your application still has to rebuild that state, and rebuilding it is the restart.
The same thing happens when the node is taken from you rather than broken. A SkyPilot issue reports that preempting a single node triggers a teardown and setup of the entire cluster from scratch again.
What the NCCL error and the watchdog timeout mean
Most of these reports carry one of two strings, an NCCL error with nothing specific in it or a watchdog timeout, and a third kind of failure arrives with no string at all.
An unhandled system error means a call below NCCL failed, but the message alone does not identify the cause, so NVIDIA's troubleshooting guide recommends enabling warning logs to see the underlying error.
The second string comes from the watchdog, a timer that limits how long a collective operation can take, and people post its line as "Watchdog caught collective operation timeout". PyTorch's explanation describes a collective that has not completed within the configured timeout. So the line tells you where progress stopped, not necessarily why, and the watchdog can abort the run.
On some hardware there is no error code to read at all. NVIDIA's release notes for the DGX GB200 NVL72 list a known issue in which "2+ domain NCCL All-to-All testing may result in system hangs or deadlocks with no error codes", with no workaround yet.
A job that hangs at startup is a different problem
If your job hangs in the process-group constructor or at rendezvous and never gets past initialization, no step has run, so there is nothing to save and nothing to come back to, and a checkpoint does not help you. Those hangs are usually a networking fault or a version mismatch, and you settle them by reading the configuration. Everything else on this page assumes your job got past init.
What people do today when a node fails
Recovery begins at the last checkpoint your training code saved, for every option here except torchft. CoreWeave, which replaces failed nodes in its own fleet, describes the difference between "a job that recovers in minutes and one that requires a full restart from an eight-hour-old checkpoint".
A restart costs you more than the work it repeats, because the job has to be built again before it computes anything. Loading the weights takes the longest: Alibaba Cloud's documentation for deploying DeepSeek-R1 says loading the full-version model might take 20 to 30 minutes on a node with 8 GPUs. Every rank then runs the initialization handshake again, which in our experience takes 15 to 30 seconds.
torchrun, the elastic launcher in PyTorch, automates that restart for you. PyTorch's documentation gives worker-group restarts to torchrun and node replacement to the job manager, and what it calls re-rendezvous is the remaining workers meeting again, agreeing on a new membership, and starting from the snapshot on disk. The workers that did not fail go too: "On failures or membership changes ALL surviving workers are killed immediately. Make sure to checkpoint your progress." Its train script page states the cost, warning that "you will lose progress up to the most recent checkpoint".
Ray Train restarts the group the same way. "When a failure is detected, all the workers are shut down, new nodes are added if necessary, and a new set of workers is started." Whether any progress survives is left to your training code, which has to "implement logic for both saving and loading checkpoints. Otherwise, the training will just start from scratch."
torchft is the one option that does not restart the group, because it runs the job as replicas. Its README describes "techniques for doing a per-step fault tolerance so you can keep training if errors occur without interrupting the entire training job". When one replica dies it is dropped, and a replacement is rebuilt using "Checkpoint transports that can be used to do live recovery from a healthy peer".
The price is a replicated job and a script written around torchft's primitives, and the dead worker's state is saved nowhere. Crusoe, weighing the options on its own page, calls fault-tolerant frameworks like torchft "emerging but not yet widely adopted in production".
So what these tools change is how quickly the restart happens and how much of it you had to write yourself. Only torchft keeps the group running, through its healthy replicas, and only for a job built that way.
Why does a resume sometimes start over anyway?
The checkpoint file holds only what your training code put into it. People who did save a checkpoint report the job restarting and beginning at zero anyway: one report on Ray's tracker has the model starting from the beginning instead of picking up from the checkpoint, because the worker died. Others report a resume that loads and then runs a different loss curve.
An application-level checkpoint is one the training code writes itself, at points it chooses. If the epoch counter, the optimizer state, the data loader position, or the learning-rate schedule never went into the file, the resumed run does not have them, and it repeats the epoch.
The file does not carry the communicator state either, so the group is rebuilt by the handshake whatever else is in it. In our experience, an application-level checkpoint also freezes the job for longer, so teams take them less often and each failure loses more work.
What changes when every GPU is caught at the same step boundary?
Every GPU has to be captured at the same point in the distributed computation, with no collective left half-completed in flight, so the restored ranks agree on where they are. Say one rank is saved just after it sent its contribution to an all-reduce and another just before it received anything. The two saved states disagree about where that operation stands, and the job comes back with its ranks out of step. Silent corruption is worse than a restart, so the boundary has to be exact.
Below the application, a kernel-level primitive can checkpoint one process on one GPU, but it does not handle the collective communication in transit between GPUs at the moment of capture. The CRIUgpu paper of February 2025 notes that the driver-level checkpoint utility it builds on did not support checkpoint and restore with NCCL at the time. So coordinating the ranks of one job is the layer above single-GPU capture. The hard part is coordination, not capture.
That coordinating layer is what we build at Cedana. What ships today is every GPU of a job on one node, captured at the same boundary with the NCCL state included, and we capture the whole group at regular heartbeat intervals while the job runs. When a failure occurs, the last group checkpoint is restored on healthy GPUs, the communicator state comes back with it instead of being built again, and the job continues from the boundary where it was captured.
So the case in this page's title, a job whose GPUs sit in several machines, needs the multi-node tier. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. A neocloud is a company that rents out GPU capacity. If your job spans nodes and this is the case you need solved, talk to Cedana about a design partnership.
| What the job holds | After a restart | After a restore |
|---|---|---|
| Model weights | Read from storage again | Copied back with the rest of GPU memory |
| CUDA context and compiled graphs | Rebuilt during initialization | Restored as captured |
| NCCL communicator state | Aborted on every rank, then rebuilt by the handshake | Restored from the checkpoint, with no handshake |
| KV cache and in-flight sessions | Lost, and the interrupted requests run again | Restored as of the last checkpoint |
| The interrupted step | Begins again from the start, or from the last application checkpoint | Continues from the boundary it reached |
You do not have to start the recovery by hand. On Kubernetes, when the node is preempted and a new one comes up, the workload restores from its checkpoint instead of starting from scratch. On Slurm, the job is requeued and restored on a compatible node. Heartbeat checkpointing and automatic failover are live today.
We have published one restore benchmark for a single-node job: on a single node with 8 NVIDIA B200 GPUs, fully initialized engines for frontier models came back in 57 to 70 seconds, timed to ready-to-serve. The checkpoints were held in tmpfs, which lives in the node's memory rather than on disk. Those runs measured the restore on its own, and did not measure recovery after an NCCL failure.
What a coordinated checkpoint does not do
The hardware you restore onto has to match the hardware the checkpoint was taken on. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. So a compatible node is one running all four of those versions.
A checkpoint covers the GPU and the process, but side effects the job already committed are not undone when it resumes from an earlier boundary.
A failure on one rank costs the whole job only while the healthy ranks have nothing to come back to. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a distributed job, that means every GPU is captured at the same boundary, below the framework and with no change to your script, on Slurm and Kubernetes. So when one rank fails, the GPUs that were healthy do not throw their work away, and your job comes back from its last checkpoint rather than starting again from nothing.
Related:
- What happens to a training job when a GPU fails
- What torchrun, torchft and Ray Train restart from when a node dies
- What to do when vLLM or SGLang stops responding and nothing crashed
- Does multi-node checkpointing ship today?
- What cannot be checkpointed in a GPU workload?
Common questions
Why does one node failing stop every GPU in my distributed job?
The set of ranks is fixed when the job starts, so once one rank crashes, loses a network path, or stops making progress, the collective cannot complete. Every other rank waits for a contribution that never arrives, and the job hangs or aborts. NVIDIA's guidance is to call ncclCommAbort on the communicator and rebuild it, and every rank has to do that, not only the one that failed.
Why does my job restart from the beginning even though I saved a checkpoint?
An application-level checkpoint is written by your training code at points it chooses, and it holds only what that code decided to write. If the epoch counter, the optimizer state, the data loader position, or the learning-rate schedule never went into the file, the resumed run does not have them and it repeats the epoch.
Why does my job hang at startup instead of during training?
That is a different problem, and a checkpoint does not help with it. If the job hangs in the process-group constructor or at rendezvous and never gets past initialization, no step has run, so there is nothing to save and nothing to come back to. Those hangs are usually a networking fault or a version mismatch, and you settle them by reading the configuration.
How do I checkpoint and restore a distributed job so one node's failure does not mean starting over?
A kernel-level primitive can checkpoint one process on one GPU, but it does not handle the collective communication in transit between GPUs at the moment of capture, so coordinating the ranks of one job is the layer above single-GPU capture. The hard part is coordination, not capture. Closing that gap means capturing the whole group at regular heartbeat intervals and restoring the last group checkpoint on healthy GPUs, with the communicator state restored rather than rebuilt. With Cedana that ships today for every GPU of a job on one node. A job whose GPUs sit in several machines needs the multi-node tier, which is in design partnership and not yet in production.
