TL;DR
- GPU failure prediction works: models reading ordinary telemetry reached 84.0% precision inside a one-day window. What it does not tell you is what to do with the job running on the card.
- Every fleet health system's repair path after a cordon ends that job, so acting on the warning costs about what the failure would have cost.
- At 84% precision, about 16 of every 100 firings are wrong. Each wrong one destroys a healthy worker's in-flight work on a machine that was going to be fine.
- Checkpoint the job and restore it on healthy hardware instead, and acting costs a restore plus the work since the last checkpoint. The detector's precision then stops deciding what you can afford.
- In this piece we walk through how good prediction is today, what a fleet health system does when the warning fires, and what keeping bad nodes out of the scheduler already buys you. Then we cover the rule for when acting is worth it, and what changes when acting costs a checkpoint and a move.
How good is GPU failure prediction today?
Prediction is good enough that its accuracy is not what stops you acting on it. Liu and colleagues, in "Prediction of GPU Failures Under Deep Learning Workloads", built failure models on a 4-month production dataset from deep learning clusters. They raised prediction precision from 46.3% to 84.0% inside a one-day window, reading ordinary telemetry including temperature, power consumption, GPU and memory utilization, and machine uptime.
NVIDIA predicts failures from its own fleet data too, under the name Fleet Intelligence. A prediction capability alone does not establish a fixed warning interval for an individual GPU, so the useful question is not how far ahead the warning comes but what you do with it.
The reason to predict at all is what a single failure costs you. In distributed training, one GPU failure can interrupt a long-running job and lose hours of work across multiple machines, and Liu and colleagues name that loss, along with inference-service interruptions, as the reason to predict failures.
Some failures give you no warning at all. A GPU can become unavailable at the driver or the interconnect and drop off the bus, and for that class Bidollahkhani and co-authors report that conventional GPU telemetry "often remains nominal until failure". In most of the configurations they tested, the median lead ahead of the failure was zero.
Where a prediction does arrive, it gives you a probability: your fleet says a card is likely to fail in the next day or two, and you have to decide what that is worth to the running job on it. The prediction work leaves that decision open, because the day-ahead paper suggests proactive maintenance and rescheduling but evaluates prediction performance rather than the cost of either response to a running job.
What a fleet health system does when the warning fires
NVIDIA, Crusoe, and Nscale publish fleet health systems that take a cluster from a GPU health signal to a repaired node with nobody in the loop. Their defaults differ, but every repair path ends by taking the affected hardware away from the job running on it.
| System | Its published default | The action at the end of the loop |
|---|---|---|
| NVIDIA NVSentinel | "By default, only health monitoring is enabled", with fault quarantine, the node drainer and fault remediation switched on through Helm values as an operator builds confidence; NVIDIA's own deployments run with full remediation enabled | Cordon and drain, node conditions the scheduler can read, and external hooks that reset the GPU or reprovision the hardware |
| Crusoe AutoClusters | "When a failure is detected, the default behavior is automatic node replacement from our spare pool." | Cordon, then "a configurable grace period to checkpoint and terminate cleanly (default 30 seconds)", then a pre-validated spare joins in place of the failed node |
| Nscale Control Center | "Observability detects anomalies across clusters; Control Center responds automatically to remediate nodes, redeploy workloads, and restore balance before issues escalate." | Repair, reallocate, or retire the hardware |
Only the cordon leaves a running job alive, by stopping new work from landing on the node while what is already there keeps going. Everything from the drain onward ends the work, because a drain evicts the job and a reset, a reprovision, or a replacement takes the hardware away from it. Crusoe's grace period is the closest any of them comes to saving the job, but whatever a workload cannot write out in 30 seconds goes with the node.
Two providers hold the repair back rather than run it the moment the signal fires. CoreWeave times its response to the job's schedule, so a catastrophic error means an immediate reboot while a less severe issue stages the node "to be moved out of production whenever a customer's job finishes". Together AI has the detection and the repair actions built, but deliberately puts a person between them: "Your training checkpoints and inference replicas are too valuable to risk on automated drains."
Keeping bad nodes out of the scheduler already cuts failures
Detection pays off before a job ever starts. Kokolis and colleagues, in "Revisiting Reliability in Large-Scale Machine Learning Research Clusters" (HPCA 2025), describe a detector for lemon nodes, the machines that fail far more often than the rest. It found them on Meta's research clusters with more than 85% accuracy, and taking them out of scheduling cut the failure rate from 14% to 4% for jobs of 512 GPUs and larger.
Taking those nodes out of scheduling protects work that has not started, because declining to place a job on a suspect node avoids exposing it to that node without interrupting anything. Meta's paper describes the cost of doing this by hand: excluding suspect nodes "is, however, not scalable and aggressive exclusion of nodes may lead to capacity starvation". So keeping suspect nodes out of scheduling reduces the fleet's failure rate, but it does nothing for the job already running on a node that is about to go bad.
Acting is worth it when it costs less than the expected cost of the failure
Act when the cost of acting is less than the chance of failure multiplied by the cost of failing. Of those three terms, the cost of acting is the only one you can change directly.
Today that cost depends on what the response does to the running job, and every response after a cordon terminates it. So acting on a warning means paying the failure's full cost at once and with certainty, to avoid a cost that was only probable.
With close to the same number on both sides of the comparison, the rule turns into a demand for near certainty, and no published detector meets it. At 84% precision, about 16 of every 100 firings are wrong, and under a response that ends the job each of those 16 destroys a healthy worker's in-flight work. The warning causes the loss it was meant to prevent, on a machine that was going to be fine.
The decision also arrives too often to settle one alert at a time, so you have to write down in advance the threshold where the two costs meet.
What changes when acting costs a checkpoint and a move
Saving the workload's state instead costs a restore, plus whatever ran since the last checkpoint, and both of those are numbers rather than guesses. Cedana's published benchmark measures the restore time on one node of 8 NVIDIA B200 GPUs: four frontier models cold start natively in 564 to 2,051 seconds on that node, and restoring the same models from a checkpoint takes 57 to 70 seconds. Frontier here means the latest open-weight models trailing the frontier labs. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count.
How much work you repeat is set by the checkpoint interval. A restore brings the workload back as it stood at its last checkpoint, so everything done after that checkpoint is done again, and with Cedana you set that interval yourself through a heartbeat checkpointing policy.
With a restore as the response, the cost of acting becomes the restore plus the last interval. The cost of failing falls to the same size, because a job checkpointed on a heartbeat also comes back from its last checkpoint after a failure nobody predicted.
The 16 firings in every 100 that used to cost a healthy worker's work now cost a restore each, so the detector's precision stops deciding what you can afford. Moving early becomes a scheduling choice rather than a bet on the warning, because you pay the recovery cost earlier and with certainty, and when you pay it is up to you. The same move covers the failures that arrive with no warning at all, because what protects the work is what happens after the event.
The prediction and the health signal still come from systems outside Cedana. Our control plane consumes DCGM, Prometheus, and NVSentinel events, so you do not run a parallel monitoring stack, and the model that forecasts the failure stays somebody else's.
Heartbeat checkpointing is live, and a workload resumes automatically through a failure on a new node. Acting before a predicted failure requires you to wire the prediction to the move, because a health signal does not trigger a migration on its own today.
Restoring on another node requires a compatible environment. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
How many nodes the job spans also decides what ships today. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
Plenty of tools predict and flag failures, and the question each warning leaves open is what happens to the job on the card. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a failure warning, that means the job comes back on healthy hardware with everything up to its last checkpoint intact. Acting on a prediction then costs a restore and an interval, and both are numbers you can measure on your own fleet.
Related:
- What to do when DCGM flags a GPU that has a job running on it
- GPU failure frequency scales with the fleet, not the on-call rotation
- Five fleet signals, five policies: from alert to automatic action
Common questions
Is there a way to see a GPU or job going bad before it fails, when every standard health metric still looks normal?
You often can see it coming. Prediction models built on ordinary telemetry, including temperature, power consumption, GPU and memory utilization, and machine uptime, reached 84.0% precision inside a one-day window in Liu and colleagues' study. Some failures still give no warning at all, because GPUs that drop off the bus at the driver or the interconnect had a median lead time of zero in the configurations researchers tested.
What do I do with a running job when a GPU failure warning arrives?
The fleet health systems from NVIDIA, Crusoe, and Nscale end in cordon-and-drain, node replacement, or remediation, and only the cordon leaves the running job alive. CoreWeave's variant defers a less severe repair until the customer's job finishes. With Cedana you checkpoint the job and restore it on healthy hardware instead, so the warning costs a restore rather than the running work.
Once a GPU is flagged or predicted to fail, what happens to the job that is already running on it?
Every fleet health system's repair path eventually takes the hardware away from the running job, because a drain evicts it and a reset, a reprovision, or a replacement ends it outright. Only a cordon, which just stops new work landing on the node, leaves what is already running alone.
Can a false-positive prediction end up killing a perfectly healthy, running job?
A false-positive prediction can do exactly that. At 84% precision, about 16 of every 100 firings are wrong, and under a response that ends the job each of those destroys a healthy worker's in-flight work, which is the very loss the warning was meant to prevent. With a checkpoint-and-restore response instead, that wrong firing costs a restore, not the run.
