TL;DR
- Some GPU jobs are worth saving so they can resume after an interruption, and some are cheaper to run again from the beginning. The question is which of yours is which.
- The classic checkpoint-interval formulas do not settle it, because they tell you how often to save once you have already decided to save. Saving everything charges every run for a failure that most runs never meet.
- Four things decide it: how often work of this size is interrupted, how much work an interruption destroys, what it costs to rebuild that work, and whether the schedule has room to rebuild it.
- A retry is the cheaper answer when the state can be rebuilt from something durable. GPU checkpointing earns its cost when the only copy of the state is in GPU memory.
- In this piece we walk through the four inputs one at a time, the table that puts them together, and the work that is clearly on the retry side. Then we cover the work where a retry costs the most, and the side effects that neither a retry nor a resume can undo.
How often a job gets interrupted follows how many GPUs it spans
An 8-GPU job on two of Meta's multi-tenant A100 research clusters had a mean time to failure of 47.7 days. The same measure for a job spanning 1,024 GPUs was 7.9 hours. Kokolis and colleagues took both figures from 11 months and 4 million jobs on those clusters and published them in Revisiting Reliability in Large-Scale Machine Learning Research Clusters.
Those are failure rates at a stated job size on particular hardware, so they are not a per-GPU rate you can lift onto your own fleet.
Attributed hardware failures touched 19% of GPU runtime and less than 1% of jobs in the same study, so most jobs never encounter a failure. The ones that do carry most of the runtime, because failures land on the long jobs with the most work in flight. So a short job on a handful of GPUs is unlikely to be interrupted at all.
Under a retry, the work at risk is everything since the job started
A retry runs the job again from the beginning, so it destroys whatever exists only inside the running job. Anything the job already wrote to a database or a file survives. The loss grows with elapsed time rather than with the failure, so an hour into the job an interruption costs an hour, and a day in it costs a day.
A resume brings back saved state and carries on from it, so the loss is limited to work done since the last save, and the length of the run no longer sets it.
What it costs to rebuild the work depends on where the state came from
For a short stateless request, the rebuild costs one more request, because you send it again and the model computes it again.
A large serving worker costs far more, because its weights have to be read back into GPU memory before it answers anything. Alibaba Cloud's deployment documentation puts that load at 20 to 30 minutes for the full-version DeepSeek-R1, and the same document says that model needs an 8-GPU node. None of that time serves a request.
The rest of the state has to come back too, and where it came from decides how hard that is. What was read out of a database or an object store can be read again, but what the job computed itself has to be computed again.
A deadline can override the other three inputs
If the result has to land tonight, a second full run does not fit, however cheap the rebuild would be. Work with slack in the schedule can absorb a retry, and work without it cannot, whatever the interruption rate says.
Saving state costs you on every run, whether or not a failure arrives
Saving state costs time, storage, and bandwidth every time you do it, and you pay that on the runs that finish without incident.
Building the saving into the application costs engineering time on top. The authors of CRAFT, a library for application-level checkpoint and restart, call the approach "the most effective CR technique in terms of overhead efficiency" and say in the same paper that it "takes a lot of implementation effort". In Cedana's experience, teams take application-level checkpoints less often because each one is so expensive, which makes every failure cost more.
Young's first-order approximation in 1974 and Daly's higher-order estimate in 2006 tell you how often to save once you have decided to save, and both of them start from the assumption that you are already saving. The question here is whether to save at all.
The four inputs, and where each one points
| The input | A retry is the right call when | A resume is the right call when |
|---|---|---|
| How often work of this size is interrupted | The job is short and spans few GPUs, so an interruption is unlikely before it finishes | The job is long or spans many GPUs, so an interruption is likely before it finishes |
| How much work an interruption destroys | Losing the work done since the job started would cost little | Losing that work would cost a lot, and it costs more the longer the job runs |
| What it costs to rebuild that work | The state can be rebuilt from something durable, such as a database, an object store, or the request itself | The only way to rebuild the state is to do the work over |
| Whether the schedule has room to rebuild it | The deadline can absorb a second full run | The deadline cannot absorb a second full run |
A workload on the retry side of all four rows does not need saved state, as long as it is safe to run twice. If a retry would destroy a lot of work and nothing durable can bring that work back, saving state usually pays, even when interruptions are rare.
Work whose state lives somewhere else is the clearest retry case
A dead web server just restarts and reconnects, because its state lives in a database. That works because the state is small and sits somewhere durable rather than inside the process using it.
A short request sits on the low-value side of the same decision, because recomputing it is cheap and the worker serving it stays up, so the cost of checkpointing there can exceed the benefit. Retry logic in the application handles it, and we say the same about our own product at Cedana.
Stateless work that is safe to run twice belongs on the same side, and the word for that is idempotent, meaning that running the step again gives the same result as running it once. If your work is idempotent, its state lives in a database or an object store, and the deadline has room for a second run, then a retry is the cheaper answer and a checkpoint is spending you did not need.
A retry costs the most when the only copy of the state is in GPU memory
Inference now accumulates session-scoped state that lives only in VRAM, can be reconstructed only by redoing the work, and is destroyed by any interruption. VRAM is the memory on the GPU itself, and nothing above it holds a copy of what the session built.
Retrying one request costs that request's prefill again, meaning the pass in which the model reads the whole prompt before it produces a token, and restarting the whole worker costs its start-up on top. The rebuild cost for an interrupted session is the whole session's context, and it grows with every turn the session runs. The retry re-runs rather than resumes.
Work that reaches outside the machine needs idempotency keys either way
Neither a retry nor a resume prevents an external side effect from happening twice. A checkpoint captures GPU state and not external world state, so if a step has already sent an email or written a row, running it again sends the email again, and bringing back saved state does not undo it either.
The application layer is the only layer that can prevent duplicate execution, and idempotency tokens, deduplication, and retry logic live there. Work of that kind needs them whichever recovery mechanism sits underneath.
Of the four inputs, only one of them changes when the state itself can be saved and brought back. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. What that changes here is the cost of rebuilding the work, because state that lives only in GPU memory is copied back rather than computed again. The other three inputs you still read off your own jobs, and the jobs that come out on the retry side of all four rows are the ones to leave alone.
