TL;DR
- Products, libraries and driver features all say they checkpoint GPU workloads, but how much of your running job comes back depends on which level does the work.
- Application-level checkpointing works well and is cheap in bytes, but only inside the one program it was written for, and only for the state its author chose to save.
- System-level GPU checkpointing is what you reach for when you cannot or do not want to change the application. It is the only way to save state the application never chose to include in its checkpoints, such as an inference worker's sessions in flight.
- A driver primitive is a third thing. It covers CUDA state on one GPU and leaves the file system, the network, the scheduler and the coordination across GPUs to you.
- In this piece we walk through what separates the two levels, where application-level checkpointing stops, and what a checkpoint primitive covers. Then we cover why every GPU in one job has to be captured at the same instant, and the six criteria any system-level claim has to clear.
What is the difference between application-level and system-level GPU checkpointing?
Application-level checkpointing lives inside the program, so what comes back is whatever the program's author decided to save. A molecular dynamics program writes a checkpoint file at intervals its author chose, in a format its author designed, and resumes through a restart path its author wrote. A training script saves the model weights at the end of an epoch. The program decides what matters and saves only that.
System-level checkpointing lives below the program, in a layer between the operating system and the container runtime, and it captures the whole running process. The capture includes the memory on the GPUs and the CUDA context around it, the process and its own memory, the files it has open, its network connections, and what the scheduler knows about the job. Nothing in the program changes, so a researcher on Slurm keeps using sbatch and squeue, and an operator on Kubernetes keeps the same pod spec apart from the lines that enable checkpointing.
Both levels save state, but they differ in how much of the running job comes back and in whether checkpointing support had to be written into the program in the first place. That is the decision in front of you: if you can change the application and the state you care about is state the program already knows about, the application level is enough, and if you cannot or do not want to change it, the work has to happen below it.
Application-level checkpointing works well, but only inside one program
GROMACS ships application-level checkpointing as a feature of the program, which makes it the clearest case of what the level gives you and what it asks of you. Its manual sets out the mechanism: "With option -maxh a simulation is terminated and a checkpoint file is written at the first neighbor search step where the run time exceeds -maxh*0.99 hours." The file holds the positions, velocities, and state the algorithms need to restart.
GROMACS also writes a checkpoint on a timer, at an interval set by its -cpt flag, and a run continues from the latest one with -cpi. So a researcher who sets -maxh stops cleanly at the wall-time limit with a checkpoint just written, but one who forgets it is cut off mid-run and repeats the work done since the last timed checkpoint.
For any of that to work, the program has to know which state matters, serialize it correctly, write it safely, and expose a restart path that works, and the user has to know to set the switch. One of the most mature programs in HPC needs a dedicated command-line option to stop cleanly at a scheduler's wall-time limit. None of that work carries over to the next program, because every program holds different state and writes it in its own format and on its own schedule.
The authors of CRAFT, a library built to make checkpoint and restart (CR) easier for scientific programs, put the tradeoff in one sentence: "Application-level CR is the most effective CR technique in terms of overhead efficiency but it takes a lot of implementation effort." It is efficient because the user picks what to save and can save the minimum. It takes implementation work for the same reason.
A scheduler cannot save the state that the program did not save. Slurm's sbatch documentation says that "When a job is requeued, the batch script is initiated from its beginning." So Slurm can put your job back in the queue, but it has nothing to resume unless the program finds a valid checkpoint and loads it.
None of this is new: the classic analyses of how often to checkpoint go back to Young's work in 1974. What has not changed is that the work sits inside each program, so every team writes and maintains its own checkpoint logic.
A cheaper checkpoint can cost more when the job fails
The CRAFT verdict counts bytes, and on bytes it is right: a program saves only the data it knows it needs, so its checkpoint is small. That is a real advantage, but the other costs do not show up in the byte count.
The checkpoint is taken at the point the program chose, so the work done since then is gone. In many training systems, that point is the end of an epoch, which can span hundreds of steps. Someone has to notice the failure and act on it, and the saving code has to be written and maintained one program at a time.
A system-level checkpoint saves the whole process, so it is larger. It needs no code change, and it runs on a schedule you set without any checkpoint logic in the program, so it can run often, which leaves less work at risk between checkpoints.
Our own view is that application-level checkpointing carries more overhead in practice once you count freeze time, how often checkpoints are taken, and the cost of each failure together. That does not contradict the CRAFT finding, which measures bytes per checkpoint. Which cost matters more depends on how often your jobs fail and how much of a job you can afford to lose.
A checkpoint primitive covers the GPU and leaves the rest to you
Within system-level checkpointing there is a second split, between a primitive in the driver and a system built around it. A kernel-level primitive pauses one process, waits for the GPU work already submitted to finish, copies device memory to the host, and leaves the process in a state that a CPU checkpointer can save.
That CPU checkpointer is a tool such as CRIU (checkpoint and restore in userspace), which saves Linux kernel resources such as memory, threads, files, and sockets. GPU state lives outside the process address space, so CRIU reaches it through vendor-specific plugins rather than on its own. Today's kernel-level primitives are single-GPU and ship as building blocks rather than products.
Everything the job needs beyond CUDA state has to come from somewhere else. The file system, the network, the scheduler's view of the job, the coordination across the GPUs in that job, and policy all sit outside the primitive, and a checkpoint primitive on its own does not decide when to act.
Every GPU in one job has to be captured at the same instant
A copy of a large model does not sit on one GPU. It is sharded across a tensor-parallel group whose members exchange partial results through collective operations over NVLink and NCCL, the NVIDIA Collective Communications Library. Every GPU in that group has to be captured at the same point in the distributed computation, with no collective left half-complete.
If you get that boundary wrong, the restored state is silently corrupt: it corresponds to no real moment in the computation, but no error is raised. The boundary also decides whether a checkpoint is worth anything at all, because losing one GPU on a 16-GPU system forces all 16 to stop, so a checkpoint that cannot restore the whole group restores nothing. The hard part is coordination, not capture.
Six criteria any system-level claim has to clear
Products, libraries, and driver features all say they checkpoint GPU workloads, but the amount of running state they save varies with which of these two levels does the work, so what you want from a vendor is a test rather than a claim. The six criteria below apply to any system-level claim, ours included, and each row names a criterion, explains what it means, and shows how to test it in a vendor's product or something you build yourself.
| Criterion | What it means | How to test it |
|---|---|---|
| Consistency | Every GPU in the job is captured at one boundary in the computation; across-node consistency applies only where the product claims across-node support | A restore continues the same execution instead of starting a new one and produces the same output as an uninterrupted run |
| Transparency | No code change in the application and no checkpoint logic in the serving path | sbatch and squeue, or the serving engine, are unchanged; the researcher's workflow is the same before and after |
| Coverage | Workloads (inference, fine-tuning, scientific programs), engines, runtimes and schedulers, and GPU generations | A support matrix the vendor will show you, because the rows never stop growing |
| Speed | Checkpoint and restore in seconds, with a restore time that tracks the bytes in the checkpoint rather than the model's parameter count | A measurement against a native start on the same hardware, with the clock defined, meaning what starts it and what stops it |
| Correctness | The restored worker continues the same execution from the exact in-flight position, with CUDA semantics preserved | Output continuity across the restore; silent corruption is worse than a restart |
| Overhead | Low enough to leave the checkpointer on during normal operation | Steady-state throughput with the checkpointer running, compared with native throughput on the same hardware over a run long enough to amortize the cost |
Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. That is why the speed row asks you for a clock and a comparison rather than a number. Checkpoint bytes can also grow with a model's parameters, so a slower restore on a larger model does not by itself prove that the product is reloading weights rather than restoring the job. The byte count and the defined clock are what settle that.
A system-level checkpoint also covers the programs that checkpoint themselves
The two levels are not rivals. We have demonstrated Cedana's system-level checkpoint and restore on GROMACS, the program that ships its own wall-time flag, and on Boltz-2, with no code change in either. Capturing below the program covers programs that never wrote a checkpoint of their own, and it holds state that an application-level checkpoint was never designed to hold, such as the in-flight sessions of an inference worker.
The Consistency criterion also applies to our own product scope. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
So the level is the thing to settle first, and the six criteria are how you hold whoever you pick to it. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. In this comparison, that means we work at the system level, capturing below the application and the serving engine, so the state your program never chose to write down comes back with the job. Hold us to the six criteria above just as you would hold anyone else to them.
Related:
- Which checkpointing approach brings back the state your job is holding?
- What CRIU and cuda-checkpoint do when you wire them together yourself
- Does Kubernetes, Docker, Slurm or Nextflow already restore a running job?
- GROMACS shows both the value and the limit of application checkpointing
- What is GPU checkpointing? A plain explanation
Common questions
System-level or application-level checkpointing: which do I need?
Application-level checkpointing works well, but only inside the one program it was written for, and only for the state its author chose to save, so it needs a code change and its own maintenance in every program that wants it. Choose system-level checkpointing when you cannot or do not want to change the application, or when the job holds state an application-level checkpoint was never designed to hold, such as an inference worker's in-flight sessions: it needs no code change and runs on a schedule you set without checkpoint logic in the program.
Does Slurm's requeue restart a job from where it stopped?
No. Slurm's own documentation says that when a job is requeued, the batch script is initiated from its beginning. A scheduler can put a job back in the queue, but it has nothing to resume unless the program itself finds a valid checkpoint and loads it.
