TL;DR
- You can take a container checkpoint on Kubernetes, Docker, Podman or containerd, and none of those platforms will restore it onto a different machine for you.
- Kubernetes built the checkpoint and left restore out on purpose. Docker's checkpoint and restore is experimental. Podman's archive does cross hosts, for the Linux side of a container. Slurm's requeue starts your batch script again at line one. Nextflow and Snakemake resume at the boundary of a task rather than inside the one that died.
- So on every one of these platforms the work inside a running GPU job has no saved copy, and an interruption costs the whole run so far.
- What changes that is a checkpoint taken below the application that holds the GPU state as well as the Linux state. It is restored on a node whose hardware and versions match.
- In this piece we walk through what each platform's own feature does, where its own trackers show the restore failing, and what happens to a live network connection. Then we cover why the two groups of tools stop where they do, and what it takes to restore a running job on a different machine.
The table shows what each platform's feature covers and where it stops, based on its documentation as of September 2026.
| Platform | What its own feature does | What comes back | In its own words |
|---|---|---|---|
| Kubernetes kubelet checkpoint API | Writes a stateful copy of a running container to disk | Nothing, since restore is left outside Kubernetes | "For the first implementation we do not want to support restore in the kubelet" |
| Docker checkpoint | Freezes a container into files on disk, as an experimental feature | The container, on the same host | "This command is experimental" |
| Podman checkpoint | Checkpoints every process in a container, into an archive that crosses hosts | The container and its processes | "The exported checkpoint can be used to import the container on another system" |
| Slurm requeue | Puts the job back in the queue | Nothing of the run so far | "When a job is requeued, the batch script is initiated from its beginning" |
| Nextflow -resume | Skips tasks whose cached hash and outputs still match | The finished tasks, not the running one | "If both conditions are met, the task is resumed" |
| Snakemake --rerun-incomplete | Runs the jobs whose output is incomplete again | The finished jobs, not the running one | "Re-run all jobs the output of which is recognized as incomplete" |
| DMTCP | Checkpoints a process tree in user space, with no change to the code | The processes; MPI needs its MANA plugin | "A DMTCP module for Slurm is coming" |
| CRaC | Checkpoints a warmed-up JVM so a restart skips the warm-up | The Java instance, given the privileges | "you may have to update your criu permissions" |
| Cedana | Not a platform feature; it installs beside Kubernetes and Slurm through its own integrations | The workload state, up to the last checkpoint | An operator or an agent starts the move |
Can Kubernetes restore a checkpointed container, or only create one?
Kubernetes will make the checkpoint and will not restore it for you. The kubelet, the agent Kubernetes runs on every node, has a CheckpointContainer API that writes a stateful copy of a running container to disk. The feature's own name is forensic container checkpointing, and forensic is the operative word: the announcement post of 2022 describes the copy as something you can analyze, and restore in a sandbox environment more than once, without the original container being aware of it.
Restore was left out deliberately. "For the first implementation we do not want to support restore in the kubelet," the enhancement proposal says, and the same document rules out reading it as a migration feature: checkpoint and restore can be used to implement container migration, but the proposal covers the forensic case only.
The API reached beta on 8 February 2024, and bringing the copy back is still work you do outside Kubernetes: the 2022 post says the checkpoint archive has to be converted into an image first.
Engineers keep asking for the missing half. One wants a way to checkpoint some containers and restore them on other nodes. Two enhancement proposals exist for that gap, KEP-2008 from 2020 and KEP-5823 from January 2026, filed five years apart, because container-level restore was not enough once a pod held more than one container.
On 21 January 2026 Kubernetes announced a Checkpoint/Restore Working Group, and its stated use cases include interruption-aware scheduling with transparent checkpoint and restore, so that lower-priority pods can be preempted while their work is preserved. The working group is addressing restore, and Kubernetes has not shipped it.
Docker never shipped the command people search for, and Podman went furthest
Docker can freeze a running container, and the command you would search for to bring it back does not exist. Its own command page calls checkpoint and restore an experimental feature, and experimental there means the feature may change between releases without warning or be removed entirely in a future release. There is no docker checkpoint restore command to find. Restore is a flag on a different command, docker start with --checkpoint, documented as restoring from that checkpoint.
CRIU is the mechanism under all of this, the Linux project that freezes a running process and writes it to disk, and its own Docker page recommends Podman as an alternative to Docker.
Podman is the most complete container implementation here. It checkpoints all the processes in one or more containers, and the archive it writes is meant to leave the machine: "The exported checkpoint can be used to import the container on another system and thus enabling container live migration." That claim covers the Linux side of a container. Neither the checkpoint nor the restore man page mentions a GPU, CUDA, or an accelerator, and Podman's issue tracker has no checkpoint report that mentions either.
The trackers are full of restores that hang or fail on cgroups and mounts
What people report failing is the restore. One Docker user cannot get past mountpoints and cgroups when starting a new container from a checkpoint. Another waits many minutes on the last command, and the container never restarts. On the Podman tracker, repeated runs of the same command took as little as 1.4 seconds and others as long as 31.5 seconds.
Crossing a machine boundary is where the documentation thins out. One asker finds many examples of checkpointing and restoring on the same machine and only one document about migrating checkpoints between different machines. On Reddit, someone asking about switching a pod from one node to another is told that the checkpoint API does not provide live pod migration.
A live network connection is the part that breaks first
CRIU can save and restore an established TCP connection, and it makes you ask for that explicitly. You pass the --tcp-established option, which CRIU's own page presents as the caller declaring it is aware of the transitional state of the netfilter: a per-connection firewall rule CRIU installs to block that connection's packets while it saves the socket. With the option set, CRIU reads the socket state and restores it back, letting the protocol resurrect the data sequence. One condition comes with that, and it is the one that matters when you change machines: the IP address the connection was using has to be available where the process comes back.
Container-platform users report failures when the connection is live at the moment of the checkpoint. One report describes a container with no active TCP connection restoring into a new one without trouble, and the same restore failing when a connection was open. Others record sockets dumped and not restored, listen sockets dropped silently, and ICMP not supported at all. Asked about the Kubernetes and CRI-O version of this, a CRIU maintainer answers that it is "completely unsupported and untested".
Slurm's checkpoint plugin is gone, and requeue restarts the script
Slurm had a checkpoint feature once, and it does not have one now. A 2016 capture of its checkpoint page says Slurm was integrated with Berkeley Lab Checkpoint/Restart, known as BLCR, to provide automatic job checkpoint and restart support. That page returns a 404 today, and the current slurm.conf reference has no checkpoint parameter in it.
What remains is requeue, which moves the job back in the queue rather than saving anything of it. The scontrol page defines requeue as putting a running, suspended or finished Slurm batch job into the pending state, and sbatch --requeue makes a job eligible for that upon preemption by a higher priority job. The sbatch documentation says what the job comes back as. "When a job is requeued, the batch script is initiated from its beginning." So nothing the run had built up comes back with it.
Administrators have asked the Slurm users list about the gap for years. One wants to checkpoint a running task, shut it down, and then restore it somewhere. A thread from May 2026 answers the question with the documentation, which says checkpointed jobs are not automatically restarted.
Nextflow and Snakemake resume the pipeline, not the process
Both tools resume at the boundary of a task, so a task that dies halfway through runs again from its own beginning. Nextflow's -resume reads a cache of the tasks that finished: it looks up a matching hash, checks that the task outputs are still in the work directory, and resumes the task when both conditions are met. Snakemake says its equivalent in one line, since --rerun-incomplete re-runs all the jobs whose output it recognizes as incomplete.
That leaves you with a pipeline that remembers everything except the step that was running. One user on the Seqera community has a pipeline stuck on step2(19) while the other 18 tasks cached successfully. On the Snakemake tracker, another reports that Snakemake will delete the checkpoint, wiping out days of work. A third finds that you cannot resume a previous run on a new compute environment, and one asker is told to check the checkpoint files and the S3 lifecycle policy, because missing or 0-byte files leave nothing to resume from.
DMTCP checkpoints MPI in its own description and single-node jobs in the reports
DMTCP describes itself as transparently checkpointing a single-host or distributed computation in user-space, with no change to the user's code, and university clusters recommend it on that basis. The reports from those clusters are narrower than the description. A sysadmin on r/HPC sums up what that looks like in practice: "For single-node jobs it seems to work OK. MPI jobs? Forget about it. Memory footprint on a checkpoint grows at about 8GB/sec until the OOM-killer does it's thing."
DMTCP's own pages support the reports more than they contradict them. For MPI, the standard by which the processes of one job talk to each other across nodes, DMTCP sends you to a second project: MANA, an implementation of transparent checkpointing for MPI built as a plugin on top of DMTCP. The Slurm integration is described as unfinished work, since "A DMTCP module for Slurm is coming". Neither the DMTCP site nor its README mentions CUDA or a GPU.
CRaC, the Java version of the same idea, has a different blocker. It checkpoints a warmed-up JVM so a restart skips the warm-up, and the CRIU binary underneath needs a privileged container or a setuid root binary to run at all. A platform engineer asking about CRaC on Kubernetes reports that in vanilla Docker it had to run with the flag --privileged or errors arose, and that question is unanswered.
Each feature stops at the edge of its own layer
Everything in the table stops at one of two places: a process-level checkpoint stops at the device, and a scheduler or a workflow manager never holds the running state in the first place.
Docker, Podman, containerd, the kubelet API, DMTCP, and CRaC all take a process-level checkpoint, and process-level checkpointing saves the Linux side and stops at the device. CRIU freezes a running container or a single application, writes its state to disk, and brings it back running exactly as it was at the moment of the freeze. The GPU sits outside what CRIU handles, deliberately, because device memory, contexts, and queues live outside normal process address space, so the project relies on vendor-specific plugins for them.
Slurm sits in the other group, and Nextflow and Snakemake sit above it. A scheduler places work and cannot move a running job, because it decides where work starts and whether it stops and holds nothing of the running state, which is why a preempted job is cancelled, requeued, or suspended. A workflow manager records which tasks finished, which is state about the pipeline rather than state inside the process.
What changes when you can restore a job on a different node
Moving a running job takes a checkpoint that holds what all of these features leave behind, the GPU side as well as the Linux side, and it has to be taken below the application so the program needs no checkpoint code of its own. That is the layer we work on at Cedana. Cedana preserves the entire workload state (CUDA context, KV cache, file system, network, scheduler), so workloads automatically resume on a new instance as of the last checkpoint. The saved work is the work inside the machine up to the last checkpoint. Checkpoints are taken on an interval the operator sets, so an interruption costs the restore plus whatever the job computed after the last one.
The destination is not any free node. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds.
None of this changes who decides where a job goes. We do not make that call: an operator or an agent watching the cluster starts the move, and the platform underneath keeps doing what it already does. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We capture below the application and bring the surrounding state along with the GPU state, so an interrupted job can come back on a compatible node instead of starting over. Whether that is worth adding to the platform you already run depends on how often your jobs are interrupted and what a lost run costs you.
Related:
- What CRIU and cuda-checkpoint do when you wire them together yourself
- Which checkpointing approach brings back the state your job is holding?
- Draining a GPU node in Kubernetes without losing the work on it
- Rebooting Slurm nodes for a kernel update without losing the running jobs
Common questions
I can checkpoint a container on one machine. Can I restore or migrate it onto a different machine?
None of Kubernetes', Docker's, Slurm's or Nextflow's own features restore a running GPU job onto a different machine. Podman's checkpoint archive does cross machines, for the Linux side of a container: its own documentation says the exported checkpoint can be used to import the container on another system, and neither its checkpoint nor its restore page mentions a GPU. Crossing a machine boundary is otherwise where the documentation of these tools thins out, and on Reddit someone asking about switching a pod between nodes was told that the checkpoint API does not provide live pod migration.
Can Kubernetes restore a checkpointed container, or only create one?
It can only create one. Kubernetes checkpoints a container through the kubelet's CheckpointContainer API, and restore was left out deliberately: "For the first implementation we do not want to support restore in the kubelet," the enhancement proposal says, and the same document rules out reading the feature as a migration feature. A Checkpoint/Restore Working Group announced on 21 January 2026 is addressing restore, and Kubernetes has not shipped it.
Are Docker's, Podman's and containerd's checkpoint features usable for anything real?
Not for moving a running GPU job, and the restore path is where the reports pile up. Docker's own command page calls checkpoint and restore an experimental feature that may change between releases without warning or be removed entirely, and there is no single restore command, only a --checkpoint flag on docker start. Podman is the most complete implementation, checkpointing every process in a container into an archive that can import into another system, but its own tracker shows the same command taking as little as 1.4 seconds and as long as 31.5 seconds to run. Trackers for all of them are full of restores that hang on mountpoints and cgroups or simply never restart.
Does Slurm restore a running job automatically after a checkpoint?
No, and Slurm's own checkpoint feature is gone: a 2016 capture of its checkpoint page now returns a 404, and the current slurm.conf reference has no checkpoint parameter. What remains is requeue, which the sbatch documentation says restarts the batch script from its beginning rather than resuming it, and a May 2026 thread on the Slurm users list answers the question with the documentation: checkpointed jobs are not automatically restarted.
Can Nextflow's -resume or Snakemake's --rerun-incomplete pick up a step that is still running?
No. Nextflow's -resume reads a cache of finished tasks and resumes only if the hash and outputs still match, and Snakemake's --rerun-incomplete re-runs jobs whose output is recognized as incomplete. Both work at the boundary of a task: a task that dies halfway through runs again from its own beginning, not from where it stopped.
What happens to a live TCP connection when a checkpointed process is restored on a different machine?
CRIU can save and restore an established TCP connection using its --tcp-established option, reading the socket state and restoring it so the protocol resurrects the data sequence, and the address the connection was using has to be available where the process comes back. In practice, users report that a container with no active TCP connection restores cleanly, but restoring one with an active connection fails, and a CRIU maintainer has called the Kubernetes and CRI-O version of this completely unsupported and untested.
