CRIU GPU checkpointing with cuda-checkpoint

TL;DR

CRIU, the standard Linux checkpoint and restore tool, does not checkpoint a GPU workload on its own. Its own documentation rules out any process that has opened a hardware device.
CRIU's CUDA plugin closes part of that gap by calling NVIDIA's cuda-checkpoint utility. That utility suspends the CUDA state of one process so an ordinary CPU checkpointer can take it from there.
The pairing is real, documented and maintained, and it stops short of a running workload. The limits are one process at a time, a restore pinned to the same GPU type, count, order and driver version, no MIG and no MPS, and open issues on vLLM, NCCL and killing a restored process.
What is left over is the layer above the tools. That layer holds every GPU of one job at the same point, captures the container, the storage and the scheduler around it, and decides when to checkpoint and where to restore.
In this piece we walk through what CRIU covers, why a GPU needs a plugin, what cuda-checkpoint captures, and what a restore requires. We also cover the open issues in the askers' own words, and what is still yours to build.

CRIU covers the CPU side of a GPU workload

If you have been asked to price the open-source option first, CRIU and NVIDIA's cuda-checkpoint are that option. Both are real, documented and maintained, and they are used by Cedana, NVIDIA's Dynamo Snapshot and MemVerge's Kubernetes operator.

CRIU stands for Checkpoint/Restore in Userspace, a Linux project that saves a running program to disk and brings it back later. It can freeze a running container, or a single application, and checkpoint its state to disk, and it is integrated into OpenVZ, LXC/LXD/Incus, Docker, Podman, Kubernetes and other software.

Your GPU workload has a CPU host process wrapped around the accelerator work, holding a process tree, open file descriptors, sockets to its peers and to the scheduler, mapped weight files, and host memory. That host process is the part CRIU knows how to save. NVIDIA's cuda-checkpoint README describes the coverage as every kernel mode resource associated with a process: anonymous memory, threads, regular files, sockets, and pipes between checkpointed processes.

Does CRIU support GPU workloads?

Not on its own, and the reason has nothing to do with accelerators in particular. CRIU's page on what cannot be checkpointed gives a general rule about devices: a task that has opened or mapped any character or block device wants a connection to the hardware, and in that case "dump (and restore) is impossible."

A GPU is a character device. The kernel presents it to a program as a file, and the program reads and writes that file directly. Whatever the GPU holds for the process sits behind that interface, so a general-purpose checkpointer has no portable way to ask for it back.

That is why the GPU needs a plugin, which is code that teaches CRIU how to handle a resource it does not understand by itself. CRIU's page on GPU checkpointing says that GPU state such as device memory, contexts and queues lives outside normal process address space and needs special handling, so CRIU relies on vendor-specific plugins.

The answer on CRIU's tracker is blunter: "GPU workloads in general are not supported by CRIU, may just crash on restore."

The CUDA plugin hands the GPU to a driver utility

The plugin does not checkpoint the GPU either, because it calls out to NVIDIA for that part. CRIU's CUDA plugin, in the project's own source tree, runs on GPU driver r555 or higher and needs NVIDIA's cuda-checkpoint utility for four operations, named lock, checkpoint, restore, and unlock. Lock runs with the PAUSE_DEVICES hook while the process is still running, to quiesce the application, meaning to bring it to a standstill where a checkpoint can be taken.

The plugin's documentation records three limits of its own, as of September 2026: what a checkpoint costs in host memory, where it can be restored, and which CUDA features are excluded.

Device memory leaves through system RAM. GPU memory contents are brought into main system memory and CRIU then checkpoints that as part of the normal procedure, which on systems with many GPUs and high GPU memory usage can cause memory thrashing.
A restore currently requires a system with similar GPUs and the same GPU count, so it is pinned to the hardware the checkpoint came from.
Three CUDA features are excluded outright: NVIDIA UVM Managed Memory, MIG (Multi Instance GPU) and MPS (Multi-Process Service) are not currently supported for checkpointing.

The work that put GPU support into CRIU is called CRIUgpu, and its changes shipped upstream in CRIU version 4.0.

What the driver primitive captures

cuda-checkpoint is a command-line utility from NVIDIA that checkpoints and restores CUDA state inside a running Linux process, with no change to the process itself. Suspending that process runs four steps.

CUDA driver calls are locked.
Work already submitted to the GPU is completed.
Device memory is copied to the host, into allocations managed by the CUDA driver.
All CUDA GPU resources are released.

The fourth step is the hand-off. Once its GPU resources are released, a suspended CUDA process no longer refers to any GPU hardware at the operating system level, so an ordinary CPU checkpointer can pick it up from there, which is the job CRIU does in the pairing.

The primitive leaves the CPU side alone by design. A standalone invocation of the driver-level tool does not handle the state of processes and threads running on the CPU, and the CRIUgpu paper, submitted on 23 February 2025, records that this can result in undefined behavior. So whoever builds the layer above has to stop and capture the threads.

Does cuda-checkpoint need spare host memory equal to the GPU's?

Yes. The third step copies device memory into host memory, so a process has to be able to allocate as much main memory as it holds on the GPU before a checkpoint can be taken. An issue on the cuda-checkpoint tracker records what that costs: the application being checkpointed has to be able to allocate sufficient main memory to hold a copy of all of its allocated GPU memory, which the reporter calls "quite prohibitively expensive" for some applications. The request behind it, writing straight to a file instead of going through host memory, is still open.

An engineer on NVIDIA's developer forum asked for the same thing from another angle, direct access to the dumped memory so the data does not have to be routed through CRIU. There is no direct-to-disk option today. That requirement applies to Cedana too, because we call the same driver primitive: the host still has to hold a copy of what is on the GPU.

Can you restore a checkpoint on a different GPU than the one it was taken on?

You can move a checkpoint between machines, but only onto GPUs that match the ones it was taken on. Start with the README, which publishes cuda-checkpoint's limits as flat lines.

"x64 only."
"Acts upon a single process, not a process tree."
"Doesn't support UVM or IPC memory created with cuMemExportToShareableHandle()."
"Doesn't support GPU migration."

The same list says the utility "waits for already-submitted CUDA work to finish before completing a checkpoint", and that it "supports display driver version 550 and higher". That list predates driver 580, and the migration line is the one that has since changed.

The CUDA driver API specifies the hardware requirement: restoring needs a GPU of "the same chip type as the old GPU" with enough memory.

An NVIDIA engineer answered the question of restoring on a different machine, on the public tracker, in issue 18, opened 12 November 2024. "Right now on the restore end it's required that the GPU type and order be the same as the checkpoint side."

Within that rule a checkpoint can move. "So you can migrate your CRIU checkpoint from machine to machine as long as the layouts are identical at the moment." The engineer's example is a checkpoint taken on a 4-GPU A100 system, which fails to restore on an 8-GPU A100 system.

The same thread rules out a different driver version and a different architecture. "The driver version to restore to for CUDA has to be the same." Asked whether a checkpoint taken on one architecture would ever restore on another, the engineer wrote, "We don't have any plans for it at the moment."

Driver 580 did add GPU migration, in the form of remapping: the vLLM project's checkpoint and restore proposal of 10 February 2026 records that "Driver 580+ supports GPU UUID remapping via CUcheckpointRestoreArgs". So the rule is the same chip type, the same count, the same order and the same driver version, and the remapping decides which card inside that layout the work lands on.

The open issues, in the askers' own words

Every issue below was open on the cuda-checkpoint or CRIU tracker as of September 2026.

A vLLM server restores on one GPU and on four, and not on two. Issue 27, opened 22 April 2025, reports that "cuda-checkpoint can successfully restore a vLLM server when it is using 1 or 4 GPUs, but cannot restore it when it is running in 2 GPUs specifically." Reports kept arriving through 2026, and by February 2026 driver 590 had changed the symptom to a segmentation fault on the checkpoint itself, still unresolved in March 2026.

Checkpointing one process of a distributed job hangs instead of returning. "I'm seeing cuda-checkpoint --toggle hang when checkpointing one process from a PyTorch torch.distributed (NCCL) job." That is issue 45, opened 10 February 2026, and nobody had commented on it by September 2026. A second reporter on the same hang was told that NCCL applications are not supported for checkpointing yet.

NVIDIA's own bundled example fails on a partitioned GPU. "When MIG is enabled on a GPU, running the basic counter demo also fails." An NVIDIA engineer wrote on 17 December 2025 that "MIG isn't supported right now", and on 24 January 2026 the same engineer pointed askers at the Kubernetes Checkpoint/Restore working group rather than at a date.

CRIU's AMD GPU plugin, the one GPU path that does not run through NVIDIA's utility, breaks on a real pod. "When I try to restore a PyTorch example workload running in podman container, the workload hangs after restore. Both checkpoint and restore logs shows success." Six threads on CRIU's tracker report that pattern, two of them filed by CRIU's own maintainer against the plugin.

Killing a restored process took the machine with it. "The problem is killing it: terminate a live restored process and ~35 seconds later the whole machine is gone." That was on bare metal with driver 595.71.05, in issue 53, opened 12 June 2026.

A restore that fails after running out of memory leaves nothing to retry. "The process is permanently stuck: restore returns 304, unlock returns CUDAERRORILLEGALSTATE (401)."

One team assembled it and published what it cost

Two Reddit posts, on r/mlops and r/Vllm, describe the same working result: a live 27B-parameter model and its vLLM server were checkpointed and restored in 11 seconds, against a 104-second cold start. The poster also listed the parts that fought back. "The tricky parts were restoring the full vLLM process tree, CUDA state, IPC/shared memory, and dealing with io_uring".

The mechanism works, but getting one workload to checkpoint and restore takes integration work. Other people are still asking CRIU's tracker how to do the same thing. "So what is the correct procedure to successfully dump a GPU workload (SGLang) running inside an NVIDIA container with cuda-checkpoint + CRIU?" Threads asking it are open without a confirmed fix.

What is covered, and what is still yours

What has to be saved, or decided	CRIU on its own	CRIU with the CUDA plugin	Still yours to build
The CPU process, meaning its memory, threads, files, sockets and pipes	Covered	Covered	Nothing
GPU device state, meaning device memory, CUDA contexts, and command queues	Not covered, and the dump is refused	Covered for a single Linux process, on a supported display driver	Matching the destination to the GPU type, count and order the checkpoint came from
Every GPU of one job held at the same point in the computation	Not covered	Not covered, with no NCCL support as of the CRIUgpu paper	The coordination across the job's GPUs
The state around the workload, meaning file system, network, container and scheduler	The process's own files and sockets	The same	The container, the storage the job uses, and the scheduler's view of it
Whether to checkpoint now, and where to restore	Neither, because CRIU is a command somebody runs	Neither	The layer that decides

The top two rows are covered, with the restore conditions stated above applying to the second. Everything in the bottom three rows is yours to implement. The hard part is coordination, not capture.

What does it take to build this yourself?

A first working checkpoint is a scoped project, covering one process on one GPU, one serving engine, one scheduler and one container runtime. After that, the system has to support more workloads and configurations. A production system has to handle multi-GPU coordination across NCCL collectives, RDMA networking captured mid-collective, multi-node workloads, every inference engine your users bring, and every workload type. That list grows with each new GPU generation.

Re-validation continues after that, because a driver update or a new engine release sends the whole matrix back for testing.

How Cedana builds on these tools

That missing layer is the one we build. Cedana calls CRIU for the CPU side of a workload and the driver primitive for the GPU side: the daemon takes a checkpoint request, runs it through a list of adapters, and then sends that request to CRIU, with checkpoint streaming from a public fork of CRIU's image streamer. On the GPU side our documentation ships two installable backends, the plugin we wrote and the community CRIU CUDA plugin.

NVIDIA's restore rule is our rule too. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

Our coordination support differs between single-node and multi-node workloads. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. Design partnership means we are building that tier with those customers rather than shipping it.

The category you have been pricing is automated checkpoint and restore for GPU workloads, where CRIU and cuda-checkpoint give you the capture and the rest of the table stays with you. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. Next to CRIU and cuda-checkpoint, that means we build on the primitive rather than replacing it, and we do not decide where a job goes: an operator or an agent starts the move.

A job that comes back still costs a restore plus whatever it did after its last checkpoint. You will be running CRIU and the driver primitive either way, so what you are deciding is how much of the remaining integration work you want to own.

Common questions

Does CRIU support checkpointing GPU workloads?

Not by itself: its own tracker states that "GPU workloads in general are not supported by CRIU, may just crash on restore." CRIU's CUDA plugin closes part of that gap by calling NVIDIA's cuda-checkpoint utility to suspend the GPU state first, then checkpointing the rest of the process itself.

Can a checkpoint be restored on a different physical GPU than the one it was taken on?

Restoring needs a GPU of the same chip type as the one the checkpoint was taken on, with enough memory, and NVIDIA's engineers have confirmed the GPU type, count and order all have to match on the restore side. A checkpoint can move between machines as long as that layout is identical. Driver 580 added GPU UUID remapping, which decides which card inside that identical layout the work lands on.

Should you build your own checkpoint and migration tooling, or use an existing one?

A first working checkpoint, covering one process, one GPU, one serving engine, one scheduler and one container runtime, is a scoped project. A production system also has to handle multi-GPU coordination, RDMA networking, multi-node scale and every inference engine and workload type your users bring, and re-validate all of it every time a driver or engine updates. You will be running CRIU and the driver primitive either way, so the decision is how much of that remaining integration work you want to own.

Why does cuda-checkpoint --toggle hang on a distributed training or inference job?

One open issue reports cuda-checkpoint --toggle hanging when checkpointing one process from a PyTorch torch.distributed (NCCL) job, and a second reporter on the same hang was told that NCCL applications are not supported for checkpointing yet.

Does cuda-checkpoint work with MIG?

No. NVIDIA's own bundled example fails when MIG is enabled on a GPU, and an NVIDIA engineer confirmed that MIG is not supported right now, pointing askers at the Kubernetes Checkpoint/Restore working group rather than at a date.

Does CRIU's AMD GPU plugin work in practice?

Not reliably. Six threads on CRIU's tracker, two filed by CRIU's own maintainer against the plugin, report the same pattern: a restored PyTorch workload in a podman container hangs after restore even though the checkpoint and restore logs both show success.

What CRIU and cuda-checkpoint do when you wire them together yourself