GPU checkpointing: build or buy?

TL;DR

When your GPU job stops, the only state that comes back is the state something saved. A running job holds far more than the weights and optimizer values a developer thought to write out.
Checkpoints written into the application are cheap to run and expensive to build, and every team that runs a job pays that build separately. The open-source floor below them, CRIU plus NVIDIA's cuda-checkpoint, captures a single process and restores it only onto a matching GPU layout.
Getting from there to a production system is nine more components plus re-validation against every driver, engine release and GPU generation. That is the work no build estimate carries. MemVerge sells a system-level answer with two published limits.
Build when your scope is fixed. Buy when your cluster is mixed, when a job spans several GPUs, or when something other than a person has to trigger the checkpoint.
In this piece we walk through what each of the five checkpointing approaches captures, why application-level checkpointing costs what it costs, and where the open-source floor stops. Then we cover the nine components between a working checkpoint and a production system, when to build and when to buy, what MemVerge publishes about GPU checkpointing, and what a restore costs.

Where the state is captured decides what comes back

State here means everything the workload is holding at the moment it stops. That is the memory on the GPU, the process on the CPU with its threads and open files, the network connections, and what the scheduler knows about the job. The five approaches below differ in how much state they capture, and anything outside an approach's capture boundary is gone when the job stops.

Approach	What it captures	What it leaves	Who maintains it
Checkpoints written into the application	The values the program was told to save	Anything the developer did not list, and the state around it	The team that wrote the program
Process-level checkpoint and restore	The Linux side of the program, its memory, threads, open files and sockets	Device state, which sits outside the normal process address space	The CRIU project, plus whoever wires it in
A checkpoint primitive in the GPU driver	The device state of one process, after submitted work finishes and its memory reaches the host	The CPU threads, which keep running, and coordination across a job's processes	NVIDIA, plus whoever calls it
A serverless snapshot of an initialized worker	A worker that finished initializing, before any request arrives	The state of requests in flight	The serving platform
Scheduling and utilization tooling	Nothing of the running job, because it decides where work starts and whether it stops	The running state, so a preempted job is cancelled, requeued or suspended	The scheduler

So match the approach to the state your workload is holding when it stops. A checkpoint written into the program saves the state the program selects. A capture that stops at the process boundary misses state that sits on the GPU, while a device-level capture gets the GPU and leaves the CPU side running. A snapshot taken before the requests arrived does not hold them, and a placement decision does not capture a running job's state at all.

Application-level checkpointing is the most efficient technique and the most work

Application-level checkpointing costs the least to run and the most to build. The run-time cost is low because only the data a developer named by hand gets written, and the build cost is that naming, done once for every program that has to survive an interruption. The authors of CRAFT, a library built to make application-level checkpointing easier to write, put the trade-off in one line of their abstract, submitted on 7 August 2017. "Application-level CR is the most effective CR technique in terms of overhead efficiency but it takes a lot of implementation effort." CR there is checkpoint and restart.

The libraries that make application-level checkpointing easier are built for the storage problem rather than the GPU. The repository for SCR, the Scalable Checkpoint/Restart library, said in September 2026 that it lets MPI applications, meaning programs that split work across machines with the message-passing interface, use distributed storage on Linux clusters to reach high file input and output bandwidth for checkpointing, restarting and output in large-scale jobs. VeloC describes itself as a multi-level checkpoint and restart runtime that delivers high performance and scalability for complex heterogeneous storage hierarchies. Neither page states GPU support one way or the other.

For training, PyTorch's distributed checkpoint is the standard answer. Its documentation says that the library saves and loads a model from several ranks in parallel, and that it reshards at load time, so a checkpoint saved on one cluster topology loads into another. A checkpoint written on one GPU count therefore loads on a different one, and that is the case system-level capture handles least well.

All three are libraries that the application calls at points chosen in its code, and they save the state the application hands them. Loaded weights outside the checkpoint call, open network connections, and warmed-up CUDA graphs are not in the file.

Whether what the file leaves out matters is a question about your own workload. A Local LLM hobbyist on r/LocalLLaMA treated model snapshotting as a solved matter of saving and loading state and dumping it to disk, and asked whether everyone was not already doing that. A platform engineer on Hacker News named the three things that settle it for a given workload: how large the heap is, how long a blackout period the application can handle, and whether it survives having all of its open connections destroyed.

Application-level checkpointing forces each team to maintain its own checkpointing code, because only that team knows which state its own code holds. That cost never appears in a build estimate. Every group that runs a job absorbs it.

The open-source floor is CRIU plus a driver primitive

Two projects wired together are the floor anyone building system-level GPU checkpointing starts from. CRIU, short for checkpoint and restore in userspace, describes itself on criu.org as able to freeze a running container or an individual application, checkpoint its state to disk, and bring it back exactly as it was during the time of the freeze. The GPU sits outside what CRIU handles by itself, so its CUDA plugin calls NVIDIA's cuda-checkpoint, a utility written to transparently checkpoint and restore CUDA state within a running Linux process.

Both projects publish their limits, and the limits are narrow. CRIU's CUDA plugin states that a restore currently requires a system with similar GPUs and the same GPU count. The cuda-checkpoint README says that the utility acts upon a single process rather than a process tree, that it does not support GPU migration, and that it supports display driver version 550 and higher.

CRIUgpu, a paper submitted on 23 February 2025, builds on the same driver capability. Its abstract reports that the approach works with a variety of deep learning and high-performance computing workloads running across multiple GPUs, eliminates steady-state performance overhead completely, and cuts recovery times well below the transparent GPU checkpointing mechanisms that came before it.

A working mechanism is still not a tool your team can adopt. A platform engineer on Hacker News wrote that CRIU turned out not to be an option for his team back then, one problem being that it could not be used as anything other than root, so they wrote their own fork server instead of using CRIU or DMTCP. A platform engineer on r/kubernetes who was checking the feasibility of building this in house was told that CRIU is one piece, and that networking, IP addresses and volumes still have to be solved.

Nine components stand between a working checkpoint and a production system

A good team can build the first working checkpoint. Take one process on one GPU, running one serving engine, started by one scheduler through one container runtime, save the GPU memory and the CUDA state, save the process and the files around it, and bring it back.

The estimate that comes out of that demonstration covers the demonstration machine, not the range of configurations in your cluster. A production system needs the following nine components.

The capture on the GPU, for every process of a job, against every driver version in the fleet.
The file system, network, container and scheduler state, restored consistently with the GPU state.
Every GPU of one job stopped at the same boundary, with the communication layer rebuilt on restore.
Workloads that span more than one node, holding that boundary across nodes.
Every container runtime and scheduler the cluster runs, and how each one starts and stops a job.
Every serving engine users bring, and training, fine-tuning, batch and interactive work.
Storage for thousands of large checkpoints, and a route to the machine where the work resumes.
The layer that senses the event, applies the policy and restores the work.
Re-validation of every item above.

Re-validation is the item that continues after the first version ships, because a driver update or a new serving engine release sends the whole matrix back for testing, and each new GPU generation requires another pass. Most of that work is small and specific, which is why it stays out of estimates.

We went through the same work at Cedana to get from a first version to full coverage. A large enterprise account handed us the workloads running on a production cluster and asked us to cover them. Most of that list did not work at first, and closing the gap took us most of a year.

What broke the product was ordinary use. The cluster's users were putting all sorts of things into their YAML job files and doing non-standard things with them. By the end of that year we supported single-node, multi-GPU, three major inference engines (vLLM, SGLang, Triton), and a wide registry of CUDA test cases. What a build like this produces is that list of covered cases.

Should you build GPU checkpointing yourself, or buy it?

Build when your scope is fixed, meaning one scheduler, one container runtime, one serving engine, and one workload type, on hardware and drivers you control. A team with that fixed scope has to implement the capture described in the first item above, and it also owns storage, re-validation, and the decision to act. If you have already built and deployed a system that covers your own workload mix, you do not need to buy at all.

Buy when your cluster is mixed rather than uniform, when a job spans several GPUs, or when something other than a person has to trigger the checkpoint.

MemVerge sells the system-level answer with published limits

MemVerge is the one commercial product making the same claim we make about GPU checkpointing for arbitrary workloads on a cluster. The serverless snapshot products in the table above are scoped to an initialized worker, so they do not make that claim. Its Kubernetes Transparent Checkpoint Operator lists "GPU-Aware Checkpointing (Optional)" and "Optional GPU Operator Integration: Enables checkpointing of GPU-enabled workloads using Kubernetes CDI" among its features, and its spot FAQ answers the GPU question with a condition attached.

"Is checkpoint and restore supported on GPU-enabled instances? Yes, as long as the GPU-enabled instances are supported by MMCloud."

Two published limits bound what MemVerge's GPU support covers. The snapshot taken in the reclaim window, the short notice before a spot instance is taken back, has a practical ceiling of about 64 GB, and none of the pages reviewed describes multi-node distributed training.

What a restore costs

A restore costs the restore time plus the work done since the last checkpoint. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. Those bytes have to reach the destination and be read into GPU memory before execution continues.

For Cedana, the destination has to be a compatible node. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

The other limit is how many nodes run the workload. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds.

Cedana does not decide where a job goes. No scheduler or runtime calls Cedana on its own today, so an operator or an agent starts a move under a policy written once rather than job by job. Cedana's steady-state overhead is under 2%, often closer to zero. What a GPU checkpointing layer costs while the workload runs gives the conditions behind that figure.

What a team buys is a maintained list of what a checkpoint covers, not the first working checkpoint. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We capture below the application, bring the file system, network, container and scheduler state along with the GPU state, and re-validate that coverage against each new driver, engine release and GPU generation. So the question in front of you is not whether a checkpoint can be made to work on your cluster, but which of these approaches you are willing to keep working as your drivers, your engines and your GPUs change.

Common questions

Is it enough to have my application write its own checkpoints, or do I need a system-level, transparent tool?

Checkpoints written into the application save the values the developer told it to save, such as model weights and optimizer state, and miss the rest of what is on the GPU, the open network connections, and the process around it. A system-level, transparent checkpoint and restore tool captures that surrounding state without changing the application. Which one you need depends on how large a blackout period your workload can tolerate and whether it can handle its open connections being destroyed.

Should you build GPU checkpointing yourself, or buy it?

Build when your scope is fixed: one scheduler, one container runtime, one serving engine and one workload type, on hardware and drivers you control. Buy when your cluster is mixed rather than uniform, when a job spans several GPUs, or when something other than a person has to trigger the checkpoint. A production system also needs nine components beyond the first working checkpoint, including re-validation every time a driver or engine updates, and that work beyond the first checkpoint rarely appears in a build estimate.

What does it mean that cuda-checkpoint does not support GPU migration?

It means a checkpoint cannot be restored on a different kind of GPU, or on a different layout of GPUs, from the one it was taken on. The README states that the utility does not support GPU migration and acts upon a single process, not a process tree, on display driver version 550 and higher, and CRIU's CUDA plugin states the same condition from its side: restore requires a system with similar GPUs and the same GPU count. A move between two machines with identical layouts is allowed, and a move to a different chip type or count is the one that is excluded.

Which checkpointing approach brings back the state your job is holding?