What cannot be checkpointed in a GPU workload?

Almost everything the managed process holds is inside the checkpoint

Most of what a running job holds sits on the machine it is running on, and a checkpoint of a managed process takes all of it. Your open network connections survive a failure, because the kernel's socket queues are checkpointed along with the process. Files come back as they were, because the container's read and write layers are captured, so file contents and the process's open-file state come back together. The process resumes from a snapshot of its memory, so an agent's conversation history comes back too, sitting in that memory as bytes rather than as text the checkpointing system can read.

On the GPU side, the checkpoint holds the key-value (KV) cache, which stores the attention state of every request in flight, and the CUDA context with its compiled graphs, streams, and contexts. It also holds the NCCL (NVIDIA Collective Communications Library) communicator state between the GPUs in one job. A restore gives that state back to the job instead of making it rebuild it.

The record of a session was already durable before GPU checkpointing arrived. Transcripts, workspace checkpoints, and sandbox snapshots of full process memory from Daytona, E2B, and Google's Agent Sandbox form an industry layer built to preserve it. The computation itself is the only part of the stack with no saved copy. A checkpoint adds that computation to what is saved, but only for one machine.

What the job already sent somewhere else is not in the checkpoint

A checkpoint captures GPU state, not the state of the outside world. A result a tool call already returned is inside the checkpoint, because it sits in the captured process memory with the rest of the job's data. What is not saved is what those calls did elsewhere: the API call that went out, the database row that was written, and any other action that reached past the machine. Once an effect reaches another system, it lives in that system's memory, and that memory does not roll back when your process does.

A restore picks up from the saved moment, so whatever the job did between that moment and the failure happens a second time. With Cedana, we take a checkpoint, KV cache included, at a heartbeat interval, meaning a fixed checkpoint interval, and the most a restore can lose is that interval multiplied by the decode rate. The stretch of work it loses is the same stretch it repeats.

So a call that is not safe to repeat can be repeated: an email may go out twice, or a row be written twice, and nothing at the checkpoint layer prevents it.

A serving platform, an enterprise coding agent and a long research job each reach past the machine in a different place.

• On a serving platform, it is an agent session whose tool calls reach into a customer's own systems.
• In an enterprise running its own coding models, a coding agent 30 turns into a session has already opened a pull request or run a command against a system outside the box.
• A research job may have written its results to a shared database an hour before the node died.

Some jobs cannot be checkpointed at all

The CPU side of what we capture goes through CRIU, the Linux checkpointing project, and CRIU publishes a page called "What cannot be checkpointed". Three entries on that page decide whether a GPU job is a candidate at all: a raw device, a debugger, and a file that belongs to something outside what is being dumped.

A process that has opened or mapped a character or block device wants a connection to some piece of hardware, and CRIU cannot dump it. CRIU's own answer for why is short: "we don't know how all the devices work". The GPU is the exception. GPU state such as device memory, contexts, and queues sits outside the normal process address space, so CRIU hands it to vendor-specific plugins. That is why GPU checkpointing became a field of its own, and it is also why a job talking to a capture card or a custom accelerator is out of scope.

A task under gdb or strace cannot be dumped, because the checkpointer needs the same kernel interface the debugger is holding and the kernel allows one holder per task. So while someone is tracing a job, that job cannot be checkpointed, whatever your eligibility rules say.

A file that belongs to something outside what is being dumped is what CRIU calls an external resource, meaning one whose state sits partly outside the container being saved. Restoring it takes help from whoever calls CRIU. A process holding a file open on a lazily unmounted filesystem is an everyday example, and closing the file makes the process checkpointable again.

The rest are limits of today's tooling, not of the approach

A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. Our reading is that a checkpoint holds driver and engine state as it was laid out in memory, and only the same versions can restore that layout. So read the version compatibility matrix before a move, and in practice keep the CUDA and operating system versions consistent across checkpoints and pin them on whatever machine you restore onto.

Our GPU checkpoint and restore works on NVIDIA GPUs today and on no others. It also works only on processes and containers we manage, meaning those started with cedana run --gpu-enabled or adopted with cedana manage --gpu-enabled, so a job launched any other way is not a candidate. That requirement is also what makes checkpointing opt-in for each workload, rather than something applied to everything on the node. Third-party job servers that run their own job-management layer, such as systems that health-check their own sub-jobs, are untested, and we do not present them as covered.

The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

A restore returns the job to the saved moment while outside clocks keep running

A restore continues the execution that was underway when the checkpoint was taken, while a restart from scratch begins a new one. The guarantee applies to the saved state, not to the present. So a job that was reading a live feed comes back at a moment the feed has already moved past, and time-dependent workloads ingesting real-time data can be affected.

Anything on the other end of a connection keeps its own clock too. A peer's timeout or a lease may have run out while your checkpoint sat on storage, and the socket comes back either way, so whether the peer is still waiting is up to the peer.

The application owns everything on the far side of the line

A call is idempotent when making it twice leaves the world in the same place as making it once. Idempotency keys and deduplication are how your application gets there, and compensating transactions undo the effects that cannot be made safe to repeat. All of that lives in the application, because the application is the only layer that knows which of its own calls are safe to make twice. So a system-level checkpoint and application-level idempotency work together rather than as alternatives.

Four rules put that ownership into practice, and all four start from the same assumption, that the stretch since the last checkpoint will run twice.

1. Put an idempotency key on every call that leaves the machine, and record the intent before acting, so the receiving side recognizes a replayed call for what it is.
2. Treat the stretch since the last checkpoint as work that will run again, and design the outside-world calls in it to be safe to repeat.
3. Do not checkpoint short, cheap requests. Checkpointing one can cost more than it saves, and an application-layer retry handles the failure. If the work lost is seconds and the job touched nothing outside the machine, retrying is the right call.
4. Keep the workloads that should not be interrupted this way off the list entirely. Checkpoint eligibility is set by partition, and a job whose checkpoint would exceed a size threshold is skipped. So live-data workloads, license-bound software, and job servers with their own management layer can sit on their own partition.

Any system-level checkpoint is limited to the process and resources it captures on one machine, and that machine boundary is what we build Cedana around. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. In practice, that means we capture what the managed process and its container hold on the machine side of the line, at the operating-system level, below the serving engine, with no application code changes, so an interruption costs the job the stretch since its last checkpoint instead of the whole run. Your application still has to handle the effects of the calls that left the machine during that stretch, because the replay repeats them.

Common questions

What cannot be checkpointed, and why is GPU checkpointing hard?

A checkpoint does not save anything the job has already sent to another system, and it does not save anything the job does after the checkpoint was taken. Some jobs cannot be checkpointed at all: a process holding a raw device other than the GPU, a task under a debugger (the checkpointer needs the same kernel interface the debugger is holding, and the kernel allows one holder per task), or a file that belongs to something outside what is being dumped. The GPU is hard for the same reason a raw device is: GPU state such as device memory, contexts, and queues sits outside the normal process address space, so it needs vendor-specific plugins rather than an ordinary process dump.

How often should a job be checkpointed, and what does the interval cost?

How often is a policy setting, and you set it from how much work you are willing to repeat: a heartbeat interval for serving and agent work, and a step or epoch boundary for training. With Cedana, we take a checkpoint, KV cache included, at a heartbeat interval, and the most a restore can lose is that interval multiplied by the decode rate. The stretch of work it loses is the same stretch it repeats.