What is inside a GPU checkpoint?

Everything around the weights is state the running worker holds

When a serving worker is ready to take requests, the model weights are already sitting in GPU memory, read once from a file on storage. Next to them is the key-value cache, or KV cache, which holds the attention keys and values the model has already computed for every token in every session in flight, so the next token does not make the model compute them again.

The CUDA context is the driver's record of the process on the GPU, including its memory allocations and streams, and a CUDA graph is a sequence of GPU operations captured once and replayed as a unit. The serving engine captures these graphs during startup, so the worker built them rather than loaded them.

If the job spans several GPUs, it also holds a set of communicators from NCCL, the NVIDIA Collective Communications Library, which are the channels those GPUs opened to one another.

The engine's own process runs on the CPU, and it is where the record of every session lives. It holds its heap, the scheduler with its queues of waiting and running requests, and the block tables that map each session to its cache pages, along with buffers for partly written responses, open files, and sockets to clients.

Resuming the workload cleanly takes the file system, network, container and scheduler state as well, because the GPU context on its own is not enough.

Cedana's rough estimates for a 70B model serving at 128K context are in the last column, and they are orders of magnitude rather than measurements.

The KV cache row carries no single figure, because its size depends on the cache precision and on how many tokens are in flight, and the next section works one case through. The scheduler, block-table and buffer rows are parts of the engine's process memory listed on their own, so they sit inside the CPU RAM row rather than adding to it. Everything else on the list except the weights and the engine's host memory is under a gigabyte.

By bytes, a checkpoint is mostly weights

A checkpoint runs roughly to the size of the worker's memory, and model weights make up most of that memory, with their share growing as the model grows. A serving worker for a frontier open-weight model holds half a terabyte or more of weights, sharded across a whole node, before the first byte of session state arrives. Kimi K2.6 runs at roughly 600 GB even in natively quantized INT4.

The KV cache is set by the model's attention design, the cache precision, and the number of tokens in flight. For GLM-5.2 at FP8, a 100K-token context works out to roughly 4.5 GB of KV cache based on the model's own configuration file, and that figure belongs to that model at that precision and does not carry over to another one.

Context length changes the balance between the weights and the cache. At long context, a single request's KV cache grows large enough to rival the model's own weights in VRAM, which is what the AnTKV paper reported in 2025.

Measured whole checkpoints run to hundreds of gigabytes. In our published benchmark we ran frontier models on a single node of 8 B200 GPUs, and their checkpoints came out at 244 to 873 GiB. Each checkpoint restored in 57 to 70 seconds.

Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. Across those four checkpoints the size grew 3.6 times and the restore time 1.2 times, so on that node the restore barely moved, and four runs cannot show more than that on their own.

The clock on those measurements started at the beginning of the restore and stopped when the same fully initialized engine was ready to serve. We served each model with SGLang, following the model's official recipe, and those runs read from tmpfs, which is memory rather than disk. So a restore in your own fleet will also depend on where the checkpoint sits and on the network in between.

A restart rebuilds most of the inventory on its own

Most of the state on a failed node has a source somewhere else, so a restart goes and gets it. Reloading a frontier model's weights takes minutes, and the runtime state around the weights has to be built a second time before the worker can take a request.

The scheduler starts over instead of coming back, because a new instance opens with empty queues that refill only as clients reconnect and retry, so a job loses its place in the line. Every part of the inventory works one of these two ways: it either comes back from somewhere, at the cost of time, or it does not come back at all, at the cost of the work itself.

The KV cache and the step in progress have no copy anywhere

Every row in that table costs time to rebuild except the last two, the KV cache and the step in progress, and those two cost the work itself, because nothing outside the GPU held a copy of either one.

On a cache miss the session is prefilled again from its transcript, which repeats the same computation the worker already did once. The step that was running when the worker stopped is not rebuilt at all but regenerated, and the regenerated step may diverge from the computation that was lost.

FailSafe, a paper by Xu, Xie, Gandhi, and Kozyrakis, draws the same line: static model weights can be reloaded from persistent storage such as host memory or disk at moderate cost, while the KV cache is the dynamic per-request state that has to be recomputed from scratch. In their own serving experiments, recomputing the lost cache alone took over 20 seconds.

The weights and the runtime state both have somewhere to come from. The computation itself is the only part of the stack with no saved copy. So reloading a model file gives you the worker back, but not the work it had in progress.

A system-level checkpoint holds the KV cache as it stood at capture

The KV cache is GPU memory, so a checkpoint taken at the system level captures it, along with the CUDA context, the process memory, the files, and the network state. What comes back is the cache as it stood at the moment the checkpoint was taken. We take checkpoints with Cedana on a heartbeat while the worker runs, so a failure costs the work done since the last one.

A KV offload tier does a different job: it can reload the prefixes it already wrote out, once a new worker is running to read them, and it holds none of the in-flight state of the worker. So what the tier never wrote out, including the step that was running, is recomputed from scratch.

A checkpoint stops at the edge of the machine and at the next version change

Anything that has already left the machine is outside the checkpoint. A tool call the agent made, or a database row it wrote, now lives in another system, so a checkpoint captures GPU state rather than the state of the outside world.

A client connection is captured as socket state, so the connection comes back with the worker, and whether the client on the other end is still waiting depends on that client.

The conversation history is captured too, as bytes of process memory, but it is not organized for reading, so the transcript remains the readable record of a session.

A checkpoint also stops at a version boundary. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

Getting the work back means checkpointing the worker itself

An agent 30 turns into a session loses none of its record when the node under it fails, as long as the transcript, the workspace, and the sandbox are kept in another system, and agent platforms usually keep them there rather than only in the worker's process memory. What went with the GPU is the cache that encoded those 30 turns and the turn that was running, and that is the expensive part.

A restart reloads the weights and rebuilds the runtime state, then sends the transcript of those 30 turns back through prefill and generates the interrupted turn a second time. A resume instead needs a checkpoint of the worker itself, taken below the application, with the KV cache in it. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. Here that means we save every component in the inventory above, including the KV cache and the work in flight, and bring it back on compatible hardware. So after a failure, your cost is the restore plus whatever the worker did since the last checkpoint.

Common questions

What is inside a GPU checkpoint, and how big is it?

A GPU checkpoint runs roughly to the size of the worker's memory, and model weights make up most of that memory, with their share growing as the model grows, so measured whole checkpoints run to hundreds of gigabytes. It also holds the key-value (KV) cache, the CUDA context and compiled graphs, the NCCL communicator state, and the CPU process's own memory, files, and sockets.