KV-cache offload vs a full worker checkpoint

TL;DR

A cache offload tier in vLLM or SGLang copies completed key-value cache blocks out of GPU memory, so a prefix you have already computed does not have to be computed again. You add it for latency, and it often gets counted as the answer to a node failure as well.
After the worker dies, those blocks survive and nothing on the new node can read them yet. The weights, the CUDA context, the compiled graphs, the engine's queues and the turn in flight all went with the process.
GPU checkpointing saves the worker itself, so what comes back is a process that was already running, with its own cache as of the last checkpoint. The two mechanisms are cut along different lines, and a fleet running long sessions on hardware that fails usually wants both.
In this piece we walk through what an offload tier holds, what dies with the worker, and why a saved block is unreadable until a new worker exists. Then we cover what a checkpoint holds instead, why prefix sharing belongs to the cache layer alone, and which mechanism answers which failure.

An offload tier keeps a computed prefix so the next request does not pay for it again

A serving engine keeps the keys and values it has computed for every token it has already seen. That means it does not have to run attention over the whole context again for each new token. The store is the key-value cache, usually written KV cache, and it sits in the GPU's own memory.

A prefix is the run of tokens at the start of a request, and two requests that open with the same system prompt share one. Automatic prefix caching in vLLM lets the second request reuse the first one's cache for the shared part, which works because a token's cache entry depends on nothing that comes after it.

The saving lands in prefill, the pass where the model reads the input tokens and builds their attention state, and nowhere else. Decode is what follows, writing the output tokens one at a time, and it runs at the same speed whether the prefix was cached or not. The vLLM documentation puts the bound in one sentence: automatic prefix caching "only reduces the time of processing the queries (the prefilling phase) and does not reduce the time of generating new tokens (the decoding phase)."

An offload tier is what lets the cache outgrow the GPU. The engine allocates the cache in fixed-size blocks, and the offloading connector in vLLM writes completed blocks out to slower and larger tiers as they are produced, host memory first and optional secondary tiers after that. A hit in one of those tiers is promoted back to the GPU on demand. SGLang builds the same idea as three levels, with GPU memory as L1, host memory as L2, and distributed storage as L3.

The gain gets large when many of your requests share a long prefix. The vLLM project measured a shared cache pool on a set of Codex traces, where the hit rate went from 1.7%, with only the system prompt cached, to 92.2%, with nearly the whole prefix cached. Throughput rose 3.8x on 12 GB200 GPUs.

When the worker dies, the tier keeps its blocks and nothing that can read them

When the engine process dies, the blocks it had already written out to host memory or storage do not die with it. LMCache's documentation makes that its headline claim, that its "KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines)". Which of those blocks a replacement worker can read depends on the tier they reached, and a tier local to the node dies with the node.

Everything else the worker held goes with the process. The weights have to be read off storage again on the next start. The failure also destroys the CUDA context and the compiled graphs, so they are built from scratch. The engine's scheduler state goes too, so the new instance comes up with an empty queue and an empty block table.

The turn the worker was in the middle of is not in the tier either, because the offload path is written from completed blocks as they are produced. Tokens that had not finished a block when the GPU failed were never written out anywhere.

Losing the turn in flight is a live research problem. GhostServe, a fault-tolerant serving system published at MLSys 2026, builds parity shards in host memory so that a lost cache can be reconstructed quickly rather than computed again in full.

A saved block is unreadable until a new worker exists

A KV block only means something inside a running engine that has the model's weights loaded and its CUDA context built. Until such an engine exists on the destination node, the blocks sit there and nothing can read them. So unless a warm worker of the same model is already running somewhere, the destination first goes through a full cold start, the same start it would need with no tier at all, and reads the cache afterwards.

A second instance cannot borrow the first one's host memory either, because that tier belongs to one inference instance's process. SGLang's HiCache documentation states the rule plainly, that "L2 is node-local and instance-private." The blocks one instance writes become visible to another only once they reach the distributed level.

A checkpoint saves the worker, so nothing has to be rebuilt

A checkpoint cuts the other way and saves the worker rather than the blocks the worker produced, so at Cedana we capture the process with the device memory inside it. What arrives on the new instance is a worker that was already running, with its weights in place, its graphs compiled, and its own KV cache, and none of that has to be rebuilt.

What comes back is the state as of the last checkpoint, so tokens generated between that checkpoint and the failure are not in it. A restore takes time to copy the saved state back.

The tier holds the completed blocks and the prefix reuse, and a checkpoint of the worker holds everything else.

Piece of state	An offload tier, after the worker dies	A checkpoint of the worker
Model weights in VRAM	Not held. The new instance reloads them from storage on every restart.	Held. The saved process image carries the weights in place.
CUDA context and compiled graphs	Not held. Destroyed on failure, recompiled from scratch on restart.	Held, and restored with the process.
KV cache of completed blocks	Held, and unusable until a new worker is built to read them back.	Held, as of the last checkpoint.
The turn in progress	Not held. The tier is written from completed blocks.	Held as of the last checkpoint. Tokens generated after it are gone.
The engine's queues and block tables	Not held. The new instance starts empty.	Held, with the rest of the process memory.
Network connections	Not held. The connections drop when the process dies.	Held. Network state is captured with the process.
Prefix reuse by another worker	Held. Shared blocks across requests with a common prefix are the tier's own capability.	Not held. A checkpoint is taken per process rather than per token sequence, so its cache serves that one worker.

Both mechanisms cost something while the worker runs, and the tier's side of that cost is documented. The offloading transfers in vLLM run asynchronously alongside the model's own computation, so the cost its documentation describes is processor overhead rather than a stall. What our own layer costs while a worker runs is in What a GPU checkpointing layer costs while the workload runs.

Only the cache layer shares a prefix between workers

What a restore hands back is one worker's own cache, not a pool the rest of the fleet can draw on. A later session that lands on that same worker can still hit the prefixes in that worker's cache, because that is what the engine's own prefix caching does. What the restore does not do is put the prefix anywhere another worker can read.

A replica you add beside a restored worker meets the same limit. It comes up from a checkpoint of an initialized worker, so it gets warm weights and warm runtime state, which is what makes an instance cheap to produce. What it does not get is a warm cache for the traffic it is about to take, because the cache in that checkpoint belongs to the worker that was captured.

Reuse across the fleet is where most of the cache layer's measured value comes from, because one prefill gets spread across every later session that shares the prefix, wherever that session is served. A checkpoint handles failure inside a session, and it amortizes nothing beyond the worker it brings back.

Which one you need depends on the failure you need to handle

The two mechanisms answer different questions, so if you run long sessions on hardware that fails you usually want both. A cache tier lowers the cost of a prefill you would otherwise repeat across requests, and checkpointing cuts what an interruption costs when it takes the worker away. Neither one covers the other's case.

The case	The answer
A short request fails	Retry it. A retry costs seconds, the application layer already handles it, and the cost of checkpointing can exceed what it saves.
Many sessions share a prefix across the fleet	The cache tier, and only the cache tier. Prefix caching amortizes the prefill across thousands of sessions on many workers, which a checkpoint does not do.
A long session on a node that fails	A checkpoint of the worker. The session's cache has to come back attached to a running worker, and the tier holds neither the worker nor the turn in flight.
A scale-up adds a replica	A restore. The cache tier cannot help here, because the weights reload and the CUDA graphs recompile whatever the cache holds. A restore brings back initialized weights and compiled graphs, though not a warm cache for the new traffic.
A drain moves work between nodes	Both do part of it. A tier with a shared level can transfer its cache to the destination, and the destination still needs a cold start before it can use the transferred blocks. Moving the running process is what removes that start.

How far that last case reaches depends on how many nodes the job itself spans, and a job spread across many nodes is not one you can drain this way. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

So the two saves are not alternatives, and the half the tier leaves out is the worker that has to exist before its blocks are worth anything. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a serving fleet, that means the worker itself comes back, with its own cache as of the last checkpoint. The node taking over your long session does not have to build a worker from nothing before it can use that cache.

Common questions

When a worker dies and a new one has to start on the destination node, how long does that cold start take?

On Cedana's benchmark node of 8 B200 GPUs, native starts of frontier models ran 9.4 to 34.2 minutes. A cache tier does not shorten that, because the destination pays a full cold start before it can read a single block, the same start it would have paid with no cache tier at all. The weights have to be read off storage again, the CUDA context and compiled graphs are destroyed and rebuilt from scratch, and the engine's scheduler state comes up with an empty queue and an empty block table.

KV-cache offload versus a saved worker: what each survives