Moving workers in disaggregated LLM serving

TL;DR

NVIDIA Dynamo splits large language model inference across two pools of GPUs, one that processes the prompt and one that generates the answer.
The objection to worker migration is that once the phases are separate, nothing needs to move a running worker any more. That falls down because the handoff leaves every session's state on the decode worker. A decode worker that goes out of service takes its sessions with it.
Dynamo works around the fixed worker by moving requests instead. Every correction the planner makes to the pool split is executed as a worker start, which is a cold start whose length follows the model.
What changes when the worker itself can move is that a decode worker comes back ready to serve from a checkpoint. Generation picks up at the next token instead of prefilling the context again.
In this piece we walk through why disaggregated serving splits the phases, where a session's state sits after the handoff, and why Dynamo moves requests rather than workers. Then we cover what changing the split costs, and what moving the worker changes.

Dynamo splits inference because the two phases want different hardware

NVIDIA Dynamo is an open source serving framework that runs above the inference engines. It does not replace SGLang, TensorRT-LLM, or vLLM. It coordinates them across many GPUs and nodes, and disaggregated serving is the first capability on its list.

Dynamo separates the phases because prefill and decode have different computation characteristics and memory footprints. Prefill runs every token of the prompt in one pass, so a longer prompt costs it more compute. Decode produces the answer one token at a time, so it is limited by memory bandwidth and by the room to hold the key-value (KV) cache of every session it is serving.

A disaggregated request runs in three steps: a prefill engine computes the prompt and produces its KV cache, that cache moves straight from the prefill engine's GPU memory into the decode engine's, and the decode engine generates the tokens. A router in front of the two pools picks a worker by load and by how much of the request's cache that worker already holds, and a planner sizes the pools against latency targets.

The industry splits the phases because the arrangement wins on both throughput and cost. Splitwise measured clusters built this way reaching up to 1.4x higher throughput at 20% lower cost. The design is sound, and it also settles where the state of a session lives.

After the handoff, the session lives on the decode worker

Once the KV cache has crossed into the decode engine, the accumulated work of the session sits on the decode side and stays there. The prefill worker computed one request's cache and let it go. Every turn after that adds to the cache the decode worker is holding, so the two kinds of worker end up in very different positions.

	Prefill worker	Decode worker
Bound by	Compute. It runs every prompt token in one pass	Memory bandwidth and capacity. It generates one token at a time across many sessions
State it holds	One request's KV cache, until the transfer completes	The KV cache of every session in progress, and the generation state in flight
How long that state lives	Seconds, ending at the handoff	As long as the session runs, growing every turn
Cost of replacing it	A model load. No session needs what it held	The sessions themselves, whose context is recomputed wherever they land next

How much a decode worker holds depends on the traffic, and agentic traffic is the heavy case. A published trace study of a coding agent found a median of 33 turns per session, and the cache grows with every turn. So in a long-context agentic session, losing the decode worker loses all of the cache it was holding.

The state is large as well as long-lived. The same study puts a 100K-token context at roughly 3.8 GB of cache for Kimi-2.5 at 8-bit precision, and it says what moving a session costs: the instance it moves to has never seen the prefix, so it must recompute it from scratch. A shared cache pool can make a prefix reachable from another instance, but it does not move the computation that is in progress.

The requests move because the workers cannot

Moving a worker is worth something only on the side that holds the state. Nothing a prefill worker holds outlives the handoff, so replacing one costs a model load and no session notices. A decode worker taken out of service takes its sessions with it, and decode is the pool you have to resize and repair as traffic shifts and hardware fails.

Dynamo is designed around workers that do not move. Its router tracks which worker holds which prefix across the cluster, so a session's next turn goes back to the worker that already has its context. Its request migration handles a worker failing mid-request by carrying that request's tokens to a new worker, which rebuilds the context by prefilling it again, and the client sees an unbroken stream. Both mechanisms work around the same limit, which is that the worker cannot move.

Changing the split starts a worker, and on the decode side it discards session state

The ratio between the two pools is not fixed. Dynamo's documentation says workers can be added and removed at runtime, and the planner can use runtime load and capacity signals to redistribute the pools.

Removing a worker and adding one cost very different things. A removed worker drains its active requests and deregisters from discovery, while adding one means scheduling a pod and loading the model into GPU memory before the new worker can register and take traffic. So the planner's decision interval has to be longer than the worker's startup time, and the planner's documentation says how long that start takes: "Spinning up a GPU worker takes minutes, not seconds."

So on a fixed pool, every correction the planner makes to the split is executed as a worker start, which means a cold start whose length follows the model. Cedana's published benchmark set ran on a single node of 8 NVIDIA B200 GPUs, serving frontier open-weight models. These are the latest open-weight models trailing the frontier labs (DeepSeek, Kimi, Qwen, GLM). Native starts of those models took 9.4 to 34.2 minutes, from engine launch to ready to serve.

On the decode side, the correction also discards the cached state of the sessions the removed worker held. Active requests finish during the drain, but the sessions themselves continue, and each one's accumulated context is recomputed wherever its next turn lands.

None of this is a flaw in Dynamo. It is what a disaggregated fleet costs while the worker itself cannot move. The split matches each phase to the hardware that suits it on the day you deploy, and the match gets worse as the fleet and its traffic change. The only correction available today is teardown followed by a replacement's model load and initialization before it can take traffic, at a cost that grows with model size.

Dynamo's own list of reasons to use the framework includes fast cold starts when new replicas are spun up, and it ships a component that streams model weights GPU to GPU so that new replicas start faster. A fleet that cannot move its workers can only change by starting new ones, so a faster start is where the effort goes.

Moving the worker leaves the sessions alive

Worker migration means saving the whole state of a running worker and bringing it back on other hardware. That state is the model weights in GPU memory, the CUDA context, the KV cache, the sessions in flight, and the file system, network, and scheduler state around the process. Cedana captures it below the serving engine, so the capture works the same way whether the worker is running SGLang, vLLM, or TensorRT-LLM, and the worker comes back on the engine and version it was captured with.

Cedana integrates with Dynamo today. In that integration, a new decode worker returns ready to serve from a checkpoint instead of loading the model, so generation picks up at the very next token rather than prefilling the context again. A preemption notice becomes a move because we watch for it and shift the active inference tasks onto healthy nodes before the node is taken down. When the traffic mix moves, the hardware under a lightly loaded decode worker can come back into service as prefill. A worker whose best place has changed can migrate and resume there mid-request, on a GPU of the same model.

Restores of those same models on the 8 B200 node took 57 to 70 seconds. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. The published runs read their checkpoints from tmpfs, a file system that lives in the node's own memory, so those times do not include a fetch from disk or across a network.

The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

Dynamo's planner still decides the split, and its router still decides where requests land. A checkpoint on its own decides nothing.

What migration adds is a way to execute those decisions without destroying decode-side state, and Cedana's control plane, under the policy you set, decides when to save a worker and where to bring it back. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. In a disaggregated deployment, that means a decode worker can move off a node instead of being drained, and the sessions it holds resume where it lands, from its last checkpoint. So separating the phases does not remove the need to move a worker. It concentrates that need on the decode pool, which is where your sessions live.

Common questions

Does adding or removing a worker in a disaggregated Dynamo deployment mean a cold start?

Every correction the planner makes to the pool split is executed as a worker start, so yes. A removed worker drains its active requests and deregisters, but adding one means scheduling a pod and loading the model into GPU memory before the new worker can take traffic, and that start is a cold start whose length follows the model. Cedana's published benchmark set, on a single node of 8 NVIDIA B200 GPUs serving frontier open-weight models, measured native starts of those models at 9.4 to 34.2 minutes, from engine launch to ready to serve.