TL;DR
- One model in production is really two workloads: prefill, which reads your prompt in one pass, and decode, which writes the answer a token at a time. Prefill wants arithmetic, and decode wants memory bandwidth and memory capacity.
- Running both phases on one GPU underfeeds one of them at most operating points. Interleaving them in time only changes which phase is starved in a given slice.
- Splitting the phases onto separate pools, which is what disaggregation means, fixes that mismatch. It costs a KV cache transfer between the pools before it returns anything.
- The right ratio between the pools keeps moving with your traffic, and a running worker cannot change phase or GPU. So correcting the ratio means standing down workers that are serving perfectly well. GPU checkpointing, which saves the complete state of a running worker so it can be brought back later, lets you stand one down without throwing away the sessions it holds.
- In this piece we walk through what prefill and decode each ask of the hardware, why one pool cannot be good at both, and what the split costs and returns. Then we cover why the ratio keeps moving, and what it takes to correct it without losing the work in flight.
Prefill and decode ask the GPU for different things
Prefill is the phase that takes your prompt. Splitwise, a study from Microsoft Azure Research published at ISCA 2024, describes it as the pass in which every input prompt token runs through the model in parallel to generate the first output token, and the study says that phase tends to be computationally intensive.
Decode is everything after that first token. Each token is produced from only the last generated token and the KV cache, which holds the keys and values the model already computed for the session, so the model does not redo that work for every new token. The same study describes decode as more memory bandwidth and capacity intensive than the phase before it.
What separates the two phases is how much arithmetic each one gets out of every byte it moves. Prefill processes a whole prompt of tokens per read of the weights, so it does a large amount of arithmetic per byte. Decode has one token, but the same weights still have to come out of memory to serve it, so the same bytes move for a fraction of the work.
Decode runs out of memory before it runs out of arithmetic
Below a certain amount of arithmetic per byte, a faster arithmetic unit stops helping, and NERSC's roofline documentation calls that crossover the machine balance point. Its general rule is that a workload whose arithmetic intensity, meaning the arithmetic it does for each byte it moves, falls below the point is bandwidth bound: it is limited by how fast data moves through the memory system rather than by how fast the calculations run. Decode sits under the crossover, so more compute does not raise its token rate.
Memory capacity is the other place decode stops. Splitwise reports that the token phase is limited by memory capacity, because token throughput rises as the batch grows until a batch size of 64, at which point the machine runs out of memory. That batch size of 64 belongs to the study's own configuration and is not a general limit. So larger batches raise decode throughput until memory capacity prevents further growth, even though compute is still available.
One GPU serving both phases usually leaves something idle
Put both phases on the same GPU, and at most operating points one of them is underfed. Prefill leaves memory bandwidth unused, which is the resource decode is short of, and decode leaves the arithmetic units far below what prefill would have asked of them. Interleaving the two in time changes which phase is starved in a given slice, but the silicon underneath is the same either way.
The same mismatch turns up across a fleet whenever a pool is given the phase its hardware cannot feed. Decode sits on compute-rich, bandwidth-poor silicon while bandwidth-rich silicon runs prefill. Neither pool is failing at anything, so the only sign of the problem is fewer tokens from hardware that could have produced more.
Separating the phases fixes that mismatch, because each pool can then be sized and placed for the resource its own phase is bound by, instead of one pool trying to be good at both.
The split costs a KV cache transfer before it returns anything
Prefill computes the KV cache and decode reads it, so once the phases sit on separate machines, the cache has to cross between them. Splitwise names that transfer as the main overhead the split introduces. It measured the transfer at 0.8% of end-to-end latency, and separately at 16.5% added to the second token. Those two figures measure the same transfer in two ways and are never added together, because the first is the overhead spread across a whole request and the second is the delay between the first token and the second.
In return for that transfer overhead, Splitwise reports clusters reaching up to 1.4x higher throughput at 20% lower cost. The study presents the same result a second way, as 2.35x more throughput under the same power and cost budgets, so those are two operating points on one result rather than two separate gains and you pick one of them. That is the trade in front of you: a latency cost on the second token, set against a throughput and cost gain across the cluster.
Your traffic keeps moving the right ratio
The ratio of prefill work to decode work follows your traffic, and your traffic keeps changing. Prompt lengths and output lengths move with what people are doing with the model, so the balance between the two pools moves with them.
DOPD, a serving system described in a paper on arXiv, retunes that ratio against live load and reports up to 1.5x better goodput, meaning throughput that meets the latency target. It measures that against vLLM and DistServe, which it names as representative aggregation-based and disaggregation-based approaches.
Agent traffic can sit almost entirely at the decode end. In a study by Yuan, Nayak, Kundu, and Talati of ReAct-style agents across five benchmarks, with context caching in play, decode accounted for 91.0 to 98.6% of model time, because most input tokens are reused across turns and the run becomes decode-dominated. That measurement covers those agents and says nothing about inference in general, but a fleet serving that traffic has a different workload from the one used to size its pool ratio.
So classify your own traffic by phase before you touch an engine flag. Measure how much of your model time goes to prefill and how much goes to decode at the prompt and output lengths you serve today, because the hardware and the tuning both have to match that ratio.
Correcting the ratio means draining or killing workers that are doing nothing wrong
A pool gets its phase when it is deployed, and a running worker keeps that phase and stays on the GPU it started on. Requests can move between workers, and you can change which pools carry which phase, but a worker itself cannot change role or GPU while it runs. So acting on a measurement means draining or killing workers that are serving perfectly well and building their replacements somewhere else.
A worker that can be saved does not have to be killed
Saving the full state of a running workload so it can be brought back later is called checkpointing. A worker saved that way does not have to be drained to empty or killed with its sessions still in flight. It stops, and it comes back on compatible hardware with those sessions still in its cache, running the phase it was already running.
A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. So a worker sent to a different GPU model cold-starts on the destination, and no checkpoint carries a worker into a new role. Reassigning the pool is still your move.
Capturing the worker's state preserves the in-flight work on the pool you are leaving, and that is what we build at Cedana. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. For a serving fleet, that means we save the state of a running worker, meaning the model in memory and the cache holding every session in flight, and bring it back on compatible hardware. The pools your new ratio asks for are still built the way they always were, and what changes is that the sessions on the workers you stand down can finish somewhere else.
