TL;DR
- Your product owes the user a fast first token. What that promise costs in GPU memory grows as the conversation grows, because a long session is cheap to answer only while the tokens it already sent are still in the cache.
- Sizing against peak request rate is the usual answer. It produces a concurrency number the memory will not support, so the sessions that lose their place in memory prefill the whole conversation again.
- Your latency target and your concurrency target draw on the same memory budget, so they are one decision. GPU checkpointing changes only one term in it, which is what an interruption costs.
- In this piece we walk through what the first token's wait is paying for, how long real agent runs get, and why the KV cache is what fills the memory the weights leave. Then we cover the two ways a session loses its place without anything failing, the three settings that spend the budget, and the four inputs that share it.
The first token's cost is the part of the prompt that has to be read again
A coding agent or a long support thread keeps one conversation open across many turns, and your product still owes the user a fast first token on every one of them. Time to first token (TTFT) is the gap between a request arriving and the first output token coming back. It is set in prefill, the phase where the model reads the prompt before it produces anything. Prefill is compute bound. Decode, the phase that produces every token after the first, is bandwidth bound, on the same silicon running the same model.
So the prefill part of that wait scales with the number of tokens the model has to push through the forward pass, and queueing and the network add to it. That number drops below the length of the prompt whenever the earlier work is still in GPU memory. The KV cache holds that earlier work, and it is the per-token attention state the model keeps in GPU memory so it does not recompute tokens it has already seen.
The KV cache for a run of tokens depends only on those tokens, so two requests that share a prompt prefix can reuse the same cache instead of computing it twice, which cuts both the redundant computation and the memory it would take. When the prefix is still in memory, the engine prefills only the tokens the new turn added. The SGLang paper reports an average reduction of 1.7x in first-token latency for Vicuna-33B in production, and that is an average measured on one model in one deployment.
By turn 30 a coding agent is carrying roughly 80K tokens it has already sent
A multi-turn session is one conversation or agent run made of many requests, each carrying everything said before it, and agent runs sit at the long end of that. The vLLM project describes an agent run as a long-horizon loop that alternates between a reasoning step, where the model works through the context, and an action step, where it issues tool calls and takes in the results.
In a trace study the same project published with Mooncake in May 2026, the median run across 610 traces of Codex and GPT-5.4 on the SWE-bench Pro dataset was 33 turns. Context grew to roughly 80K tokens by turn 30, and the longest contexts ran beyond 180K. Each turn typically introduced only a few hundred to a few thousand new tokens.
So a late turn adds almost nothing and carries every token before it. While its prefix is in memory, that turn prefills only the new tokens and still pays its own decode. Once the prefix is gone, the prefill costs as much as reading the whole conversation from the beginning. Those traces are coding agents rather than chat traffic, so a support conversation that ends after a handful of turns sits at the other end of the same curve.
The cache is what fills the memory the weights leave
The KV cache grows with every token of context and with every concurrent request, so on long-context or high-concurrency traffic it is the growing cache that exhausts the memory left after the weights. Every token a session accumulates adds cache for that session, and every session running beside it adds its own.
A single 100K-token context can occupy gigabytes of that memory, about 3.8 GB for Kimi-2.5 with an FP8 KV cache in the vLLM post's example, and both the model and the cache precision move that figure.
A whole node has a ceiling of its own, and at a million tokens of context it holds a low single-digit number of sessions. Together AI's engineering write-up of its DeepSeek-V4 bring-up reports that the model's compressed cache layouts raised total KV cache capacity on a single NVIDIA HGX B200 node from roughly 1.2M tokens to 3.7M tokens, measured at a 1M-token context and varying with which layout is used. So divide the larger number by a 1M-token context, and that is the count you get, before anything else on the box is accounted for.
A session loses its place in memory without anything failing
Eviction and routing are the two things that take a prefix away, and neither of them is a crash. SGLang's RadixAttention holds the active caches in a radix tree that maps sequences of tokens to their cache tensors, and it applies a simple LRU policy that evicts the least recently used leaf first.
A new request walks as far down the tree as its prefix matches and prefills only the part that is new. So under memory pressure, the leaf that has gone longest without a request is the one that goes, and going a while without a request is exactly what your session does between turns.
Routing takes a prefix away the second way, and it does so even when the memory was there. A router balancing load across instances may not send the next turn of a session back to the instance that served the last one. The vLLM project notes that an instance which has never seen the prefix has to recompute it from scratch. The router balanced load as intended, but sending the session to another instance required a full prefill.
Three settings decide how the memory gets spent, and each one gives something up
Prefill batching sets how much prompt goes through in one chunk, and SGLang's cookbook prints the trade on its chunked prefill size flag, which is that lower values favor latency and higher values favor throughput. The cookbook regenerates its example values per model release and engine version, so a number copied from it dates quickly, while the trade it names does not.
The static memory fraction is the share of the whole card set aside for the weights plus the cache pool, so once the weights are loaded it decides how much memory the engine may hold as cache. The same cookbook names that flag as the one to set for memory utilization. Whatever value you choose, the trade underneath it does not move, because memory handed to the cache is memory the engine cannot use for anything else.
The ceiling on requests in flight fixes how many sessions compete for what is left. Modern engines use continuous batching, where requests join and leave the running batch token by token, so a finished request gives up its slot immediately and a waiting one takes it. Raising the ceiling raises throughput when there is traffic to fill it, but each session admitted under that ceiling holds its own cache in that remaining memory. The right ceiling therefore depends on your model and your distribution of context lengths, and it comes out of measurement rather than a default.
Four inputs draw on one budget
The budget is the memory left once the weights are loaded, and context length, residency, concurrency and your first-token target all draw on it, so settling three of them leaves little room in the fourth.
| Input | What sets it | What raising it costs | Where to measure it |
|---|---|---|---|
| Context length per session | The product, meaning how much history a turn carries and how long a run goes on | More cache per session, so fewer sessions fit on the node | The distribution of context lengths in your own traffic, at the tail as well as the median |
| Residency, how long a session's tokens stay in GPU memory | The memory left after the weights, the eviction policy, and the routing policy | Cache held by idle sessions between turns, which the active ones wanted | Cache hit rate per turn, and the share of turns that missed |
| Concurrency, how many sessions run at once | The ceiling on requests in flight, and the cache each session holds | Pressure on the cache, so more prefixes go before their next turn arrives | Requests in flight against cache occupancy at peak |
| The first-token target | The commitment you made to the product | A stricter number needs prefixes resident, which takes memory from concurrency | Time to first token at the tail, on the sessions that missed |
Start with the first-token number you committed to. Take the context lengths from real sessions rather than an average, because the long sessions are the ones with the most prefix to lose. Turn those lengths into memory using the model and the cache precision you serve, then decide how many of those sessions have to keep their prefix between turns. What remains after the weights and the runtime is the memory your concurrency has to fit inside.
If you size against peak request rate alone, you get a concurrency number that memory does not support, and the sessions that lose their prefix pay for it at the tail.
A median first-token time measured over traffic that mostly hits cache is mostly a median of the hits, so it is not a latency target on its own. In the trace study the cache hit rate was 94.2%, so the sessions that decide whether you meet your latency commitment sit in the small remainder, which the median never shows.
The budget assumes the worker stays where it is
If your deployment keeps its cache in the serving workers, every resident prefix lives inside one worker and nothing above the GPU holds a copy. That state can be reconstructed only by redoing the work that produced it, so a node failure, a rebalance, or a maintenance window turns an incremental prefill back into a full one for every session that was resident at the time.
Saving the full state of a running worker so it can be brought back later is called checkpointing. Checkpointing does not make prefill faster, and it does not raise the number of sessions a node can hold. What it changes is the cost of an interruption.
The restore destination must be compatible hardware. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
What the budget above cannot price is the moment a worker leaves and takes every prefix inside it. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. In practice that means we save the state of a running worker, its KV cache included, and bring it back on another GPU, so a session that survives its worker keeps the prefix it had at the last save instead of prefilling the whole run again. Your four inputs still have to fit one budget, and what a node failure takes out of it is the prefill since the last save rather than the whole conversation.
