How long do agentic sessions run, and where does their state live?

Almost everything an agent sends the model has already been computed

An agent session is one task from start to finish. A coding agent receives a task and works until it is done. A turn is one call to the model inside that session, and every call carries the whole conversation so far as input, including every earlier turn.

So the input grows with every turn, but most of what it carries is not new. In May 2026 the vLLM project published traces of coding agent sessions, where the median session ran 33 turns, and those traces show how fast the growth is: by turn 30 the context length grows to roughly 80K tokens, the longest contexts grow beyond 180K tokens, and each turn typically introduces only a few hundred to a few thousand new tokens. Everything else is prefix the model has already seen.

The prefix is the part of the input the model has already processed. On any turn after the first, it includes the system prompt, the agent's memory and skills, and the history of earlier turns.

The serving engine does not process that part again. It keeps the attention state for those tokens in GPU memory as the key-value cache, or KV cache, and the SGLang paper explains why that works: the cached state of a token depends only on the tokens before it, so a request that begins with the same tokens can reuse the cache already built for that prefix.

The share of input tokens that the engine finds already in the cache is the cache hit rate, and across the vLLM traces that rate was 94.2%.

The cache also grows for as long as the session does. CONCUR, a paper on running many agents on one system, finds that an agent's context and its cache footprint grow steadily over the agent's lifetime, and the vLLM post puts that growth at roughly 3.8 GB of KV cache for a 100K-token context on Kimi-2.5 at FP8.

So the cache is the session's accumulated work rather than per-request scratch. It holds the computation from every earlier turn, and a new turn adds only the newly appended tokens and the model's own output.

A second study on different models and tasks found the same thing

One dataset from one project is a thin base for a claim about how agents behave, so it is worth checking a second. Yuan and coauthors traced two other models, Qwen3.6-27B and Gemma4-31B, across five agentic benchmarks, and their measured cache-hit ratio ranged from 84.6 to 99.5%.

Their turn counts varied much more than a single median would suggest, and the paper reports no median at all, so the 33-turn figure belongs to the vLLM dataset alone. Both studies count turns rather than elapsed minutes, so what they report is a number of model calls and not a number of hours. What the two datasets agree on is the cache reuse.

Their traces also show where the time goes inside a turn: the model spent 91.0 to 98.6% of its time generating output rather than reading input, because the only input it has to read is the part that was appended since the last turn.

The next turn has to land on the worker that already holds the cache

A serving instance, or worker, is one running copy of the model on one or more GPUs, and each worker has its own memory and its own cache. A router in front of many workers picks one of them for each incoming request. A session's cache sits in the memory of the worker that built it, so unless the fleet copies it out to a shared pool, no other worker can read it.

A router that balances load will sometimes send the next turn of a session to a different instance, and the vLLM post spells out the consequence: that instance has never seen the prefix, so it must recompute it from scratch. Keeping the session where it is has its own limit, because large prefix caches on a busy worker fill its memory and force the engine to discard cached entries to make room for new ones.

The same post measured a miss on its Codex traces, using 12 GB200 GPUs: with only the system prompt cached, the cache hit rate was 1.7%. With a shared cache pool that made nearly the whole prefix reachable from any worker, it was 92.2%.

A shared pool moves the prefix, but it does not move the work in flight. It can hand back what it already wrote out, while the turn that was running when a worker died has no copy anywhere.

The other approach in use today is routing by cache location. NVIDIA Dynamo tracks where each session's cache sits across the cluster, so that a request lands on the worker already holding its context. Both approaches accept the same constraint: a worker's own GPU memory is private, so a turn either goes back to the worker holding the cache or reads a copy written somewhere outside that worker.

The vLLM post draws the general conclusion that "we can no longer treat an inference service as a set of isolated vLLM replicas."

A long session pins one worker for the length of the task

Sending every request from one session to the same server is called session affinity, and it is an ordinary load-balancing practice. What is new is how long the worker has to stay assigned. A short request releases its worker the moment it returns, but an agent session holds one through every turn and every pause between turns.

Those pauses happen while the agent runs its tools, and the vLLM traces time them: 5.2 seconds at the median and 81.4 seconds at the 99th percentile. The cache stays resident through all of them, because the next turn is going to need it.

So on a fleet where each worker keeps its own cache, the session's next turn has to return to that worker until the session ends, and the worker keeps the cache resident through the pauses. Rerouting the session buys load balance at the price of rebuilding its prefix somewhere else. A shared pool avoids some of that recomputation, because the prefix blocks it wrote out can be read back on another worker, but the turn in flight cannot be recovered from that pool.

Draining a worker means waiting for every session on it to finish, because a request boundary is not enough. And if the worker goes down, planned or not, its memory is cleared and the accumulated work of every session on it is gone.

The session itself continues after the worker is gone, because the transcript above the GPU survives, but the computation the model did on it does not. So the next turn sends the whole transcript back through the model, which computes it again from the beginning. The user sees a slow turn rather than a lost conversation, and the tool calls already made are not repeated. What is repeated is the prefill, the work of reading the whole input before the model writes anything new.

At Cedana, we call this dependence on one worker long-horizon session affinity. Accumulated work is pinned to one worker, and the longer the session runs, the more of it is pinned. That affinity is a property of the traffic rather than of any one serving stack, so you can measure it on your own fleet the way both studies did, by counting the turns per session and the share of input tokens served from cache.

The accumulated work only survives the worker if something outside the worker holds a copy of it. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. For an agent session, that means a checkpoint of the whole worker holds the KV cache together with the process around it, so the session's accumulated work no longer depends on the survival of the worker that computed it. Restoring the checkpoint brings back the state as of the last checkpoint. Then your agent session survives the worker that was holding it, and its next turn continues instead of sending the whole transcript through the model again.

Related:

• Your KV cache now rivals your weights
• KV-cache offload versus a saved worker: what each survives
• Does disaggregated serving remove the need to move GPU workers?
• Why an idle notebook keeps its GPU until you kill it