The engine layer: batching, caching, and configuration as a commitment

The hardware sets a ceiling that no serving engine moves

Performance on a machine is bounded before any request arrives. The flags you set on a vLLM or SGLang worker control batching, caching, and parallelism, so they decide how close the worker gets to that bound.

The roofline model, set out in NERSC's performance documentation, bounds floating-point performance using the machine's peak performance, its peak bandwidth, and the arithmetic intensity of the application. Arithmetic intensity means the arithmetic an application does for each byte it loads. So there are two ceilings, one on how fast the machine can compute and one on how fast it can move data to the compute, and which one you run into depends on where your workload's arithmetic intensity falls.

NERSC reads the model this way: below the point where the two lines cross, a workload is usually bound by how fast the data can be moved through the memory system rather than by how fast the calculations can be done. Generating tokens one at a time sits on that side, because each step reads the weights and does comparatively little arithmetic with them. Applying the roofline to token generation is our own reasoning.

Nothing in the serving engine moves either ceiling. What the engine changes is how close your deployment gets to the ceiling that binds it, and three mechanisms do most of that work: continuous batching, paged key-value caching, and prefix caching.

Batching shares one pass over the weights, but adds waiting

Continuous batching keeps a group of requests running together and admits a new request as soon as a slot frees, instead of waiting for a whole batch to finish. A single pass over the model weights serves every request in the group, so the cost of moving those weights is spread across all of them. On a workload bound by memory bandwidth, that sharing is the largest lever the engine has.

The PagedAttention paper, presented at SOSP in 2023, compared vLLM with FasterTransformer and Orca, and reports that vLLM improved the throughput of popular LLMs by 2 to 4x at the same level of latency on that paper's benchmarks.

The sharing costs the requests that cannot get into the batch. A request that arrives when the running batch is full waits for a slot, and the wait shows up in tail latency rather than in the average.

Every request in the batch also holds a key-value cache while it runs, and they all draw on the same pool of memory. That cache stores the attention state of a request so each new token does not recompute the whole conversation. So the memory you have limits how large the batch can be, and the PagedAttention paper traces that limit to how the memory is managed, because memory managed inefficiently is significantly wasted by fragmentation and redundant duplication.

Paging removes waste from the cache without slowing how fast it fills

PagedAttention manages the memory the key-value cache sits in, and its authors took the idea from virtual memory and paging in operating systems. The paper reports two results from that design, almost no wasted key-value cache memory and flexible sharing of the cache within and across requests.

Paging removes waste, but it does not slow growth. Every token generated adds to the cache, so a request that runs longer needs more memory at the end than it did at the start. Paging changes how efficiently that memory is used, but the rate at which the cache fills is the same as it was.

Prefix caching pays only where your traffic repeats itself

Prefix caching stores the key-value cache of a prompt prefix so that a later request sharing that prefix does not recompute it. SGLang's RadixAttention paper rests the mechanism on one property, which is that the cache for a prefix depends only on the prefix tokens. Requests that start the same way can therefore share it. But the saving exists only where later requests do share a prefix, and only while that prefix is still in the cache.

RadixAttention holds the live caches in a radix tree that maps sequences of tokens to their cache tensors, and it evicts the least recently used leaf first. So a prefix nothing has touched for a while can be gone by the time the next request that shares it arrives, which is why reuse is not guaranteed under load.

Prefix caching removes prefill work, the computation of the prompt before generation starts, so first-token latency improves. The paper reports an average reduction of 1.7x for Vicuna-33B in production. That is an average on one named model, and what it measures is a reduction in first-token latency, not a latency that holds flat as a conversation grows.

Your own traffic decides how much prefix caching returns

Agentic traffic repeats itself heavily, and the reuse has been measured on agentic benchmarks. A workload characterization study of agentic AI by Yuan, Nayak, Kundu, and Talati found most input tokens reused from the cache across turns, so each turn prefills only the appended region. Its empirical cache-hit ratio ranged from 84.6 to 99.5%.

Decode accounts for 91.0 to 98.6% of LLM time in the same study, and prefill for the rest, because the reuse leaves only the appended region of each turn to compute.

Traffic like that asks two things of one worker at once. Its arrival pattern looks like online serving, while its state looks like a long-running job accumulated over many turns, so the worker has to schedule for the arrivals and keep the state resident at the same time. A configuration tuned for one without the other is short of the traffic.

Those figures belong to that paper and its benchmarks. They are a reason to measure reuse on your own traffic rather than a rate to plan a deployment around.

Two presets for one model are two answers to the same question

SGLang's cookbook ships presets for the same model that point in opposite directions. Its GLM-5.1 source data file defines a preset named high-throughput-dp and a preset named speculative-mtp, and it marks the second one as optimizing for low latency.

The low-latency preset, on the GLM-5.1 page under SGLang v0.5.10, turns on EAGLE speculative decoding. The high-throughput preset on the same page and version turns on data-parallel attention with the enable-dp-attention flag at dp 8, and the page says that setting "trades off low-concurrency latency for high-concurrency throughput."

So choosing one of them means choosing which traffic the worker is right for. The batching, parallelism, and speculative-decoding settings named here are fixed when the worker launches, so changing them means a new worker.

An engine update makes published flag values outdated

The flag values above are facts about one model page at one engine version, which is why none of them is a recommendation. The cookbook gives the reason in its own scope note: "Because commands are generated from data and pinned to an SGLang version, exact flag values change per model release and engine version." The same holds for any value published anywhere else, so read it against the version you run.

Changing the regime means a new worker, and the old worker's state is lost

When the traffic changes shape, changing the regime means launching a worker with different flags and moving traffic onto it. The worker already running cannot follow the change, because its flags were fixed when it launched.

What that worker is holding does not come across with the traffic either, because inference accumulates that session-scoped state, everything the sessions in flight have built up since they started, only in VRAM. Any interruption destroys the state, and it can be reconstructed only by redoing the work.

What that costs a production fleet depends on how much of that state has to be rebuilt and on how the traffic moves onto the new worker. We put no figure on it.

Moving a worker keeps its flags and saves its work

A worker that can be moved still runs the flags it launched with, so moving it is not a way to change the regime. Moving it means saving its state and bringing that state back somewhere else, on the same engine and the same settings. That puts a condition on the move, because a Cedana checkpoint can restore only when the versions match.

A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. An engine upgrade makes published flag values outdated and invalidates checkpoints taken against the older engine.

So the flags stay with the worker, but the work the worker has done does not have to end with it. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a serving worker, that means we save the full state of a running worker and bring it back on compatible hardware, so the sessions it was holding can finish instead of being redone. A different regime is still a new worker, with its flags chosen at launch and its own cold start. What changes is the worker you are replacing, because stopping it no longer throws away the work it was in the middle of.