TL;DR
- A serving node short of GPU memory is up against one of two different limits. It has run out of memory bandwidth, so every token waits on bytes, or it has run out of capacity, so it cannot admit more work.
- Adding another node raises both ceilings at once, so it can clear the symptom without ever telling you which limit you were under.
- The two limits take different fixes. Quantizing the weights cuts the bytes each token has to read, and quantizing the key-value cache buys back the space that grows with every session.
- Both levers change numerical precision, so the quality bar is yours to set and yours to measure. The change itself costs you the sessions your workers are holding when they restart.
- In this piece we walk through what fills GPU memory on a serving node, how to tell a bandwidth limit from a capacity limit, and what each quantization lever buys. Then we cover what a precision change costs on a fleet that is already serving.
The cache is the part of memory that keeps growing
The weights and the key-value cache are the two things that fill high-bandwidth memory (HBM) while a model serves, and only one of them grows. The weights arrive when the model loads and hold the same space for as long as it serves. The key-value cache, or KV cache, holds the attention state of every request in flight. It uses what is left and keeps growing.
The cache grows for two reasons, longer sessions and more of them at once, and both are things you want more of. Every token of context leaves keys and values behind for later tokens to attend to, so a longer session holds a bigger cache. AnTKV, a 2025 arXiv paper on cache quantization, says the KV cache grows rapidly with context length. Every session running beside it holds a cache of its own.
One long session is already worth gigabytes. In an example from the vLLM project's post on distributed key-value caching, a 100,000-token context occupies about 3.8 GB for one model's FP8 caches. That is one model at one cache precision, mentioned in passing in a post about something else, so treat it as a rough sense of scale rather than a way to size your own fleet.
Managing that memory better inside the engine gets some of it back, but it does not slow the growth. The PagedAttention paper, published at SOSP in 2023, describes cache memory that is large for each request and changes size while the request runs, so when that memory is managed badly, fragmentation and duplication waste it and the batch size stays down. PagedAttention is aimed at that waste and reports leaving almost none of it, so more of the memory you already have is usable. What it leaves unchanged is the rate at which the cache grows per session.
Bandwidth is a speed problem and capacity is an admission problem
Running short of GPU memory reaches you as slow tokens in one case and as refused work in the other. Tokens slow down when the arithmetic units sit waiting on bytes from memory. Work is refused when the weights plus the cache no longer fit, and then requests queue, cached prefixes are evicted, or an allocation fails.
The speed limit is the one the roofline model describes, and it turns on how much arithmetic a piece of work does per byte it moves. NERSC's roofline documentation defines arithmetic intensity as the ratio of the floating-point operations a piece of work performs to the bytes of data movement those operations need, so it is arithmetic done per byte moved. Every machine has a value of that ratio where its two limits cross, which NERSC calls the machine balance point.
Work below that point is usually bandwidth bound, which means it is held back by how fast data moves through the memory system rather than by how fast the calculations run. Work above the point is more likely to be limited by the calculations themselves. NERSC qualifies both statements, and a real kernel can fall short of either ceiling for reasons of its own.
Decoding sits below the crossing on most hardware, because generating one token reads the model's weights out of memory to do a small amount of arithmetic per byte. NERSC describes the crossing in general terms, and applying it to decoding is our own reading. Together AI's engineering post on serving DeepSeek-V4 describes the same pressure from the operator's side, where long-context, decode-heavy workloads spend much of their time reading cache.
Capacity is not on the roofline at all. A roofline describes a rate, so it says nothing about a workload that does not fit in the first place. GPU memory has no automatic spill, so an allocation that does not fit waits or fails instead of running slower, which puts the trouble in admission and eviction rather than in the throughput curve.
Your serving stack already reports enough to tell the two apart. The symptoms are clues rather than proof, so confirm the reading with the memory-bandwidth counter before you change anything.
| What you see | Which wall it is | What to change |
|---|---|---|
| Tokens per second flattens or falls while the arithmetic units are not saturated | Bandwidth | Quantize whichever dominates the bytes read, the weights or the cache |
| Requests queue, cached prefixes are evicted, or an allocation fails | Capacity | Quantize the cache, or pick a model whose attention layout bounds the cache |
Both limits can apply at once, and they take different changes, so work out which one you are under before you change anything. When a serving node is short of GPU memory the next question is usually whether to add another node, and another node raises both ceilings at the same time. That is why it can clear the symptom without ever telling you what the limit was.
Quantizing the weights cuts the bytes each token has to read
Holding each parameter in fewer bytes means fewer bytes streamed for every token you decode. The weights account for part of the bytes streamed per token, and on long sessions the cache accounts for another part. How much faster weight quantization makes decoding depends on the model, the batch size and the GPU, so measure it on your own hardware rather than reading it off a page.
Quantizing the weights also raises the compute ceiling, because the tensor cores run low-precision arithmetic faster. NVIDIA's Blackwell datasheet puts the FP4 tensor core peak on a single B200 at twice its FP8 peak and four times its FP16 and BF16 peak.
Quantizing the cache buys you more sessions
Cache entries at lower precision shrink the part of memory that grows, so the node can admit more sessions at once, or longer ones. On a long session they also cut the bytes each token reads back, which is why a decode-heavy fleet can hit the bandwidth wall and reach for this lever instead of the weights.
Cache quantization is already a setting you can turn on rather than a research idea. The SGLang Cookbook's page for DeepSeek-V4 states that "KV cache dtype options include mxfp8". That is what one recipe for one model carries, and the cookbook does not offer it as a default. Its commands are generated from versioned data files pinned to an engine version, so the flags move from one model release to the next.
The other way to cap the cache belongs to the model rather than to a setting, and it is an attention design that keeps only a bounded window of recent context exact. DeepSeek-V4 does that in part, with a sliding window of around 128 tokens on some of its layers mixed with other layouts, so the window bounds part of the state rather than all of it. The trade is some modeling reach for a cache that grows more slowly with the session.
DeepSeek-V4 also uses compressed cache layouts, which are a different mechanism from the window. Together AI's engineering post on serving DeepSeek-V4 reports that they raised the total KV cache capacity of a single NVIDIA HGX B200 node from roughly 1.2M tokens to 3.7M tokens. How much capacity you get varies with the layout in use.
A faster configuration is not automatically a better one
Quantizing the weights and quantizing the cache both alter numerical precision, so both can change what the model produces. Set the quality bar for your own task first, then measure each model against that bar after the change.
A published accuracy loss for FP4, for an mxfp8 cache, or for any other scheme is a measurement of someone else's model on someone else's task, so it does not transfer to yours.
A precision change costs you the sessions your workers are holding
A precision change is not an edit you make on a running worker, because the worker restarts to pick it up. So on a fleet that is already serving, you drain workers a group at a time, and a drain that lets requests finish first costs only the wait, while stopping a worker mid-session ends the sessions it holds.
Those sessions are work rather than scratch. Inference now accumulates session-scoped state that lives only in VRAM, can be reconstructed only by redoing the work, and is destroyed by any interruption. A stop before the sessions finish is one of those interruptions.
Saving the full state of a running worker so it can be brought back later is called checkpointing. A worker saved that way does not have to lose its sessions when it stops.
A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. So a restore brings a worker back as it was, and it cannot bring one back at the new precision.
The new precision therefore arrives as a new worker either way, and the open question is whether the old worker's sessions survive the swap. Keeping them is what we work on at Cedana. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. Here that means we save the running state of a worker and bring it back on compatible hardware, so a drain moves the sessions it was holding instead of ending them. When you plan a precision change of your own, count the sessions your workers are holding as part of what it costs, and then decide whether they have to be part of it.
