TL;DR
- Serving long context runs a node out of GPU memory before it runs out of compute. Every live session keeps its attention state in that memory, and that state grows with the conversation.
- Paging and sharing the cache removes the waste around that growth, but nothing there changes the growth itself.
- Sliding window and state space designs do change it, and both put a ceiling on how far the memory cost can climb.
- In the one configuration with published numbers, the sliding window layout costs more per token than the model before it. The large capacity gain reported beside it came from the compressed layouts rather than from the window.
- In this piece we walk through what sets sliding window memory use, what the published figures do and do not show, why there is no state space figure here, and what every design leaves sitting in GPU memory.
Memory management removes the waste around the cache and leaves the growth alone
Under full attention, every token attends to every token before it, so the key-value cache (KV cache), the memory holding the attention state of a request while it runs, grows with the length of the conversation. That growth is what decides how many sessions fit on a node, and it is what you run into when the next session will not start.
The PagedAttention paper works on the waste around that growth rather than on the growth itself. It pages the cache and shares it across requests, and it reports "near-zero waste in KV cache memory". Nothing in it changes how the cache scales with sequence length, and that last reading is ours rather than a claim the paper makes.
The GPU memory ceiling itself is a hard one, and serving long-context traffic runs into it well before it runs out of compute. A KV cache is resident in the GPU's own memory, its VRAM, and it is read constantly while the model decodes. Together AI's engineering post on serving DeepSeek-V4 reports that long-context, decode-heavy workloads spend much of their time reading cache.
Nothing else is fast enough to page that cache out to, so the memory a node has is the memory the workload gets. What is left to change is how the cache grows, and that is a property of the model rather than of the scheduler.
Two families of design change how fast the cache grows
Sliding window and state space models are usually offered as the fix, on the grounds that they use less memory. A sliding window keeps a short span of the most recent tokens exact and handles older context some other way. A state space design carries a recurrent state whose size is set by the design rather than by the number of tokens the session has seen, so what a session holds stops tracking how long it has run.
Both put a ceiling on how far the memory cost can climb, and both give up a measure of modeling reach in exchange. How much reach a given design gives up is a property of that design rather than of the family it belongs to.
In one configuration, a sliding window raised the cost of holding one token
Only one configuration has published numbers in the sources this post uses, and it is DeepSeek-V4 as Together AI served it. Together AI's engineering post covers serving that model on a single NVIDIA HGX B200 node, and the model uses a short window of around 128 tokens to keep recent context exact.
The model has 61 layers, so the 128-token window counted once per layer comes to the roughly 8K tokens of recompute the post puts against it. That is a count of window positions rather than a prompt length, because an 8K-token prompt runs 8K tokens through every layer, which is a different quantity. The figure is that window size multiplied by those 61 layers, so it does not carry over to another window size or another depth.
The cost of holding one token went up under that layout. In the same post the sliding window layout takes roughly 3.8 KB per token, against roughly 3.4 KB for the previous model, on that node.
The capacity gain came from holding fewer token positions
The same post does report a large capacity gain, and it does not belong to the window. Compressed cache layouts increased total KV-cache capacity on a single NVIDIA HGX B200 node from roughly 1.2M tokens to 3.7M tokens depending on cache policy, so that gain belongs to those layouts as a set.
| Layer type | What it holds | What the post reports about it |
|---|---|---|
| Sliding Window Attention | Recent context, exact, inside a window of around 128 tokens | Roughly 3.8 KB per token, against roughly 3.4 KB for the previous model |
| Compressed Sparse Attention | Context compressed with stride 4, each entry summarizing an 8-token neighborhood | The stride and the neighborhood size, with no separate capacity figure |
| Heavily Compressed Attention | Context compressed with stride 128 | At a 1M-token context length, the cache falls from 1M token positions to roughly 8K compressed entries |
Every cell comes from that one post, on DeepSeek-V4 on one NVIDIA HGX B200 node. The three rows are layer types inside a single model's cache design rather than three designs competing for the same job, so in DeepSeek-V4 the sliding window layers hold the recent range at full resolution while the two compressed layouts hold the longer range at lower resolution.
The roughly 8K compressed entries in the last row are cache entries at a 1M-token context, which is a different quantity from the roughly 8K tokens of recompute above.
Production models mix sliding window and full attention layers
The SGLang cookbook's DeepSeek-V4 recipe carries a flag named swa-full-tokens-ratio, set to 0.1. The cookbook states the flag and its value, and the rest is our reading: a name like that splits a token budget between sliding window layers and full attention layers, so it means something only in a model that has both. DeepSeek-V4's cache design is therefore a mixture rather than one mechanism applied to every layer.
Cookbook values are also pinned to an engine version and a model release, so the number moves with releases and you should read it off the recipe you are running.
This post has no state space figure to put beside these
Kimi K3 is a recent model whose public material does not place it in either family. Its release blog names Kimi Delta Attention (KDA) and Attention Residuals (AttnRes) as its memory mechanism, but a blog is not an architecture paper, so it does not say how those mechanisms bound memory.
A mechanism name tells you what a design is called and nothing about what it holds, and that rule is the same for both families, so take the memory bound from the architecture paper of the design you are looking at. Every figure above belongs to one sliding window configuration on one node, and the absence of a comparable state space figure here reflects what this post's sources cover rather than the merits of the two approaches.
Every design leaves the session state in GPU memory
Whichever family you end up serving, the serving process holds the session's state while the session runs, whether that state is a KV cache, a recurrent state, or both. Inference now accumulates session-scoped state that lives only in VRAM. That state can be reconstructed only by redoing the work, and any interruption destroys it. The attention design sets how large that state grows, and it changes nothing about what happens to the state when the node fails or the spot instance is reclaimed.
Saving the full state of a running workload so it can be brought back later is called checkpointing, and a checkpoint turns an interruption into a pause rather than a restart. That is what we work on at Cedana. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We take the checkpoint below the serving engine, so the cache travels with the rest of the workload and comes back on healthy hardware as of the last checkpoint. How large that saved state is depends on the attention design you picked.
A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. So if you are weighing a move to a model with a different attention design, that move is a model change, and it invalidates the checkpoints you are holding the same way a driver change does.
