The fleet's real yield metric is tokens per gigabyte of VRAM

Serving pays for the gap between compute and memory bandwidth

Compute has pulled away from memory for 20 years. Peak compute scaled 3.0 times every 2 years over that period while DRAM bandwidth scaled 1.6 times, measured across server hardware by Gholami, Yao, Kim, Hooper, Mahoney, and Keutzer in "AI and Memory Wall" (IEEE Micro, 2024). That disparity has made memory, rather than compute, the primary bottleneck in AI applications, and serving is where it bites hardest.

Whether the gap reaches your workload depends on its arithmetic intensity, meaning how much arithmetic it does for each byte it reads out of memory. NERSC's documentation of the roofline model, which plots what a machine can deliver against that ratio, puts a crossover point on it. An application below the point is bandwidth bound, so its speed is set by how fast data moves through the memory system rather than by how fast the calculations run.

Decode sits below that point. Decode is the phase that produces every token after the first, and it sweeps the model's weights out of GPU memory once for each token it emits. The Splitwise paper (Patel and colleagues, ISCA 2024) arrives at the same place from the input side: once the first token exists, the only inputs left are the last token generated and the key-value (KV) cache, which is the working state a session builds up as it runs. So token generation is intensive in memory bandwidth and capacity, and adding compute to a decode worker does not change that.

Decode is almost the whole workload on agentic traffic

On agentic traffic, decode is almost all of the work. Yuan, Nayak, Kundu, and Talati characterized ReAct-style agents, which alternate reasoning steps with tool calls, running two open models across five benchmarks with context caching. They found that decode accounts for 91.0 to 98.6% of the model's time, and that prefill, the pass that reads the prompt, takes 1.4 to 9.0%.

The finding is about agentic serving rather than about all inference. On that traffic, the phase that consumes the bandwidth is also the phase that produces nearly every token.

The cache and the weights compete for the same gigabytes

Capacity is the second bound, because the weights and the cache draw on the same pool of VRAM. Together AI's engineering write-up on serving DeepSeek-V4 reports that a change to the cache layout raised the usable KV-cache capacity on one NVIDIA HGX B200 node from roughly 1.2 million tokens to 3.7 million, so a node holds a few million tokens of cache at best. The weights take the rest of the memory, and every live session claims a share of what they leave.

Cold starts, failures and immobility fill VRAM with gigabytes that make no tokens

A cold start makes you hold gigabytes of weights before the traffic arrives. A frontier worker, meaning one serving the latest open-weight models trailing the frontier labs, takes a long time to start, and Alibaba Cloud's deployment documentation puts the load of a full-version DeepSeek-R1 at 20 to 30 minutes. You cannot take that wait during a traffic spike, so the answer today is a warm pool, replicas kept loaded and idling ahead of demand. A warm replica is half a terabyte or more of weights spread across a whole node before the first byte of session state arrives, and it produces nothing until traffic reaches it.

A failure fills the same gigabytes twice for one result. GhostServe, a paper on fault-tolerant serving at MLSys 2026, describes the loss: "In the event of a failure, this volatile state is lost, forcing the system to restart the inference job from the very beginning." Recovery reloads the weights and recomputes the cache into the same memory a second time, so the fleet paid for two fills and delivered one result. For each failure on your own fleet, multiply the restart time by the number of GPUs the worker spans, then total those losses over the period.

Immobility is work that cannot move once it has been placed, and its everyday form is fragmentation. Weng and colleagues, in a USENIX ATC 2023 study of fragmentation, describe a production-trace cluster of more than 6,200 GPUs that was fully packed and could not accept new tasks while 500 of its GPUs sat unallocated. Those 500 were scattered across nodes that were otherwise full, and a frontier worker needs several GPUs together.

A failure wakes somebody up, and a slow start shows in the launch-time metrics, but powered gigabytes that produce nothing appear nowhere.

Decode earns the H200 premium and prefill does not

The phase a worker is running determines which GPU it belongs on. Prefill runs against the compute ceiling, which the H100 and H200 share.

An H200 running prefill pays a premium for bandwidth that the phase does not use. A decode worker uses all of that extra bandwidth, because the two memory rows above set how fast it emits tokens and how many sessions it can hold at once. So ask the question per phase rather than per model: write down the phase each worker is running and the memory bandwidth of the GPU under it.

Splitting the two phases onto hardware matched to each has been measured, and the Splitwise authors report up to 1.4 times higher throughput at 20% lower cost. That came from a research system on modeled clusters, with the placement settled when the cluster was designed. If your traffic mix moves through the day, holding that match means re-placing workers as it moves, and a worker sent to a different GPU model cold-starts on the destination.

VRAM should hold only the state that is producing tokens now

An idle replica holds gigabytes of your fleet's memory and serves nobody. It is there because starting one from scratch takes tens of minutes, so the pool is sized for the peak rather than for the traffic in front of it.

A checkpoint of the whole running workload, including weights, cache, and CUDA context, makes it safe to take an idle replica out of VRAM. The checkpoint is taken while the workload runs and written to cheaper storage. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count.

Cedana's benchmark page reports a test on one node of 8 NVIDIA B200 GPUs in which frontier models restored in 57 to 70 seconds, against native starts of up to 34.2 minutes. Both clocks stop when the engine is ready to serve: the native clock starts at engine launch and includes loading the weights, and the restore clock starts when the same initialized engine begins coming back.

The checkpoints in that test sat on tmpfs, a file system held in the node's own memory, so those seconds are what a restore costs when the checkpoint is already on the node. A restore from a disk or an object store takes longer, so measure it on your own path.

A checkpoint does not expire on a clock, and you can suspend a workload and resume it a week later. What invalidates one is a change underneath it. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead. Engine and driver updates land every week or two, so a replica parked through an update window comes back as a cold start.

Which replica is idle enough to leave VRAM is a policy call, and it stays yours. What changes is that acting on the decision no longer means destroying the worker.

That ability, to take a live workload out of VRAM and bring it back, is what we work on at Cedana. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We save the full state of a running workload and bring it back on compatible hardware, so an idle replica can leave VRAM instead of holding it, and after a failure the same gigabytes are refilled from a saved copy in about a minute instead of rebuilt over tens of minutes. The fleet stays the same size and delivers more tokens.

Tokens per gigabyte is the yield on the input that is hardest to buy

The arithmetic is the tokens your fleet delivered in a period, divided by the gigabytes of VRAM it held during that period. For a fleet whose size does not change, the denominator is its installed VRAM, idle gigabytes included, and a node carrying 8 H200s holds 1,128 GB of it. Because the denominator is fixed, a gigabyte that stops producing lowers the number, whatever put it in that state.

None of this replaces the scoreboard you already keep. NVIDIA's own framing calls cost per token the one measure of total cost of ownership that accounts for hardware performance, software optimization, ecosystem support, and real-world utilization. Cost per token is revenue per megawatt read from the opposite side, and both improve when the same fleet delivers more tokens. Tokens per gigabyte puts that same output over the gigabytes you already own.

TrendForce's July 2026 survey has the DRAM market staying extremely tight through the third quarter of 2026, contract prices still rising, and suppliers continuing to prioritize AI when they allocate production capacity. As new memory gets more expensive, the yield on the memory already installed is worth more, and it is the one number on your dashboard you can move without buying a gigabyte.

Related:

• How to put a dollar figure on the GPU-hours that produced nothing
• Cost per token and revenue per megawatt are the same number
• Your KV cache now rivals your weights
• Can LLM inference scale to zero without paying for warm replicas?

Common questions

How much is idle, queued, or unused GPU capacity costing us, and does a high utilization number even mean the GPUs are doing useful work?

A warm replica is half a terabyte or more of weights spread across a whole node before the first byte of session state arrives, and it produces nothing until traffic reaches it. A failure fills the same gigabytes twice for one result, and immobility strands GPUs that are otherwise healthy while a frontier worker needs several of them together. The failure raises an incident and the slow start shows in the launch-time metrics, but the idle and stranded gigabytes appear nowhere, and all three lower what the fleet delivers.

What's the right yardstick to compare what different GPUs or generations deliver for the money, and how do I compute it for my own fleet?

The arithmetic is the tokens the fleet delivered in a period divided by the gigabytes of VRAM it held during that period. For a fleet whose size does not change, the denominator is its installed VRAM, idle gigabytes included. NVIDIA's own framing calls cost per token the one measure of total cost of ownership that accounts for hardware performance, software optimization, ecosystem support, and real-world utilization, and it is revenue per megawatt read from the opposite side.