TL;DR
- Two vendors hand you a GPU checkpointing benchmark page and each claims a multiple against a baseline of its own. So you have to decide from the page alone whether either number means anything.
- A multiplier describes two runs and most pages describe one. The run that goes missing is usually the baseline: what the comparison ran against, on what hardware, and whether it was timed the same way.
- Nine items decide it, from the model and the checkpoint size through both clocks and the baseline to the storage path and the spread across runs.
- If those are on the page and the configuration is public, you can rebuild the run yourself. If they are not, the number is a claim about a setup you cannot see, and that test applies to our pages too.
- In this piece we walk through what makes two results comparable, the nine items, and why an overhead percentage on its own compares with nothing. Then we cover what a restore time depends on, and how our own published benchmark scores against the same nine.
Fixed conditions are what make two results comparable
Two results are comparable when the workload was fixed before either system was measured against it. MLCommons, the consortium that publishes the MLPerf benchmarks, does that for inference. It defines each datacenter benchmark by a dataset and a quality target, runs the scenarios under a standard load generator that sends requests in a set pattern, and sorts submissions into a closed division that requires the reference models or an open division that allows different ones. All of that is settled before anyone measures a system, so the results describe the systems rather than the test.
A vendor benchmark page has no such rules behind it, so the conditions are whatever the vendor chose and whatever the page reports. A page that prints a multiplier is describing two runs, and most pages describe one. The item most often missing is the baseline: what the comparison ran against, on what hardware, and whether it was timed the same way. Without it, the multiplier is a statement about a setup you cannot see.
Nine items decide whether the number means anything
The baseline is one of nine items, and every one of them moves the number when it changes, so a page that leaves one blank has handed you a variable you cannot account for.
| The item | What the page should say | What a missing entry hides |
|---|---|---|
| Model and parameter count | The exact model and its size | Whether the run resembles anything you serve |
| Checkpoint size | The bytes each restore had to move | The variable that predicts the restore column |
| GPU count and type | The accelerator, how many, and whether they sat in one node | That a single-GPU result and an 8-GPU result are different measurements |
| Serving engine and configuration | The engine, its version, and the recipe or arguments used | Whether the run can be rebuilt at all |
| What each clock starts and stops on | The start and stop events for both numbers, in words | That two pages can use one word for two different intervals |
| The baseline | What the comparison ran against, and on what hardware | The second of the two runs a multiplier describes |
| Storage path | Where the checkpoint was read from and written to | A variable that moves restore time on its own |
| Number of runs and spread | How many times each figure was measured, and how much it varied | Whether one run or a mean is being reported |
| Performance after the restore | How the workload ran once it was serving again | Whether the restored workload matches a natively started one |
An overhead percentage on its own compares with nothing
If you are weighing application-level checkpointing against system-level checkpointing, the figure you want for each is an overhead percentage. It becomes comparable only when the page states the workload it ran, the hardware it ran on, and the window the measurement covers, because a cost amortized over a long run and a cost measured per call are different quantities.
The two kinds of checkpoint also capture different amounts of the workload, so their timings are not comparable even when the conditions match. An application-level checkpoint saves what that one application knows about, while a system-level one saves the process, its GPU state, and the state around it. So compare capture scope first and timings second.
A restore time means whatever the page's clock says it means
A restore time is worth no more than the clock behind it, so the definition of the clock matters more than the figure beside it. Cedana's published cold-start page for 8x B200 gives the hardware and the engine before it gives either clock. All runs were on a single node with 8x NVIDIA B200 (Blackwell) and 1.7 TB of system memory, CUDA 12.9.
Models were served with SGLang using the official SGLang cookbook recipes for each model, unmodified. Those recipes are public, so anyone with the same node can pull the same recipe from the SGLang cookbook and run it, which makes the engine item checkable rather than merely stated.
Each clock is then defined by the events it starts and stops on, which is what most pages leave the reader to guess. Native cold start is the time from engine launch to ready-to-serve, including weight loading and full engine initialization. Cedana restore is the time to restore the same fully-initialized engine from a Cedana checkpoint to ready-to-serve. Both clocks stop at the same state on the same node, so the speedup column is a comparison rather than a pair of unrelated timings.
The first of those two definitions also explains why a large model takes so long to come up. A native cold start is not only weight loading, because everything else the engine does before it can serve a request sits inside that number.
Checkpoint size is the variable that predicts the restore column
The same page prints one row per model with the checkpoint size beside the times, which is what lets you see which column the restore time tracks.
| Model | Parameters | Checkpoint | Native cold start | Cedana restore | Speedup |
|---|---|---|---|---|---|
| MiniMax-M2.7 | 229B | 244 GiB | 564 s | 57 s | 9.9x |
| GLM-5.2-FP8 | 753B | 734 GiB | 1,322 s | 61 s | 21.7x |
| Kimi-K2.6 | 1100B | 670 GiB | 1,217 s | 63 s | 19.3x |
| DeepSeek-V4-Pro | 1,600B | 873 GiB | 2,051 s | 70 s | 29.3x |
The restore column does not follow the parameter column. Parameter counts grow sevenfold across these four rows while the restore times run from 57 to 70 seconds. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. Four rows cannot demonstrate that on their own, and the 734 GiB checkpoint here came back slightly faster than the 670 GiB one. But the reading rule holds either way: a page that gives parameter counts without checkpoint sizes omits the variable that determines how many bytes a restore has to move.
The page also makes two claims that no row on it measures. We capture at the operating system and CUDA driver level, so the results do not depend on the serving framework, and equivalent behavior applies across vLLM and SGLang. Accelerated cold starts are also validated on H100 and A100 systems. Neither claim has a benchmark row beside it, so the measured set is still four models served by SGLang on one node of 8x B200. Split any page you are reading the same way, into what was timed and what was asserted.
Against the nine items, that page discloses the model and parameter count, the checkpoint size, the GPU count and type, the engine and recipe, both clocks, and the baseline. It does not disclose the storage path, the number of runs and their spread, or how the workload performed after the restore, so three of the nine are missing from our own page.
Where the checkpoint is read from changes the restore time
Where a checkpoint is written and read back is a separate choice from how it is captured, and it moves the restore time on its own. Reading the state out of memory is the capture: our checkpoints stream directly from memory with no disk I/O. Those bytes can then be written to NFS, S3, Google Cloud Storage, or our own storage, depending on the security posture. A restore reads them back over whichever path was chosen, and bandwidth differs from one destination to the next, so the path belongs beside any published restore time.
The four runs in the table above used tmpfs, a filesystem that lives in system memory rather than on disk. The public benchmark page does not say so, which is why the storage path is one of the three items it leaves open.
Ask every vendor, including us, which of its own runs came out worse
The nine items are a standard only if a vendor answers them for the runs that went badly too, so we also have to report where our own runs performed worse. In multi-GPU runs on smaller models, a workload has run slower under Cedana than it runs natively, and we publish no number for those runs. Put the same question to every benchmark page you are handed, ours included: which runs came out worse?
The test is whether you could rebuild the run from the page
If the hardware, the engine configuration, and the baseline are named, and the configuration is public, you can rebuild the run. If the page hands you a multiplier and a logo, you cannot, and the number is a claim about a setup nobody outside the vendor has seen.
Our FLUX page shows those disclosures for a different workload class. It prints the exact arguments the run used, down to the compile mode and the autotune search space, so the configuration sits on the page instead of being described. It also labels its headline speedup as computed from about 50 minutes against 73 seconds, rather than presenting a rounded native time as a measured one. FLUX is image generation, so that pair of numbers says nothing about a language model on the same hardware.
A published paper can take disclosure further than a vendor page usually does. CRIUgpu, an evaluation of transparent GPU checkpointing published on arXiv, lists the hardware for every server it used, the operating system, the kernel version on each machine, and the driver and CUDA versions. It names its baseline instead of leaving the comparison implied, and it says how many times each figure was measured: "We run each experiment 10 times and calculate the mean and standard deviation of each value in the collected data."
We publish benchmark pages of our own, so these nine items are ours to answer too. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. The numbers in the table above are ours, and the page they come from describes the run well enough for you to rebuild it, while still owing you three of the nine disclosures. Read the next page you are handed for what it discloses before you read what it claims.
Related:
- Which checkpointing approach brings back the state your job is holding?
- What a GPU checkpointing layer costs while the workload runs
- Where the minutes go when an LLM worker cold starts
- What NVIDIA Dynamo Snapshot and Modal GPU memory snapshots restore today
Common questions
How do I read a GPU checkpointing benchmark?
Read it for nine items: the model and parameter count, the checkpoint size, the GPU count and type, the serving engine and its configuration, what each clock starts and stops on, the baseline it compares against, the storage path, the number of runs and spread, and performance after the restore. If the hardware, the engine configuration, and the baseline are all named and the configuration is public, you can rebuild the run. If the page hands you only a multiplier and a logo, you cannot, and the number is a claim about a setup nobody outside the vendor has seen.
Do application-level and system-level checkpoints produce comparable overhead numbers?
No. The two kinds of checkpoint capture different amounts of the workload, so their timings are not comparable even when the measurement conditions match. An application-level checkpoint saves only what that one application knows about, while a system-level one saves the process, its GPU state, and the state around it. So compare capture scope first and timings second.
