TL;DR
- Multi-node checkpointing means saving a job whose GPUs sit in several machines at the same time. Whether one of your jobs has that shape decides whether it can be checkpointed today.
- Watching a job get saved on one machine and brought back on another looks like the answer. But that job was only ever running inside one machine at a time, so it tells you nothing about a job whose GPUs are in several machines while it runs.
- The answer depends on the shape of one job, because a job is covered when all of its GPUs sit in the same machine. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On this question that means your single-machine jobs are covered today, including when you move one of them to a different machine. So the number to count is machines per job.
- In this piece we walk through the three sizes a GPU workload comes in and which of them ship, why a job that moves between machines is inside the tier that ships, and what every published restore figure was measured on. Then we cover why coordination across machines is the harder problem, and how to tell which tier your own inference, coding and research jobs sit in.
Two of the three sizes ship today
The three sizes a GPU workload comes in are single-GPU, multi-GPU-on-a-single-node, and multi-node, and checkpointing a job of any size means saving its full state while it runs so it can be brought back later.
| The size | What a job of that size looks like |
|---|---|
| Single-GPU | One workload on one GPU inside one machine |
| Multi-GPU-on-a-single-node | Several GPUs in the same machine put to work as one unit. Modern inference and high-performance computing (HPC) workloads span 4, 8, or 16 GPUs on a single node |
| Multi-node | One job whose GPUs sit in several machines at the same time and talk to each other over the network between those machines while the job runs |
The GPUs inside one machine coordinate through Nvidia's NVLink interconnect and frameworks like NCCL or MPI, short for the NVIDIA Collective Communications Library and the Message Passing Interface. Those frameworks let the processes driving those GPUs exchange data while a job runs.
The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
A job that moves between machines is inside the tier that ships
A job on the 8 GPUs of one node can be saved there and restored on the 8 GPUs of another node, so it moves between machines, but at no point is it running on two of them at once, because the save and the restore each happen inside one machine. That is what happens when we move a workload to an idle node or bring a job back on healthy hardware after a failure.
Multi-node describes the shape of the job itself, so while such a job runs, its GPUs are in more than one machine and they have to stay in step with each other the whole time.
The boundary is whether the job itself runs on more than one machine at a time, and it decides what you can promise the person who submitted the job. A job that fits inside one machine can be taken off a node before a maintenance window and put back afterwards, on that node or another one, but for a job whose GPUs sit in several machines we have no such path today.
Every restore figure we have published was measured on one node
Frontier models, meaning the latest open-weight models trailing the frontier labs, restore in 57 to 70 seconds on 8x B200 with Cedana, and a native start takes 9 to 34 minutes. Every run behind those figures was on a single node with 8x NVIDIA B200 GPUs and 1.7 TB of system memory on CUDA 12.9, and each model was served by the SGLang engine from its official recipe, unmodified.
Both figures come from timing the same engine. The native cold start runs from launch to ready-to-serve, so it includes loading the weights and initializing the engine in full, and the restore starts from a checkpoint of that finished engine and ends at the same ready-to-serve point.
Swapping one model for another on the same node sits in the same tier, because the running model is checkpointed and the target model is restored, and both of those happen inside one machine.
Both of the tiers that ship carry one limit on where a checkpoint can be brought back. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
The hard part is coordination, not capture
A job spread over many GPUs is running one computation, so its state is spread over those GPUs with it. Capturing that state cleanly requires synchronizing all the GPUs at exactly the same boundary in their distributed computation. If that synchronization is off, what comes back from one GPU does not fit what comes back from the rest.
We do that inside one machine today, but we have not finished doing it across machines, because single-GPU checkpointing is a challenging problem and distributed checkpointing across many nodes is an order of magnitude harder.
Frontier Labs (OpenAI, Anthropic, Meta, Google DeepMind) already have their own application-level solutions at that scale, where application-level means the checkpointing lives in the training or serving code itself. Each of those solutions is built for its own company's infrastructure, so it covers the code it was written for.
Most inference and coding work already sits inside one machine
If you run an inference platform or a neocloud, you usually run a frontier replica on one node of 8 GPUs, which is the second tier, so that worker can be checkpointed while it serves and restored later on a compatible node in the fleet. A replica big enough to need more than one machine to hold the model belongs to the third tier instead.
If you run your own coding models inside an enterprise, you usually keep a portfolio of them on one machine, which is the second tier again, and counting the machines your own deployment spans is how you confirm that.
Research computing is where the answer changes, because a university cluster, a national lab, or a pharma R&D group can have one job spread over many machines. What ships today does not cover a job of that shape, and we have not published a date, a quarter, or a definition of done for it.
A big cluster does not by itself put a site in the third tier, because coverage depends on where the GPUs of one job run, so a cluster of any size whose jobs each fit inside one machine is inside the tier that ships. If your site runs both kinds of work, measure the share of the queue that fits on one node, because that share is the part covered today.
Today we cover supported single-node workloads, meaning jobs whose GPUs are inside one machine, including moving such a job to a different machine. Count the machines your job's GPUs sit in, and you have your answer.
Common questions
Does multi-node checkpointing ship today?
The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds.
