Does moving a job change its results?

A restore continues the run, a restart begins a new one

A job moved off its node, because the node failed or a maintenance window opened, comes back either as a restart, which begins the job again, or as a restore, which continues the run that was already underway. A restart begins from what is on disk, the model file and the job script, so everything the run had built on top of those since it launched has to be built again. A restore puts the built state back, and the run carries on from there, so nothing computed before the checkpoint is derived a second time.

For the run to carry on, the saved state has to hold what the run was holding. What we save with Cedana is the running process together with its GPU state, and we guarantee CUDA semantics along with the process and file system state. The saved state holds the driver state inside the process, including the CUDA context, which is the working environment the driver keeps for a process. It also holds the intermediate and scratch output still sitting in GPU memory, the files as they were in their read and write layers, and the socket queues.

So the only work that happens twice is the work that ran after the checkpoint was taken. Work that was half finished at that moment is not done over, because its intermediate output was saved, and values the run had already computed but not yet consumed come back with it, so those calculations are not repeated either.

An application that writes its own checkpoints and a checkpoint taken below the application both return the run to a saved point, and the work after that point runs again either way, so what differs is what the saved point holds. The application saves the values it chose to write, at the step it chose. A checkpoint taken below the application holds the whole running process and its GPU state as they were at the moment of capture, with no change to the application.

A checkpoint only restores where the versions match

On a catastrophic failure, the workload automatically resumes on a different GPU from its last checkpoint. That GPU is a different physical unit of the same model, running the same driver, the same engine, and the same model version, and that is the scope of the guarantee.

A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

So the thing people picture, a job landing on unfamiliar hardware and quietly printing a different number, does not happen, because the checkpoint is not valid there in the first place. The job either restores on matching versions or cold-starts.

Four things are held fixed between a checkpoint and its restore, and two are free to change.

Only one of the two exceptions can change a number

In two cases the run does not simply continue where it stopped, and only the first of them can leave you with a different number.

1. A workload that ingests real-time data can come back to different data, because the feed moved on while the job was stopped.
2. A checkpoint whose versions no longer match does not restore, so the workload cold-starts and does the work again from the beginning.

The first one belongs to the workload rather than to the checkpoint. We guarantee correctness, but a time-dependent workload reading real-time data can still be affected, and nothing about the saved state changed while the job was stopped. What changed is the world the job was reading. GROMACS, one of the workloads on our tested single-GPU list, takes its inputs from files on disk, so a run like that is not exposed to a moving feed at all.

The second one belongs to whoever runs the cluster. Nothing goes quietly wrong when versions drift between the capture and the restore. What you get is a cold start, which costs time and not correctness, so keeping the four fixed parts of the run consistent between a capture and its restore keeps a checkpoint usable.

We run correctness tests, detect when a restore has a problem, and do not let it go silently wrong. In the worst case, there is no restore and the workload cold-starts, which is what it would have done without us.

Run-to-run variance on GPUs has nothing to do with the move

Two GPU runs of the same code can differ even when nothing moved, and that is the reason the question gets asked at all.

Work is scheduled across the parallel units in an order that is not fixed, so floating-point results accumulate in whatever order they arrive. A kernel library such as cuDNN picks the routine that suits the shapes it sees, so the same operation can run different code from one occasion to the next. A model that samples adds variation of its own.

Weights, configuration, and code are necessary to reproduce a run, but they are not sufficient on their own. None of this variation is caused by a checkpoint, and none of it is removed by one, so it is there whether or not the job ever moves.

A number that shifted between two runs of the same code is not, on its own, evidence that the move did anything to it. The comparison in How to know a restored GPU workload is correct is how to tell ordinary variance from an effect of the move.

A job that resumed at all resumed on matching versions, because anything else would have cold-started. The hardware model and software versions on the far side of the move match the ones the run started on, so the variance you are looking at is the variance you would have seen if the job had never left.

A checkpoint is a record of the run at one moment

A stored checkpoint holds the state itself rather than a description of it, so the moment it was taken can be recovered from, rerun, or compared against another moment later, and a run can be rolled back to any step of a training, evaluation, or inference workflow.

Those capabilities are properties of a stored checkpoint. They are not a catalog of past runs an operator can browse today. For a researcher, that means the state that produced a result still exists wherever a checkpoint was taken, instead of having to be rebuilt later from the weights, the configuration, and the code.

Recovery, comparison and rollback all follow from saving the state itself, so a run can be continued instead of repeated. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We save the whole running workload rather than the model file, so a job that has to leave its node carries on somewhere else instead of starting over. The same holds for an inference session that has to come back as the session it was and for a coding task interrupted in the middle of its work, so if your job is moved and comes back, it is the run you started that carries on.

Common questions

Does a restored job give the same result as an uninterrupted one?

The saved computation continues from the checkpoint, so nothing computed before it is derived a second time, and the run carries on with the same GPU model, driver, engine and model version it started on. There are two exceptions: a workload reading real-time data comes back to a feed that moved on, and a checkpoint whose versions no longer match cold-starts instead of restoring. The ordinary run-to-run variation of GPU arithmetic is there whether or not the job ever moved.