More science from the GPUs
you already paid for.
Your code already checkpoints. Just not when it matters.
Every one of these codes writes a restart file. The catch is granularity and timing: too coarse, too rare, or only at the very end. Ordered by how much of a research cluster each field actually runs.
DFT & Ab Initio
The workhorse of materials science and chemistry on every campus cluster. The plane-wave wavefunction and charge density live in GPU memory across the SCF loop.
- Snapshot the live wavefunction and charge density mid-SCF, which VASP writes to WAVECAR only after the loop converges
- A maintenance drain becomes checkpoint-and-resume, not a restart from the last completed ionic step
- For multi-node HSE and GW jobs, coordinated checkpoint captures every MPI rank and its in-flight CUDA-aware MPI transfers consistently
Six ways a shared cluster loses the work.
The restart file is only half the story. The cluster takes the GPU between writes, and device memory was never in it.
A drain takes the node mid-run
Maintenance reserves the node and kills every job on it. A VASP relaxation mid-SCF drops to its last ionic step, or to zero. Researchers stop submitting days ahead to dodge the window.
One bad GPU takes the whole job down
Every rank syncs behind each collective, so one bad GPU, NVLink, or NIC kills the whole job. Meta's Llama 3 405B run logged 419 unexpected interruptions over 54 days on 16,384 H100s, about one every three hours.
Packed scheduler, idle silicon
To insure against interruption, researchers oversize walltime and hoard spare GPUs. Slurm shows a full cluster while nvidia-smi reads 30-40% SM utilization.
Topology strands whole GPUs
Jobs want contiguous ranks in one NVLink domain, but odd shapes never fit the free 8-GPU islands. The scheduler idles nodes to assemble a slot, stranding 21-42% of the fleet.
Preempted with 80 GB still on the device
An owning lab's job lands: SIGTERM, about 30 seconds, then SIGKILL. A full dump of 80 GB of HBM to shared storage rarely finishes in that window, so the borrowed run dies instead of pausing. Owners stop lending.
A spot reclaim wipes the in-flight run
Spot and cloud-burst run up to 90% cheaper, but a reclaim takes the GPU mid-step on the same brief grace window. One eviction erases a multi-day run, so real science stays on-prem.
Six ways a shared cluster wins the work back.
Each gain closes one of the six losses above — earned by a specific capability, not a slogan, and nothing for researchers to change.
Maintenance windows stop killing jobs
A failed node costs a minute, not days
No reason left to over-request “insurance” GPUs
Defragment the queue, shrink the wait
Condo sharing becomes policy, not politics
Spot & cloud-burst safe for week-long runs
Your run survives the GPU underneath it.
Cedana runs at the OS and kernel level and works transparently with Slurm or Kubernetes, keeping every run moving across the GPUs underneath it.
Running
A run lives in GPU memory for days.
Save
Cedana snapshots full GPU state, below the app.
Migrate
A fault or drain triggers a live migration.
Resume
It resumes at the same step. Nothing lost.
vMotion for GPU workloads. Researchers never touch it; their runs just stop dying.
Runs everywhere your GPUs already live.
Questions we hear first.
sbatch, srun, and squeue stay exactly as they are. It installs as a bootstrap script plus a plugin, typically in under an hour, transparent at the OS and kernel level. See the installation instructions →Command your
compute.
Your compute, liquid — checkpoint, migrate, and resume live GPU jobs across the fleet.