Turn idle GPUs into capacity. Automatically.

The numbers below are not simulated or projected; they come from a single continuous 24.6-hour run on a real Slurm cluster we stood up on AWS EC2 with 8 NVIDIA A10 GPUs spread across multiple nodes. The experiment modeled the standard scavenger-queue problem: a set of high-priority jobs arrives on an unpredictable schedule and must get GPUs immediately, while a second queue of lower-priority “scavenger” jobs is allowed to run only on capacity the priority jobs are not using. In a conventional Slurm setup, scavenger jobs are killed on preemption and lose all progress. In this run, Cedana checkpointed each scavenger workload at the moment Slurm preempted it, and Slurm requeued and restored it, often on a different node, as soon as idle capacity reappeared. Over the run, priority jobs consumed 34% of the cluster’s total GPU time, leaving 66% idle. The eight scavenger workloads were preempted 588 times in total, and all eight progressed through every preemption and ran to completion. Of the idle GPU time, 87% was converted into kept, useful scavenger work (91% during windows when jobs were actively waiting), with 8% consumed by checkpoint and restore overhead. The median handover, measured from a priority job’s submission to its first CUDA kernel executing on the GPU it preempted, was 11.4 seconds, and 265 of the restores landed on a different node than the one the job was checkpointed from.