TL;DR
- For over 20 years, academic HPC centers have run condo nodes. PIs buy nodes, and their groups get a dedicated partition.
- To broaden access to computing and to use idle capacity, centers open condo nodes to other researchers through a scavenger queue. When an owner submits a job, the scavenger job on that node can be cancelled, and it loses its work.
- Cedana saves the preempted job's work, lets the owner run, and resumes the job when resources free up. That cuts lost work and raises batch throughput, which on GPU nodes also means less power and cooling spent on work that gets thrown away. A fully automated solution that works across inference, training, and HPC workloads.
Condo nodes and scavenger queues
Condo nodes have been part of academic HPC for over 20 years. A PI buys nodes, usually with grant or startup money. The HPC center installs them in the shared cluster and often gives the research group its own queue, a dedicated partition the group submits to.
On their own, condo nodes serve only the groups that can buy them. Researchers without funding for their own hardware rely on the general queue. The owners also don't use their nodes all the time, so capacity that someone paid for sits idle.
The usual answer is a scavenger queue. Condo nodes are made available to others, often to everyone on campus, through a separate partition that runs on the owners' idle hardware. UC Berkeley's Savio states the trade on its condo page: "Operating costs for managing and housing PI-owned compute nodes are waived in exchange for letting other users make use of any idle compute cycles on the PI-owned nodes."
The owner keeps priority, so scavenger jobs live under one of two rules, and often both. They have short maximum walltimes, or they can be preempted. Preemption means that when a condo owner submits a job to one of their nodes, the scavenger job running on that node can be cancelled.
What preemption costs
A cancelled scavenger job loses everything it computed since its last save. Most researchers do not implement checkpointing, and if they do it is application-specific code they have to implement. Administrators can’t rely on workloads being able to save their progress.
If the center requeues the job instead of cancelling it, Slurm starts the batch script over from the beginning. Slurm's default grace time, the warning a job gets before it is killed, is zero seconds.
Centers say this plainly and put the fix on the user. Purdue's guide to its preemptible GPU queue tells users "it is up to you to implement checkpointing for your workflow." Boston College gives preempted jobs "a 5 minute grace time" and notes that "Some software cannot use checkpointing directly, requiring custom wrapper scripts." Duke tells users of its scavenger H200 partition that jobs "will be cancelled and requeued" and to "make sure you check-point."
So the scavenger queue offers capacity on terms that only some jobs can accept. A job that writes its own checkpoints, or finishes inside a short walltime, does fine. A long job whose code can't save itself on a few minutes' notice either stays out or gambles on the owner not coming back. When it loses, it runs again from the start.
Short walltimes avoid the loss by keeping long jobs out of the scavenger queue altogether. Preemption lets long jobs in and then throws their work away. Either way, idle capacity goes unused or gets used twice or more for the same result.
Why it matters
The whole point of condo nodes and scavenger queues is getting more science out of what a university already owns: more science per dollar, per unit of time and per watt. Lost work cuts into all three.
Every preempted job that restarts from the top burns node-hours that produce nothing new. The scavenger queue then delivers less useful work than its utilization numbers suggest, and the borrower's result arrives later.
On GPU nodes the waste is larger, because GPU nodes cost more and draw far more power. NVIDIA lists a maximum system power of 10.2 kW for an 8-GPU DGX H100. At that maximum, a node like that uses about 120 kWh in 12 hours, before cooling. If the job loses its work, all of that energy bought nothing, and the job spends it again when it reruns.
At NVIDIA's 10.2 kW maximum system power, before cooling. A job that loses its work spends it again when it reruns.
What Cedana changes
On a Slurm cluster, the owner's priority doesn't change. When the owner's job arrives, Slurm selects the scavenger job for preemption as it does today. Cedana checkpoints it to storage inside the grace time and Slurm requeues it. When the job starts again on a node with the same GPU model, Cedana restores it from that checkpoint. The job carries on from where it stopped instead of starting over.
The checkpoint is taken below the application, so the job needs no checkpoint code of its own. That matters for a scavenger queue, because it lets supported jobs that can't save themselves use the queue, the ones centers currently tell to stay out or take their chances.
The result is less lost work and higher batch throughput. Scavenger hours that used to be thrown away become finished work, and long jobs that stayed out of the scavenger queue can use it. See actual results from Cedana’s automated scavenger queue for Slurm here.
INTERACTIVE REPLAYScavenger Queue - SLURMReplay a real 24-hour Slurm run where Cedana turns idle GPU time into finished batch work.
What a center has to set
Two settings matter. The scavenger partition runs PreemptMode=REQUEUE, so preempted jobs go back in the queue rather than being cancelled. Its GraceTime has to be long enough to write a checkpoint, and the default of zero isn't. Size the grace time for the worst case, when one owner job preempts several scavenger jobs on the same node and their checkpoints all hit storage at once. Time a real checkpoint on your own storage before you pick a number.
The owner waits no longer than the grace time the center sets. What changes is what happens to the job that was there.
Where it won't help
- Short jobs. If a job finishes in minutes, rerunning it costs less than saving it.
- State too large for the grace time. If a job's memory can't reach storage within the grace time owners will accept, it belongs on reserved capacity.
- Jobs that span nodes. Cedana's checkpoint covers one GPU or several GPUs in one node today.
The idle hours are already paid for
Condo nodes and scavenger queues are how universities share hardware that individual groups paid for. Preemption is the price of that sharing, and today the price falls entirely on the scavenger job, which loses its work. Saving that work lets the owner run on schedule, lets the borrower finish, and turns idle hours the university has already paid for into science.
Related:
