TL;DR
- Your GPUs sit reserved and mostly idle while other teams wait in the queue, so your GPU utilization stays well below what you are paying for.
- Every scheduler on this page has its own name for lending that capacity out, borrowing, backfill, over quota, or a preempt partition. Each comes with a way to guarantee the paying account a floor while the rest gets lent out.
- The catch is how the loan ends. Almost every one of them takes the GPU back by ending the job that borrowed it. Slurm can suspend the job instead, which holds the node, and Google's Flex Start never reclaims, so the wait for capacity is the price.
- So the borrower turns the offer down rather than risk the run, and the idle GPUs stay idle while both sides behave sensibly. A borrowed job that was checkpointed while it ran breaks that stalemate. The reclaim then costs the time since the last checkpoint plus a restore rather than the whole run.
- In this piece we walk through what administrators ask for, how each scheduler lends a GPU and takes it back, and how Runway runs the trade on its own fleet. Then we cover why a reclaim has to end the borrowed job, what that costs the borrower, what the cheaper capacity is worth, and what changes when the borrowed job can be saved instead of killed.
What admins ask for when their GPUs sit idle
Idle GPU harvesting is the practice of lending unused GPU capacity to work that is waiting, and the administrators asking for it want two halves of the same arrangement: a guaranteed share for the account that paid, and the run of whatever the other accounts are not using.
A commenter on Hacker News asks why a company with a private cluster would not put a workload scheduler on it and let validated customer partners push batch jobs to the cluster. An administrator on the Slurm users list wants both halves written into policy, "guaranteeing availability of GPU resources to a Slurm account, while allowing this account to use other available GPU resources as well". Another on the same list, asking for a per-user cap, describes what a cluster looks like with no lending at all. "Person A submits 100 jobs, 2 would run, and 98 would be pending, 38 gpus would be idle."
People also ask for a job that grows into capacity as it frees up. On the Kueue tracker, a user describes two workloads of two pods each, one GPU per pod, and says that "When one of the workload completed, the other workload should have 4 GPUs to use". Another wants jobs that start with whatever GPU capacity is currently free and grow automatically as more becomes available. The first two asks describe one arrangement: a floor for the account that paid, the rest lent out while idle, and a handback when the owner returns. Growing a running job into more GPUs is a different ask, and no scheduler here grants it.
How each scheduler lends an idle GPU and takes it back
Every project below documents a lending policy, called fair share, borrowing, or backfill, and all but Google's Flex Start document a reclaim policy, as of September 2026.
| Tool | How it lends idle capacity | How it takes it back | What the borrower loses | In its own words |
|---|---|---|---|---|
| Kueue | A ClusterQueue with no spare quota draws on its cohort | The reclaimWithinCohort policy | The admitted workload | "Preemption is the process of evicting one or more admitted Workloads to accommodate another Workload." |
| Volcano | A queue runs above its deserved value while others sit below theirs | The reclaim action | The evicted tasks' work | "this queue must return the borrowed resources until its usage matches its Deserved value" |
| NVIDIA KAI Scheduler | A queue runs over quota on what another queue is not using | Reclaim, step 3 of the cycle | The running process | "Reclaim, Inter-queue resource recovery. Evicts over-quota preemptible workloads from other queues." |
| NVIDIA Run:ai | Projects take a node pool's unused resources beyond their quota | Rank-based reclaim, written in its docs as "reclaim (preempt)" | The preempted workload | "Over quota resources can only be used by preemptible workloads." |
| Slurm | Backfill fills gaps with lower priority jobs, a low PriorityTier partition holds the borrowers, and a MAGNETIC reservation with the flex flag keeps a floor for one account | Suspend, requeue, or cancel, set per partition | Requeue and cancel lose the work, suspend holds the node | "jobs in higher PriorityTier partitions will suspend jobs in lower PriorityTier partitions to run on the released resources" |
| Pitt CRC's preempt partition | Any account may run on idle resources at no charge | Preemption when the owner returns | The job | "use idle resources at no Service Unit cost. In exchange, the jobs are preemptible" |
| Google Cloud Dynamic Workload Scheduler | Flex Start holds a capacity request and provisions once capacity appears | No reclaim, the allocation runs for the duration requested | Nothing, and the wait is the price | "you can simply terminate the VMs to free up the resources and only pay for what your workload actually consumed" |
| Cedana | Not a lender, the scheduler above still lends | Not a reclaim mechanism | The interval since the last checkpoint, plus the restore | Not a scheduler. Nothing here calls us, so an operator or an agent starts the move. |
Filling an idle GPU does not count against you later. NVIDIA's January 2026 post on balanced GPU allocation in Kubernetes clusters states the accounting rule as "Usage is measured against cluster capacity, not against what others consumed". None of these projects settles how to hand the GPU back without losing the borrowed job's work.
Runway lends its inference GPUs to research overnight
Runway lends its inference GPUs to its research teams and has written the arrangement up twice. Its post of 27 April 2026, "No Idle GPUs: Managing Research Compute at Runway", gives the lending rule as "Default can borrow the 64 idle GPUs and run batch evaluations, experiments and other preemptible workloads", and a second post of 2 July 2026 puts the whole trade in its title, "Borrowing the Night: Reclaiming Idle Inference GPUs for Research".
Evaluations and experiments are the work that goes into the borrowing lane because losing one costs nobody an answer they were waiting for.
Why taking a GPU back means ending a job
A scheduler ends the borrowed job because a GPU cannot be handed back in pieces. Google's Borg paper, from EuroSys 2015, sorts a machine's resources into two kinds. CPU cycles and disk input and output bandwidth are compressible, which the paper defines as rate-based and reclaimable from a task by decreasing its quality of service without killing it. Memory and disk space are non-compressible, and those, the paper says, generally cannot be reclaimed without killing the task.
Borg was harvesting idle capacity on that basis a decade ago, running batch jobs in resources reclaimed from service jobs. The paper never mentions GPUs or accelerators. A GPU is allocated as one non-compressible unit, so a scheduler cannot claw back a share of it by slowing the borrower down. It can only take the whole device, which means ending the process that holds it.
The borrower declines the bet, so the idle GPUs stay idle
Borrowing is therefore a bet: you get GPUs sooner, and you lose everything computed so far if the owner comes back first. Few jobs are built to checkpoint, so most borrowers decline the bet, and the hours they could have used go unused. The owner's GPUs stay reserved and idle under the owner's name while other groups wait in the queue.
Losing that bet is charged twice, because Slurm charges an account for the allocation it held whether the job finished or was killed at a boundary. A job preempted late in its run and requeued from the top pays a second time for the computation it repeats. The fair-share factor, Slurm's own term for that account's standing in the queue, is computed from those same charges, so the group that was preempted waits longer for its next turn.
The same budget buys 2.5 to 10 times the GPU-hours
Cheap capacity is what makes the bet worth taking. In our experience at Cedana, spot runs 60 to 90% below on-demand, and where it lands inside that range depends on the pool and the region. At a 60% discount, the same dollar buys two and a half times as many GPU-hours. At an 80% discount it buys 5 times as many, and at 90% it buys 10 times as many. On a cluster you own, the cheapest capacity is the hour when nothing is running on it, and it is paid for either way.
The range is our own estimate and the multiples are arithmetic, not measured savings. None of it holds unless the jobs survive the interruptions, because a job that restarts pays the recompute after every one, and enough interruptions cancel the discount out. A job that is interrupted often finishes later in wall-clock time than the same job on reserved capacity.
The published saving we have is a spot result on a customer's cloud capacity, and idle windows on an owned cluster were not part of that measurement.
The kill becomes a pause when the filler can be checkpointed
A borrowed job that has been saved does not have to be killed to give the GPU back. A checkpoint is a saved copy of the full state of a running job, complete enough to bring the job back later, and taken below the application, at the operating-system level, it needs no checkpoint code in the job itself. Cedana takes one on a timer, so a recent copy always exists, and how often that happens is a policy the operator sets.
On a cluster configured for requeue preemption, the save fits inside the window the scheduler already provides. The owner's job arrives, the scheduler selects the borrowed job for preemption, and the job is checkpointed during the grace time. The hardware goes back to the owner on the schedule the grace time already sets. When the requeued job is scheduled again, it resumes from that checkpoint and keeps what it had computed.
The one number to get right is the grace time, because the window has to be long enough to write the checkpoint. If one cannot be written before the job is killed, the job comes back from its last periodic checkpoint, and starts fresh only if the policy never took one. So the grace time gets sized for the largest job in the lane, and periodic checkpoints set by policy are the fallback.
What the borrowed job still pays
The scheduler still decides who borrows and when the owner takes the GPU back, because we add no queue, no priority tier, and no quota. Fair share still orders the queue, and the owner's tier still wins.
What the borrower still pays is a restore plus whatever it computed after its last checkpoint. On one node of 8x B200 GPUs, Kimi K2.6, a trillion-parameter model with a 670 GiB checkpoint, restored in 63 seconds, on hardware where the native start took 20.3 minutes, in our published benchmark. Those runs kept their checkpoints in tmpfs, a file system that lives in the node's own memory, so the times include no fetch from disk or across a network.
The restore cannot go to any free node. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
A job on the 8 GPUs of one node can be saved there and restored on the 8 GPUs of another, because at no point was it running on two machines at once. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
We do not decide where a job goes, and no scheduler on this page calls us today. We document an automatic checkpoint and restore path for Slurm and one for Kubernetes, so when the node under a workload is taken away, the workload restores on the new one instead of starting from scratch. Beyond those paths, an operator or an agent watching the queue decides when a job moves.
The GPU still goes back to the owner on the owner's schedule, and what changes is what the handback costs the borrower. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. For a borrowing lane, that means a job can use capacity the owner is allowed to take back, because a reclaim costs the time since the last checkpoint rather than the whole run. Whether that is worth setting up depends on how much of your fleet sits reserved and idle, and on the grace time on that partition.
Related:
- How much of your GPU utilization was work you kept?
- Why an idle notebook keeps its GPU until you kill it
- How long do agentic sessions run, and where does their state live?
- When the node dies under a Jupyter session
- Sharing GPUs without fixed MIG slices
- Slurm has no suspend option for GPU jobs. What preemption without killing the job looks like
Common questions
Is there a way to close the gap when GPUs sit reserved and mostly idle, instead of accepting the waste?
Yes. Every scheduler in the table above can lend idle capacity to waiting work, whether it calls that borrowing, backfill, over quota, or a preempt partition, and all but Google's Flex Start take the capacity back when the owner needs it. The catch is that all but two of them take the GPU back by ending the job that borrowed it, so the idle hours only get used if the borrower can survive the reclaim, or if the borrowed job is checkpointed first.
How do I guarantee my team a minimum share of GPU capacity on a shared cluster while still letting other teams use it when it's idle?
On Slurm, a MAGNETIC reservation with the flex flag keeps a floor for one account, while backfill and other accounts' unused allocations lend out whatever sits idle beyond that floor. An administrator on the Slurm users list asked for exactly that pairing, guaranteeing availability of GPU resources to an account while letting that account use other available resources too.
