What would a GPU scheduler do differently if it could move running jobs?

Every scheduler decides where a job starts, and none can move it afterwards

Whichever scheduler runs your cluster, it decides where a job starts and then leaves it there. Volcano, Apache YuniKorn, NVIDIA KAI Scheduler, NVIDIA Run:ai, and Slurm each pick the node a job lands on, while Kueue decides when a job is admitted and leaves the node to the scheduler beneath it. None of them can move a job once it is running, so each works around that limit with its own rule instead: evicting the job, gang-scheduling the whole group at once, or suspending it in place. Each rule costs the running job something different, and the table sets out what each one costs, quoting each project's own documentation as of September 2026.

Gang scheduling is the half that is settled

Gang scheduling means a group of pods starts together or not at all. If your job needs 8 GPUs across 8 Pods and only 5 GPUs are free, starting those 5 Pods leaves them holding hardware while they wait for the remaining 3, which is how partial allocation strands resources. Run several such jobs at once and Kubernetes ends up scheduling one worker from each job, so none of them ever reaches its full set.

Kubernetes now does gang scheduling itself. Since Kubernetes 1.35, the native feature "ensures that a group of Pods are scheduled on an 'all-or-nothing' basis".

The earlier queues and plugins are still widely used, and Kueue's answer is to evict the group and requeue it. YuniKorn either fails the group or scatters it, and the coscheduling plugin keeps the whole group waiting, which in its own demo leaves "all nginx pods ... in pending state".

A group that never started holds no state to lose, so getting it started is a scheduling problem and not a checkpointing one. Everything below is about the jobs that did start.

What do Kueue, Volcano, and DRA each control?

Kueue decides whether and when a job is admitted, and Volcano is the batch scheduler that places the admitted pods, so a job passes Kueue's admission gate first and Volcano's placement decision second.

Dynamic Resource Allocation, the third of the three and stable since v1.35, is how Kubernetes requests and allocates devices. Its published limitations settle what happens to a job that wants a busy device. "The Kubernetes scheduler doesn't support preemption for DRA resources." The waiting pod stays pending "until the device becomes available, which happens when the conflicting Pod terminates or is manually deleted", so the wait ends when the job holding the device ends, and not before.

Ray behaves the same way with a group of bundles, because "If a bundle can't fit in any of the current nodes, Ray reserves no resources for the placement group". All of them decide the first placement, and once a pod is bound to its node the only move left is an eviction.

Packing is settled the moment a job is placed

Free GPUs scatter as jobs come and go, so your cluster can be busy and unable to start anything large. A large model replica needs several GPUs sitting together, and as jobs finish at different times the free ones end up spread across nodes that are already partly occupied.

On the production cluster that Weng and colleagues call cluster H, 500 GPUs sat unallocated while the cluster was roughly 90% packed and could not take new tasks. They measured it for their USENIX ATC 2023 paper on GPU fragmentation.

Slurm has the same placement problem, and it tells you so in the job's pending reason: "The job is waiting for resources to become available". Its pending-job reason codes separate that case, hardware in the way, from priority and policy limits. Slurm's own view of the hardware is flat: it counts GPUs per node as a generic resource, and without its optional topology plugin it "considers nodes as a one-dimensional array".

What a stock scheduler can change is where the next job lands. A bin-packing policy fills one node before it uses another, and on a Kubernetes fleet running Volcano it took the count of nodes with all 4 GPUs free from 18 to 214, in numbers NVIDIA published on its developer blog. Before the change, 115 nodes had 3 free GPUs each, which a job needing 4 cannot use.

The policy leaves every running job where it already sits, so it improves the shape of the cluster from here on and does nothing about the fragmentation already in it. Users keep asking the scheduler projects for the other half: consolidate running pods onto the minimal set of hardware, including the pods holding fractions of a GPU, because inference and similar workloads leave a cluster jagged with no room for a full node.

Preemption makes you choose between the work and the hardware

Slurm publishes four preemption modes, and each one costs the losing job something different.

• OFF disables preemption.
• CANCEL ends the job, because "The preempted job will be cancelled".
• REQUEUE puts the job back in the queue, and "When a job is requeued, the batch script is initiated from its beginning".
• GANG suspends the job in place, which keeps the work.

GANG is the only one of the four that keeps the work, and it is the one that does not release the hardware. A suspended job holds the memory on its nodes, which is why Slurm has to account for memory to offer the mode at all, "because the suspended jobs will still use memory on the allocated nodes". The job holds the node it was meant to release, so the higher priority job gets no hardware out of it.

Kubernetes hands over the hardware the same way, by evicting lower priority pods so the new pod can be scheduled once they are gone. The evicted pods get a graceful termination period, which is time to shut down rather than a way to keep the work.

The time limit kills the job because the job has nowhere to continue

When an enforced wall-time limit expires, Slurm terminates the job and records it as a TIMEOUT. Termination is the only enforcement available, because a job that has run out of time has nowhere to continue.

The scheduler needs the limit to plan around a job's end time. Backfill can only slot a small job into a gap if it knows when the gap ends, and a maintenance window needs a time by which the cluster has drained.

The way out today is an exception granted by hand. The University of Utah's Center for High Performance Computing documents a long QOS, a quality of service setting that exceeds the maximum walltime limit, given case by case.

An administrator planning a reservation has no better option. One asked the Slurm users list how to stop long jobs from starting on a machine that is about to be reserved, because nothing takes a long job off the node later.

Everything in the serving stack moves except the worker

Inference platforms cannot move a running worker either, and not for want of moving parts elsewhere in the stack. Requests, key-value cache blocks and model weights all move between running instances today.

• Llumnix, at OSDI 2024, migrates requests between instances while they run.
• Mooncake Store, with vLLM, moves key-value cache blocks between GPU memory, host memory, and other nodes.
• Tensor R-Fork, in SGLang, reads weights out of a running instance into a new one.

Every one of them leaves the worker where it is. A serving worker is one model instance, and it holds the weights, runtime state, captured CUDA graphs, and key-value cache for every request in flight, all built on the GPUs it sits on. So the scheduler above it has the same two options as a batch scheduler, place the worker or kill it.

Each rule changes when a running job can move

Packing, preemption and wall-time enforcement are all built around a job that cannot be moved, so all three change once a running job's full state can be saved and brought back on other hardware. Nobody has published a comparison of cluster packing with and without the ability to move running jobs, so what follows describes what changes, not measured results.

Packing stops being a decision taken once. The small jobs holding several half-empty nodes are checkpointed and restored onto fewer nodes, and the job that needs 8 GPUs starts on a node they free. Repacking happens as jobs finish or demand changes.

Preemption stops being a choice between the work and the hardware. The hardware goes to the higher priority job, and the lower priority job comes back from its last checkpoint when capacity returns.

The wall-time limit stays where your administrator set it, but reaching it triggers a different response: checkpoint the process as the limit approaches, release the allocation, requeue the job, and restore it on a compatible node. That path is what we build at Cedana, and it means a week-long simulation under a 24-hour limit still finishes, in 24-hour allocations that carry its state forward.

Something outside the scheduler has to start the move

No scheduler decides on its own to checkpoint a running job and migrate it today, and none of the ones in the table calls us, so something above them has to make that call.

A control layer outside the scheduler runs the checkpoint and the restore. We document an automatic checkpoint and restore path for Slurm and one for Kubernetes, so when the node under a workload is taken away, the workload restores on the new one instead of starting from scratch. Beyond those two paths, the decision is yours: an operator or an agent watching the queue says when a job moves.

A move costs a restore plus the work since the last checkpoint

Checkpoints are taken on an interval you set, so a move costs the job the restore plus whatever it did after the last one. In our published benchmark, on a single node with 8 B200 GPUs, Kimi K2.6, a trillion-parameter model with a 670 GiB checkpoint, resumed in 63 seconds, where starting the same model from scratch took 20.3 minutes.

Both times were measured to ready-to-serve on a node with 1.7 TB of system memory, and the checkpoint was read from tmpfs, a file system that lives in the node's memory, so the restore does not include a fetch from disk or across a network.

The destination cannot be whichever node is free. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

A job on the 8 GPUs of one node can be saved there and restored on the 8 GPUs of another, because at no point was it running on two machines at once. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

Every rule in the table, evicting the group, failing it, suspending it in place, or letting the clock end the job, exists because a running job cannot be moved. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a cluster, that means the job in the way can be set down and picked up on other hardware instead of being thrown away. Which rule is worth changing first depends on your own partition limits, priority tiers, and packing policy.

Related:

• Why a cluster with free GPUs still cannot place an 8-GPU job
• Why a healthy inference worker can still be in the wrong place
• Does disaggregated serving remove the need to move GPU workers?
• H100 for prefill, H200 for decode: the mismatch you pay for twice
• Sharing GPUs without fixed MIG slices
• Slurm has no suspend option for GPU jobs. What preemption without killing the job looks like

Common questions

My job needs several workers placed together at once. Why does the scheduler seat only part of the group, and how do I stop that partial-seating deadlock?

Gang scheduling means a group of pods starts together or not at all, and without it the scheduler seats whatever happens to be free. If your job needs 8 GPUs across 8 Pods and only 5 GPUs are free, those 5 Pods start and then wait for the remaining 3, which is a deadlock rather than a running job. Kubernetes now does this itself, scheduling a group of Pods on an all-or-nothing basis natively since Kubernetes 1.35, while the earlier queues and plugins still in wide use evict and requeue the group, fail it, or leave it waiting.

Can already-running jobs be consolidated onto fewer nodes so a whole node frees up, instead of staying scattered until something finishes on its own?

Not by the scheduler itself: no scheduler named on this page consolidates running jobs today. With checkpoint and restore, and an operator or an agent starting the moves, packing stops being a decision taken once, because the small jobs holding several half-empty nodes are checkpointed and restored onto fewer nodes, and a job that needs more GPUs than any single node has free starts on a node they free.

Why does my cluster fail to place a job that needs several GPUs together, even though the total free GPU count looks like enough?

Free GPUs scatter as jobs come and go. A large model replica needs several GPUs sitting together, and the free ones end up spread across nodes that are already partly occupied, so the total looks like enough while no single node can host the job. A stock scheduler can only change where the next job lands, filling one node before using another, and that leaves every running job where it already sits.

I know a job will take a long time to run. How do I keep it from starting on a machine that is about to be reserved?

Slurm will not start a new job whose declared time limit runs past the start of a reservation, so new jobs are held automatically. The job to worry about is the one already running when the reservation is placed, which nothing today takes off the node. With that job checkpointed, it can be moved off before the reservation starts and restored afterwards.