TL;DR
- Slurm preemption on a GPU cluster ends the job. Slurm can suspend a job, but a suspended job keeps its GPUs. So the two modes that do free the card, REQUEUE and CANCEL, free it by ending the running process.
- The policies administrators reach for instead, a scavenger QOS, PreemptExemptTime, tiered access, change how much the kill costs and who is surprised by it. None of them keeps the job.
- The long QOS is the other lever, and it makes completion times less predictable for the scheduler. So the jobs queued behind the grant wait longer for a cost nobody is billed for.
- What changes the answer is a checkpoint taken below the application. The owner still gets the card on the schedule the policy already sets, and the preempted job comes back to the point it had reached.
- In this piece we walk through what each PreemptMode value does, why a suspended job keeps its GPU, and what other administrators are doing about it. Then we cover why the other allocation policies also end in a kill, why granting a long QOS makes the problem worse, and what changes when a preempted job can be checkpointed instead.
Can Slurm suspend a GPU job instead of killing it?
Slurm can suspend a job, but it cannot free the GPU that way, so preemption on a GPU partition ends up killing the job. A cluster administrator wrote to the Slurm users list in June 2026 about avoiding killed GPU jobs during preemption, and the one reply the thread got offered no way to keep the job. The administrator put it in a sentence:
"there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available."
Preemption behavior on a Slurm cluster is set by PreemptMode, which can be set cluster wide, on a partition, or on a QOS, the quality of service setting that names the class a job is submitted under. SchedMD's preemption page lists four values for PreemptMode: SUSPEND, REQUEUE, CANCEL, and GANG.
SUSPEND and REQUEUE are the two values that say what happens to the displaced job, and the difference between them is the whole problem. Under SUSPEND, "The preempted jobs will be suspended, and later the Gang scheduler will resume them." Under REQUEUE, Slurm "Preempts jobs by requeuing them (if possible) or canceling them. For jobs to be requeued they must have the '--requeue' sbatch option set or the cluster wide JobRequeue parameter in slurm.conf must be set to 1."
On a first reading, SUSPEND is the mode this administrator wants, because the job stops, it keeps what it has done, and the scheduler brings it back when the preempting job is finished. But a suspended job holds on to its GPUs, so the card the preempting job came for never comes free.
What happens to the GPUs when a job is suspended?
The suspended job keeps its GPUs. A GPU is configured in Slurm as a generic resource, which the scheduler calls a GRES, and Slurm's configuration reference states the rule without qualification: "Suspended jobs will not release GRES. Higher priority jobs will not be able to preempt to gain access to GRES."
That leaves REQUEUE and CANCEL, the two modes that do free the card, and both of them end the running process. A preempted job can be requeued if it was submitted with --requeue, and the University of Utah's preemption page says that when it comes back "the job will start from the beginning".
So the administrator's sentence was accurate. Suspend keeps the job and holds the GPU, requeue and cancel free the GPU by ending the job, and no value in the list both keeps the job and frees the GPU.
What are other administrators doing about it?
The one reply came on 29 June 2026, and it was about expectations rather than mechanism. Suspend works for CPU and memory, so a site can use it where no GPU is involved. For the GPU case the reply suggested three policies:
- Run the borrowed work under a scavenger QOS with shorter runtimes, so a job that gets killed loses less.
- Use PreemptExemptTime, which protects a job from preemption for a set period after it starts. It is not GraceTime, the notice a job gets after selection.
- Give groups tiered QOS access, so the people most likely to be preempted know that in advance.
These policies reduce the impact of preemption without preserving the running job. They are sensible, but a site running all three still kills the GPU job. They change how much the kill costs and who is surprised by it.
Administrators ask about both of those settings on the Slurm users list, and what comes back is a delay rather than a pause. One asker wants the wait to be long: "Does anyone have a working example using PreemptExemptTime? My goal is to make a higher priority job wait 24 hours before actually preempting a lower priority job." The answer on record is PreemptExemptTime together with PreemptMode set to CANCEL, which delays the kill and then kills.
GraceTime is the other half of the pair, the notice a job gets after it has been selected and before it is killed. Neither setting pauses the job, so both of them only say how long it has left, which is the window a checkpoint has to fit inside.
Why do the other policies end with a job being killed too?
Preemption is not the only policy that ends a running job. Three policies decide who gets the GPUs on a shared research cluster: fair share orders the jobs that are waiting, and the other two, the runtime ceiling and owner preemption, are enforced by ending a job that is already running.
Many research clusters run on a condo model, where a lab buys GPUs out of its grant, the hardware goes into the cluster, and the lab keeps priority access to what it paid for. Owners do not keep those GPUs busy every hour, so the center lends the spare hours out through a requeue partition, a lane where any account may run on hardware it does not own. When the owner's job arrives, Slurm preempts whatever is running in that lane.
Fair share settles which pending job runs next. Slurm computes a fair-share factor from the gap between the share of the machine an account was promised and what its jobs have already consumed, and the account furthest below its promised share goes first. It is not a quota, so running over your share puts you later in the queue rather than out of it.
The runtime ceiling does a different job from queue ordering: it is what makes an allocation return to the pool at all. Backfill scheduling, where the scheduler slots a short job into a gap while a bigger job waits for nodes, needs an accurate end time before it can do that. A node also cannot be drained for maintenance until the work on it stops, so a bounded runtime bounds that wait.
Owner preemption is the third policy, and it is how a lab gets its own hardware back. The owner's partition sits in a higher priority tier than the borrowing lane, so an owner's job displaces whatever is running there. The grace time is the interval between the moment a job is selected for preemption and the moment it is killed, and the administrator sets it on the partition or the QOS. Its default value is zero.
The borrower gets GPUs sooner but loses everything computed so far if the owner comes back first. Few jobs are built to checkpoint, so most borrowers decline the bet, and the hours they could have used go unused. The owner's GPUs then stay reserved and idle under the owner's name while other groups wait in the queue, and none of it shows up as a failure, because no job crashed and no limit was hit.
The account pays for the lost work as well. Slurm charges consumed resources to the account that ran the job, and an account is charged for the allocation it held whether the job finished or was killed at a boundary. A job preempted late in its run and requeued from the top pays a second time for the computation it repeats. The fair-share factor is then computed from those same charges, so the group that was preempted waits longer for its next turn.
Can Slurm enforce a soft time limit?
Slurm has no soft wall-time limit, meaning one that only ends a job when something else needs the hardware. A user on the Slurm list asked for one: "Is it possible to implement 'soft' wallclock limits in slurm, namely ones which would not be enforced unless necessary to run more jobs?" The nearest setting is PreemptExemptTime with PreemptMode set to CANCEL, which delays the kill and then kills. Owner preemption comes closer for the borrowing lane, because a borrowed job runs until the owner needs the hardware rather than until a clock runs out.
Checkpoint the borrowed job and you have the soft limit the asker wanted: the job runs for as long as nothing else needs the GPUs, and when something does, it is saved and requeued instead of killed.
Why does granting a long QOS make the problem worse?
The other lever an administrator reaches for is the exception, and it moves the cost rather than removing it. A long QOS is a standing exception to a cluster's maximum wall-time limit, handed out one request at a time, and Utah publishes the rule in a single line: "Special access is given to a long qos to exceed the MAX walltime limit on a case-by-case basis."
On Utah's general nodes, that limit is 72 hours, and the QOS there carries the time limit, so the standard QOS allows 3 days and the long QOS allows 14. Those are Utah's own published numbers and not a description of what centers do in general.
Granting one puts nothing on anyone's bill, which is why it is easy to say yes, but it makes job completion times less predictable for the scheduler. Backfill uses the time limits people declared to work out where the gaps in the schedule are, which is why Slurm's scheduling configuration guide says that "reasonably accurate time limits are important for backfill scheduling to work well". A job running under a 14-day limit hands the scheduler a completion time that can sit 14 days out, and every reservation behind it is projected against that endpoint.
The same problem shows up at ordinary scale, and the University of Maryland's HPC documentation warns its own users that requesting excessive amounts of wall-time "may prevent the job from being backfilled thereby causing significant delays in scheduling". Longer exception limits also keep nodes allocated for longer, so they return to the pool less often, and the longest grant on any node extends the worst-case maintenance window. The declared limit is never billed, so the whole cost lands as longer waits for the jobs behind the grant.
Utah caps the exception at two nodes running under the long QOS at any one time. One grant is a special case, and the case-by-case process exists for exactly that. When the requests keep arriving and more groups request exceptions for the same application each quarter, the problem is no longer confined to an individual user, because a whole class of workloads no longer fits the standard boundary. The cost accumulates without anyone deciding to take it on.
What changes if the preempted job can be checkpointed instead?
A checkpoint is a saved copy of the full state of a running job, complete enough to bring the job back later. Taken below the application, at the operating-system level, it needs no checkpoint code in the job itself, so it reaches the jobs in the borrowing lane that were never built to checkpoint.
That is what Cedana's Slurm integration does. A job is checkpointed automatically when it is preempted and resumed on its next run after Slurm requeues it, provided Slurm is configured to requeue the job. The high-priority job takes the node on whatever schedule the policy already sets, and the preempted job comes back to the point it had reached.
Checkpointing does not change PreemptMode, so the cluster still runs REQUEUE, the preempted job still gives up the GPU on the owner's schedule, and the owner still gets the card. What changes is the state the job is in when it comes back.
You have to make the grace-time window long enough to write the checkpoint. If one cannot be written before the job is killed and no earlier checkpoint exists, the job starts fresh, which is what happens when the grace time on a partition is too short for the size of the job running there. So you size the window for the largest job in the lane, and periodic checkpoints set by policy are the fallback, because they bound the loss from a missed checkpoint to the work done since the last one.
| Policy | What it guarantees | What the running job pays today | What changes when the job can be checkpointed |
|---|---|---|---|
| Fair share | Under-served accounts run first, and over-served accounts run when nothing else wants the machine | Nothing while it runs, because fair share orders the jobs that are waiting. The hours an account spends rerunning lost work still count against its share, so its next job waits longer | The queue order is untouched, and those rerun hours stop counting against the account's share |
| Runtime ceiling | Every allocation returns to the pool at its time limit | The job is terminated and recorded as TIMEOUT. Nothing requeues it, and it starts from the first line of its script whenever it is submitted again | The limit stays, and reaching it triggers a checkpoint and a requeue |
| Owner preemption | The owner's job takes the hardware back inside the grace time | Cancelled, requeued from the start, or suspended while still holding the GRES | Checkpointed inside the grace time, and resumed from that checkpoint when it next runs |
| Account charging | Consumed resources are charged to the account that ran the job | Charged for the allocation it held, finished or killed, and charged again for the rerun | Still charged for each allocation, but charged once for the computation |
Checkpointing also reduces requests for wall-time exceptions. A job that carries its state across allocations finishes a long computation in standard-length pieces, so on a cluster with a 24-hour limit a week-long simulation still completes, running in 24-hour allocations and carrying its state forward. The jobs in that class stop asking for the long QOS, and the wall-time limit stays where it is.
The same checkpoint covers the job that reaches its declared time limit rather than being preempted, which is the subject of Your Slurm job was cancelled due to time limit. What to do now.
Preemption on a GPU cluster is a choice between the owner's card and the borrower's work only for as long as the kill is the only way to free the card. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On Slurm we take the checkpoint below the application, inside the grace time you already set, so the owner gets the node back inside the same window and the borrowed job carries on in its next allocation. Every limit and priority tier stays exactly where you set it.
Related:
- Your Slurm job was cancelled due to time limit. What to do now
- DMTCP, application checkpoints, workflow managers and system-level checkpointing: what each covers on a Slurm cluster
- GROMACS shows both the value and the limit of application checkpointing
- Stateful AI has the same boundary problem as long HPC runs
- Measuring the cost of wall-time termination from your sacct data
- A 90-day wall-time PoC: what to measure before you change policy
Common questions
Can Slurm pause a preempted GPU job and resume it instead of killing and requeuing it from scratch?
No native Slurm mode both keeps the job and frees the GPU. SUSPEND keeps the job, but Slurm does not release GRES from a suspended job, and REQUEUE or CANCEL free the GPU by ending the process, which then starts from the beginning. With Cedana's Slurm integration the job comes back in a different state: it is checkpointed automatically when it is preempted and resumed on its next run after Slurm requeues it, while the GPU still goes to the higher-priority job on whatever schedule the policy already sets.
My job will lose all its progress if it's preempted or hits its time limit. How do I make it save its state so it can pick up where it left off?
A checkpoint is a saved copy of the full state of a running job, taken below the application at the operating-system level, so it needs no checkpoint code in the job itself. With Cedana's Slurm integration, a job is checkpointed automatically when it is preempted and resumed on its next run after Slurm requeues it, provided Slurm is configured to requeue the job.
When Slurm requeues my job, does it resume where it left off or restart the whole script?
A requeue restarts the whole script. A preempted job that was submitted with --requeue comes back from the queue and, in the words of the University of Utah's preemption page, "the job will start from the beginning". Checkpointing the job before it is killed changes that, because the preempted job then comes back to the point it had reached.
How do I get access to a queue that allows a longer maximum wall-time?
A long QOS is a standing exception to a cluster's maximum wall-time limit, handed out one request at a time. Utah's long QOS raises its 72-hour general limit to 14 days on a case-by-case basis, capped at two nodes running under it at any one time.
Can Slurm enforce a soft wall-time limit that only kills my job if the resources are needed elsewhere, instead of always killing it at a fixed time?
The runtime ceiling always kills a job at its declared time regardless of demand elsewhere, but owner preemption already works closer to a soft limit, because an owner's job displaces a borrowed job only when the owner's job arrives, not on a fixed schedule. The borrower still loses everything computed so far when the job is not checkpointed, so the softer trigger costs the same interrupted work unless the job can be checkpointed and resumed instead.
Can I make Slurm wait before preempting a lower-priority job, giving it a grace period instead of killing it the moment a higher-priority job shows up?
Grace time is the interval between the moment a job is selected for preemption and the moment it is killed, set by the administrator on the partition or the QOS, with a default value of zero. That window has to be long enough to write a checkpoint, or the job comes back from its last periodic checkpoint, which is why periodic checkpoints set by policy are the fallback that bounds the loss to the work done since the last one.
