Recently I was at PEARC talking with HPC and AI practitioners from research-computing centers across the country. One issue came up repeatedly, from both administrators and researchers. Jobs reach their wall-time limits and lose hours or days of in-memory progress. However, the limit itself is not the problem.
In Slurm, each job runs against a declared time limit. Partitions and QOS policies set defaults and maximums. When the enforced limit expires, Slurm terminates the job and records it as a TIMEOUT.
One group I spoke with uses a 24-hour maximum on a key partition. When that allocation expires, the scheduler terminates the workload whether it had just started or was 95% complete.
Those limits exist for good reasons.
Backfill needs bounded runtimes.
Slurm’s backfill scheduler starts lower-priority jobs when doing so will not delay higher-priority reservations. To make that decision, it needs an expected endpoint for every running and candidate job. Slurm’s Scheduling Configuration Guide notes that reasonably accurate time limits are important for effective backfill. Operators say the same. The University of Maryland’s HPC group notes that reasonable walltime estimates matter for backfill performance, and that excessive requests can prevent a job from being backfilled at all (HPC@UMD, Overview of Job Scheduling). Research on runtime estimation shows the same mechanism. Work on refining user walltime estimates for conservative backfilling, built on OpenPBS’s soft walltime feature, demonstrates how estimate accuracy affects wait times and scheduler efficiency.
Fairness needs turnover.
Shared clusters serve many users and research groups. Slurm’s fair-share policy helps determine which pending job should run next, but a runtime ceiling also ensures that current allocations eventually return their resources to the pool.
Operations need drain points.
Maintenance, firmware upgrades, hardware replacement, and node reconfiguration require administrators to drain nodes. Bounded runtimes place an upper limit on how long a clean drain should take.
Runaway jobs need a backstop.
Hung processes and infinite loops can otherwise consume an allocation indefinitely. The wall-time limit is the final safeguard.
Wall-time limits are part of how a shared cluster remains schedulable. The problem is what happens to the workload when the limit is reached.
Many important workloads legitimately run longer than one allocation. Molecular dynamics simulations, computational fluid dynamics, protein pipelines, parameter sweeps, model training. With AI this is long running stateful inference and fine-tuning jobs. All of these cross allocation boundaries.
Some mature applications handle this well. GROMACS writes its simulation state to a checkpoint file and resumes later, as described in its guide to managing long simulations. It ships a flag for this exact situation. With -maxh, mdrun terminates the run and writes a checkpoint at the first neighbor-search step after 99% of the specified time has elapsed, so the job stops itself just short of the wall time (gmx mdrun documentation).
But GROMACS also illustrates the underlying asymmetry. The application has to know which state matters, serialize it correctly, write it safely, expose a reliable restart path, and give the user a walltime-aware switch they must know to set. Every code has different state, checkpoint formats, intervals, scripts, and recovery behavior. One of the most mature codes in HPC needs a dedicated command-line option to survive a scheduler policy.
Application-level checkpointing can be efficient because the application knows exactly which data must be preserved. It can also require substantial engineering. The authors of CRAFT, a library designed to simplify application-level checkpoint and restart, describe that approach as efficient but implementation-intensive.
Researchers are left with a few options.
- Break the workload into application-specific stages.
- Configure or build a checkpoint-and-restart path for each application.
- Request access to a special long-running partition or QOS.
- Or restart the job from the beginning after a timeout.
This creates complexity as it has to be done by workload type and by researcher.
Administrators then maintain exception queues and long-running policies for workloads that do not fit cleanly inside standard limits. This is standard practice. The University of Utah’s CHPC documents a long QOS that exceeds the maximum walltime limit, granted case by case (CHPC General HPC Cluster Policies). Every such exception weakens some of the predictability the wall-time policy was meant to provide.
Slurm can requeue a job, but requeueing alone does not preserve the running process. As the sbatch documentation makes clear, a requeued batch script begins again from the start. Unless the application finds and loads a valid checkpoint, requeue means restart, not continue.
That is the missing primitive: a general way to carry the running workload into its next allocation.
Cedana brings transparent checkpoint, migrate, and resume to solve this tradeoff, supporting CPU and GPU workloads across training, inference, and HPC.
Cedana’s Slurm integration captures workload state below the application layer. As a job approaches its time limit, Cedana checkpoints the running process, releases the allocation, requeues the job, and restores it on a compatible node. No application code changes or application-specific checkpoint path are required.
The application continues from the captured point rather than beginning again. We have demonstrated the same underlying system-level checkpoint/restore mechanism with scientific workloads including GROMACS and Boltz-2, described in Supercharging Scientific Computing with Cedana.
On a cluster with a 24-hour limit, a week-long simulation still completes. It runs in 24-hour allocations, carrying its state forward from one allocation to the next.
The wall-time limit remains in place. Reaching it triggers a checkpoint and a requeue instead of a kill.
For administrators, this preserves everything short limits provide: useful bounds for backfill scheduling, predictable allocation turnover, bounded node-drain windows, and protection from runaway jobs. For researchers, the limit no longer means losing the running process or implementing a different checkpoint system for every application.
Checkpointing has always been an economic trade. The classic analyses of the optimal checkpoint interval, from Young’s first-order approximation (1974) through Daly’s higher-order estimate (2006), balance the time required to save state against recovery cost and the amount of computation at risk.
That trade changes when both checkpoint and resume are fast.
Cedana uses system-level checkpoints to capture the running workload directly. For workloads we have measured, Cedana completes the checkpoint roughly an order of magnitude faster than the existing application-level checkpoint path. Across the inference workloads in our published dataset, restores ran 9.9x to 41.1x faster than cold starts on the same hardware. On 8x B200 GPUs, a large model resumed in 63 seconds against a 20.3-minute native start, and a compiled FLUX pipeline resumed in 73 seconds against a 50-minute initialization.
Less time goes to writing state, reloading data and models, reconstructing memory, and repeating computation. The checkpoint starts reducing the total time and cost required to complete the workload, on top of insuring against interruption.
There is still real overhead. Checkpoint storage, I/O, transfer time, and compatibility requirements at the destination all cost something. Cedana optimizes these costs significantly, reducing the overhead to make a clear ROI: eliminating recomputing hours or days of work, out of long-running exception queues, and out of a separate restart path for every application. Up to 65% of compute time can be spent on recovering from failures according to research.
In many cases, a bounded checkpoint-and-resume cost is the better trade.
The first step is understanding how often the problem occurs and how much compute reaches the boundary.
Slurm’s sacct interface exposes job state, elapsed time, time limit, allocated resources, and TIMEOUT termination. From an anonymized period of accounting history, we can identify:
- How many allocations end in TIMEOUT.
- Which applications, users, and partitions experience repeated timeouts.
- How many node-hours or GPU-hours are attached to those allocations.
- How often similar jobs are resubmitted.
- Where long-running QOS exceptions are accumulating.
- Which workloads are the best candidates for transparent checkpointing.
Not every timed-out allocation represents completely lost work. Some applications may have written native checkpoints before termination, and scheduler logs alone cannot tell us how recent or usable those checkpoints were.
The logs do reveal where the boundary is being crossed, how much compute reaches it, and where a deeper application-level analysis should begin.