Stateful AI has the same boundary problem as long HPC runs

Every job meets the limit the same way

The cap on a shared research cluster was written when the long jobs were simulations, and inference services, fine-tuning runs and agent sessions now wait in the same queues under the same rule. In Slurm, every job runs against a declared time limit, and the partition and quality of service (QOS) policies set the defaults and the maximums. A QOS is the policy object a scheduler applies to a queue, and maximum wall time is one of the limits it carries.

When the enforced limit expires, Slurm terminates the job and records it as a TIMEOUT. The job is terminated whether it had just started or was 95% complete.

The limit is doing real work for the cluster. Slurm's backfill scheduler starts a lower-priority job early when doing so will not delay a higher-priority reservation, and it can only make that call if every running job has an expected end. A runtime ceiling is also what returns an allocation's nodes to the pool, and it puts an upper bound on how long a clean drain takes before maintenance. Without one, a hung process would hold an allocation indefinitely.

Administrators keep a documented exception for work that does not fit, granted case by case rather than by default. The University of Utah's Center for High Performance Computing publishes a maximum of 72 hours for jobs on its general nodes and gives special access to a long QOS above that limit case by case. That is one center's published policy, and other sites set their own numbers.

Nothing puts a killed job back in the queue on its own, so it returns when somebody submits it again or when something requeues it. Slurm's sbatch documentation says where a requeued job starts: "When a job is requeued, the batch script is initiated from its beginning."

The same page lists the events that requeue an eligible job: an administrator acting by hand, a node failure, and preemption by a higher-priority job. Whatever the trigger, the script runs again from its first line, so nothing about the process that was running comes back with it.

Simulation programs cross the limit by saving their own state

High-performance computing, or HPC, has lived with this boundary for decades, and the mature simulation programs answer it by saving their own state and starting again from it. GROMACS, a molecular dynamics engine, ships a flag for this exact situation. With -maxh set, a run stops itself and writes a checkpoint at the first neighbor search step after the run time passes 99 percent of the hours specified, so the save happens just short of the wall time rather than after the kill.

The checkpoint holds a full-precision copy of the positions and velocities, along with the state information the algorithms need to restart, so the run continues from it as if there had been no interruption. None of that is free, and the bill is paid inside the application: GROMACS has to know which state matters, serialize it correctly, write it safely, expose a restart path you can point at, and hand you a switch you have to know to set.

Every program holds different state and makes its own choices about formats, intervals, and recovery, so none of the work done for one carries over to the next. One of the most mature applications in HPC needs a dedicated command-line option to survive a scheduler policy.

AI workloads arrive with no copy of their state anywhere

Inference now accumulates session-scoped state that lives only in VRAM and can be reconstructed only by redoing the work. Any interruption destroys it. VRAM is the GPU's own memory, and nothing outside the GPU holds a copy of what is in it unless something has written one.

A serving worker, a fine-tuning run and an agent session each keep their progress there, in three different shapes.

• A serving worker holds the weights it loaded plus the key-value (KV) cache of every session in flight, spread across the GPUs of one instance.
• A fine-tuning run that has gone on for days carries everything computed since its last framework checkpoint.
• An agent session is context accumulated one turn at a time.

None of this is misuse of the queue. Molecular dynamics simulations, computational fluid dynamics, protein pipelines, parameter sweeps, and model training all legitimately run longer than one allocation, and with AI so do long-running stateful inference and fine-tuning jobs. All of them cross allocation boundaries.

Most serving engines and training frameworks have no checkpoint path of their own, so the requeued script starts at its first line and the worker rebuilds from nothing. GhostServe, a 2026 MLSys paper on inference serving, puts the loss in one sentence: "In the event of a failure, this volatile state is lost, forcing the system to restart the inference job from the very beginning." That sentence describes a failure rather than a scheduler kill, but the mechanism is the same either way.

The AI side loses more than time

A simulation's state is what the program computed and can name, which is why GROMACS can write out positions, velocities, and the information its algorithms need. A serving worker's state is harder to name, because it includes the weights, the cache for every session it is holding, the CUDA context and the compiled graphs built while the engine started, and the requests that are partway through generation. That state is spread across several GPUs, and most serving engines write no copy of any of it.

A batch simulation is judged against a completion date, so a lost interval costs time and the researcher submits the job again. A serving endpoint has people waiting on it now, so the same lost hour interrupts someone's request and affects whatever availability your team has promised. The two kinds of work owe their users different things.

A simulation restarted from last week's checkpoint loses time, and nobody outside the group sees it. A fine-tuning run also falls back to its last framework checkpoint, usually across a longer stretch of work. A killed serving worker is visible the moment it goes, and the sessions it was holding are gone because no copy of them exists anywhere.

An AI team that answers this the way GROMACS did has to do the same engineering again. For every serving engine or training framework that lacks a checkpoint path today, the team needs a separate one, written and maintained by whoever owns that program.

Today, a team whose job does not fit inside the limit chooses among four options by hand.

• Split the workload into stages that each fit inside an allocation.
• Configure or build a checkpoint and restart path for the application.
• Request access to a long-running partition or QOS.
• Restart the job from the beginning after a TIMEOUT.

Each answer is arranged for one application, one queue, or one researcher, and the next workload starts from nothing.

A restore costs about a minute where a rebuild costs tens of minutes

Many teams end up with the last of those four, restarting after the TIMEOUT, so the number that matters is what starting over costs. Cedana's published benchmark covers four frontier models, each served with SGLang from its official cookbook recipe, unmodified.

The native cold start is the time from engine launch to ready-to-serve, including weight loading and full engine initialization. The restore clock covers bringing that same fully initialized engine back from a checkpoint until it is ready to serve.

All four runs were on a single node with 8x NVIDIA B200 (Blackwell) GPUs and 1.7 TB of system memory. They used CUDA 12.9 and ran on tmpfs, a filesystem that lives in system memory rather than on a disk or across a network.

The restores stay inside a narrow band, but the parameter counts do not. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. Across these four rows the checkpoint grows 3.6 times and the restore time 1.2 times, so on this node the restore column barely moves. Four rows cannot show more than that on their own.

A worker killed at a boundary pays its cold start again on the other side, and again at the boundary after that. A worker restored from a checkpoint pays the restore instead and keeps the state it had.

A checkpoint taken below the application makes the crossing general

A checkpoint taken below the application does not need the application to know about it. It captures the running process as it stands, including the files, the network connections, and the scheduler's view of the job, so one mechanism covers every job on the cluster rather than only the programs whose owners built a path. The limit stays exactly where the administrator set it, and what changes is what the job does when it gets there.

So the programs that never got a checkpoint path of their own cross the boundary the way GROMACS does, without anybody writing them one. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. Cedana's Slurm integration checkpoints a running job as it approaches its time limit, releases the allocation, requeues the job, and restores it on a compatible node. A week-long run then finishes on a cluster with a 24-hour limit by carrying its state from one 24-hour allocation to the next, and so does the stateful AI job queued beside it on your cluster, with no checkpoint path of its own.

Related:

• Your Slurm job was cancelled due to time limit. What to do now
• Slurm has no suspend option for GPU jobs. What preemption without killing the job looks like
• DMTCP, application checkpoints, workflow managers and system-level checkpointing: what each covers on a Slurm cluster
• GROMACS shows both the value and the limit of application checkpointing
• Measuring the cost of wall-time termination from your sacct data

Common questions

My job will be killed when it hits its time limit and I will lose all the progress since my last save. How do I make it save its state so it can pick up where it left off?

When the program cannot save itself, the save has to happen below it. A checkpoint taken below the application captures the running process as it stands, including the files, the network connections, and the scheduler's view of the job, so the program needs no checkpoint code of its own. Cedana's Slurm integration checkpoints a running job as it approaches its time limit, releases the allocation, requeues the job, and restores it on a compatible node.

When a requeued job comes back after a timeout, does it resume where it left off or restart the whole script?

It restarts the whole script. The requeued script starts at its first line and the worker rebuilds from nothing, because most serving engines and training frameworks have no checkpoint path of their own. A checkpoint taken below the application changes that, since one mechanism then covers every job on the cluster rather than only the programs whose owners built a path.

My computation needs more time than the fixed wall-time limit allows. How do I split or chain it across multiple job submissions so the whole thing still finishes?

One answer is to split the workload into stages that each fit inside an allocation, which somebody arranges by hand for that one application. The other is to let the checkpoint carry the job across: Cedana's Slurm integration checkpoints a running job as it approaches its time limit, releases the allocation, requeues the job, and restores it on a compatible node. A week-long run then finishes on a cluster with a 24-hour limit by carrying its state from one 24-hour allocation to the next.

Why did my job get killed, and what does the TIMEOUT state mean?

When the enforced limit expires, Slurm terminates the job and records it as a TIMEOUT, and the job is terminated whether it had just started or was 95 percent complete. The limit is doing real work for the cluster: backfill starts a lower-priority job early only when every running job has a predictable expected end, and a runtime ceiling is what returns an allocation's nodes to the pool.