TL;DR
- Kubernetes ended your GPU pod, and unless something saved its state while it ran, the work it was holding in GPU memory is gone. No command you run afterwards brings it back.
- Kubernetes has five ways to end a running pod, with the Linux kernel's out of memory kill underneath them. A spot reclaim and Kubernetes preemption add no sixth way, because each of them triggers a path that already exists.
- Karpenter and the AWS Node Termination Handler bring the node and the pods back, but neither one saves what was running. A grace period is time to shut down cleanly rather than time to move GPU memory anywhere.
- A pod checkpointed while it ran restores from its last checkpoint on the replacement node instead of starting from nothing. So an eviction costs the restore plus the work done since that checkpoint.
- In this piece we walk through the five ways Kubernetes ends a pod, what GKE does on a spot reclaim, and what Karpenter and the node termination handler bring back. Then we cover whether pod preemption gives you a chance to pause, why none of these paths preserves GPU memory, whether a replacement node shows up, and what a checkpointed pod restores instead.
Kubernetes has five ways to end a running pod, plus the kernel underneath
Kubernetes ends running pods on purpose, and five separate parts of it can do it, with the Linux kernel's out of memory kill underneath them. A spot node being reclaimed and a higher priority pod needing room add no new paths, because each of them triggers a path that already exists. The five differ in who acts, what sets them off, and how much warning the pod gets.
| Path | Who acts | What sets it off | The grace the pod gets |
|---|---|---|---|
| Node-pressure eviction | The kubelet, the agent Kubernetes runs on every node | Memory, disk, or inodes running short on that node | None under a hard threshold, and no disruption budget is consulted |
| Priority preemption | The scheduler | A pending pod of higher priority has nowhere to go | The evicted pod's termination grace period |
| Taint-based eviction | The node, once a taint lands on it | A NoExecute taint the pod does not tolerate | None, the pod goes immediately |
| API-initiated eviction | Whoever calls the Eviction API, as kubectl drain does | Maintenance on the node | Graceful termination, and disruption budgets are respected |
| Autoscaler disruption | The autoscaler, which sits outside Kubernetes | Consolidation of an underused node, or a spot interruption | The node is drained first |
| Out of memory kill | The Linux kernel | A container going past its memory limit | None |
The kubelet acts alone when a node runs short of memory, disk, or inodes, and the two protections you may be counting on do not apply to it. The Kubernetes documentation states that it "does not respect your configured PodDisruptionBudget or the pod's terminationGracePeriodSeconds". A PodDisruptionBudget caps how many pods in a set may be down at once, and terminationGracePeriodSeconds is the time a pod gets to shut down cleanly. Neither applies.
The scheduler uses priority to decide which workload gets scarce capacity. A pod carries a PriorityClass, an object mapping a name to an integer. Pod preemption is how a cluster hands a production job the GPUs a batch job is holding, and the victim gets its graceful termination period and nothing more.
A taint on the node can clear your pod too. A taint is a mark on the node, and a toleration on the pod lets it stay, so a NoExecute taint clears pods without a matching toleration. Draining is the deliberate version of the same thing, where kubectl drain evicts every pod through the Eviction API, and a graceful termination is still a termination.
The kernel's row is not an eviction at all. When Linux enforces a memory limit with an out of memory kill, no Eviction object is created or budget consulted, and the pod's status records OOMKilled. Whether the pod is evicted or OOM-killed, the workload loses its in-memory state.
Does GKE drain and cordon the node before it preempts, or does it just shut down hard?
On GKE the node is meant to be drained rather than cut off. A spot reclaim starts graceful node shutdown after the preemption notice, which is the warning that Google is taking the machine back, and the default graceful termination period is 30 seconds.
The sequence does not always finish cleanly. One engineer on Stack Overflow asks whether the node does a drain and cordon or just a hard shutdown. Another reports GKE 1.20 preemptible nodes whose pods zombie into Failed or Shutdown status, cleared by a cron job that force deletes them every 9 minutes.
What Karpenter and the node termination handler bring back
Karpenter, a common autoscaler, starts the replacement as soon as it sees the reclaim. Its disruption documentation describes monitoring an AWS SQS queue for the 2-minute termination warning and draining the affected node while provisioning a replacement in parallel. The same page says what happens once that time is up. "Once the terminationGracePeriod elapses, remaining pods will be forcibly deleted and the underlying instance will be terminated."
The AWS Node Termination Handler does the Kubernetes half on its own. Its README describes cordoning the node so no new work is scheduled there, "then drain it, removing any existing work".
So the node comes back and the pods come back. The work does not, and Karpenter says as much in the notes on its do-not-disrupt annotation, which exists for "a long batch job (such as you might have with machine learning) that would need to start over if it were interrupted".
Does Kubernetes preemption give the pod any chance to pause first?
No, the pod is interrupted and whatever progress it had reached is part of the price. On a shared cluster the trigger is priority rather than the cloud, and the maintainers of Kueue wrote that price into their own preemption design issue: preemption reclaims borrowed capacity, and the obvious tradeoff is interrupting workloads and potentially losing significant progress.
Engineers who ask for a hook inside that window do not get a reliable one. An issue on the node termination handler asks for a script that runs when the notice appears, so that files generated in the pod can be transferred back to S3 inside the 2 minute window. A Karpenter issue reports that the preStop hook is not respected when a spot termination is what ends the pod.
None of these paths preserves what is in GPU memory
A grace period is time to shut down cleanly, and that is all it is. A container that handles the termination signal can flush a log and close its connections, but nothing in that contract moves GPU memory anywhere.
The work is in that memory. A serving worker holds its model weights in VRAM, the GPU's own memory, together with the key-value (KV) cache that carries the attention state of every request in flight. That state exists nowhere else, so it can be rebuilt only by redoing the work. A 2026 MLSys paper on fault-tolerant serving, GhostServe, describes a failure taking that volatile state and forcing the system to restart the inference job from the very beginning.
The replacement pod starts from nothing, so every step of the start-up runs again. It pulls its image, brings up the serving engine, fetches and loads the weights, initializes the distributed group, profiles the cache, and captures its CUDA graphs before it can answer a request. Cedana's published benchmark measures that on one node of 8 NVIDIA B200 GPUs, where a model of 1.6 trillion parameters takes 34 minutes to become ready to serve after a start with no checkpoint.
Will a replacement node even show up?
Sometimes there is no spot capacity to give you, which is a different problem from the one the rest of this page is about. Teams hold on demand nodes in reserve against spot risk, and one report on the Kubernetes autoscaler describes a GPU pool where spot instances are frequently unavailable, with the autoscaler trying the spot ASG for 15 minutes before falling back. That is a provisioning question. Checkpointing does not launch a machine, and it helps only once a replacement exists.
A checkpointed pod restores instead of starting from nothing
A checkpoint is the saved state of a running workload at one moment. It is taken below the application, so nothing in the code changes. It holds the GPU memory and the CUDA context, the process and its files, the open network connections, and what the scheduler knows about the job.
With that copy in hand, recovery happens inside the Kubernetes lifecycle, and that is what we build at Cedana. For a workload kind that reschedules its pod after a node failure or an eviction, and provided the pod was checkpointed while it ran, the pod restores from the latest checkpoint instead of starting from scratch. The same holds on the replacement node when an autoscaler preempts the one underneath, which is the spot case. A pod evicted by priority resubmits with its checkpoint, and a pod that is OOM-killed resumes at its last checkpoint rather than starting over.
Recovery starts after the pod has gone, once it is rescheduled or the replacement node comes up. Every model in our published benchmark set restores in 57 to 70 seconds on the benchmark node.
What ships today depends on how many machines the job spans. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.
How much you lose between checkpoints is a policy you set
A restore brings back the latest checkpoint, so any work done since that checkpoint must be repeated. Heartbeat checkpointing takes one on an interval you set, which holds the loss to that interval. A triggered policy runs on a condition instead, and the documented example fires a checkpoint when a pod passes 80% memory utilization in a namespace or deployment, so a recent copy can exist before the kernel kills the container.
Turning this on is one install and one line per workload. Cedana's Helm chart installs a daemon on every node as a DaemonSet, and it stays dormant until you opt a workload in with one line in its pod spec, the runtime class.
A checkpoint does not survive a version change and does not stop the evictions
A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.
None of this stops the evictions. The actors in the table keep their authority to end pods, which is how a cluster reclaims memory and places urgent work, so what changes is the price of each one. A failure costs the resume time, not the state. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. For a pod on a Kubernetes GPU node, that means an eviction costs a restore plus whatever the pod did since its last checkpoint, instead of everything it had done. The next time the kubelet, the scheduler or an autoscaler ends your pod, that is what it costs you.
Related:
- What happens to a training job when a spot instance is reclaimed
- What SkyPilot, Ray Train, SageMaker and MemVerge bring back after a spot reclaim
- Draining a GPU node in Kubernetes without losing the work on it
- What cannot be checkpointed in a GPU workload?
Common questions
When my spot node is reclaimed, will Karpenter or the node termination handler stand up replacement capacity before my application goes down, or do I have to build that failover myself?
Karpenter brings a replacement node up on its own, starting to provision a new node while it drains the old one. The AWS Node Termination Handler only cordons and drains the node so no new work is scheduled there. Either way the node comes back and the pods come back, but the work does not, because neither tool saves what was running.
When GKE preempts a node, does it drain and cordon the pods gracefully first, or does it just hard-shut-down the node and leave the pods stuck?
On GKE, a spot reclaim starts graceful node shutdown after the preemption notice, with a default graceful termination period of 30 seconds. Whether that sequence finishes cleanly is disputed: one engineer reports GKE 1.20 preemptible nodes whose pods zombie into Failed or Shutdown status, cleared only by a cron job that force deletes them every 9 minutes.
When a Kubernetes node gets drained for a spot interruption or a scheduler preemption (Kueue, aws-node-termination-handler), the pod is evicted or killed immediately with no chance to pause it. Is there a way to avoid that?
No. The eviction still happens, and a grace period gives the container time to shut down cleanly, which is not time to save GPU state. The maintainers of Kueue say as much themselves: preemption reclaims borrowed capacity, and interrupting workloads and potentially losing significant progress is the tradeoff. With Cedana checkpointing the pod while it runs, a pod evicted by priority resubmits with its checkpoint instead of starting over.
