LoginSchedule demo
cedana / throughput · v1.4 · build 2826

Cedana is
GPU job migration
infrastructure
for increasing AI
revenue per MW.

Automatically checkpoints, migrates, and resumes live GPU jobs across instances to increase throughput, reliability, and performance.
- ● DETECT
┌──────────────────────────────────────┐
│  NODE-A · us-east-1c      [FAILING]  │
├──────────────────────────────────────┤
│ GPU-7 B200 ████████████████   88 GiB │
│  iteration 14,821 / 50,000           │
└──────────────────────────────────────┘
│     ··· awaiting checkpoint ···      │
│······································│
│······································│
│······································│
│[░░░░░░░░░░░░░░░░░░░░░░░░]   0%       │
┌──────────────────────────────────────┐
│  NODE-B · us-east-1d      [STANDBY]  │
├──────────────────────────────────────┤
│ GPU-3 B200 ░░░░░░░░░░░░░░░░    0 GiB │
│  awaiting checkpoint                 │
└──────────────────────────────────────┘
GPU utilization
██████▆▄▃▃▄▅▆▆▅▃▁▁86%
checkpoint sent
00/96 GB
loss 0.00%SLA OKt+ 0.00s
~ / cedana / · ● recording
$ cedana migrate --job=train-llama-7b --to=node-04
[+] inspect :: pid 4318 / cuda 12.4 / 2.4 GiB
[+] checkpoint :: ok (1.7s · cuda-aware)
[+] verify hash :: ok (sha256: 9f4c…e2)
[+] transfer :: ok (96 GiB/s)
[+] restore on B :: ok (gpu-7 → gpu-3)
[+] resume @ step :: 14 832 / 50 000 (Δ 0.42s)
[✓] migrated. SLA preserved.
$
01The Platform

unlock() your scheduler.

// 01

Work with what you have

No rip-and-replace. No code changes. No disruption to your teams.

// 02

Kubernetes & SLURM

Built for AI and HPC. Native support for SLURM and Kubernetes.

// 03

First migration in <30 min

K8s helm chart or SLURM plug-in. No changes to your config.

02The Problem → The Fix

AI workloads cannot move once running.
Cedana makes them liquid.

Four ways stranded compute drains budgets — and how Cedana resolves each, live and in place.

— the problemstatus: degraded
→ with cedanastatus: ok
cluster · 64 gpus · 25% utilized● 75% IDLE
active 16 idle 48
// 01ERR.IDLE

Idle GPUs

Valuable compute remains stranded while critical work is delayed.

75% idle
cluster · 64 gpus · 88% utilized● 88% ACTIVE
active 56 idle → reassigned
// 01RESOLVED

Maximize throughput

Workloads shift to idle GPUs, reclaiming stranded capacity and maximizing cluster throughput.

up to 88% utilization
training · 50 000 stepsrunning…
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
0%compute saved: 0
// 02FATAL.RESTART

Expensive failures

Failures and preemptions force workloads to restart from scratch — up to 65% of compute wasted.

65% compute lost
NODE-A · FAULT✗ checkpoint step 14 832migrate()NODE-B · OKresumedstep 14 832 / 50 000● Δ 0.42s↳ live migration · sla preserved
// 02RESOLVED

Automated reliability

Workloads automatically migrate to healthy infrastructure and resume after failures.

0 lost progress
capacity · 10 nodes30% buffer · idle
active 7 safety buffer 3
// 03WASTE.30%

Overprovisioned GPUs

Capacity is routinely over-provisioned by 10–50% just to maintain reliability and hit SLAs.

10–50% overprovisioned
capacity · 10 nodes● 0% buffer · full
active 10 reclaimed 3
// 03RESOLVED

Eliminate overprovisioning

Automatic migration and recovery remove the need for large safety buffers to meet SLAs.

0% safety buffer
scheduler · static routing● ROUTE FAULT
SCHEDULERw1w2✗w3w4
1 lane down · scheduler can't reroutew3 stuck
// 04STUCK.SCHEDULE

Rigid infrastructure

Schedulers cannot dynamically adapt to failures, demand, or changing priorities.

cannot adapt
scheduler · adaptive routing● REROUTED
SCHEDULERw1w2↻w3w4
load reroutes in real timeall lanes healthy
// 04RESOLVED

Adaptive infrastructure

Kubernetes and SLURM adapt workloads in real time to failures and demand.

real-time
03The Difference

The Cedana diff.

— without migrationSTATUS · DEGRADED
✗
Expensive failures
Up to 65% compute lost
✗
Over-provisioned GPUs
10–50% capacity buffers
✗
Idle GPUs
Stranded compute while jobs wait
✗
Rigid infrastructure
Schedulers cannot adapt
+ with cedanaSTATUS · OK
✓
Automated reliability
Workloads resume automatically
✓
Eliminate overprovisioning
SLAs without safety buffers
✓
Maximize throughput
Workloads migrate to idle GPUs
✓
Adaptive infrastructure
Workloads adjust in real time
AWS●Google Cloud●NVIDIA●SLURM●Kubernetes●Lambda●CoreWeave●AWS●Google Cloud●NVIDIA●SLURM●Kubernetes●Lambda●CoreWeave●
04Cold-Start Race

Same model. Same hardware.
21.7× faster to first token.

GLM-5.2-FP8 · 753B MoE · Multimodal · vLLM v0.19.0 · TP=8 · 8× B200 · 734 GiB ckpt · elapsed 00:00.00s
NATIVE · vLLM v0.19.0 · TP=8
● COLD START
00:00.00
$ vllm serve zai-org/GLM-5.2-FP8 --tp=8
INFO 00:00:03   spawning 8 workers · EP=8
INFO 00:00:40   loading shards 12/94 …
INFO 00:05:00   loading shards 40/94 …
INFO 00:12:00   loading shards 78/94 …
INFO 00:16:20   loading shards 94/94 …
INFO 00:18:40   building cuda graph (×8)
INFO 00:20:30   warming kv cache · vision tower
INFO 00:21:30   jit compile attention
CEDANA · RESTORE
▲ WARM RESUME
00:00.00
$ cedana resume glm-5.2-fp8 --from=snap.az-1
[+] fetch snapshot     :: ok    (734 GiB · memlock)
[+] verify hash        :: ok    (sha256: c4e1…9a)
[+] restore cuda ctx   :: ok    (8× driver attached)
[+] repopulate gpu mem :: ok    (weights + KV)
[+] thaw connections   :: ok    (sockets)
[+] register endpoint  :: ok    (port 8000)
READY 00:01:01   first token
NATIVE
0.0s
CEDANA
0.0s
delta · −1261s savedspeedup · 21.7×verdict · cedana wins
04BStateful Reliability

Hardware fails.
Your training run doesn't.

scenario · sev-1 · power loss · node-A
01 ·RUNNING
02 ·POWER LOSS
03 ·FAILURE DETECTED
04 ·MIGRATING
05 ·RESUMED ON B
NODE-A · GPU-7
● RUNNING
■■■■■■■
$ python train.py --batch=64 --gpu=7
INFO step 13 200/50 000  loss=2.41   lr=3e-5
INFO step 14 050/50 000  loss=2.34   lr=3e-5
INFO step 14 800/50 000  loss=2.31   lr=3e-5
FATAL gpu-7 :: power_loss · pcie link down
ERR   nvml :: device unreachable (-117)
WARN  cedana :: node-A flagged unhealthy
WARN  cedana :: snapshot @ step 14 832 secured
NODE-B · GPU-3
● STANDBY
□□□□□□□□
$ cedana watch --cluster=us-east-1
INFO node-B :: idle  (gpu-3 free)
INFO node-B :: idle  (gpu-3 free)
[+] receive snapshot   :: ok   (96 GiB/s)
[+] verify hash        :: ok   (sha256: 9f4c…e2)
[+] map gpu pages      :: ok   (cuda-aware)
[+] thaw kv cache      :: ok   (no jit)
[+] restore optimizer  :: ok   (state matched)
[✓] resume @ step 14 832/50 000  Δ 0.42s
INFO step 14 900/50 000  loss=2.30   lr=3e-5
TRAINING STEP ·
PERSISTENT
1,530 / 50 000
checkpoint · continuousdowntime · thawing…steps lost · 0state · preserved
05Advanced · Status: SCALING

Built for the hardest workloads.

01 · ADVANCED WORKLOADS

Distributed by default.
Works transparently with MPI & NCCL.

// distributed migration · mpi · ncclRUNNING
cluster-a8× gpu
g0g1g2g3g4g5g6g7
nccl ring
checkpoint
────▶
idle
new partitioncluster-b
g0g1g2g3g4g5g6g7
standby
mpi · oknccl · ringsteps lost · 0code changes · none

02 · SCALE

From a single node to an AI factory.

// scale tiers · 1 → ai factoryACTIVE · SINGLE NODE
single node
4× gpu
cluster
14× gpu
ai factory
28× gpu
on-prem · okhybrid · okcloud · okscale · linear

●READY · COMMAND YOUR COMPUTE$ cedana run --anywhere

Your compute,
liquid_

Schedule a demo
// 01
AWS
// 02
Google Cloud
// 03
NVIDIA
// 04
K8s
// 05
SLURM
// 06
Nvidia Dynamo