LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 99

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicAll topics

Cedana. Blueprint of a board of eight node rectangles in two rows, most holding grey job blocks each fixed with a black push-pin where it landed. One job in the top right is a dashed blue outline carrying a saved-state icon and no pin, with a long blue dashed arrow curving from it to a solid blue block that has landed in the one empty node in the row below.What would a GPU scheduler do differently if it could move running jobs?
· Cedana Editorial

What would a GPU scheduler do differently if it could move running jobs?

Compare how GPU schedulers handle placement, preemption and time limits, and what checkpointing could change once a job has already started.

read →
Cedana. Blueprint of four tall worker columns each under a green tick, marked all healthy. The first is filled almost to the top in red; the other three hold a shallow blue fill. A blue dashed arc carries the first worker to a fifth, dashed column at the right, filled pale blue to the same height, and a save icon marked checkpoint sits under the first column.Why a healthy inference worker can still be in the wrong place
· Cedana Editorial

Why a healthy inference worker can still be in the wrong place

Understand how GPU topology, decode load and hardware fit affect inference workers, and why admission-time placement cannot rebalance running sessions.

read →
Cedana. Blueprint of a serving engine as a chain: a scatter of small request circles waiting on the left, an arrow into an API server box that already holds three, an arrow to an engine-loop box with a circular arrow inside, and an arrow to a grid of eight worker squares, seven with blue status bars and one filled pale red with a dashed red outline and no bar. Under the chain a timeline of blue heartbeat marks runs up to a red bar, the hang, with the last mark before it ringed under a saved-state icon.What to do when vLLM or SGLang stops responding and nothing crashed
· Cedana Editorial

What to do when vLLM or SGLang stops responding and nothing crashed

Separate a stalled serving engine from a hung GPU. Understand health-check limits and why recovery needs a checkpoint from before the engine stopped.

read →
Cedana. Blueprint of two rows of the same steps in different orders. On top: a warning triangle, a cordon barrier, a node whose job is red and struck out as it drains, then a reset arrow. Below: the warning triangle, a blue saved-state icon, a healthy node holding the job in blue, then the cordon, an empty node draining, and the same reset arrow.What to do when DCGM flags a GPU that has a job running on it
· Cedana Editorial

What to do when DCGM flags a GPU that has a job running on it

Interpret DCGM and Xid alerts, distinguish repair from workload recovery, and decide which checkpoint to restore before draining a degraded GPU node.

read →
Cedana. Blueprint of a ten by ten grid of dots, eighty-four grey and sixteen red scattered among them, the wrong predictions in a hundred. To the right two thick bars from one baseline: a long grey bar ending in a red cross, what each wrong one costs today, and a short blue bar under a saved-state icon, what it costs when acting means a restore.Is it worth acting on a GPU failure prediction?
· Cedana Editorial

Is it worth acting on a GPU failure prediction?

Assess GPU failure predictions using precision, warning time and the cost of acting. Understand what checkpoints change and which failures give no warning.

read →
Cedana. Blueprint of a thick horizontal bus with eight GPU cards hanging from it on short stems, their status bars grey. One stem is broken into red dashes and its card has dropped away below the row, tilted, outlined in dashed red and struck out. From a saved-state icon above the bus a blue dashed arrow curves down to a ninth card on a second, shorter bus at the lower right, its status bar blue.What happens to a training job when a GPU fails
· Cedana Editorial

What happens to a training job when a GPU fails

Understand Xid faults, GPU resets and the state a training job loses. Learn why recovery depends on a checkpoint taken before the hardware fails.

read →
Cedana. Blueprint of sixteen GPUs in a ring as one training group, one filled red and struck out, the ring joining them broken into dashes, and the collective at the hub struck out. A blue ring encloses the whole group.One node failed and the whole training job died
· Cedana Editorial

One node failed and the whole training job died

Learn why one failed rank stops a distributed training job, what NCCL timeouts mean, and how checkpoint state determines what a restart can recover.

read →
Cedana. Blueprint of a long progress track filled blue up to a red cut, with small blue heartbeat marks along it and a grey document icon standing some way back. From the cut, grey dashed arcs loop back to the very start of the track and to the document, another climbs to a second grey lane above that runs on unbroken, and a short blue arc drops to the last heartbeat mark just before the cut.What torchrun, torchft and Ray Train restart from when a node dies
· Cedana Editorial

What torchrun, torchft and Ray Train restart from when a node dies

Compare how torchrun, torchft, Ray Train and Kubeflow recover after a node fails, including saved state, code changes and checkpoint intervals.

read →
Cedana. Blueprint of two tracks of one job ending at the same red line marked reclaim, with a pale red band just before it. On the top track the tick marks are far apart and a red hatched block marked work lost runs back the whole interval. On the bottom track the blue ticks are dense and the hatched loss is a sliver.What happens to a training job when a spot instance is reclaimed
· Cedana Editorial

What happens to a training job when a spot instance is reclaimed

Compare spot interruption windows and recovery costs for GPU training. Learn what must be saved before reclaim and when spot remains worth using.

read →
Cedana. Blueprint of one pod at the centre holding a blue block of GPU memory, with six lines converging on it from small circles at the left and right edges, some solid and some dashed, each ending in an arrowhead. Beneath the pod its floor is a red dashed line over a red cross. Lower right, a dashed outline of a replacement pod with an empty slot, and a blue saved-state icon pointing into it.What happens to a GPU pod when Kubernetes ends it
· Cedana Editorial

What happens to a GPU pod when Kubernetes ends it

Compare the ways Kubernetes ends GPU pods, the warning each path provides, and what checkpoint recovery needs after eviction or spot-node termination.

read →
Cedana. Blueprint of a dashed, struck-out instance on the left with three lines leaving it to the right: two grey dashed lines to an empty machine outline and to a small stack of documents, and one solid blue line, through a saved-state icon, to a machine holding the job in blue as it was running.What SkyPilot, Ray Train, SageMaker and MemVerge bring back after a spot reclaim
· Cedana Editorial

What SkyPilot, Ray Train, SageMaker and MemVerge bring back after a spot reclaim

Compare spot recovery tools by the machines, files or running state they restore, the checkpoint code they require and their published limits.

read →
Cedana. Blueprint of five process columns on one node under a dashed line marked limit; the tallest breaches the line, its top painted red and struck out. A blue dashed arrow carrying a save icon marked checkpoint leads right to a second node whose dashed limit sits higher, where the same column stands in blue with room above it.What happens when a cluster job runs out of memory
· Cedana Editorial

What happens when a cluster job runs out of memory

Identify which memory limit killed your cluster job, where to find the evidence, and why recovery requires state saved before the OOM kill.

read →

Showing 73–84 of 99 posts