LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
Featured
Blueprint schematic of an eight-GPU node accumulating session state, with a state gauge rising toward the VRAM capacity lineBlueprint schematic of an eight-GPU node accumulating session state, with a state gauge rising toward the VRAM capacity line
FeaturedEngineering

The Era of Stateful Inference: How to Improve Cost per Token.

We are entering the stateful inference era, driven by frontier models with longer context windows, longer in-flight sessions, and single instances spanning 8, 16, or more GPUs.

Updated · Cedana Editorial
read →
All posts · 98

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicGPU checkpointing

Cedana. Blueprint of one machine drawn as a large outlined box holding six pale-blue tiles with line glyphs: a chip, a server, a folder, a network, a calendar, and stacked layers. A blue dashed outline with a checkpoint mark at its corner encloses all six. Three grey dashed arrows leave the machine's right edge to three grey circles outside it holding a paper plane, a database, and a webhook glyph.
· Cedana Editorial

What Cedana saves, and what it does not

See which GPU, process, file, network and scheduler state Cedana saves, what remains outside the checkpoint, and what a compatible restore requires.

read →
Cedana. Blueprint of a memory block with grey stripes copied by a blue arrow into a document with the same stripes. From the document, two blue arrows go to a folder and to a bucket inside a large dashed boundary, and a grey dashed arrow leaves the boundary to a cloud shape below it. A small dashed padlock sits under the document.
· Cedana Editorial

Where checkpoints live, how long they stay, and who can read them

Choose where GPU checkpoints live, how long they remain and who can read them. Identify the encryption and key-management answers a security review still needs.

read →
Cedana. Blueprint of a multiplication written in shapes across the frame: a blue block marked twenty gigabytes, times a stack of five pale blue copies, times a grid of sixteen small squares, equals a database cylinder.
· Cedana Editorial

How much storage does checkpointing a GPU cluster need?

Estimate checkpoint storage from workload memory, retained copies and job count. Review quota controls, storage bandwidth and unanswered lifecycle behaviors.

read →
Cedana. Blueprint of four horizontal slider tracks with end stops, a faint dashed line at their midpoint, a circular restart arrow at the left end and a saved-state icon at the right. On every track a hollow grey knob sits near the left and a solid blue knob sits near the right.
· Cedana Editorial

When retrying the job is the right call

Decide whether a GPU job needs checkpoints using interruption risk, work at stake, rebuild cost and deadline slack. Keep application idempotency in either path.

read →
Cedana. Blueprint of two rows of small grey blocks, each ending in an outlined output box. The top row is unbroken. The second has one block replaced by a blue save icon marked checkpoint. A bracket joins the two output boxes and the word match sits beside it.
· Cedana Editorial

How to know a restored GPU workload is correct

Test whether a restored GPU workload continues the same computation. Compare repeatable runs, check multi-GPU boundaries and record the exact environment.

read →
Cedana. Blueprint of two identical GPU cards at the top joined by a blue dashed arc carrying a saved-state icon. Beneath them a wide dashed band with a faint centre line holds a dozen dark grey dots scattered at different heights, and near its right end one blue dot with a ring around it sits inside the band, under the second card.
· Cedana Editorial

Does moving a job change its results?

Understand what a checkpoint preserves when a GPU job moves, which versions must match, and how live inputs and ordinary GPU variation affect reproducibility.

read →
Cedana. Blueprint of a dashed worker outline struck out in red above a shelf holding six grey cache blocks, with a long segmented grey bar beneath the shelf leading to a small dashed outline. On the right a large worker outlined in blue, with a saved-state icon above it, holds a dark weights block, three small grey blocks, and six blue cache blocks inside.
· Cedana Editorial

KV-cache offload versus a saved worker: what each survives

Compare the state preserved by KV-cache offload and a full worker checkpoint. Learn what survives a crash, what must reload, and where prefix reuse still helps.

read →
Cedana. Blueprint of one wide bar divided by bytes: a large grey block marked weights, four small pale segments, and a solid blue segment at the right marked kv cache. Just past the bar's end a small red dashed square with a cross floats, marked in flight. A blue bracket under the bar spans everything, with a save icon at its centre.
· Cedana Editorial

What is inside a GPU checkpoint?

See the weights, KV cache, CUDA context and process state inside a GPU checkpoint, how they affect its size, and what a restart has to rebuild.

read →
Cedana. Blueprint of four columns on one timeline, each holding two small machine boxes: a grey dashed arrow between them under os, solid arrows under vms and containers, and under gpus an empty blue dashed slot.
· Cedana Editorial

Every era of computing needed a migration primitive. GPUs are next

Trace migration through operating systems, virtual machines, containers and databases to understand why GPU fleets need portable running state.

read →
Cedana. Blueprint of a node in cross-section: three engine boxes on top, a wide process band beneath them, a runtime band of small tiles, then one solid blue band, then a darker kernel band, then a driver band holding four GPU cells with blue status bars. From the blue band one arrow drops into the driver band, one leads right to a process box outside the stack, and a dashed line carries a saved-state icon down to a storage disc.
· Cedana Editorial

How Cedana works: checkpoint, restore, and migration below the serving engine

Explore Cedana's daemon, CRIU and GPU capture layers, the state they restore, and the policies, storage paths and compatibility limits around them.

read →
Cedana. Blueprint of a hexagon cut into six wedges in alternating pale blue and grey, each holding a small line glyph: a GPU card, a chip, two joined dots, a folder, three list lines, a triangle of dots. At the centre a white circle holds one small grey document. A blue dashed outline runs around the whole hexagon.
· Cedana Editorial

What is GPU checkpointing? A plain explanation

Learn what GPU checkpointing saves, how checkpoint, snapshot, restore and migration differ, and why saving model weights alone cannot resume a workload.

read →
Cedana. Blueprint of two pairs of horizontal lanes. In the upper pair the top lane holds three long blue blocks and the lane beneath is almost empty, with a thin black tick and a small hatched sliver at the start of each block. In the lower pair the top lane holds twenty short blue blocks and the lane beneath is crowded with a tick and a hatched sliver under every one.
· Cedana Editorial

What a GPU checkpointing layer costs while the workload runs

Understand steady-state GPU checkpointing overhead, why driver-call patterns matter, and what Cedana's published single-GPU measurements leave unanswered.

read →

Showing 12 of 23 posts