LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 99

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicAll topics

Cedana. Blueprint of a red dashed vertical line marked time limit with two horizontal lanes crossing it. In the upper lane, marked simulation, a small molecule of joined circles carries a document past the line and appears whole on the far side. In the lower lane, marked serving worker, a GPU worker stacked with blue slabs approaches the line and only a dashed outline stands beyond it, except that a save icon marked checkpoint below the line sends a blue arrow to a second worker outlined in blue with its slabs intact.Stateful AI has the same boundary problem as long HPC runs
· Cedana Editorial

Stateful AI has the same boundary problem as long HPC runs

See why inference, fine-tuning and agent workloads face the same allocation limits as HPC simulations, but carry different state and recovery obligations.

read →
Cedana. Blueprint of one wide bar divided by bytes: a large grey block marked weights, four small pale segments, and a solid blue segment at the right marked kv cache. Just past the bar's end a small red dashed square with a cross floats, marked in flight. A blue bracket under the bar spans everything, with a save icon at its centre.What is inside a GPU checkpoint?
· Cedana Editorial

What is inside a GPU checkpoint?

See the weights, KV cache, CUDA context and process state inside a GPU checkpoint, how they affect its size, and what a restart has to rebuild.

read →
Cedana. Blueprint of a row of forty-three small squares under a bracket that spans them all: the first twenty solid red, the next ten red-hatched, the rest empty outlines. Below the row a red line runs under the twenty and continues dashed under the ten. Further down, a single blue square sits beside a saved-state icon.One frontier restart burns half a monthly 99.9% error budget
· Cedana Editorial

One frontier restart burns half a monthly 99.9% error budget

Calculate how model restart time consumes an inference error budget, what replicas change, and how to compare a measured restore with your availability SLO.

read →
Cedana. Blueprint of a document with a folded corner, ruled lines, one boxed field holding a short blue mark, and a blue circled tick. To the right, two candidates for the field: a long grey dashed bar marked twenty to thirty minutes, and a very short blue bar beside a timer icon marked about one minute, from which a thin blue line leads back into the field.The recovery time you can put in a filing
· Cedana Editorial

The recovery time you can put in a filing

Define and test LLM recovery time from failure to serving again. Separate restore benchmarks from the detection, placement and recovery your team must document.

read →
Cedana. Blueprint of four columns on one timeline, each holding two small machine boxes: a grey dashed arrow between them under os, solid arrows under vms and containers, and under gpus an empty blue dashed slot.Every era of computing needed a migration primitive. GPUs are next
· Cedana Editorial

Every era of computing needed a migration primitive. GPUs are next

Trace migration through operating systems, virtual machines, containers and databases to understand why GPU fleets need portable running state.

read →
Cedana. Blueprint of a node in cross-section: three engine boxes on top, a wide process band beneath them, a runtime band of small tiles, then one solid blue band, then a darker kernel band, then a driver band holding four GPU cells with blue status bars. From the blue band one arrow drops into the driver band, one leads right to a process box outside the stack, and a dashed line carries a saved-state icon down to a storage disc.How Cedana works: checkpoint, restore, and migration below the serving engine
· Cedana Editorial

How Cedana works: checkpoint, restore, and migration below the serving engine

Explore Cedana's daemon, CRIU and GPU capture layers, the state they restore, and the policies, storage paths and compatibility limits around them.

read →
Cedana. Blueprint of a hexagon cut into six wedges in alternating pale blue and grey, each holding a small line glyph: a GPU card, a chip, two joined dots, a folder, three list lines, a triangle of dots. At the centre a white circle holds one small grey document. A blue dashed outline runs around the whole hexagon.What is GPU checkpointing? A plain explanation
· Cedana Editorial

What is GPU checkpointing? A plain explanation

Learn what GPU checkpointing saves, how checkpoint, snapshot, restore and migration differ, and why saving model weights alone cannot resume a workload.

read →
Cedana. Blueprint of two pairs of horizontal lanes. In the upper pair the top lane holds three long blue blocks and the lane beneath is almost empty, with a thin black tick and a small hatched sliver at the start of each block. In the lower pair the top lane holds twenty short blue blocks and the lane beneath is crowded with a tick and a hatched sliver under every one.What a GPU checkpointing layer costs while the workload runs
· Cedana Editorial

What a GPU checkpointing layer costs while the workload runs

Understand steady-state GPU checkpointing overhead, why driver-call patterns matter, and what Cedana's published single-GPU measurements leave unanswered.

read →
Cedana. Blueprint of three rows of five outlined tiles. The first column is enclosed in a dashed blue fence and its top tile is tinted blue with a saved-state icon. From the right-hand tile of the middle row a grey dashed arrow leads out to a red-outlined box filled with red diagonal hatching.Which GPU workloads should I checkpoint first, and which should I leave alone?
· Cedana Editorial

Which GPU workloads should I checkpoint first, and which should I leave alone?

Choose a small first workload set for GPU checkpointing. Review opt-in controls, cheap retries, real-time inputs, external side effects and untested job servers.

read →
Cedana. Blueprint of two identical clock faces side by side, each with a red cross above twelve o'clock. On the left the sweep is a thick grey arc running most of the way around with the hand near ten o'clock. On the right the sweep is a short thick blue arc and the hand sits just past one o'clock. A dot marks the end of each sweep outside the rim.How do I run a proof of concept for GPU checkpointing?
· Cedana Editorial

How do I run a proof of concept for GPU checkpointing?

Plan a GPU checkpointing evaluation with a matched baseline, real interruptions, correctness checks and agreed success criteria before expanding the install.

read →
Cedana. Blueprint of eleven horizontal bars stacked top to bottom, each longer than the one above, grey along almost their whole length with a short blue block at the right end of every bar. From the longest bar a grey dashed arrow runs to a small GPU card at the lower right with a black push-pin above it.How long do agentic sessions run, and where does their state live?
· Cedana Editorial

How long do agentic sessions run, and where does their state live?

Explore published coding-agent traces to see how turns, cache reuse and pauses shape session affinity, GPU memory use and the cost of a lost worker.

read →
Cedana. Blueprint of a small document on the left with four code cells and a green tick, marked saved, and on the right a large circle marked kernel holding a grey band, a dark block and a GPU card. Three red icons sit around the circle, a cross, a clock and a moon, each joined to it by a red dashed line. A blue bracket spans the circle beneath it with a save icon marked checkpoint.When the node dies under a Jupyter session
· Cedana Editorial

When the node dies under a Jupyter session

Learn what a notebook file cannot recover after a GPU node fails, how process checkpoints preserve kernel state, and which restoration limits still apply.

read →

Showing 25–36 of 99 posts