LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 98

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicGPU checkpointing

Cedana. Blueprint of three rows of five outlined tiles. The first column is enclosed in a dashed blue fence and its top tile is tinted blue with a saved-state icon. From the right-hand tile of the middle row a grey dashed arrow leads out to a red-outlined box filled with red diagonal hatching.
· Cedana Editorial

Which GPU workloads should I checkpoint first, and which should I leave alone?

Choose a small first workload set for GPU checkpointing. Review opt-in controls, cheap retries, real-time inputs, external side effects and untested job servers.

read →
Cedana. Blueprint of two identical clock faces side by side, each with a red cross above twelve o'clock. On the left the sweep is a thick grey arc running most of the way around with the hand near ten o'clock. On the right the sweep is a short thick blue arc and the hand sits just past one o'clock. A dot marks the end of each sweep outside the rim.
· Cedana Editorial

How do I run a proof of concept for GPU checkpointing?

Plan a GPU checkpointing evaluation with a matched baseline, real interruptions, correctness checks and agreed success criteria before expanding the install.

read →
Cedana. Blueprint of two documents side by side. The left carries a large bold cross beside a solid black block at its top and nine lines beneath, three of them solid with ticks and six dashed, with the sixth ringed in dashed red. The right carries two small ruled boxes at its top and nine solid lines each with a blue tick.
· Cedana Editorial

How to read a checkpoint benchmark

Evaluate checkpoint benchmarks using nine disclosures, from capture scope and clock boundaries to storage, repeated runs and performance after restoration.

read →
Cedana. Blueprint of three machines. A small one on the left holds one solid blue square. A wider one in the middle holds eight solid blue squares on a pale-blue tray. On the right three grey dashed machines each hold four dashed squares, linked by short dashed lines and spanned by one dashed bracket above them.
· Cedana Editorial

Does multi-node checkpointing ship today?

Distinguish a workload moving between nodes from one spanning nodes. See Cedana's shipped single-node coverage and the multi-node tier still in design partnership.

read →
Cedana. Blueprint split by a faint dashed line. On the left two identical nodes stand side by side, one holding a blue block, the other a grey one, with three coins and a dotted line beneath the grey. On the right one node holds a blue block and points to a small storage cylinder with a saved-state icon, from which a blue arrow drops to a dashed empty node outline below.
· Cedana Editorial

Hot standby versus checkpoint recovery on the same hardware

Compare a spare GPU node with checkpoint recovery by cost, outage tolerance and saved session state. Choose per service tier and include time to find capacity.

read →
Cedana. Blueprint of a large rounded rectangle, the machine, holding a blue process block with a saved-state icon and three faint boxes beneath it. Three lines cross the machine's right edge to an envelope, a database cylinder and a round service pill outside. Behind the envelope a second envelope is drawn in dashed red, a second red line runs to it, and one row in the cylinder is red.
· Cedana Editorial

What cannot be checkpointed in a GPU workload?

Understand GPU checkpoint boundaries, unsupported resources, version constraints and external side effects your application must handle after a restore.

read →
Cedana. Blueprint of a grid four rows deep and six columns wide, with a saved-state icon and four small glyphs down the left: a GPU card, a chip, a gear, a document. Most cells are solid blue; six are white with dashed red edges and a red cross. Three column headers are solid blue with a white tick; the other three are white with a grey circular arrow.
· Cedana Editorial

Version mismatch and the compatibility matrix

Check the GPU, driver, engine and model versions a checkpoint records. Learn why a supported driver range does not mean checkpoints restore across versions.

read →
Cedana. Blueprint of a track with six hurdles in a row, each a pair of posts and a top bar, and one blue dotted arc clearing all six from end to end. Above the track at the left, two program boxes: one holding a small grey document inside it, the other standing on a solid blue band with a saved-state icon at its end.
· Cedana Editorial

System-level vs application-level GPU checkpointing: the category and the bar

Compare application and system-level GPU checkpoints by saved state, code changes, overhead and six criteria for evaluating a checkpointing claim.

read →
Cedana. Blueprint of four horizontal strata, the layers of state a running job holds, with six vertical core samples drilled into them: a thin one reaching partway, one stopping at a red bar on the top stratum, one taking only the top stratum, one drawn dashed, one that is only a small ring on the bottom stratum, and one solid blue core, marked with a saved-state icon, running through all four.
· Cedana Editorial

Which checkpointing approach brings back the state your job is holding?

Compare GPU checkpointing approaches by the state they save, their limits and the integration work needed to decide whether to build or buy.

read →
Cedana. Blueprint of three processes of one GPU job stacked as bars, each split into a grey CPU half and a blue-outlined GPU half. Under every bar a solid bracket spans the CPU half and another the GPU half, the two tools that cover them. Over all three bars a single dashed bracket, and one blue dashed line dropping through every GPU half at the same point, mark the layer nobody has built for you.
· Cedana Editorial

What CRIU and cuda-checkpoint do when you wire them together yourself

See how CRIU and cuda-checkpoint save GPU workloads, where their support stops, and what your team must build around the open-source tools.

read →
Cedana. Blueprint of two machines facing each other across a gap, a running GPU job in the left one and an empty dashed slot in the right. Five grey arrows leave the left machine and stop at increasing distances, each ending in a short tick, the longest at a red dashed line just before the right machine. One thick blue arrow, marked with a saved-state icon, crosses the whole gap and lands in the slot.
· Cedana Editorial

Does Kubernetes, Docker, Slurm or Nextflow already restore a running job?

Compare what Kubernetes, Docker, Podman, Slurm and Nextflow preserve, what they restart, and where restoring a running GPU job needs additional tooling.

read →

Showing 13–23 of 23 posts