LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 99

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicAll topics

Cedana. Blueprint split by a faint dashed line. On the left two identical nodes stand side by side, one holding a blue block, the other a grey one, with three coins and a dotted line beneath the grey. On the right one node holds a blue block and points to a small storage cylinder with a saved-state icon, from which a blue arrow drops to a dashed empty node outline below.Hot standby versus checkpoint recovery on the same hardware
· Cedana Editorial

Hot standby versus checkpoint recovery on the same hardware

Compare a spare GPU node with checkpoint recovery by cost, outage tolerance and saved session state. Choose per service tier and include time to find capacity.

read →
Cedana. Blueprint of a large rounded rectangle, the machine, holding a blue process block with a saved-state icon and three faint boxes beneath it. Three lines cross the machine's right edge to an envelope, a database cylinder and a round service pill outside. Behind the envelope a second envelope is drawn in dashed red, a second red line runs to it, and one row in the cylinder is red.What cannot be checkpointed in a GPU workload?
· Cedana Editorial

What cannot be checkpointed in a GPU workload?

Understand GPU checkpoint boundaries, unsupported resources, version constraints and external side effects your application must handle after a restore.

read →
Cedana. Blueprint of a key drawn in blue, its bow a saved-state icon and its blade cut with four teeth of different heights. A blue line leads from it to a node box outlined in blue whose bottom edge is cut with four matching notches and a filled blue tick. A grey dashed line leads to a second node box below, outlined in ink, whose third notch is red and shallower; the line ends in a red cross, and a grey circular restart arrow sits inside the box.Driver, CUDA, and engine upgrades with workloads running, and the one limit
· Cedana Editorial

Driver, CUDA, and engine upgrades with workloads running, and the one limit

Plan rolling GPU driver, CUDA and engine upgrades around compatible capacity. Learn why existing checkpoints cannot carry a workload across a version change.

read →
Cedana. Blueprint of a grid four rows deep and six columns wide, with a saved-state icon and four small glyphs down the left: a GPU card, a chip, a gear, a document. Most cells are solid blue; six are white with dashed red edges and a red cross. Three column headers are solid blue with a white tick; the other three are white with a grey circular arrow.Version mismatch and the compatibility matrix
· Cedana Editorial

Version mismatch and the compatibility matrix

Check the GPU, driver, engine and model versions a checkpoint records. Learn why a supported driver range does not mean checkpoints restore across versions.

read →
Cedana. Blueprint of a track with six hurdles in a row, each a pair of posts and a top bar, and one blue dotted arc clearing all six from end to end. Above the track at the left, two program boxes: one holding a small grey document inside it, the other standing on a solid blue band with a saved-state icon at its end.System-level vs application-level GPU checkpointing: the category and the bar
· Cedana Editorial

System-level vs application-level GPU checkpointing: the category and the bar

Compare application and system-level GPU checkpoints by saved state, code changes, overhead and six criteria for evaluating a checkpointing claim.

read →
Cedana. Blueprint of three grey job bars at staggered heights all ending just short of a red dashed vertical line under a red calendar icon marked patch date, each with a blue save icon at its end marked checkpoint. A narrow dashed column with a wrench icon stands just past the line, and blue bars continue from the far side of it.Patch the GPU cluster on the security calendar, not the job calendar
· Cedana Editorial

Patch the GPU cluster on the security calendar, not the job calendar

Plan GPU security patches around the bulletin deadline. Move compatible workloads before maintenance and account for the last nodes crossing a driver upgrade.

read →
Cedana. Blueprint of three pairs of bars on one baseline, each pair a grey bar beside a blue one of different height, with one dashed horizontal line running unbroken across all three at the same height. Beneath, a ruled ninety-day strip with a black marker at its midpoint and a saved-state icon under it, and a small ticked box just before the strip begins.A 90-day wall-time PoC: what to measure before you change policy
· Cedana Editorial

A 90-day wall-time PoC: what to measure before you change policy

Design a Slurm checkpointing proof of concept using accounting history, application review and agreed thresholds for completion time, queue impact and exceptions.

read →
Cedana. Blueprint of four horizontal strata, the layers of state a running job holds, with six vertical core samples drilled into them: a thin one reaching partway, one stopping at a red bar on the top stratum, one taking only the top stratum, one drawn dashed, one that is only a small ring on the bottom stratum, and one solid blue core, marked with a saved-state icon, running through all four.Which checkpointing approach brings back the state your job is holding?
· Cedana Editorial

Which checkpointing approach brings back the state your job is holding?

Compare GPU checkpointing approaches by the state they save, their limits and the integration work needed to decide whether to build or buy.

read →
Cedana. Blueprint of two long bars running toward a red dashed wall on the right. The top bar is filled blue to just short of the wall, with a small blue document above its end and a black switch below it with the knob to the right. The bottom bar is filled grey all the way to the wall and ends in a red cross, with a dashed empty switch below it and, beneath that, a blue bracket spanning the bar with a saved-state icon near its end.GROMACS shows both the value and the limit of application checkpointing
· Cedana Editorial

GROMACS shows both the value and the limit of application checkpointing

Use GROMACS maxh, cpt and cpi to understand checkpointing across allocation limits, and see what other applications must build to offer the same recovery path.

read →
Cedana. Blueprint of ten ledger rows of small grey fields, four of them tagged red at the left with two of their fields outlined red. Beside each tagged row a dark grey bar of a different length extends to the right, and under the column of rows a single blue bar the width of the longest.Measuring the cost of wall-time termination from your sacct data
· Cedana Editorial

Measuring the cost of wall-time termination from your sacct data

Use sacct to count Slurm TIMEOUT records, calculate exposed node-hours and identify repeat jobs. Separate accounting evidence from proof that work was lost.

read →
Cedana. Blueprint of three processes of one GPU job stacked as bars, each split into a grey CPU half and a blue-outlined GPU half. Under every bar a solid bracket spans the CPU half and another the GPU half, the two tools that cover them. Over all three bars a single dashed bracket, and one blue dashed line dropping through every GPU half at the same point, mark the layer nobody has built for you.What CRIU and cuda-checkpoint do when you wire them together yourself
· Cedana Editorial

What CRIU and cuda-checkpoint do when you wire them together yourself

See how CRIU and cuda-checkpoint save GPU workloads, where their support stops, and what your team must build around the open-source tools.

read →
Cedana. Blueprint of two machines facing each other across a gap, a running GPU job in the left one and an empty dashed slot in the right. Five grey arrows leave the left machine and stop at increasing distances, each ending in a short tick, the longest at a red dashed line just before the right machine. One thick blue arrow, marked with a saved-state icon, crosses the whole gap and lands in the slot.Does Kubernetes, Docker, Slurm or Nextflow already restore a running job?
· Cedana Editorial

Does Kubernetes, Docker, Slurm or Nextflow already restore a running job?

Compare what Kubernetes, Docker, Podman, Slurm and Nextflow preserve, what they restart, and where restoring a running GPU job needs additional tooling.

read →

Showing 49–60 of 99 posts