LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 99

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicAll topics

Cedana. Blueprint of eight circles arranged in a ring, each with a small empty square inside. Every circle is joined to every other by a straight blue line, so the middle of the ring is a dense blue mesh while the circles themselves are pale and empty.MoE serving is a network problem, and more GPUs will not fix it
· Cedana Editorial

MoE serving is a network problem, and more GPUs will not fix it

Diagnose compute, memory and interconnect limits in MoE inference. See why activated parameter counts and extra GPUs cannot replace communication measurements.

read →
Cedana. Blueprint of a memory block with grey stripes copied by a blue arrow into a document with the same stripes. From the document, two blue arrows go to a folder and to a bucket inside a large dashed boundary, and a grey dashed arrow leaves the boundary to a cloud shape below it. A small dashed padlock sits under the document.Where checkpoints live, how long they stay, and who can read them
· Cedana Editorial

Where checkpoints live, how long they stay, and who can read them

Choose where GPU checkpoints live, how long they remain and who can read them. Identify the encryption and key-management answers a security review still needs.

read →
Cedana. Blueprint of a bordered panel holding four rows. Three rows, marked with a blue plus and labelled cluster, scheduler and workload, carry blue chips reading plus helm chart one daemonset, plus three lines in slurm.conf, and plus CEDANA_ENABLE equals one. The fourth row, marked with a grey minus and labelled your code, carries a grey chip reading unchanged.What actually changes on my cluster when I install Cedana?
· Cedana Editorial

What actually changes on my cluster when I install Cedana?

Review the Helm install, Slurm plugins and workload opt-in settings Cedana adds to a cluster, with configuration examples and the application-code boundary.

read →
Cedana. Blueprint of a multiplication written in shapes across the frame: a blue block marked twenty gigabytes, times a stack of five pale blue copies, times a grid of sixteen small squares, equals a database cylinder.How much storage does checkpointing a GPU cluster need?
· Cedana Editorial

How much storage does checkpointing a GPU cluster need?

Estimate checkpoint storage from workload memory, retained copies and job count. Review quota controls, storage bandwidth and unanswered lifecycle behaviors.

read →
Cedana. Blueprint of four horizontal slider tracks with end stops, a faint dashed line at their midpoint, a circular restart arrow at the left end and a saved-state icon at the right. On every track a hollow grey knob sits near the left and a solid blue knob sits near the right.When retrying the job is the right call
· Cedana Editorial

When retrying the job is the right call

Decide whether a GPU job needs checkpoints using interruption risk, work at stake, rebuild cost and deadline slack. Keep application idempotency in either path.

read →
Cedana. Blueprint of two rows of small grey blocks, each ending in an outlined output box. The top row is unbroken. The second has one block replaced by a blue save icon marked checkpoint. A bracket joins the two output boxes and the word match sits beside it.How to know a restored GPU workload is correct
· Cedana Editorial

How to know a restored GPU workload is correct

Test whether a restored GPU workload continues the same computation. Compare repeatable runs, check multi-GPU boundaries and record the exact environment.

read →
Cedana. Blueprint of two identical GPU cards at the top joined by a blue dashed arc carrying a saved-state icon. Beneath them a wide dashed band with a faint centre line holds a dozen dark grey dots scattered at different heights, and near its right end one blue dot with a ring around it sits inside the band, under the second card.Does moving a job change its results?
· Cedana Editorial

Does moving a job change its results?

Understand what a checkpoint preserves when a GPU job moves, which versions must match, and how live inputs and ordinary GPU variation affect reproducibility.

read →
Cedana. Blueprint of two fractions. On the left, marked cost per token, a receipt icon sits over a line with a row of twelve blue dots beneath. On the right, marked revenue per megawatt, the same row of blue dots sits over a line with a lightning icon beneath. A blue dashed curve joins the two rows of dots, marked tokens delivered.Cost per token and revenue per megawatt are the same number
· Cedana Editorial

Cost per token and revenue per megawatt are the same number

Calculate cost per delivered token from fleet costs and serving logs, then see how cold starts, failures and stranded capacity also affect revenue per megawatt.

read →
Cedana. Blueprint of one long horizontal bar under a bracket: its left half solid blue with columns of small blue dots rising above it, then a grey segment, then a red-hatched segment, then a dashed empty segment at the right, with nothing rising above any of those three.The fleet's real yield metric is tokens per gigabyte of VRAM
· Cedana Editorial

The fleet's real yield metric is tokens per gigabyte of VRAM

Measure delivered tokens against installed VRAM. See how memory bandwidth, warm replicas, failures and stranded GPUs affect the output of an inference fleet.

read →
Cedana. Blueprint of a dashed worker outline struck out in red above a shelf holding six grey cache blocks, with a long segmented grey bar beneath the shelf leading to a small dashed outline. On the right a large worker outlined in blue, with a saved-state icon above it, holds a dark weights block, three small grey blocks, and six blue cache blocks inside.KV-cache offload versus a saved worker: what each survives
· Cedana Editorial

KV-cache offload versus a saved worker: what each survives

Compare the state preserved by KV-cache offload and a full worker checkpoint. Learn what survives a crash, what must reload, and where prefix reuse still helps.

read →
Cedana. Blueprint of a fraction. Above the line, twenty-four small outlined boxes marked four dollars per GPU-hour, twenty-four hours billed, the first eight and last four hatched grey and two of the middle ones capped dark. Below the line, columns of blue ticked squares marked tasks finished stand under the middle boxes only.Tasks per dollar: the economics of running your own coding models
· Cedana Editorial

Tasks per dollar: the economics of running your own coding models

Compare coding APIs and self-hosted GPUs using cost per completed task. Account for idle node hours, model swaps and task completion on your own traffic.

read →
Cedana. Blueprint of a power line running flat, stepping down at a bolt-shaped signal, holding a lower level with the reduction hatched beneath the original line, then stepping back up. Under it a grey band runs unbroken across the whole width, and beneath that three blue bars stop at the signal, three saved-state icons sit in the gap, and the bars resume at the return.Demand response for GPU fleets: what the programs require
· Cedana Editorial

Demand response for GPU fleets: what the programs require

Review two GPU demand response trials, the workloads they could flex, and why deeper power cuts require saved job state. Separate trial results from design targets.

read →

Showing 13–24 of 99 posts