LoginSchedule demo
cedana / blog · field reports & engineering deep-dives

Writing from the
cedana team.

Engineering notes, benchmarks, and field reports covering how we build the automation layer for AI factories.
All posts · 99

insights from the team.

Field reports, engineering deep-dives, and benchmarks from the Cedana team.

TopicAll topics

Cedana. Blueprint of two long dashed outlines marked requested, each holding a solid blue fill marked used: a row of eight GPU cells with only two lit and the rest hatched, and a memory bar filled a third of the way with the remainder hatched.Why everyone over-requests memory on a shared cluster
· Cedana Editorial

Why everyone over-requests memory on a shared cluster

Understand why cluster users request extra memory, how Slurm limits and sampled peaks affect sizing, and what changing an allocation costs.

read →
Cedana. Blueprint of three panels headed by an icon and a word: a red cross marked kill, a pause glyph marked pause, a blue save icon marked checkpoint. Each panel is a machine's memory stacked to a dashed limit with a process column beside it: emptied and struck out under kill, still dark and full under pause, and emptied with its top block rising in blue to the save icon under checkpoint.Can Linux pause a process instead of killing it when memory runs out?
· Cedana Editorial

Can Linux pause a process instead of killing it when memory runs out?

Learn why pausing a process does not free RAM, what earlyoom and systemd-oomd can do, and when checkpointing must act to preserve running work.

read →
Cedana. Blueprint of the same node drawn three times in a row: on the left holding three blue session blocks, in the middle standing empty inside a dashed grey window with a nut-and-spanner mark above it, on the right holding the three blue blocks again. Three saved-state icons sit above the empty node, and blue dashed arrows carry the sessions up from the first node into them and down from them into the third.Draining a GPU node in Kubernetes without losing the work on it
· Cedana Editorial

Draining a GPU node in Kubernetes without losing the work on it

Understand what a Kubernetes drain does to GPU pods, how disruption budgets affect it, and how to plan checkpoint and restore around maintenance.

read →
Cedana. Blueprint of a bar chart of node occupancy by day: high grey bars that fall step by step over four days to almost nothing at a dashed grey window, then climb again after it. The last sliver before the window is red and struck out. Blue dashed outlines stand on top of the falling bars up to the height they would have kept, and a saved-state icon sits above the window.Rebooting Slurm nodes for a kernel update without losing the running jobs
· Cedana Editorial

Rebooting Slurm nodes for a kernel update without losing the running jobs

Compare Slurm reboot and reservation workflows, account for the idle time before maintenance, and plan checkpoint recovery around a kernel update.

read →
Cedana. Blueprint of three columns of three nodes. The left column is tinted blue and empty, freshly upgraded. The middle and right columns are outlined in ink and hold grey work blocks, with a blue block beside each grey one in the middle column where work arrived from the left along blue dashed arrows. A red dashed line runs between the left column and the rest, and a grey arrow trying to cross it back ends in a red cross.Why an NVIDIA GPU Operator upgrade waits for your workloads
· Cedana Editorial

Why an NVIDIA GPU Operator upgrade waits for your workloads

Learn why GPU Operator upgrades wait for active workloads, what causes driver pods to stall, and how compatibility limits shape a rolling upgrade.

read →
Cedana. Blueprint of an allocation drawn as a clock face, a thick blue arc of work running almost the whole way round to a red cut marked time limit. Two arrows leave to the right: a grey dashed one to an empty bar labelled requeue, and a blue one to a bar filled almost full, labelled checkpoint.Your Slurm job was cancelled due to time limit. What to do now
· Cedana Editorial

Your Slurm job was cancelled due to time limit. What to do now

Confirm a Slurm TIMEOUT, check what progress survived, and compare checkpoints, requeue and job chains for runs that exceed the wall-time limit.

read →
Cedana. Blueprint of a two-by-two grid. Along the top, a GPU card with its bar filled and one with its bar empty; down the side, a job block with a pause mark and a red dashed job block struck out. The top-left cell holds the paused job with the held GPU; the bottom-right holds the struck-out job with the freed GPU; the bottom-left is a faint dash; and the top-right, tinted blue, holds a saved-state icon with an arrow to a freed GPU outlined in blue.Slurm has no suspend option for GPU jobs. What preemption without killing the job looks like
· Cedana Editorial

Slurm has no suspend option for GPU jobs. What preemption without killing the job looks like

Learn why Slurm suspend keeps GPUs allocated, how requeue and grace time work, and where checkpointing can preserve a preempted job's progress.

read →
Cedana. Blueprint of the same workload silhouette four times in a row: a GPU card over a wider process box holding lines of a program, over a short pipeline of four small steps. In the first only a dark strip inside the process is filled. In the second the first two pipeline steps are filled and the third is dashed. In the third the process box is filled dark and a red slash crosses the GPU card. In the fourth every part is solid blue under a saved-state icon.DMTCP, application checkpoints, workflow managers and system-level checkpointing: what each covers on a Slurm cluster
· Cedana Editorial

DMTCP, application checkpoints, workflow managers and system-level checkpointing: what each covers on a Slurm cluster

Compare application checkpoints, workflow resume, DMTCP and system-level GPU checkpoints for Slurm jobs, including setup responsibilities and limits.

read →
Blueprint roofline plot: performance rises along the memory-bandwidth slope to a ridge point, then flattens at the compute ceilingBlueprint roofline plot: performance rises along the memory-bandwidth slope to a ridge point, then flattens at the compute ceiling
Updated · Cedana Editorial

Roofline Analysis and the Inference Value Chain

Once you stop renting intelligence, you own performance.

read →
Blueprint schematic of an eight-GPU node accumulating session state, with a state gauge rising toward the VRAM capacity lineBlueprint schematic of an eight-GPU node accumulating session state, with a state gauge rising toward the VRAM capacity line
Updated · Cedana Editorial

The Era of Stateful Inference: How to Improve Cost per Token.

We are entering the stateful inference era, driven by frontier models with longer context windows, longer in-flight sessions, and single instances spanning 8, 16, or more GPUs.

read →
Blueprint fleet of GPU cells with only a third lit, next to a utilization gauge stuck at a dashed 30 percent ceiling and a second gauge breaking through itBlueprint fleet of GPU cells with only a third lit, next to a utilization gauge stuck at a dashed 30 percent ceiling and a second gauge breaking through it
Updated · Cedana Editorial

The Utilization Ceiling: Why AI and HPC Schedulers Hit 30% and How to Fix This

GPU utilization across AI and HPC workloads is fundamentally capped at 30% because schedulers cannot migrate running jobs. Cedana's CPU and GPU migration capability surpasses this limitation, unlocking near-full utilization.

read →
Blueprint chart of three trajectories diverging from a point, with an accent curve rising steeply off the chartBlueprint chart of three trajectories diverging from a point, with an accent curve rising steeply off the chart
· Niranjan Ravichandra

The Fork in the Road Post GPT-5

On GPT-5 being underwhelming, scaling laws and where we can get reasonable progress from. Also, should AGI still be the goal?

read →

Showing 85–96 of 99 posts