One frontier restart burns half a monthly 99.9% error budget

One restart costs half to two-thirds of a 99.9% monthly budget

A 30-day month has 43,200 minutes. If you hold a time-based availability SLO (service level objective) of 99.9% on a single deployment, 0.1% of those minutes may be violated, which is 43.2 minutes for the whole month. A restart that takes 20 minutes uses 46% of that budget, and one that takes 30 minutes uses 69%.

A tighter target leaves you less room, and at 99.99% one restart is already past the month's allowance before you count anything else.

Each row is scoped to a service on a single deployment and assumes one incident and a full restart.

Replicas move the cost rather than remove it. If you run replicas, a failed worker drops capacity instead of breaching the SLO, so less of the availability budget is at risk and you pay an infrastructure cost instead. Hiding the restart means holding a warm standby, a second worker with the model already loaded and paid to sit idle all month, and it can take traffic at once.

What the standby does not have is the failed worker's in-flight sessions, so those requests start over. Once the standby is serving, it is no longer spare, so covering the next failure means another worker going through the same 20-to-30-minute load.

Either way, restart time is the cost you are trying to reduce. On one deployment it lands on the error budget, and behind replicas it lands in the idle worker you keep warm.

Recovery takes 20 to 30 minutes because it is a full restart

Recovery from a failure is a full restart, and a frontier model's startup is a long chain of steps, each waiting on the one before it. Alibaba Cloud's documentation for deploying DeepSeek models says that on a full-version DeepSeek-R1, "the model loading process might take 20 to 30 minutes". That figure covers the model load alone, on the 8-GPU node the full model requires, so it is the floor for the restart rather than the whole of it.

On a serving stack, the model load is one step in a sequence that runs from placement to the first token:

1. Placement across GPUs, or across nodes for a model that needs more than one node.
2. Runtime startup.
3. Fetching the weights.
4. Materializing the weights in GPU memory.
5. Initializing NCCL (the NVIDIA Collective Communications Library) and the rest of the distributed communication.
6. Profiling the key-value (KV) cache, the memory that holds the working state of every request in flight.
7. Capturing the CUDA graphs.
8. Prefill, and the first token.

A failure discards the initialized worker, the KV cache, and the in-flight computation, which had no copy anywhere. Nothing above the GPU keeps a copy of that state, so an interrupted request is re-run from the beginning rather than continued.

GhostServe, a paper on fault-tolerant serving presented at MLSys 2026, describes the loss: "In the event of a failure, this volatile state is lost, forcing the system to restart the inference job from the very beginning."

Both inputs to the restart keep growing

The two inputs that set recovery time, model size and context length, are both increasing. Model size sets how much has to be reloaded. Kimi K2.6 is a trillion-parameter mixture-of-experts model whose official serving configuration is a single H200 node with 8 GPUs, so before any session state arrives the worker already occupies a full node with its weights, and every step in the sequence above scales with model size.

Context length sets how much cached state has to be rebuilt for the requests that were active. DeepSeek-V4's published context length is 1 million tokens, and Together AI's bring-up notes put a full HGX B200 node's usable KV capacity at roughly 1.2 to 3.7 million tokens, depending on cache policy. A single request at that context length can therefore occupy a large share of a node's cache, and whatever was cached is recomputed after a restart.

More weights to reload and more cache to rebuild mean a longer restart, so a fixed monthly budget absorbs fewer restarts as models and contexts grow.

A resume from a checkpoint costs about a minute

A resume skips the model-load path entirely. It copies the saved state of the running worker back into GPU memory and continues from where the worker was, with the cache and the in-flight sessions intact.

Cedana ran the benchmark behind these numbers, on 8x B200 GPUs. Kimi K2.6 resumed in 63 seconds from a 670 GiB checkpoint, compared with a 20.3-minute native start on the same hardware. Across the models in the published set, restores took 57 to 70 seconds, including a 1.6-trillion-parameter model that natively takes 34 minutes.

Both numbers were timed to the same endpoint, ready-to-serve. Every run was on one node with 8x B200 GPUs and 1.7 TB of system memory, and the models were served with SGLang using its official recipes. The native cold start was timed from engine launch, including weight loading and full engine initialization, and the restore was timed from the start of restoring the same fully initialized engine.

Recovery time in your own fleet will also depend on the checkpoint's size, the storage it is read from, and the network between them, so measure it there.

Against a 43-minute budget, a resume of about a minute is roughly 2% of the monthly budget. That figure is the restore alone, and detection and placement come on top, so the room it leaves for further failures is less than the arithmetic suggests.

A resume does not add nines by itself. Reliability nines come from how often failures happen, how visible each failure is to users, and how much redundancy stands behind the service, and a resume shrinks the middle term, the visible impact of each failure, by collapsing the recovery time. The other two terms do not move, because the failure rate is set by the hardware and the redundancy by the architecture.

The SLO arithmetic is yours to do. Use about a minute of recovery instead of half an hour as the input.

The recovery time you promise becomes a number you can test

With a checkpoint in place, recovery time stops being an estimate you defend in a meeting and becomes a measurement you can rerun. A recovery time objective, or RTO, is the time within which a service must be back after a failure, and teams in regulated industries are asked to state one for critical systems and show that it holds.

A recovery objective built on a 20-to-30-minute reload holds inside a 99.9% budget only while incidents stay rare. One incident fits inside the 43.2 minutes, but a second one, or one incident plus any other downtime that month, takes the budget to its edge or past it. An objective built on a one-minute restore can be tested on a schedule.

The reload time is not a number you control, because it tracks the model and the context. A restore follows the size of the saved state rather than the size of the model.

So divide your measured restart time by your SLO's monthly error-budget minutes. If the answer is more than half, restart time is the number worth changing, because the month's minutes are fixed and the failure rate is set by the hardware.

With a checkpoint, the state survives, so the cost of the failure is the time it takes to resume. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We save the state of a workload while it is running, so recovery restores that state on a healthy GPU instead of loading the model again from nothing. On your own deployment that changes one input to the arithmetic above, the restart time you measure, and leaves the error budget you were given where it was.

Related:

• What happens to a training job when a GPU fails
• One node failed and the whole training job died
• What torchrun, torchft and Ray Train restart from when a node dies
• What to do when vLLM or SGLang stops responding and nothing crashed
• The recovery time you can put in a filing