TL;DR
- Running inference on hardware you own puts four duties on your team: availability, capacity planning, token economics, and audits.
- One 8-GPU node has about a 47% chance of at least one interruption in a 30-day month. A 99.9% availability target leaves you 43.2 minutes of violation to spend, for a service on a single deployment.
- A rebuild of 5 to 15 minutes spends between 12 and 35% of that allowance, so one interruption at the middle of the range takes roughly a quarter of the month.
- The usual answer is a second node held warm. That doubles what you buy, and it covers the hardware rather than the sessions in flight, which are lost either way. Saving the state of the running workload turns the rebuild into a copy instead. Cedana's published benchmark measured restores of 57 to 70 seconds on one 8x B200 node.
- In this piece we walk through the four duties, the interruption rate for one 8-GPU node, and what a rebuild costs against a 99.9% month. Then we cover what a warm standby covers, and what changes when the state survives the node it was running on.
Three duties move to your team, and a fourth changes shape
When you buy the hardware, three jobs the provider used to do move to your team, and the audit expectation you always had starts to apply to a system you operate yourself.
| The duty | Who carried it on a provider's API | Who carries it on hardware you own |
|---|---|---|
| Availability | The provider. A node that failed inside its fleet was its incident, and you saw an error rate. | You. A node that stops is your outage, measured against the target the business holds you to. |
| Capacity planning | The provider. It decided how much hardware stood behind the tokens it sold. | You. You decide how much spare capacity to hold, and you pay for it while it waits. |
| Token economics | The provider. It set a price per token and carried the cost of producing one. | You. The question becomes whether enough is coming out of those GPUs to justify owning them, counted in completed tasks per dollar. |
| Audit expectations | The provider, for the systems it ran. | You, for a system you now run, under the standards your industry already applies to production systems. |
Maintenance arrives with the availability duty, because patching or upgrading means shutting those GPUs down, and they cost money by the hour whether or not they are serving anything. Whatever was running on them stops when they stop.
The audit expectation arrives as paperwork rather than as an incident. Regulated industries are starting to ask whether stateful AI workloads can meet the operational-continuity standards they already apply to their other digital systems, so what reaches your team is a request for a written recovery procedure and a recovery time you have measured rather than assumed.
Of the four, availability is the one you can size on paper in advance, and the calculation that follows is for a single 8-GPU node.
One 8-GPU node has roughly an even chance of an interruption in any month
The public number that fits a fleet this size comes from Meta: Kokolis and colleagues studied 11 months of operations on two of Meta's machine-learning research clusters, on A100 hardware, and report a mean time to failure (MTTF) of 47.7 days for 8-GPU jobs against 7.9 hours for 1024-GPU jobs.
Those failures are counted from every cause, from user programs through system software to hardware faults, so 47.7 days is a mean time between interruptions of every kind rather than a rate at which GPUs break. The figure also describes a job on a research cluster rather than a node in an inference fleet, so treat it as a planning input for an inference node of that size rather than as a measurement taken from one.
The authors flag the small-job end of their own range as its least dependable part: "for smaller jobs, we observe less predictable MTTFs, mostly due to experimental usage patterns that cause correlated NODE_FAIL." Allow for that uncertainty when you plan from the 8-GPU figure.
Assuming interruptions arrive at a constant average rate, a 47.7-day mean gives one node about a 47% chance of at least one interruption in a 30-day month. That is a chance rather than a schedule, so it says nothing about which month the interruption lands in.
One rebuild spends about a quarter of a 99.9% month's allowance
A 99.9% availability target leaves you 43.2 minutes of violation in a 30-day month, for a service on a single deployment. A 30-day month holds 43,200 minutes, and the target allows one thousandth of them. Tighter targets shrink the allowance on the same scope, to 21.6 minutes at 99.95% and 4.32 minutes at 99.99%.
What one interruption spends out of that allowance is set by how long your model takes to come back, and the 5 to 15 minute range in the table is a planning estimate for a rebuild on a node like yours. Every share in the table rests on that estimate. Cedana's published benchmark, further down, measured native starts of 9.4 to 34.2 minutes for frontier models on an 8-GPU B200 node, so if your model is one of those, put its measured start in the rebuild column instead. Both columns are scoped to a service on a single deployment.
| Rebuild time | Share of a 43.2-minute allowance, at 99.9% | Share of a 21.6-minute allowance, at 99.95% |
|---|---|---|
| 5 minutes | 12% | 23% |
| 10 minutes | 23% | 46% |
| 15 minutes | 35% | 69% |
At the middle of the range, one interruption spends roughly a quarter of a 99.9% month's allowance. Everything else that takes your endpoint down in the same month draws on the same 43.2 minutes, so the margin on one node is thin.
Three answers are open to you, and the choice belongs to whoever signs for the hardware.
- Accept a lower availability target, which is a reasonable answer for plenty of services.
- Hold a second node warm.
- Run a workload whose state survives the node it was running on, so it can be brought back on another node.
A warm standby covers the hardware and not the work
A second node held warm is real redundancy and it works. Traffic moves to hardware that is already running, so once the failover completes a hardware failure on the first node need not be visible to the people using the service. That is the usual pattern in inference, and it is why serving teams keep so many replicas.
The standby is also a second node bought to insure the first, so the fleet doubles and the insurance half produces nothing while the primary is healthy. Put that next to the lower availability target in the business case, because the two are answers to the same question.
The standby does not save the work that was in flight. The sessions on the failed node are destroyed whether or not a second node is standing by, so a user whose session was on that node starts again on the standby.
Saving the state replaces the rebuild time with a copy time
Saving the whole state of a running workload so it can be brought back later is called checkpointing. If the state is captured while the workload runs, an interruption costs you the time to copy that state back rather than the time to build the model up from nothing. The capture happens below the application, so nothing in the application changes. Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count.
Cedana's published benchmark measures that copy. Every run used one node with 8x NVIDIA B200 GPUs, the models were served by SGLang from its official recipes, and the checkpoints were held on tmpfs, which is memory rather than disk. Restores of the models in that set took 57 to 70 seconds, on a clock that ran from the saved checkpoint to the engine being ready to serve.
A resume of about a minute is roughly 2% of the same 43.2-minute allowance, for a service on one deployment. The 2% is arithmetic on a one-minute assumption, and the published 57 to 70 seconds is what makes a minute realistic to plan with.
The benchmark measured the restore itself, but the allowance counts the whole outage, so it also includes the time to notice the failure and to find a free node with the same GPUs, driver, and engine. The checkpoint has to be reachable from that node as well. This is still a single deployment with no standby, so the free node has to exist, and what the checkpoint removes is the rebuild rather than the need for hardware to rebuild on.
Nothing else in the arithmetic moves, because interruptions arrive as often as they did and a standby is still a standby. What changes is the number you put in the rebuild column. A ten-minute initialization becomes a state copy.
Measure your own rebuild time before you sign
Three numbers of your own settle the question before the purchase order clears: how many nodes you will run, the availability target the business will hold you to, and how long a rebuild takes on the hardware you are about to buy. Measure that last one rather than carrying an estimate into the plan, because it is the number that decides whether one node can hold the target.
Checkpointing is what changes that last number. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a node like yours, that means the state of a running workload is captured and brought back on healthy hardware, as of its last checkpoint, so what goes into your availability arithmetic is a copy time rather than a rebuild time. Put that copy time into the arithmetic in place of your rebuild time, and you will see whether the node you are about to buy can hold your target on its own.
Related:
- Which utilization number goes into your own-versus-rent calculation?
- How to put a dollar figure on the GPU-hours that produced nothing
- Tasks per dollar: the economics of running your own coding models
- Where the minutes go when an LLM worker cold starts
Common questions
How much is idle, queued, or unused GPU capacity costing us?
In this arithmetic the idle capacity is the warm standby, a second node bought to insure the first, which produces nothing while the primary is healthy. The standby does not save the work that was in flight either, so a user whose session was on the failed node starts again on the standby regardless. To price that idle half at your node's hourly rate, use How to put a dollar figure on the GPU-hours that produced nothing.
