The recovery time you can put in a filing

A recovery time objective for a model endpoint ends at the first request served

The recovery time you can defend is the one you have measured.

A recovery time objective, or RTO, is the time within which a service has to be back after a failure. A recovery point objective, or RPO, is how much work may be lost. For a model endpoint, being back means ready to serve requests, so the time between the process starting and the endpoint answering is where the minutes go.

A frontier model is one of the latest open-weight models trailing the frontier labs (DeepSeek, Kimi, Qwen, GLM). Before one of them answers anything, it fetches its weights into GPU memory, brings up its runtime, initializes communication between its GPUs, profiles its key-value cache, and captures its CUDA graphs. The key-value cache, or KV cache, is the memory a model builds as it works through a session, so it holds that session's accumulated work.

A failure destroys the cache along with everything else in GPU memory, and how much you lose then depends on how you recover. A worker restart loses everything in flight, because none of that work existed anywhere else. A restore brings back the state of the last checkpoint, so what is lost is whatever the worker did after it.

The rules ask for a documented procedure, a target, a yearly test, and a record

The rules that ask for a recovery time were not written for AI, but they apply to systems that support a function the firm depends on, and a model endpoint serving such a function is one of them.

In the EU, the Digital Operational Resilience Act (DORA), Regulation (EU) 2022/2554 has applied to financial entities since January 2025, and ICT in its text means information and communication technology. Article 12(1) requires documented procedures "for the purpose of ensuring the restoration of ICT systems and data with minimum downtime, limited disruption and loss".

Article 12(6) tells financial entities to set "recovery time and recovery point objectives for each function", taking into account whether each function is critical or important, and it adds that "Such time objectives shall ensure that, in extreme scenarios, the agreed service levels are met."

Under Article 11, financial firms have to "test the ICT business continuity plans and the ICT response and recovery plans in relation to ICT systems supporting all functions at least yearly", and that testing has to include "switchovers between the primary ICT infrastructure and the redundant capacity". Firms also have to "keep readily accessible records of activities before and during disruption events".

For a US hospital, the nearest equivalent is the contingency-plan standard of the Health Insurance Portability and Accountability Act (HIPAA), at 45 CFR 164.308(a)(7). The standard covers systems that contain electronic protected health information, so it reaches a model endpoint in a clinical workflow rather than every model the hospital runs. It requires "procedures to restore any loss of data" and, as an addressable item, periodic testing of the contingency plan, but it sets no recovery time.

At a US bank you might expect model-risk guidance to set the recovery time for a model, and it does not. SR 26-2, issued by the US banking agencies on 17 April 2026 in place of SR 11-7, governs how models are validated, sets no recovery time, and states that generative and agentic AI models "are not within the scope of this guidance". So the recovery commitment for a model endpoint comes from the bank's continuity obligations instead.

A restart puts the model's reload time in the plan

Recovery today means restarting the instance. The weights are read back into GPU memory, and the cache is rebuilt from nothing, because no copy of the cache existed outside the GPU.

Alibaba Cloud's documentation for deploying DeepSeek models on its PAI platform says that for the full-version DeepSeek-R1, which needs an 8-GPU node, "the model loading process might take 20 to 30 minutes". Cedana's published benchmark measured native starts of 9.4 to 34.2 minutes across four frontier models on one node of 8x NVIDIA B200 GPUs. We timed each run from engine launch to ready-to-serve.

That reload has to fit inside the service level committed to in the same filing. For a service on a single deployment, a 99.9% objective allows a tenth of a percent of a 30-day month in violation, which is 43.2 minutes out of 43,200. So one 20-minute restart spends 46% of that budget, and one 30-minute restart spends 69%.

Article 12(6) asks the objective to hold in extreme scenarios, and a target set by a reload holds only while incidents stay rare. One 20-minute restart fits inside a 99.9% month, but a second one does not, and neither does a first one in a 99.95% month.

Reload time also increases as the weights grow with every model generation and the cache grows with every longer context window. So the number you file this year is not the number the plan will meet next year.

Testing a reload means sitting through those 20 to 30 minutes every time you exercise the plan. You can do that on spare hardware or in a maintenance window rather than on a live endpoint, but either way it is a long exercise to schedule and staff. In our experience, disaster recovery drills for GPU and AI workloads rarely exist, so the recovery objectives that rest on them go untested. A plan whose only exercise is the next real failure has not met the yearly-test clause.

Restoring a saved checkpoint measures 57 to 70 seconds on the same hardware

A checkpoint is the saved state of the running worker, and restoring it copies that state back into GPU memory. The worker carries on from where it was with its cache and sessions intact, so nothing has to be reloaded or recomputed.

In the same benchmark, restores of the four frontier models took 57 to 70 seconds on that one node with 8x NVIDIA B200 GPUs, timed from the start of the restore of the same fully-initialized engine to ready-to-serve. Kimi-K2.6 gives the direct comparison: it restored in 63 seconds from a 670 GiB checkpoint, compared with a 20.3-minute native start on the same 8x B200 node.

Those runs read their checkpoints from tmpfs, a file system held in the node's own memory rather than on disk. A restore in your own fleet will depend on the size of the checkpoint, the storage it is read from, and the network between them, so measure it there before you write it down.

The RTO clock starts at the failure, so the number your filing needs is the whole interval from failure to ready-to-serve. That interval has two parts: the time to detect the failure and start the resume, which Cedana's control plane does without an operator, and the restore itself. The published 57 to 70 seconds measures only the second part, from the start of the restore to ready-to-serve. The test below measures both.

A resume shortens how long each failure is visible, but it does not change how often GPUs fail, and it does not replace redundant capacity. Both still belong in your availability calculation.

The test that produces the number is the test the rule already asks for

Running the recovery once takes four steps, and the fourth is the number you file.

1. Checkpoint the workload while it is serving.
2. Fail the primary over to a secondary node.
3. Restore the checkpoint on the secondary node.
4. Record the time from the failure to ready-to-serve, including detection, with the hardware, model, and checkpoint size beside it.

The secondary node has to match the primary. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

That sequence is the switchover between primary infrastructure and redundant capacity that Article 11(6)(a) names, and it produces the kind of record Article 11(8) asks for during a real disruption. The recovery runs without an operator and the restore takes seconds rather than the minutes a native start takes, so the exercise is short enough to be a scheduled test instead of an incident.

Before the first test, measure your native start and restore times on your own hardware, and write both down with what each measurement covers. Then put the test on the calendar the rule already requires, and file the time it records.

The number that reaches the filing is then a switchover you have run rather than a reload you have assumed. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We save the state of a running model worker while it serves and bring it back on healthy hardware, so the recovery time in your plan is a restore you have measured and can run again. When risk or compliance asks you for the number and the date it was last tested, both come out of the same exercise.

Related:

• What happens to a training job when a GPU fails
• One node failed and the whole training job died
• What torchrun, torchft and Ray Train restart from when a node dies
• What to do when vLLM or SGLang stops responding and nothing crashed
• One frontier restart burns half a monthly 99.9% error budget