Can LLM inference scale to zero without paying for warm replicas?

What one warm replica adds to your LLM serving cost

An idle GPU is billed like a busy one, so the floor is a bill for readiness. Baseten's cold-start documentation states the trade: "You pay for that replica while it's idle, but requests avoid a scale-from-zero wait or rejection."

So the height of that floor follows the length of a start. One engineer paying that bill describes acquiring extra GPUs with the models already loaded, purely so that requests can be served quickly, and calls that over-provisioning.

When someone on r/LLMDevs asked whether true scale to zero is feasible for 30B to 70B models in production, the reply was that it rarely happens: "Most teams we see end up with scale-to-one instead, keeping a single warm instance and eating the cost."

A replica is one running copy of a model, loaded onto GPUs, and at the frontier that copy is a whole node: Alibaba Cloud's deployment documentation puts the full-version DeepSeek-R1 on a node of 8 GPUs.

So the monthly bill is GPUs per replica, times hours in the month, times price per GPU-hour. A 30-day month is 720 hours, so an 8-GPU replica burns 5,760 GPU-hours whether it served every request in its region or none of them. On 1 September 2026, GMI Cloud's pricing page listed the B200 from $4.00 per GPU-hour, and Lambda listed its on-demand 8-GPU B200 instance at $6.69 per GPU-hour plus tax. One warm replica for the month is $23,040 at the first price and $38,534 at the second.

A platform serving real traffic holds more than one spare. Take a fleet of 100 replicas at average load with 20 to 80 spare replicas above it, an illustration rather than any customer's fleet. Baseten's engineers have described single models scaling to 200 replicas, and models running 50 replicas in one region against 80 in another. The 20 to 80 spares come from our own experience and our discussions with providers, where over-provisioning above average load runs from 20 to 80% depending on the models a platform hosts.

A reserved rate is lower than either list price and changes none of the arithmetic. Put in your own rate and count.

Every platform ships a min replicas setting

Wherever you serve, the floor is a setting with a name, and the vendor documents it.

Baseten also defaults its scale-down delay to 15 minutes, and Together does not scale up from zero at all, where "A deployment at zero replicas stays at zero until you raise its floor above zero".

RunPod prices the setting in the same breath as it defines it, calling active workers the "Minimum number of workers that remain warm and ready at all times" and adding that "Active workers incur charges continuously, including when idle".

Removing the floor has a cost of its own, which Ray Serve names: "setting min_replicas=0 causes higher tail latencies". KServe defaults minReplicas to 1, and tells you to "set minReplicas to 0" to scale down to zero, "especially for use cases like serving on GPUs".

On Fireworks the first request into a deployment at zero does not wait at all. "When a deployment is scaled to zero and receives a request, the system immediately returns a 503 error while the deployment scales up. Your application should implement retry logic to handle this."

The same pattern runs outside GPU serving, where Amazon's EC2 Auto Scaling guide calls a warm pool "a pool of pre-initialized EC2 instances that sits alongside an Auto Scaling group", and every setting above is that pattern with a GPU price per hour on it.

Autoscaling LLM inference cannot add a replica faster than a replica starts

Letting the autoscaler add replicas when traffic arrives is the obvious answer for inference autoscaling, but each new replica still has to load its weights and initialize before it can serve, so the autoscaler inherits the cold start. Baseten documents the same steps for a scale-up as for a start from zero. The page on where the minutes go when an LLM worker cold starts breaks each one down and times it.

A user reports that "cold starts with bitsandbytes NF4 take 2+ minutes on first load and 45-120 seconds on warm restarts, which kills serverless and autoscaling use cases". Another, on Knative's own serving tracker, cannot measure the wait at all, and reports that Knative has no metric for cold start time.

The number that governs this is the start window, the time from the decision to add a replica until that replica serves its first request. An autoscaler cannot add capacity any faster than that, so the pool has to hold whatever demand can arrive inside one window. If the window is 20 minutes, the decision to scale up has to be made 20 minutes ahead of the need.

Modal states the same relation for its own product: "the size of the buffer you need to absorb spikes of a given magnitude scales with the speed you can create new replicas". Reading that as a rule for any warm pool is our own extension of it.

Ray Serve autoscaling adds a decision lag on top, separate from the cold start. Its upscale_delay_s defaults to 30 seconds and "defines how long Serve waits before scaling up the number of replicas in your deployment". Shortening the start does not shorten that wait, but it does change how much the wait costs you.

What has to be true before an endpoint can go to zero

Scaling to zero, the promise of a serverless GPU platform, removes the running replicas when traffic stops, so the next request waits for a replica to start. Spheron's page puts scale to zero "at the cost of a 40-90 second penalty". The cost does not go away when the replicas do, but moves onto the first request after a quiet period.

Scale to zero already works for small models, and it stops below the frontier. A frontier model, meaning one of the latest open-weight models trailing the frontier labs (DeepSeek, Kimi, Qwen, GLM), takes 20 to 30 minutes to build from nothing on the 8-GPU node it needs, which is where Alibaba Cloud's documentation puts the load of a full-version DeepSeek-R1. So three conditions have to hold before a model that size can be turned off and brought back on demand.

1. The worker has to be saved after the start has finished. By then the weights are in GPU memory, the GPUs have been set up to talk to each other, and the CUDA graphs are captured, so every later start reads that state back instead of building it again. To find out whether your own worker is saved or merely restarted, stop a warmed replica and bring it back. If the weights reload and the graphs recompile, what you have is a restart.
2. The restore has to be short enough that the first request can wait for it.
3. Something outside the worker has to decide when to save it and what brings it back. A worker sitting at zero is not running, so the request that should wake it cannot reach it.

That third condition is the one you have to wire into your own service. Cedana's documented policies for Kubernetes run as a trigger, then a filter that says which workloads it applies to, then an action: one checkpoints on a schedule you set, a second checkpoints on a webhook, and a third restores when a node is preempted. None of them catches a first request for a worker at zero, so that wake comes from the autoscaler, the router, an agent, or a policy on your own service, and holding the request while the worker comes back is that component's job, not Cedana's.

What changes when a replica restores from a checkpoint

We measured the same worker started both ways, from nothing and from a checkpoint. Every run used one node with 8x NVIDIA B200 GPUs and 1.7 TB of system memory, on CUDA 12.9, with each of four frontier models served by SGLang from its official recipe. Started natively, from engine launch to ready to serve, the four took 564 to 2,051 seconds, which is 9.4 to 34.2 minutes. Restored from a checkpoint of that same finished engine, they were ready to serve in 57 to 70 seconds.

Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. The checkpoints ran from 244 to 873 GiB while the restores ran 57 to 70 seconds, so on that node the restore time barely moved across a 3.6-fold range of sizes. They were read from tmpfs, a file system that lives in the node's own memory, so those times leave out a read from disk or across a network, and a checkpoint held elsewhere comes back at the speed of that path. A checkpoint does not make the weights load from storage any faster.

So the startup wait becomes the restore time, and whether your endpoint can take that wait depends on your traffic. An in-house coding assistant, an agent session, a batch job, and an endpoint that goes quiet overnight can all wait about a minute for a first response. A sub-second interactive endpoint cannot, so it keeps its warm replica.

For a busy model the pool does not reach zero either, because demand can still jump inside a restore of about a minute. What changes for that pool is where the standby sits. A checkpoint costs storage while it waits, and the GPU bill starts when it is restored onto GPUs, so the standby moves off GPUs and onto NFS, S3, or Cedana's own storage.

What a checkpoint does not remove

The checkpoint has to exist before any of this works, and making it means paying one cold start. That start runs once for each model and each version, so a new model or a new version pays it again.

A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

Every figure above was measured on one node. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. A frontier replica on one node of 8 GPUs sits inside the tier that ships.

What sets the floor on your endpoint is how long the next worker takes to become ready. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On an inference endpoint, that means we save a worker once it has finished initializing, so the next one reads that state back instead of running the start again. Your floor then covers the restore time and whatever demand arrives inside it, not a start that runs tens of minutes, and if the first request on your endpoint can wait for that restore, the floor can be zero.

Related:

• Where the minutes go when an LLM worker cold starts
• How to swap models on one GPU without a cold start
• What NVIDIA Dynamo Snapshot and Modal GPU memory snapshots restore today
• Your KV cache now rivals your weights
• Does disaggregated serving remove the need to move GPU workers?
• KV-cache offload versus a saved worker: what each survives

Common questions

Cold start is slow enough that we keep a replica warm and pay for the idle GPU. How do I get fast starts without paying to keep GPUs idle?

Save the worker after it has finished initializing, so the next start reads that state back instead of building the engine again. On our benchmark, four frontier models on one node of 8 NVIDIA B200 GPUs took 564 to 2,051 seconds to start natively and 57 to 70 seconds to restore from a checkpoint of the finished engine, so your floor covers the restore time rather than a start that runs tens of minutes. The saved worker has to come back on a matching node. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

My autoscaler adds a replica once traffic picks up, but the spike has passed by the time it finishes starting. How do I make autoscaling work?

An autoscaler cannot add capacity faster than a replica starts, so the pool has to hold whatever demand can arrive inside one start window, the time from the decision to add a replica until it serves its first request. Shortening the start is what shortens that window: on our benchmark a replica restored from a checkpoint of a finished engine was ready to serve in 57 to 70 seconds, against 564 to 2,051 seconds started natively. The autoscaler's own decision lag is separate and does not move, so on Ray Serve the upscale_delay_s default of 30 seconds still runs before it scales up at all.