Why a healthy inference worker can still be in the wrong place

A scheduler that only counts GPUs does not see NVLink or PCIe

Two free GPUs are not interchangeable with two other free GPUs. Inside a node, GPUs talk to each other over NVLink or PCIe, and across nodes they talk over the network, so where the parts of a workload land decides how fast those parts can talk to each other. A scheduler that counts free GPUs does not account for that layout.

Ignoring the layout can cost about half the throughput. An issue on NVIDIA's Dynamo tracker reports that "DGD lacks NUMA/PCI-root topology awareness for prefill/decode placement (~50% throughput loss, ~30x latency)".

Topology-aware placement uses connectivity alongside the GPU count. In NVIDIA Run:ai, node labels describe racks, blocks, and NVLink domains, so the scheduler can place a distributed workload's pods close together. Those labels are read when the workload is admitted, so they decide where the worker starts and leave it wherever it started.

Load imbalance in a decode pool shows up as tail latency

A decode pool goes lopsided on its own, because the requests it serves are not the same size. A decode instance is one running copy of the model that generates output tokens for its requests, one token at a time. Every request it serves holds a key-value cache, or KV cache, in GPU memory, and that cache is the attention state of the session so far, so it grows with every token the answer adds.

Nobody knows at admission how long an answer will run. A request may stop after a sentence or work through a long reasoning trace, emitting tokens long after the shorter requests have finished, and the instance serving the long ones accumulates their caches and holds them until those answers finish.

So the imbalance lasts at least as long as the slowest request on that instance. While it lasts, that one worker sets your tail latency and the other GPUs stay idle, because P99 latency is set by the tail of the request distribution rather than by the typical request. Your fleet average stays healthy the whole time.

Once memory runs short, that instance starts preempting requests and rebuilding caches it had already built once.

A well-loaded worker can still be on the wrong silicon

A worker can be fully loaded and still be on hardware that suits the other phase of the work. Prefill and decode are bound by different resources: prefill runs every prompt token in one parallel pass, so it is compute-bound, while decode emits tokens one at a time, and each token streams the full weights and a growing KV cache out of memory. Splitwise, the paper on splitting the two phases across machines, says of the tokens after the prompt that "This makes the subsequent token generation more memory bandwidth and capacity intensive."

GPU models differ in compute capacity and memory performance. NVIDIA lists 1,979 TFLOPS FP16 and 3,958 TFLOPS FP8, with sparsity, for the SXM part of both the H100 and the H200. It lists 4.8 TB/s of memory bandwidth and 141 GB of HBM3e on the H200, against 3.35 TB/s on the H100 SXM.

You do not pick the silicon when the worker starts, because the worker is deployed on whichever GPUs are available at the time. The fleet's hardware mix changes afterwards, and so does the traffic's mix of prefill and decode. So a well-loaded worker can end up running decode on compute-rich, bandwidth-poor GPUs while bandwidth-rich silicon runs prefill.

Relabelling the nodes does not move a worker that is already running. Kubernetes node affinity uses node labels to constrain where a pod is scheduled, and its IgnoredDuringExecution behavior leaves a running pod in place if those labels later change, so labels that guide admission do not move the worker already on the wrong node. Moving that worker to the other part is not a restore either, as the last section explains: an H100 checkpoint restores only on H100 capacity.

A health check cannot see a worker in the wrong place

A health check asks one narrow question, whether the worker is up and the GPU behind it is sound. A worker with its GPUs on the wrong side of a bus, a worker buried under long requests, and a worker running decode on the wrong silicon all pass that check.

NVIDIA's fault-management system for GPU clusters, NVSentinel, draws the same boundary. Its documentation says that "By default, only health monitoring is enabled." The GPU health monitor it describes "watches for thermal issues, ECC errors, and XID events". Every item on that list is a fault in the hardware, so none of them says whether the worker is in the right place. The waste never trips an alarm.

Topology-aware scheduling decides the first placement

Vendors do answer the topology question, and every answer decides where a workload is admitted. NVIDIA Run:ai's documentation states its scope: "Topology-aware scheduling in NVIDIA Run:ai is applied at the workload level."

Nebius builds its version on Kueue's topology constraints and gives it two modes. In Required mode, "Kueue admits the workload only if enough resources are available within the same GPU cluster. Otherwise, the workload remains pending until resources become available or the topology constraint changes." In Preferred mode, Kueue tries the same domain first and otherwise places the pods across several domains.

On plain Kubernetes, the NodeResourceTopology plugin scores nodes by NUMA layout rather than by a flat resource total, and it buys that accuracy with waiting: its safer caching mode "prevents these suboptimal decisions at the cost of leaving pods pending longer".

CoreWeave's SUNK scheduler groups nodes so an allocation stays inside one group, and its documentation names the limitation that comes with the grouping: "No bin-packing. Slurm doesn't fill partially-used Nodes before moving to idle ones. This can spread Pods across Nodes and lead to GPU fragmentation, especially during scaling."

Run:ai can consolidate after placement by preempting workloads: "During consolidation, the Scheduler may preempt workloads from other queues ... consolidation is performed only if all preempted workloads can be immediately re-scheduled and continue running."

So the topology features act before a worker starts, and Run:ai's consolidation acts later but stops and restarts the workload. None of these vendor pages describes moving the running worker itself.

Routers and autoscalers cannot move existing sessions

Neither a router nor an autoscaler touches a session that is already running. A router decides where the next request lands, so it can send the next thousand requests to the quietest instances and change nothing about the sessions on the busy one. An autoscaler can add a replica, but that replica starts empty and only takes new arrivals, so the fleet ends up larger and no better balanced, with one instance still handling the slowest requests.

Scaling down ends the requests that are still in flight. NVIDIA's Dynamo planner is an autoscaler built for inference workers, and its documentation is direct about it: "When the Planner scales down a worker, the worker is terminated without waiting for in-flight requests to complete. Requests that were mid-prefill on the terminated worker will fail. In disaggregated deployments, this can also affect decode workers that were waiting on KV cache transfers from the terminated prefill worker." The documented answer is to trigger the scale-down less often, with a floor set by min_endpoint or a lower load_scaling_down_sensitivity.

What changes when the worker itself can migrate

The alternative to rebuilding a cache is to move it. Llumnix, a request-scheduling system published at OSDI 2024, migrates a request together with its in-memory state from one model instance to another while generation is still running.

On a 16-GPU cluster of NVIDIA A10 GPUs, the downtime of that move was roughly 20 to 30 milliseconds, nearly constant as sequences grew and shorter than a single decode step. On that 2024 research testbed of A10 GPUs, Llumnix reports up to 15 times better P99 first-token latency than the INFaaS scheduler while running LLaMA-7B and LLaMA-30B.

What Llumnix moves is one request's cache, and every destination it uses is a second instance already running the same model. So a pool with no identical warm replica has nothing to route to, and free GPUs on nodes running other workloads are not destinations that mechanism can use.

Moving the worker instead of the request lifts both of those conditions. With Cedana, the worker itself moves to a compatible node. It does not need another running instance of the same model, and free GPUs on a node running another workload can qualify as a destination. The weights in GPU memory, the KV cache, and the sessions in flight travel with the worker, so those sessions carry on where it lands instead of recomputing their context.

Is there a layer that moves work to where the free GPUs are?

Kueue and Ray both stop short of it, because neither one watches for capacity that frees up later. A Kueue issue notes that "Kueue itself does not watch for freed resources and trigger scale-up". A Ray user found that when spot capacity for one instance type ran out, "the autoscaler keeps retrying to acquire that node type... but never moves on to the next node type and so my cluster doesn't scale".

With Cedana, running work can move to free GPUs on a compatible node. Cedana does not decide where a job goes: an operator or an agent starts the move, and nothing named on this page calls us today. Work that has never started is still the scheduler's to place.

What has to be true before a worker can move

Cedana preserves the worker in a checkpoint and restores that state on a compatible node, under a policy written once rather than job by job. The move costs a restore plus the work done since the last checkpoint. The saved bytes have to travel to the destination and be read back into GPU memory before execution continues.

The scheduler still controls initial placement, including whether it accounts for NUMA and NVLink. Migration makes that placement changeable after the worker starts.

The destination has to be a compatible node. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

So the wrong-silicon case runs into a limit the other two do not. A worker cannot be moved from an H100 and resumed on an H200. The H100 worker's loaded weights and cache come back on other H100 capacity.

The last limit is how many nodes run the workload at once, not how far it moves. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

So the question stops being whether the first placement was right and becomes whether you can change it afterwards. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On an inference fleet, that means we save a running worker in full, with its GPU memory, its KV cache, and the sessions it is serving, and bring it back on compatible hardware. A placement made under last week's traffic becomes something you can change today.

Related:

• Why a cluster with free GPUs still cannot place an 8-GPU job
• What would a GPU scheduler do differently if it could move running jobs?
• Does disaggregated serving remove the need to move GPU workers?
• H100 for prefill, H200 for decode: the mismatch you pay for twice

Common questions

Can placement be made topology-aware, so a workload's GPUs land close together on NUMA, PCIe, or NVLink instead of wherever the scheduler counts them free?

Vendors do offer topology-aware placement: node labels describe racks, blocks, and NVLink domains, and the scheduler uses that alongside the GPU count to place a distributed workload's pods close together. Those constraints only govern admission. Once a worker is running, it stays exactly where it landed even if the traffic mix, the fleet's hardware, or a session's load changes underneath it, because migration, not admission-time placement, is what makes it changeable afterward.

Free GPU capacity exists somewhere else in the fleet. Is there a layer that moves my pending or misplaced work to where that capacity is?

A router can send new requests to the quietest instances, and an autoscaler can add a replica, but neither one moves the sessions already running on an overloaded or misplaced worker. With Cedana, the worker itself moves to a compatible node instead, carrying its GPU memory, its KV cache, and the sessions in flight with it, so those sessions carry on where it lands instead of recomputing their context.