vLLM or SGLang hangs: detection and recovery

TL;DR

Your vLLM, SGLang or TensorRT-LLM server has stopped answering requests while the process is still running and GPU memory is still allocated. So there is no crash log and no clean error to read.
A GPU hang looks the same from outside, so check whether the device is still answering before you blame the engine.
The health endpoints do not save the work either way. SGLang's health check kills the service and everything it held in GPU memory, and vLLM's health call can miss a dead worker entirely. So an external check has to catch the failure and something has to restart the service.
That restart is a full cold start: the weights load again, the GPUs are wired together again, the engine rebuilds everything it had in GPU memory, and the sessions in flight start over. On one node of 8x NVIDIA B200 GPUs, published native starts of four frontier models ran from 564 to 2,051 seconds. Restores of the same engines ran from 57 to 70 seconds.
In this piece we walk through what the reports show, what is hanging underneath the API server, how to tell a wedged engine from a hung GPU, and what the health endpoints do. Then we cover what a restart costs, what a checkpoint taken before the hang changes, and who decides that the engine has stopped answering.

What the reports describe

On the trackers of all three engines the same failure keeps coming back: the server stops answering and nothing in the logs says why. A vLLM issue from October 2023 describes a server that hangs forever without producing an error message. A later report carries the one line the engine does print when it gives up, "AsyncEngineDeadError: Background loop is stopped."

The engine usually serves normally for hours before it goes quiet. A vLLM report from May 2026 describes an engine that runs fine for a period of hours, then has a worker process hang, and the failure cascades from there. An SGLang report from April 2026 shows the watchdog catching the same thing and logging "health check failed. server could not get a response from detokenizer for last 20 seconds", after which the process exits.

Other reports carry no error line at all. A TensorRT-LLM report from July 2025 records no further action for hours. An engineer on Reddit says vLLM hangs after that with no errors, and a user on the Ray forum describes a job that hangs and eventually quits.

The same silence can arrive before the server has served anything. A vLLM report from August 2026 describes an engine that never comes up: the process goes silent before the log line "init engine took" appears, and the API server never starts. Nothing has been served in that case, so there is no state to bring back, and the answer is in the configuration rather than in recovery.

Once a running engine has hung, recovery without a checkpoint means restarting it. A second vLLM report from August 2026 describes what that took on one fleet: "Only a full fleet restart recovers. Reproduced 3/3 boots."

Most of these threads are still open, or closed with no confirmed fix, and the fixes that do get reported are narrow: a flag such as --disable-custom-all-reduce, or a process manager such as systemd or llama-swap put in front of the service to restart it.

What is happening below the API server

A modern serving engine is not one process, which is why one part of it can stop while the rest looks healthy. The API server takes the requests, an engine loop drives the scheduling, and worker processes hold the model, one per GPU in a tensor-parallel deployment. They talk to each other over sockets and shared memory, so when one of those parts dies or stops making progress, the part in front of it waits.

A worker can die without the parent process finding out. A vLLM forum post from June 2025 reports that when a worker subprocess crashes, whether from an out-of-memory kill, a segfault or a manual kill, the parent process never notices, because the engine's check_health() call does nothing.

A worker can also be alive and stuck, with the engine waiting on a call that never returns. In the May 2026 report, one of the 8 tensor-parallel workers stops responding, the remote call to it times out, and the engine is marked dead afterwards.

A request can stall before it reaches a worker at all. The August 2026 report that needed a full fleet restart traces the stall to the shared-memory channel between the API server and the scheduler, where requests never reach the scheduler.

The API server can stay alive even when a worker has died or stopped responding. Memory remains allocated, but requests do not finish. A different pattern, where one node's NCCL error stalls an entire distributed training job rather than a single serving engine, is covered in One node failed and the whole training job died.

A hung GPU looks the same from outside

The GPU itself can be the thing that stopped, and from the front it looks identical: requests stop finishing, the process cannot be killed, and no signal clears it. That is a hardware failure rather than an engine failure, so it ends in a GPU reset or a node reboot, and it is covered on the page about what happens to a training job when a GPU fails.

Before you assume the engine is at fault, check whether the device is still answering.

What the health endpoints do about a stalled engine

vLLM and SGLang handle a stalled engine differently, but neither of them preserves the work.

SGLang's health check kills the service. In the April 2026 report, the check that could not reach the detokenizer for 20 seconds ends the SGLang process, and everything the service held in GPU memory goes with it: the loaded weights, the compiled CUDA graphs, and the key-value (KV) cache holding every session in flight.

vLLM has the opposite problem, which is a health call that reports nothing wrong. The June 2025 forum post reports that the call does not detect a dead child process at all, so the health endpoint gives the orchestrator no warning even though the service has stopped answering.

Either way, an external check has to detect the failed service, and a person or a process has to restart the service from scratch.

The restart is a full cold start

A new process reads the weights off storage, writes them into GPU memory, brings up the communication between the GPUs, reserves the KV cache, compiles, captures its CUDA graphs, and only then answers a request. The sessions the old process was serving are not in the new one, so the clients that were waiting start over.

Cedana's published benchmark times that sequence against a restore of the same engine. Every run used one node with 8x NVIDIA B200 GPUs and 1.7 TB of system memory, on CUDA 12.9. Each model was served by SGLang from its official recipe, and the checkpoints were read from tmpfs, which lives in the node's own memory rather than on disk. Across four frontier models, the native starts ran from 564 to 2,051 seconds, timed from engine launch to ready-to-serve, and the restores of the same fully initialized engines ran from 57 to 70 seconds.

Restore time follows the size of the checkpoint, meaning how many bytes have to move, not the parameter count. Tmpfs is the fastest path available, so treat those restore times as a floor rather than a promise about your own fleet.

What a checkpoint taken before the hang changes

A checkpoint taken while the engine was healthy turns the recovery into a restore. Checkpointing means writing down the state of a running process so it can be brought back later, and the state worth having here is the fully initialized engine: the weights in GPU memory, the compiled graphs, the cache, and the sessions in it. That is what we build at Cedana. We take the checkpoint below the serving engine, at the operating-system and CUDA-driver levels, so vLLM, SGLang and TensorRT-LLM are captured the same way and none of them needs a code change, and heartbeat checkpointing saves the workload on an interval you set.

Cedana preserves the entire workload state (CUDA context, KV cache, file system, network, scheduler), so workloads automatically resume on a new instance as of the last checkpoint. What comes back is the engine as of the last checkpoint, with the sessions and the cache it held at that moment, and the requests that arrived afterwards are not in it. Recovery from a wedged engine then means restoring onto a healthy worker rather than starting from nothing, so you wait for the restore instead of the whole initialization sequence.

A restore continues the workload from the state it captured, so a checkpoint taken after the engine wedged brings back a wedged engine. You need a checkpoint from before the engine stopped answering, which is why both the checkpoint interval and the delay before detection matter.

Who decides that the engine has stopped answering?

You do, through your health check, your probe, or the alert somebody wakes up to, exactly as it works now. We do not notice a silent engine for you.

Our published policies act on four things: a failure that automatic failover resumes through, the schedule the heartbeat runs on, a pod's memory use crossing a threshold you set, or an agent asking for the action. An engine that is up but not answering is none of those unless something you run turns it into one. The control plane gives an admin a UI for manual checkpointing and for redeploying workloads, and a program can request the same action instead, so in this case your own signal has to call for the restore.

A restore requires compatible hardware and software. A checkpoint records the GPU, driver, engine and model versions it was taken against. If any of them changes, the checkpoint is invalid and the workload cold-starts instead.

A serving replica that fits on one machine is covered. The single-GPU and multi-GPU-on-a-single-node tiers ship in production today. Multi-node, where a single workload spans hundreds or thousands of GPUs across many nodes, is in design partnership with leading enterprises and neoclouds. If your jobs span nodes and this is the case you need solved, talk to Cedana about a design partnership.

A hung engine costs you the whole cold start only while nothing saved the healthy worker. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a serving fleet that means capturing the worker while it is healthy and bringing that state back on a compatible node, so a wedged engine costs the restore plus the work since the last healthy checkpoint rather than the whole cold start. It does not stop the engine from hanging, so the next time yours goes quiet at three in the morning you still have to notice it. What you no longer have to do is rebuild the engine from nothing while your clients wait.

Common questions

How do I detect and recover from vLLM or SGLang hanging with no crash and no error?

Your own health check or probe has to catch it, because the API server can stay alive even when a worker has died or stopped responding, with memory still allocated and requests not finishing. Without a checkpoint, the recovery is a restart, and that restart is a full cold start that reloads the weights and rebuilds the cache from nothing. With a checkpoint taken while the engine was still healthy, the same check triggers a restore of that checkpoint onto a healthy worker instead, so you wait for the restore rather than the whole initialization.

Why does the engine never come up instead of hanging after it starts serving?

That is a different failure, and it happens at startup rather than in service. A vLLM report describes an engine that never comes up, with the process going silent before the API server starts. Nothing has been served yet at that point, so there is no state to bring back, and the answer there is in the configuration rather than in recovery.

What to do when vLLM or SGLang stops responding and nothing crashed