TL;DR
- A mixture-of-experts (MoE) model's router picks which experts each token uses, and no serving setting you own changes that choice.
- What you do own is the layout: which device holds which expert, how attention and the experts are each split across devices, and how much of the exchange between devices hides behind computation.
- So a count of free GPUs does not tell you whether a replica will fit, and a flag copied out of someone else's recipe is not a plan. A layout has to suit the model, the interconnect and the traffic in front of you.
- All three decisions are cheap to make before a replica starts and expensive to revisit while it serves. The work already done for every request in flight lives only in GPU memory.
- In this piece we walk through what the router fixes, how sharding turns a routing decision into network traffic, and why free GPUs are not placeable GPUs. Then we cover why attention and the experts want different splits, what overlap decides, and what changing a layout costs.
The router's choice belongs to the model, and the layout belongs to you
A mixture-of-experts layer holds many expert blocks where a dense model holds one feed-forward block, and a router in front of those blocks sends each token to a few of them. The router's weights were learned in training, so the rule that picks the experts is fixed, and which experts a given token lands on is decided at inference, token by token, by that rule. Nothing you set on the serving side touches it. Kimi K2.6's model card gives a current example with 384 experts and 8 selected for each token.
So a token's work can land in 8 of those 384 places at each layer, and no serving configuration changes which ones. Those counts describe the model. Where the blocks sit across your GPUs is a different question, and it is the one you answer. You own three parts of that layout: placement, parallelism, and overlap.
Everything below is the steady state, meaning the model is resident in GPU memory and the deployment is serving traffic. What it costs to start a replica and what a failure costs are separate questions.
Sharding the experts is what turns a routing decision into network traffic
A 235B-parameter model, which is a different model from the one above, does not fit on one GPU, so the experts are sharded across the node. Each decode step then triggers an all-to-all exchange across the interconnect. An all-to-all is the collective where every device holding part of the layer sends to every other one.
Each device hands the tokens it holds to the devices that own the experts those tokens picked, and the results have to come back before the layer is finished. So a decision made inside the model becomes traffic on the network between your GPUs.
Placement is the decision about which device holds which expert. It sets how far a token's slice travels on each decode step and how many links it crosses on the way. Change the layout and the router makes the same choices it made before, so the same model on the same node can put a different amount of traffic on the interconnect.
The engine's job is to keep the interconnect from becoming the binding limit. Placement and overlap are the two moves that do that job. We count parallelism as a third decision because attention and the experts are not handled the same way. That separation is ours rather than something the architecture forces.
| Decision | What it sets | What it does not set |
|---|---|---|
| Placement | Which device holds which expert, so how far a token's slice travels on each decode step | Which experts the router picks for a token |
| Parallelism | How attention and the feed-forward block are each split across devices, so which devices exchange results inside a layer | How many experts the model holds, or how many of them a token uses |
| Overlap | How much of the exchange between devices runs behind arithmetic the device is doing anyway | How large the exchange is |
No layout is correct everywhere, so you choose one against the model's expert count, the interconnect between the devices, and the traffic being served. None of those three is the same from one fleet to the next.
A count of free GPUs does not describe a placeable set
A replica needs a set of devices that can carry its exchange on every decode step, and a count of free GPUs does not tell you whether such a set exists. Free GPUs scatter as jobs start and finish. A frontier replica, meaning one serving instance of a current open-weight model, has to land on a contiguous multi-GPU or multi-node resource.
So your fleet can report spare GPUs and still have nowhere to put a new replica, because the free devices are scattered and no group of them can carry the exchange.
Placement is settled when the replica starts, and the exchange it creates then runs for as long as the replica serves.
Attention and the experts do not want the same split
The two halves of a mixture-of-experts model want different treatment, because attention likes data parallelism and the experts like expert parallelism. Data parallelism gives each device its own copy of the part it runs and splits the incoming work between the copies. Tensor parallelism cuts one layer's weights across devices instead, so each device holds a slice and the devices exchange results inside the layer.
SGLang ships a hybrid built on that difference, splitting attention one way and the feed-forward block another. Its server-arguments guide describes the data-parallel attention option as data parallelism for attention alongside tensor parallelism for the feed-forward network (FFN) block. Expert parallelism is the other arrangement, and it also exists in SGLang: it runs the experts across devices, with DeepEP carrying its all-to-all traffic. The two labels therefore name different devices talking to each other, and reading them as the same thing gives you the wrong answer about which devices a layer's traffic runs between.
The guide's model tips say that data-parallel attention raises throughput under high concurrency and costs latency under low concurrency. That makes the split a decision about the traffic you serve. So a throughput number is a reading of one deployment, because the same model under a different split reports a different figure.
An engine's shipped recipes carry specific settings, but a setting copied out of one is not a plan for your deployment. The cookbook says why: "Because commands are generated from data and pinned to an SGLang version, exact flag values change per model release and engine version."
Overlap decides how much of the exchange you wait on
The all-to-all is communication, and communication can be hidden behind computation. While one layer's tokens are crossing the interconnect, the device can work on arithmetic it already holds. So time spent moving data does not have to be time the pipeline waits. A good MoE backend is mostly a scheduler for hiding the network behind the compute, and SGLang has a dedicated all-to-all backend for that traffic called DeepEP.
How much of the exchange ends up hidden is the open question in any layout. We have no measurement of it for a named engine version and a named model, so you have to measure it on your own fleet rather than look it up. The other two decisions set the size of the exchange, and overlap sets the part of it the pipeline waits on.
Changing a layout while the replica is serving costs the sessions it holds
All three decisions are cheap to set before the replica starts and expensive to change afterwards, because a replica holds session state while it serves. That state is the work already done for every request in flight, and it lives only in GPU memory, where any interruption destroys it. The only way to get that state back is to run the work again.
So restarting a replica to re-place it means rebuilding everything it was holding, unless that state is moved off it first. A new layout is a new replica: the placement, the parallelism and the overlap are chosen at launch, so the replica that runs them starts cold, and you are the one who decides to build it.
The state is what makes a layout hard to revisit once the replica is serving. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We save the full state of a running replica and bring it back on compatible hardware, so moving one does not cost the sessions it was serving. None of that raises the interconnect ceiling or changes an expert's placement. What it changes is the price of standing the old replica down, because the sessions it held can finish elsewhere instead of being rebuilt. That is the price you weigh when your traffic stops matching the layout you launched with.
