TL;DR
- On a shared cluster, a job that needs a fraction of a card holds all of it. So the usual answer is to cut the cards into fixed NVIDIA MIG instances and hand out the pieces.
- MIG gives you isolation in hardware, but the sizes come from a fixed menu and the GPUs have to be empty before you can change them. So a split chosen once a term has to fit work that changes week to week.
- Sharing the whole card between processes instead, with time-slicing or MPS, takes away the sizing problem and the fault isolation with it. That is not a risk you can hand to strangers on a queue.
- The third way shares the card in time: checkpoint the job holding the GPU, give the whole card to the next job, and restore the first one later from where it stopped. Nothing is sized in advance, and each swap costs a save and a restore.
- In this piece we walk through what MIG guarantees and what changing a split costs, and what time-slicing and MPS do and do not isolate. Then we cover how sharing by checkpoint works, what each swap costs, and which of the three fits which cluster.
MIG is the only one of the three with isolation in hardware
Multi-Instance GPU, or MIG, divides a supported card into as many as 7 instances, each with its own memory and its own fault isolation at the hardware layer. NVIDIA's MIG user guide puts the guarantee in one sentence: "MIG ensures one client cannot impact the work or scheduling of other clients, in addition to providing enhanced isolation for customers."
That guarantee is what lets you hand one card to people who do not know each other. A job that leaks memory or dies badly stays inside its own instance, and each instance delivers a known quality of service rather than a share of whatever the neighbors leave. So for steady workloads shared between tenants on one card, MIG is the right tool.
Changing the split means emptying the GPUs first
The instance sizes come from a fixed menu, and predefined is NVIDIA's own word for them, so the profiles your card supports are the profiles you get.
Changing a card's profile layout costs you the work running on it. NVIDIA's GPU Operator documentation says MIG Manager "requires that no user workloads are running on the GPUs being configured", and that a reconfiguration stops the GPU pods on those cards before it applies the new layout.
A reconfiguration can also involve rebooting the node, where enabling MIG mode requires a reset. Setting MIG mode on an A100 or an A30 requires a GPU reset, but from the Hopper generation on it does not. All of them require idle GPUs for reconfiguration.
MIG Manager supports dynamic reconfiguration of the geometry, which means it applies a new layout without an administrator running the steps by hand. Dynamic there describes the tooling rather than the operation, because the GPUs still have to be empty before it runs.
Slurm does not close the gap, because it only consumes the MIG instances that already exist in its configuration. It treats those instances as individual GPUs, with cgroup isolation and task binding, but it does not create or resize them, so a new split is an operation outside the scheduler, and the node then has to be described to Slurm again.
The sizes are a forecast of work that cannot be forecast
Clearing the GPUs is expensive enough that the split rarely moves once it is set. Sizes picked once a term have to fit computational biology next to materials science next to a student's notebook, and the proportions move week to week.
Research computing teams tell us at Cedana that the sizes were chosen in advance, the GPUs have to be cleared to change them, and the mix of work the cluster runs never matches what was chosen. A slice larger than the job inside it is still counted as allocated, so the cluster's own numbers report it as used.
Sharing without partitions means sharing without fault isolation
You can also share a card without cutting it up, which is what NVIDIA's time-slicing and MPS do: both put several processes on the whole card at the same time. The GPU Operator documentation says of the time-slicing configuration for Kubernetes that there is "no memory or fault-isolation between replicas, but for some workloads this is better than not being able to share at all".
The Multi-Process Service, or MPS, differs on memory, and NVIDIA's MPS documentation says "MPS client processes have fully isolated GPU address spaces". Faults are the part MPS does not cover, and the MIG user guide says "MPS currently does not offer error isolation between clients".
When one team runs cooperating processes that already share memory, either one is a reasonable answer. But on a shared research cluster the jobs belong to strangers. A memory leak in one user's notebook or a crash in another's simulation can stop unrelated work on the same GPU, and you cannot hand that risk to everyone on the queue, so a shared cluster ends up back at fixed slices for the jobs that have to run at the same moment.
The third way shares the card in time, and needs no sizes at all
Checkpointing swaps the jobs instead of resizing the card. You save the running job in full, meaning its GPU memory, its CUDA state, the process and its memory, its files, and its network connections. The whole card then goes to the next job, and the first one is restored later and carries on from where it stopped.
Nothing is partitioned, so no set of GPUs has to be emptied to make room for a different mix of work. Each job runs alone on the whole card at its full size and then vacates it for the next one, because the previous job is checkpointed off before the next one starts.
No two jobs share GPU memory or execution at the same moment, so the separation here is in time, where MIG isolates instances in hardware as they run side by side. The two are different guarantees.
NVIDIA also calls its interleaving approach time-slicing, so the two methods are easy to confuse. Sharing by checkpoint is serial sharing, meaning one job holds the whole card for a window and the next one starts on an empty card.
The same swap covers preemption, so batch work can run on idle GPUs and be checkpointed off when the owner of those GPUs has a job to run. The borrowed job is saved rather than killed, and the owner waits for a checkpoint instead of for someone else's job to finish.
Each swap has a cost, and the queue keeps running
Each swap is a save on the way out, a restore on the way back, and a pause for the job in between. Cedana's published floor for model restores on H100 is about 30 seconds on a single GPU and about 80 seconds across 8 GPUs, on the product page. A different workload's restore is governed by the byte size of its own checkpoint, not its parameter count, so the number that matters is the one you measure on your own jobs.
Changing a MIG geometry stops whatever is running on the affected GPUs, because those cards have to be idle first, and on some systems the node reboots as well. NVIDIA's time-slicing has nothing to size, so there is nothing to change.
Which one fits depends on whether the jobs trust each other
| Approach | Isolation | Changing the sizes | Where it fits |
|---|---|---|---|
| Fixed partitions with MIG | Memory and fault isolation in hardware, with a known quality of service per instance | The affected GPUs must be idle first, so the work stops or the node drains, and on some systems a reboot is in the path | Steady, known workloads that need hard isolation while they run side by side |
| Time-slicing or MPS | Time-slicing gives none, for memory or for faults; MPS isolates each client's GPU memory but not its faults | Nothing to size, and every process sees the whole GPU | Cooperating workloads from one team that can absorb a neighbor's crash |
| Serial sharing through checkpoints | One job on the card at a time, so nothing is shared while it runs | Nothing to size, and a save and a restore per swap | Diverse, bursty work where jobs want the whole card for a window and the mix changes |
You do not have to pick one of the three for the whole cluster. One node can hold MIG instances for the steady tenants while another gives whole cards to jobs in turn, so the choice follows the work on each node.
Serial sharing needs a checkpoint that covers the whole job
For this to work on a real queue, the checkpoint has to capture the whole job below the application, so that any job on the cluster can be saved and resumed without its owner writing checkpoint code for it. Something also has to decide when to swap, and that is a scheduling policy rather than a property of the checkpoint.
A checkpoint that covers the whole job is the part that has to exist underneath before any swapping policy can run. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. We capture the whole job at the operating-system level on Slurm and Kubernetes, with no change to the application, and the swap depends on that system-level checkpoint. Deciding when to swap is still your policy to write.
Related:
- How much of your GPU utilization was work you kept?
- How to fill idle GPUs without killing the job that fills them
- Why an idle notebook keeps its GPU until you kill it
- What would a GPU scheduler do differently if it could move running jobs?
Common questions
On our shared cluster a job that needs one GPU ties up a whole multi-GPU node, and some jobs get handed a full A100 for work that barely uses it. How do we stop wasting the rest?
A slice larger than the job inside it is still counted as allocated, whether that slice is a whole node or a fixed MIG instance, so the cluster's own numbers report it as used while most of it sits idle. Checkpointing the job and swapping in the next one shares the card in time instead, so nothing has to be sized in advance. A job holding the whole card still leaves whatever it does not use idle. What changes is that the card goes to the next job when the first is checkpointed off, rather than sitting in a fixed slice sized months ago.
