LoginSchedule demo

» Cedana / Solutions / Power

Break the GPU power bottleneck

Expand power capacity. Accelerate permitting.

A year at a 40 MW GPU data center. Ten times the grid operator asks for 8 MW back, 20% of the site's capacity, with as little as five minutes of warning. The site runs 70% inference and 30% fine-tuning and training. Each time it suspends the workloads on 408 to 493 of its 2,000 B200 servers, saves them to disk, and powers those servers down. When the grid releases the site, every workload resumes from where it stopped. Across the whole year the site is curtailed for 45 hours, 0.5% of the time.

40 MW
Facility capacity at the utility meter.
≈ 16,000
B200 GPUs, in 2,000 eight-GPU servers at 14.3 kW each.
8 MW
Curtailment per event, 20% of capacity.
0%
Work lost. Every suspended workload resumes from its checkpoint.
0.5%
Of the year spent curtailed: 45 hours across ten events.
Normal operationsThu Jan 1 · 00:00
5 min60 min5 minTime between the grid's curtailment request and the start of curtailment.
10%90%20% · 8 MWShare of the 40 MW capacity the site returns to the grid in each event.

Event 1 of 10

Metered loadLoad without the eventPower returned to the grid

Warning window

05:00
Every workload saved 02:30 after the requestGPUs powered down 03:00 after the requestHeadroom before the deadline 02:00, 40% of the window
  1. Request received
  2. Suspend and save to disk
  3. Power down GPUs
  4. Curtailed
  5. Power up
  6. Resume from checkpoint

Server floor

Each square is one eight-GPU B200 server. 1,400 run inference and 600 run fine-tuning and training.
RunningSelected for curtailmentSaving to disk, darker as it completesSaved to diskPowered downRestoring from disk to running

Checkpoint ledger

Servers selected to cover 8 MW.
Saving, then saved to disk0 / 0
B200s powered down0 / 0
Workloads resumed and running0 / 0
GPU memory on disk 0 TBProgress preserved 100%Work lost 0%

Suspendable workloads, automatically checkpointed and resumed through each event

Ten examples from the servers suspended in each event.
WorkloadTypeProgressState
0.0 MW
Facility load
0.0 MW
Returned to the grid
0 of 2,000
Servers powered down
0
B200s powered down
0 of 10
Events answered this year
0.0 h · 0.00%
Time curtailed so far, and its share of the 8,760 hours in the year
0 MWh
Energy returned to the grid
0 of 0
Workloads resumed from checkpoint
0%
Work lost. 100% of progress preserved.

What happens in each event

Request

The grid operator asks for 8 MW. The warning window is the time until curtailment must start. The site selects enough servers to cover it, in the same 70/30 mix as the fleet.

Suspend and save

The save starts the moment the request arrives. Workloads on those servers are suspended and their full GPU memory is written to disk. One server takes about a minute.

Power down and hold

With their state on disk, those GPUs power down safely, three minutes after the request. The rest of the warning window is headroom. Every other server keeps running.

Resume

On release the servers power up and state is restored from disk. Training continues from the same step. Inference comes back warm, with no model reload.

The ten events

Timed for the hours the grid is tightest: winter cold-snap peaks in the early morning and evening, and summer heat-wave afternoons. Durations vary from two to six hours. Select a row to replay it.

#DateTriggerRequestStartEndHoursLoad beforeReturnedServers suspendedMWh
1Tue Jan 20Cold snap, evening peak16:2516:3018:302.0033.8 MW8 MW473 · 3,784 GPUs16.5
2Wed Jan 21Cold snap, morning peak05:4005:4509:003.2532.5 MW8 MW493 · 3,944 GPUs26.5
3Tue Jun 23Heat wave, afternoon peak15:2515:3020:154.7539.2 MW8 MW408 · 3,264 GPUs38.5
4Wed Jun 24Heat wave, afternoon peak14:2514:3018:153.7539.1 MW8 MW409 · 3,272 GPUs30.5
5Wed Jul 8Heat wave, afternoon peak15:2515:3019:153.7537.1 MW8 MW431 · 3,448 GPUs30.5
6Thu Jul 16Heat wave, afternoon peak13:5514:0018:454.7539.1 MW8 MW410 · 3,280 GPUs38.5
7Fri Jul 17Heat wave, afternoon peak13:2513:3019:306.0037.5 MW8 MW427 · 3,416 GPUs48.5
8Wed Aug 5Heat wave, afternoon peak14:2514:3020:005.5039.0 MW8 MW410 · 3,280 GPUs44.5
9Thu Aug 13Heat wave, afternoon peak14:1014:1520:156.0038.0 MW8 MW421 · 3,368 GPUs48.5
10Wed Sep 2Late-summer heat, evening peak16:2516:3021:004.5035.9 MW8 MW445 · 3,560 GPUs36.5

Assumptions

  1. Sizing

    2,000 servers × 14.3 kW maximum = 28.6 MW. Network, storage and management add 10%, giving 31.5 MW of IT load. Cooling and electrical overhead of 1.27 at the design peak brings the facility to 40 MW. That is 16,000 B200s.

  2. Load

    IT utilization averages about 90% with a small daily cycle. Cooling overhead rises with outdoor temperature, so the site draws about 32 MW in winter and 37 to 39 MW on hot afternoons.

  3. Curtailment depth

    Set by the slider, 10% to 90% of the 40 MW capacity. The default is 20%, or 8 MW. Because the site runs below capacity in cooler weather, the same depth takes a larger share of the servers in winter. The site always keeps at least 1 MW on the grid, so at very high depths the curtailment is limited by the load itself.

  4. Workload mix

    70% of servers run inference and 30% run fine-tuning and training. Servers are selected for each event in the same proportion.

  5. Warning window

    Set by the slider, 5 to 60 minutes. The save begins when the request arrives. Whatever is left of the window after the GPUs are off is headroom, so a longer warning means more headroom and slightly more energy returned.

  6. Checkpoint

    A full save of one server's GPU memory (8 × 180 GB) takes about one minute. Saves are staggered across the selected servers over 90 seconds. Each server powers down 30 seconds after its save completes, so every selected GPU is off three minutes after the request.

  7. Resume

    Servers power up over 90 seconds after release, take 30 seconds to boot and about one minute to restore. All workloads are running again three minutes after release.

  8. Playback

    The year runs fast between events. Save and resume play at about one minute of site time per 1.2 seconds so the per-server progress is visible.