cedana / benchmarks · AWS Trainium2 · 64 devices
Trainium2 cold start: 44 minutes to 45 seconds
Llama 3.3 70B on vLLM across 64 AWS Trainium2 devices. Native cold start takes 2,647 seconds, about 44 minutes. Cedana restores the same serving state in 45 seconds, all 64 devices in parallel.
2,647s
native cold start
45s
Cedana restore
58.8×
speedup
64
Trainium2 devices
The result
44 minutes of cold start, back in 45 seconds.
All 64 devices restore in parallel.
AWS Trainium2, 64 devices · Llama 3.3 70B on vLLM · speedup computed from 2,647s vs 45s
The checkpoint
Taking the checkpoint is a separate step.
The checkpoint is written once and restored as often as needed, so its time is not part of the 45 seconds. On this run the dump wrote 577 GB of device state in 32 seconds and 93 GB of container state in 78 seconds.
CheckpointWritten once, separate from the restore time
Device state577 GB
32s
Container state93 GB
78s
RestoreBack to ready-to-serve, all 64 devices in parallel
Serving state64 Trainium2 devices
45s
Configuration
What was measured.
- Model
- Llama 3.3 70B
- Engine
- vLLM
- Accelerator
- AWS Trainium2
- Devices
- 64
- Native cold start
- The time from launch to ready-to-serve.
- Cedana restore
- The time to bring the same serving state back from a Cedana checkpoint to ready-to-serve.
Multiply the fleet you already have.
Same hardware, same models, same serving stack. Bring one workload and we will show you the GPU-hours it gets back and the tokens it delivers.