Cold start time (time to first token, excluding container init)
workload : transformers-interface
Boost inference performance. Elevate your customer experience and retention.
- Industry’s fastest time-to-first-token for enterprise-grade inferencing - up to 10x faster.
- Reduce over-provisioning while improving SLAs
The industry’s fastest cold starts are here
47% faster time to first token for Llama 3.1-70B (38GiB model, H100 80GiB)
Common Tags
Updated 2024-10-24T01:38:53.880809
Mock internet speed 500 Mib/s
Disk write speed: 852 Mb/s
CPU memory (DRAM): 83.56 GiB
CPU cache (L2): 104 MiB (26 instances)
CPU cache (L1 data): 832 KiB (26 instances)
CPU clock speed (max):
CPU threads/core: 1
GPU API: 12.4
GPU SM clock speed (max): 1755 MHz
CPU cache (L3): 416 MiB (26 instances)
GPU memory (total): 81559 MiB
GPU compute capability: 9.0
CPU model intel(R) Xeon(R) CPU @ 2.20GHz
GPU memory clock speed (max): 1593 MHz
GPU model: NVIDIA H100 PCIe
Disk read speed: 1.8 GB/s
COLD START TIME
CPU cores: 26
GPU driver: 550.127.05
C/R tool: native
COLD START TIME
CPU cores: 12
GPU driver: 535.129.03
C/R tool: cedana (v0.9.232-1-ga951e5b)
GPU runtime: cedana
Works across models with no code modifications or changes to your stack
2-10x faster cold starts drives enterprise AI success
Test-time inference, multi-modal and compound AI are all shifting more compute to inference, increasing the importance of fast cold start times.
Customer experience and retention
Deliver more responsive chatbots, copilots, video, and reasoning systems by improving Time to First Token (TTFT)
Cut inference costs while scaling elastically
Minimize over-provisioning while elastically scaling to meet traffic spikes.
Expand your product capabilities
Accelerate workloads and free up GPU resources for advanced features, personalization, and improved reasoning.
Get Started
Play in the sandbox
We’ve deployed a test cluster for you to play with where you can interact and experiment with the system.
SandboxGet a demo
We’ve deployed a test cluster for you to play with where you can interact and experiment with the system.
ConnectAPI Reference & Guides
We’ve deployed a test cluster for you to play with where you can interact and experiment with the system.
ViewDocs