TL;DR
- If your engineers use coding models, you are paying an API by the token or paying for GPUs by the hour. A cost per token and a cost per GPU-hour are quoted in different units, so you cannot hold them up against each other.
- The published rate cards settle only half of it. They tell you what an hour or a million tokens costs, and nothing about how many coding tasks come out of the hours you paid for.
- A finished coding task is the unit both bills share. On self-hosted GPUs one task costs the node's hourly bill divided by the tasks that came out of it.
- So the hours that finish no task, the idle ones, the model swaps, the work that has to start again, are the half of the calculation you can change.
- In this piece we walk through the two formulas, the published prices as they stood on 1 September 2026, and where the paid hours go on a coding node. Then we cover what one model swap costs in node time, and the three measures that test the economics on your own traffic.
The only unit both bills share is a finished coding task
On a per-token API, one task costs you the tokens that task consumed times the price per token. Your bill already holds the tokens it charged you for, input, output and cached, so the work is attributing them to the tasks that used them, and that is where your tokens per task comes from.
On self-hosted GPUs, whether you rent them or own them, one task costs the hourly rate times the hours you paid for, divided by the tasks that came out.
The two formulas behave differently when nobody is working. The API bill stops with the traffic, but a node you hold is billed for every hour, busy or idle, so an hour in which nobody submitted a task still sits in the cost you are dividing.
The published prices settle the numerator and nothing else
Both sides publish what they charge, so the price of an hour or of a million tokens is the easy half of this calculation. The rates below are from 1 September 2026, and they change without notice.
| Side | Provider | Model or hardware | Price and unit |
|---|---|---|---|
| Per token | Anthropic | Claude Opus 5 | $5 per million input tokens, $25 per million output tokens |
| Per token | Anthropic | Claude Sonnet 5 | $2 per million input tokens, $10 per million output tokens |
| Per token | Anthropic | Claude Haiku 4.5 | $1 per million input tokens, $5 per million output tokens |
| Per token | DeepSeek | DeepSeek-V4-Pro | $0.66 to $1.32 per million input tokens on a cache miss, $1.98 to $3.96 per million output tokens, off-peak to peak |
| Per GPU-hour | Lambda | H100 SXM, on-demand 8x instance | $3.99 |
| Per GPU-hour | Lambda | B200 SXM6, on-demand 8x instance | $6.69 |
| Per GPU-hour | GMI Cloud | H100 | from $2.00 |
| Per GPU-hour | GMI Cloud | B200 | from $4.00 |
The API rows are base rates, so what you pay can be lower than the table says. Prompt caching and batch discounts both bring it down, DeepSeek's cached input costs far less than its cache-miss rate, and its off-peak rates are half its peak rates.
On the two rental pages above, current-generation parts run from $2.00 to $6.69 per GPU-hour, and you are charged for as long as you hold the node.
NVIDIA writes the same equation for the self-hosted side. It published an analysis of inference economics in April 2026, "Rethinking AI TCO" (total cost of ownership), and the top of its fraction, the numerator, is the cost per GPU per hour, either the rate paid to a cloud provider or the amortized hourly cost of hardware you own. You divide that by the output delivered in the same hour. NVIDIA measures output in tokens. For a coding node, use tasks instead.
NVIDIA's own reading is that buyers concentrate on the numerator while the denominator is where cost moves. You can see why from the two halves: the hourly rate is printed on a page you can go and read, but how many tasks come out of the node is set by how you run it.
Hours that finish no task make the finished ones more expensive
Suppose you hold 100 H100s at $4 per GPU-hour and run them 24 hours a day: that is $9,600 a day and $288,000 over a 30-day month. If the cluster sits idle a third of the time, $96,000 of that month bought nothing.
These figures are Cedana's illustration, not a measurement, though the $4 rate sits inside the September 2026 range on the two rental pages above. We estimate that a typical cluster runs near 30% effective utilization.
Three kinds of hour on a coding node go on the bill with no task to show for them: idle time, model swaps, and work that has to start again.
- Idle time is the plainest of the three, because nights and weekends bill the same as a Tuesday afternoon.
- A model swap adds paid minutes while the node cold-starts a model and serves nothing.
- A long job pushed off the node adds more when it has to start again from the beginning.
Swaps are not optional on a coding node
A portfolio of coding models does not fit on one node, so hard tasks run on whichever model happens to be resident. A model swap is itself a cold start, so fast restores let one node hold a portfolio of models instead of one or two.
Our published benchmark measured four open-weight models on one node.
| Model | Checkpoint size | Native start | Restore from checkpoint |
|---|---|---|---|
| MiniMax-M2.7 | 244 GiB | 564 seconds | 57 seconds |
| Kimi-K2.6 | 670 GiB | 1,217 seconds | 63 seconds |
| GLM-5.2-FP8 | 734 GiB | 1,322 seconds | 61 seconds |
| DeepSeek-V4-Pro | 873 GiB | 2,051 seconds | 70 seconds |
Every run was on one node of 8x NVIDIA B200 GPUs with 1.7 TB of system memory, on CUDA 12.9, with the checkpoints held in tmpfs, an in-memory file system rather than a disk. The native start was timed from engine launch to ready-to-serve, including weight loading and full engine initialization. The restore was timed on the same fully initialized engine, from a saved checkpoint back to ready-to-serve. What the same restore takes from a disk or across a network is not in these numbers.
A checkpoint holds the memory the engine had at one moment, including the weights, the key-value (KV) cache, and the runtime state. The checkpoint includes host memory alongside GPU memory, so its size measures the whole saved state rather than what the model occupies on a GPU while it serves.
When the portfolio does not fit on one node, you can keep one node per model, each billed around the clock and mostly idle, or run a single node that changes which model it holds. The single-node option makes the swap cost part of the comparison.
A swap costs its minutes times the node's hourly rate, and the native starts above run from 9.4 to 34.2 minutes. At Lambda's on-demand B200 rate of $6.69 per GPU-hour on 1 September 2026, the 8 GPUs of one node cost $53.52 an hour. So the 2,051-second native start of DeepSeek-V4-Pro is $30.49 of node time serving nothing, against $1.04 for the 70-second restore. That restore figure is the restore alone. The checkpoint it reads was written earlier, while the model was serving.
That arithmetic prices the minutes of one swap, but it says nothing about how many tasks the node finishes in an hour.
No GPU count answers this for you
Most teams want a threshold, a fleet size above which self-hosting starts to win, but the providers that sell both options have published such a threshold and then disagreed with themselves.
Fireworks AI wrote in June 2024 that "Cost-wise, we generally see that it makes sense to switch when achieving volumes nearing ~500,000 tokens/minute", while the summary of the same post starts the range at "~100k+ tokens per minute". Those are 2024 prices on 2024 models, and one post carries two different thresholds. Whatever explains that gap, the line moves as models and hardware change.
The threshold only means something once you run your own inference. If you buy coding models through someone else's API, you have no GPU bill to divide by tasks, whatever you spend on tokens, and from the point where you run your own inference on rented or owned GPUs, the calculation above applies to you.
Three measures test the economics on your own traffic: tasks completed per node-hour, the completion rate, and cost per task.
- Tasks completed per node-hour comes from your own logs, measured on your own task mix.
- The completion rate is tasks completed against tasks requested, and it belongs next to the first number.
- Cost per task is the node's hourly bill divided by tasks per node-hour. Compare that figure with the per-token cost of completing the same tasks through an API.
Hours spent reloading a model and re-running lost work are the part of that bill you can change, because you paid for them and no task came out of them. Cedana is automated GPU checkpointing and migration infrastructure that increases the useful work your GPUs deliver. On a coding node, that means a swap can cost a restore instead of a cold start, and a job pushed off the node can resume from its last checkpoint instead of starting over. Whichever way your own three measures come out, the comparison stays in one unit: what a finished coding task costs you.
Related:
- Which utilization number goes into your own-versus-rent calculation?
- How to put a dollar figure on the GPU-hours that produced nothing
- How much utilization improvement do you need to break even on GPU checkpointing?
- How to swap models on one GPU without a cold start
Common questions
What does it cost, per token, to run a coding model yourself once you count hardware, power and utilization, and how does that compare with an API?
On a per-token API, a task costs the tokens that task consumed times the price per token. On self-hosted GPUs, rented or owned, a task costs the hourly rate times the hours you paid for, divided by the tasks that came out. The two part ways when nobody is working, because the API bill stops with the traffic while a node you hold is billed for every hour, busy or idle.
At what point does it make more sense to own or self-host GPUs than to keep paying an API by the token?
There is no single number that holds for every team. Fireworks AI has published a self-hosting threshold near 500,000 tokens a minute in one place and 100,000-plus in the summary of the same post, so the line moves as models and hardware change. A team paying for coding models entirely through someone else's API has no GPU bill to divide by tasks. Once it runs its own inference on rented or owned GPUs, the cost-per-task arithmetic here applies.
How much is idle or wasted GPU time costing a coding fleet?
Idle time is the plainest waste on a coding node: nights and weekends bill the same as a Tuesday afternoon. Model swaps add paid minutes while the node cold-starts a model and serves nothing, and a long job pushed off the node adds more when it has to start again from the beginning.
What's the right yardstick for comparing what different GPUs or providers deliver for the money on a coding workload?
A finished coding task is the unit that lets a per-token API and self-hosted GPUs be compared. Measure tasks completed per node-hour and the completion rate against tasks requested, then divide the node's hourly bill by tasks per node-hour to get cost per task, and compare that with the per-token cost of the same tasks through an API.
