Over the same window, on a single key, three systems reported three sums: $0.20 in the provider's dashboard, $0.07231 in Langfuse, $0.0711 in Grafana. The key existed only for this bench, so no foreign spend could have leaked in. The question looked like “which of the three is lying”.
The right question turned out to be a different one.
How many independent measurements are there
Langfuse and Grafana are not two witnesses. Both took their number from the same formula: tokens_from_the_response × a static price in the config. They agreed with each other not because both were right, but because they shared an input. There is exactly one independent source, and it is the provider's ledger — the thing the money is actually drawn against.
| Source | Where the number comes from | Independent |
|---|---|---|
| provider ledger | what you are billed against | yes |
| Langfuse | the “tokens × price” formula | no |
| Prometheus / Grafana | the same formula | no |
Two dashboards agreeing reads as corroboration and is not. Witnesses are counted by data source, not by number of panels. After that the investigation narrowed to one question: why does the single formula undercount by nearly three times.
The prices were correct
First suspicion: the prices in the config had drifted. That hypothesis is cheap to kill — the aggregator's model catalogue is public and needs no key. The check matched to the third decimal: 0.43 / 0.87 against 0.435 / 0.87 for one model, an exact 0.10 / 0.40 for the other.
One detail surfaced along the way and mattered later: for a reasoning model the catalogue carries a separate price line, internal_reasoning, not equal to the completion price.
The clue: the same multiplier in both directions
A direct probe with usage accounting enabled showed the gap was not everywhere:
| Model | My formula | Provider charged | Ratio |
|---|---|---|---|
| gemini class | $6.67e-5 | $6.67e-5 | 1.00 |
| deepseek class | $2.81e-4 | $4.82e-4 | 1.7× |
One model matched to the cent, the other was off by 1.7. What decides the case is not the gap but its shape. In cost_details the prompt portion had been priced at 0.745/M instead of 0.435/M — and the 1.7 coefficient was identical on the prompt and on the completion.
With correct prices, one and the same multiplier in both directions at once can mean only one thing: the token counts in the response are not the counts you are billed on.
What sat underneath the discrepancy
The aggregator billed on the provider model's native tokens, while usage.prompt_tokens and usage.completion_tokens returned normalized ones — mapped onto a common scale so different models can be compared against each other. One model has a denser native tokenizer, hence the 1.7×. For the other, normalized equals native, which is why it matched.
The rest is arithmetic: the most expensive and most heavily used alias on the bench was exactly the one that diverged. It dragged the whole total from $0.20 down to $0.07.
The split between the two counters has not disappeared — it still lives in the API. The generation-stats endpoint documents both sets as separate fields: tokens_prompt and tokens_completion against native_tokens_prompt, native_tokens_completion, native_tokens_reasoning, native_tokens_cached — the latter marked “as reported by provider”.
The rule that outlives any single vendor
Your own cost formula is an estimate, not a measurement. Its place is the fallback path. The authoritative number comes from whoever issues the bill.
In practice that means four things:
- Take
usage.costandcost_detailsfrom the provider's response as the source of truth. - Keep the static price table as a safety net: you need it where the provider returns no price at all — a local tier, for instance, where the price is zero anyway.
- Write into the config that the prices are a fallback. Otherwise in six months they will be read as truth again.
- Verify that the client SDK does not strip non-standard fields.
costandcost_detailsare not part of the OpenAIusageschema and land among the response model's extra fields; in practice they survive the SDK and are reachable throughgetattr,model_dumpandmodel_extra— but confirm that with a probe rather than with faith.
The recipe expired, the rule did not
The measurement was taken on 14 July 2026. Before publishing I checked the provider's documentation, and it describes different behaviour.
| What | Measured 14 Jul 2026 | Docs as of 23 Aug 2026 |
|---|---|---|
| counters in the response | normalized | “calculated using the model's native tokenizer” |
| billing basis | native tokens | “pricing are based on these native token counts” |
| enabling usage accounting | a parameter in the request body | “always included automatically” |
The provider aligned the behaviour: the counters in the response became native, and the former usage: {include: true} parameter is now deprecated and, per the docs, has no effect. There is no reason to set it any more — and it is exactly the kind of line you should not copy out of a six-month-old article.
None of that refutes the measurement: the prices were checked against the live catalogue, one model matched to the cent, the other diverged by an identical multiplier in both directions. The vendor changed, the fact did not. Reading the authoritative number instead of recomputing it yourself is still the only way not to diverge from the ledger — and it says nothing about spending fewer tokens, which is a separate conversation.
Why two terms of arithmetic are not enough
The formula prompt × price_in + completion × price_out assumes the whole generation is described by two numbers. For reasoning models that is wrong in at least two ways.
First, the separate price line: reasoning can be billed at its own rate, different from completion. Second, non-obvious nesting. In my probe the reasoning tokens sat inside completion_tokens (261 of 317) — but that is a fact about one provider-model pair, not a general rule. If another pair puts them alongside, the formula silently loses an entire cost line.
Caching belongs here too: input_cache_read and cache_write_tokens are billed separately. In my run they were zero, so how the formula behaves with a warm cache is something I did not test.
There is also a neighbouring class where no price exists at all: a generation that ends in a provider-side error arrives with status 200, no content and no cost line — no protective layer sees it. There the price is missing. Here it exists and is simply computed on the wrong tokens.
The check lies too
The reconciliation did not land on zero at the first attempt, and both misses were in the check itself rather than in the system.
First, a grep went against a metric name that did not exist and returned zero. Zero looks like “the fix did not work” while it means “I asked the wrong question”.
Then the ledger delta, read immediately after the run, came out smaller than what had been recorded: child workflows were still sending their calls last, and the provider's account counter caught up with a lag. A couple of minutes later the numbers matched exactly — $0.019571 against $0.019571.
Both traps are one class: the check lied about itself, not about its subject. The same class as a green security-policy header that proves nothing, because it was taken in a state where the relevant requests do not happen yet.
What remains once the vendor fixes everything
The native-versus-normalized split is a detail of one aggregator, and it has already been smoothed over. The mechanism underneath it has not.
Any layer between you and the model normalizes: it maps counters onto a common scale, aggregates, rounds, adds billing lines of its own. As long as cost is computed by multiplication on your side, you are measuring your model of the spend rather than the spend. It will start diverging quietly, on exactly the day something changes upstream.
A simple rule: cost gets one source, and it is not your formula.