Files
ai/model/cache_tokens_test.go
zeekay 46951769f6 read the cached tokens a provider already reports
Prompt caching has never been visible here. Measured across the whole span
table: 775 chat requests, and cost_cache_read and cost_cache_write are zero on
every single one — not sparse, zero.

Nothing was broken downstream. usageRecord carries CacheReadTokens, cloud_usage
has the columns, pricing has a cache-read rate, and both the gen_ai span and the
genai event emit the counts when they are above zero. The pipeline was built end
to end. What it never had was a SOURCE: CacheReadTokens was assigned in exactly
three places in this module and all three are test files.

ModelResult was the gap. Every provider fills it in and every controller reads
it, but it had nowhere to put the cached portion of a prompt, so the number died
at the carrier however well a provider cached. Both OpenAI wires report it and
we were discarding both — chat/completions in usage.prompt_tokens_details, and
responses in usage.input_tokens_details.

The counts are a SUBSET of the prompt, not an addition: a provider charges those
tokens at the cache rate instead of the input rate, so summing them into a total
would bill the same tokens twice.
2026-08-13 10:51:16 -07:00

19 lines
756 B
Go

package model
import "testing"
// The cached portion of a prompt has to survive the carrier, because every
// consumer downstream reads it from there: the usage record, the ClickHouse
// columns, the per-rate pricing, the gen_ai span attributes.
func TestModelResultCarriesCacheCounts(t *testing.T) {
r := &ModelResult{PromptTokenCount: 10339, CacheReadTokenCount: 10318}
if r.CacheReadTokenCount != 10318 {
t.Fatalf("cache read = %d, want 10318", r.CacheReadTokenCount)
}
// Cached tokens are a SUBSET of the prompt, never additional to it —
// they are the same tokens priced at the cache-read rate.
if r.CacheReadTokenCount > r.PromptTokenCount {
t.Fatalf("cache read %d exceeds prompt %d", r.CacheReadTokenCount, r.PromptTokenCount)
}
}