mirror of
https://github.com/hanzoai/ai.git
synced 2026-11-20 14:13:23 +00:00
Prompt caching has never been visible here. Measured across the whole span table: 775 chat requests, and cost_cache_read and cost_cache_write are zero on every single one — not sparse, zero. Nothing was broken downstream. usageRecord carries CacheReadTokens, cloud_usage has the columns, pricing has a cache-read rate, and both the gen_ai span and the genai event emit the counts when they are above zero. The pipeline was built end to end. What it never had was a SOURCE: CacheReadTokens was assigned in exactly three places in this module and all three are test files. ModelResult was the gap. Every provider fills it in and every controller reads it, but it had nowhere to put the cached portion of a prompt, so the number died at the carrier however well a provider cached. Both OpenAI wires report it and we were discarding both — chat/completions in usage.prompt_tokens_details, and responses in usage.input_tokens_details. The counts are a SUBSET of the prompt, not an addition: a provider charges those tokens at the cache rate instead of the input rate, so summing them into a total would bill the same tokens twice.
19 lines
756 B
Go
19 lines
756 B
Go
package model
|
|
|
|
import "testing"
|
|
|
|
// The cached portion of a prompt has to survive the carrier, because every
|
|
// consumer downstream reads it from there: the usage record, the ClickHouse
|
|
// columns, the per-rate pricing, the gen_ai span attributes.
|
|
func TestModelResultCarriesCacheCounts(t *testing.T) {
|
|
r := &ModelResult{PromptTokenCount: 10339, CacheReadTokenCount: 10318}
|
|
if r.CacheReadTokenCount != 10318 {
|
|
t.Fatalf("cache read = %d, want 10318", r.CacheReadTokenCount)
|
|
}
|
|
// Cached tokens are a SUBSET of the prompt, never additional to it —
|
|
// they are the same tokens priced at the cache-read rate.
|
|
if r.CacheReadTokenCount > r.PromptTokenCount {
|
|
t.Fatalf("cache read %d exceeds prompt %d", r.CacheReadTokenCount, r.PromptTokenCount)
|
|
}
|
|
}
|