Files
wbin90620andClaude Opus 5.5 de7f17646d feat(router): keep org traffic off a replica whose prefix cache ignores cache_salt (cloud#798)
hanzo-router salts chat and completions bodies per org (#126), which
partitions vLLM's and Hanzo Engine's prefix caches. evo serves Halogen
0.12.3, whose prompt cache is a longest-prefix match across every request
and takes no partition key, so the salt reaches it and changes nothing:
two orgs that send one system prompt share its cached state, and each
answer's usage reports how much of the prompt was cached.

A replica now states `cache_salt: false` in the pool when its engine
ignores the salt. A request the router salts (an org, or a client salt)
is placed only on replicas that honour it while one that can hold the
request is up, and waits for a slot there rather than spill. Room still
comes first, and with no such replica up the shared one serves, so the
preference never becomes an outage. Unsalted requests, explicit targets
and replicas that do not set the key are unchanged.

The tests are in the kv-isolation gate; four of the five new ones fail
with the preference disabled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-20 06:17:44 -08:00
..