mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
hanzo-router salts chat and completions bodies per org (#126), which partitions vLLM's and Hanzo Engine's prefix caches. evo serves Halogen 0.12.3, whose prompt cache is a longest-prefix match across every request and takes no partition key, so the salt reaches it and changes nothing: two orgs that send one system prompt share its cached state, and each answer's usage reports how much of the prompt was cached. A replica now states `cache_salt: false` in the pool when its engine ignores the salt. A request the router salts (an org, or a client salt) is placed only on replicas that honour it while one that can hold the request is up, and waits for a slot there rather than spill. Room still comes first, and with no such replica up the shared one serves, so the preference never becomes an outage. Unsalted requests, explicit targets and replicas that do not set the key are unchanged. The tests are in the kv-isolation gate; four of the five new ones fail with the preference disabled. Co-Authored-By: Claude Opus 5.5 <[email protected]>