Files
ai/bootstrap.go
55da8c8a68 fix(router): explore honors true servability + 5s org-settings TTL & Valkey invalidation bus (#112)
* fix(router): explore/heuristic honor true servability (served ∩ enabled), not just the org allowlist

The per-org policy override replaced the process's real servability predicate
(c.Known) with the org's Enabled allowlist, so the exploration floor and the
servable-first heuristic could pick an ENABLED-but-UNSERVED model (e.g. one still
allowlisted after its provider disconnected) — which 404s downstream. With
ec.Allow left unset the last-resort also relaxed to ALL models, ignoring disabled.

Intersect instead of replace: ec.Known = served ∩ enabled (explore + servable-first
heuristic only pick actually-served models); set ec.Allow = rp.Enabled as the HARD
allowlist so the last-resort may relax servability to a listed model (beats an empty
id) but never routes outside the allowlist. Adds TestExploreStaysServableUnderPolicy
+ TestExploreSamplesOnlyServedAlternatives.

* feat(router): 5s org-settings cache TTL + opt-in Valkey cross-pod invalidation bus

Reconcile the org-settings (router policy / savings↔quality dial) cache TTL from
60s to 5s so a console change converges fleet-wide within 5s (overridable via
ROUTER_SETTINGS_CACHE_TTL_SECONDS). Add an opt-in cross-pod invalidation bus:
when KV_URL is set, every org-settings write broadcasts over Valkey pub/sub so all
pods drop the stale entry sub-second, with the TTL as the fail-safe backstop; a
no-op without KV_URL. Mirrors InitDatastore/InitTelemetry; wired from Bootstrap.

---------

Co-authored-by: hanzo-dev <[email protected]>
2026-07-22 15:51:35 -07:00

264 lines
12 KiB
Go

// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package ai
import (
"encoding/json"
"fmt"
"sync"
"time"
"github.com/hanzoai/ai/log"
"github.com/hanzoai/ai/web"
"github.com/hanzoai/ai/conf"
"github.com/hanzoai/ai/controllers"
"github.com/hanzoai/ai/model"
"github.com/hanzoai/ai/object"
"github.com/hanzoai/ai/proxy"
"github.com/hanzoai/ai/routers"
"github.com/hanzoai/ai/util"
)
// Bootstrap performs the AI runtime initialization shared by BOTH
// entrypoints — the standalone server (cmd/aid) and the embedded unified
// cloud binary (Mount, per HIP-0106). It is the SINGLE source of the
// runtime boot sequence: DB + adapter + tables, model/pricing config, the
// HTTP client, GeoIP/parser, the background maintenance tasks, the
// Commerce-backed balance gate + tier cache + per-key rate limiter, the
// full BeforeRouter/AfterExec filter chain, the session config, and
// the billing usage queue. After wiring those it publishes the
// fully-configured native router via SetHandler so the unified
// binary's /v1/ai/* adapter stops returning 503 "ai runtime not
// initialized".
//
// What Bootstrap deliberately does NOT do (those are standalone-only
// concerns that the embedded binary owns differently, and several would
// break or collide when co-resident in the unified process):
//
// - It binds NO listeners. The native ZAP inference node (port 9999),
// the inter-service ZAP transport (CLOUD_ZAP_PORT, default 9320) and
// the standalone HTTP listener (:httpport) stay in cmd/aid. The unified
// binary serves routers.App through zip on its own :8080 and runs its own
// ZAP at :9653; starting the legacy nodes here would collide.
// - It calls NO util.StopOldInstance — that races on the legacy standalone
// port and is meaningless when the embedded binary never listens there.
// - It installs NO signal handler and never calls os.Exit. cloud.Serve
// owns graceful shutdown for the unified binary; cmd/aid keeps its own
// drain goroutine, wired to the handles returned here.
//
// Bootstrap is guarded by sync.Once: it is safe to call more than once, so
// Mount can call it at mount time and the standalone can call it explicitly
// without double-initializing global runtime state. The cached result
// (including the error) is returned on every call.
//
// It returns an error rather than panicking: several init steps (notably
// the DB open) panic deep inside on a missing/unreachable backend, which
// would take down the whole multi-subsystem cloud binary. Bootstrap
// recovers those into a precise error so Mount can surface
// "mount ai: bootstrap: ..." and the operator sees exactly what failed.
//
// The rate limiter and billing queue created here are exposed via
// RateLimiter() and BillingQueue() for the standalone's graceful-drain
// wiring. Either may be nil (e.g. the billing queue is nil when no Commerce
// endpoint is configured).
func Bootstrap() error {
bootstrapOnce.Do(func() {
bootErr = doBootstrap()
})
return bootErr
}
// RateLimiter returns the per-key rate limiter created by Bootstrap, or nil
// if Bootstrap has not run (or failed before creating it).
func RateLimiter() *routers.RateLimiter { return bootRateLimiter }
// BillingQueue returns the Commerce billing usage queue created by
// Bootstrap, or nil if none was configured / Bootstrap did not reach it.
func BillingQueue() *util.BillingQueue { return bootBillingQueue }
var (
bootstrapOnce sync.Once
bootErr error
bootRateLimiter *routers.RateLimiter
bootBillingQueue *util.BillingQueue
)
func doBootstrap() (err error) {
// Convert the deep DB-open panic (and any other init panic) into a
// precise error so a missing backend fails the AI mount cleanly instead
// of crashing the entire cloud process.
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("ai: bootstrap: %v", r)
}
}()
// Persistence: flags, ORM adapter, schema, DB connection.
object.InitFlag()
object.InitAdapter()
object.CreateTables()
object.InitDb()
// Analytics datastore (datastore): the OLAP ledger behind the usage/spend
// Overview (hanzo.cloud_usage) and the per-tenant trace ledger
// (hanzo.observations). Opt-in via DATASTORE_ADDR; connects in the background
// and is a no-op when unset, so boot never blocks on the warehouse. This runs
// in BOTH cmd/aid and the unified cloud binary (the ai ZAP node does not, so
// the datastore is reached directly, not via a ZAP peer). See object.InitDatastore.
object.InitDatastore()
// OTel GenAI telemetry: one gen_ai span per LLM call → o11y OTLP
// collector. Opt-in via OTEL_EXPORTER_OTLP_ENDPOINT; connects in the
// background and is a no-op when unset, so boot never blocks on the
// collector. Runs in BOTH cmd/aid and the unified cloud binary, mirroring
// InitDatastore. See object.InitTelemetry.
object.InitTelemetry()
// Cross-pod cache-invalidation bus: broadcast org-settings (router policy /
// savings↔quality dial) writes over Valkey pub/sub so a console change converges
// sub-second fleet-wide. Opt-in via KV_URL; connects in the background and is a
// no-op when unset, so boot never blocks on Valkey and the per-pod TTL still
// backstops. Mirrors InitDatastore/InitTelemetry. See object.InitCacheBus.
object.InitCacheBus()
// Model routing/pricing config from YAML. Non-fatal: static fallback.
configPath := conf.GetConfigString("modelConfigPath")
if configPath == "" {
configPath = "conf/models.yaml"
}
if err := controllers.InitModelConfig(configPath); err != nil {
log.Warn("Model config: %v (using static fallback)", err)
}
// Wire the YAML-backed context-window resolver: models.yaml context_window
// becomes the per-model source of truth for max context, overriding the
// static getContextLength table. Without this, glm-5.2 (1M context) fell
// through the table to a 16K fallback and dead-ended long prompts + /compact.
// Nil-safe: if ModelConfig failed to load, the table-only path stays.
if mc := controllers.GetModelConfig(); mc != nil {
model.SetContextWindowResolver(mc.ContextWindow)
}
// Provider HTTP client, GeoIP, request parsing, maintenance tasks.
proxy.InitHttpClient()
util.InitMaxmindFiles()
util.InitIpDb()
util.InitParser()
object.InitCleanupChats()
object.InitStoreCount()
object.InitCommitRecordsTask()
object.InitScanJobProcessor()
object.InitMessageTransactionRetry()
// The in-pod balance ledger (object.GlobalBalanceLedger) reserves/settles in
// process memory — correct ONLY at a SINGLE replica (deploy strategy=Recreate
// + HPA min=max=1). If the operator stamped CLOUD_API_REPLICAS > 1, refuse to
// start rather than silently double-spend across pods; scaling out requires a
// Commerce-atomic conditional reserve first.
if n, ok := object.SinglePodReplicaHint(); ok && n > 1 {
log.Error("FATAL: CLOUD_API_REPLICAS=%d — the in-pod balance ledger double-spends across pods. Refusing to start; implement a Commerce-atomic reserve before scaling out.", n)
panic("cloud-api: in-pod balance ledger requires a single replica (CLOUD_API_REPLICAS>1)")
}
log.Info("balance ledger: single-pod invariant active (reservation ledger requires replicas=1, strategy=Recreate, HPA min=max=1)")
// Commerce balance gate (pre-request balance enforcement).
routers.InitBalanceGate()
// Commerce-backed tier cache — must precede InitRateLimiter so
// DefaultTierFunc can resolve tiers through it.
routers.InitTierCache()
// Per-key rate limiting (env override → tier cache → zen-free).
bootRateLimiter = routers.InitRateLimiter(routers.DefaultTierFunc)
log.Info("Per-key rate limiter initialized (tiers: free=10/min, starter=60/min, pro=300/min, enterprise=1000/min)")
// Filter chain on the ai router, in the same order as the standalone
// surface so the handler enforces the same rewrite/auth/tenant/balance
// pipeline. The router caches the request body itself (no CopyRequestBody
// knob), and the /swagger static is served at the edge, not here.
routers.InstallFilters()
// Session store. API requests authenticate per-request via bearer key/JWT;
// sessions back only the cookie web-admin flow, so their contents are
// incidental. An in-process memory store needs no filesystem (the runtime
// ships read-only) and suits the single-replica runtime. Built once and
// bound to the router so every request has a session bound before the
// filters run (setSessionUser needs a present store). SameSite=Lax.
if web.Sessions == nil {
web.Sessions = web.NewMemorySessions("cloud_session_id", 365*24*time.Hour)
}
routers.App.UseSessions(web.Sessions)
// Optional log adapter reconfig. Guarded: in the embedded binary there
// is no conf/app.conf, so logConfig is empty — json.Unmarshal("") would
// panic. Skip when unset and keep the default logger.
if raw := conf.GetConfigString("logConfig"); raw != "" {
logConfigMap := make(map[string]interface{})
if err := json.Unmarshal([]byte(raw), &logConfigMap); err != nil {
log.Warn("logConfig parse failed: %v (keeping default logger)", err)
} else {
logAdapter := "file"
if v, ok := logConfigMap["adapter"].(string); ok {
logAdapter = v
}
if logAdapter == "console" {
log.Reset()
}
if err := log.SetLogger(logAdapter, raw); err != nil {
log.Warn("logConfig SetLogger failed: %v (keeping default logger)", err)
}
}
}
// Commerce-backed billing usage queue (retried with backoff). nil when
// no Commerce endpoint is configured.
bootBillingQueue = controllers.InitBillingQueue()
if bootBillingQueue != nil {
log.Info("Billing queue started (Commerce endpoint configured)")
}
// Publish the router as the request handler. Every BeforeRouter filter
// inserted above runs on each request the unified binary forwards into
// /v1/ai/*; the session store was bound to the router above.
SetHandler(routers.App)
// Enso router flywheel (HIP-510): the learned-routing loop that trains on real
// traffic. Started HERE — in the SINGLE sync.Once boot sequence BOTH cmd/aid and
// ai.Mount run — so it is the ONE launch site (DRY): the deployed unified binary
// mounts via cloud.Wire → ai.Mount → Bootstrap, so wiring it here (not in a caller)
// is what makes the flywheel actually run in production, where it previously never
// did (retrains: [], rewarded_events: 0). Both Start* self-gate on env and default
// OFF (ROUTER_TRAIN_ENABLED for the trainer; ROUTER_PROBE_RPH+ROUTER_PROBE_TOKEN for
// the probe), so this is dark-by-default and turns real only via deployment config.
// sync.Once guarantees the goroutines are launched exactly once per process.
// StartRouterJudge arms the LLM-as-a-judge dense quality reward (openai_api.go's
// post-response hook): it turns served (prompt, response) pairs into 0..1 quality
// rewards for the flywheel — anonymous, consent-gated, content-transient. Unlike
// the probe/trainer it is DB-backed and ON by default: it reads its config live
// from the "*" GlobalDefaultOwner OrgSettings row (judge.enabled/models/url/sample),
// tunable at admin.hanzo.ai with no restart, and runs the Mean-Field Judge Panel on
// ~10% of eligible (opted-in, non-EU-protected) traffic out of the box. Point
// judge.url at a TEE endpoint for confidential inference (controllers/router_judge.go).
// SeedInternalTrainingConsent first flips our OWN reserved orgs to explicit opt-in
// so the EU per-request guard never suppresses our internal traffic.
controllers.SeedInternalTrainingConsent()
controllers.StartRouterProbe()
controllers.StartRouterTrainer()
controllers.StartRouterJudge()
return nil
}