mirror of
https://github.com/hanzoai/ai.git
synced 2026-11-21 02:36:49 +00:00
* fix(router): explore/heuristic honor true servability (served ∩ enabled), not just the org allowlist The per-org policy override replaced the process's real servability predicate (c.Known) with the org's Enabled allowlist, so the exploration floor and the servable-first heuristic could pick an ENABLED-but-UNSERVED model (e.g. one still allowlisted after its provider disconnected) — which 404s downstream. With ec.Allow left unset the last-resort also relaxed to ALL models, ignoring disabled. Intersect instead of replace: ec.Known = served ∩ enabled (explore + servable-first heuristic only pick actually-served models); set ec.Allow = rp.Enabled as the HARD allowlist so the last-resort may relax servability to a listed model (beats an empty id) but never routes outside the allowlist. Adds TestExploreStaysServableUnderPolicy + TestExploreSamplesOnlyServedAlternatives. * feat(router): 5s org-settings cache TTL + opt-in Valkey cross-pod invalidation bus Reconcile the org-settings (router policy / savings↔quality dial) cache TTL from 60s to 5s so a console change converges fleet-wide within 5s (overridable via ROUTER_SETTINGS_CACHE_TTL_SECONDS). Add an opt-in cross-pod invalidation bus: when KV_URL is set, every org-settings write broadcasts over Valkey pub/sub so all pods drop the stale entry sub-second, with the TTL as the fail-safe backstop; a no-op without KV_URL. Mirrors InitDatastore/InitTelemetry; wired from Bootstrap. --------- Co-authored-by: hanzo-dev <[email protected]>
264 lines
12 KiB
Go
264 lines
12 KiB
Go
// Copyright 2023-2025 Hanzo AI Inc. All Rights Reserved.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package ai
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/hanzoai/ai/log"
|
|
"github.com/hanzoai/ai/web"
|
|
|
|
"github.com/hanzoai/ai/conf"
|
|
"github.com/hanzoai/ai/controllers"
|
|
"github.com/hanzoai/ai/model"
|
|
"github.com/hanzoai/ai/object"
|
|
"github.com/hanzoai/ai/proxy"
|
|
"github.com/hanzoai/ai/routers"
|
|
"github.com/hanzoai/ai/util"
|
|
)
|
|
|
|
// Bootstrap performs the AI runtime initialization shared by BOTH
|
|
// entrypoints — the standalone server (cmd/aid) and the embedded unified
|
|
// cloud binary (Mount, per HIP-0106). It is the SINGLE source of the
|
|
// runtime boot sequence: DB + adapter + tables, model/pricing config, the
|
|
// HTTP client, GeoIP/parser, the background maintenance tasks, the
|
|
// Commerce-backed balance gate + tier cache + per-key rate limiter, the
|
|
// full BeforeRouter/AfterExec filter chain, the session config, and
|
|
// the billing usage queue. After wiring those it publishes the
|
|
// fully-configured native router via SetHandler so the unified
|
|
// binary's /v1/ai/* adapter stops returning 503 "ai runtime not
|
|
// initialized".
|
|
//
|
|
// What Bootstrap deliberately does NOT do (those are standalone-only
|
|
// concerns that the embedded binary owns differently, and several would
|
|
// break or collide when co-resident in the unified process):
|
|
//
|
|
// - It binds NO listeners. The native ZAP inference node (port 9999),
|
|
// the inter-service ZAP transport (CLOUD_ZAP_PORT, default 9320) and
|
|
// the standalone HTTP listener (:httpport) stay in cmd/aid. The unified
|
|
// binary serves routers.App through zip on its own :8080 and runs its own
|
|
// ZAP at :9653; starting the legacy nodes here would collide.
|
|
// - It calls NO util.StopOldInstance — that races on the legacy standalone
|
|
// port and is meaningless when the embedded binary never listens there.
|
|
// - It installs NO signal handler and never calls os.Exit. cloud.Serve
|
|
// owns graceful shutdown for the unified binary; cmd/aid keeps its own
|
|
// drain goroutine, wired to the handles returned here.
|
|
//
|
|
// Bootstrap is guarded by sync.Once: it is safe to call more than once, so
|
|
// Mount can call it at mount time and the standalone can call it explicitly
|
|
// without double-initializing global runtime state. The cached result
|
|
// (including the error) is returned on every call.
|
|
//
|
|
// It returns an error rather than panicking: several init steps (notably
|
|
// the DB open) panic deep inside on a missing/unreachable backend, which
|
|
// would take down the whole multi-subsystem cloud binary. Bootstrap
|
|
// recovers those into a precise error so Mount can surface
|
|
// "mount ai: bootstrap: ..." and the operator sees exactly what failed.
|
|
//
|
|
// The rate limiter and billing queue created here are exposed via
|
|
// RateLimiter() and BillingQueue() for the standalone's graceful-drain
|
|
// wiring. Either may be nil (e.g. the billing queue is nil when no Commerce
|
|
// endpoint is configured).
|
|
func Bootstrap() error {
|
|
bootstrapOnce.Do(func() {
|
|
bootErr = doBootstrap()
|
|
})
|
|
return bootErr
|
|
}
|
|
|
|
// RateLimiter returns the per-key rate limiter created by Bootstrap, or nil
|
|
// if Bootstrap has not run (or failed before creating it).
|
|
func RateLimiter() *routers.RateLimiter { return bootRateLimiter }
|
|
|
|
// BillingQueue returns the Commerce billing usage queue created by
|
|
// Bootstrap, or nil if none was configured / Bootstrap did not reach it.
|
|
func BillingQueue() *util.BillingQueue { return bootBillingQueue }
|
|
|
|
var (
|
|
bootstrapOnce sync.Once
|
|
bootErr error
|
|
bootRateLimiter *routers.RateLimiter
|
|
bootBillingQueue *util.BillingQueue
|
|
)
|
|
|
|
func doBootstrap() (err error) {
|
|
// Convert the deep DB-open panic (and any other init panic) into a
|
|
// precise error so a missing backend fails the AI mount cleanly instead
|
|
// of crashing the entire cloud process.
|
|
defer func() {
|
|
if r := recover(); r != nil {
|
|
err = fmt.Errorf("ai: bootstrap: %v", r)
|
|
}
|
|
}()
|
|
|
|
// Persistence: flags, ORM adapter, schema, DB connection.
|
|
object.InitFlag()
|
|
object.InitAdapter()
|
|
object.CreateTables()
|
|
object.InitDb()
|
|
|
|
// Analytics datastore (datastore): the OLAP ledger behind the usage/spend
|
|
// Overview (hanzo.cloud_usage) and the per-tenant trace ledger
|
|
// (hanzo.observations). Opt-in via DATASTORE_ADDR; connects in the background
|
|
// and is a no-op when unset, so boot never blocks on the warehouse. This runs
|
|
// in BOTH cmd/aid and the unified cloud binary (the ai ZAP node does not, so
|
|
// the datastore is reached directly, not via a ZAP peer). See object.InitDatastore.
|
|
object.InitDatastore()
|
|
|
|
// OTel GenAI telemetry: one gen_ai span per LLM call → o11y OTLP
|
|
// collector. Opt-in via OTEL_EXPORTER_OTLP_ENDPOINT; connects in the
|
|
// background and is a no-op when unset, so boot never blocks on the
|
|
// collector. Runs in BOTH cmd/aid and the unified cloud binary, mirroring
|
|
// InitDatastore. See object.InitTelemetry.
|
|
object.InitTelemetry()
|
|
|
|
// Cross-pod cache-invalidation bus: broadcast org-settings (router policy /
|
|
// savings↔quality dial) writes over Valkey pub/sub so a console change converges
|
|
// sub-second fleet-wide. Opt-in via KV_URL; connects in the background and is a
|
|
// no-op when unset, so boot never blocks on Valkey and the per-pod TTL still
|
|
// backstops. Mirrors InitDatastore/InitTelemetry. See object.InitCacheBus.
|
|
object.InitCacheBus()
|
|
|
|
// Model routing/pricing config from YAML. Non-fatal: static fallback.
|
|
configPath := conf.GetConfigString("modelConfigPath")
|
|
if configPath == "" {
|
|
configPath = "conf/models.yaml"
|
|
}
|
|
if err := controllers.InitModelConfig(configPath); err != nil {
|
|
log.Warn("Model config: %v (using static fallback)", err)
|
|
}
|
|
|
|
// Wire the YAML-backed context-window resolver: models.yaml context_window
|
|
// becomes the per-model source of truth for max context, overriding the
|
|
// static getContextLength table. Without this, glm-5.2 (1M context) fell
|
|
// through the table to a 16K fallback and dead-ended long prompts + /compact.
|
|
// Nil-safe: if ModelConfig failed to load, the table-only path stays.
|
|
if mc := controllers.GetModelConfig(); mc != nil {
|
|
model.SetContextWindowResolver(mc.ContextWindow)
|
|
}
|
|
|
|
// Provider HTTP client, GeoIP, request parsing, maintenance tasks.
|
|
proxy.InitHttpClient()
|
|
util.InitMaxmindFiles()
|
|
util.InitIpDb()
|
|
util.InitParser()
|
|
object.InitCleanupChats()
|
|
object.InitStoreCount()
|
|
object.InitCommitRecordsTask()
|
|
object.InitScanJobProcessor()
|
|
object.InitMessageTransactionRetry()
|
|
|
|
// The in-pod balance ledger (object.GlobalBalanceLedger) reserves/settles in
|
|
// process memory — correct ONLY at a SINGLE replica (deploy strategy=Recreate
|
|
// + HPA min=max=1). If the operator stamped CLOUD_API_REPLICAS > 1, refuse to
|
|
// start rather than silently double-spend across pods; scaling out requires a
|
|
// Commerce-atomic conditional reserve first.
|
|
if n, ok := object.SinglePodReplicaHint(); ok && n > 1 {
|
|
log.Error("FATAL: CLOUD_API_REPLICAS=%d — the in-pod balance ledger double-spends across pods. Refusing to start; implement a Commerce-atomic reserve before scaling out.", n)
|
|
panic("cloud-api: in-pod balance ledger requires a single replica (CLOUD_API_REPLICAS>1)")
|
|
}
|
|
log.Info("balance ledger: single-pod invariant active (reservation ledger requires replicas=1, strategy=Recreate, HPA min=max=1)")
|
|
|
|
// Commerce balance gate (pre-request balance enforcement).
|
|
routers.InitBalanceGate()
|
|
|
|
// Commerce-backed tier cache — must precede InitRateLimiter so
|
|
// DefaultTierFunc can resolve tiers through it.
|
|
routers.InitTierCache()
|
|
|
|
// Per-key rate limiting (env override → tier cache → zen-free).
|
|
bootRateLimiter = routers.InitRateLimiter(routers.DefaultTierFunc)
|
|
log.Info("Per-key rate limiter initialized (tiers: free=10/min, starter=60/min, pro=300/min, enterprise=1000/min)")
|
|
|
|
// Filter chain on the ai router, in the same order as the standalone
|
|
// surface so the handler enforces the same rewrite/auth/tenant/balance
|
|
// pipeline. The router caches the request body itself (no CopyRequestBody
|
|
// knob), and the /swagger static is served at the edge, not here.
|
|
routers.InstallFilters()
|
|
|
|
// Session store. API requests authenticate per-request via bearer key/JWT;
|
|
// sessions back only the cookie web-admin flow, so their contents are
|
|
// incidental. An in-process memory store needs no filesystem (the runtime
|
|
// ships read-only) and suits the single-replica runtime. Built once and
|
|
// bound to the router so every request has a session bound before the
|
|
// filters run (setSessionUser needs a present store). SameSite=Lax.
|
|
if web.Sessions == nil {
|
|
web.Sessions = web.NewMemorySessions("cloud_session_id", 365*24*time.Hour)
|
|
}
|
|
routers.App.UseSessions(web.Sessions)
|
|
|
|
// Optional log adapter reconfig. Guarded: in the embedded binary there
|
|
// is no conf/app.conf, so logConfig is empty — json.Unmarshal("") would
|
|
// panic. Skip when unset and keep the default logger.
|
|
if raw := conf.GetConfigString("logConfig"); raw != "" {
|
|
logConfigMap := make(map[string]interface{})
|
|
if err := json.Unmarshal([]byte(raw), &logConfigMap); err != nil {
|
|
log.Warn("logConfig parse failed: %v (keeping default logger)", err)
|
|
} else {
|
|
logAdapter := "file"
|
|
if v, ok := logConfigMap["adapter"].(string); ok {
|
|
logAdapter = v
|
|
}
|
|
if logAdapter == "console" {
|
|
log.Reset()
|
|
}
|
|
if err := log.SetLogger(logAdapter, raw); err != nil {
|
|
log.Warn("logConfig SetLogger failed: %v (keeping default logger)", err)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Commerce-backed billing usage queue (retried with backoff). nil when
|
|
// no Commerce endpoint is configured.
|
|
bootBillingQueue = controllers.InitBillingQueue()
|
|
if bootBillingQueue != nil {
|
|
log.Info("Billing queue started (Commerce endpoint configured)")
|
|
}
|
|
|
|
// Publish the router as the request handler. Every BeforeRouter filter
|
|
// inserted above runs on each request the unified binary forwards into
|
|
// /v1/ai/*; the session store was bound to the router above.
|
|
SetHandler(routers.App)
|
|
|
|
// Enso router flywheel (HIP-510): the learned-routing loop that trains on real
|
|
// traffic. Started HERE — in the SINGLE sync.Once boot sequence BOTH cmd/aid and
|
|
// ai.Mount run — so it is the ONE launch site (DRY): the deployed unified binary
|
|
// mounts via cloud.Wire → ai.Mount → Bootstrap, so wiring it here (not in a caller)
|
|
// is what makes the flywheel actually run in production, where it previously never
|
|
// did (retrains: [], rewarded_events: 0). Both Start* self-gate on env and default
|
|
// OFF (ROUTER_TRAIN_ENABLED for the trainer; ROUTER_PROBE_RPH+ROUTER_PROBE_TOKEN for
|
|
// the probe), so this is dark-by-default and turns real only via deployment config.
|
|
// sync.Once guarantees the goroutines are launched exactly once per process.
|
|
// StartRouterJudge arms the LLM-as-a-judge dense quality reward (openai_api.go's
|
|
// post-response hook): it turns served (prompt, response) pairs into 0..1 quality
|
|
// rewards for the flywheel — anonymous, consent-gated, content-transient. Unlike
|
|
// the probe/trainer it is DB-backed and ON by default: it reads its config live
|
|
// from the "*" GlobalDefaultOwner OrgSettings row (judge.enabled/models/url/sample),
|
|
// tunable at admin.hanzo.ai with no restart, and runs the Mean-Field Judge Panel on
|
|
// ~10% of eligible (opted-in, non-EU-protected) traffic out of the box. Point
|
|
// judge.url at a TEE endpoint for confidential inference (controllers/router_judge.go).
|
|
// SeedInternalTrainingConsent first flips our OWN reserved orgs to explicit opt-in
|
|
// so the EU per-request guard never suppresses our internal traffic.
|
|
controllers.SeedInternalTrainingConsent()
|
|
controllers.StartRouterProbe()
|
|
controllers.StartRouterTrainer()
|
|
controllers.StartRouterJudge()
|
|
|
|
return nil
|
|
}
|