Compare commits

..
Author SHA1 Message Date
zandGitHub 180fd74369 chore(deps): bump commerce v1.49.3 — bounded org-resolution cache on the auth path (#340)
Auth-path org resolution hit the datastore on every request, allocating the
Organization before the blocking store call, so requests stalled on the
connection pool each pinned one and the heap tracked the backlog. Confirmed
from a live goroutine profile: 46 waiters in sql.(*DB).conn under
org.Resolve <- IAMTokenRequired, organization.New at 26.4% of a 1301MB heap.

Carries three fixes uncovered while landing it:
- follow commerce's resolver consolidation (middleware/svcorg -> pkg/org)
- point the go-unit test list at clients/flags; the stale clients/featureflags
  path failed setup on a missing directory and had CI/CD red on main
- assert the post-#331 flags contract: runtime flags ignore env, boot-time
  ReadOnly rows still read it. That test asserted the override #331 removed
  and never ran because of the stale path above.
2026-07-19 02:47:02 -07:00
78e0d35199 analytics(ingest): PostHog-wire uuid->idempotent MessageID + utm_* attribution mapping (#338)
Co-authored-by: hanzo-dev <dev@hanzo.ai>
2026-07-19 00:05:43 -07:00
hanzo-dev e80a6fc641 chore(cloud): vendor hanzoai/ai v1.826.6 — DO model catalog + mean-field + judge-panel
Brings the full run into the deployed service: 55-model DO GenAI catalog (Claude
opus-4.8/sonnet-5/fable-5/haiku, GPT-5.6/5.5/4o/o3, deepseek-v4-pro, llama-4, qwen,
glm, kimi — capabilities declared per live probe), the mean-field congestion router
(gated), the live /v1/router/judge-panel endpoint, the Mean-Field Judge Panel, and
geo-aware consent. Prod model ConfigMap (universe) syncs the catalog data separately.

Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
2026-07-18 23:45:24 -07:00
10 changed files with 147 additions and 583 deletions
-1
View File
@@ -103,7 +103,6 @@ func routes(app *zip.App, s *cloud.Service[core.State]) {
app.Get("/v1/admin/products", core.Guard(s, products))
app.Get("/v1/admin/compute", core.Guard(s, compute))
app.Get("/v1/admin/o11y", core.Guard(s, o11y))
app.Get("/v1/admin/aimetrics", core.Guard(s, aimetrics))
app.Post("/v1/admin/sync", core.Guard(s, syncNow))
// Product analytics — org-scoped (SuperAdmin: all-orgs; org admin: their own org).
-392
View File
@@ -1,392 +0,0 @@
// Copyright 2023-2026 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package admin
// aimetrics — GET /v1/admin/aimetrics, the GLOBAL fleet-wide AI / training / eval
// read that powers the operator's AI-metrics board on admin.hanzo.ai. It is the
// AI-and-eval-focused companion to o11y (o11y.go): where o11y answers "how is the
// FLEET behaving" (RED metrics, logs, usage), this answers "how are the MODELS and
// EVALS doing" — LLM generations, per-model spend, and eval-run quality/progress —
// over the SAME ONE datastore (Datastore), the SAME shared client
// (aiobject.DatastoreQuery), no second connection.
//
// Signals, each from its canonical table in the one datastore:
// - LLM generations → langfuse.observations : generations, cost (USD), latency
// (fleet-wide; honest-empty until the
// Langfuse ingest lands rows)
// - Per-model usage → hanzo.cloud_usage : requests, tokens, cost per model
// (the live usage ledger the ai gateway
// writes — populated today)
// - Eval runs → hanzo.eval_traces : traces, runs, datasets, models under
// test, per-trace latency
// - Eval progress → hanzo.eval_scores : score count, avg score, per-score-name
// distribution, recent-run averages, and
// the avg-score-over-time TREND — the
// training/eval progress signal
//
// The eval_traces / eval_scores tables are OWNED and written by the eval telemetry
// store (clients/eval/telemetry.go) — the SAME warehouse, same db ("hanzo"), same
// shared aiobject client. admin only READS them here. There is deliberately no
// "training_progress" table: the router's per-request training events live in the ai
// OLTP Postgres (object.RoutingEvent), NOT the OLAP warehouse, so the honest
// warehouse-side progress signal is the eval-score trend, not a routing table.
//
// SUPERADMIN ONLY (the core.Guard wrap in admin.go), all-orgs, no org filter — the
// one place a fleet operator crosses tenants for AI/eval metrics; a non-admin bearer
// is refused 403 before a single row is read. Fail-closed.
//
// Honest by construction, exactly like o11y/compute: no datastore connected → the
// real empty aggregate, never a fabricated fleet; and every signal degrades
// INDEPENDENTLY — a table that is absent or a column that differs contributes its
// zero-value (the enclosing `if err == nil`), never a failure, so the board always
// renders what the datastore actually holds. admin READS only; it owns and creates
// NO table. Money from cloud_usage is USD cents, from langfuse is USD; latency is
// milliseconds; time bounds are POSITIONAL parameters (never interpolated), and the
// bucket interval is a server-side constant — injection-safe.
import (
"strconv"
"time"
aiobject "github.com/hanzoai/ai/object"
"github.com/hanzoai/cloud"
"github.com/hanzoai/cloud/clients/admin/core"
"github.com/zap-proto/zip"
)
// Fully-qualified datastore tables. admin only READS these — the ai gateway owns
// hanzo.cloud_usage, Langfuse owns langfuse.observations, and the eval telemetry
// store (clients/eval) owns hanzo.eval_traces / hanzo.eval_scores.
const (
aimUsageTable = "hanzo.cloud_usage"
aimLangfuseObs = "langfuse.observations"
aimEvalTraces = "hanzo.eval_traces"
aimEvalScores = "hanzo.eval_scores"
aimTopN = 12
)
// aiMetrics is the whole AI-metrics board payload.
type aiMetrics struct {
Range string `json:"range"`
Start string `json:"start"`
End string `json:"end"`
Langfuse aimLangfuse `json:"langfuse"`
Usage aimUsage `json:"usage"`
Evals aimEvals `json:"evals"`
TopModels []aimModelStat `json:"topModels"` // cloud_usage per-model (populated today)
LangfuseModels []aimLfModelStat `json:"langfuseModels"` // langfuse per-model (honest-empty today)
ScoreNames []aimScoreStat `json:"scoreNames"` // eval_scores per score-name
EvalRuns []aimRunStat `json:"evalRuns"` // recent eval runs (progress)
ScoreSeries []aimScorePoint `json:"scoreSeries"` // avg eval score over time (progress trend)
}
// aimLangfuse is the fleet-wide Langfuse generation rollup (honest-empty today).
// Cost is USD (Langfuse's native unit); latency is milliseconds (end_time-start_time).
type aimLangfuse struct {
Generations int64 `json:"generations"`
CostUsd float64 `json:"costUsd"`
LatencyMsAvg float64 `json:"latencyMsAvg"`
LatencyMsP95 float64 `json:"latencyMsP95"`
}
// aimUsage is the fleet LLM-usage KPI band from the live cloud_usage ledger.
type aimUsage struct {
Requests int64 `json:"requests"`
Tokens int64 `json:"tokens"`
PromptTokens int64 `json:"promptTokens"`
CompletionTokens int64 `json:"completionTokens"`
CostCents int64 `json:"costCents"`
Models int64 `json:"models"`
}
// aimEvals is the fleet eval KPI band: the trace half (eval_traces) and the score
// half (eval_scores). LatencyMsAvg is the mean model-under-test call window.
type aimEvals struct {
Runs int64 `json:"runs"`
Traces int64 `json:"traces"`
Datasets int64 `json:"datasets"`
Models int64 `json:"models"`
LatencyMsAvg float64 `json:"latencyMsAvg"`
Scores int64 `json:"scores"`
ScoreNames int64 `json:"scoreNames"`
AvgScore float64 `json:"avgScore"`
}
// aimModelStat is one row of the per-model usage leaderboard (cloud_usage).
type aimModelStat struct {
Model string `json:"model"`
Requests int64 `json:"requests"`
Tokens int64 `json:"tokens"`
CostCents int64 `json:"costCents"`
}
// aimLfModelStat is one row of the per-model Langfuse leaderboard (honest-empty today).
type aimLfModelStat struct {
Model string `json:"model"`
Generations int64 `json:"generations"`
CostUsd float64 `json:"costUsd"`
}
// aimScoreStat is one row of the per-score-name eval leaderboard (eval_scores).
type aimScoreStat struct {
Name string `json:"name"`
Count int64 `json:"count"`
AvgValue float64 `json:"avgValue"`
MinValue float64 `json:"minValue"`
MaxValue float64 `json:"maxValue"`
}
// aimRunStat is one recent eval run: its dataset, how many scores it recorded, its
// mean score, and when it last ran — the run-level eval-progress row.
type aimRunStat struct {
RunName string `json:"runName"`
Dataset string `json:"dataset"`
Scores int64 `json:"scores"`
AvgValue float64 `json:"avgValue"`
LastTs string `json:"lastTs"`
}
// aimScorePoint is one bucket of the avg-eval-score-over-time trend.
type aimScorePoint struct {
Ts string `json:"ts"`
AvgValue float64 `json:"avgValue"`
Count int64 `json:"count"`
}
// aimetrics answers GET /v1/admin/aimetrics. ?range=24h|7d|30d bounds the window
// (default 30d). SUPERADMIN ONLY (core.Guard). Every signal degrades independently:
// a table that is absent or errors contributes its zero-value, never a failure — the
// board always renders what the datastore actually holds.
func aimetrics(s *cloud.Service[core.State], c *zip.Ctx) error {
ctx := c.Context()
rangeLabel := o11yRange(c.Query("range"))
since := computeSince(rangeLabel)
payload := aiMetrics{
Range: rangeLabel,
Start: since.Format(time.RFC3339),
End: time.Now().UTC().Format(time.RFC3339),
TopModels: []aimModelStat{},
LangfuseModels: []aimLfModelStat{},
ScoreNames: []aimScoreStat{},
EvalRuns: []aimRunStat{},
ScoreSeries: []aimScorePoint{},
}
// Honest-empty when the warehouse is not connected: the board renders its zero
// state, never a fabricated fleet.
if !aiobject.DatastoreEnabled() {
return core.OK(c, payload)
}
sinceTS := chTS(since) // DateTime literal — cloud_usage.timestamp, langfuse.start_time, eval_*.ts
interval := o11yBucket(rangeLabel)
// ── Langfuse generations (fleet) — honest-empty until ingest lands rows ──
if rows, err := aiobject.DatastoreQuery(ctx, aimLangfuseTotalsSQL(), sinceTS); err == nil {
r := firstRowOr(rows)
payload.Langfuse.Generations = chInt64(r["gens"])
payload.Langfuse.CostUsd = chFloat64(r["cost"])
}
// Langfuse latency (separate query so a Nullable end_time / column mismatch never
// zeroes the proven generations+cost number above).
if rows, err := aiobject.DatastoreQuery(ctx, aimLangfuseLatencySQL(), sinceTS); err == nil {
r := firstRowOr(rows)
payload.Langfuse.LatencyMsAvg = chFloat64(r["lat_avg"])
payload.Langfuse.LatencyMsP95 = chFloat64(r["lat_p95"])
}
// Langfuse per-model.
if rows, err := aiobject.DatastoreQuery(ctx, aimLangfuseModelsSQL(), sinceTS); err == nil {
payload.LangfuseModels = lfModelsFromRows(rows)
}
// ── Per-model usage (fleet) from the live cloud_usage ledger ──
if rows, err := aiobject.DatastoreQuery(ctx, aimUsageTotalsSQL(), sinceTS); err == nil {
fillAimUsage(&payload.Usage, firstRowOr(rows))
}
if rows, err := aiobject.DatastoreQuery(ctx, aimTopModelsSQL(), sinceTS); err == nil {
payload.TopModels = aimModelsFromRows(rows)
}
// ── Evals (fleet): traces + scores + progress ──
if rows, err := aiobject.DatastoreQuery(ctx, aimEvalTracesSQL(), sinceTS); err == nil {
fillAimEvalTraces(&payload.Evals, firstRowOr(rows))
}
if rows, err := aiobject.DatastoreQuery(ctx, aimEvalScoresSQL(), sinceTS); err == nil {
fillAimEvalScores(&payload.Evals, firstRowOr(rows))
}
if rows, err := aiobject.DatastoreQuery(ctx, aimScoreNamesSQL(), sinceTS); err == nil {
payload.ScoreNames = scoreNamesFromRows(rows)
}
if rows, err := aiobject.DatastoreQuery(ctx, aimEvalRunsSQL(), sinceTS); err == nil {
payload.EvalRuns = evalRunsFromRows(rows)
}
if rows, err := aiobject.DatastoreQuery(ctx, aimScoreSeriesSQL(interval), sinceTS); err == nil {
payload.ScoreSeries = scoreSeriesFromRows(rows)
}
return core.OK(c, payload)
}
// ── pure SQL builders (static SQL + one positional time bound; unit-tested) ──
func aimLangfuseTotalsSQL() string {
return "SELECT count() AS gens, toFloat64(sum(total_cost)) AS cost FROM " + aimLangfuseObs +
" WHERE type = 'GENERATION' AND start_time >= ?"
}
func aimLangfuseLatencySQL() string {
lat := "(toUnixTimestamp64Milli(end_time) - toUnixTimestamp64Milli(start_time))"
return "SELECT round(avg(" + lat + "), 2) AS lat_avg, round(quantile(0.95)(" + lat + "), 2) AS lat_p95 " +
"FROM " + aimLangfuseObs + " WHERE type = 'GENERATION' AND start_time >= ? AND end_time > start_time"
}
func aimLangfuseModelsSQL() string {
return "SELECT provided_model_name AS model, count() AS gens, toFloat64(sum(total_cost)) AS cost " +
"FROM " + aimLangfuseObs + " WHERE type = 'GENERATION' AND start_time >= ? AND provided_model_name != '' " +
"GROUP BY model ORDER BY gens DESC LIMIT " + strconv.Itoa(aimTopN)
}
func aimUsageTotalsSQL() string {
return "SELECT count() AS requests, sum(total_tokens) AS tokens, " +
"sum(prompt_tokens) AS prompt_tokens, sum(completion_tokens) AS completion_tokens, " +
"sum(cost_cents) AS cost_cents, uniqExact(model) AS models " +
"FROM " + aimUsageTable + " WHERE timestamp >= ?"
}
func aimTopModelsSQL() string {
return "SELECT model, count() AS requests, sum(total_tokens) AS tokens, " +
"sum(cost_cents) AS cost_cents FROM " + aimUsageTable +
" WHERE timestamp >= ? AND model != '' GROUP BY model ORDER BY requests DESC LIMIT " + strconv.Itoa(aimTopN)
}
func aimEvalTracesSQL() string {
lat := "(toUnixTimestamp64Milli(end_time) - toUnixTimestamp64Milli(start_time))"
return "SELECT count() AS traces, uniqExact(run_name) AS runs, uniqExact(dataset) AS datasets, " +
"uniqExact(model) AS models, round(avgIf(" + lat + ", end_time > start_time), 2) AS lat_avg " +
"FROM " + aimEvalTraces + " WHERE ts >= ?"
}
func aimEvalScoresSQL() string {
return "SELECT count() AS scores, round(avg(value), 4) AS avg_value, uniqExact(name) AS score_names " +
"FROM " + aimEvalScores + " WHERE ts >= ?"
}
func aimScoreNamesSQL() string {
return "SELECT name, count() AS n, round(avg(value), 4) AS avg_value, " +
"round(min(value), 4) AS min_value, round(max(value), 4) AS max_value " +
"FROM " + aimEvalScores + " WHERE ts >= ? AND name != '' GROUP BY name ORDER BY n DESC LIMIT " + strconv.Itoa(aimTopN)
}
func aimEvalRunsSQL() string {
return "SELECT run_name, any(dataset) AS dataset, count() AS scores, round(avg(value), 4) AS avg_value, " +
"max(ts) AS last_ts FROM " + aimEvalScores + " WHERE ts >= ? AND run_name != '' " +
"GROUP BY run_name ORDER BY last_ts DESC LIMIT " + strconv.Itoa(aimTopN)
}
func aimScoreSeriesSQL(interval string) string {
return "SELECT toStartOfInterval(ts, INTERVAL " + interval + ") AS ts, " +
"round(avg(value), 4) AS avg_value, count() AS n FROM " + aimEvalScores +
" WHERE ts >= ? GROUP BY ts ORDER BY ts"
}
// ── pure row parsers (unit-tested) ──
func fillAimUsage(u *aimUsage, r map[string]any) {
u.Requests = chInt64(r["requests"])
u.Tokens = chInt64(r["tokens"])
u.PromptTokens = chInt64(r["prompt_tokens"])
u.CompletionTokens = chInt64(r["completion_tokens"])
u.CostCents = chInt64(r["cost_cents"])
u.Models = chInt64(r["models"])
}
func fillAimEvalTraces(e *aimEvals, r map[string]any) {
e.Traces = chInt64(r["traces"])
e.Runs = chInt64(r["runs"])
e.Datasets = chInt64(r["datasets"])
e.Models = chInt64(r["models"])
e.LatencyMsAvg = chFloat64(r["lat_avg"])
}
func fillAimEvalScores(e *aimEvals, r map[string]any) {
e.Scores = chInt64(r["scores"])
e.AvgScore = chFloat64(r["avg_value"])
e.ScoreNames = chInt64(r["score_names"])
}
func aimModelsFromRows(rows []map[string]any) []aimModelStat {
out := make([]aimModelStat, 0, len(rows))
for _, r := range rows {
out = append(out, aimModelStat{
Model: chStr(r["model"]),
Requests: chInt64(r["requests"]),
Tokens: chInt64(r["tokens"]),
CostCents: chInt64(r["cost_cents"]),
})
}
return out
}
func lfModelsFromRows(rows []map[string]any) []aimLfModelStat {
out := make([]aimLfModelStat, 0, len(rows))
for _, r := range rows {
out = append(out, aimLfModelStat{
Model: chStr(r["model"]),
Generations: chInt64(r["gens"]),
CostUsd: chFloat64(r["cost"]),
})
}
return out
}
func scoreNamesFromRows(rows []map[string]any) []aimScoreStat {
out := make([]aimScoreStat, 0, len(rows))
for _, r := range rows {
out = append(out, aimScoreStat{
Name: chStr(r["name"]),
Count: chInt64(r["n"]),
AvgValue: chFloat64(r["avg_value"]),
MinValue: chFloat64(r["min_value"]),
MaxValue: chFloat64(r["max_value"]),
})
}
return out
}
func evalRunsFromRows(rows []map[string]any) []aimRunStat {
out := make([]aimRunStat, 0, len(rows))
for _, r := range rows {
out = append(out, aimRunStat{
RunName: chStr(r["run_name"]),
Dataset: chStr(r["dataset"]),
Scores: chInt64(r["scores"]),
AvgValue: chFloat64(r["avg_value"]),
LastTs: chTime(r["last_ts"]),
})
}
return out
}
func scoreSeriesFromRows(rows []map[string]any) []aimScorePoint {
out := make([]aimScorePoint, 0, len(rows))
for _, r := range rows {
out = append(out, aimScorePoint{
Ts: chTime(r["ts"]),
AvgValue: chFloat64(r["avg_value"]),
Count: chInt64(r["n"]),
})
}
return out
}
-171
View File
@@ -1,171 +0,0 @@
// Copyright 2023-2026 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package admin
import (
"strings"
"testing"
)
// TestAimSQL_ReadsCanonicalTables proves every AI-metrics query reads the ONE
// datastore's canonical table, binds the time bound as a POSITIONAL param (one
// `?`), and never interpolates user input. The bucket interval is the only rendered
// value in the series query and it is a server-side constant.
func TestAimSQL_ReadsCanonicalTables(t *testing.T) {
cases := []struct {
name, sql, table string
wantQMarks int
}{
{"langfuseTotals", aimLangfuseTotalsSQL(), "langfuse.observations", 1},
{"langfuseLatency", aimLangfuseLatencySQL(), "langfuse.observations", 1},
{"langfuseModels", aimLangfuseModelsSQL(), "langfuse.observations", 1},
{"usageTotals", aimUsageTotalsSQL(), "hanzo.cloud_usage", 1},
{"topModels", aimTopModelsSQL(), "hanzo.cloud_usage", 1},
{"evalTraces", aimEvalTracesSQL(), "hanzo.eval_traces", 1},
{"evalScores", aimEvalScoresSQL(), "hanzo.eval_scores", 1},
{"scoreNames", aimScoreNamesSQL(), "hanzo.eval_scores", 1},
{"evalRuns", aimEvalRunsSQL(), "hanzo.eval_scores", 1},
{"scoreSeries", aimScoreSeriesSQL("1 DAY"), "hanzo.eval_scores", 1},
}
for _, c := range cases {
if !strings.Contains(c.sql, "FROM "+c.table) {
t.Errorf("%s must read %s; got %q", c.name, c.table, c.sql)
}
if n := strings.Count(c.sql, "?"); n != c.wantQMarks {
t.Errorf("%s: %d bind params, want %d (time bound only) — no interpolation; got %q", c.name, n, c.wantQMarks, c.sql)
}
}
}
// TestAimLangfuseScopedToGeneration proves the Langfuse lens is scoped to
// generations only (not spans/events), matching the o11y LLM lens.
func TestAimLangfuseScopedToGeneration(t *testing.T) {
for _, sql := range []string{aimLangfuseTotalsSQL(), aimLangfuseLatencySQL(), aimLangfuseModelsSQL()} {
if !strings.Contains(sql, "type = 'GENERATION'") {
t.Errorf("langfuse lens must scope to GENERATION observations; got %q", sql)
}
}
}
// TestAimTop_LimitAndOrder proves the leaderboards bound + order the result.
func TestAimTop_LimitAndOrder(t *testing.T) {
if !strings.Contains(aimTopModelsSQL(), "ORDER BY requests DESC LIMIT 12") {
t.Errorf("topModels must order by requests desc, limit %d", aimTopN)
}
if !strings.Contains(aimScoreNamesSQL(), "GROUP BY name ORDER BY n DESC LIMIT 12") {
t.Errorf("scoreNames must group+order+limit %d", aimTopN)
}
if !strings.Contains(aimEvalRunsSQL(), "ORDER BY last_ts DESC LIMIT 12") {
t.Errorf("evalRuns must order by last_ts desc, limit %d", aimTopN)
}
}
// TestAimScoreSeries_IntervalBound proves the (constant) bucket interval is
// rendered into the score-trend series query and grouped/ordered by the bucket.
func TestAimScoreSeries_IntervalBound(t *testing.T) {
for _, iv := range []string{"1 HOUR", "6 HOUR", "1 DAY"} {
s := aimScoreSeriesSQL(iv)
if !strings.Contains(s, "INTERVAL "+iv) || !strings.Contains(s, "GROUP BY ts ORDER BY ts") {
t.Errorf("score series must bucket by INTERVAL %s; got %q", iv, s)
}
}
}
// TestAimEvalLatencyGuarded proves the latency expressions guard end_time>start_time
// so a zero/default end_time never contributes a garbage (negative) latency.
func TestAimEvalLatencyGuarded(t *testing.T) {
if !strings.Contains(aimEvalTracesSQL(), "end_time > start_time") {
t.Errorf("eval traces latency must guard end_time>start_time; got %q", aimEvalTracesSQL())
}
if !strings.Contains(aimLangfuseLatencySQL(), "end_time > start_time") {
t.Errorf("langfuse latency must guard end_time>start_time; got %q", aimLangfuseLatencySQL())
}
}
// TestFillAimUsage reads a cloud_usage row into the KPI band across the numeric
// variants the driver returns (uint64/int64/float64), honest zeros on an empty row.
func TestFillAimUsage(t *testing.T) {
var empty aimUsage
fillAimUsage(&empty, map[string]any{})
if empty.Requests != 0 || empty.Tokens != 0 || empty.Models != 0 {
t.Fatalf("empty row must yield honest zeros; got %+v", empty)
}
var got aimUsage
fillAimUsage(&got, map[string]any{
"requests": uint64(274), "tokens": uint64(102597), "prompt_tokens": uint64(60000),
"completion_tokens": uint64(42597), "cost_cents": uint64(216), "models": uint64(42),
})
if got.Requests != 274 || got.Tokens != 102597 || got.CostCents != 216 || got.Models != 42 {
t.Fatalf("usage totals mis-parsed: %+v", got)
}
}
// TestFillAimEvals maps both eval halves (traces + scores) into the KPI band,
// including the float latency/score columns (round()/avg() land as float64; a
// Decimal-as-string is parsed).
func TestFillAimEvals(t *testing.T) {
var e aimEvals
fillAimEvalTraces(&e, map[string]any{
"traces": uint64(1280), "runs": uint64(16), "datasets": uint64(4),
"models": uint64(6), "lat_avg": float64(842.5),
})
fillAimEvalScores(&e, map[string]any{
"scores": uint64(1280), "avg_value": "0.8125", "score_names": uint64(3),
})
if e.Traces != 1280 || e.Runs != 16 || e.Datasets != 4 || e.Models != 6 || e.LatencyMsAvg != 842.5 {
t.Fatalf("eval traces mis-parsed: %+v", e)
}
if e.Scores != 1280 || e.ScoreNames != 3 || e.AvgScore != 0.8125 { // string→float64 path
t.Fatalf("eval scores mis-parsed: %+v", e)
}
}
// TestAimParsers map datastore rows into the view-models and preserve order (the
// SQL already ORDER BYs; a parser must not reorder or drop rows), with empty input
// yielding an empty (non-nil) slice rather than a panic.
func TestAimParsers(t *testing.T) {
models := aimModelsFromRows([]map[string]any{
{"model": "glm-5.2", "requests": uint64(154), "tokens": uint64(38966), "cost_cents": uint64(114)},
{"model": "deepseek-v4-flash", "requests": uint64(118), "tokens": uint64(61550), "cost_cents": uint64(101)},
})
if len(models) != 2 || models[0].Model != "glm-5.2" || models[1].Model != "deepseek-v4-flash" || models[0].Requests != 154 {
t.Fatalf("top models mis-parsed/reordered: %+v", models)
}
lf := lfModelsFromRows([]map[string]any{
{"model": "gpt-4o", "gens": uint64(42), "cost": float64(1.25)},
})
if len(lf) != 1 || lf[0].Model != "gpt-4o" || lf[0].Generations != 42 || lf[0].CostUsd != 1.25 {
t.Fatalf("langfuse models mis-parsed: %+v", lf)
}
names := scoreNamesFromRows([]map[string]any{
{"name": "accuracy", "n": uint64(320), "avg_value": float64(0.82), "min_value": float64(0), "max_value": float64(1)},
})
if len(names) != 1 || names[0].Name != "accuracy" || names[0].Count != 320 || names[0].AvgValue != 0.82 || names[0].MaxValue != 1 {
t.Fatalf("score names mis-parsed: %+v", names)
}
runs := evalRunsFromRows([]map[string]any{
{"run_name": "nightly-2026-07", "dataset": "gsm8k", "scores": uint64(200), "avg_value": float64(0.9), "last_ts": nil},
})
if len(runs) != 1 || runs[0].RunName != "nightly-2026-07" || runs[0].Dataset != "gsm8k" || runs[0].Scores != 200 || runs[0].AvgValue != 0.9 {
t.Fatalf("eval runs mis-parsed: %+v", runs)
}
// Empty input → empty (non-nil) slices, never a panic.
if got := scoreSeriesFromRows(nil); got == nil || len(got) != 0 {
t.Errorf("nil rows must yield empty slice, got %v", got)
}
if got := aimModelsFromRows(nil); got == nil || len(got) != 0 {
t.Errorf("nil rows must yield empty slice, got %v", got)
}
}
+22
View File
@@ -33,7 +33,11 @@ import (
)
// insightsEvent is the PostHog wire shape (subset that matters for ingest).
// UUID is the top-level per-event id PostHog SDKs mint for idempotency; the rest
// of the identity/attribution the SDKs carry rides inside Properties (mapped in
// toCapture).
type insightsEvent struct {
UUID string `json:"uuid"`
Event string `json:"event"`
DistinctID string `json:"distinct_id"`
Timestamp string `json:"timestamp"`
@@ -65,6 +69,11 @@ func (e insightsEvent) toCapture() CaptureEvent {
typ = "pageview"
}
return CaptureEvent{
// Idempotency id: PostHog SDKs carry a top-level event `uuid`; some send it
// as an `$insert_id` property instead. Preserve it as the client MessageID so
// a retried batch (insights-go retries with backoff) keeps a STABLE row id
// rather than the server minting a fresh one per attempt.
MessageID: firstNonEmptyStr(strings.TrimSpace(e.UUID), strings.TrimSpace(str("$insert_id"))),
Type: typ,
Event: e.Event,
Timestamp: e.Timestamp,
@@ -73,6 +82,19 @@ func (e insightsEvent) toCapture() CaptureEvent {
URL: str("$current_url"),
Path: str("$pathname"),
Referrer: str("$referrer"),
// UTM attribution: PostHog SDKs put campaign params in BARE `utm_*`
// properties (not $-prefixed — confirmed against the SDK/ingest source).
// hanzo.events has first-class utm_* columns and the native capture path
// maps CaptureEvent.UTM into them (capture.go), so surfacing them here is
// what lets the web/commerce lens attribute traffic to a campaign. They were
// previously dropped on the PostHog-wire front door.
UTM: UTM{
Source: str("utm_source"),
Medium: str("utm_medium"),
Campaign: str("utm_campaign"),
Term: str("utm_term"),
Content: str("utm_content"),
},
Product: str("product"),
Library: str("$lib"),
LibraryVer: str("$lib_version"),
+97
View File
@@ -0,0 +1,97 @@
// Copyright 2023-2026 Hanzo AI Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// See the License for the specific language governing permissions and
// limitations under the License.
package analytics
import (
"testing"
"time"
)
// TestToCapture_PreservesUTMAttribution proves the PostHog-wire adapter carries the
// BARE utm_* campaign params (what PostHog SDKs emit) through to the native
// CaptureEvent — and thence, via normalizeEvent, into the hanzo.events utm_*
// columns the INSERT binds. Regression guard: these were previously dropped, so
// every campaign-attributed pageview lost its source/medium/campaign on the
// /v1/insights/e front door and the web/commerce lens could never attribute it.
func TestToCapture_PreservesUTMAttribution(t *testing.T) {
e := insightsEvent{
Event: "$pageview",
DistinctID: "visitor-1",
Properties: map[string]any{
"utm_source": "newsletter",
"utm_medium": "email",
"utm_campaign": "launch",
"utm_term": "analytics",
"utm_content": "hero-cta",
"$current_url": "https://hanzo.ai/insights",
},
}
cap := e.toCapture()
if cap.UTM.Source != "newsletter" || cap.UTM.Medium != "email" ||
cap.UTM.Campaign != "launch" || cap.UTM.Term != "analytics" || cap.UTM.Content != "hero-cta" {
t.Fatalf("UTM not mapped from PostHog wire: %+v", cap.UTM)
}
// End-to-end through the normalizer into the positional row the INSERT binds.
row, ok := normalizeEvent("acme", time.Now(), cap)
if !ok {
t.Fatal("want ok")
}
if row.utmSource != "newsletter" || row.utmMedium != "email" ||
row.utmCampaign != "launch" || row.utmTerm != "analytics" || row.utmContent != "hero-cta" {
t.Fatalf("UTM lost before the events row: src=%q med=%q camp=%q term=%q content=%q",
row.utmSource, row.utmMedium, row.utmCampaign, row.utmTerm, row.utmContent)
}
}
// TestToCapture_IdempotencyID proves the client event id (PostHog top-level `uuid`,
// or the `$insert_id` property fallback) is preserved as the stable row id, so a
// retried batch does not mint a fresh id per attempt — while an absent id still
// falls back to a server-minted one (existing behavior unchanged).
func TestToCapture_IdempotencyID(t *testing.T) {
// top-level uuid wins
row, _ := normalizeEvent("acme", time.Now(),
insightsEvent{Event: "signup", DistinctID: "u1", UUID: "evt-abc"}.toCapture())
if row.id != "evt-abc" {
t.Fatalf("top-level uuid not preserved as row id, got %q", row.id)
}
// $insert_id property fallback when no top-level uuid
row2, _ := normalizeEvent("acme", time.Now(),
insightsEvent{Event: "signup", DistinctID: "u1", Properties: map[string]any{"$insert_id": "ins-9"}}.toCapture())
if row2.id != "ins-9" {
t.Fatalf("$insert_id fallback not preserved, got %q", row2.id)
}
// absent → server still mints a non-empty id
row3, _ := normalizeEvent("acme", time.Now(),
insightsEvent{Event: "signup", DistinctID: "u1"}.toCapture())
if row3.id == "" {
t.Fatal("server must still mint an id when the client sends none")
}
}
// TestToCapture_MapsCoreFields guards that the pre-existing $-property mappings
// still hold alongside the new UTM/idempotency mappings (no regression).
func TestToCapture_MapsCoreFields(t *testing.T) {
cap := insightsEvent{
Event: "$pageview",
DistinctID: "v1",
Properties: map[string]any{
"$session_id": "s1",
"$current_url": "https://hanzo.ai/x",
"$pathname": "/x",
"$referrer": "https://news.ycombinator.com/",
"$lib": "insights-go",
"$lib_version": "1.2.3",
"product": "console",
},
}.toCapture()
if cap.Type != "pageview" || cap.SessionID != "s1" || cap.URL != "https://hanzo.ai/x" ||
cap.Path != "/x" || cap.Referrer != "https://news.ycombinator.com/" ||
cap.Library != "insights-go" || cap.LibraryVer != "1.2.3" || cap.Product != "console" {
t.Fatalf("core PostHog-wire mapping regressed: %+v", cap)
}
}
+15 -8
View File
@@ -46,33 +46,40 @@ func newTestClient(t *testing.T) *Client {
}
func TestEnvFallbackAndDefault(t *testing.T) {
// No engine/store configured -> env fallback, then literal default.
// No engine/store configured. Runtime flags carry no Env, so they resolve to
// their literal default and an env var must NOT move them — that is what makes
// /v1/flags the single source of truth, flippable live without a redeploy.
// Boot-time ReadOnly rows keep Env, because env IS their boot mechanism.
prev := mounted
mounted = &Client{} // not configured
t.Cleanup(func() { mounted = prev })
// waitlist_open default is "true" (no env set).
// waitlist_open default is "true".
t.Setenv("WAITLIST_OPEN", "")
if !Bool("waitlist_open") {
t.Fatalf("waitlist_open default should be true")
}
// env override beats the literal default.
// A runtime flag ignores env: the default still wins.
t.Setenv("WAITLIST_OPEN", "false")
if Bool("waitlist_open") {
t.Fatalf("waitlist_open env=false should win")
if !Bool("waitlist_open") {
t.Fatalf("waitlist_open is a runtime flag: env must not override its default")
}
// int default.
if Int("waitlist_access_capacity") != 0 {
t.Fatalf("capacity default should be 0")
}
t.Setenv("WAITLIST_ACCESS_CAPACITY", "250")
if Int("waitlist_access_capacity") != 250 {
t.Fatalf("capacity env should be 250, got %d", Int("waitlist_access_capacity"))
if Int("waitlist_access_capacity") != 0 {
t.Fatalf("waitlist_access_capacity is a runtime flag: env must not override its default")
}
// network id read-only default.
// A boot-time ReadOnly row still reads env, and falls back to its default.
if Int("network_id_localnet") != 1337 {
t.Fatalf("localnet id default should be 1337")
}
t.Setenv("LUX_NETWORK_ID_LOCALNET", "1338")
if Int("network_id_localnet") != 1338 {
t.Fatalf("network_id_localnet is boot-time: env must win, got %d", Int("network_id_localnet"))
}
// unknown key is safe.
if Bool("nope") || Int("nope") != 0 || String("nope") != "" {
t.Fatalf("unknown key must be zero-valued")
@@ -16,15 +16,15 @@ import (
"github.com/hanzoai/commerce/datastore"
"github.com/hanzoai/commerce/db"
commercemid "github.com/hanzoai/commerce/middleware"
"github.com/hanzoai/commerce/middleware/svcorg"
"github.com/hanzoai/commerce/pkg/org"
"github.com/hanzoai/commerce/util/bit"
"github.com/hanzoai/commerce/util/permission"
)
// seedInMemDatastore installs a fresh in-memory default datastore so the
// service-token branch's svcorg.Resolve (GetOrCreate on the org table) succeeds
// service-token branch's org.Resolve (GetOrCreate on the org table) succeeds
// and the request reaches the handler — the funded path — rather than 503ing on a
// resolve failure. Mirrors svcorg/resolver_test.go's setup.
// resolve failure. Mirrors pkg/org's lookup_test.go setup.
func seedInMemDatastore(t *testing.T) {
t.Helper()
mgr, err := db.NewManager(&db.Config{DataDir: t.TempDir(), EnableVectorSearch: false, IsDev: true})
@@ -65,8 +65,8 @@ func TestInProcMeteringDispatch_ServiceTokenAuthPath(t *testing.T) {
const svc = "svc-token-metering-xyz789"
t.Setenv("COMMERCE_SERVICE_TOKEN", svc)
seedInMemDatastore(t)
svcorg.Invalidate("funded")
svcorg.Invalidate("broke")
org.Invalidate("funded")
org.Invalidate("broke")
// finance NOT co-resident → the metering balance read goes over commerceinproc,
// through the real commerce middleware chain, exactly like the topology that 500'd.
+2 -2
View File
@@ -14,7 +14,7 @@ require (
github.com/google/go-containerregistry v0.21.7
github.com/google/go-github/v52 v52.0.0
github.com/hanzoai/account v0.2.0
github.com/hanzoai/commerce v1.49.2
github.com/hanzoai/commerce v1.49.3
github.com/hanzoai/decimal v0.1.1
github.com/hanzoai/go-openai v1.41.0
github.com/hanzoai/goa v1.0.0
@@ -841,7 +841,7 @@ require (
github.com/hanzo-ds/go v1.0.1
github.com/hanzo-ds/native v0.72.0 // indirect
github.com/hanzoai/agent v0.1.3
github.com/hanzoai/ai v1.826.4
github.com/hanzoai/ai v1.826.6
github.com/hanzoai/authz v1.10.7
github.com/hanzoai/base v1.5.7
github.com/hanzoai/licensing v0.1.5
+4 -2
View File
@@ -1212,8 +1212,8 @@ github.com/hanzoai/account v0.2.0 h1:WxIut3YMz8JNdHerlRIqP1a+k3P79BIKM1mjfRFDG0Y
github.com/hanzoai/account v0.2.0/go.mod h1:8OzIGRphAhlabOI74O4GoL3RM0y8mbUV0pQUKgXLjkw=
github.com/hanzoai/agent v0.1.3 h1:zzV4t8kN/m/wTLrqzEy0fxxONSZbx3XSVH7TIR9gZNU=
github.com/hanzoai/agent v0.1.3/go.mod h1:Z3hCBdSeN/nGV4o+3F4psQ2bbFk17+tMP5l+G2ssNNA=
github.com/hanzoai/ai v1.826.4 h1:k4kooLThqCHSfzqMy1y1UdZ5YtgwS3oPElNg8VfYkKc=
github.com/hanzoai/ai v1.826.4/go.mod h1:LkSrjXJjFS9weIQmhXl53x/Dmt90KgkulEkx8O1Gd5U=
github.com/hanzoai/ai v1.826.6 h1:qcsQN2buAR9CdEPgQ6u6HIbkQdKCS54dOjQYfS/a2wM=
github.com/hanzoai/ai v1.826.6/go.mod h1:LkSrjXJjFS9weIQmhXl53x/Dmt90KgkulEkx8O1Gd5U=
github.com/hanzoai/authz v1.10.7 h1:JrHljH29mbmVi8u6/6EVG7R0NiFhIYYm2WUBBuBmFq0=
github.com/hanzoai/authz v1.10.7/go.mod h1:9wf6n6BvrvxRULUtL3yc+vmprwwvAGZLGJoMHhiQSK4=
github.com/hanzoai/authzstore v0.1.1 h1:4GsvB+bKs+gFtfKDMoYq/C7KxJAHrXLwtwdSutTakbo=
@@ -1228,6 +1228,8 @@ github.com/hanzoai/captable v1.0.0 h1:utXPsOaPL+QV0rJaoNDFHvWFv7etf2kfm1bsFh6JRD
github.com/hanzoai/captable v1.0.0/go.mod h1:czdMnzEvb8FWNKCzmh+z8PmwYeQp4MGyTPgo+Q5O+mk=
github.com/hanzoai/commerce v1.49.2 h1:zIuXaCz6CvjiF0/z/9ub+y0dkKTaonewNpmRHroVimg=
github.com/hanzoai/commerce v1.49.2/go.mod h1:+Bp6hVxJQmv71D/ZPzs5qiEVHyDNUchNsqakd+778ps=
github.com/hanzoai/commerce v1.49.3 h1:nrHFEdndavp5DX3cnXPSqYDamC9MAtDp1gka+wGs4X4=
github.com/hanzoai/commerce v1.49.3/go.mod h1:+Bp6hVxJQmv71D/ZPzs5qiEVHyDNUchNsqakd+778ps=
github.com/hanzoai/csqlite v0.1.0 h1:suwC3dh0INlfP/U0Es6cDf6JNQ+2+GVLLATPWCUux6k=
github.com/hanzoai/csqlite v0.1.0/go.mod h1:H31a/O6VXuklR9UBkgY++bmAK5uzVfXPqU0F6P9Wsos=
github.com/hanzoai/dashscope-go-sdk v0.0.2 h1:L/FlStjXeehrNSBqU8y/ecSG/MnaAJqfyfjmLYmY344=
+2 -2
View File
@@ -21,7 +21,7 @@ images:
test:
- name: native-flags
# The flags evaluator staticlib clients/featureflags links under CGO=1.
# The flags evaluator staticlib clients/flags links under CGO=1.
run: |
set -e
if ! command -v cargo >/dev/null 2>&1; then
@@ -56,7 +56,7 @@ test:
./clients/catalogsync/ \
./clients/bots/ \
./clients/admin/finance/ \
./clients/featureflags/ \
./clients/flags/ \
./clients/ml/
kms: