research: redact api key in fmt, validate since as object id, harden transport

- Research implements Stringer + GoStringer, so %v/%+v/%#v/%s render the key
  as <redacted>; a struct dump no longer leaks hk-* in plaintext.
- commitNarrative gates `since` to a git object id (hex, <=64) before it
  reaches `git log <since>..HEAD`, closing arg-injection from a server value
  (parity with the C++ port).
- http.Client refuses redirects (CheckRedirect -> ErrUseLastResponse) and the
  response read is capped at 8 MiB, so a hostile server cannot bounce the
  Bearer key to another host or OOM the client.
- Docs state the accurate contract: records are semantically identical across
  languages (the server keys on project + id = kind:subject:task), not
  byte-identical.

Tests: TestKeyNeverLeaksViaFmt, TestSinceMustBeAGitObjectID.
This commit is contained in:
zooqueen
2026-07-22 23:21:01 -07:00
parent 50604f9c32
commit 34e62c05d1
6 changed files with 111 additions and 22 deletions
+13 -8
View File
@@ -5,8 +5,10 @@ surface (HIP-0512). Stdlib-only (`net/http` + `encoding/json`), zero external mo
any service — cloud, tools, marketing/growth, ops — imports it without dragging a
dependency graph.
Mirrors the Python producer (`hanzo/python-sdk/pkg/hanzo-research`) verb-for-verb and
byte-for-byte: every language emits structurally identical records into the one store.
Mirrors the Python producer (`hanzo/python-sdk/pkg/hanzo-research`) verb-for-verb. Records
are semantically identical across languages: the server keys each experiment on `(project,
id = kind:subject:task)`, so every language upserts the same row regardless of JSON
serialization (Go emits compact bytes; Python/C++ emit spaced — immaterial to the store).
## Surface
@@ -45,13 +47,16 @@ the host, and the calling code site (`runtime.Callers`). Serialized into `meta`
exact key order shared with Python/TS: `doc, commits, note, host, hypothesis, predict,
verdict, because, log`.
## Byte-compatibility
## Cross-language identity — semantic, not byte
`encode` marshals compact with `SetEscapeHTML(false)` so `& < >` travel literally, matching
Python's `json.dumps`. `research_test.go` proves Go's canonical meta byte-equals real
CPython `json.dumps(meta, sort_keys=True)`. Whitespace and numeric shortest-form (`0` vs
`0.0`) differ on the wire but are JSON-insignificant; the server parses and is idempotent
by stable id, so records interleave cleanly.
Records are keyed by the server on `(project, id = kind:subject:task)` (experiments) and
`(project, benchmark, item, model)` (attempts), so every language upserts the same row
regardless of serialization — that is the identity guarantee, not byte-matching. `encode`
marshals compact with `SetEscapeHTML(false)` so `& < >` travel literally rather than as
escaped sequences; `research_test.go` proves Go's canonical meta is the SAME JSON document
as CPython `json.dumps(meta, sort_keys=True)`. Whitespace and numeric shortest-form (`0` vs
`0.0`) differ on the wire but are JSON-insignificant — the server parses and dedupes by
stable id, so records from every language interleave into one row cleanly.
## Auth
+4 -2
View File
@@ -2,8 +2,10 @@
// Package research is the ONE way a Go service records and queries R&D evidence on the
// unified /v1/research surface (HIP-0512). It mirrors the Python and TypeScript producers
// verb-for-verb and byte-for-byte, so every language emits structurally identical records
// into the one store.
// verb-for-verb. Records are semantically identical across languages, not byte-identical:
// the server keys each experiment on (project, id = kind:subject:task) and each attempt on
// (project, benchmark, item, model), so every language upserts the same row regardless of
// JSON serialization.
//
// A tiny, zero-config surface makes hand-rolling the obviously worse choice:
//
+4 -3
View File
@@ -42,9 +42,10 @@ type Result struct {
Status string
}
// meta carries the scientific frame plus the self-documenting narrative. Its field order
// and json names are the cross-language contract — identical to the Python and TypeScript
// producers — so every language emits structurally identical records into the one store.
// meta carries the scientific frame plus the self-documenting narrative. Its json names are
// the cross-language contract (field order is shared for readability, but JSON objects are
// unordered, so order is not load-bearing). Records are keyed and deduped semantically by
// the server, so every language's meta lands in the same row regardless of serialization.
type meta struct {
Doc string `json:"doc"`
Commits []string `json:"commits"`
+23 -4
View File
@@ -58,11 +58,13 @@ func gitState(repo string) vcs {
}
// commitNarrative is the commit-message story: subjects SINCE the last recorded run's sha
// when known (<since>..HEAD), else the last window commits. Always non-nil (an empty slice
// serializes to [], byte-identically to the other producers), never nil.
// when known (<since>..HEAD), else the last window commits. Always non-nil so it serializes
// to an empty array [], not null — the shape every producer sends and the server expects.
func commitNarrative(repo, since string, window int) []string {
var raw string
if since != "" {
// since can arrive from a server response; only a git object id (hex) reaches the git
// arg, so it can never be read as a flag or a path — anything else falls back to window.
if isObjectID(since) {
raw = git(repo, "log", since+"..HEAD", "--format=%s")
} else {
raw = git(repo, "log", "-"+strconv.Itoa(window), "--format=%s")
@@ -76,10 +78,27 @@ func commitNarrative(repo, since string, window int) []string {
return out
}
// isObjectID reports whether s is a git object id — hex only, 1..64 chars (sha1=40,
// sha256=64). The since value can arrive from a server response and reaches
// `git log <since>..HEAD`; gating it to hex means it can never be read as a flag (a leading
// '-') or a path ('/'), which closes the git arg-injection. Parity with the C++ port.
func isObjectID(s string) bool {
if s == "" || len(s) > 64 {
return false
}
for i := 0; i < len(s); i++ {
c := s[i]
if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')) {
return false
}
}
return true
}
// libVersions is the Go-native {module: version} snapshot from the build info — the
// longitudinal "which lib version regressed X" record. It always names the Go toolchain
// and the main module; extra module-path prefixes pull in named dependencies. Always
// non-nil so it serializes to {} byte-identically to the other producers.
// non-nil so it serializes to an empty object {}, not null — the shape every producer sends.
func libVersions(extra []string) map[string]string {
out := map[string]string{"go": runtime.Version()}
bi, ok := debug.ReadBuildInfo()
+32 -5
View File
@@ -18,7 +18,8 @@ const (
defaultBase = "https://api.hanzo.ai"
defaultProject = "default"
defaultTimeout = 120 * time.Second
narrativeDepth = 10 // commits to record when the last run's sha is unknown
narrativeDepth = 10 // commits to record when the last run's sha is unknown
maxResponse = 8 << 20 // response-body cap (8 MiB): a hostile server cannot OOM the client
)
// Config configures a client. The zero value is valid: Base defaults to api.hanzo.ai (or
@@ -63,10 +64,29 @@ func New(cfg Config) *Research {
project: firstNonEmpty(cfg.Project, os.Getenv("RESEARCH_PROJECT"), defaultProject),
repo: repo,
libs: cfg.Libs,
http: &http.Client{Timeout: timeout},
http: &http.Client{
Timeout: timeout,
// Never follow redirects: a /v1/research endpoint has no legitimate redirect, and
// refusing them denies a hostile 30x any chance to bounce the request (and its
// Bearer key) to another host or an SSRF target. Return the 30x as-is.
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
},
}
}
// String renders the client for logs and debug dumps WITHOUT its key. The api key is a
// secret and must never reach a log line, so implementing Stringer + GoStringer makes every
// fmt verb (%v, %+v, %#v, %s) redact it — closing the plaintext-key leak a struct dump
// would otherwise open. Mirrors the Rust SDK's Debug redaction. Value receiver, so a
// *Research and a Research value are both covered.
func (c Research) String() string {
return fmt.Sprintf("research.Research{base:%q, project:%q, repo:%q, key:<redacted>}",
c.base, c.project, c.repo)
}
// GoString redacts the key under the %#v (Go-syntax) verb, which does not consult String.
func (c Research) GoString() string { return c.String() }
// Experiment gets or creates the handle for (kind, subject, task) and posts it in-flight
// so the ops board sees it immediately. kind is an OPEN string — benchmark, kernel-perf,
// training, ablation, policy-eval AND marketing-experiment, ad-test, pricing-test,
@@ -221,7 +241,12 @@ func (c *Research) do(method, path string, body, out any) error {
return err
}
defer resp.Body.Close()
data, _ := io.ReadAll(resp.Body)
// Bound the read: a hostile or broken server cannot stream an unbounded body to OOM the
// client. Read one byte past the cap so an over-limit body is detected, not silently cut.
data, _ := io.ReadAll(io.LimitReader(resp.Body, maxResponse+1))
if len(data) > maxResponse {
return fmt.Errorf("research: %s %s: response exceeds %d-byte cap", method, path, maxResponse)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("research: %s %s: %s: %s", method, path, resp.Status, serverError(data))
}
@@ -310,8 +335,10 @@ type GrantRequest struct {
Publishable *bool `json:"publishable,omitempty"`
}
// encode marshals v as compact JSON with HTML escaping OFF, so <, >, and & travel
// literally — byte-for-byte as the other language producers emit them.
// encode marshals v as compact JSON with HTML escaping OFF, so <, >, and & in notes and
// commit messages travel literally rather than as escaped sequences. The wire bytes are
// compact (Python/C++ emit spaced json.dumps) — immaterial to the store: the server keys
// each record on (project, id) and parses the JSON, so every language upserts the same row.
func encode(v any) ([]byte, error) {
var b bytes.Buffer
enc := json.NewEncoder(&b)
+35
View File
@@ -4,6 +4,7 @@ package research
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
@@ -206,6 +207,40 @@ func TestVerdictValidation(t *testing.T) {
}
}
// TestKeyNeverLeaksViaFmt proves the client redacts its api key under every fmt verb. A
// structured logger, a panic, or a %+v debug dump of the client must never expose the key —
// a plaintext-secret leak. Both the *Research pointer and the Research value are checked.
func TestKeyNeverLeaksViaFmt(t *testing.T) {
const key = "hk-SUPERSECRET-abc123"
c := New(Config{Base: "https://api.hanzo.ai", Key: key, Project: "p"})
for _, verb := range []string{"%v", "%+v", "%#v", "%s"} {
for _, dump := range []string{fmt.Sprintf(verb, c), fmt.Sprintf(verb, *c)} {
if strings.Contains(dump, "SUPERSECRET") || strings.Contains(dump, key) {
t.Fatalf("verb %s leaked the key: %s", verb, dump)
}
if !strings.Contains(dump, "<redacted>") {
t.Fatalf("verb %s did not redact the key: %s", verb, dump)
}
}
}
}
// TestSinceMustBeAGitObjectID proves the since value that reaches `git log <since>..HEAD` is
// gated to a git object id (hex), so a server-returned value can never inject a git flag or
// path (the arg-injection Red flagged). Parity with the C++ hex validation.
func TestSinceMustBeAGitObjectID(t *testing.T) {
for _, s := range []string{"abc123", strings.Repeat("a", 40), strings.Repeat("F", 64)} {
if !isObjectID(s) {
t.Fatalf("isObjectID(%q) = false, want true (a valid object id)", s)
}
}
for _, s := range []string{"", "--output=/tmp/pwned", "HEAD~5", strings.Repeat("a", 65), "../etc", "a b", "-x"} {
if isObjectID(s) {
t.Fatalf("isObjectID(%q) = true, want false (must fall back to the window)", s)
}
}
}
// ── test helpers ────────────────────────────────────────────────────────────────────
// mock is a stand-in /v1/research server that records the ingest bodies it receives.