hanzoai/python-sdk
2
0
Fork 0
mirror of https://github.com/hanzoai/python-sdk.git synced 2026-10-23 20:23:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

hanzo-research 0.2.0: every research and arena route, the crate's records, provenance

Browse Source 
The Python client now reaches what the Rust crate does and the server
serves. Research covers every /v1/research route: record() files
benchmarks, studies, runs and papers in one batch a kind; runs(),
get_run() and compare() read them; grant(), revise() and retract() move
consent and versions; the experiment plane and artifacts as before.
Arena covers /v1/benchmark: catalog, leaderboard, claims (written with
protocol and source), history, compare, presets, and runs, which queue a
paid execution and are refused without confirm=True.

wire holds the crate's Run, Benchmark, Measure, Study, Paper and
Comparison field for field; canonical writes RFC 8785 bytes equal to the
crate's (tests/vectors/rust.json, written by the crate); lineage and wal
are the crate's records, and a WAL the crate wrote verifies here.
capture() reads the harness commit (the diff's SHA-256 when dirty), the
weights' SHA-256, host, CPU, GPU, device, dtype and packages, and
apply() folds them into a run.

runs(org=) sends no credential: the server answers ?org= only to a
caller with no org. An org the token does not belong to is refused
before the request, since the server would act in the home org. The
command is hanzo-research; the native CLI owns `hanzo`.
This commit is contained in:
zeekay
2026-10-15 12:11:45 -07:00
parent 81c0caf426
commit 42124b1f45
35 changed files with 4905 additions and 414 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+45
View File
@@ -416,6 +416,51 @@ runs them against `conformance/fake.py`, and `uv run python tests/live_conforman
against api.hanzo.ai and prints pass or fail per rule. Release: tag `hanzo-kai-v<version>`, and
`https://pypi.org/pypi/hanzo-kai/json` is the proof, not a green run.
## hanzo-research — measurements into `/v1/research` and `/v1/benchmark`
`pkg/hanzo-research` (`import hanzo_research`, command `hanzo-research`) is the Python peer
of the Rust crate `hanzo-research` (hanzoai/ml): `Research` covers every route
hanzo-inc/cloud `apps/research` serves (one write, `record(benchmarks=, studies=, runs=,
papers=)`; reads `benchmarks`, `runs`, `studies`, `papers`, `get_run`, `compare`; `grant`,
`revise`, `retract`), `Arena` every route of `apps/benchmark`,
`provenance.capture()` reads harness commit (and the diff's SHA-256 when dirty), model,
weights SHA-256 (a file, a directory by its SHA256SUMS, or a value), host, CPU, GPU,
device, dtype, Python and packages, and `Provenance.apply(run)` folds them into a `Run`.
Standard library only, Python ≥ 3.10. Tests: `uv run --no-project --with pytest pytest
pkg/hanzo-research` over an in-process fake of both apps (`tests/fake.py`); `-m integration`
reads api.hanzo.ai with no credential. The README's worked example is executed by the
suite, and the docs page (`docs/content/docs/research.mdx`) must show the same code.
What the server does that a client has to know (read in `research.go`, `record.go`,
`claims.go`, and on the wire):
- **Writes are batches**: `{"benchmarks": [...]}`, `{"runs": [...]}`, `{"studies": ...}`,
`{"papers": ...}`, at most 20,000; zero is 400 `nothing to record`. Every write lands
private; only `POST /v1/research/grants` moves visibility and consents.
- **`GET /v1/research/runs?org=X` is a public read only for a caller with NO org.** A
credentialled caller gets its own org's runs whatever `org` says, so `runs(org=...)`
sends no credential.
- **The acting org is `X-Org-Id`, honoured only when the token's signed `orgs` names it**
(or the token is a SuperAdmin's); otherwise the server silently acts in the home org.
The client refuses such an org before the request. The project is the token's `project`
claim (`X-Project-Id` is minted from it), so a client asking another project is refused.
- **Typed ops refuse in RFC 7807** (`{"detail", "status", "title", "type"}`); the blob
route and the gateway answer `{"error"}`. Read `detail` first.
- The leaderboard's claims are org `admin`'s public ones only; `claims` shows every org's
public ones and the caller's private ones; `history`/`compare` read the arena's attempts
plane, not research runs; `POST /v1/benchmark/runs` queues a paid run (`confirm=True`).
The Rust crate at hanzoai/ml 7c967bf79 differs from the server in four places, none
patched here: `post_run`/`post_benchmark` send a bare record where the server reads a
batch (400), `post_batch` sends benchmarks inside the runs body (ignored; Python's
`record` posts them first), `Comparison`
reads `differences` where the server writes `config`, and its 8 MiB answer cap refuses the
hanzo org's run listing at about 1,800 runs (1.5 MB at 330 today). Its canonical numbers
are Rust `Display`, not RFC 8785, past 1e21 and below 1e-6 and for `-0.0`
(`tests/vectors/rust.json` pins both).
Release: version by hand in `pkg/hanzo-research/pyproject.toml` (the package reads it back
from its metadata), tag `hanzo-research-v<version>`, pushed by a person.
## Plan usage — `hanzoai.usage`
`read_usage(headers)` → `Usage(usage, usage_class, paid_by, fallback, served, reason)` from
docs/content/docs/meta.json
+1
View File
@@ -11,6 +11,7 @@
"embeddings",
"models",
"files",
"research",
"---Agent SDK---",
"...agents",
"---MCP---",
docs/content/docs/research.mdx
+163
View File
@@ -0,0 +1,163 @@
---
title: Research
description: File a measurement made in Python to /v1/research and /v1/benchmark with its provenance
---
# Research
`pip install hanzo-research` ([pkg/hanzo-research](https://github.com/hanzoai/python-sdk/tree/main/pkg/hanzo-research)).
File a measurement made in Python to Hanzo's research plane (`/v1/research`) and its
benchmark arena (`/v1/benchmark`), with what it takes to believe or repeat it: the
harness commit, the model and its weights' SHA-256, the host, CPU and GPU, the device and
dtype, the Python and package versions, the date. The Python peer of the Rust crate
`hanzo-research` (hanzoai/ml): the same wire records, the same RFC 8785 digests, the same
lineage and WAL records. Standard library only.
```bash
pip install hanzo-research # or: uv run --with hanzo-research python measure.py
```
The credential is `HANZO_API_KEY`, else what `hanzo auth token` prints (a CLI token is
fetched again when it expires). Records land in the token's org (its `owner`) and project
(its `project` claim, else `default`). `Research(org="lux")` acts in another org the
token belongs to (`X-Org-Id`); an org it does not belong to is refused before any request,
since the server would file under the home org instead.
## Worked example
Measure something, file it with its provenance, make it public, read it back.
```python
import hashlib
import platform
import time
from hanzo_research import Benchmark, Research, Run, Split, capture, now
r = Research()
bench = Benchmark(
id="demo.sha256",
title="SHA-256 over 16 MiB",
splits=[Split("16mib", 30, "30 hashes of 16 MiB of zeros, one process, after one discarded")],
metric="p50_ms",
metrics=["p50_ms", "p95_ms", "mb_s"],
)
p = capture(device="cpu") # harness commit, host, CPU, GPU, Python, packages, date
start = now()
data = bytes(16 << 20)
hashlib.sha256(data).digest()
ms = []
for _ in range(30):
t = time.perf_counter()
hashlib.sha256(data).digest()
ms.append(1000 * (time.perf_counter() - t))
ms.sort()
run = Run(
id=f"demo.sha256-16mib-cpu-{p.host}",
benchmark=bench.id,
split="16mib",
system="hashlib",
version=platform.python_version(),
questions=30,
answered=30,
when=start,
ended=now(),
)
run.put("all", "p50_ms", ms[14], n=30, of=30)
run.put("all", "p95_ms", ms[28], n=30, of=30)
run.put("all", "mb_s", 16 / (ms[14] / 1000))
p.apply(run) # fills commit and by, appends the provenance sentence and its digest to notes
r.record(benchmarks=[bench], runs=[run]) # the definition, then the run: both private
r.grant(run.id, visibility="public", publishable=True)
back = r.get_run(run.id, org=r.org) # the public read: no credential
print(back.completion, back.measure("all", "p50_ms").value, back.commit)
```
`completion` is the server's: `complete` only when `answered == questions` and `ended` is
set; `partial`, `inconsistent` or `unknown` otherwise. A model run names its weights:
`capture(model="kai-1.3", weights="/models/kai-1.3", device="cuda", dtype="bf16")` hashes a
file, or a directory by its `SHA256SUMS` (else every file under it), and `apply` sets
`system` to the model and `version` to `<model>@<sha256>`. A dirty harness tree is recorded
with the SHA-256 of `git diff HEAD` and the untracked files, so the number stays pinned to
the code that made it.
## The research plane: `Research`
| call | route |
|---|---|
| `record(benchmarks=, studies=, runs=, papers=)` | `POST /v1/research/{benchmarks,studies,runs,papers}`, one batch a non-empty kind, in that order: `{"runs": [...]}` |
| `benchmarks(id=, project=)`, `get_benchmark(id)` | `GET /v1/research/benchmarks` |
| `runs(id=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
| `runs(org=...)`, `get_run(id, org=...)` | the same route with `?org=` and no credential: that org's public runs |
| `studies(...)`, `papers(..., cites=)` | `GET /v1/research/{studies,papers}` |
| `compare(a, b)` | `GET /v1/research/compare`: the reasons the two runs may not be compared, the config they differ in, the deltas |
| `grant(id, visibility=, publishable=, trainable=)`, `grant(sha256=, visibility=)` | `POST /v1/research/grants`: on every version of the id; a write is always private until this |
| `revise(record)` | a new version marked `corrected` |
| `retract(id, why)` | a new version marked `retracted`: the id leaves every listing, earlier versions stay |
| `ingest`, `query`, `totals`, `projects`, `experiment(...)` | the experiment plane: `/v1/research/{experiments,totals,projects}` |
| `upload_artifact`, `upload_json_artifact`, `artifacts`, `get_artifact` | `/v1/research/artifacts[/<sha256>]`: `snapshot` or `report` bytes, addressed by the server's hash |
| `url(run_id, org=)` | the public read address of one run: the `source` a claim cites |
Every write is a batch of at most 20,000 and is idempotent by content: the same record
again records nothing. A mapping is sent as given; a `Run` is sent with every field. A
refusal raises `Denied` (401, 403), `Missing` (404), `Invalid` (400, 409, 422), `Limited`
(413, 429) or `Unavailable` (502 to 504, no answer), each carrying the server's own
sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves:
`json` refuses it before the request.
## The arena: `Arena` (or `Research().arena`)
| call | route |
|---|---|
| `catalog()` | `GET /v1/benchmark/catalog` |
| `leaderboard(benchmark)` | `GET /v1/benchmark/leaderboard`: each model's measured score (its latest run) beside a published claim. Only claims a SuperAdmin wrote under org `admin`, and public, reach it. |
| `claims(benchmark=, model=, provider=, source=, protocol=, org=)` | `GET /v1/benchmark/claims`: every org's public claims, labelled, and your org's private ones |
| `post_claims(rows, visibility=)` | `POST /v1/benchmark/claims`: under your acting org; each row names its `protocol` and its `source` (for a measured score, `Research.url(run_id)`), or it is refused here |
| `history(benchmark, model=)`, `compare(a, b, benchmark=)` | `GET /v1/benchmark/{history,compare}`: read the arena's per-item attempts plane, not the research runs |
| `presets()`, `post_preset(preset)` | `GET/POST /v1/benchmark/presets`: validated and echoed as `enso-<name>`, not stored |
| `post_run(benchmarks, model=, endpoint=, confirm=True)` | `POST /v1/benchmark/runs`: queues a paid execution, so it is refused without `confirm=True` |
## The command
`hanzo-research` (or `python -m hanzo_research`); every answer is JSON on stdout.
```bash
hanzo-research post runs.jsonl --grant public --publishable # a run per line
hanzo-research post plan.json # {"benchmarks": [...], "runs": [...]}
hanzo-research post plan.json --dry # print the bodies, file nothing
hanzo-research list runs --benchmark demo.sha256
hanzo-research list runs --org hanzo # hanzo's public runs, no credential
hanzo-research get demo.sha256-16mib-cpu-dgx --org hanzo
hanzo-research grant RUN_ID ... --visibility public --publishable
hanzo-research retract RUN_ID --why "read on the wrong split"
hanzo-research compare RUN_A RUN_B
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
hanzo-research arena leaderboard --benchmark gpqa_diamond
```
It is not `hanzo research`: the `hanzo` command is the native CLI, and no Python package
in this workspace may install one.
## Records beside the API
- `canonical.canonicalize(value)`, `canonical.digest(value)`: RFC 8785 bytes and their
SHA-256, equal to the crate's `rfc8785_canonicalize`/`rfc8785_digest` on every string,
integer within 2^53 and float from 1e-6 to 1e21. Past that the crate writes Rust's
`Display` (`1e21` as 22 digits, `-0.0` as `-0`), which is not RFC 8785; this package
writes the RFC's `1e+21` and `0`. `tests/vectors/rust.json` holds the crate's own output.
- `lineage`: `DataPurpose`, `LineagePolicy` (a child never less restrictive than a parent),
`ContaminationError`, `verify_promotion_eligibility`, and the crate's evidence records
(`EnvironmentV1`, `GateSpec`, `Claim`, `TeacherInteraction`, `PairEvidence`,
`DeterminismReport`, `ArtifactRef`, `ResearchEvent`), each with the crate's digest.
- `wal.ResearchWal`: an fsynced, hash-chained event log. A log the crate wrote verifies here
(`wal.verify`), and the reverse.
## Tests
```bash
uv run --no-project --with pytest pytest pkg/hanzo-research # no network
uv run --no-project --with pytest pytest pkg/hanzo-research -m integration # api.hanzo.ai, no credential
```
pkg/hanzo-research/README.md
+138 -46
View File
@@ -1,64 +1,156 @@
# hanzo-research
Record and query Hanzo R&D evidence in one import. The ONE way every producer
(enso-bench benchmarks, hanzo-engine/hanzo-ml kernel-perf, any research project) files
runs into the unified `/v1/research` plane (HIP-0512) — versioned, append-only,
idempotent by content, private by default.
File a measurement made in Python to Hanzo's research plane (`/v1/research`) and its
benchmark arena (`/v1/benchmark`), with what it takes to believe or repeat it: the
harness commit, the model and its weights' SHA-256, the host, CPU and GPU, the device and
dtype, the Python and package versions, the date. The Python peer of the Rust crate
`hanzo-research` (hanzoai/ml): the same wire records, the same RFC 8785 digests, the same
lineage and WAL records. Standard library only.
```python
import hanzo_research as research
# get/create the experiment; provenance is auto-captured (see below)
exp = research.experiment("benchmark", "grok-4.5", "gpqa_diamond")
exp.record("q1", "grok-4.5", {"answer": "A", "correct": True}) # an attempt, idempotent
exp.snapshot(png_bytes) # a board-snapshot artifact
exp.report(markdown_text) # a generated-report artifact
exp.finish(value=94.3) # seal the run
rows = research.query(project="enso-bench", kind="benchmark") # read canonical
```bash
pip install hanzo-research # or: uv run --with hanzo-research python measure.py
```
## Benchmarks, runs, studies, papers
The credential is `HANZO_API_KEY`, else what `hanzo auth token` prints (a CLI token is
fetched again when it expires). Records land in the token's org (its `owner`) and project
(its `project` claim, else `default`). `Research(org="lux")` acts in another org the
token belongs to (`X-Org-Id`); an org it does not belong to is refused before any request,
since the server would file under the home org instead.
The record hanzo.ai/benchmarks draws from. A benchmark defines what is measured; a run
is one execution with its measures; a study is the question runs answer; a paper cites
runs. Each write is a batch, versioned and idempotent by content.
## Worked example
Measure something, file it with its provenance, make it public, read it back.
```python
research.record(
benchmarks=[{"id": "enso.route", "title": "...", "metric": "route_p50_us"}],
runs=[{"id": "enso.route-dgx", "benchmark": "enso.route", "system": "enso",
"measures": [{"category": "route", "metric": "route_p50_us", "value": 12.576}],
"visibility": "public", "canonical": True}],
import hashlib
import platform
import time
from hanzo_research import Benchmark, Research, Run, Split, capture, now
r = Research()
bench = Benchmark(
id="demo.sha256",
title="SHA-256 over 16 MiB",
splits=[Split("16mib", 30, "30 hashes of 16 MiB of zeros, one process, after one discarded")],
metric="p50_ms",
metrics=["p50_ms", "p95_ms", "mb_s"],
)
research.runs(benchmark="enso.route") # yours
research.runs(benchmark="mcp.injection", org="hanzo") # another org's public runs
research.compare("run-a", "run-b") # side by side, with what stops it
p = capture(device="cpu") # harness commit, host, CPU, GPU, Python, packages, date
start = now()
data = bytes(16 << 20)
hashlib.sha256(data).digest()
ms = []
for _ in range(30):
t = time.perf_counter()
hashlib.sha256(data).digest()
ms.append(1000 * (time.perf_counter() - t))
ms.sort()
run = Run(
id=f"demo.sha256-16mib-cpu-{p.host}",
benchmark=bench.id,
split="16mib",
system="hashlib",
version=platform.python_version(),
questions=30,
answered=30,
when=start,
ended=now(),
)
run.put("all", "p50_ms", ms[14], n=30, of=30)
run.put("all", "p95_ms", ms[28], n=30, of=30)
run.put("all", "mb_s", 16 / (ms[14] / 1000))
p.apply(run) # fills commit and by, appends the provenance sentence and its digest to notes
r.record(benchmarks=[bench], runs=[run]) # the definition, then the run: both private
r.grant(run.id, visibility="public", publishable=True)
back = r.get_run(run.id, org=r.org) # the public read: no credential
print(back.completion, back.measure("all", "p50_ms").value, back.commit)
```
## Zero-config auto-instrumentation
`completion` is the server's: `complete` only when `answered == questions` and `ended` is
set; `partial`, `inconsistent` or `unknown` otherwise. A model run names its weights:
`capture(model="kai-1.3", weights="/models/kai-1.3", device="cuda", dtype="bf16")` hashes a
file, or a directory by its `SHA256SUMS` (else every file under it), and `apply` sets
`system` to the model and `version` to `<model>@<sha256>`. A dirty harness tree is recorded
with the SHA-256 of `git diff HEAD` and the untracked files, so the number stays pinned to
the code that made it.
The caller supplies **no** provenance. On `experiment()` the SDK reads the run's
environment and captures three self-documenting narrative sources, plus machine context:
## The research plane: `Research`
- **the calling code's docstring** — *what this experiment is* (write a normal module
docstring);
- **the commit messages since this experiment's last recorded run** — *what changed +
why* (commit normally);
- an optional **`note=`** you pass.
| call | route |
|---|---|
| `record(benchmarks=, studies=, runs=, papers=)` | `POST /v1/research/{benchmarks,studies,runs,papers}`, one batch a non-empty kind, in that order: `{"runs": [...]}` |
| `benchmarks(id=, project=)`, `get_benchmark(id)` | `GET /v1/research/benchmarks` |
| `runs(id=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
| `runs(org=...)`, `get_run(id, org=...)` | the same route with `?org=` and no credential: that org's public runs |
| `studies(...)`, `papers(..., cites=)` | `GET /v1/research/{studies,papers}` |
| `compare(a, b)` | `GET /v1/research/compare`: the reasons the two runs may not be compared, the config they differ in, the deltas |
| `grant(id, visibility=, publishable=, trainable=)`, `grant(sha256=, visibility=)` | `POST /v1/research/grants`: on every version of the id; a write is always private until this |
| `revise(record)` | a new version marked `corrected` |
| `retract(id, why)` | a new version marked `retracted`: the id leaves every listing, earlier versions stay |
| `ingest`, `query`, `totals`, `projects`, `experiment(...)` | the experiment plane: `/v1/research/{experiments,totals,projects}` |
| `upload_artifact`, `upload_json_artifact`, `artifacts`, `get_artifact` | `/v1/research/artifacts[/<sha256>]`: `snapshot` or `report` bytes, addressed by the server's hash |
| `url(run_id, org=)` | the public read address of one run: the `source` a claim cites |
Plus `git_sha` / `git_branch` / `git_dirty`, `lib_versions`, and host. The project
self-documents as a side effect of running.
Every write is a batch of at most 20,000 and is idempotent by content: the same record
again records nothing. A mapping is sent as given; a `Run` is sent with every field. A
refusal raises `Denied` (401, 403), `Missing` (404), `Invalid` (400, 409, 422), `Limited`
(413, 429) or `Unavailable` (502 to 504, no answer), each carrying the server's own
sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves:
`json` refuses it before the request.
## Importer-agnostic, idempotent
## The arena: `Arena` (or `Research().arena`)
`ingest(experiments=…, attempts=…)` is one idempotent POST. Any source — a SQLite
backfill, git history, a kernel-perf campaign — maps its records to stable ids and calls
it; re-import is a no-op. "Load all research in" is just running importers.
| call | route |
|---|---|
| `catalog()` | `GET /v1/benchmark/catalog` |
| `leaderboard(benchmark)` | `GET /v1/benchmark/leaderboard`: each model's measured score (its latest run) beside a published claim. Only claims a SuperAdmin wrote under org `admin`, and public, reach it. |
| `claims(benchmark=, model=, provider=, source=, protocol=, org=)` | `GET /v1/benchmark/claims`: every org's public claims, labelled, and your org's private ones |
| `post_claims(rows, visibility=)` | `POST /v1/benchmark/claims`: under your acting org; each row names its `protocol` and its `source` (for a measured score, `Research.url(run_id)`), or it is refused here |
| `history(benchmark, model=)`, `compare(a, b, benchmark=)` | `GET /v1/benchmark/{history,compare}`: read the arena's per-item attempts plane, not the research runs |
| `presets()`, `post_preset(preset)` | `GET/POST /v1/benchmark/presets`: validated and echoed as `enso-<name>`, not stored |
| `post_run(benchmarks, model=, endpoint=, confirm=True)` | `POST /v1/benchmark/runs`: queues a paid execution, so it is refused without `confirm=True` |
## Config
## The command
`RESEARCH_BASE` (default `https://api.hanzo.ai`), `HANZO_API_KEY` (per-org key →
Bearer), `RESEARCH_PROJECT`. Uploads are private; public visibility, training, and
commons rights are each a separate grant. Stdlib only (urllib) — importable in any
zero-dep harness.
`hanzo-research` (or `python -m hanzo_research`); every answer is JSON on stdout.
```bash
hanzo-research post runs.jsonl --grant public --publishable # a run per line
hanzo-research post plan.json # {"benchmarks": [...], "runs": [...]}
hanzo-research post plan.json --dry # print the bodies, file nothing
hanzo-research list runs --benchmark demo.sha256
hanzo-research list runs --org hanzo # hanzo's public runs, no credential
hanzo-research get demo.sha256-16mib-cpu-dgx --org hanzo
hanzo-research grant RUN_ID ... --visibility public --publishable
hanzo-research retract RUN_ID --why "read on the wrong split"
hanzo-research compare RUN_A RUN_B
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
hanzo-research arena leaderboard --benchmark gpqa_diamond
```
It is not `hanzo research`: the `hanzo` command is the native CLI, and no Python package
in this workspace may install one.
## Records beside the API
- `canonical.canonicalize(value)`, `canonical.digest(value)`: RFC 8785 bytes and their
SHA-256, equal to the crate's `rfc8785_canonicalize`/`rfc8785_digest` on every string,
integer within 2^53 and float from 1e-6 to 1e21. Past that the crate writes Rust's
`Display` (`1e21` as 22 digits, `-0.0` as `-0`), which is not RFC 8785; this package
writes the RFC's `1e+21` and `0`. `tests/vectors/rust.json` holds the crate's own output.
- `lineage`: `DataPurpose`, `LineagePolicy` (a child never less restrictive than a parent),
`ContaminationError`, `verify_promotion_eligibility`, and the crate's evidence records
(`EnvironmentV1`, `GateSpec`, `Claim`, `TeacherInteraction`, `PairEvidence`,
`DeterminismReport`, `ArtifactRef`, `ResearchEvent`), each with the crate's digest.
- `wal.ResearchWal`: an fsynced, hash-chained event log. A log the crate wrote verifies here
(`wal.verify`), and the reverse.
## Tests
```bash
uv run --no-project --with pytest pytest pkg/hanzo-research # no network
uv run --no-project --with pytest pytest pkg/hanzo-research -m integration # api.hanzo.ai, no credential
```
pkg/hanzo-research/hanzo_research/__init__.py
+148 -39
View File
@@ -1,36 +1,133 @@
"""Hanzo Research SDK — record and query R&D evidence in one import.
"""Hanzo Research: file measurements to `/v1/research` and `/v1/benchmark` with their provenance.
import hanzo_research as research
from hanzo_research import Research, Run, capture, now
exp = research.experiment("benchmark", "grok-4.5", "gpqa_diamond")
exp.record("q1", "grok-4.5", {"answer": "A", "correct": True})
exp.snapshot(png_bytes)
exp.report(markdown_text)
exp.finish(value=94.3)
r = Research() # HANZO_API_KEY, else `hanzo auth token`
p = capture(model="kai-1.3", weights="/models/kai-1.3", device="cpu", dtype="f32")
run = Run(id="demo.latency-dev-cpu-kai", benchmark="demo.latency", split="dev",
questions=200, answered=200, when=p.date)
run.put("all", "p50_ms", 38.3, n=200, of=200)
run.ended = now()
p.apply(run) # commit, system, version, by, notes
r.record(runs=[run])
r.grant(run.id, visibility="public", publishable=True)
r.get_run(run.id, org="hanzo") # the public read, no credential
rows = research.query(project="enso-bench", kind="benchmark")
Zero-config: provenance (git sha/branch + the commit-message narrative since the last
recorded run, lib versions, host) is auto-captured — the caller supplies none. Records
mirror to cloud /v1/research (HIP-0512): versioned, append-only, idempotent by content,
private by default.
The Rust peer is the `hanzo-research` crate (hanzoai/ml): the same wire records, the same
RFC 8785 digests, the same lineage and WAL records.
"""
from .client import Research, Experiment
__all__ = ["Research", "Experiment", "experiment", "query", "totals", "configure", "client",
"record", "benchmarks", "runs", "studies", "papers", "compare"]
from . import wal, wire, lineage, benchmark, canonical, provenance
from .wal import ResearchWal
from .wire import (
Ack,
Gap,
Run,
Batch,
Delta,
Paper,
Split,
Study,
Census,
Record,
Stored,
Totals,
Listing,
Measure,
Summary,
Artifact,
Benchmark,
KindTotal,
Comparison,
Difference,
now,
left,
scope,
token,
claimed,
completion,
)
from .client import DEFAULT_LIBS, Research, Experiment
from .errors import Error, Denied, Invalid, Limited, Missing, Unavailable
from .benchmark import Arena
from .canonical import digest, canonicalize
from .transport import VERSION
from .provenance import Provenance, capture
_default = None
__version__ = VERSION
__all__ = [
"Research",
"Experiment",
"Arena",
"DEFAULT_LIBS",
"Split",
"Benchmark",
"Measure",
"Run",
"Study",
"Paper",
"Difference",
"Delta",
"Comparison",
"Census",
"Gap",
"Listing",
"Batch",
"Stored",
"Artifact",
"Record",
"Ack",
"KindTotal",
"Totals",
"Summary",
"completion",
"token",
"claimed",
"left",
"scope",
"now",
"canonicalize",
"digest",
"Provenance",
"capture",
"ResearchWal",
"Error",
"Denied",
"Missing",
"Invalid",
"Limited",
"Unavailable",
"benchmark",
"canonical",
"lineage",
"provenance",
"wal",
"wire",
"configure",
"client",
"experiment",
"query",
"totals",
"record",
"benchmarks",
"runs",
"studies",
"papers",
"compare",
]
_default: Research | None = None
def configure(**kwargs):
"""Set the process-wide default client (base, api_key, project, repo, libs)."""
def configure(**kwargs) -> Research: # type: ignore[no-untyped-def]
"""Set the process-wide default client (base, api_key, project, repo, libs, timeout, org)."""
global _default
_default = Research(**kwargs)
return _default
def client():
def client() -> Research:
"""The process-wide default client, built from the environment on first use."""
global _default
if _default is None:
@@ -38,45 +135,57 @@ def client():
return _default
def experiment(kind, subject, task, metric="accuracy", n_total=0, note="",
hypothesis="", predict=""):
"""Get/create an experiment handle on the default client. Pass `hypothesis` + `predict`
to structure it as a falsifiable test that conclude(verdict) can prove or refute."""
return client().experiment(kind, subject, task, metric=metric, n_total=n_total, note=note,
hypothesis=hypothesis, predict=predict)
def experiment(
kind: str,
subject: str,
task: str,
metric: str = "accuracy",
n_total: int = 0,
note: str = "",
hypothesis: str = "",
predict: str = "",
) -> Experiment:
"""An experiment handle on the default client (see `Experiment`)."""
return client().experiment(
kind, subject, task, metric=metric, n_total=n_total, note=note, hypothesis=hypothesis, predict=predict
)
def query(project=None, kind=None, canonical=True):
"""Read canonical experiments on the default client."""
return client().query(project=project, kind=kind, canonical=canonical)
def query(project: str | None = None, kind: str | None = None) -> list[Record]:
"""Canonical experiments on the default client."""
return client().query(project=project, kind=kind)
def totals(project=None):
"""Read the headline totals (canonical + retained) on the default client."""
def totals(project: str | None = None) -> Totals:
"""The headline totals on the default client."""
return client().totals(project=project)
def record(benchmarks=(), runs=(), studies=(), papers=()):
"""Record benchmark definitions, runs, studies and papers on the default client."""
def record(benchmarks=(), runs=(), studies=(), papers=()) -> dict[str, Batch]: # type: ignore[no-untyped-def]
"""File benchmarks, studies, runs and papers on the default client (see `Research.record`)."""
return client().record(benchmarks=benchmarks, runs=runs, studies=studies, papers=papers)
def benchmarks(**filters):
def benchmarks(**filters) -> list[Benchmark]: # type: ignore[no-untyped-def]
"""Canonical benchmark definitions on the default client."""
return client().benchmarks(**filters)
def runs(**filters):
"""Runs, e.g. runs(benchmark="coding.own", org="hanzo") for another org's public ones."""
def runs(**filters) -> Listing: # type: ignore[no-untyped-def]
"""Canonical runs on the default client; `org=` reads another org's public runs."""
return client().runs(**filters)
def studies(**filters):
def studies(**filters) -> list[Study]: # type: ignore[no-untyped-def]
"""Canonical studies on the default client."""
return client().studies(**filters)
def papers(**filters):
def papers(**filters) -> list[Paper]: # type: ignore[no-untyped-def]
"""Canonical papers on the default client."""
return client().papers(**filters)
def compare(a, b):
def compare(a: str, b: str) -> Comparison:
"""Two runs side by side on the default client."""
return client().compare(a, b)
pkg/hanzo-research/hanzo_research/__main__.py
+7
View File
@@ -0,0 +1,7 @@
"""`python -m hanzo_research`: the `hanzo-research` command."""
import sys
from .cli import main
sys.exit(main())
pkg/hanzo-research/hanzo_research/benchmark.py
+273
View File
@@ -0,0 +1,273 @@
"""`Arena`: the `/v1/benchmark` routes hanzo-inc/cloud `apps/benchmark` serves.
Two planes, never blended. The MEASURED plane is per-item attempts the arena's own harness
made; `leaderboard` reads each model's latest run of them, and `history` and `compare`
read every run of them (`compare` pairs two models on the items both answered and gives an
exact McNemar p). The PUBLISHED plane is claims: someone's reported score, each with the
source it was read from.
What reads where:
* `leaderboard` shows, beside the measured score, only claims an org-`admin` SuperAdmin
wrote, and those public. Every other org's claims are at `claims`, labelled with their
org, and never on the board.
* `claims` shows public claims of every org, plus the caller's own org's private ones.
* `post_claims` writes under the caller's acting org, private unless `visibility="public"`.
Writing under org `admin` needs a SuperAdmin token (owner `admin`).
* `post_run` asks the arena to measure a model: a PAID execution, so it is refused here
unless `confirm=True`. The answer is an admission, not a result.
* `post_preset` validates a blend and echoes it as `enso-<name>`; the arena stores nothing.
"""
from __future__ import annotations
from typing import Any, Mapping, Iterable
from dataclasses import field, dataclass
from .wire import Wire, dump
from .errors import Invalid
from .transport import Transport
__all__ = [
"Arena",
"Entry",
"Row",
"Board",
"Pairing",
"Point",
"History",
"Claim",
"Filed",
"Admission",
"Preset",
"Accepted",
]
DEFAULT = "gpqa_diamond"
@dataclass
class Entry(Wire):
"""One canonical public test of the catalog; `native` marks the ones the harness runs today."""
id: str = ""
title: str = ""
axis: str = ""
items: int = 0
native: bool = False
source: str = ""
@dataclass
class Row(Wire):
"""One model on a board: its measured score (latest run, Wilson 95%) beside the platform's claim."""
model: str = ""
measured: float | None = None
n: int = 0
published: float | None = None
gap: float | None = None
protocol: str = ""
claims: int = 0
spread: float | None = None
mean: float | None = None
run: str = ""
measuredAt: str | None = None # noqa: N815 (the wire's name)
ciLow: float | None = None # noqa: N815
ciHigh: float | None = None # noqa: N815
@dataclass
class Board(Wire):
"""A benchmark's leaderboard, measured score descending."""
benchmark: str = ""
rows: list[Row] = field(default_factory=list)
@dataclass
class Pairing(Wire):
"""Two models paired on the items both completed: rescues each way and an exact McNemar p."""
benchmark: str = ""
a: str = ""
b: str = ""
n_common: int = 0
a_correct: int = 0
b_correct: int = 0
rescue_a_over_b: int = 0
rescue_b_over_a: int = 0
net_a_minus_b: int = 0
mcnemar_p: float = 1.0
@dataclass
class Point(Wire):
"""One run of a model on a benchmark: its score, items, and change from the run before."""
run: str = ""
at: str | None = None
score: float = 0.0
n: int = 0
delta: float | None = None
@dataclass
class History(Wire):
"""A model's runs on a benchmark in time order, and the first-to-last change."""
model: str = ""
points: list[Point] = field(default_factory=list)
trend: float | None = None
@dataclass
class Claim(Wire):
"""A reported score and its source. A write sends the first six fields; a read fills the rest."""
benchmark: str = ""
provider: str = ""
model: str = ""
score: float = 0.0
protocol: str = ""
source: str = ""
org: str = ""
visibility: str = ""
origin: str = ""
at: str | None = None
by: str = ""
@dataclass
class Filed(Wire):
"""A claims write: how many rows landed, which were refused and why, under which org and visibility."""
recorded: int = 0
rejected: list[str] = field(default_factory=list)
org: str = ""
visibility: str = ""
@dataclass
class Admission(Wire):
"""A queued arena run: admitted, not finished."""
status: str = ""
org: str = ""
by: str = ""
model: str = ""
endpoint: str = ""
benchmarks: list[str] = field(default_factory=list)
note: str = ""
@dataclass
class Preset(Wire):
"""A router blend served as `enso-<name>`: its arms, escalation rank and panel width."""
name: str = ""
owner: str = ""
arms: list[str] = field(default_factory=list)
rank: list[str] = field(default_factory=list)
panel: int = 0
note: str = ""
@dataclass
class Accepted(Wire):
"""A preset write's echo: validated, not stored."""
status: str = ""
served_as: str = ""
preset: Preset = field(default_factory=Preset)
by: str = ""
note: str = ""
_CLAIM = ("benchmark", "provider", "model", "score", "protocol", "source")
class Arena:
"""The benchmark arena on a credential and an acting org (both as `Research` resolves them)."""
def __init__(
self,
base: str | None = None,
api_key: str | None = None,
org: str | None = None,
timeout: float = 120.0,
transport: Transport | None = None,
):
self.http = transport or Transport(base, api_key, org, None, timeout)
def catalog(self) -> list[Entry]:
return [Entry.load(x) for x in self.http.get("/v1/benchmark/catalog").get("data") or []]
def leaderboard(self, benchmark: str = DEFAULT) -> Board:
return Board.load(self.http.get("/v1/benchmark/leaderboard", {"benchmark": benchmark}))
def compare(self, a: str, b: str, benchmark: str = DEFAULT) -> Pairing:
return Pairing.load(self.http.get("/v1/benchmark/compare", {"benchmark": benchmark, "a": a, "b": b}))
def history(self, benchmark: str = DEFAULT, model: str | None = None) -> list[History]:
out = self.http.get("/v1/benchmark/history", {"benchmark": benchmark, "model": model})
return [History.load(x) for x in out.get("data") or []]
def claims(
self,
benchmark: str | None = None,
model: str | None = None,
provider: str | None = None,
source: str | None = None,
protocol: str | None = None,
org: str | None = None,
) -> list[Claim]:
q = {
"benchmark": benchmark,
"model": model,
"provider": provider,
"source": source,
"protocol": protocol,
"org": org,
}
return [Claim.load(x) for x in self.http.get("/v1/benchmark/claims", q).get("data") or []]
def post_claims(self, rows: Iterable[Claim | Mapping[str, Any]], visibility: str = "private") -> Filed:
"""Write claims under the acting org (at most 500 a call, 2,000 an org a UTC day).
Every row names its `protocol` (how the score was made: hanzo-measured, provider-reported,
oracle-ceiling, ...) and its `source` (for a measured score, the research run it came from:
`Research.url(run_id)`); a row without either is refused here. The server refuses a row
outside the catalog or a score outside 0..100 and says which in `rejected`.
"""
body = []
for i, r in enumerate(rows):
d = dump(r)
row = {k: d.get(k, 0.0 if k == "score" else "") for k in _CLAIM}
if not str(row["protocol"]).strip() or not str(row["source"]).strip():
raise Invalid(f"claim row {i} needs a protocol and a source")
body.append(row)
return Filed.load(self.http.post("/v1/benchmark/claims", {"data": body, "visibility": visibility}))
def presets(self) -> list[Preset]:
return [Preset.load(x) for x in self.http.get("/v1/benchmark/presets").get("data") or []]
def post_preset(self, preset: Preset | Mapping[str, Any]) -> Accepted:
return Accepted.load(self.http.post("/v1/benchmark/presets", dump(preset)))
def post_run(
self,
benchmarks: list[str],
model: str | None = None,
endpoint: str | None = None,
attempts: int | None = None,
*,
confirm: bool = False,
) -> Admission:
"""Queue the arena's harness on a model or your endpoint. It spends money: `confirm=True` or it is refused."""
if not confirm:
raise Invalid("POST /v1/benchmark/runs queues a paid execution; pass confirm=True to queue it")
body: dict[str, Any] = {"benchmarks": benchmarks}
for k, v in (("model", model), ("endpoint", endpoint), ("attempts", attempts)):
if v:
body[k] = v
return Admission.load(self.http.post("/v1/benchmark/runs", body))
pkg/hanzo-research/hanzo_research/canonical.py
+134
View File
@@ -0,0 +1,134 @@
"""RFC 8785 JSON canonicalization (JCS) and SHA-256 digests over it.
The Python peer of `canonical.rs` in the Rust crate: keys sorted by UTF-16 code
units, no whitespace, strings escaped by RFC 8785's table, numbers written as
ECMAScript writes them. Over strings, booleans, null, integers within 2**53 and
floats from 1e-6 up to 1e21 in magnitude the bytes equal the crate's
(`tests/vectors/rust.json` holds the crate's own output). Outside that range the
crate writes Rust's `Display` (`1e21` as 22 digits, `-0.0` as `-0`), which is not
RFC 8785; this module writes the RFC's `1e+21` and `0`.
"""
from __future__ import annotations
import enum
import math
import hashlib
import dataclasses
from typing import Any
__all__ = ["canonicalize", "digest", "number"]
_ESCAPES = {'"': '\\"', "\\": "\\\\", "\b": "\\b", "\f": "\\f", "\n": "\\n", "\r": "\\r", "\t": "\\t"}
def canonicalize(value: Any) -> bytes:
"""The RFC 8785 canonical bytes of a JSON value, a wire dataclass, or a mix of them."""
out: list[str] = []
_write(_plain(value), out)
try:
return "".join(out).encode("utf-8")
except UnicodeEncodeError as e:
raise ValueError(f"canonical JSON needs valid Unicode: {e}") from None
def digest(value: Any) -> str:
"""Lowercase hex SHA-256 of `canonicalize(value)`, as `rfc8785_digest` in the crate."""
return hashlib.sha256(canonicalize(value)).hexdigest()
def number(x: int | float) -> str:
"""A number as ECMAScript's Number.prototype.toString writes it (RFC 8785 section 3.2.2.3)."""
if isinstance(x, bool):
raise TypeError("a boolean is not a number")
if isinstance(x, int) and abs(x) <= 2**53:
return str(x)
try:
f = float(x)
except OverflowError:
raise ValueError(f"{x} is past the range of a double") from None
if not math.isfinite(f):
raise ValueError(f"{f} has no JSON form")
if f == 0:
return "0"
sign = "-" if f < 0 else ""
# repr is the shortest decimal that reads back as f, as ECMAScript's digits are
mantissa, _, exp = repr(abs(f)).partition("e")
whole, _, frac = mantissa.partition(".")
every = whole + frac
digits = every.lstrip("0")
# value = 0.<digits> * 10**n
n = len(whole) + int(exp or 0) - (len(every) - len(digits))
digits = digits.rstrip("0")
k = len(digits)
if k <= n <= 21:
return sign + digits + "0" * (n - k)
if 0 < n <= 21:
return sign + digits[:n] + "." + digits[n:]
if -6 < n <= 0:
return sign + "0." + "0" * -n + digits
e = n - 1
mark = "+" if e > 0 else "-"
head = digits if k == 1 else digits[0] + "." + digits[1:]
return f"{sign}{head}e{mark}{abs(e)}"
def _plain(value: Any) -> Any:
"""Wire objects to plain JSON values: dataclasses by their fields, enums by value."""
if dataclasses.is_dataclass(value) and not isinstance(value, type):
dump = getattr(value, "dump", None)
return _plain(dump() if callable(dump) else dataclasses.asdict(value))
if isinstance(value, enum.Enum):
return _plain(value.value)
if isinstance(value, dict):
return {k: _plain(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [_plain(v) for v in value]
return value
def _write(v: Any, out: list[str]) -> None:
if v is None:
out.append("null")
elif v is True:
out.append("true")
elif v is False:
out.append("false")
elif isinstance(v, (int, float)):
out.append(number(v))
elif isinstance(v, str):
_string(v, out)
elif isinstance(v, list):
out.append("[")
for i, x in enumerate(v):
if i:
out.append(",")
_write(x, out)
out.append("]")
elif isinstance(v, dict):
for k in v:
if not isinstance(k, str):
raise TypeError(f"a JSON object key is a string, not {type(k).__name__}")
out.append("{")
for i, k in enumerate(sorted(v, key=lambda s: s.encode("utf-16-be", "surrogatepass"))):
if i:
out.append(",")
_string(k, out)
out.append(":")
_write(v[k], out)
out.append("}")
else:
raise TypeError(f"{type(v).__name__} has no JSON form")
def _string(s: str, out: list[str]) -> None:
out.append('"')
for c in s:
e = _ESCAPES.get(c)
if e is not None:
out.append(e)
elif c < " ":
out.append(f"\\u{ord(c):04x}")
else:
out.append(c)
out.append('"')
pkg/hanzo-research/hanzo_research/cli.py
+223
View File
@@ -0,0 +1,223 @@
"""`hanzo-research`: file, grant, read and compare research records from a shell.
hanzo-research post runs.jsonl --grant public --publishable
hanzo-research post plan.json # {"benchmarks": [...], "runs": [...], ...}
hanzo-research list runs --benchmark decision.latency
hanzo-research list runs --org hanzo # an org's public runs, no credential
hanzo-research get decision.latency-loopback-dgx.cpu-kai --org hanzo
hanzo-research grant RUN_ID ... --visibility public --publishable
hanzo-research retract RUN_ID --why "read on the wrong split"
hanzo-research compare RUN_A RUN_B
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
hanzo-research arena leaderboard --benchmark gpqa_diamond
Every answer is JSON on stdout. The token is HANZO_API_KEY, else `hanzo auth token`.
"""
from __future__ import annotations
import os
import sys
import json
import argparse
from typing import Any, Iterable
from .wire import Run, Paper, Study, Benchmark, dump
from .client import Research
from .errors import Error
from .benchmark import Arena
from .provenance import capture
KINDS = {"run": "runs", "benchmark": "benchmarks", "study": "studies", "paper": "papers"}
def read(path: str) -> Any:
"""A JSON document, or the records of a JSON Lines file, from a path or `-` for stdin."""
text = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read()
try:
return json.loads(text)
except json.JSONDecodeError:
return [json.loads(line) for line in text.splitlines() if line.strip()]
def plan(doc: Any, kind: str) -> dict[str, list[Any]]:
"""What a document files, by kind: a plan's keys, else its records as one kind."""
if isinstance(doc, dict) and any(k in doc for k in KINDS.values()):
return {k: list(doc[k]) for k in ("benchmarks", "studies", "runs", "papers") if doc.get(k)}
return {KINDS[kind]: doc if isinstance(doc, list) else [doc]}
def main(argv: Iterable[str] | None = None) -> int:
p = argparse.ArgumentParser(
prog="hanzo-research", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
p.add_argument("--base", help="API base (RESEARCH_BASE, else https://api.hanzo.ai)")
p.add_argument("--org", help="act in this org (X-Org-Id); the token must belong to it")
p.add_argument("--project", help="the token's project (default: its claim)")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("post", help="file records from a JSON or JSON Lines file")
s.add_argument("file")
s.add_argument("--kind", choices=KINDS, default="run", help="what a bare record or list is (default run)")
s.add_argument("--grant", choices=("private", "org", "public"), help="then set the filed runs' visibility")
s.add_argument("--publishable", action=argparse.BooleanOptionalAction, help="with the grant: commons consent")
s.add_argument("--trainable", action=argparse.BooleanOptionalAction, help="with the grant: training consent")
s.add_argument("--dry", action="store_true", help="print the bodies, file nothing")
s = sub.add_parser("list", help="canonical records")
s.add_argument(
"what",
nargs="?",
default="runs",
choices=("runs", "benchmarks", "studies", "papers", "experiments", "artifacts", "projects", "totals"),
)
for f in ("id", "benchmark", "system", "study", "completion", "since", "until", "kind", "run", "cites"):
s.add_argument(f"--{f}")
s.add_argument("--org", dest="public", help="runs: that org's public runs, no credential")
s.add_argument("--in", dest="scope", help="narrow to one project of the org")
s = sub.add_parser("get", help="one record's canonical version")
s.add_argument("id")
s.add_argument("--kind", choices=KINDS, default="run")
s.add_argument("--org", dest="public", help="read that org's public runs, no credential")
s = sub.add_parser("grant", help="visibility and consent for runs, or an artifact by --sha256")
s.add_argument("ids", nargs="*")
s.add_argument("--sha256")
s.add_argument("--visibility", choices=("private", "org", "public"))
s.add_argument("--publishable", action=argparse.BooleanOptionalAction)
s.add_argument("--trainable", action=argparse.BooleanOptionalAction)
s = sub.add_parser("retract", help="withdraw a run: a version marked retracted")
s.add_argument("id")
s.add_argument("--why", required=True)
s = sub.add_parser("compare", help="two runs side by side")
s.add_argument("a")
s.add_argument("b")
s = sub.add_parser("provenance", help="print what capture() reads here")
s.add_argument("--repo")
s.add_argument("--model", default="")
s.add_argument("--weights")
s.add_argument("--device", default="")
s.add_argument("--dtype", default="")
s.add_argument("--package", action="append", help="a distribution whose version pins the result")
s.add_argument("--no-gpu", action="store_true")
s = sub.add_parser("arena", help="the /v1/benchmark arena")
s.add_argument("what", choices=("catalog", "leaderboard", "claims", "history", "compare", "presets"))
s.add_argument("models", nargs="*", help="compare: two model ids")
for f in ("benchmark", "model", "provider", "source", "protocol"):
s.add_argument(f"--{f}")
s.add_argument("--of", dest="owner", help="claims: only this org's")
a = p.parse_args(list(argv) if argv is not None else None)
try:
out = run(a)
except Error as e:
print(f"hanzo-research: {e}", file=sys.stderr)
return 1
except (OSError, ValueError) as e:
print(f"hanzo-research: {e}", file=sys.stderr)
return 2
try:
json.dump(dump(out), sys.stdout, indent=1, ensure_ascii=False)
sys.stdout.write("\n")
sys.stdout.flush()
except BrokenPipeError:
# the reader stopped early (`| head`): end quietly
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
return 0
def run(a: argparse.Namespace) -> Any:
r = Research(base=a.base, project=a.project, org=a.org, repo=".")
if a.cmd == "post":
filed = plan(read(a.file), a.kind)
if a.dry:
return filed
out: dict[str, Any] = dict(r.record(**filed))
if a.grant or a.publishable is not None or a.trainable is not None:
ids = [dump(x)["id"] for x in filed.get("runs", [])]
out["granted"] = {
i: r.grant(i, visibility=a.grant, publishable=a.publishable, trainable=a.trainable) for i in ids
}
return out
if a.cmd == "list":
w = a.what
if w == "runs":
return r.runs(
id=a.id,
benchmark=a.benchmark,
system=a.system,
study=a.study,
completion=a.completion,
since=a.since,
until=a.until,
project=a.scope,
org=a.public,
)
if w == "benchmarks":
return r.benchmarks(id=a.id, project=a.scope)
if w == "studies":
return r.studies(id=a.id, project=a.scope)
if w == "papers":
return r.papers(id=a.id, study=a.study, cites=a.cites, project=a.scope)
if w == "experiments":
return r.query(project=a.scope, kind=a.kind)
if w == "artifacts":
return r.artifacts(run=a.run, project=a.scope)
if w == "projects":
return r.projects()
return r.totals(project=a.scope)
if a.cmd == "get":
if a.kind == "run":
return r.get_run(a.id, org=a.public)
if a.kind == "benchmark":
return r.get_benchmark(a.id)
rows: list[Any] = list(r.studies(id=a.id)) if a.kind == "study" else list(r.papers(id=a.id))
if not rows:
raise Error(f"no canonical {a.kind} {a.id!r}", 404)
return rows[0]
if a.cmd == "grant":
if a.sha256:
return {a.sha256: r.grant(sha256=a.sha256, visibility=a.visibility)}
if not a.ids:
raise ValueError("grant needs run ids or --sha256")
return {i: r.grant(i, visibility=a.visibility, publishable=a.publishable, trainable=a.trainable) for i in a.ids}
if a.cmd == "retract":
return r.retract(a.id, a.why)
if a.cmd == "compare":
return r.compare(a.a, a.b)
if a.cmd == "provenance":
return capture(
repo=a.repo,
model=a.model,
weights=a.weights,
device=a.device,
dtype=a.dtype,
packages=a.package,
gpu=not a.no_gpu,
)
arena = Arena(transport=r.http)
bench = a.benchmark or "gpqa_diamond"
if a.what == "catalog":
return arena.catalog()
if a.what == "leaderboard":
return arena.leaderboard(bench)
if a.what == "claims":
return arena.claims(
benchmark=a.benchmark, model=a.model, provider=a.provider, source=a.source, protocol=a.protocol, org=a.owner
)
if a.what == "history":
return arena.history(bench, a.model)
if a.what == "compare":
if len(a.models) != 2:
raise ValueError("arena compare needs two model ids")
return arena.compare(a.models[0], a.models[1], bench)
return arena.presets()
if __name__ == "__main__":
sys.exit(main())
pkg/hanzo-research/hanzo_research/client.py
+401 -222
View File
@@ -1,305 +1,484 @@
"""Hanzo Research SDK — the ONE way every producer records and queries research.
"""`Research`: every route hanzo-inc/cloud `apps/research` serves at `/v1/research`.
Generated-shape from the openapi/research contract, wrapped in a tiny, intuitive,
ZERO-CONFIG surface so hand-rolling is obviously the worse choice:
Two planes on one client. The RECORD plane names what a number needs to be believed:
benchmarks (the task), runs (the execution, its completion derived by the server),
studies (the question) and papers (the writing that cites runs). The EXPERIMENT plane is
the older stream of one measured number per id plus its attempts and diary artifacts; the
`experiment()` handle files to it.
import hanzo_research as research
# A benchmark run — accumulate attempts, seal with the score.
exp = research.experiment("benchmark", "grok-4.5", "gpqa_diamond") # get/create → handle
exp.record("q1", "grok-4.5", {"answer": "A", "correct": True}) # an attempt, idempotent
exp.snapshot(png_bytes) # a board snapshot artifact
exp.finish(value=94.3) # seal the run (or use `with`)
# A kernel-perf run structured as a falsifiable test — state the claim, log what you
# see, then PROVE or REFUTE it. A refutation is a first-class, durable result.
k = research.experiment("kernel-perf", "matvec_q4k_f32_blk", "vulkan/6144x2048",
metric="ratio_vs_hand",
hypothesis="the DSL f32-direct matvec beats the hand kernel",
predict="DSL/hand >= 1.0 cold in-engine at the dominant FFN shape")
k.log("cold in-engine A/B, evo gfx1151, quiet window, 3 runs, bit-exact 2.3e-6")
k.conclude("proven", because="1.022x at 6144 rows (loses small shapes → gate >=4096)",
value=1.022) # git sha + lib versions auto-stamped
rows = research.query(project="enso-bench", kind="benchmark") # read canonical
Auto-instrumentation (the caller supplies NO provenance): on experiment()/finish() the
SDK reads git sha/branch/dirty, the commit messages SINCE the last recorded run of this
experiment (the narrative of what changed), the installed lib versions, and the host —
and weaves them into the record. Research self-documents as a side effect of running.
Auth is the per-org key (private-by-default is enforced server-side). Records mirror to
the cloud /v1/research surface (HIP-0512), which is versioned + append-only + idempotent
by content, so re-recording the same result is a no-op.
Everything a caller writes lands private, under the caller's org, in the token's project.
Visibility (private, org, public) and the training and commons consents are a separate
call, `grant`. Records are versioned and append-only: a correction is a new version under
the same id (`revise`), a withdrawal is a version marked retracted (`retract`), and every
earlier version stays retained.
"""
from __future__ import annotations
import base64
import hashlib
import json
import os
import urllib.error
import urllib.parse
import urllib.request
from typing import Any, Mapping, TypeVar, Iterable
from . import provenance
from . import canonical, provenance
from .wire import (
Ack,
Run,
Batch,
Paper,
Study,
Record,
Stored,
Totals,
Listing,
Summary,
Artifact,
Benchmark,
Comparison,
dump,
)
from .errors import Invalid, Missing
from .transport import Transport
__all__ = ["Research", "Experiment", "DEFAULT_LIBS"]
DEFAULT_LIBS = [
"hanzo-kernel", "hanzo-ml", "hanzo-engine", "hanzo-research",
"enso-bench", "datasets", "huggingface-hub", "torch", "transformers",
"hanzo-kernel",
"hanzo-ml",
"hanzo-engine",
"hanzo-research",
"enso-bench",
"datasets",
"huggingface-hub",
"torch",
"transformers",
]
VERDICTS = ("proven", "refuted", "inconclusive")
R = TypeVar("R", Benchmark, Run, Study, Paper)
# the batch key, and route, of each record kind
_KEYS: dict[type, str] = {Benchmark: "benchmarks", Run: "runs", Study: "studies", Paper: "papers"}
class Research:
"""A configured research client. Base URL, per-org key, and project come from the
environment by default (HANZO_API_KEY, RESEARCH_BASE, RESEARCH_PROJECT); the repo the
caller runs in is auto-detected for provenance."""
"""A research client. Base URL, credential, project and acting org come from the environment
by default (`RESEARCH_BASE`, `HANZO_API_KEY` else `hanzo auth token`, the token's `project`
claim, the token's `owner`); `org` overrides the org, which the token must belong to.
`repo` is the checkout whose git state stamps the experiment plane (default: the git
toplevel of the working directory).
"""
def __init__(self, base=None, api_key=None, project=None, repo=None, libs=None, timeout=120):
self.base = (base or os.environ.get("RESEARCH_BASE", "https://api.hanzo.ai")).rstrip("/")
self.api_key = api_key or os.environ.get("HANZO_API_KEY", "")
self.project = project or os.environ.get("RESEARCH_PROJECT", "default")
def __init__(
self,
base: str | None = None,
api_key: str | None = None,
project: str | None = None,
repo: str | None = None,
libs: list[str] | None = None,
timeout: float = 120.0,
org: str | None = None,
transport: Transport | None = None,
):
self.http = transport or Transport(base, api_key, org, project, timeout)
self.repo = repo or provenance.find_repo()
self.libs = libs if libs is not None else DEFAULT_LIBS
self.timeout = timeout
self._git: dict[str, Any] | None = None
# ── transport ──────────────────────────────────────────────────────────────
def _headers(self):
# Auth is ONLY the per-org key (the gateway mints the validated principal from it).
# The client NEVER mints X-User-Id/X-Org-Id — that is a cross-tenant forge the
# gateway strips anyway (SanitizeIdentity); any local/dev bypass belongs SERVER-side
# behind an explicit unsafe flag, never in the client.
h = {"Content-Type": "application/json", "X-Project-Id": self.project}
if self.api_key:
h["Authorization"] = "Bearer " + self.api_key
return h
@property
def base(self) -> str:
return self.http.base
def _req(self, method, path, payload=None):
data = json.dumps(payload).encode() if payload is not None else None
req = urllib.request.Request(self.base + path, data=data, headers=self._headers(), method=method)
with urllib.request.urlopen(req, timeout=self.timeout) as r:
return json.loads(r.read() or b"{}")
@property
def project(self) -> str:
return self.http.project
# ── low-level surface (mirrors the contract; used by the bulk uploader) ──────
def ingest(self, experiments=None, attempts=None):
"""POST a batch of experiments + attempts. Idempotent by content; returns the
cloud counts (canonical + retained)."""
return self._req("POST", "/v1/research/experiments",
{"experiments": experiments or [], "attempts": attempts or []})
@property
def org(self) -> str | None:
return self.http.org
def artifact(self, art):
return self._req("POST", "/v1/research/artifacts", art)
@property
def arena(self) -> Any:
"""The `/v1/benchmark` arena on this client's credential and org."""
from .benchmark import Arena
def grant(self, **g):
return self._req("POST", "/v1/research/grants", g)
return Arena(transport=self.http)
# ── ergonomic surface ───────────────────────────────────────────────────────
def experiment(self, kind, subject, task, metric="accuracy", n_total=0, note="",
hypothesis="", predict=""):
"""Get/create the experiment for (kind, subject, task) and return a handle. Auto-
captures provenance + the self-documenting narrative (the caller's docstring, the
commit messages since this experiment's last run, and an optional note).
def url(self, run_id: str, org: str | None = None) -> str:
"""The address that reads one run of an org's public runs, with no credential: a claim's source."""
from urllib.parse import quote
Structure it as a falsifiable test: `hypothesis` is the claim under test and
`predict` is the observation that would confirm it — stated up front so a later
conclude(verdict) can PROVE or REFUTE it unambiguously. A refutation is a
first-class result, recorded as clearly as a proof."""
return Experiment(self, kind, subject, task, metric=metric, n_total=n_total,
note=note, hypothesis=hypothesis, predict=predict)
org = org or self.org
if not org:
raise ValueError("no org to read the run from: pass org=")
return f"{self.base}/v1/research/runs?org={quote(org, safe='')}&id={quote(run_id, safe='')}"
def query(self, project=None, kind=None, canonical=True):
"""Read canonical experiments (the answered latest-run view). canonical=False is
reserved for the full retained-version history (a follow-up endpoint; the retained
COUNTS are already on /v1/research/totals)."""
q = []
if project or self.project:
q.append("project=" + urllib.parse.quote(project or self.project))
if kind:
q.append("kind=" + urllib.parse.quote(kind))
path = "/v1/research/experiments" + ("?" + "&".join(q) if q else "")
out = self._req("GET", path)
return out.get("data", [])
# ── record plane: writes ─────────────────────────────────────────────────────
def totals(self, project=None):
path = "/v1/research/totals" + ("?project=" + urllib.parse.quote(project) if project else "")
return self._req("GET", path)
def record(
self,
benchmarks: Iterable[Benchmark | Mapping[str, Any]] = (),
runs: Iterable[Run | Mapping[str, Any]] = (),
studies: Iterable[Study | Mapping[str, Any]] = (),
papers: Iterable[Paper | Mapping[str, Any]] = (),
) -> dict[str, Batch]:
"""File records: one write a non-empty kind, at most 20,000 records each, benchmarks and
studies before the runs that name them and papers last. The answers by kind; nothing
given sends nothing. A `Run` (or any record) is sent with every field, a mapping as given.
"""
return self._write({"benchmarks": benchmarks, "studies": studies, "runs": runs, "papers": papers})
# ── the record: benchmarks, runs, studies, papers ───────────────────────────
# A benchmark defines what is measured; a run is one execution of it with its
# measures (each {category, metric, value, lo?, hi?, n?, of?}); a study is the
# question runs answer; a paper is writing that cites runs. Every write is a
# batch, versioned and idempotent by content: re-recording the same row is a
# no-op, and a changed one becomes its next revision.
def record(self, benchmarks=(), runs=(), studies=(), papers=()):
"""POST each non-empty batch and return the cloud's answers by kind."""
def _write(self, batches: Mapping[str, Iterable[Any]]) -> dict[str, Batch]:
out = {}
for kind, rows in (("benchmarks", benchmarks), ("studies", studies), ("runs", runs), ("papers", papers)):
if rows:
out[kind] = self._req("POST", "/v1/research/" + kind, {kind: list(rows)})
for key, rows in batches.items():
body = [dump(r) for r in rows]
if body:
out[key] = Batch.load(self.http.post(f"/v1/research/{key}", {key: body}))
return out
def _list(self, kind, **params):
q = urllib.parse.urlencode({k: v for k, v in params.items() if v is not None})
return self._req("GET", "/v1/research/" + kind + ("?" + q if q else "")).get("data") or []
def revise(self, record: R, notes: str | None = None) -> Batch:
"""File a correction: a new version of the record under its id, marked corrected."""
record.revision = "corrected"
if notes is not None and isinstance(record, Run):
record.notes = notes
return self._write({_KEYS[type(record)]: [record]})[_KEYS[type(record)]]
def benchmarks(self, id=None, project=None):
"""Benchmark definitions, by id or project."""
return self._list("benchmarks", id=id, project=project)
def retract(self, record: str | R, why: str) -> Batch:
"""Withdraw a record: a version marked retracted. A run id reads its canonical run first.
def runs(self, id=None, benchmark=None, system=None, study=None, completion=None, since=None, until=None, org=None):
"""Runs, filtered as GET /v1/research/runs filters them; `org` reads another
org's public runs."""
return self._list("runs", id=id, benchmark=benchmark, system=system, study=study,
completion=completion, since=since, until=until, org=org)
A retracted latest version takes the id out of every listing; the earlier versions stay
retained. `why` becomes the version's notes for a run (the only kind with notes).
"""
if not why.strip():
raise Invalid("a retraction says why")
rec: Any = self.get_run(record) if isinstance(record, str) else record
rec.revision = "retracted"
if isinstance(rec, Run):
rec.notes = why
return self._write({_KEYS[type(rec)]: [rec]})[_KEYS[type(rec)]]
def studies(self, id=None, project=None):
return self._list("studies", id=id, project=project)
def grant(
self,
id: str | None = None,
*,
sha256: str | None = None,
visibility: str | None = None,
trainable: bool | None = None,
publishable: bool | None = None,
project: str | None = None,
) -> int:
"""Set a run's visibility (private, org, public) and consents, or an artifact's visibility.
def papers(self, id=None, study=None, cites=None):
return self._list("papers", id=id, study=study, cites=cites)
Lands on every retained version of the id. `None` leaves a field as it is. Returns how
many records it moved; none is `Missing`.
"""
body: dict[str, Any] = {"project": project or self.project}
if sha256:
body["sha256"] = sha256
else:
body["id"] = id
for k, v in (("visibility", visibility), ("trainable", trainable), ("publishable", publishable)):
if v is not None:
body[k] = v
return int(self.http.post("/v1/research/grants", body).get("updated", 0))
def compare(self, a, b):
"""Two runs side by side, with what stops the comparison."""
return self._req("GET", "/v1/research/compare?" + urllib.parse.urlencode({"a": a, "b": b}))
# ── record plane: reads ──────────────────────────────────────────────────────
# ── internal ────────────────────────────────────────────────────────────────
def _last_run_sha(self, exp_id):
"""The git sha of this experiment's last recorded run, for the since-narrative."""
def benchmarks(self, id: str | None = None, project: str | None = None) -> list[Benchmark]:
"""Canonical benchmark definitions of the org (`project` narrows; empty reads every project)."""
out = self.http.get("/v1/research/benchmarks", {"id": id, "project": project})
return [Benchmark.load(x) for x in out.get("data") or []]
def get_benchmark(self, id: str) -> Benchmark:
return _one(self.benchmarks(id=id), "benchmark", id)
def runs(
self,
id: str | None = None,
benchmark: str | None = None,
system: str | None = None,
study: str | None = None,
completion: str | None = None,
since: str | None = None,
until: str | None = None,
project: str | None = None,
org: str | None = None,
) -> Listing:
"""Canonical runs, with the completion census and the fields runs leave unstated.
Without `org`, the caller's own org, private runs included. With `org`, that org's
PUBLIC runs, read with no credential: the server answers `?org=` only to a caller
that names no org of its own. `since`/`until` bound `when` (RFC 3339).
"""
q = {
"id": id,
"benchmark": benchmark,
"system": system,
"study": study,
"completion": completion,
"since": since,
"until": until,
"project": project,
"org": org,
}
out = Listing.load(self.http.get("/v1/research/runs", q, auth=not org))
out.data = [Run.load(x) for x in out.data]
return out
def get_run(self, id: str, org: str | None = None) -> Run:
"""One run's canonical version, or `Missing`."""
return _one(self.runs(id=id, org=org).data, "run", id)
def studies(self, id: str | None = None, project: str | None = None) -> list[Study]:
out = self.http.get("/v1/research/studies", {"id": id, "project": project})
return [Study.load(x) for x in out.get("data") or []]
def papers(
self, id: str | None = None, study: str | None = None, cites: str | None = None, project: str | None = None
) -> list[Paper]:
out = self.http.get("/v1/research/papers", {"id": id, "study": study, "cites": cites, "project": project})
return [Paper.load(x) for x in out.get("data") or []]
def compare(self, a: str, b: str, project: str | None = None) -> Comparison:
"""Two canonical runs side by side: what blocks the comparison, the config they differ in, the deltas."""
return Comparison.load(self.http.get("/v1/research/compare", {"a": a, "b": b, "project": project}))
# ── experiment plane ─────────────────────────────────────────────────────────
def ingest(
self, experiments: list[Mapping[str, Any]] | None = None, attempts: list[Mapping[str, Any]] | None = None
) -> Ack:
"""File experiment and attempt versions in one batch, idempotent by content."""
return Ack.load(
self.http.post("/v1/research/experiments", {"experiments": experiments or [], "attempts": attempts or []})
)
def query(self, project: str | None = None, kind: str | None = None) -> list[Record]:
"""Canonical experiments of a project (default the client's), narrowed to one kind."""
out = self.http.get("/v1/research/experiments", {"project": project or self.project, "kind": kind})
return [Record.load(x) for x in out.get("data") or []]
def totals(self, project: str | None = None) -> Totals:
return Totals.load(self.http.get("/v1/research/totals", {"project": project}))
def projects(self) -> list[Summary]:
return [Summary.load(x) for x in self.http.get("/v1/research/projects").get("data") or []]
def artifact(self, art: Artifact | Mapping[str, Any]) -> Stored:
"""File an artifact record whose `content` is base64; the server hashes the bytes."""
return Stored.load(self.http.post("/v1/research/artifacts", dump(art)))
def upload_artifact(
self,
kind: str,
run_id: str,
data: bytes,
media_type: str = "application/octet-stream",
canonicalization: str = "raw",
compression: str = "none",
artifact_id: str | None = None,
) -> Stored:
"""File bytes as a `snapshot` or `report` of a run, stamped with this checkout's git state."""
sha = hashlib.sha256(data).hexdigest()
g = self._state()
return self.artifact(
Artifact(
sha256=sha,
content=base64.b64encode(data).decode(),
kind=kind,
run_id=run_id,
git_sha=g["git_sha"],
git_branch=g["git_branch"],
git_dirty=g["git_dirty"],
lib_versions=provenance.lib_versions(self.libs),
artifact_id=artifact_id or f"{run_id}:{kind}",
content_sha256=sha,
media_type=media_type,
canonicalization=canonicalization,
compression=compression,
uncompressed_size=len(data),
)
)
def upload_json_artifact(self, kind: str, run_id: str, value: Any) -> Stored:
"""File a value as RFC 8785 JSON; its SHA-256 is `canonical.digest(value)`."""
return self.upload_artifact(kind, run_id, canonical.canonicalize(value), "application/json", "rfc8785")
def artifacts(self, run: str | None = None, since: int | None = None, project: str | None = None) -> list[Artifact]:
"""The diary feed, newest first, as metadata; bytes are read by `get_artifact`."""
out = self.http.get("/v1/research/artifacts", {"run": run, "since": since, "project": project})
return [Artifact.load(x) for x in out.get("data") or []]
def get_artifact(self, sha256: str, project: str | None = None) -> bytes:
"""An artifact's bytes by content hash, within the caller's org and project."""
from urllib.parse import quote
return self.http.call(
"GET", f"/v1/research/artifacts/{quote(sha256, safe='')}", {"project": project}, None, True
)
def experiment(
self,
kind: str,
subject: str,
task: str,
metric: str = "accuracy",
n_total: int = 0,
note: str = "",
hypothesis: str = "",
predict: str = "",
) -> Experiment:
"""Get or create the experiment `kind:subject:task` and post it in flight (see `Experiment`)."""
return Experiment(
self, kind, subject, task, metric=metric, n_total=n_total, note=note, hypothesis=hypothesis, predict=predict
)
def _state(self) -> dict[str, Any]:
if self._git is None:
self._git = provenance.git_state(self.repo)
return self._git
def _last_run_sha(self, exp_id: str) -> str:
try:
for e in self.query():
if e.get("id") == exp_id:
return e.get("git_sha", "")
return next((e.git_sha for e in self.query() if e.id == exp_id), "")
except Exception:
return ""
return ""
def _one(rows: list[Any], what: str, id: str) -> Any:
if not rows:
raise Missing(f"no canonical {what} {id!r}", 404)
return rows[0]
class Experiment:
"""A handle to one experiment (a run of kind:subject:task). record() files attempts;
snapshot()/report() file diary artifacts; finish() seals the run with its final
number + the auto-captured narrative. Usable as a context manager (finish on exit)."""
"""One experiment `kind:subject:task` on the experiment plane, structured as a falsifiable test.
def __init__(self, client, kind, subject, task, metric="accuracy", n_total=0, note="",
hypothesis="", predict=""):
Posts itself in flight on creation; `record` files attempts, `snapshot`/`report` file
artifacts, `log` keeps a running trail, `conclude`/`finish` seal it. The git state, the
commit subjects since its last recorded run, the installed versions and the host are
stamped without being asked for. A context manager seals it complete, or faulted on an
exception.
"""
def __init__(
self,
client: Research,
kind: str,
subject: str,
task: str,
metric: str = "accuracy",
n_total: int = 0,
note: str = "",
hypothesis: str = "",
predict: str = "",
):
self.c = client
self.kind, self.subject, self.task, self.metric = kind, subject, task, metric
self.id = f"{kind}:{subject}:{task}"
self.n_total = n_total
self._ok = 0
self._n = 0
# The falsifiable frame, stated up front so the verdict is unambiguous: the claim
# under test, and the observation that would confirm it.
self.hypothesis = hypothesis
self.predict = predict
self.verdict = "" # proven | refuted | inconclusive — set by conclude()
self.because = "" # the reasoning that earns the verdict
self._log = [] # the running notes/logs that travel with the run
# Zero-config, self-documenting provenance. THREE narrative sources, captured with
# nothing asked of the caller: (a) the calling code's docstring — what this
# experiment IS; (b) the commit messages since this experiment's last recorded run
# — what changed + why; (c) an optional note. Plus git state, lib versions, host.
self.prov = provenance.git_state(self.c.repo)
self.verdict = ""
self.because = ""
self._log: list[str] = []
self.prov = client._state()
self.doc = provenance.caller_doc()
self.note = note
last = self.c._last_run_sha(self.id)
self.commits = provenance.commit_narrative(self.c.repo, since_sha=last)
self.libs = provenance.lib_versions(self.c.libs)
self.commits = provenance.commit_narrative(client.repo, since_sha=client._last_run_sha(self.id))
self.libs = provenance.lib_versions(client.libs)
self.host = provenance.host()
# Post the run as in-flight so the ops board sees it immediately.
self._post(status="running", value=0.0)
def log(self, text):
"""Append to the running log — the 'what I saw / thought' trail that travels with
the run into the record. Chainable. (`note=` on experiment() is the one-shot note;
log() accumulates the ongoing entries.)"""
def log(self, text: str) -> Experiment:
"""Append to the running log that travels with the run."""
self._log.append(text)
return self
def conclude(self, verdict, because="", value=None):
"""Seal the experiment with its epistemic VERDICT ∈ {proven, refuted, inconclusive}
(distinct from execution status) and the reasoning that earns it, then finish the
run. A refutation is recorded as clearly and durably as a proof — the whole point of
an evidentiary layer. Returns the sealed record."""
if verdict not in ("proven", "refuted", "inconclusive"):
def conclude(self, verdict: str, because: str = "", value: float | None = None) -> Ack:
"""Seal with a verdict (proven, refuted or inconclusive) and the reasoning that earns it."""
if verdict not in VERDICTS:
raise ValueError(f"verdict must be proven|refuted|inconclusive, got {verdict!r}")
self.verdict = verdict
self.because = because
return self.finish(value=value)
def _post(self, status, value):
e = {
"id": self.id, "kind": self.kind, "subject": self.subject, "task": self.task,
"metric": self.metric, "value": value, "n": self._n, "n_total": self.n_total,
"status": status, "git_sha": self.prov["git_sha"], "git_branch": self.prov["git_branch"],
"git_dirty": self.prov["git_dirty"], "lib_versions": self.libs,
# meta carries the scientific frame + the self-documenting narrative. hypothesis/
# predict/verdict/because make the run a falsifiable claim with a clear result;
# verdict is queryable evidence (e.g. every refuted lever) — promoting it to a
# first-class column is the follow-up, the shape is fixed here.
"meta": {"doc": self.doc, "commits": self.commits, "note": self.note, "host": self.host,
"hypothesis": self.hypothesis, "predict": self.predict,
"verdict": self.verdict, "because": self.because, "log": self._log},
}
return self.c.ingest(experiments=[e])
def record(self, item, model, result):
"""File one attempt (idempotent by stable id). `result` is a dict:
{answer, correct, response?, gold?, source?, status?}."""
def record(self, item: str, model: str, result: Mapping[str, Any]) -> Ack:
"""File one attempt: `result` holds answer, correct, and optionally response, gold, source, status.
A `faulted` or `failed` attempt is kept and left out of the score."""
a = {
"benchmark": self.task, "item": item, "model": model,
"answer": result.get("answer", ""), "correct": bool(result.get("correct")),
"response": result.get("response", ""), "gold": result.get("gold", ""),
"source": result.get("source", "hanzo-measured"), "status": result.get("status", "complete"),
"benchmark": self.task,
"item": item,
"model": model,
"answer": result.get("answer", ""),
"correct": bool(result.get("correct")),
"response": result.get("response", ""),
"gold": result.get("gold", ""),
"source": result.get("source", "hanzo-measured"),
"status": result.get("status", "complete"),
}
if a["status"] not in ("faulted", "failed"):
self._n += 1
if a["correct"]:
self._ok += 1
self._ok += int(a["correct"])
return self.c.ingest(attempts=[a])
def snapshot(self, data, run_id=None):
"""File a board-snapshot artifact. `data` is the PNG bytes (or a path to the PNG).
The SDK submits the bytes; the SERVER content-addresses them by sha256."""
raw = data if isinstance(data, (bytes, bytearray)) else _read_bytes(data)
return self._artifact("snapshot", raw, run_id)
def snapshot(self, data: bytes | str, run_id: str | None = None) -> Stored:
"""File PNG bytes (or a path to them) as a snapshot of this run."""
raw = data if isinstance(data, (bytes, bytearray)) else open(data, "rb").read()
return self.c.upload_artifact(
"snapshot", run_id or self.id, bytes(raw), "image/png", artifact_id=f"{self.id}:snapshot"
)
def report(self, data, run_id=None):
"""File a generated-report artifact — HTML/Markdown TEXT (or bytes)."""
def report(self, data: bytes | str, run_id: str | None = None) -> Stored:
"""File HTML or Markdown as a report of this run."""
raw = data if isinstance(data, (bytes, bytearray)) else str(data).encode()
return self._artifact("report", raw, run_id)
return self.c.upload_artifact(
"report", run_id or self.id, bytes(raw), "text/plain", artifact_id=f"{self.id}:report"
)
def _artifact(self, kind, raw, run_id):
# Submit the BYTES; the server hashes them and owns the sha256 + ref. The client
# sha256 travels only so the server can reject a mismatch (a mis-hash or tamper).
return self.c.artifact({
"content": base64.b64encode(raw).decode(),
"sha256": hashlib.sha256(raw).hexdigest(),
"kind": kind, "run_id": run_id or self.id,
"git_sha": self.prov["git_sha"], "git_branch": self.prov["git_branch"],
"git_dirty": self.prov["git_dirty"], "lib_versions": self.libs,
})
def finish(self, value=None):
"""Seal the run: post the final number (computed from recorded attempts if not
given) + status complete. A new version supersedes the in-flight one; the prior
is retained. If a hypothesis was stated but no verdict declared, it defaults to
`inconclusive` — a finished run never silently reads as a proof."""
def finish(self, value: float | None = None) -> Ack:
"""Seal complete with the final number (from the recorded attempts when not given). A stated
hypothesis with no verdict seals inconclusive: a finished run never reads as a proof by default."""
if value is None:
value = round(100.0 * self._ok / self._n, 2) if self._n else 0.0
if self.hypothesis and not self.verdict:
self.verdict = "inconclusive"
return self._post(status="complete", value=value)
def __enter__(self):
def _post(self, status: str, value: float) -> Ack:
e = {
"id": self.id,
"kind": self.kind,
"subject": self.subject,
"task": self.task,
"metric": self.metric,
"value": value,
"n": self._n,
"n_total": self.n_total,
"status": status,
"git_sha": self.prov["git_sha"],
"git_branch": self.prov["git_branch"],
"git_dirty": self.prov["git_dirty"],
"lib_versions": self.libs,
"meta": {
"doc": self.doc,
"commits": self.commits,
"note": self.note,
"host": self.host,
"hypothesis": self.hypothesis,
"predict": self.predict,
"verdict": self.verdict,
"because": self.because,
"log": self._log,
},
}
return self.c.ingest(experiments=[e])
def __enter__(self) -> Experiment:
return self
def __exit__(self, exc_type, exc, tb):
# A run that raised is sealed faulted (retained), else complete.
def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None:
if exc_type is not None:
self._post(status="faulted", value=0.0)
else:
self.finish()
return False
def _read_bytes(path):
with open(path, "rb") as f:
return f.read()
pkg/hanzo-research/hanzo_research/errors.py
+78
View File
@@ -0,0 +1,78 @@
"""Errors a call raises, one class a kind of refusal, each carrying the server's own sentence."""
from __future__ import annotations
import json
from typing import Any
__all__ = ["Error", "Denied", "Missing", "Invalid", "Limited", "Unavailable", "refusal"]
class Error(Exception):
"""A call that did not succeed: the HTTP status (0 when no answer came), what the server said, and where."""
def __init__(self, message: str, status: int = 0, method: str = "", path: str = "", body: Any = None):
super().__init__(f"{method} {path}: {status}: {message}" if method else message)
self.message = message
self.status = status
self.method = method
self.path = path
self.body = body
class Denied(Error):
"""401 or 403: no credential, an expired one, or a caller this org or route does not admit."""
class Missing(Error):
"""404: no such record. A record another org holds reads the same, by design."""
class Invalid(Error):
"""400, 409 or 422: the request does not fit the route; the message names the field."""
class Limited(Error):
"""413 or 429: too large, or too many; `retry` is the server's wait in seconds when it gave one."""
retry: float | None = None
class Unavailable(Error):
"""502, 503, 504, or no answer at all: retry later."""
def refusal(status: int, raw: bytes, method: str, path: str, retry: float | None = None) -> Error:
"""The typed error for a non-2xx answer, its message read from whichever envelope the server used."""
body: Any = None
message = ""
try:
body = json.loads(raw)
except ValueError:
message = raw[:1024].decode("utf-8", "replace").strip()
if isinstance(body, dict):
e = body.get("error")
if isinstance(e, dict):
message = str(e.get("message") or e.get("code") or "")
elif e:
message = str(e)
# typed ops answer RFC 7807 ({detail, title}); the blob route and the gateway {error}
message = message or str(
body.get("detail") or body.get("message") or body.get("msg") or body.get("title") or ""
)
message = message or f"HTTP {status}"
cls: type[Error] = Error
if status in (401, 403):
cls = Denied
elif status == 404:
cls = Missing
elif status in (400, 409, 422):
cls = Invalid
elif status in (413, 429):
cls = Limited
elif status in (502, 503, 504):
cls = Unavailable
err = cls(message, status, method, path, body)
if isinstance(err, Limited):
err.retry = retry
return err
pkg/hanzo-research/hanzo_research/lineage.py
+258
View File
@@ -0,0 +1,258 @@
"""Lineage, contamination guards and evidence records: the crate's `lineage.rs`, field for field.
A child's lineage only ever narrows: its purpose is the most restrictive of its own and its
parents', and derivation is allowed only when every parent allows it and the purpose is not
an evaluation one. A record's `digest()` is the SHA-256 of its RFC 8785 bytes, the same
bytes the crate hashes, so a Rust harness and a Python one agree on every address.
"""
from __future__ import annotations
import enum
from typing import Any, Iterable
from dataclasses import field, dataclass
from . import canonical
from .wire import Wire
__all__ = [
"DataPurpose",
"LineagePolicy",
"ContaminationError",
"ArtifactRef",
"EnvironmentV1",
"DeterminismReport",
"TeacherInteraction",
"PairEvidence",
"GateSpec",
"Claim",
"ResearchEvent",
"derive_lineage_policy",
"verify_promotion_eligibility",
]
class DataPurpose(enum.Enum):
"""What an item may be used for, least to most restrictive."""
TRAIN = "train"
TEACHER_MINING = "teacher_mining"
DEV = "dev"
EVAL_ONLY = "eval_only"
SEALED_EVAL = "sealed_eval"
@property
def rank(self) -> int:
return list(DataPurpose).index(self)
@staticmethod
def most_restrictive(a: DataPurpose, b: DataPurpose) -> DataPurpose:
return a if a.rank >= b.rank else b
def is_training_allowed(self) -> bool:
return self in (DataPurpose.TRAIN, DataPurpose.TEACHER_MINING)
def is_eval(self) -> bool:
return self in (DataPurpose.DEV, DataPurpose.EVAL_ONLY, DataPurpose.SEALED_EVAL)
_SEALED = (DataPurpose.EVAL_ONLY, DataPurpose.SEALED_EVAL)
class ContaminationError(Exception):
"""STAGE_REJECTED_CONTAMINATION: an artifact whose lineage forbids training reached it."""
def __init__(self, kind: str, artifact_id: str = "", purpose: DataPurpose | None = None, message: str = ""):
self.kind, self.artifact_id, self.purpose, self.message = kind, artifact_id, purpose, message
if kind == "contaminated_purpose" and purpose is not None:
text = f"artifact {artifact_id} has purpose {purpose.name.title().replace('_', '')} (forbidden in training)"
elif kind == "derivation_disallowed":
text = f"artifact {artifact_id} has derivation_allowed = false"
else:
text = f"contamination detected in stage: {message}"
super().__init__(f"STAGE_REJECTED_CONTAMINATION: {text}")
@dataclass
class LineagePolicy(Wire):
"""The purpose an artifact carries and whether anything may be derived from it."""
purpose: DataPurpose = DataPurpose.TRAIN
derivation_allowed: bool = False
@classmethod
def root(cls, purpose: DataPurpose, derivation_allowed: bool) -> LineagePolicy:
"""A source artifact's policy; an evaluation purpose never allows derivation."""
return cls(purpose, False if purpose in _SEALED else derivation_allowed)
@classmethod
def derive(cls, purpose: DataPurpose, derivation_allowed: bool, parents: Iterable[LineagePolicy]) -> LineagePolicy:
"""A child's policy: never less restrictive than any parent."""
ps = list(parents)
top = purpose
for p in ps:
top = DataPurpose.most_restrictive(top, p.purpose)
allowed = derivation_allowed and all(p.derivation_allowed for p in ps) and top not in _SEALED
return cls(top, allowed)
def assert_trainable(self, context: str) -> None:
"""Raise ContaminationError unless this policy permits training."""
if not self.purpose.is_training_allowed():
raise ContaminationError("contaminated_purpose", context, self.purpose)
if not self.derivation_allowed:
raise ContaminationError("derivation_disallowed", context)
def digest(self) -> str:
return canonical.digest(self)
def derive_lineage_policy(
purpose: DataPurpose, derivation_allowed: bool, parents: Iterable[LineagePolicy]
) -> LineagePolicy:
"""`LineagePolicy.derive`, as the crate names it."""
return LineagePolicy.derive(purpose, derivation_allowed, parents)
def verify_promotion_eligibility(artifact_id: str, policy: LineagePolicy, ancestors: Iterable[LineagePolicy]) -> None:
"""Raise unless the artifact and every transitive ancestor allow derivation and training."""
if not policy.derivation_allowed:
raise ContaminationError("derivation_disallowed", artifact_id)
if not policy.purpose.is_training_allowed():
raise ContaminationError("contaminated_purpose", artifact_id, policy.purpose)
for i, a in enumerate(ancestors):
where = f"{artifact_id}->transitive_ancestor[{i}]"
if not a.derivation_allowed:
raise ContaminationError("derivation_disallowed", where)
if not a.purpose.is_training_allowed():
raise ContaminationError("contaminated_purpose", where, a.purpose)
class _Digest(Wire):
def digest(self) -> str:
"""Lowercase hex SHA-256 of the record's RFC 8785 bytes."""
return canonical.digest(self)
@dataclass
class ArtifactRef(_Digest):
"""An immutable reference to an artifact at `/v1/research/artifacts/<sha256>`."""
artifact_id: str = ""
sha256: str = ""
content_sha256: str = ""
media_type: str = "application/octet-stream"
canonicalization: str = "raw"
compression: str = "none"
uncompressed_size: int = 0
ref_uri: str = ""
lineage: LineagePolicy = field(default_factory=LineagePolicy)
@classmethod
def new(cls, sha256: str, lineage: LineagePolicy) -> ArtifactRef:
return cls(f"artifact:{sha256}", sha256, sha256, ref_uri=f"sha256:{sha256}", lineage=lineage)
@dataclass
class EnvironmentV1(_Digest):
"""The reproducibility state of the host a result came from."""
git_sha: str = ""
git_branch: str = ""
git_dirty: bool = False
host: str = ""
os: str = ""
kernel_version: str = ""
cargo_lock_hash: str = ""
image_digest: str = ""
precision_config: str = ""
determinism_config: str = ""
compiler_version: str = ""
lib_versions: dict[str, str] = field(default_factory=dict)
@dataclass
class DeterminismReport(_Digest):
"""A stepped determinism probe's result."""
steps_compared: list[int] = field(default_factory=list)
loss_exact_match: bool = False
gradient_norm_exact_match: bool = False
max_weight_divergence: float = 0.0
passed: bool = False
probe_duration_secs: float = 0.0
notes: str = ""
@dataclass
class TeacherInteraction(_Digest):
"""One call harvested from a teacher model; money in integer nanodollars."""
model: str = ""
input_prompt_hash: str = ""
output_text: str = ""
tokens_prompt: int = 0
tokens_completion: int = 0
cost_nanodollars: int = 0
latency_ms: int = 0
lineage: LineagePolicy = field(default_factory=LineagePolicy)
when: str = ""
@dataclass
class PairEvidence(_Digest):
"""A pairwise preference, chosen over rejected, by a margin."""
prompt_hash: str = ""
chosen_hash: str = ""
rejected_hash: str = ""
margin: float = 0.0
cost_nanodollars: int = 0
lineage: LineagePolicy = field(default_factory=LineagePolicy)
@dataclass
class GateSpec(_Digest):
"""A preregistered, immutable promotion gate."""
gate_id: str = ""
candidate_discovery_recall_floor: float = 0.0
reranking_top1_min: float = 0.0
reranking_mrr_min: float = 0.0
conditional_reranking_top1_min: float = 0.0
allow_loss_regression_pct: float = 0.0
notes: str = ""
@dataclass
class Claim(_Digest):
"""A formal claim about a benchmark result, bound to the gate it was judged by."""
id: str = ""
run_id: str = ""
benchmark_id: str = ""
metric: str = ""
claimed_value: float = 0.0
verified: bool = False
gate_sha256: str = ""
lineage: LineagePolicy = field(default_factory=LineagePolicy)
created_at: str = ""
@dataclass
class ResearchEvent(_Digest):
"""A WAL entry: ordered by `sequence`, chained by `previous_event_hash`, its payload hashed."""
run_id: str = ""
worker_id: str = "worker-0"
event_id: str = ""
sequence: int = 0
previous_event_hash: str = ""
payload_hash: str = ""
payload: Any = None
created_at: str = ""
@classmethod
def new(
cls, run_id: str, worker_id: str, event_id: str, sequence: int, previous: str, payload: Any, created_at: str
) -> ResearchEvent:
return cls(run_id, worker_id, event_id, sequence, previous, canonical.digest(payload), payload, created_at)
pkg/hanzo-research/hanzo_research/provenance.py
+318 -33
View File
@@ -1,23 +1,66 @@
"""Zero-config provenance auto-capture — like OpenTelemetry auto-instrument, for
experiments. The caller supplies nothing; the SDK reads the run's environment: git
sha/branch/dirty, the recent COMMIT MESSAGES (the narrative of what changed), the
installed lib versions, and the host. Import and go.
"""What a filed measurement must carry to be believed or repeated: the harness commit, the model
and its weights, the machine, the device, the software, the date.
`capture()` reads all of it from the environment; `Provenance.apply(run)` folds it into a
run: `commit`, `system`, `version` (`<model>@<weights sha256>`), `by`, and a sentence in
`notes` ending in the SHA-256 of the whole record (RFC 8785), so the record can be filed
beside the run as an artifact and checked against it. Best-effort throughout: a missing
tool or a directory outside git yields an empty field, never an error.
The functions the experiment handle uses (`git_state`, `commit_narrative`, `lib_versions`,
`host`, `find_repo`, `caller_doc`) are the crate's `provenance.rs`, record for record.
"""
import inspect
from __future__ import annotations
import os
import re
import sys
import socket
import hashlib
import inspect
import platform
import subprocess
import dataclasses
from typing import Any
from importlib import metadata
from dataclasses import field, dataclass
from . import canonical
from .wire import Run, now
__all__ = [
"Provenance",
"capture",
"digest_file",
"digest_tree",
"weights",
"git_state",
"commit_narrative",
"lib_versions",
"host",
"find_repo",
"caller_doc",
]
PACKAGES = ("hanzo-research", "numpy", "torch", "transformers", "datasets", "vllm", "jax", "hanzo-kai")
_HEX = re.compile(r"^[0-9a-fA-F]{1,64}$")
_SHA = re.compile(r"^(sha256:)?([0-9a-f]{64})$")
def _git(repo, *args):
def _run(*args: str, cwd: str | None = None) -> str:
try:
return subprocess.check_output(["git", "-C", repo, *args], stderr=subprocess.DEVNULL, text=True).strip()
except Exception:
out = subprocess.run(args, capture_output=True, text=True, timeout=30, cwd=cwd)
except (OSError, subprocess.SubprocessError):
return ""
return out.stdout.strip() if out.returncode == 0 else ""
def git_state(repo):
def _git(repo: str, *args: str) -> str:
return _run("git", "-C", repo, *args)
def git_state(repo: str) -> dict[str, Any]:
"""The producing repo's commit sha, branch, and whether the tree was dirty."""
return {
"git_sha": _git(repo, "rev-parse", "HEAD"),
@@ -26,45 +69,287 @@ def git_state(repo):
}
def commit_narrative(repo, since_sha="", window=10):
"""The commit-message narrative — the 'what changed + why' story. Commit subjects
SINCE the last recorded run's sha when known (`<since>..HEAD`), else the last
`window` commits. Research self-documents as a side effect of running."""
rng = f"{since_sha}..HEAD" if since_sha else f"-n{window}"
raw = _git(repo, "log", rng, "--format=%s") if since_sha else _git(repo, "log", f"-{window}", "--format=%s")
return [ln for ln in raw.splitlines() if ln.strip()]
def commit_narrative(repo: str, since_sha: str = "", window: int = 10) -> list[str]:
"""Commit subjects since the last recorded run's sha (`<since>..HEAD`), else the last `window`.
`since_sha` can come from a server answer, so only a git object id (hex) is used; anything
else, which could read as a flag or a path, falls back to the window.
"""
if since_sha and _HEX.match(since_sha):
raw = _git(repo, "log", f"{since_sha}..HEAD", "--format=%s")
else:
raw = _git(repo, "log", f"-{int(window)}", "--format=%s")
return [ln.strip() for ln in raw.splitlines() if ln.strip()]
def lib_versions(names):
"""{lib: installed version} for the relevant libs, skipping any not installed."""
def lib_versions(names: Any) -> dict[str, str]:
"""{distribution: installed version} for the named ones that are installed."""
out = {}
for n in names or []:
try:
out[n] = metadata.version(n)
except Exception:
except metadata.PackageNotFoundError:
pass
return out
def host():
"""The box the run executed on — enough to correlate a result to hardware."""
return {"hostname": socket.gethostname(), "platform": os.uname().sysname if hasattr(os, "uname") else os.name}
def host() -> dict[str, str]:
"""The box the run executed on: hostname and OS name."""
return {"hostname": socket.gethostname(), "platform": platform.system() or os.name}
def find_repo(start=None):
"""The git repo the caller runs in (the toplevel), auto-detected from the CWD."""
def find_repo(start: str | None = None) -> str:
"""The git toplevel the caller runs in, else the directory itself."""
start = start or os.getcwd()
top = _git(start, "rev-parse", "--show-toplevel")
return top or start
return _git(start, "rev-parse", "--show-toplevel") or start
def caller_doc():
"""The docstring of the first caller frame OUTSIDE this SDK — 'what this experiment
is', captured zero-config from the researcher's normal module docstring. Together
with the commit narrative and an optional note, the project self-documents as a side
effect of running: write a normal docstring + commit normally, nothing extra."""
def caller_doc() -> str:
"""The module docstring of the first frame outside this package: what the experiment is."""
for fr in inspect.stack():
mod = fr.frame.f_globals.get("__name__", "")
if not mod.startswith("hanzo_research"):
if not str(fr.frame.f_globals.get("__name__", "")).startswith("hanzo_research"):
return (fr.frame.f_globals.get("__doc__") or "").strip()
return ""
# ── digests ─────────────────────────────────────────────────────────────────────
def digest_file(path: str) -> str:
"""`sha256:<hex>` of a file's bytes, read in 1 MiB blocks."""
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return "sha256:" + h.hexdigest()
def digest_tree(path: str) -> str:
"""`sha256:<hex>` of a directory: of its SHA256SUMS when it holds one, else of the
`sha256sum` listing of every file under it, sorted by path, written as that file would be.
"""
sums = os.path.join(path, "SHA256SUMS")
if os.path.isfile(sums):
return digest_file(sums)
lines = []
for root, dirs, files in os.walk(path):
dirs.sort()
for name in sorted(files):
full = os.path.join(root, name)
rel = os.path.relpath(full, path).replace(os.sep, "/")
lines.append(f"{digest_file(full)[7:]} {rel}\n")
return "sha256:" + hashlib.sha256("".join(sorted(lines, key=lambda s: s[66:])).encode()).hexdigest()
def weights(value: str) -> str:
"""Weights as `sha256:<hex>`: a file or a directory hashed, or a digest given as one."""
if os.path.isdir(value):
return digest_tree(value)
if os.path.isfile(value):
return digest_file(value)
m = _SHA.match(value.strip().lower())
if not m:
raise ValueError(f"weights {value!r} is neither a path nor a sha256")
return "sha256:" + m.group(2)
# ── the record ──────────────────────────────────────────────────────────────────
@dataclass
class Provenance:
"""Where a number came from: harness commit, model and weights, machine, device, software, date."""
repo: str = ""
remote: str = ""
commit: str = ""
branch: str = ""
dirty: bool = False
diff: str = ""
model: str = ""
weights: str = ""
host: str = ""
os: str = ""
machine: str = ""
cpu: str = ""
cores: int = 0
gpu: str = ""
device: str = ""
dtype: str = ""
python: str = ""
packages: dict[str, str] = field(default_factory=dict)
date: str = ""
def dump(self) -> dict[str, Any]:
return dataclasses.asdict(self)
def digest(self) -> str:
"""`sha256:<hex>` of the record's RFC 8785 bytes: the address of the record filed as an artifact."""
return "sha256:" + canonical.digest(self.dump())
def note(self) -> str:
"""The record as one sentence for a run's notes, ending in its digest."""
parts = []
if self.commit:
where = self.remote or self.repo
tree = f", dirty (diff {self.diff})" if self.dirty else ""
parts.append(f"harness {where} at {self.commit}{tree}")
if self.model:
parts.append(f"model {self.model}" + (f", weights {self.weights}" if self.weights else ""))
box = ", ".join(
x
for x in (
self.os,
self.machine,
self.cpu and f"CPU {self.cpu}",
self.cores and f"{self.cores} cores",
self.gpu and f"GPU {self.gpu}",
)
if x
)
if self.host:
parts.append(f"host {self.host}" + (f" ({box})" if box else ""))
if self.device or self.dtype:
parts.append(
" ".join(
x
for x in (
f"device {self.device}" if self.device else "",
f"dtype {self.dtype}" if self.dtype else "",
)
if x
)
)
soft = ", ".join(f"{k} {v}" for k, v in sorted(self.packages.items()))
parts.append(f"python {self.python}" + (f" with {soft}" if soft else ""))
if self.date:
parts.append(f"captured {self.date}")
return "Provenance: " + "; ".join(parts) + f". Record {self.digest()}."
def apply(self, run: Run | dict[str, Any]) -> Run | dict[str, Any]:
"""Fold the record into a run, filling what the run leaves empty and appending `note()`."""
version = f"{self.model}@{self.weights.removeprefix('sha256:')}" if self.model and self.weights else self.model
fill = {"commit": self.commit, "system": self.model, "version": version, "by": self.host}
get = run.get if isinstance(run, dict) else lambda k, d=None: getattr(run, k, d)
for key, value in fill.items():
if value and not get(key):
_set(run, key, value)
notes = str(get("notes") or "")
_set(run, "notes", (notes.rstrip() + " " if notes.strip() else "") + self.note())
return run
def environment(self) -> Any:
"""The record as the crate's `EnvironmentV1`, whose digest a Rust harness computes alike."""
from .lineage import EnvironmentV1
return EnvironmentV1(
git_sha=self.commit,
git_branch=self.branch,
git_dirty=self.dirty,
host=self.host,
os=self.os,
kernel_version=platform.release(),
cargo_lock_hash="",
image_digest="",
precision_config=self.dtype,
determinism_config="",
compiler_version=f"python {self.python}",
lib_versions=dict(self.packages),
)
def _set(run: Any, key: str, value: Any) -> None:
if isinstance(run, dict):
run[key] = value
else:
setattr(run, key, value)
def capture(
repo: str | None = None,
model: str = "",
weights: str | None = None,
device: str = "",
dtype: str = "",
packages: Any = None,
gpu: bool = True,
) -> Provenance:
"""Read the provenance of a measurement made now.
`repo` is the harness checkout (default: the git toplevel of the working directory);
`weights` a file, a directory (its SHA256SUMS, else every file) or a sha256 given as a
value; `packages` the distributions whose versions pin the result (default: the
common ML stack, those installed). A dirty tree is recorded with the SHA-256 of
`git diff HEAD`, so the uncommitted state the number came from is pinned too.
"""
top = find_repo(repo) if repo is None or os.path.isdir(repo) else repo
state = git_state(top)
diff = ""
if state["git_dirty"]:
listing = _git(top, "diff", "HEAD", "--binary") + "\n" + _git(top, "ls-files", "--others", "--exclude-standard")
diff = "sha256:" + hashlib.sha256(listing.encode()).hexdigest()
names = list(packages) if packages is not None else list(PACKAGES)
return Provenance(
repo=top,
remote=_remote(top),
commit=state["git_sha"],
branch=state["git_branch"],
dirty=state["git_dirty"],
diff=diff,
model=model,
weights=_weights(weights) if weights else "",
host=socket.gethostname(),
os=f"{platform.system()} {platform.release()}".strip(),
machine=platform.machine(),
cpu=_cpu(),
cores=os.cpu_count() or 0,
gpu=_gpu() if gpu else "",
device=device,
dtype=dtype,
python=platform.python_version(),
packages=lib_versions(names),
date=now(),
)
_weights = weights # capture() takes a parameter of that name
def _remote(repo: str) -> str:
url = _git(repo, "remote", "get-url", "origin")
# never a credential: https://user:token@host/... keeps only the host and path
return re.sub(r"^(\w+://)[^@/]+@", r"\1", url)
def _cpu() -> str:
if sys.platform == "darwin":
return _run("sysctl", "-n", "machdep.cpu.brand_string")
try:
with open("/proc/cpuinfo") as f:
text = f.read()
except OSError:
return platform.processor()
for key in ("model name", "Model", "Hardware"):
m = re.search(rf"^{key}\s*:\s*(.+)$", text, re.M)
if m:
return m.group(1).strip()
lscpu = _run("lscpu")
m = re.search(r"^Model name:\s*(.+)$", lscpu, re.M)
return m.group(1).strip() if m else platform.processor()
def _gpu() -> str:
nv = _run("nvidia-smi", "--query-gpu=name,driver_version", "--format=csv,noheader")
if nv:
rows = [r.split(",") for r in nv.splitlines() if r.strip()]
return "; ".join(f"{r[0].strip()} (driver {r[1].strip()})" if len(r) > 1 else r[0].strip() for r in rows)
info = _run("rocminfo")
if info:
names = re.findall(r"Marketing Name:\s*(.+)", info)
gfx = [
n.strip()
for n, kind in zip(names, re.findall(r"Device Type:\s*(\w+)", info), strict=False)
if kind == "GPU"
]
return "; ".join(gfx)
return ""
pkg/hanzo-research/hanzo_research/py.typed
View File
pkg/hanzo-research/hanzo_research/transport.py
+190
View File
@@ -0,0 +1,190 @@
"""One HTTP path to api.hanzo.ai for both clients: the credential, the acting org, retries, refusals.
Stdlib only (urllib), so a zero-dependency harness can import it.
"""
from __future__ import annotations
import os
import json
import time
import urllib.error
import urllib.parse
import urllib.request
from typing import Any, Mapping
from importlib import metadata
from . import wire
from .errors import Error, Denied, Unavailable, refusal
DEFAULT_BASE = "https://api.hanzo.ai"
try:
# one statement of the version: the installed distribution's (pyproject.toml)
VERSION = metadata.version("hanzo-research")
except metadata.PackageNotFoundError:
VERSION = "source"
# A body past this is refused rather than read (the crate's cap is 8 MiB, which the hanzo
# org's run listing, 1.5 MB at 330 runs, passes at about 1,800 runs).
MAX_RESPONSE_BYTES = 64 << 20
# 503 is the research store's failover answer ("retry"): asked again after 2, 4 and 8 s.
WAITS = (2, 4, 8)
ADMIN = "admin"
class _NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl): # type: ignore[no-untyped-def]
raise Error(f"redirect to {newurl} refused", code, req.get_method(), urllib.parse.urlsplit(req.full_url).path)
class Transport:
"""Base URL, credential, acting org and project, shared by `Research` and `Arena`.
`token` None resolves `HANZO_API_KEY`, else `hanzo auth token`, on the first call that
needs one; "" sends no credential. `org` selects the org a call acts in (`X-Org-Id`);
the server honours it only when the token's signed `orgs` names it, so a JWT that does
not is refused here instead of filing under its home org. `project` must be the
token's own (`project` claim, else `default`): the server files under the claim.
"""
def __init__(
self,
base: str | None = None,
token: str | None = None,
org: str | None = None,
project: str | None = None,
timeout: float = 120.0,
):
self.base = (base or os.environ.get("RESEARCH_BASE") or DEFAULT_BASE).rstrip("/")
if urllib.parse.urlsplit(self.base).scheme not in ("http", "https"):
raise ValueError(f"base {self.base!r} is not an http(s) URL")
self.timeout = timeout
self._token = token
self._cli = False
self._org = (org or "").strip() or None
self._project = project
self._checked = False
self._opener = urllib.request.build_opener(_NoRedirect)
# ── identity ────────────────────────────────────────────────────────────────
@property
def token(self) -> str:
if self._token is None:
env = os.environ.get("HANZO_API_KEY", "").strip()
self._token = env or wire.token()
self._cli = not env and bool(self._token)
if not self._checked:
self._check()
return self._token
@property
def project(self) -> str:
"""The project records are filed under: the token's claim, else the one asked for, else `default`."""
tok = self.token
c = wire.claimed(tok) if tok else None
return c or wire.scope(self._project or os.environ.get("RESEARCH_PROJECT", ""))
@property
def org(self) -> str | None:
"""The org calls act in: the one asked for, else the token's home org (`owner`); None when unknown."""
if self._org:
return self._org
c = wire.claims(self.token) if self.token else None
return str(c.get("owner")) if c and c.get("owner") else None
def _check(self) -> None:
tok = self._token or ""
claimed = wire.claimed(tok) if tok else None
if claimed is not None and self._project and wire.scope(self._project) != claimed:
raise Error(
f"the token's project is {claimed}, not {self._project}: records would be filed under {claimed}"
)
c = wire.claims(tok) if tok else None
if c is not None and self._org:
orgs = [str(m.get("org")) for m in c.get("orgs") or [] if isinstance(m, dict)]
home = orgs[0] if orgs else str(c.get("owner") or "")
if home != ADMIN and self._org != home and (self._org == ADMIN or self._org not in orgs):
raise Denied(
f"this token cannot act in org {self._org} (its orgs: {', '.join(orgs) or home}); "
f"the server would act in {home}"
)
self._checked = True
# ── calls ───────────────────────────────────────────────────────────────────
def get(self, path: str, query: Mapping[str, Any] | None = None, auth: bool = True) -> Any:
return json.loads(self.call("GET", path, query, None, auth) or b"null")
def post(self, path: str, body: Any, query: Mapping[str, Any] | None = None) -> Any:
return json.loads(self.call("POST", path, query, body, True) or b"null")
def call(self, method: str, path: str, query: Mapping[str, Any] | None, body: Any, auth: bool) -> bytes:
"""One request; its raw answer on 2xx, else the typed error. 503 is asked again after 2, 4 and 8 s."""
url = self.base + path + _query(query)
data = None if body is None else json.dumps(body, allow_nan=False).encode()
refreshed = False
for wait in (*WAITS, None):
# the base is http(s), checked in __init__
req = urllib.request.Request(url, data=data, method=method, headers=self._headers(auth, data is not None)) # noqa: S310
try:
with self._opener.open(req, timeout=self.timeout) as r:
return _read(r, method, path)
except urllib.error.HTTPError as e:
raw = _read(e, method, path) if e.fp else b""
if e.code == 503 and wait is not None:
time.sleep(wait)
continue
if e.code == 401 and auth and self._cli and not refreshed:
# a CLI token lives about an hour: fetch the current one and ask once more
refreshed = True
self._token = wire.token()
continue
err = refusal(e.code, raw, method, path, _retry(e.headers))
if isinstance(err, Denied) and auth and not self._token:
err.message += " (no credential: set HANZO_API_KEY or run `hanzo auth login`)"
err.args = (f"{method} {path}: {err.status}: {err.message}",)
raise err from None
except Error:
raise
except (urllib.error.URLError, TimeoutError, ConnectionError, OSError) as e:
raise Unavailable(f"no answer: {getattr(e, 'reason', e)}", 0, method, path) from None
raise AssertionError("unreachable")
def _headers(self, auth: bool, body: bool) -> dict[str, str]:
h = {"Accept": "application/json", "User-Agent": f"hanzo-research/{VERSION} (python)"}
if body:
h["Content-Type"] = "application/json"
if auth:
tok = self.token
if tok:
h["Authorization"] = f"Bearer {tok}"
h["X-Project-Id"] = self.project
if self._org:
h["X-Org-Id"] = self._org
return h
def _query(q: Mapping[str, Any] | None) -> str:
pairs = []
for k, v in (q or {}).items():
if v is None or v == "":
continue
if isinstance(v, bool):
v = "true" if v else "false"
pairs.append(f"{urllib.parse.quote(str(k), safe='')}={urllib.parse.quote(str(v), safe='')}")
return "?" + "&".join(pairs) if pairs else ""
def _read(r: Any, method: str, path: str) -> bytes:
raw = r.read(MAX_RESPONSE_BYTES + 1)
if len(raw) > MAX_RESPONSE_BYTES:
raise Error(f"answer past {MAX_RESPONSE_BYTES} bytes refused", getattr(r, "status", 0) or 0, method, path)
return bytes(raw)
def _retry(headers: Any) -> float | None:
v = headers.get("Retry-After") if headers else None
try:
return float(v) if v else None
except ValueError:
return None
pkg/hanzo-research/hanzo_research/wal.py
+91
View File
@@ -0,0 +1,91 @@
"""A write-ahead log of research events: appended, fsynced, hash-chained (the crate's `wal.rs`).
Each line is one `ResearchEvent`; its `previous_event_hash` is the digest of the line before
(64 zeros for the first), so editing, dropping or reordering a line breaks the chain, which
`verify` finds. A log written by the Rust crate reads here, and the reverse.
"""
from __future__ import annotations
import os
import json
from typing import Any, Iterable
from .wire import now
from .lineage import ResearchEvent
__all__ = ["ResearchWal", "GENESIS", "read", "verify", "ordered"]
GENESIS = "0" * 64
class ResearchWal:
"""One worker's log for one run, at `<dir>/wal.jsonl` (`wal-<worker>.jsonl` past worker-0)."""
def __init__(self, path: str, run_id: str, worker_id: str, sequence: int, last: str):
self.path, self.run_id, self.worker_id = path, run_id, worker_id
self.sequence, self.last_event_hash = sequence, last
@classmethod
def open(cls, run_dir: str, run_id: str, worker_id: str = "worker-0") -> ResearchWal:
"""Open or create the log, resuming its sequence and chain from what is on disk."""
os.makedirs(run_dir, exist_ok=True)
name = "wal.jsonl" if worker_id in ("", "worker-0") else f"wal-{worker_id}.jsonl"
path = os.path.join(run_dir, name)
sequence, last = 0, GENESIS
for event in read(path):
sequence, last = event.sequence, event.digest()
return cls(path, run_id, worker_id, sequence, last)
def append(self, event_type: str, payload: Any) -> ResearchEvent:
"""Append one event and fsync it before returning."""
plain = json.loads(json.dumps(payload, allow_nan=False, sort_keys=True))
seq = self.sequence + 1
event = ResearchEvent.new(
self.run_id,
self.worker_id,
f"{self.run_id}-{self.worker_id}-{seq}-{event_type}",
seq,
self.last_event_hash,
plain,
now(),
)
line = json.dumps(event.dump(), ensure_ascii=False, separators=(",", ":"))
with open(self.path, "a", encoding="utf-8") as f:
f.write(line + "\n")
f.flush()
os.fsync(f.fileno())
self.sequence, self.last_event_hash = seq, event.digest()
return event
def read(path: str) -> list[ResearchEvent]:
"""Every event of a log, in file order; a missing file is an empty log, a blank line is skipped."""
if not os.path.exists(path):
return []
with open(path, encoding="utf-8") as f:
return [ResearchEvent.load(json.loads(line)) for line in f if line.strip()]
def verify(path: str) -> str:
"""Check the chain and every payload hash; the last event's digest, or ValueError at the first break."""
last = GENESIS
for i, event in enumerate(read(path), 1):
if event.sequence != i:
raise ValueError(f"{path}: line {i} has sequence {event.sequence}")
if event.previous_event_hash != last:
raise ValueError(f"{path}: line {i} does not chain to the line before it")
if (
ResearchEvent.new(
event.run_id, event.worker_id, event.event_id, event.sequence, last, event.payload, event.created_at
).payload_hash
!= event.payload_hash
):
raise ValueError(f"{path}: line {i}'s payload does not hash to its payload_hash")
last = event.digest()
return last
def ordered(events: Iterable[ResearchEvent]) -> list[ResearchEvent]:
"""Several workers' events in the run's one order: sequence, time, worker, payload hash."""
return sorted(events, key=lambda e: (e.sequence, e.created_at, e.worker_id, e.payload_hash))
pkg/hanzo-research/hanzo_research/wire.py
+529
View File
@@ -0,0 +1,529 @@
"""Wire types of `/v1/research`: the records hanzo-inc/cloud `apps/research` stores and returns.
Field names, order and JSON are the Rust crate's (`wire.rs`, `lib.rs`), which are the
server's (`record.go`, `research.go`). Reads are tolerant as the crate's are: an unknown
key is ignored, a missing one takes its default, and a null list reads as empty. A
pointer the server declares (`k`, `temperature`, `questions`, a measure's `lo`) stays
`None` when absent, never 0: absence is the record declining to say.
"""
from __future__ import annotations
import os
import enum
import json
import math
import types
import base64
import typing
import binascii
import datetime
import subprocess
import dataclasses
from typing import Any, Mapping, TypeVar, Iterator
from dataclasses import field, dataclass
__all__ = [
"Split",
"Benchmark",
"Measure",
"Run",
"Study",
"Paper",
"Difference",
"Delta",
"Comparison",
"Census",
"Gap",
"Listing",
"Batch",
"Stored",
"Artifact",
"Record",
"Ack",
"KindTotal",
"Totals",
"Summary",
"completion",
"load",
"dump",
"token",
"claims",
"claimed",
"left",
"scope",
"now",
]
T = TypeVar("T")
class Wire:
"""A record on the wire: `dump()` is its JSON object, `load(data)` reads one back."""
def dump(self) -> dict[str, Any]:
return {f.name: _out(getattr(self, f.name)) for f in dataclasses.fields(self)} # type: ignore[arg-type]
@classmethod
def load(cls: type[T], data: Mapping[str, Any] | None) -> T:
return load(cls, data)
@dataclass
class Split(Wire):
"""One named division of a benchmark's items, as the benchmark defines it."""
name: str = ""
items: int = 0
definition: str = ""
@dataclass
class Benchmark(Wire):
"""The task: a standardized test, its dataset and terms, its splits, its official metric."""
project: str = ""
id: str = ""
title: str = ""
version: str = ""
dataset: str = ""
license: str = ""
origin: str = ""
splits: list[Split] = field(default_factory=list)
metric: str = ""
metrics: list[str] = field(default_factory=list)
citation: str = ""
notes: str = ""
revision: str = ""
canonical: bool = False
when: str = ""
@dataclass
class Measure(Wire):
"""One reported number: the slice it covers, the quantity, its value and interval."""
category: str = ""
metric: str = ""
value: float = 0.0
lo: float | None = None
hi: float | None = None
n: int | None = None
of: int | None = None
@dataclass
class Run(Wire):
"""One execution: what was measured, by which system, at which commit, over which data."""
project: str = ""
id: str = ""
benchmark: str = ""
split: str = ""
system: str = ""
version: str = ""
baseline: bool = False
study: str = ""
embedder: str = ""
reader: str = ""
k: int | None = None
temperature: float | None = None
max_tokens: int | None = None
prompt: str = ""
prompt_digest: str = ""
dataset_digest: str = ""
store_digest: str = ""
facts_digest: str = ""
commit: str = ""
questions: int | None = None
answered: int | None = None
when: str = ""
ended: str = ""
by: str = ""
completion: str = ""
measures: list[Measure] = field(default_factory=list)
notes: str = ""
revision: str = ""
canonical: bool = False
visibility: str = ""
trainable: bool = False
publishable: bool = False
def put(
self,
category: str,
metric: str,
value: Any,
n: int | None = None,
of: int | None = None,
lo: float | None = None,
hi: float | None = None,
) -> Run:
"""Record a measure when `value` is a finite number; anything else records nothing."""
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
return self
self.measures.append(Measure(category, metric, float(value), lo, hi, n, of))
return self
def measure(self, category: str, metric: str) -> Measure | None:
"""The measure recorded for (category, metric), or None."""
return next((m for m in self.measures if m.category == category and m.metric == metric), None)
def completion(run: Run) -> str:
"""Completion as the server derives it from `questions`, `answered` and `ended`."""
q, a = run.questions, run.answered
if q is None or a is None:
return "unknown"
if a > q:
return "inconsistent"
if a < q:
return "partial"
return "complete" if run.ended.strip() else "unknown"
@dataclass
class Study(Wire):
"""The question a set of runs was run to answer, and the answer it reached."""
project: str = ""
id: str = ""
title: str = ""
question: str = ""
finding: str = ""
revision: str = ""
canonical: bool = False
when: str = ""
@dataclass
class Paper(Wire):
"""Writing that cites runs; the edge points from the paper to the runs only."""
project: str = ""
id: str = ""
title: str = ""
authors: list[str] = field(default_factory=list)
venue: str = ""
doi: str = ""
url: str = ""
status: str = ""
cites: list[str] = field(default_factory=list)
study: str = ""
revision: str = ""
canonical: bool = False
when: str = ""
@dataclass
class Difference(Wire):
"""One configuration field two runs disagree about, both values as text."""
field: str = ""
a: str = ""
b: str = ""
@dataclass
class Delta(Wire):
"""One (category, metric) both runs measured, and the change from the first to the second."""
category: str = ""
metric: str = ""
a: float = 0.0
b: float = 0.0
change: float = 0.0
overlap: bool | None = None
@dataclass
class Comparison(Wire):
"""Two runs side by side; `blocks` are why they may not be compared, read first."""
a: Run | None = None
b: Run | None = None
comparable: bool = False
blocks: list[str] = dataclasses.field(default_factory=list)
config: list[Difference] = dataclasses.field(default_factory=list)
deltas: list[Delta] = dataclasses.field(default_factory=list)
@dataclass
class Census(Wire):
"""How many runs of a listing read complete, partial, inconsistent and unknown."""
complete: int = 0
partial: int = 0
inconsistent: int = 0
unknown: int = 0
@dataclass
class Gap(Wire):
"""A run field, and how many runs of a listing do not state it."""
field: str = ""
runs: int = 0
@dataclass
class Listing(Wire):
"""A listing's rows. A run listing also carries its completion census and its gaps."""
data: list[Any] = dataclasses.field(default_factory=list)
total: int = 0
completion: Census | None = None
gaps: list[Gap] = dataclasses.field(default_factory=list)
def __iter__(self) -> Iterator[Any]:
return iter(self.data)
def __len__(self) -> int:
return len(self.data)
def __getitem__(self, i: int) -> Any:
return self.data[i]
@dataclass
class Batch(Wire):
"""What a record write filed: the project, the versions new to it, the canonical count."""
project: str = ""
recorded: int = 0
canonical: int = 0
@dataclass
class Stored(Wire):
"""An artifact write: the server's hash of the bytes, its address, and whether it was new."""
sha256: str = ""
ref: str = ""
created: bool = False
rolled_up: bool = False
@dataclass
class Artifact(Wire):
"""A diary artifact (a snapshot or a report), content-addressed by the server.
The server reads the first block; the second is the crate's transport metadata,
sent as the crate sends it and ignored by today's server.
"""
sha256: str = ""
content: str = ""
kind: str = ""
ref: str = ""
run_id: str = ""
project: str = ""
visibility: str = ""
retention_class: str = ""
git_sha: str = ""
git_branch: str = ""
git_dirty: bool = False
lib_versions: dict[str, str] = dataclasses.field(default_factory=dict)
ts: int = 0
artifact_id: str = ""
content_sha256: str = ""
media_type: str = ""
canonicalization: str = ""
compression: str = ""
uncompressed_size: int = 0
@dataclass
class Record(Wire):
"""One stored experiment version, as the canonical experiment listing returns it."""
project: str = ""
id: str = ""
revision: str = ""
status: str = ""
canonical: bool = False
visibility: str = ""
trainable: bool = False
publishable: bool = False
kind: str = ""
subject: str = ""
task: str = ""
metric: str = ""
value: float = 0.0
n: int = 0
n_total: int = 0
cost_usd: float = 0.0
meta: Any = None
git_sha: str = ""
git_branch: str = ""
git_dirty: bool = False
lib_versions: dict[str, str] = dataclasses.field(default_factory=dict)
ts: int = 0
@dataclass
class Ack(Wire):
"""An experiment ingest: what this call added and what the org's store now holds."""
project: str = ""
experiments_ingested: int = 0
attempts_ingested: int = 0
canonical_experiments: int = 0
experiments_retained: int = 0
canonical_attempts: int = 0
attempts_retained: int = 0
rolled_up: bool = False
@dataclass
class KindTotal(Wire):
"""One experiment kind's slice of the totals."""
kind: str = ""
experiments: int = 0
cost_usd: float = 0.0
@dataclass
class Totals(Wire):
"""The headline aggregate, canonical and retained side by side, with a per-kind breakdown."""
project: str = ""
projects: int = 0
experiments: int = 0
experiments_retained: int = 0
attempts: int = 0
attempts_retained: int = 0
models: int = 0
benchmarks: int = 0
cost_usd: float = 0.0
by_kind: list[KindTotal] = dataclasses.field(default_factory=list)
@dataclass
class Summary(Wire):
"""One project of the org with its canonical and retained counts."""
project: str = ""
experiments: int = 0
experiments_retained: int = 0
attempts: int = 0
attempts_retained: int = 0
models: int = 0
benchmarks: int = 0
cost_usd: float = 0.0
kinds: list[str] = dataclasses.field(default_factory=list)
# ── (de)serialization ─────────────────────────────────────────────────────────
def dump(record: Any) -> Any:
"""A wire record, a mapping, or a list of either, as plain JSON values."""
return _out(record)
def load(cls: type[T], data: Mapping[str, Any] | None) -> T:
"""Read a dataclass from JSON: unknown keys ignored, missing ones defaulted, null lists empty."""
hints = typing.get_type_hints(cls)
kw = {}
src = data or {}
for f in dataclasses.fields(cls): # type: ignore[arg-type]
if f.name in src:
kw[f.name] = _in(hints[f.name], src[f.name])
return cls(**kw)
def _out(v: Any) -> Any:
if isinstance(v, Wire):
return v.dump()
if dataclasses.is_dataclass(v) and not isinstance(v, type):
return {f.name: _out(getattr(v, f.name)) for f in dataclasses.fields(v)}
if isinstance(v, enum.Enum):
return v.value
if isinstance(v, (list, tuple)):
return [_out(x) for x in v]
if isinstance(v, Mapping):
return {k: _out(x) for k, x in v.items()}
return v
def _in(tp: Any, v: Any) -> Any:
origin = typing.get_origin(tp)
args = typing.get_args(tp)
if origin in (typing.Union, types.UnionType):
if v is None:
return None
inner = [a for a in args if a is not type(None)]
return _in(inner[0], v) if len(inner) == 1 else v
if origin is list:
return [_in(args[0], x) for x in (v or [])] if args else list(v or [])
if origin is dict:
return {k: _in(args[1], x) for k, x in (v or {}).items()} if args else dict(v or {})
if isinstance(tp, type) and dataclasses.is_dataclass(tp):
cls: Any = tp
return load(cls, v) if isinstance(v, Mapping) else cls()
if isinstance(tp, type) and issubclass(tp, enum.Enum):
return tp(v)
if tp is str:
return "" if v is None else v
if tp is bool:
return bool(v)
if tp is float:
return 0.0 if v is None else float(v)
if tp is int:
return 0 if v is None else int(v)
return v
# ── the token, its claims, and time ─────────────────────────────────────────────
def token() -> str:
"""`HANZO_API_KEY`, else what `hanzo auth token` prints; "" when neither gives one."""
t = os.environ.get("HANZO_API_KEY", "").strip()
if t:
return t
try:
out = subprocess.run(["hanzo", "auth", "token"], capture_output=True, text=True, timeout=60)
except (OSError, subprocess.SubprocessError):
return ""
return out.stdout.strip() if out.returncode == 0 else ""
def claims(tok: str) -> dict[str, Any] | None:
"""A JWT's claims, read without verifying it (the server verifies); None for a token that is not a JWT."""
parts = tok.split(".")
if len(parts) != 3:
return None
body = parts[1]
try:
out = json.loads(base64.urlsafe_b64decode(body + "=" * (-len(body) % 4)))
except (ValueError, binascii.Error):
return None
return out if isinstance(out, dict) else None
def claimed(tok: str) -> str | None:
"""The project a JWT's claims name, `default` when none; None for a token that is not a JWT."""
c = claims(tok)
return None if c is None else scope(str(c.get("project") or ""))
def left(tok: str) -> int | None:
"""Seconds until a JWT's `exp`; None for a token that is not a JWT or names no expiry."""
c = claims(tok)
exp = None if c is None else c.get("exp")
if not isinstance(exp, (int, float)) or isinstance(exp, bool):
return None
return int(exp) - int(datetime.datetime.now(datetime.timezone.utc).timestamp())
def scope(project: str) -> str:
"""A project as the server scopes it: empty is the default project."""
return project.strip() or "default"
def now() -> str:
"""Now as RFC 3339 in UTC, to the second."""
return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
pkg/hanzo-research/pyproject.toml
+26 -3
View File
@@ -1,15 +1,32 @@
[project]
name = "hanzo-research"
version = "0.2.0"
description = "Hanzo Research SDK — record and query R&D evidence (benchmarks, runs, studies, papers, experiments, artifacts) via /v1/research. Zero-config provenance auto-capture."
description = "File measurements to Hanzo's /v1/research and /v1/benchmark with their provenance: harness commit, model weights, host, device."
readme = "README.md"
license = "MIT"
authors = [{ name = "Hanzo AI", email = "[email protected]" }]
requires-python = ">=3.9"
dependencies = [] # stdlib only (urllib) — importable in any zero-dep harness
requires-python = ">=3.10"
dependencies = [] # stdlib only (urllib): importable in any zero-dependency harness
keywords = ["hanzo", "research", "benchmark", "provenance", "evaluation"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Typing :: Typed",
]
[project.scripts]
hanzo-research = "hanzo_research.cli:main"
[project.urls]
Homepage = "https://hanzo.ai"
Documentation = "https://github.com/hanzoai/python-sdk/blob/main/docs/content/docs/research.mdx"
Repository = "https://github.com/hanzoai/python-sdk"
[dependency-groups]
dev = ["pytest>=8"]
[build-system]
requires = ["hatchling"]
@@ -17,3 +34,9 @@ build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["hanzo_research"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
markers = ["integration: reads api.hanzo.ai with no credential (run with -m integration)"]
addopts = "-m 'not integration'"
pkg/hanzo-research/tests/conftest.py
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
import os
import sys
from typing import Iterator
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from fake import Server, jwt # noqa: E402
from hanzo_research import wire, transport # noqa: E402
HANZO = jwt(
{
"sub": "u-1",
"owner": "hanzo",
"orgs": [{"org": "hanzo", "role": "admin"}, {"org": "lux", "role": "admin"}],
"exp": 4102444800,
}
)
SUPER = jwt({"sub": "u-0", "owner": "admin", "orgs": [{"org": "admin", "role": "admin"}], "exp": 4102444800})
ATLAS = jwt({"sub": "u-2", "owner": "hanzo", "orgs": [{"org": "hanzo"}], "project": "atlas", "exp": 4102444800})
@pytest.fixture(autouse=True)
def quiet(monkeypatch: pytest.MonkeyPatch) -> None:
"""No test reads the caller's own credential or waits out a real backoff."""
monkeypatch.delenv("HANZO_API_KEY", raising=False)
monkeypatch.delenv("RESEARCH_BASE", raising=False)
monkeypatch.delenv("RESEARCH_PROJECT", raising=False)
monkeypatch.setattr(transport, "WAITS", (0, 0, 0))
monkeypatch.setattr(wire, "token", lambda: "")
@pytest.fixture
def server() -> Iterator[Server]:
with Server() as s:
yield s
pkg/hanzo-research/tests/fake.py
+583
View File
@@ -0,0 +1,583 @@
"""An in-process stand-in for api.hanzo.ai's `/v1/research` and `/v1/benchmark`.
It answers as hanzo-inc/cloud `apps/research` (research.go, record.go, store.go) and
`apps/benchmark` (benchmark.go, claims.go, presets.go, history.go) do at the rules a client
can observe: the batch keys and their bounds, required fields, versions appended by
content and read back canonical (the latest that is not retracted), completion derived,
writes forced private, grants on every version, the public `?org=` read only to a caller
with no org, RFC 7807 refusals, the acting org chosen from the token's signed `orgs`, and
claims on the leaderboard only from org admin. JWTs are read, not verified.
"""
from __future__ import annotations
import json
import base64
import hashlib
import threading
import urllib.parse
from typing import Any
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
CATALOG = [
{
"id": "gpqa_diamond",
"title": "GPQA-Diamond",
"axis": "science-reasoning",
"items": 198,
"native": True,
"source": "hendrydong/gpqa_diamond_mc",
},
{
"id": "livecodebench",
"title": "LiveCodeBench (v6)",
"axis": "competitive-coding",
"native": True,
"source": "livecodebench/code_generation_lite test6",
},
]
KEYS = {"benchmarks": "benchmarks", "runs": "runs", "studies": "studies", "papers": "papers"}
NEEDS = {"benchmarks": ("id",), "runs": ("id", "benchmark"), "studies": ("id",), "papers": ("id",)}
DERIVED = ("project", "canonical", "completion", "visibility", "trainable", "publishable")
GAPS = (
"embedder",
"facts_digest",
"k",
"prompt",
"prompt_digest",
"store_digest",
"reader",
"temperature",
"max_tokens",
"dataset_digest",
"commit",
"questions",
"answered",
"when",
"ended",
"by",
"split",
"system",
"version",
)
def jwt(claims: dict[str, Any]) -> str:
"""An unsigned JWT carrying `claims` (the fake reads, never verifies)."""
enc = lambda d: base64.urlsafe_b64encode(json.dumps(d).encode()).decode().rstrip("=") # noqa: E731
return f"{enc({'alg': 'none'})}.{enc(claims)}.sig"
def completion(r: dict[str, Any]) -> str:
q, a = r.get("questions"), r.get("answered")
if q is None or a is None:
return "unknown"
if a > q:
return "inconsistent"
if a < q:
return "partial"
return "complete" if (r.get("ended") or "").strip() else "unknown"
class Problem(Exception):
def __init__(self, status: int, detail: str):
super().__init__(detail)
self.status, self.detail = status, detail
class Fake:
"""The server's state and its handlers; `requests` records every request it answered."""
def __init__(self) -> None:
self.versions: dict[tuple[str, str], list[dict[str, Any]]] = {} # (org, kind) -> versions
self.experiments: dict[str, list[dict[str, Any]]] = {}
self.attempts: dict[str, list[dict[str, Any]]] = {}
self.artifacts: dict[str, dict[str, dict[str, Any]]] = {}
self.claims: list[dict[str, Any]] = []
self.attempt_rows: list[dict[str, Any]] = []
self.requests: list[dict[str, Any]] = []
self.unavailable = 0 # answer the next N requests 503
self.expired: set[str] = set() # bearer tokens answered 401
self.seq = 0
self.lock = threading.Lock()
# ── identity ────────────────────────────────────────────────────────────────
def principal(self, headers: Any) -> dict[str, Any] | None:
auth = headers.get("Authorization") or ""
if not auth.startswith("Bearer "):
return None
parts = auth[7:].split(".")
if len(parts) != 3:
return None
c = json.loads(base64.urlsafe_b64decode(parts[1] + "=" * (-len(parts[1]) % 4)))
orgs = [m["org"] for m in c.get("orgs") or []] or [c.get("owner", "")]
home = orgs[0]
want = headers.get("X-Org-Id") or ""
org = want if want and (home == "admin" or (want != "admin" and want in orgs)) else home
return {
"org": org,
"home": home,
"super": home == "admin",
"user": c.get("sub", ""),
"project": c.get("project") or "default",
}
# ── research: records ───────────────────────────────────────────────────────
def canonical(self, org: str, kind: str) -> list[dict[str, Any]]:
latest: dict[tuple[str, str], dict[str, Any]] = {}
for v in self.versions.get((org, kind), []):
latest[(v["project"], v["id"])] = v
out = []
for v in latest.values():
if (v.get("revision") or "") == "retracted":
continue
r = dict(v)
r.pop("_hash", None)
r.pop("_seq", None)
r["canonical"] = True
if kind == "runs":
r["completion"] = completion(r)
out.append(r)
return sorted(out, key=lambda r: (r["project"], r["id"]))
def put(self, who: dict[str, Any], kind: str, body: dict[str, Any]) -> dict[str, Any]:
rows = body.get(KEYS[kind]) or []
if not rows:
raise Problem(400, "nothing to record")
if len(rows) > 20000:
raise Problem(413, "batch too large; split the upload")
for r in rows:
if any(not r.get(f) for f in NEEDS[kind]):
raise Problem(422, {"runs": "run needs an id and a benchmark"}.get(kind, f"{kind[:-1]} needs an id"))
added = 0
store = self.versions.setdefault((who["org"], kind), [])
for r in rows:
rec = {k: v for k, v in r.items() if k not in DERIVED}
rec["revision"] = (
(rec.get("revision") or "").strip().lower()
if (rec.get("revision") or "").strip().lower() in ("corrected", "retracted")
else "original"
)
h = hashlib.sha256(json.dumps(rec, sort_keys=True).encode()).hexdigest()
if any(v["_hash"] == h and v["id"] == r["id"] and v["project"] == who["project"] for v in store):
continue
self.seq += 1
store.append(
{
**rec,
"project": who["project"],
"visibility": "private",
"trainable": False,
"publishable": False,
"_hash": h,
"_seq": self.seq,
}
)
added += 1
return {"project": who["project"], "recorded": added, "canonical": len(self.canonical(who["org"], kind))}
def runs(self, who: dict[str, Any] | None, q: dict[str, str]) -> dict[str, Any]:
if who is None:
if not q.get("org"):
raise Problem(403, "a validated principal is required")
rows = [r for r in self.canonical(q["org"], "runs") if r["visibility"] == "public"]
if not rows:
raise Problem(404, "Not Found")
else:
rows = self.canonical(who["org"], "runs")
if q.get("completion") and q["completion"] not in ("complete", "partial", "inconsistent", "unknown"):
raise Problem(422, "completion must be complete, partial, inconsistent, or unknown")
for f in ("project", "id", "benchmark", "system", "study", "completion"):
if q.get(f):
rows = [r for r in rows if r.get(f) == q[f]]
if q.get("since"):
rows = [r for r in rows if (r.get("when") or "") >= q["since"]]
if q.get("until"):
rows = [r for r in rows if r.get("when") and r["when"] <= q["until"]]
census = {
k: sum(1 for r in rows if r["completion"] == k) for k in ("complete", "partial", "inconsistent", "unknown")
}
gaps = [{"field": f, "runs": n} for f in GAPS if (n := sum(1 for r in rows if r.get(f) in (None, "")))]
return {"data": rows, "total": len(rows), "completion": census, "gaps": gaps}
def compare(self, who: dict[str, Any], q: dict[str, str]) -> dict[str, Any]:
if not q.get("a") or not q.get("b"):
raise Problem(400, "compare needs two run ids, a and b")
rows = {
r["id"]: r
for r in self.canonical(who["org"], "runs")
if r["project"] == (q.get("project") or who["project"])
}
for k in ("a", "b"):
if q[k] not in rows:
raise Problem(404, f'no canonical run "{q[k]}"')
a, b = rows[q["a"]], rows[q["b"]]
blocks = []
if a.get("benchmark") != b.get("benchmark"):
blocks.append(f'different benchmarks: "{a.get("benchmark")}" and "{b.get("benchmark")}"')
for r in (a, b):
if r["completion"] != "complete":
blocks.append(f'run "{r["id"]}" is not established as complete (completion {r["completion"]})')
config = [
{"field": f, "a": str(a.get(f) or ""), "b": str(b.get(f) or "")}
for f in ("system", "version", "commit", "dataset_digest")
if (a.get(f) or "") != (b.get(f) or "")
]
index = {(m["category"], m["metric"]): m for m in b.get("measures") or []}
deltas = []
for m in a.get("measures") or []:
n = index.get((m["category"], m["metric"]))
if n:
overlap = (
None
if None in (m.get("lo"), m.get("hi"), n.get("lo"), n.get("hi"))
else max(m["lo"], n["lo"]) <= min(m["hi"], n["hi"])
)
deltas.append(
{
"category": m["category"],
"metric": m["metric"],
"a": m["value"],
"b": n["value"],
"change": n["value"] - m["value"],
"overlap": overlap,
}
)
return {
"a": a,
"b": b,
"comparable": not blocks,
"blocks": blocks or None,
"config": config or None,
"deltas": deltas or None,
}
def grant(self, who: dict[str, Any], body: dict[str, Any]) -> dict[str, Any]:
if not body.get("id") and not body.get("sha256"):
raise Problem(400, "grant needs a stable id or an artifact sha256")
vis = body.get("visibility")
if vis is not None and vis not in ("private", "org", "public"):
raise Problem(422, "visibility must be private, org, or public")
project = body.get("project") or who["project"]
n = 0
if body.get("sha256"):
if vis is None:
raise Problem(400, "artifact grant needs a visibility")
a = self.artifacts.get(who["org"], {}).get(body["sha256"])
if a and a["project"] == project:
a["visibility"], n = vis, 1
else:
for v in self.versions.get((who["org"], "runs"), []):
if v["id"] == body["id"] and v["project"] == project:
for k in ("visibility", "trainable", "publishable"):
if body.get(k) is not None:
v[k] = body[k]
n += 1
if n == 0:
raise Problem(404, "no records for that target")
return {"updated": n}
# ── research: experiments and artifacts ─────────────────────────────────────
def ingest(self, who: dict[str, Any], body: dict[str, Any]) -> dict[str, Any]:
exps, atts = body.get("experiments") or [], body.get("attempts") or []
if not exps and not atts:
raise Problem(400, "ingest needs at least one experiment or attempt")
for e in exps:
if not (e.get("id") and e.get("kind") and e.get("subject")):
raise Problem(422, "experiment needs id, kind, and subject")
for a in atts:
if not (a.get("benchmark") and a.get("item") and a.get("model")):
raise Problem(422, "attempt needs benchmark, item, and model")
ex = self.experiments.setdefault(who["org"], [])
at = self.attempts.setdefault(who["org"], [])
e_added = a_added = 0
for e in exps:
if {**e, "project": who["project"]} not in ex:
ex.append({**e, "project": who["project"]})
e_added += 1
for a in atts:
if a not in at:
at.append(a)
a_added += 1
return {
"project": who["project"],
"experiments_ingested": e_added,
"attempts_ingested": a_added,
"canonical_experiments": len({e["id"] for e in ex}),
"experiments_retained": len(ex),
"canonical_attempts": len(at),
"attempts_retained": len(at),
"rolled_up": True,
}
def put_artifact(self, who: dict[str, Any], a: dict[str, Any]) -> dict[str, Any]:
if a.get("kind") not in ("snapshot", "report"):
raise Problem(422, "kind must be snapshot or report")
raw = base64.b64decode(a.get("content") or "")
if not raw:
raise Problem(422, "content must be non-empty base64")
h = hashlib.sha256(raw).hexdigest()
if a.get("sha256") and a["sha256"].lower() != h:
raise Problem(422, "sha256 does not match the content bytes")
store = self.artifacts.setdefault(who["org"], {})
created = h not in store
rec = {k: a.get(k) for k in ("kind", "run_id", "git_sha", "git_branch", "git_dirty", "lib_versions", "ts")}
store.setdefault(
h,
{
**rec,
"sha256": h,
"ref": "sha256:" + h,
"project": who["project"],
"visibility": "private",
"retention_class": "raw-artifact",
"_bytes": raw,
},
)
return {"sha256": h, "ref": "sha256:" + h, "created": created, "rolled_up": True}
# ── the arena ───────────────────────────────────────────────────────────────
def put_claims(self, who: dict[str, Any] | None, body: dict[str, Any]) -> dict[str, Any]:
if who is None:
raise Problem(401, "sign in")
if who["org"] == "admin" and not who["super"]:
raise Problem(403, "SuperAdmin required")
vis = body.get("visibility") or "private"
if vis not in ("private", "public"):
raise Problem(400, 'visibility is "private" or "public"')
out = {"recorded": 0, "rejected": [], "org": who["org"], "visibility": vis}
known = {c["id"] for c in CATALOG}
for i, c in enumerate(body.get("data") or []):
if not c.get("benchmark") or not c.get("model"):
out["rejected"].append(f"row {i}: needs a benchmark and a model")
elif c["benchmark"] not in known:
out["rejected"].append(f"row {i}: unknown benchmark")
elif not str(c.get("source") or "").strip():
out["rejected"].append(f"row {i}: a claim needs its source")
elif not 0 <= float(c.get("score", -1)) <= 100:
out["rejected"].append(f"row {i}: a score is a percentage from 0 to 100")
else:
self.claims.append(
{
**c,
"org": who["org"],
"visibility": vis,
"origin": "stored",
"at": "2026-10-15T00:00:00Z",
"by": who["user"],
}
)
out["recorded"] += 1
if not out["rejected"]:
del out["rejected"]
return out
def leaderboard(self, q: dict[str, str]) -> dict[str, Any]:
bench = q.get("benchmark") or "gpqa_diamond"
rows = []
for c in self.claims:
if c["benchmark"] == bench and c["org"] == "admin" and c["visibility"] == "public":
rows.append(
{
"model": c["model"],
"measured": None,
"n": 0,
"published": c["score"],
"gap": None,
"protocol": c["protocol"],
"claims": 1,
"mean": c["score"],
}
)
return {"benchmark": bench, "rows": rows}
# ── dispatch ────────────────────────────────────────────────────────────────
def handle(self, method: str, url: str, headers: Any, raw: bytes) -> tuple[int, str, bytes]:
parts = urllib.parse.urlsplit(url)
path = parts.path
q = {k: v[-1] for k, v in urllib.parse.parse_qs(parts.query).items()}
body = json.loads(raw) if raw else None
with self.lock:
self.requests.append(
{"method": method, "path": path, "query": q, "headers": dict(headers.items()), "body": body}
)
if self.unavailable:
self.unavailable -= 1
return 503, "application/json", b'{"detail":"research store failover in progress; retry","status":503}'
if (headers.get("Authorization") or "").removeprefix("Bearer ") in self.expired:
return 401, "application/json", b'{"detail":"token expired","status":401}'
if path == "/redirect":
return 302, "text/plain", b""
who = self.principal(headers)
try:
out = self.route(method, path, q, body, who)
except Problem as p:
doc = {"detail": p.detail, "status": p.status, "title": "x", "type": "about:blank"}
return p.status, "application/json", json.dumps(doc).encode()
if isinstance(out, bytes):
return 200, "application/octet-stream", out
return out[0], "application/json", json.dumps(out[1]).encode()
def route(self, method: str, path: str, q: dict[str, str], body: Any, who: dict[str, Any] | None) -> Any:
ok = lambda x: (200, x) # noqa: E731
if path == "/v1/research/runs" and method == "GET":
return ok(self.runs(who, q))
if path.startswith("/v1/benchmark/"):
return self.arena(method, path, q, body, who)
if who is None:
raise Problem(403, "a validated principal is required")
name = path.removeprefix("/v1/research/")
if method == "POST" and name in KEYS:
return ok(self.put(who, name, body or {}))
if method == "GET" and name in ("benchmarks", "studies", "papers"):
rows = self.canonical(who["org"], name)
for f in ("id", "project", "study"):
if q.get(f):
rows = [r for r in rows if r.get(f) == q[f]]
if q.get("cites"):
rows = [r for r in rows if q["cites"] in (r.get("cites") or [])]
return ok({"data": rows, "total": len(rows)})
if name == "compare":
return ok(self.compare(who, q))
if name == "grants":
return ok(self.grant(who, body or {}))
if name == "experiments" and method == "POST":
return ok(self.ingest(who, body or {}))
if name == "experiments":
rows = [
e for e in self.experiments.get(who["org"], []) if not q.get("project") or e["project"] == q["project"]
]
return ok({"data": rows, "total": len(rows)})
if name == "totals":
ex = self.experiments.get(who["org"], [])
return ok(
{
"projects": 1,
"experiments": len({e["id"] for e in ex}),
"experiments_retained": len(ex),
"attempts": len(self.attempts.get(who["org"], [])),
"by_kind": None,
}
)
if name == "projects":
return ok(
{"data": [{"project": "default", "experiments": len(self.experiments.get(who["org"], []))}], "total": 1}
)
if name == "artifacts" and method == "POST":
return ok(self.put_artifact(who, body or {}))
if name == "artifacts":
rows = [
{k: v for k, v in a.items() if k != "_bytes"}
for a in self.artifacts.get(who["org"], {}).values()
if not q.get("run") or a["run_id"] == q["run"]
]
return ok({"data": rows, "total": len(rows)})
if name.startswith("artifacts/"):
a = self.artifacts.get(who["org"], {}).get(name.removeprefix("artifacts/"))
if not a:
raise Problem(404, "artifact not found")
return a["_bytes"]
raise Problem(404, "Not Found")
def arena(self, method: str, path: str, q: dict[str, str], body: Any, who: dict[str, Any] | None) -> Any:
name = path.removeprefix("/v1/benchmark/")
if name == "catalog":
return 200, {"data": CATALOG, "total": len(CATALOG)}
if name == "leaderboard":
return 200, self.leaderboard(q)
if name == "compare":
if not q.get("a") or not q.get("b"):
raise Problem(400, "compare needs ?a= and ?b= model ids")
return 200, {
"benchmark": q.get("benchmark") or "gpqa_diamond",
"a": q["a"],
"b": q["b"],
"n_common": 0,
"a_correct": 0,
"b_correct": 0,
"rescue_a_over_b": 0,
"rescue_b_over_a": 0,
"net_a_minus_b": 0,
"mcnemar_p": 1,
}
if name == "history":
return 200, {"benchmark": q.get("benchmark") or "gpqa_diamond", "data": [], "total": 0}
if name == "claims" and method == "POST":
return 200, self.put_claims(who, body or {})
if name == "claims":
rows = [c for c in self.claims if c["visibility"] == "public" or (who and c["org"] == who["org"])]
for f in ("benchmark", "model", "provider", "source", "protocol", "org"):
if q.get(f):
rows = [c for c in rows if c.get(f) == q[f]]
return 200, {"data": rows, "total": len(rows)}
if name == "presets" and method == "POST":
if who is None:
raise Problem(401, "sign in")
p = dict(body or {}, owner=who["org"])
if not p.get("name") or not p.get("arms"):
raise Problem(400, "preset needs a name and at least one arm")
p["rank"] = p.get("rank") or p["arms"]
p["panel"] = max(1, int(p.get("panel") or 1))
return 202, {
"status": "accepted",
"served_as": "enso-" + p["name"],
"preset": p,
"by": who["user"],
"note": "n",
}
if name == "presets":
return 200, {"data": [{"name": "ultra", "owner": "hanzo", "arms": ["a"], "rank": ["a"], "panel": 3}]}
if name == "runs" and method == "POST":
if who is None:
raise Problem(401, "sign in")
return 202, {
"status": "queued",
"org": who["org"],
"by": who["user"],
"benchmarks": (body or {}).get("benchmarks"),
}
raise Problem(404, "Not Found")
class Server:
"""The fake on a loopback port: `with Server() as s: Research(base=s.url, api_key=...)`."""
def __init__(self) -> None:
self.fake = Fake()
fake = self.fake
class Handler(BaseHTTPRequestHandler):
def log_message(self, *a: Any) -> None:
pass
def answer(self) -> None:
n = int(self.headers.get("Content-Length") or 0)
status, ctype, out = fake.handle(
self.command, self.path, self.headers, self.rfile.read(n) if n else b""
)
self.send_response(status)
if status == 302:
self.send_header("Location", "https://example.com/elsewhere")
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(out)))
self.end_headers()
self.wfile.write(out)
do_GET = do_POST = answer
self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
self.url = f"http://127.0.0.1:{self.httpd.server_address[1]}"
self.thread = threading.Thread(target=self.httpd.serve_forever, args=(0.01,), daemon=True)
def __enter__(self) -> Server:
self.thread.start()
return self
def __exit__(self, *a: Any) -> None:
self.httpd.shutdown()
self.httpd.server_close()
pkg/hanzo-research/tests/test_arena.py
+90
View File
@@ -0,0 +1,90 @@
"""`Arena` against the fake: catalog, board, claims, presets, and the paid run behind confirm."""
from __future__ import annotations
import pytest
from conftest import HANZO, SUPER
from hanzo_research import Arena, Denied, Invalid, Research
from hanzo_research.benchmark import Claim, Preset
def arena(server, token=HANZO, org=None):
return Arena(base=server.url, api_key=token, org=org)
def claim(source="https://api.hanzo.ai/v1/research/runs?org=hanzo&id=r", protocol="hanzo-measured", **kw):
return Claim(
benchmark=kw.pop("benchmark", "gpqa_diamond"),
provider="hanzo",
model=kw.pop("model", "enso-ultra"),
score=kw.pop("score", 91.4),
protocol=protocol,
source=source,
)
def test_reads(server):
a = arena(server)
assert [e.id for e in a.catalog()] == ["gpqa_diamond", "livecodebench"] and a.catalog()[0].items == 198
assert a.compare("x", "y").mcnemar_p == 1.0
assert a.history("gpqa_diamond", model="x") == []
assert a.presets()[0].panel == 3
assert server.fake.requests[-3]["query"] == {"benchmark": "gpqa_diamond", "a": "x", "b": "y"}
def test_claims_reach_the_board_only_from_org_admin(server):
a = arena(server)
f = a.post_claims([claim()], visibility="public")
assert (f.recorded, f.org, f.visibility, f.rejected) == (1, "hanzo", "public", [])
want = {k: claim().dump()[k] for k in ("benchmark", "provider", "model", "score", "protocol", "source")}
assert server.fake.requests[-1]["body"] == {"data": [want], "visibility": "public"}
assert a.leaderboard().rows == []
assert [c.org for c in a.claims(benchmark="gpqa_diamond")] == ["hanzo"]
assert arena(server, token=SUPER).post_claims([claim(model="zen5")], visibility="public").org == "admin"
rows = a.leaderboard("gpqa_diamond").rows
assert [r.model for r in rows] == ["zen5"] and rows[0].published == 91.4 and rows[0].measured is None
private = a.post_claims([claim(model="secret")])
assert private.visibility == "private"
assert "secret" in {c.model for c in a.claims()}
assert "secret" not in {c.model for c in Arena(base=server.url, api_key="").claims()}
def test_a_claim_names_its_protocol_and_source(server):
a = arena(server)
n = len(server.fake.requests)
with pytest.raises(Invalid, match="row 0 needs a protocol and a source"):
a.post_claims([claim(protocol="")])
with pytest.raises(Invalid, match="row 1"):
a.post_claims([claim(), {"benchmark": "gpqa_diamond", "model": "m", "score": 1, "protocol": "x"}])
assert len(server.fake.requests) == n
f = a.post_claims([claim(benchmark="nope"), claim(score=101)])
assert f.recorded == 0 and f.rejected == [
"row 0: unknown benchmark",
"row 1: a score is a percentage from 0 to 100",
]
def test_org_admin_needs_a_superadmin_token(server):
with pytest.raises(Denied, match="cannot act in org admin"):
arena(server, org="admin").post_claims([claim()])
def test_a_paid_run_needs_confirm(server):
a = arena(server)
n = len(server.fake.requests)
with pytest.raises(Invalid, match="paid execution"):
a.post_run(["gpqa_diamond"], model="zen5-pro")
assert len(server.fake.requests) == n
adm = a.post_run(["gpqa_diamond"], model="zen5-pro", confirm=True)
assert adm.status == "queued" and adm.org == "hanzo"
assert server.fake.requests[-1]["body"] == {"benchmarks": ["gpqa_diamond"], "model": "zen5-pro"}
def test_presets_echo(server):
acc = arena(server).post_preset(Preset(name="mine", arms=["a", "b"]))
assert acc.served_as == "enso-mine" and acc.preset.rank == ["a", "b"] and acc.preset.owner == "hanzo"
def test_research_hands_its_credential_to_the_arena(server):
r = Research(base=server.url, api_key=HANZO, org="lux", repo=".")
assert r.arena.post_claims([claim()]).org == "lux"
pkg/hanzo-research/tests/test_canonical.py
+92
View File
@@ -0,0 +1,92 @@
"""The canonical bytes and digests are the Rust crate's, read from tests/vectors/rust.json."""
from __future__ import annotations
import os
import json
import math
import pytest
from hanzo_research import Run, Measure, canonical
VECTORS = json.load(open(os.path.join(os.path.dirname(__file__), "vectors", "rust.json"), encoding="utf-8"))
@pytest.mark.parametrize("v", VECTORS["vectors"], ids=lambda v: v["name"])
def test_bytes_and_digest_equal_the_crates(v):
assert canonical.canonicalize(v["value"]).decode() == v["canonical"]
assert canonical.digest(v["value"]) == v["digest"]
def test_a_run_digests_as_the_crate_digests_it():
rec = VECTORS["records"]["run"]
run = Run.load(rec["value"])
assert canonical.digest(run) == rec["digest"]
assert canonical.digest(run.dump()) == rec["digest"]
def test_rfc8785_appendix_example():
# RFC 8785 section 3.2.2's sample, built from code points so no escape is read twice
bs, q = chr(92), chr(34)
text = chr(0x20AC) + "$" + chr(15) + chr(10) + "A'B" + q + bs + bs + q + "/"
value = {"numbers": [333333333.33333329, 1e30, 4.50, 2e-3, 1e-27], "string": text, "literals": [None, True, False]}
want = (
'{"literals":[null,true,false],"numbers":[333333333.3333333,1e+30,4.5,0.002,1e-27],"string":"'
+ chr(0x20AC)
+ "$"
+ bs
+ "u000f"
+ bs
+ "nA'B"
+ bs
+ q
+ bs * 4
+ bs
+ q
+ '/"}'
)
assert canonical.canonicalize(value).decode() == want
def test_numbers_past_the_crates_range_are_ecmascripts():
# the crate writes these with Rust's Display (recorded in rust.json "apart"); RFC 8785 says:
want = {
1e21: "1e+21",
1e-7: "1e-7",
1.5e300: "1.5e+300",
-0.0: "0",
5e-324: "5e-324",
9007199254740993: "9007199254740992",
2**64 - 1: "18446744073709552000",
}
for x, s in want.items():
assert canonical.number(x) == s
crate = {str(a["value"]): a["canonical"] for a in VECTORS["apart"]}
assert crate["1e+21"] == "1000000000000000000000"
assert crate["-0.0"] == "-0"
def test_shortest_round_trip_digits():
for x in (0.1, 0.8734610296764017, 123456.789, 1e-6, 1e20, 2.5e-5, 1 / 3, 100.0, -3.5):
assert float(canonical.number(x)) == x
def test_what_has_no_json_form_is_refused():
for bad in (math.nan, math.inf, -math.inf):
with pytest.raises(ValueError):
canonical.canonicalize({"x": bad})
with pytest.raises(ValueError):
canonical.canonicalize("\ud800")
with pytest.raises(TypeError):
canonical.canonicalize({1: "x"})
with pytest.raises(TypeError):
canonical.canonicalize({"x": object()})
assert canonical.canonicalize([True, False, None]) == b"[true,false,null]"
def test_a_dataclass_reads_as_its_fields():
m = Measure("all", "f1", 0.5, n=3)
assert canonical.canonicalize(m) == canonical.canonicalize(m.dump())
assert (
canonical.canonicalize(m) == b'{"category":"all","hi":null,"lo":null,"metric":"f1","n":3,"of":null,"value":0.5}'
)
pkg/hanzo-research/tests/test_cli.py
+95
View File
@@ -0,0 +1,95 @@
"""`hanzo-research` against the fake."""
from __future__ import annotations
import json
import pytest
from conftest import HANZO
from hanzo_research.cli import main
@pytest.fixture
def cli(server, monkeypatch, capsys):
monkeypatch.setenv("HANZO_API_KEY", HANZO)
monkeypatch.setenv("RESEARCH_BASE", server.url)
def go(*argv):
code = main(argv)
out, err = capsys.readouterr()
return code, (json.loads(out) if out.strip() else None), err
return go
def run(i):
return {
"id": f"demo-dev-c-s{i}",
"benchmark": "demo",
"system": f"s{i}",
"questions": 2,
"answered": 2,
"ended": "2026-10-15T00:00:00Z",
"measures": [{"category": "all", "metric": "accuracy", "value": 0.5}],
}
def test_post_a_jsonl_file_and_grant(cli, server, tmp_path):
f = tmp_path / "runs.jsonl"
f.write_text("\n".join(json.dumps(run(i)) for i in range(3)) + "\n")
code, out, _ = cli("post", str(f), "--grant", "public", "--publishable")
assert code == 0 and out["runs"]["recorded"] == 3
assert out["granted"] == {f"demo-dev-c-s{i}": 1 for i in range(3)}
code, out, _ = cli("list", "runs", "--org", "hanzo")
assert out["total"] == 3 and out["completion"]["complete"] == 3
def test_a_plan_files_benchmarks_before_runs(cli, server, tmp_path):
f = tmp_path / "plan.json"
f.write_text(json.dumps({"runs": [run(1)], "benchmarks": [{"id": "demo", "metric": "accuracy"}]}))
code, out, _ = cli("post", str(f), "--dry")
assert code == 0 and list(out) == ["benchmarks", "runs"] and len(server.fake.requests) == 0
code, out, _ = cli("post", str(f))
assert [q["path"] for q in server.fake.requests] == ["/v1/research/benchmarks", "/v1/research/runs"]
code, out, _ = cli("list", "benchmarks")
assert out[0]["id"] == "demo"
def test_get_grant_retract_compare(cli, server, tmp_path):
f = tmp_path / "two.json"
f.write_text(json.dumps([run(1), run(2)]))
cli("post", str(f))
code, out, _ = cli("get", "demo-dev-c-s1")
assert out["completion"] == "complete"
code, out, _ = cli("grant", "demo-dev-c-s1", "--visibility", "org", "--no-publishable")
assert out == {"demo-dev-c-s1": 1} and server.fake.requests[-1]["body"]["publishable"] is False
code, out, _ = cli("compare", "demo-dev-c-s1", "demo-dev-c-s2")
assert out["comparable"] and out["deltas"][0]["change"] == 0
code, out, _ = cli("retract", "demo-dev-c-s1", "--why", "wrong split")
assert out["recorded"] == 1
code, out, err = cli("get", "demo-dev-c-s1")
assert code == 1 and "no canonical run" in err
def test_refusals_exit_nonzero_with_the_servers_sentence(cli, tmp_path):
f = tmp_path / "bad.json"
f.write_text(json.dumps({"id": "x"}))
code, out, err = cli("post", str(f))
assert code == 1 and "run needs an id and a benchmark" in err
code, out, err = cli("--org", "zoo", "list", "runs")
assert code == 1 and "cannot act in org zoo" in err
code, out, err = cli("grant")
assert code == 2
def test_provenance_and_arena(cli, server):
code, out, _ = cli(
"provenance", "--model", "m", "--weights", "sha256:" + "c" * 64, "--no-gpu", "--package", "pytest"
)
assert code == 0 and out["model"] == "m" and out["weights"] == "sha256:" + "c" * 64 and "pytest" in out["packages"]
code, out, _ = cli("arena", "catalog")
assert out[0]["id"] == "gpqa_diamond"
code, out, _ = cli("arena", "compare", "a", "b", "--benchmark", "livecodebench")
assert out["benchmark"] == "livecodebench"
code, out, _ = cli("arena", "leaderboard")
assert out["rows"] == []
pkg/hanzo-research/tests/test_client.py
+304
View File
@@ -0,0 +1,304 @@
"""`Research` against the fake: the bodies, headers and answers of every /v1/research route."""
from __future__ import annotations
import json
import base64
import pytest
from conftest import ATLAS, HANZO, SUPER
from hanzo_research import (
Run,
Error,
Paper,
Split,
Study,
Denied,
Invalid,
Missing,
Research,
Benchmark,
Unavailable,
wire,
canonical,
transport,
)
def client(server, token=HANZO, **kw):
return Research(base=server.url, api_key=token, repo=".", **kw)
def run(id="demo.bench-dev-a-sys", **kw):
base = dict(
benchmark="demo.bench",
split="dev",
system="sys",
version="v1",
commit="c0ffee",
questions=4,
answered=4,
ended="2026-10-15T00:00:00Z",
)
r = Run(id=id, **{**base, **kw})
return r.put("all", "accuracy", 0.75, n=3, of=4, lo=0.3, hi=0.95)
def test_records_file_in_their_batch_and_read_back(server):
r = client(server)
b = r.record(
benchmarks=[Benchmark(id="demo.bench", title="Demo", splits=[Split("dev", 4, "all four")], metric="accuracy")]
)["benchmarks"]
assert (b.project, b.recorded, b.canonical) == ("default", 1, 1)
batch = r.record(runs=[run(), run("demo.bench-dev-b-sys", system="other")])["runs"]
assert batch.recorded == 2
sent = server.fake.requests[-1]
assert sent["method"] == "POST" and sent["path"] == "/v1/research/runs" and set(sent["body"]) == {"runs"}
assert sent["body"]["runs"][0] == run().dump()
h = sent["headers"]
assert h["Authorization"] == f"Bearer {HANZO}" and h["X-Project-Id"] == "default" and "X-Org-Id" not in h
assert h["Content-Type"] == "application/json" and h["User-Agent"].startswith("hanzo-research/")
listing = r.runs(benchmark="demo.bench")
assert listing.total == len(listing) == 2 and listing.completion.complete == 2
assert [x.system for x in listing] == ["sys", "other"] and listing[0].id == "demo.bench-dev-a-sys"
got = r.get_run("demo.bench-dev-a-sys")
assert isinstance(got, Run) and got.completion == "complete" and got.canonical and got.visibility == "private"
assert got.measure("all", "accuracy").lo == 0.3
assert r.get_benchmark("demo.bench").splits[0].definition == "all four"
assert {g.field for g in listing.gaps} >= {"prompt", "k"}
def test_a_mapping_is_sent_as_given(server):
r = client(server)
row = {"id": "x-dev-c-s", "benchmark": "x", "measures": [{"category": "all", "metric": "m", "value": 1}]}
r.record(runs=[row])["runs"]
assert server.fake.requests[-1]["body"] == {"runs": [row]}
def test_the_same_record_twice_records_nothing(server):
r = client(server)
assert r.record(runs=[run()])["runs"].recorded == 1
assert r.record(runs=[run()])["runs"].recorded == 0
both = r.record(runs=[run()], benchmarks=[Benchmark(id="demo.bench")])
assert (both["benchmarks"].recorded, both["runs"].recorded) == (1, 0)
assert [q["path"] for q in server.fake.requests[-2:]] == ["/v1/research/benchmarks", "/v1/research/runs"]
n = len(server.fake.requests)
assert r.record() == {} and r.record(runs=[], papers=()) == {}
assert len(server.fake.requests) == n
def test_studies_and_papers(server):
r = client(server)
out = r.record(papers=[Paper(id="p", cites=["a", "b"], study="s")], studies=[Study(id="s", question="does it?")])
assert list(out) == ["studies", "papers"]
assert r.studies(id="s")[0].question == "does it?"
assert r.papers(cites="b")[0].id == "p" and r.papers(cites="z") == []
def test_grant_then_the_public_read_needs_no_credential(server):
r = client(server)
r.record(runs=[run()])
with pytest.raises(Missing):
r.runs(org="hanzo")
assert r.grant("demo.bench-dev-a-sys", visibility="public", publishable=True) == 1
sent = server.fake.requests[-1]["body"]
assert sent == {"project": "default", "id": "demo.bench-dev-a-sys", "visibility": "public", "publishable": True}
public = Research(base=server.url, api_key="", repo=".").runs(org="hanzo")
assert public.total == 1 and public.data[0].publishable and public.data[0].visibility == "public"
q = server.fake.requests[-1]
assert q["query"] == {"org": "hanzo"} and "Authorization" not in q["headers"] and "X-Project-Id" not in q["headers"]
# a credentialled client reading ?org= still sends none: the server answers ?org= only to a caller with no org
assert r.get_run("demo.bench-dev-a-sys", org="hanzo").id == "demo.bench-dev-a-sys"
assert "Authorization" not in server.fake.requests[-1]["headers"]
with pytest.raises(Missing, match="no records for that target"):
r.grant("nope", visibility="public")
with pytest.raises(Invalid, match="visibility must be"):
r.grant("demo.bench-dev-a-sys", visibility="world")
def test_retract_withdraws_and_revise_appends(server):
r = client(server)
r.record(runs=[run()])
fixed = r.get_run("demo.bench-dev-a-sys")
fixed.measures[0].value = 0.5
assert r.revise(fixed, notes="rescored").recorded == 1
again = r.get_run("demo.bench-dev-a-sys")
assert again.revision == "corrected" and again.measures[0].value == 0.5 and again.notes == "rescored"
with pytest.raises(Invalid):
r.retract("demo.bench-dev-a-sys", " ")
assert r.retract("demo.bench-dev-a-sys", "read on the wrong split").recorded == 1
assert server.fake.requests[-1]["body"]["runs"][0]["revision"] == "retracted"
with pytest.raises(Missing):
r.get_run("demo.bench-dev-a-sys")
assert r.runs().total == 0
def test_compare_names_what_blocks_it(server):
r = client(server)
r.record(runs=[run(), run("demo.bench-dev-b-sys", system="other", answered=3)])["runs"]
c = r.compare("demo.bench-dev-a-sys", "demo.bench-dev-b-sys")
assert not c.comparable and any("not established as complete" in b for b in c.blocks)
assert c.a.id == "demo.bench-dev-a-sys" and c.config[0].field == "system"
assert c.deltas[0].change == 0 and c.deltas[0].overlap is True
with pytest.raises(Missing, match='no canonical run "zz"'):
r.compare("demo.bench-dev-a-sys", "zz")
def test_refusals_carry_the_servers_sentence(server):
r = client(server)
with pytest.raises(Invalid, match="run needs an id and a benchmark") as e:
r.record(runs=[{"id": "x"}])["runs"]
assert e.value.status == 422 and e.value.path == "/v1/research/runs"
with pytest.raises(Invalid, match="completion must be"):
r.runs(completion="done")
def test_what_json_cannot_hold_never_leaves(server):
n = len(server.fake.requests)
with pytest.raises(ValueError):
client(server).record(runs=[{"id": "x", "benchmark": "b", "measures": [{"value": float("nan")}]}])
assert len(server.fake.requests) == n
def test_503_is_asked_again(server):
r = client(server)
server.fake.unavailable = 2
assert r.record(runs=[run()])["runs"].recorded == 1
server.fake.unavailable = 9
with pytest.raises(Unavailable, match="failover"):
r.runs()
def test_no_credential_says_how_to_get_one(server):
with pytest.raises(Denied, match="HANZO_API_KEY"):
Research(base=server.url, api_key="", repo=".").runs()
def test_an_expired_cli_token_is_fetched_again(server, monkeypatch):
server.fake.expired.add("old")
fresh = iter(["old", HANZO])
monkeypatch.setattr(wire, "token", lambda: next(fresh))
r = Research(base=server.url, repo=".")
assert r.runs().total == 0
auths = [q["headers"].get("Authorization") for q in server.fake.requests]
assert auths == ["Bearer old", f"Bearer {HANZO}"]
server.fake.expired.add(HANZO)
with pytest.raises(Denied, match="token expired"):
Research(base=server.url, api_key=HANZO, repo=".").runs()
def test_the_acting_org_is_one_the_token_names(server):
r = client(server, org="lux")
r.record(runs=[run()])
assert server.fake.requests[-1]["headers"]["X-Org-Id"] == "lux"
assert r.org == "lux" and client(server).org == "hanzo"
assert client(server).runs().total == 0 and r.runs().total == 1
n = len(server.fake.requests)
with pytest.raises(Denied, match="cannot act in org zoo"):
client(server, org="zoo").runs()
with pytest.raises(Denied, match="cannot act in org admin"):
client(server, org="admin").runs()
assert len(server.fake.requests) == n
assert client(server, token=SUPER, org="zoo").runs().total == 0
def test_the_project_is_the_tokens(server):
with pytest.raises(Error, match="the token's project is default, not atlas"):
client(server, project="atlas").runs()
r = client(server, token=ATLAS)
assert r.project == "atlas"
assert r.record(runs=[run()])["runs"].project == "atlas"
assert server.fake.requests[-1]["headers"]["X-Project-Id"] == "atlas"
def test_redirects_and_oversized_answers_are_refused(server, monkeypatch):
t = client(server).http
with pytest.raises(Error, match="redirect"):
t.call("GET", "/redirect", None, None, True)
monkeypatch.setattr(transport, "MAX_RESPONSE_BYTES", 10)
with pytest.raises(Error, match="past 10 bytes"):
client(server).runs()
def test_no_answer_is_unavailable():
with pytest.raises(Unavailable, match="no answer"):
Research(base="http://127.0.0.1:9", api_key=HANZO, repo=".", timeout=2).runs()
def test_the_experiment_plane(server, tmp_path):
r = client(server)
with r.experiment("benchmark", "grok-4.5", "gpqa_diamond", n_total=2, hypothesis="H", predict="P") as e:
e.record("q1", "grok-4.5", {"answer": "A", "correct": True})
e.record("q2", "grok-4.5", {"answer": "B", "correct": False, "gold": "C"})
e.record("q3", "grok-4.5", {"answer": "", "correct": False, "status": "faulted"})
e.log("one seed")
posts = [
q["body"] for q in server.fake.requests if q["path"] == "/v1/research/experiments" and q["method"] == "POST"
]
running, done = posts[0]["experiments"][0], posts[-1]["experiments"][0]
assert running["status"] == "running" and running["meta"]["log"] == [] and running["meta"]["hypothesis"] == "H"
assert done["status"] == "complete" and done["value"] == 50.0 and done["meta"]["verdict"] == "inconclusive"
assert done["meta"]["log"] == ["one seed"] and done["id"] == "benchmark:grok-4.5:gpqa_diamond"
assert posts[1]["attempts"][0]["source"] == "hanzo-measured"
k = r.experiment("kernel-perf", "matvec", "vulkan/6144", metric="ratio_vs_hand")
ack = k.conclude("refuted", because="0.79x", value=0.79)
assert ack.experiments_ingested == 1
with pytest.raises(ValueError):
k.conclude("maybe")
with pytest.raises(RuntimeError):
with r.experiment("training", "a7", "stage-a") as t:
raise RuntimeError("boom")
assert server.fake.requests[-1]["body"]["experiments"][0]["status"] == "faulted"
assert {x.id for x in r.query()} >= {"benchmark:grok-4.5:gpqa_diamond", "kernel-perf:matvec:vulkan/6144"}
assert r.totals().experiments == 3 and r.projects()[0].project == "default"
def test_artifacts_are_addressed_by_their_bytes(server):
r = client(server)
s = r.upload_artifact("report", "demo.bench-dev-a-sys", b"# notes\n", "text/markdown")
assert s.created and s.ref == "sha256:" + s.sha256
body = server.fake.requests[-1]["body"]
assert base64.b64decode(body["content"]) == b"# notes\n" and body["content_sha256"] == s.sha256
assert body["artifact_id"] == "demo.bench-dev-a-sys:report" and body["uncompressed_size"] == 8
assert not r.upload_artifact("report", "demo.bench-dev-a-sys", b"# notes\n").created
value = {"b": 1, "a": [1.5, "é"]}
j = r.upload_json_artifact("report", "demo.bench-dev-a-sys", value)
assert j.sha256 == canonical.digest(value)
assert r.get_artifact(j.sha256) == canonical.canonicalize(value)
assert {a.sha256 for a in r.artifacts(run="demo.bench-dev-a-sys")} == {s.sha256, j.sha256}
assert r.grant(sha256=j.sha256, visibility="public") == 1
with pytest.raises(Invalid, match="kind must be"):
r.upload_artifact("blob", "x", b"x")
with pytest.raises(Missing):
r.get_artifact("0" * 64)
def test_url_is_the_public_read_of_one_run(server):
r = client(server)
assert (
r.url("enso.gpqa_diamond-test-frontier-x")
== f"{server.url}/v1/research/runs?org=hanzo&id=enso.gpqa_diamond-test-frontier-x"
)
assert r.url("a b", org="zoo").endswith("?org=zoo&id=a%20b")
with pytest.raises(ValueError, match="no org"):
Research(base=server.url, api_key="sk-opaque", repo=".").url("x")
assert json.loads(json.dumps(r.arena.catalog()[0].dump()))["id"] == "gpqa_diamond"
def test_the_default_client(server, monkeypatch):
import hanzo_research as research
monkeypatch.setattr(research, "_default", None)
research.configure(base=server.url, api_key=HANZO, repo=".")
out = research.record(
benchmarks=[{"id": "b"}], runs=[{"id": "r1", "benchmark": "b"}, {"id": "r2", "benchmark": "b"}]
)
assert list(out) == ["benchmarks", "runs"] and out["runs"].recorded == 2
assert server.fake.requests[-1]["body"] == {
"runs": [{"id": "r1", "benchmark": "b"}, {"id": "r2", "benchmark": "b"}]
}
assert [x.id for x in research.runs(benchmark="b")] == ["r1", "r2"]
assert server.fake.requests[-1]["query"] == {"benchmark": "b"}
assert research.benchmarks(id="b")[0].id == "b" and research.studies() == [] and research.papers() == []
c = research.compare("r1", "r2")
assert c.a.id == "r1" and c.b.id == "r2" and server.fake.requests[-1]["query"] == {"a": "r1", "b": "r2"}
pkg/hanzo-research/tests/test_lineage.py
+72
View File
@@ -0,0 +1,72 @@
"""Lineage narrows and never widens; records digest as the crate's do."""
from __future__ import annotations
import os
import json
import pytest
from hanzo_research.lineage import (
ArtifactRef,
DataPurpose,
EnvironmentV1,
LineagePolicy,
ResearchEvent,
ContaminationError,
derive_lineage_policy,
verify_promotion_eligibility,
)
VECTORS = json.load(open(os.path.join(os.path.dirname(__file__), "vectors", "rust.json"), encoding="utf-8"))
def test_policy_propagation():
train = LineagePolicy.root(DataPurpose.TRAIN, True)
train.assert_trainable("source")
ev = LineagePolicy.root(DataPurpose.EVAL_ONLY, True)
assert not ev.derivation_allowed
with pytest.raises(ContaminationError):
ev.assert_trainable("eval_src")
derived = LineagePolicy.derive(DataPurpose.TEACHER_MINING, True, [train, ev])
assert derived.purpose is DataPurpose.EVAL_ONLY and not derived.derivation_allowed
clean = derive_lineage_policy(DataPurpose.TEACHER_MINING, True, [train])
assert clean.purpose is DataPurpose.TEACHER_MINING and clean.derivation_allowed
clean.assert_trainable("clean_child")
def test_promotion_eligibility():
root = LineagePolicy.root(DataPurpose.TRAIN, True)
mining = LineagePolicy.derive(DataPurpose.TEACHER_MINING, True, [root])
ckpt = LineagePolicy.derive(DataPurpose.TRAIN, True, [root, mining])
verify_promotion_eligibility("ckpt-a10", ckpt, [root, mining])
with pytest.raises(ContaminationError, match=r"ckpt-a10->transitive_ancestor\[0\] has derivation_allowed = false"):
verify_promotion_eligibility("ckpt-a10", ckpt, [LineagePolicy(DataPurpose.TRAIN, False)])
with pytest.raises(ContaminationError, match="purpose EvalOnly"):
verify_promotion_eligibility("ckpt-a10", LineagePolicy(DataPurpose.EVAL_ONLY, True), [])
def test_purposes_order_and_wire_names():
assert [p.value for p in DataPurpose] == ["train", "teacher_mining", "dev", "eval_only", "sealed_eval"]
assert DataPurpose.most_restrictive(DataPurpose.DEV, DataPurpose.TRAIN) is DataPurpose.DEV
assert DataPurpose.DEV.is_eval() and not DataPurpose.DEV.is_training_allowed()
def test_records_digest_as_the_crates():
r = VECTORS["records"]
assert LineagePolicy.load(r["policy"]["value"]).digest() == r["policy"]["digest"]
assert EnvironmentV1.load(r["environment"]["value"]).digest() == r["environment"]["digest"]
ev = ResearchEvent.load(r["event"]["value"])
assert ev.digest() == r["event"]["digest"]
assert (
ResearchEvent.new(
ev.run_id, ev.worker_id, ev.event_id, 1, ev.previous_event_hash, {"loss": 1.25, "step": 1}, ev.created_at
).payload_hash
== ev.payload_hash
)
def test_artifact_ref():
a = ArtifactRef.new("ab" * 32, LineagePolicy.root(DataPurpose.TRAIN, True))
assert a.ref_uri == "sha256:" + "ab" * 32 and a.artifact_id == "artifact:" + "ab" * 32
assert a.dump()["lineage"] == {"purpose": "train", "derivation_allowed": True}
assert ArtifactRef.load(a.dump()) == a
pkg/hanzo-research/tests/test_live.py
+26
View File
@@ -0,0 +1,26 @@
"""Reads of api.hanzo.ai that need no credential: `pytest -m integration`."""
from __future__ import annotations
import pytest
from hanzo_research import Run, Arena, Research
pytestmark = pytest.mark.integration
def test_the_public_runs_of_hanzo():
r = Research(api_key="", repo=".")
page = r.runs(org="hanzo")
assert page.total == len(page.data) > 0
assert all(isinstance(x, Run) and x.visibility == "public" and x.canonical for x in page.data)
c = page.completion
assert c.complete + c.partial + c.inconsistent + c.unknown == page.total
first = page.data[0]
assert r.get_run(first.id, org="hanzo").id == first.id
def test_the_arena_reads():
a = Arena(api_key="")
assert any(e.id == "gpqa_diamond" for e in a.catalog())
assert a.leaderboard().benchmark == "gpqa_diamond"
assert all(c.source for c in a.claims(benchmark="gpqa_diamond"))
pkg/hanzo-research/tests/test_provenance.py
+100
View File
@@ -0,0 +1,100 @@
"""capture() reads the harness, the weights and the machine; apply() folds them into a run."""
from __future__ import annotations
import os
import hashlib
import subprocess
import pytest
from hanzo_research import Run, canonical, provenance
from hanzo_research.lineage import EnvironmentV1
def git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
@pytest.fixture
def repo(tmp_path):
r = tmp_path / "harness"
r.mkdir()
git(r, "init", "-q", "-b", "main")
git(r, "config", "user.email", "[email protected]")
git(r, "config", "user.name", "t")
git(r, "remote", "add", "origin", "https://user:[email protected]/hanzoai/harness.git")
(r / "bench.py").write_text("print(1)\n")
git(r, "add", ".")
git(r, "commit", "-q", "-m", "first")
(r / "bench.py").write_text("print(2)\n")
git(r, "commit", "-qam", "second")
return r
def test_capture_reads_the_harness_and_the_machine(repo):
p = provenance.capture(
repo=str(repo),
model="kai-1.3",
weights="sha256:" + "a" * 64,
device="cpu",
dtype="f32",
packages=["hanzo-research-not-installed"],
gpu=False,
)
assert p.commit == git(repo, "rev-parse", "HEAD") and p.branch == "main" and not p.dirty and p.diff == ""
assert p.remote == "https://github.com/hanzoai/harness.git"
assert p.weights == "sha256:" + "a" * 64 and p.device == "cpu" and p.dtype == "f32"
assert p.host and p.python and p.cores > 0 and p.date.endswith("Z") and p.packages == {}
(repo / "bench.py").write_text("print(3)\n")
d = provenance.capture(repo=str(repo), gpu=False)
assert d.dirty and d.diff.startswith("sha256:")
(repo / "new.py").write_text("x")
assert provenance.capture(repo=str(repo), gpu=False).diff != d.diff
def test_weights_are_hashed_by_content(tmp_path):
f = tmp_path / "model.safetensors"
f.write_bytes(b"w" * 3_000_000)
assert provenance.weights(str(f)) == "sha256:" + hashlib.sha256(b"w" * 3_000_000).hexdigest()
d = tmp_path / "folder"
(d / "sub").mkdir(parents=True)
(d / "kai.json").write_text("{}")
(d / "sub" / "b.bin").write_bytes(b"b")
listing = "".join(f"{hashlib.sha256(x).hexdigest()} {n}\n" for n, x in (("kai.json", b"{}"), ("sub/b.bin", b"b")))
assert provenance.weights(str(d)) == "sha256:" + hashlib.sha256(listing.encode()).hexdigest()
(d / "SHA256SUMS").write_text(listing)
assert provenance.weights(str(d)) == "sha256:" + hashlib.sha256(listing.encode()).hexdigest()
assert provenance.weights("E45DD919" * 8) == "sha256:" + "e45dd919" * 8
with pytest.raises(ValueError):
provenance.weights("not-a-path-or-digest")
def test_apply_fills_what_the_run_leaves_empty(repo):
p = provenance.capture(
repo=str(repo), model="kai-1.3", weights="sha256:" + "b" * 64, device="cuda", dtype="bf16", gpu=False
)
r = p.apply(Run(id="x", benchmark="b", notes="Read once."))
assert r.commit == p.commit and r.system == "kai-1.3" and r.version == "kai-1.3@" + "b" * 64 and r.by == p.host
assert r.notes.startswith("Read once. Provenance: harness https://github.com/hanzoai/harness.git at ")
assert f"Record {p.digest()}." in r.notes and "device cuda dtype bf16" in r.notes
assert p.digest() == "sha256:" + canonical.digest(p.dump())
kept = p.apply({"id": "y", "system": "kai", "version": "kai-1.3@cafe", "commit": "abc"})
assert (kept["system"], kept["version"], kept["commit"]) == ("kai", "kai-1.3@cafe", "abc")
assert kept["notes"].startswith("Provenance: ")
def test_environment_is_the_crates_record(repo):
env = provenance.capture(repo=str(repo), dtype="bf16", gpu=False).environment()
assert (
isinstance(env, EnvironmentV1) and env.precision_config == "bf16" and env.compiler_version.startswith("python ")
)
assert len(env.digest()) == 64
def test_the_narrative_takes_only_an_object_id(repo):
first = git(repo, "rev-list", "--max-parents=0", "HEAD")
assert provenance.commit_narrative(str(repo), since_sha=first) == ["second"]
assert provenance.commit_narrative(str(repo), since_sha="--output=/tmp/pwned") == ["second", "first"]
assert provenance.git_state(str(repo))["git_branch"] == "main"
assert provenance.find_repo(str(repo / ".")) == os.path.realpath(repo)
assert provenance.host()["hostname"]
pkg/hanzo-research/tests/test_readme.py
+38
View File
@@ -0,0 +1,38 @@
"""The README's worked example runs as written, against the fake, and the docs page shows the same code."""
from __future__ import annotations
import os
import re
from conftest import HANZO
HERE = os.path.dirname(__file__)
README = os.path.join(HERE, "..", "README.md")
DOCS = os.path.join(HERE, "..", "..", "..", "docs", "content", "docs", "research.mdx")
def example(path: str) -> str:
text = open(path, encoding="utf-8").read()
return re.search(r"## Worked example.*?```python\n(.*?)```", text, re.S).group(1)
def test_the_worked_example_files_reads_back(server, monkeypatch, capsys):
monkeypatch.setenv("HANZO_API_KEY", HANZO)
monkeypatch.setenv("RESEARCH_BASE", server.url)
exec(compile(example(README), "README.md", "exec"), {"__name__": "example"}) # noqa: S102
out = capsys.readouterr().out.split()
assert out[0] == "complete" and float(out[1]) > 0
paths = [(q["method"], q["path"]) for q in server.fake.requests]
assert paths == [
("POST", "/v1/research/benchmarks"),
("POST", "/v1/research/runs"),
("POST", "/v1/research/grants"),
("GET", "/v1/research/runs"),
]
run = server.fake.requests[1]["body"]["runs"][0]
assert run["commit"] == out[2] and "Provenance: harness " in run["notes"] and run["by"]
def test_the_docs_page_shows_the_same_example():
assert example(DOCS) == example(README)
pkg/hanzo-research/tests/test_record.py
-71
View File
@@ -1,71 +0,0 @@
"""The record wire against a local stand-in for /v1/research: each batch posts to its
own path under its own key, a filter reaches the query, and compare names both runs."""
import json
import threading
import unittest
from http.server import BaseHTTPRequestHandler, HTTPServer
import hanzo_research as research
SEEN = []
class Fake(BaseHTTPRequestHandler):
def _answer(self, body):
raw = json.dumps(body).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def do_POST(self):
n = int(self.headers.get("Content-Length", 0))
SEEN.append(("POST", self.path, json.loads(self.rfile.read(n)), self.headers.get("Authorization")))
self._answer({"project": "default", "recorded": 1})
def do_GET(self):
SEEN.append(("GET", self.path, None, self.headers.get("Authorization")))
self._answer({"data": [{"id": "r1"}], "total": 1} if "/runs" in self.path else {"a": {}, "b": {}})
def log_message(self, *a):
pass
class RecordWire(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.srv = HTTPServer(("127.0.0.1", 0), Fake)
threading.Thread(target=cls.srv.serve_forever, daemon=True).start()
research.configure(base=f"http://127.0.0.1:{cls.srv.server_port}", api_key="k", project="enso")
@classmethod
def tearDownClass(cls):
cls.srv.shutdown()
def setUp(self):
SEEN.clear()
def test_each_batch_posts_to_its_path(self):
out = research.record(benchmarks=[{"id": "b"}], runs=[{"id": "r", "benchmark": "b"}])
self.assertEqual(sorted(out), ["benchmarks", "runs"])
posted = {p: body for m, p, body, _ in SEEN if m == "POST"}
self.assertEqual(posted["/v1/research/benchmarks"], {"benchmarks": [{"id": "b"}]})
self.assertEqual(posted["/v1/research/runs"], {"runs": [{"id": "r", "benchmark": "b"}]})
self.assertTrue(all(auth == "Bearer k" for *_, auth in SEEN))
def test_an_empty_batch_is_not_sent(self):
self.assertEqual(research.record(), {})
self.assertEqual(SEEN, [])
def test_runs_carry_their_filters(self):
self.assertEqual(research.runs(benchmark="enso.route", org="hanzo"), [{"id": "r1"}])
self.assertEqual(SEEN[0][1], "/v1/research/runs?benchmark=enso.route&org=hanzo")
def test_compare_names_both_runs(self):
research.compare("x", "y")
self.assertEqual(SEEN[0][1], "/v1/research/compare?a=x&b=y")
if __name__ == "__main__":
unittest.main()
pkg/hanzo-research/tests/test_wal.py
+64
View File
@@ -0,0 +1,64 @@
"""The WAL chains as the crate's: a log the crate wrote verifies here, and one written here resumes."""
from __future__ import annotations
import os
import json
import shutil
import pytest
from hanzo_research import wal
from hanzo_research.wal import GENESIS, ResearchWal
HERE = os.path.join(os.path.dirname(__file__), "vectors")
VECTORS = json.load(open(os.path.join(HERE, "rust.json"), encoding="utf-8"))
def test_a_log_the_crate_wrote_verifies(tmp_path):
assert wal.verify(os.path.join(HERE, "wal.jsonl")) == VECTORS["wal"]["last"]
shutil.copy(os.path.join(HERE, "wal.jsonl"), tmp_path / "wal.jsonl")
w = ResearchWal.open(str(tmp_path), "run-1")
assert w.sequence == 3 and w.last_event_hash == VECTORS["wal"]["last"]
e = w.append("step", {"step": 4, "loss": 0.9})
assert e.sequence == 4 and e.previous_event_hash == VECTORS["wal"]["last"]
assert wal.verify(w.path) == w.last_event_hash
def test_append_chains_and_resumes(tmp_path):
w = ResearchWal.open(str(tmp_path), "run-test")
e1 = w.append("step", {"step": 1, "loss": 1.25})
assert e1.sequence == 1 and e1.previous_event_hash == GENESIS and e1.event_id == "run-test-worker-0-1-step"
e2 = w.append("step", {"step": 2, "loss": 1.10})
assert e2.previous_event_hash == e1.digest()
again = ResearchWal.open(str(tmp_path), "run-test")
assert again.sequence == 2 and again.last_event_hash == e2.digest()
w7 = ResearchWal.open(str(tmp_path), "run-test", "worker-7")
assert w7.path.endswith("wal-worker-7.jsonl") and w7.sequence == 0
def test_a_changed_line_breaks_the_chain(tmp_path):
w = ResearchWal.open(str(tmp_path), "run")
for i in range(3):
w.append("step", {"step": i})
lines = open(w.path).read().splitlines()
lines[1] = lines[1].replace('"step":1', '"step":9')
open(w.path, "w").write("\n".join(lines) + "\n")
with pytest.raises(ValueError, match="line 2's payload"):
wal.verify(w.path)
del lines[1]
open(w.path, "w").write("\n".join(lines) + "\n")
with pytest.raises(ValueError, match="sequence"):
wal.verify(w.path)
def test_workers_merge_in_one_order(tmp_path):
a = ResearchWal.open(str(tmp_path), "run", "worker-a")
b = ResearchWal.open(str(tmp_path), "run", "worker-b")
evs = [b.append("x", {"i": 1}), a.append("x", {"i": 1}), a.append("x", {"i": 2})]
got = wal.ordered(evs)
assert [(e.sequence, e.worker_id) for e in got] == [(1, "worker-a"), (1, "worker-b"), (2, "worker-a")]
def test_a_payload_with_no_json_form_is_refused(tmp_path):
with pytest.raises(ValueError):
ResearchWal.open(str(tmp_path), "run").append("step", {"loss": float("nan")})
pkg/hanzo-research/tests/test_wire.py
+68
View File
@@ -0,0 +1,68 @@
"""The wire records hold the crate's (and the server's) fields, and read the server's answers tolerantly."""
from __future__ import annotations
import os
import json
import math
from fake import jwt
from hanzo_research import Run, Listing, Measure, Benchmark, Comparison, wire
VECTORS = json.load(open(os.path.join(os.path.dirname(__file__), "vectors", "rust.json"), encoding="utf-8"))
def test_a_run_has_the_crates_fields():
rust = VECTORS["records"]["run"]["value"]
assert set(Run().dump()) == set(rust)
assert set(Measure().dump()) == set(rust["measures"][0])
assert Run.load(rust).dump() == rust
def test_reads_are_tolerant():
r = Run.load({"id": "x", "measures": None, "k": None, "unknown": 1, "temperature": 0})
assert r.measures == [] and r.k is None and r.temperature == 0.0 and isinstance(r.temperature, float)
c = Comparison.load({"comparable": True, "blocks": None, "config": None, "deltas": None, "a": None})
assert c.blocks == [] and c.config == [] and c.a is None
b = Benchmark.load({"splits": [{"name": "dev", "items": 3}], "metrics": None})
assert b.splits[0].items == 3 and b.splits[0].definition == "" and b.metrics == []
assert Listing.load({"data": [{"id": "x"}], "total": 1}).completion is None
def test_completion_is_derived_as_the_server_derives_it():
assert wire.completion(Run(questions=10, answered=10, ended="2026-10-15T00:00:00Z")) == "complete"
assert wire.completion(Run(questions=10, answered=10)) == "unknown"
assert wire.completion(Run(questions=10, answered=7, ended="t")) == "partial"
assert wire.completion(Run(questions=10, answered=11, ended="t")) == "inconsistent"
assert wire.completion(Run(questions=10)) == "unknown"
def test_put_records_only_finite_numbers():
r = Run()
r.put("all", "acc", 0.5, n=2, of=4).put("all", "nan", math.nan).put("all", "inf", math.inf)
r.put("all", "bool", True).put("all", "text", "0.5").put("all", "int", 3, lo=2.0, hi=4.0)
assert [m.metric for m in r.measures] == ["acc", "int"]
assert r.measure("all", "acc") == Measure("all", "acc", 0.5, None, None, 2, 4)
assert r.measure("all", "int").value == 3.0 and r.measure("x", "y") is None
def test_dump_passes_a_mapping_through():
d = {"id": "x", "measures": [{"category": "all"}]}
assert wire.dump(d) == d and wire.dump([Run(id="y"), d])[1] == d
def test_token_claims():
t = jwt({"owner": "hanzo", "project": "atlas", "exp": 4102444800})
assert wire.claims(t)["owner"] == "hanzo"
assert wire.claimed(t) == "atlas"
assert wire.claimed(jwt({"owner": "hanzo"})) == "default"
assert wire.claimed("sk-opaque") is None and wire.claims("a.b.c") is None
assert wire.left(t) > 0 and wire.left(jwt({})) is None
assert wire.scope(" ") == "default" and wire.scope("x") == "x"
assert wire.now().endswith("Z") and len(wire.now()) == 20
def test_token_reads_the_environment_first(monkeypatch):
monkeypatch.undo()
monkeypatch.setenv("HANZO_API_KEY", " sk-env ")
assert wire.token() == "sk-env"
pkg/hanzo-research/tests/vectors/gen/Cargo.toml
+14
View File
@@ -0,0 +1,14 @@
# The canonical-JSON, record and WAL vectors in ../rust.json and ../wal.jsonl, as the Rust
# crate writes them. Regenerate: cargo run --release --manifest-path tests/vectors/gen/Cargo.toml -- tests/vectors
# (it writes rust.json and wal/wal.jsonl; move the log to wal.jsonl).
[package]
name = "vectors"
version = "0.0.0"
edition = "2021"
publish = false
[dependencies]
hanzo-research = { git = "https://github.com/hanzoai/ml", rev = "7c967bf79ccb0605355df5522746fecfd3d4abec" }
serde_json = "1"
[workspace]
pkg/hanzo-research/tests/vectors/gen/src/main.rs
+66
View File
@@ -0,0 +1,66 @@
//! Canonical JSON vectors computed by the hanzo-research crate, for the Python port's tests.
use hanzo_research::lineage::{DataPurpose, EnvironmentV1, LineagePolicy, ResearchEvent};
use hanzo_research::{rfc8785_canonicalize, rfc8785_digest, Measure, ResearchWal, Run};
use serde_json::{json, Value};
use std::collections::BTreeMap;
fn main() {
let out = std::env::args().nth(1).expect("out dir");
let values: Vec<(&str, Value)> = vec![
("sorting", json!({"b": 2, "a": 1, "c": {"z": true, "x": false}})),
("empty", json!({"o": {}, "a": [], "s": "", "n": null})),
("escapes", json!({"s": "q\" b\\ s/ \u{8}\u{c}\n\r\t \u{1} \u{1f} \u{7f} é 😀 \u{2028}"})),
("utf16", json!({"\u{e9}": 1, "\u{1f600}": 2, "\u{ff61}": 3, "a": 4, "\u{10000}": 5, "\u{ffff}": 6})),
("ints", json!([0, 1, -1, 42, 9007199254740991i64, -9007199254740991i64])),
("floats", json!([0.5, 0.1, -1.5, 1.0, 100.0, 3.14159, 123456.789, 0.000001, 1e20, 2.5e-5, 0.8734610296764017])),
("nested", json!([{"b": [1, {"d": 1, "c": 2}], "a": "x"}, [[], {}]])),
];
let mut vecs = Vec::new();
for (name, v) in &values {
let bytes = rfc8785_canonicalize(v).unwrap();
vecs.push(json!({"name": name, "value": v, "canonical": String::from_utf8(bytes).unwrap(), "digest": rfc8785_digest(v).unwrap()}));
}
// Where the crate's numbers part from RFC 8785 (ES Number::toString): recorded, not asserted equal.
let mut apart = Vec::new();
for x in [1e21f64, 1e-7, 1.5e300, -0.0, 5e-324] {
apart.push(json!({"value": x, "canonical": String::from_utf8(rfc8785_canonicalize(&x).unwrap()).unwrap()}));
}
for x in [9007199254740993u64, u64::MAX] {
apart.push(json!({"value": x, "canonical": String::from_utf8(rfc8785_canonicalize(&x).unwrap()).unwrap()}));
}
let run = Run {
project: "default".into(), id: "demo.bench-dev-cfg-sys".into(), benchmark: "demo.bench".into(),
split: "dev".into(), system: "sys".into(), version: "m@abc".into(), k: Some(10), temperature: Some(0.0),
commit: "0123456789abcdef0123456789abcdef01234567".into(), questions: Some(100), answered: Some(100),
when: "2026-10-15T00:00:00Z".into(), ended: "2026-10-15T00:10:00Z".into(), by: "dgx".into(),
measures: vec![Measure { category: "all".into(), metric: "accuracy".into(), value: 0.81, lo: Some(0.72), hi: Some(0.88), n: Some(100), of: Some(100) }],
notes: "é 😀".into(), revision: "original".into(), ..Run::default()
};
let env = EnvironmentV1 {
git_sha: "abc".into(), git_branch: "main".into(), git_dirty: true, host: "dgx".into(), os: "Linux".into(),
kernel_version: "7.0.0".into(), cargo_lock_hash: "".into(), image_digest: "".into(), precision_config: "bf16".into(),
determinism_config: "".into(), compiler_version: "rustc".into(),
lib_versions: BTreeMap::from([("hanzo-ml".to_string(), "0.11.112".to_string())]),
};
let ev = ResearchEvent::new("run-1".into(), "worker-0".into(), "run-1-worker-0-1-step".into(), 1,
"0".repeat(64), json!({"step": 1, "loss": 1.25}), "2026-10-15T00:00:00Z".into()).unwrap();
let policy = LineagePolicy::derive(DataPurpose::TeacherMining, true, &[&LineagePolicy::root(DataPurpose::Train, true)]);
let records = json!({
"run": {"value": serde_json::to_value(&run).unwrap(), "digest": rfc8785_digest(&run).unwrap()},
"environment": {"value": serde_json::to_value(&env).unwrap(), "digest": env.digest().unwrap()},
"event": {"value": serde_json::to_value(&ev).unwrap(), "digest": ev.digest().unwrap()},
"policy": {"value": serde_json::to_value(&policy).unwrap(), "digest": rfc8785_digest(&policy).unwrap()},
});
let dir = std::path::Path::new(&out);
let _ = std::fs::remove_dir_all(dir.join("wal"));
let mut wal = ResearchWal::open(&dir.join("wal"), "run-1").unwrap();
wal.append("step", &json!({"step": 1, "loss": 1.25})).unwrap();
wal.append("step", &json!({"step": 2, "loss": 1.1})).unwrap();
wal.append("eval", &json!({"accuracy": 0.81, "split": "dev", "é": "😀"})).unwrap();
let doc = json!({
"crate": "hanzo-research", "source": "hanzoai/ml hanzo-research @ 7c967bf79",
"vectors": vecs, "apart": apart, "records": records,
"wal": {"last": wal.last_event_hash(), "sequence": wal.sequence()},
});
std::fs::write(dir.join("rust.json"), serde_json::to_string_pretty(&doc).unwrap() + "\n").unwrap();
}
pkg/hanzo-research/tests/vectors/rust.json
+228
View File
@@ -0,0 +1,228 @@
{
"apart": [
{
"canonical": "1000000000000000000000",
"value": 1e+21
},
{
"canonical": "0.0000001",
"value": 1e-7
},
{
"canonical": "1500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"value": 1.5e+300
},
{
"canonical": "-0",
"value": -0.0
},
{
"canonical": "0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005",
"value": 5e-324
},
{
"canonical": "9007199254740993",
"value": 9007199254740993
},
{
"canonical": "18446744073709551615",
"value": 18446744073709551615
}
],
"crate": "hanzo-research",
"records": {
"environment": {
"digest": "e5d401891cba38723a93a37a4bcd10bc080eb624b1460f920f220e95df5327f7",
"value": {
"cargo_lock_hash": "",
"compiler_version": "rustc",
"determinism_config": "",
"git_branch": "main",
"git_dirty": true,
"git_sha": "abc",
"host": "dgx",
"image_digest": "",
"kernel_version": "7.0.0",
"lib_versions": {
"hanzo-ml": "0.11.112"
},
"os": "Linux",
"precision_config": "bf16"
}
},
"event": {
"digest": "15830f5752208bbde0d46ae382e6d90f603af4dd1637822440a5308e98e8e654",
"value": {
"created_at": "2026-10-15T00:00:00Z",
"event_id": "run-1-worker-0-1-step",
"payload": {
"loss": 1.25,
"step": 1
},
"payload_hash": "fb017ac5a6a9a378f10ae0b180ed416b72e1a60375b0877e38c7a504ae30482e",
"previous_event_hash": "0000000000000000000000000000000000000000000000000000000000000000",
"run_id": "run-1",
"sequence": 1,
"worker_id": "worker-0"
}
},
"policy": {
"digest": "8245875bf205b34358d1302e5f78e13beb91e65a0ab2da1b38fec08f95ce2b9e",
"value": {
"derivation_allowed": true,
"purpose": "teacher_mining"
}
},
"run": {
"digest": "cd91b801cdc4d8cbdf7b0cd2932a2cea55de052974097b9bfac0238bc65a5896",
"value": {
"answered": 100,
"baseline": false,
"benchmark": "demo.bench",
"by": "dgx",
"canonical": false,
"commit": "0123456789abcdef0123456789abcdef01234567",
"completion": "",
"dataset_digest": "",
"embedder": "",
"ended": "2026-10-15T00:10:00Z",
"facts_digest": "",
"id": "demo.bench-dev-cfg-sys",
"k": 10,
"max_tokens": null,
"measures": [
{
"category": "all",
"hi": 0.88,
"lo": 0.72,
"metric": "accuracy",
"n": 100,
"of": 100,
"value": 0.81
}
],
"notes": "é 😀",
"project": "default",
"prompt": "",
"prompt_digest": "",
"publishable": false,
"questions": 100,
"reader": "",
"revision": "original",
"split": "dev",
"store_digest": "",
"study": "",
"system": "sys",
"temperature": 0.0,
"trainable": false,
"version": "m@abc",
"visibility": "",
"when": "2026-10-15T00:00:00Z"
}
}
},
"source": "hanzoai/ml hanzo-research @ 7c967bf79",
"vectors": [
{
"canonical": "{\"a\":1,\"b\":2,\"c\":{\"x\":false,\"z\":true}}",
"digest": "879f06e22167e9fb8c4c911a9682ed6df15cfd47e551347b97c2f852e95c8633",
"name": "sorting",
"value": {
"a": 1,
"b": 2,
"c": {
"x": false,
"z": true
}
}
},
{
"canonical": "{\"a\":[],\"n\":null,\"o\":{},\"s\":\"\"}",
"digest": "a0e64a9d54eb19d4d01f5430bb54ea80575a24a5436ed3d3a64a46a90f52267f",
"name": "empty",
"value": {
"a": [],
"n": null,
"o": {},
"s": ""
}
},
{
"canonical": "{\"s\":\"q\\\" b\\\\ s/ \\b\\f\\n\\r\\t \\u0001 \\u001f  é 😀 
\"}",
"digest": "a4f961f790939cd0504b7b542a6f99b1f82c561d5b85c36c39aacc3a8f237bfe",
"name": "escapes",
"value": {
"s": "q\" b\\ s/ \b\f\n\r\t \u0001 \u001f  é 😀 
"
}
},
{
"canonical": "{\"a\":4,\"é\":1,\"𐀀\":5,\"😀\":2,\"｡\":3,\"￿\":6}",
"digest": "2fde9af60a65073b84e85627130d5597de7313a31d3832dbf1d866c987506f6d",
"name": "utf16",
"value": {
"a": 4,
"é": 1,
"｡": 3,
"￿": 6,
"𐀀": 5,
"😀": 2
}
},
{
"canonical": "[0,1,-1,42,9007199254740991,-9007199254740991]",
"digest": "0339c6206cc3dd2c9dd28b6fbd5d8d1bba2eac60315c3605c0cd897f24d26b6c",
"name": "ints",
"value": [
0,
1,
-1,
42,
9007199254740991,
-9007199254740991
]
},
{
"canonical": "[0.5,0.1,-1.5,1,100,3.14159,123456.789,0.000001,100000000000000000000,0.000025,0.8734610296764017]",
"digest": "7657ff45b99b19920c4d7ac27cd79da2004a2be9fc7e09dc13d93f2560f2a768",
"name": "floats",
"value": [
0.5,
0.1,
-1.5,
1.0,
100.0,
3.14159,
123456.789,
1e-6,
1e+20,
0.000025,
0.8734610296764017
]
},
{
"canonical": "[{\"a\":\"x\",\"b\":[1,{\"c\":2,\"d\":1}]},[[],{}]]",
"digest": "ec08c4a540df2ee848163a6feae581f786a57051d3c674bffbc6d97296780dcf",
"name": "nested",
"value": [
{
"a": "x",
"b": [
1,
{
"c": 2,
"d": 1
}
]
},
[
[],
{}
]
]
}
],
"wal": {
"last": "7a9b8a8c1076a65703699887094e933bbd684e4dfdfdbfc83c1de2617af9ba93",
"sequence": 3
}
}
pkg/hanzo-research/tests/vectors/wal.jsonl
+3
View File
@@ -0,0 +1,3 @@
{"run_id":"run-1","worker_id":"worker-0","event_id":"run-1-worker-0-1-step","sequence":1,"previous_event_hash":"0000000000000000000000000000000000000000000000000000000000000000","payload_hash":"fb017ac5a6a9a378f10ae0b180ed416b72e1a60375b0877e38c7a504ae30482e","payload":{"loss":1.25,"step":1},"created_at":"2026-10-15T18:47:26Z"}
{"run_id":"run-1","worker_id":"worker-0","event_id":"run-1-worker-0-2-step","sequence":2,"previous_event_hash":"dca12f30e0f231a7b153da8f7bb54df2a19b950f535a137a3ffb9016a12832a1","payload_hash":"1598c46c5d732737c0f2b08688fe0f6f16bf44c79b498e6f5e98dac3ec57b55e","payload":{"loss":1.1,"step":2},"created_at":"2026-10-15T18:47:26Z"}
{"run_id":"run-1","worker_id":"worker-0","event_id":"run-1-worker-0-3-eval","sequence":3,"previous_event_hash":"ec20e82ff009461471828dc30955b8ccc1bdba2e624f152bdd47020eeb64dc06","payload_hash":"293df84819096b7ac185f1c58fc24cae6ee521bae499e7aced8af95c8d99fcb0","payload":{"accuracy":0.81,"split":"dev","é":"😀"},"created_at":"2026-10-15T18:47:26Z"}