python-sdk/pkg/hanzo-research

hanzo-research

File a measurement made in Python to Hanzo's research plane (/v1/research) and its benchmark arena (/v1/benchmark), with what it takes to believe or repeat it: the harness commit, the model and its weights' SHA-256, the host, CPU and GPU, the device and dtype, the Python and package versions, the date. The Python peer of the Rust crate hanzo-research (hanzoai/ml): the same wire records, the same RFC 8785 digests, the same lineage and WAL records. Standard library only.

pip install hanzo-research        # or: uv run --with hanzo-research python measure.py

The credential is HANZO_API_KEY, else what hanzo auth token prints (a CLI token is fetched again when it expires). Records land in the org the server reads from the token (a person's home org, the first of its signed orgs; a machine token's owner; an API key's own org) and in the token's project (its project claim, else default). Research(org="lux") acts in another org a person's token belongs to (X-Org-Id); a switch the server would not honour, where it would file under the home org instead, is refused before any request, as is a credential sent over plain http to another machine.

Worked example

Measure something, file it with its provenance, make it public, read it back.

import hashlib
import platform
import time

from hanzo_research import Benchmark, Research, Run, Split, capture, now

r = Research()
bench = Benchmark(
    id="demo.sha256",
    title="SHA-256 over 16 MiB",
    splits=[Split("16mib", 30, "30 hashes of 16 MiB of zeros, one process, after one discarded")],
    metric="p50_ms",
    metrics=["p50_ms", "p95_ms", "mb_s"],
)
p = capture(device="cpu")  # harness commit, host, CPU, GPU, Python, packages, date
start = now()
data = bytes(16 << 20)
hashlib.sha256(data).digest()
ms = []
for _ in range(30):
    t = time.perf_counter()
    hashlib.sha256(data).digest()
    ms.append(1000 * (time.perf_counter() - t))
ms.sort()

run = Run(
    id=f"demo.sha256-16mib-cpu-{p.host}",
    benchmark=bench.id,
    split="16mib",
    system="hashlib",
    version=platform.python_version(),
    questions=30,
    answered=30,
    when=start,
    ended=now(),
)
run.put("all", "p50_ms", ms[14], n=30, of=30)
run.put("all", "p95_ms", ms[28], n=30, of=30)
run.put("all", "mb_s", 16 / (ms[14] / 1000))
p.apply(run)  # fills commit and by, appends the provenance sentence and its digest to notes

r.record(benchmarks=[bench], runs=[run])  # the definition, then the run: both private
r.grant(run.id, visibility="public", publishable=True)
back = r.get_run(run.id, org=r.org)  # the public read: no credential
print(back.completion, back.measure("all", "p50_ms").value, back.commit)

completion is the server's: complete only when answered == questions and ended is set; partial, inconsistent or unknown otherwise. A model run names its weights: capture(model="kai-1.3", weights="/models/kai-1.3", device="cuda", dtype="bf16") hashes a file, or a directory: one holding SHA256SUMS is checked against it (every file listed and matching, none unlisted) and named by that file's digest, as a Kai release is pinned; any other is digested as its sha256sum listing. apply sets system to the model and version to <model>@<sha256>. A dirty harness tree is recorded with the SHA-256 of git diff HEAD and the contents of the untracked files, so the number stays pinned to the code that made it; a git that fails records unknown, never clean.

The research plane: Research

call	route
record(benchmarks=, studies=, runs=, papers=)	POST /v1/research/{benchmarks,studies,runs,papers}, one batch a non-empty kind, in that order: {"runs": [...]}
benchmarks(id=, project=), get_benchmark(id)	GET /v1/research/benchmarks
runs(id=, key=, benchmark=, system=, study=, completion=, since=, until=, project=)	GET /v1/research/runs: your org's runs, private ones included, with the completion census and the fields runs leave unstated
get_run(id, project=)	one canonical run in a project of your org (default the client's: an id is unique only within a project)
runs(org=...), get_run(id, org=...)	the same route with ?org= and no credential: that org's public runs
studies(...), papers(..., cites=)	GET /v1/research/{studies,papers}
compare(a, b)	GET /v1/research/compare: the reasons the two runs may not be compared, the config they differ in, the deltas
grant(id, visibility=, publishable=, trainable=), grant(sha256=, visibility=)	POST /v1/research/grants: on every version of the id; a write is always private until this
revise(record)	a new version marked corrected, the run's grants carried to it (a new version lands private)
retract(id, why)	a new version marked retracted: the id leaves every listing, earlier versions stay
ingest, query, totals, projects, experiment(...)	the experiment plane: /v1/research/{experiments,totals,projects}
upload_artifact, upload_json_artifact, artifacts, get_artifact	/v1/research/artifacts[/<sha256>]: snapshot or report bytes, addressed by the server's hash
url(run_id, org=)	the public read address of one run: the source a claim cites
measure(identity, fn, force=False)	GET /v1/research/runs?key=&completion=complete, then fn and POST /v1/research/runs only on a miss or with force

Every write is a batch of at most 20,000 and is idempotent by content: the same record again records nothing. A mapping is sent as given; a Run is sent with every field. A refusal raises Denied (401, 403), Missing (404), Invalid (400, 409, 422), Limited (413, 429) or Unavailable (502 to 504, no answer), each carrying the server's own sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves: json refuses it before the request.

Measure once

A run's key is the SHA-256 of what decides its number: benchmark, split, system, version, k, temperature, max_tokens, embedder, reader and the four digests (IDENTITY), and nothing about who ran it, when, or at which commit. The server derives it on every run and filters on it; key(run) computes the same bytes before anything is spent. measure asks for a complete run under that key and calls fn only on a miss:

identity = Run(
    id="locomo-all-k20-enso-flash",
    benchmark="locomo",
    split="all",
    system="enso-flash",
    version="1.2.0",
    k=20,
    temperature=0.0,
    dataset_digest=digest,
)
run = r.measure(identity, lambda: evaluate())  # returns measures, questions, answered, ended
r.measure(identity, evaluate, force=True)  # measure again regardless
r.runs(key=key(identity))  # every run of that measurement

A partial run is not a hit. fn may not change an identity field: a result filed under a configuration it did not measure is the false hit this exists to prevent.

The arena: Arena (or Research().arena)

call	route
catalog()	GET /v1/benchmark/catalog
leaderboard(benchmark)	GET /v1/benchmark/leaderboard: each model's measured score (its latest run) beside a published claim. Only claims a SuperAdmin wrote under org admin, and public, reach it.
claims(benchmark=, model=, provider=, source=, protocol=, org=)	GET /v1/benchmark/claims: every org's public claims, labelled, and your org's private ones
post_claims(rows, visibility=)	POST /v1/benchmark/claims: under your acting org; each row names its protocol and its source (for a measured score, Research.url(run_id)), or it is refused here
history(benchmark, model=), compare(a, b, benchmark=)	GET /v1/benchmark/{history,compare}: read the arena's per-item attempts plane, not the research runs
presets(), post_preset(preset)	GET/POST /v1/benchmark/presets: validated and echoed as enso-<name>, not stored
post_run(benchmarks, model)	POST /v1/benchmark/runs: per benchmark, the model's latest run the arena holds, or that nothing was queued (the arena has no executor); spends nothing

The command

hanzo-research (or python -m hanzo_research); every answer is JSON on stdout.

hanzo-research post runs.jsonl --grant public --publishable   # a run per line
hanzo-research post plan.json                                 # {"benchmarks": [...], "runs": [...]}
hanzo-research post plan.json --dry                           # print the bodies, file nothing
hanzo-research list runs --benchmark demo.sha256
hanzo-research list runs --public hanzo                       # hanzo's public runs, no credential
hanzo-research get demo.sha256-16mib-cpu-dgx --public hanzo
hanzo-research grant RUN_ID ... --visibility public --publishable
hanzo-research retract RUN_ID --why "read on the wrong split"
hanzo-research compare RUN_A RUN_B
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
hanzo-research arena leaderboard --benchmark gpqa_diamond

It is not hanzo research: the hanzo command is the native CLI, and no Python package in this workspace may install one.

Records beside the API

• canonical.canonicalize(value), canonical.digest(value): RFC 8785 bytes and their SHA-256, equal to the crate's rfc8785_canonicalize/rfc8785_digest on every string, integer within 2^53 and float from 1e-6 to 1e21. Past that the crate writes Rust's Display (1e21 as 22 digits, -0.0 as -0), which is not RFC 8785; this package writes the RFC's 1e+21 and 0. tests/vectors/rust.json holds the crate's own output.
• lineage: DataPurpose, LineagePolicy (a child never less restrictive than a parent), ContaminationError, verify_promotion_eligibility, and the crate's evidence records (EnvironmentV1, GateSpec, Claim, TeacherInteraction, PairEvidence, DeterminismReport, ArtifactRef, ResearchEvent), each with the crate's digest.
• wal.ResearchWal: an fsynced, hash-chained event log. A log the crate wrote verifies here (wal.verify), and the reverse.

Tests

uv run --no-project --with pytest pytest pkg/hanzo-research              # no network
uv run --no-project --with pytest pytest pkg/hanzo-research -m integration  # api.hanzo.ai, no credential