mirror of
https://github.com/hanzoai/python-sdk.git
synced 2026-10-23 20:23:00 +00:00
hanzo-research 0.2.0: every research and arena route, the crate's records, provenance
The Python client now reaches what the Rust crate does and the server serves. Research covers every /v1/research route: record() files benchmarks, studies, runs and papers in one batch a kind; runs(), get_run() and compare() read them; grant(), revise() and retract() move consent and versions; the experiment plane and artifacts as before. Arena covers /v1/benchmark: catalog, leaderboard, claims (written with protocol and source), history, compare, presets, and runs, which queue a paid execution and are refused without confirm=True. wire holds the crate's Run, Benchmark, Measure, Study, Paper and Comparison field for field; canonical writes RFC 8785 bytes equal to the crate's (tests/vectors/rust.json, written by the crate); lineage and wal are the crate's records, and a WAL the crate wrote verifies here. capture() reads the harness commit (the diff's SHA-256 when dirty), the weights' SHA-256, host, CPU, GPU, device, dtype and packages, and apply() folds them into a run. runs(org=) sends no credential: the server answers ?org= only to a caller with no org. An org the token does not belong to is refused before the request, since the server would act in the home org. The command is hanzo-research; the native CLI owns `hanzo`.
This commit is contained in:
@@ -11,6 +11,7 @@
|
||||
"embeddings",
|
||||
"models",
|
||||
"files",
|
||||
"research",
|
||||
"---Agent SDK---",
|
||||
"...agents",
|
||||
"---MCP---",
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
title: Research
|
||||
description: File a measurement made in Python to /v1/research and /v1/benchmark with its provenance
|
||||
---
|
||||
|
||||
# Research
|
||||
|
||||
`pip install hanzo-research` ([pkg/hanzo-research](https://github.com/hanzoai/python-sdk/tree/main/pkg/hanzo-research)).
|
||||
|
||||
File a measurement made in Python to Hanzo's research plane (`/v1/research`) and its
|
||||
benchmark arena (`/v1/benchmark`), with what it takes to believe or repeat it: the
|
||||
harness commit, the model and its weights' SHA-256, the host, CPU and GPU, the device and
|
||||
dtype, the Python and package versions, the date. The Python peer of the Rust crate
|
||||
`hanzo-research` (hanzoai/ml): the same wire records, the same RFC 8785 digests, the same
|
||||
lineage and WAL records. Standard library only.
|
||||
|
||||
```bash
|
||||
pip install hanzo-research # or: uv run --with hanzo-research python measure.py
|
||||
```
|
||||
|
||||
The credential is `HANZO_API_KEY`, else what `hanzo auth token` prints (a CLI token is
|
||||
fetched again when it expires). Records land in the token's org (its `owner`) and project
|
||||
(its `project` claim, else `default`). `Research(org="lux")` acts in another org the
|
||||
token belongs to (`X-Org-Id`); an org it does not belong to is refused before any request,
|
||||
since the server would file under the home org instead.
|
||||
|
||||
## Worked example
|
||||
|
||||
Measure something, file it with its provenance, make it public, read it back.
|
||||
|
||||
```python
|
||||
import hashlib
|
||||
import platform
|
||||
import time
|
||||
|
||||
from hanzo_research import Benchmark, Research, Run, Split, capture, now
|
||||
|
||||
r = Research()
|
||||
bench = Benchmark(
|
||||
id="demo.sha256",
|
||||
title="SHA-256 over 16 MiB",
|
||||
splits=[Split("16mib", 30, "30 hashes of 16 MiB of zeros, one process, after one discarded")],
|
||||
metric="p50_ms",
|
||||
metrics=["p50_ms", "p95_ms", "mb_s"],
|
||||
)
|
||||
p = capture(device="cpu") # harness commit, host, CPU, GPU, Python, packages, date
|
||||
start = now()
|
||||
data = bytes(16 << 20)
|
||||
hashlib.sha256(data).digest()
|
||||
ms = []
|
||||
for _ in range(30):
|
||||
t = time.perf_counter()
|
||||
hashlib.sha256(data).digest()
|
||||
ms.append(1000 * (time.perf_counter() - t))
|
||||
ms.sort()
|
||||
|
||||
run = Run(
|
||||
id=f"demo.sha256-16mib-cpu-{p.host}",
|
||||
benchmark=bench.id,
|
||||
split="16mib",
|
||||
system="hashlib",
|
||||
version=platform.python_version(),
|
||||
questions=30,
|
||||
answered=30,
|
||||
when=start,
|
||||
ended=now(),
|
||||
)
|
||||
run.put("all", "p50_ms", ms[14], n=30, of=30)
|
||||
run.put("all", "p95_ms", ms[28], n=30, of=30)
|
||||
run.put("all", "mb_s", 16 / (ms[14] / 1000))
|
||||
p.apply(run) # fills commit and by, appends the provenance sentence and its digest to notes
|
||||
|
||||
r.record(benchmarks=[bench], runs=[run]) # the definition, then the run: both private
|
||||
r.grant(run.id, visibility="public", publishable=True)
|
||||
back = r.get_run(run.id, org=r.org) # the public read: no credential
|
||||
print(back.completion, back.measure("all", "p50_ms").value, back.commit)
|
||||
```
|
||||
|
||||
`completion` is the server's: `complete` only when `answered == questions` and `ended` is
|
||||
set; `partial`, `inconsistent` or `unknown` otherwise. A model run names its weights:
|
||||
`capture(model="kai-1.3", weights="/models/kai-1.3", device="cuda", dtype="bf16")` hashes a
|
||||
file, or a directory by its `SHA256SUMS` (else every file under it), and `apply` sets
|
||||
`system` to the model and `version` to `<model>@<sha256>`. A dirty harness tree is recorded
|
||||
with the SHA-256 of `git diff HEAD` and the untracked files, so the number stays pinned to
|
||||
the code that made it.
|
||||
|
||||
## The research plane: `Research`
|
||||
|
||||
| call | route |
|
||||
|---|---|
|
||||
| `record(benchmarks=, studies=, runs=, papers=)` | `POST /v1/research/{benchmarks,studies,runs,papers}`, one batch a non-empty kind, in that order: `{"runs": [...]}` |
|
||||
| `benchmarks(id=, project=)`, `get_benchmark(id)` | `GET /v1/research/benchmarks` |
|
||||
| `runs(id=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
|
||||
| `runs(org=...)`, `get_run(id, org=...)` | the same route with `?org=` and no credential: that org's public runs |
|
||||
| `studies(...)`, `papers(..., cites=)` | `GET /v1/research/{studies,papers}` |
|
||||
| `compare(a, b)` | `GET /v1/research/compare`: the reasons the two runs may not be compared, the config they differ in, the deltas |
|
||||
| `grant(id, visibility=, publishable=, trainable=)`, `grant(sha256=, visibility=)` | `POST /v1/research/grants`: on every version of the id; a write is always private until this |
|
||||
| `revise(record)` | a new version marked `corrected` |
|
||||
| `retract(id, why)` | a new version marked `retracted`: the id leaves every listing, earlier versions stay |
|
||||
| `ingest`, `query`, `totals`, `projects`, `experiment(...)` | the experiment plane: `/v1/research/{experiments,totals,projects}` |
|
||||
| `upload_artifact`, `upload_json_artifact`, `artifacts`, `get_artifact` | `/v1/research/artifacts[/<sha256>]`: `snapshot` or `report` bytes, addressed by the server's hash |
|
||||
| `url(run_id, org=)` | the public read address of one run: the `source` a claim cites |
|
||||
|
||||
Every write is a batch of at most 20,000 and is idempotent by content: the same record
|
||||
again records nothing. A mapping is sent as given; a `Run` is sent with every field. A
|
||||
refusal raises `Denied` (401, 403), `Missing` (404), `Invalid` (400, 409, 422), `Limited`
|
||||
(413, 429) or `Unavailable` (502 to 504, no answer), each carrying the server's own
|
||||
sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves:
|
||||
`json` refuses it before the request.
|
||||
|
||||
## The arena: `Arena` (or `Research().arena`)
|
||||
|
||||
| call | route |
|
||||
|---|---|
|
||||
| `catalog()` | `GET /v1/benchmark/catalog` |
|
||||
| `leaderboard(benchmark)` | `GET /v1/benchmark/leaderboard`: each model's measured score (its latest run) beside a published claim. Only claims a SuperAdmin wrote under org `admin`, and public, reach it. |
|
||||
| `claims(benchmark=, model=, provider=, source=, protocol=, org=)` | `GET /v1/benchmark/claims`: every org's public claims, labelled, and your org's private ones |
|
||||
| `post_claims(rows, visibility=)` | `POST /v1/benchmark/claims`: under your acting org; each row names its `protocol` and its `source` (for a measured score, `Research.url(run_id)`), or it is refused here |
|
||||
| `history(benchmark, model=)`, `compare(a, b, benchmark=)` | `GET /v1/benchmark/{history,compare}`: read the arena's per-item attempts plane, not the research runs |
|
||||
| `presets()`, `post_preset(preset)` | `GET/POST /v1/benchmark/presets`: validated and echoed as `enso-<name>`, not stored |
|
||||
| `post_run(benchmarks, model=, endpoint=, confirm=True)` | `POST /v1/benchmark/runs`: queues a paid execution, so it is refused without `confirm=True` |
|
||||
|
||||
## The command
|
||||
|
||||
`hanzo-research` (or `python -m hanzo_research`); every answer is JSON on stdout.
|
||||
|
||||
```bash
|
||||
hanzo-research post runs.jsonl --grant public --publishable # a run per line
|
||||
hanzo-research post plan.json # {"benchmarks": [...], "runs": [...]}
|
||||
hanzo-research post plan.json --dry # print the bodies, file nothing
|
||||
hanzo-research list runs --benchmark demo.sha256
|
||||
hanzo-research list runs --org hanzo # hanzo's public runs, no credential
|
||||
hanzo-research get demo.sha256-16mib-cpu-dgx --org hanzo
|
||||
hanzo-research grant RUN_ID ... --visibility public --publishable
|
||||
hanzo-research retract RUN_ID --why "read on the wrong split"
|
||||
hanzo-research compare RUN_A RUN_B
|
||||
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
|
||||
hanzo-research arena leaderboard --benchmark gpqa_diamond
|
||||
```
|
||||
|
||||
It is not `hanzo research`: the `hanzo` command is the native CLI, and no Python package
|
||||
in this workspace may install one.
|
||||
|
||||
## Records beside the API
|
||||
|
||||
- `canonical.canonicalize(value)`, `canonical.digest(value)`: RFC 8785 bytes and their
|
||||
SHA-256, equal to the crate's `rfc8785_canonicalize`/`rfc8785_digest` on every string,
|
||||
integer within 2^53 and float from 1e-6 to 1e21. Past that the crate writes Rust's
|
||||
`Display` (`1e21` as 22 digits, `-0.0` as `-0`), which is not RFC 8785; this package
|
||||
writes the RFC's `1e+21` and `0`. `tests/vectors/rust.json` holds the crate's own output.
|
||||
- `lineage`: `DataPurpose`, `LineagePolicy` (a child never less restrictive than a parent),
|
||||
`ContaminationError`, `verify_promotion_eligibility`, and the crate's evidence records
|
||||
(`EnvironmentV1`, `GateSpec`, `Claim`, `TeacherInteraction`, `PairEvidence`,
|
||||
`DeterminismReport`, `ArtifactRef`, `ResearchEvent`), each with the crate's digest.
|
||||
- `wal.ResearchWal`: an fsynced, hash-chained event log. A log the crate wrote verifies here
|
||||
(`wal.verify`), and the reverse.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run --no-project --with pytest pytest pkg/hanzo-research # no network
|
||||
uv run --no-project --with pytest pytest pkg/hanzo-research -m integration # api.hanzo.ai, no credential
|
||||
```
|
||||
Reference in New Issue
Block a user