key(run) is the sha256 the cloud derives as a run's key: benchmark,
split, system, version, k, temperature, max_tokens, embedder, reader and
the four digests, each as text and NUL-terminated. tests/test_measure.py
holds the vectors hanzo-inc/cloud's apps/research TestKeyVector does.
Run keeps the crate's fields; the key is computed, not carried.
Research.measure(identity, fn, force=False) asks for a complete run under
that key and returns the latest to end without calling fn. A miss, a
partial run or force calls fn, records the run and returns it as read
back. fn may not change an identity field.
Arena.post_run follows the arena: it answers the latest run held per
benchmark, or that nothing was queued, and spends nothing, so confirm,
endpoint and attempts go. Admission becomes Measurements.
Claude-Session: https://claude.ai/code/session_01M8RoubcPN71KgJuZ8LdBUL
A review of 0.2.0 found the client reading identity and scope other
than the server does. The acting org is now the server's: a person's
home is the first of its signed orgs, not owner (the org of the app it
signed in through); a machine token acts in its owner alone; an API key
acts in its own org and project default. A switch the server would not
honour is refused before the request, a write answered under another
project raises, and a bearer token goes over http to this machine only.
A run id is unique only within a project, so get_run reads the client's
project and revise and retract refuse a record from another one; revise
carries the run's grants to the new version, which lands private; and
both refuse a version the server already holds. A claim needs a score,
and post_run needs confirm=True, nothing merely truthy.
capture() pins the contents of untracked files and records unknown, not
clean, when git fails; a folder with SHA256SUMS is checked against it;
listings escape names as sha256sum does; a remote keeps no userinfo. A
torn last WAL line is cut on open. The CLI's public read is --public.
The Python client now reaches what the Rust crate does and the server
serves. Research covers every /v1/research route: record() files
benchmarks, studies, runs and papers in one batch a kind; runs(),
get_run() and compare() read them; grant(), revise() and retract() move
consent and versions; the experiment plane and artifacts as before.
Arena covers /v1/benchmark: catalog, leaderboard, claims (written with
protocol and source), history, compare, presets, and runs, which queue a
paid execution and are refused without confirm=True.
wire holds the crate's Run, Benchmark, Measure, Study, Paper and
Comparison field for field; canonical writes RFC 8785 bytes equal to the
crate's (tests/vectors/rust.json, written by the crate); lineage and wal
are the crate's records, and a WAL the crate wrote verifies here.
capture() reads the harness commit (the diff's SHA-256 when dirty), the
weights' SHA-256, host, CPU, GPU, device, dtype and packages, and
apply() folds them into a run.
runs(org=) sends no credential: the server answers ?org= only to a
caller with no org. An org the token does not belong to is refused
before the request, since the server would act in the home org. The
command is hanzo-research; the native CLI owns `hanzo`.
The research client spoke only the experiments wire. The record
hanzo.ai/benchmarks draws from is /v1/research/{benchmarks,runs,studies,
papers,compare}, and producers posted it by hand (Kai's bench/code
post.py). record() posts each non-empty batch under its own key; runs(),
benchmarks(), studies() and papers() read with the API's own filters,
org= included for another org's public runs; compare() reads two runs
side by side. Tested against a stand-in server and read live.
Claude-Session: https://claude.ai/code/session_01M8RoubcPN71KgJuZ8LdBUL
zeekay