mirror of
https://github.com/hanzoai/python-sdk.git
synced 2026-10-23 20:23:00 +00:00
hanzo-research 0.3.0: measure what the org does not already hold
key(run) is the sha256 the cloud derives as a run's key: benchmark, split, system, version, k, temperature, max_tokens, embedder, reader and the four digests, each as text and NUL-terminated. tests/test_measure.py holds the vectors hanzo-inc/cloud's apps/research TestKeyVector does. Run keeps the crate's fields; the key is computed, not carried. Research.measure(identity, fn, force=False) asks for a complete run under that key and returns the latest to end without calling fn. A miss, a partial run or force calls fn, records the run and returns it as read back. fn may not change an identity field. Arena.post_run follows the arena: it answers the latest run held per benchmark, or that nothing was queued, and spends nothing, so confirm, endpoint and attempts go. Admission becomes Measurements. Claude-Session: https://claude.ai/code/session_01M8RoubcPN71KgJuZ8LdBUL
This commit is contained in:
@@ -94,7 +94,7 @@ code that made it; a git that fails records `unknown`, never clean.
|
||||
|---|---|
|
||||
| `record(benchmarks=, studies=, runs=, papers=)` | `POST /v1/research/{benchmarks,studies,runs,papers}`, one batch a non-empty kind, in that order: `{"runs": [...]}` |
|
||||
| `benchmarks(id=, project=)`, `get_benchmark(id)` | `GET /v1/research/benchmarks` |
|
||||
| `runs(id=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
|
||||
| `runs(id=, key=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
|
||||
| `get_run(id, project=)` | one canonical run in a project of your org (default the client's: an id is unique only within a project) |
|
||||
| `runs(org=...)`, `get_run(id, org=...)` | the same route with `?org=` and no credential: that org's public runs |
|
||||
| `studies(...)`, `papers(..., cites=)` | `GET /v1/research/{studies,papers}` |
|
||||
@@ -105,6 +105,7 @@ code that made it; a git that fails records `unknown`, never clean.
|
||||
| `ingest`, `query`, `totals`, `projects`, `experiment(...)` | the experiment plane: `/v1/research/{experiments,totals,projects}` |
|
||||
| `upload_artifact`, `upload_json_artifact`, `artifacts`, `get_artifact` | `/v1/research/artifacts[/<sha256>]`: `snapshot` or `report` bytes, addressed by the server's hash |
|
||||
| `url(run_id, org=)` | the public read address of one run: the `source` a claim cites |
|
||||
| `measure(identity, fn, force=False)` | `GET /v1/research/runs?key=&completion=complete`, then `fn` and `POST /v1/research/runs` only on a miss or with `force` |
|
||||
|
||||
Every write is a batch of at most 20,000 and is idempotent by content: the same record
|
||||
again records nothing. A mapping is sent as given; a `Run` is sent with every field. A
|
||||
@@ -113,6 +114,33 @@ refusal raises `Denied` (401, 403), `Missing` (404), `Invalid` (400, 409, 422),
|
||||
sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves:
|
||||
`json` refuses it before the request.
|
||||
|
||||
## Measure once
|
||||
|
||||
A run's `key` is the SHA-256 of what decides its number: benchmark, split, system, version,
|
||||
k, temperature, max_tokens, embedder, reader and the four digests (`IDENTITY`), and nothing
|
||||
about who ran it, when, or at which commit. The server derives it on every run and filters on
|
||||
it; `key(run)` computes the same bytes before anything is spent. `measure` asks for a complete
|
||||
run under that key and calls `fn` only on a miss:
|
||||
|
||||
```python
|
||||
identity = Run(
|
||||
id="locomo-all-k20-enso-flash",
|
||||
benchmark="locomo",
|
||||
split="all",
|
||||
system="enso-flash",
|
||||
version="1.2.0",
|
||||
k=20,
|
||||
temperature=0.0,
|
||||
dataset_digest=digest,
|
||||
)
|
||||
run = r.measure(identity, lambda: evaluate()) # returns measures, questions, answered, ended
|
||||
r.measure(identity, evaluate, force=True) # measure again regardless
|
||||
r.runs(key=key(identity)) # every run of that measurement
|
||||
```
|
||||
|
||||
A partial run is not a hit. `fn` may not change an identity field: a result filed under a
|
||||
configuration it did not measure is the false hit this exists to prevent.
|
||||
|
||||
## The arena: `Arena` (or `Research().arena`)
|
||||
|
||||
| call | route |
|
||||
@@ -123,7 +151,7 @@ sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never
|
||||
| `post_claims(rows, visibility=)` | `POST /v1/benchmark/claims`: under your acting org; each row names its `protocol` and its `source` (for a measured score, `Research.url(run_id)`), or it is refused here |
|
||||
| `history(benchmark, model=)`, `compare(a, b, benchmark=)` | `GET /v1/benchmark/{history,compare}`: read the arena's per-item attempts plane, not the research runs |
|
||||
| `presets()`, `post_preset(preset)` | `GET/POST /v1/benchmark/presets`: validated and echoed as `enso-<name>`, not stored |
|
||||
| `post_run(benchmarks, model=, endpoint=, confirm=True)` | `POST /v1/benchmark/runs`: queues a paid execution, so it is refused without `confirm=True` |
|
||||
| `post_run(benchmarks, model)` | `POST /v1/benchmark/runs`: per benchmark, the model's latest run the arena holds, or that nothing was queued (the arena has no executor); spends nothing |
|
||||
|
||||
## The command
|
||||
|
||||
|
||||
@@ -87,7 +87,7 @@ code that made it; a git that fails records `unknown`, never clean.
|
||||
|---|---|
|
||||
| `record(benchmarks=, studies=, runs=, papers=)` | `POST /v1/research/{benchmarks,studies,runs,papers}`, one batch a non-empty kind, in that order: `{"runs": [...]}` |
|
||||
| `benchmarks(id=, project=)`, `get_benchmark(id)` | `GET /v1/research/benchmarks` |
|
||||
| `runs(id=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
|
||||
| `runs(id=, key=, benchmark=, system=, study=, completion=, since=, until=, project=)` | `GET /v1/research/runs`: your org's runs, private ones included, with the completion census and the fields runs leave unstated |
|
||||
| `get_run(id, project=)` | one canonical run in a project of your org (default the client's: an id is unique only within a project) |
|
||||
| `runs(org=...)`, `get_run(id, org=...)` | the same route with `?org=` and no credential: that org's public runs |
|
||||
| `studies(...)`, `papers(..., cites=)` | `GET /v1/research/{studies,papers}` |
|
||||
@@ -98,6 +98,7 @@ code that made it; a git that fails records `unknown`, never clean.
|
||||
| `ingest`, `query`, `totals`, `projects`, `experiment(...)` | the experiment plane: `/v1/research/{experiments,totals,projects}` |
|
||||
| `upload_artifact`, `upload_json_artifact`, `artifacts`, `get_artifact` | `/v1/research/artifacts[/<sha256>]`: `snapshot` or `report` bytes, addressed by the server's hash |
|
||||
| `url(run_id, org=)` | the public read address of one run: the `source` a claim cites |
|
||||
| `measure(identity, fn, force=False)` | `GET /v1/research/runs?key=&completion=complete`, then `fn` and `POST /v1/research/runs` only on a miss or with `force` |
|
||||
|
||||
Every write is a batch of at most 20,000 and is idempotent by content: the same record
|
||||
again records nothing. A mapping is sent as given; a `Run` is sent with every field. A
|
||||
@@ -106,6 +107,33 @@ refusal raises `Denied` (401, 403), `Missing` (404), `Invalid` (400, 409, 422),
|
||||
sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves:
|
||||
`json` refuses it before the request.
|
||||
|
||||
## Measure once
|
||||
|
||||
A run's `key` is the SHA-256 of what decides its number: benchmark, split, system, version,
|
||||
k, temperature, max_tokens, embedder, reader and the four digests (`IDENTITY`), and nothing
|
||||
about who ran it, when, or at which commit. The server derives it on every run and filters on
|
||||
it; `key(run)` computes the same bytes before anything is spent. `measure` asks for a complete
|
||||
run under that key and calls `fn` only on a miss:
|
||||
|
||||
```python
|
||||
identity = Run(
|
||||
id="locomo-all-k20-enso-flash",
|
||||
benchmark="locomo",
|
||||
split="all",
|
||||
system="enso-flash",
|
||||
version="1.2.0",
|
||||
k=20,
|
||||
temperature=0.0,
|
||||
dataset_digest=digest,
|
||||
)
|
||||
run = r.measure(identity, lambda: evaluate()) # returns measures, questions, answered, ended
|
||||
r.measure(identity, evaluate, force=True) # measure again regardless
|
||||
r.runs(key=key(identity)) # every run of that measurement
|
||||
```
|
||||
|
||||
A partial run is not a hit. `fn` may not change an identity field: a result filed under a
|
||||
configuration it did not measure is the false hit this exists to prevent.
|
||||
|
||||
## The arena: `Arena` (or `Research().arena`)
|
||||
|
||||
| call | route |
|
||||
@@ -116,7 +144,7 @@ sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never
|
||||
| `post_claims(rows, visibility=)` | `POST /v1/benchmark/claims`: under your acting org; each row names its `protocol` and its `source` (for a measured score, `Research.url(run_id)`), or it is refused here |
|
||||
| `history(benchmark, model=)`, `compare(a, b, benchmark=)` | `GET /v1/benchmark/{history,compare}`: read the arena's per-item attempts plane, not the research runs |
|
||||
| `presets()`, `post_preset(preset)` | `GET/POST /v1/benchmark/presets`: validated and echoed as `enso-<name>`, not stored |
|
||||
| `post_run(benchmarks, model=, endpoint=, confirm=True)` | `POST /v1/benchmark/runs`: queues a paid execution, so it is refused without `confirm=True` |
|
||||
| `post_run(benchmarks, model)` | `POST /v1/benchmark/runs`: per benchmark, the model's latest run the arena holds, or that nothing was queued (the arena has no executor); spends nothing |
|
||||
|
||||
## The command
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ RFC 8785 digests, the same lineage and WAL records.
|
||||
from . import wal, wire, lineage, benchmark, canonical, provenance
|
||||
from .wal import ResearchWal
|
||||
from .wire import (
|
||||
IDENTITY,
|
||||
Ack,
|
||||
Gap,
|
||||
Run,
|
||||
@@ -40,6 +41,7 @@ from .wire import (
|
||||
KindTotal,
|
||||
Comparison,
|
||||
Difference,
|
||||
key,
|
||||
now,
|
||||
left,
|
||||
scope,
|
||||
@@ -82,6 +84,8 @@ __all__ = [
|
||||
"Totals",
|
||||
"Summary",
|
||||
"completion",
|
||||
"IDENTITY",
|
||||
"key",
|
||||
"token",
|
||||
"claimed",
|
||||
"left",
|
||||
@@ -115,6 +119,7 @@ __all__ = [
|
||||
"studies",
|
||||
"papers",
|
||||
"compare",
|
||||
"measure",
|
||||
]
|
||||
|
||||
_default: Research | None = None
|
||||
@@ -189,3 +194,8 @@ def papers(**filters) -> list[Paper]: # type: ignore[no-untyped-def]
|
||||
def compare(a: str, b: str) -> Comparison:
|
||||
"""Two runs side by side on the default client."""
|
||||
return client().compare(a, b)
|
||||
|
||||
|
||||
def measure(identity, fn, force: bool = False) -> Run: # type: ignore[no-untyped-def]
|
||||
"""The org's complete run under `key(identity)`, else `fn`'s results recorded (see `Research.measure`)."""
|
||||
return client().measure(identity, fn, force=force)
|
||||
|
||||
@@ -13,8 +13,9 @@ What reads where:
|
||||
* `claims` shows public claims of every org, plus the caller's own org's private ones.
|
||||
* `post_claims` writes under the caller's acting org, private unless `visibility="public"`.
|
||||
Writing under org `admin` needs a SuperAdmin token (owner `admin`).
|
||||
* `post_run` asks the arena to measure a model: a PAID execution, so it is refused here
|
||||
unless `confirm=True`. The answer is an admission, not a result.
|
||||
* `post_run` asks what the arena holds for a model: per benchmark, its latest measured run,
|
||||
or that nothing is held and nothing was queued (the arena has no executor). It spends
|
||||
nothing; `Research.measure` is how a harness spends only on what is not held.
|
||||
* `post_preset` validates a blend and echoes it as `enso-<name>`; the arena stores nothing.
|
||||
"""
|
||||
|
||||
@@ -38,7 +39,8 @@ __all__ = [
|
||||
"History",
|
||||
"Claim",
|
||||
"Filed",
|
||||
"Admission",
|
||||
"Measurement",
|
||||
"Measurements",
|
||||
"Preset",
|
||||
"Accepted",
|
||||
]
|
||||
@@ -149,16 +151,23 @@ class Filed(Wire):
|
||||
|
||||
|
||||
@dataclass
|
||||
class Admission(Wire):
|
||||
"""A queued arena run: admitted, not finished."""
|
||||
class Measurement(Wire):
|
||||
"""One benchmark's answer: the model's latest run the arena holds, or why there is none."""
|
||||
|
||||
benchmark: str = ""
|
||||
latest: Point | None = None
|
||||
harness: str = ""
|
||||
note: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Measurements(Wire):
|
||||
"""What the arena holds for a model, one answer per benchmark asked about, in that order."""
|
||||
|
||||
status: str = ""
|
||||
org: str = ""
|
||||
by: str = ""
|
||||
model: str = ""
|
||||
endpoint: str = ""
|
||||
benchmarks: list[str] = field(default_factory=list)
|
||||
note: str = ""
|
||||
data: list[Measurement] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -259,20 +268,10 @@ class Arena:
|
||||
def post_preset(self, preset: Preset | Mapping[str, Any]) -> Accepted:
|
||||
return Accepted.load(self.http.post("/v1/benchmark/presets", dump(preset)))
|
||||
|
||||
def post_run(
|
||||
self,
|
||||
benchmarks: list[str],
|
||||
model: str | None = None,
|
||||
endpoint: str | None = None,
|
||||
attempts: int | None = None,
|
||||
*,
|
||||
confirm: bool = False,
|
||||
) -> Admission:
|
||||
"""Queue the arena's harness on a model or your endpoint. It spends money: `confirm=True` or it is refused."""
|
||||
if confirm is not True:
|
||||
raise Invalid("POST /v1/benchmark/runs queues a paid execution; pass confirm=True to queue it")
|
||||
body: dict[str, Any] = {"benchmarks": benchmarks}
|
||||
for k, v in (("model", model), ("endpoint", endpoint), ("attempts", attempts)):
|
||||
if v:
|
||||
body[k] = v
|
||||
return Admission.load(self.http.post("/v1/benchmark/runs", body))
|
||||
def post_run(self, benchmarks: list[str], model: str) -> Measurements:
|
||||
"""What the arena holds for `model` on each benchmark: its latest run, or that nothing was queued.
|
||||
|
||||
The arena has no executor, so this spends nothing and queues nothing; a benchmark it holds
|
||||
no run for names the harness that measures it, when one does.
|
||||
"""
|
||||
return Measurements.load(self.http.post("/v1/benchmark/runs", {"benchmarks": benchmarks, "model": model}))
|
||||
|
||||
@@ -17,7 +17,7 @@ from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
from typing import Any, Mapping, TypeVar, Iterable
|
||||
from typing import Any, Mapping, TypeVar, Callable, Iterable
|
||||
|
||||
from . import wire, canonical, provenance
|
||||
from .wire import (
|
||||
@@ -34,6 +34,7 @@ from .wire import (
|
||||
Artifact,
|
||||
Benchmark,
|
||||
Comparison,
|
||||
key,
|
||||
dump,
|
||||
)
|
||||
from .errors import Error, Invalid, Missing
|
||||
@@ -217,6 +218,7 @@ class Research:
|
||||
def runs(
|
||||
self,
|
||||
id: str | None = None,
|
||||
key: str | None = None,
|
||||
benchmark: str | None = None,
|
||||
system: str | None = None,
|
||||
study: str | None = None,
|
||||
@@ -230,10 +232,12 @@ class Research:
|
||||
|
||||
Without `org`, the caller's own org, private runs included. With `org`, that org's
|
||||
PUBLIC runs, read with no credential: the server answers `?org=` only to a caller
|
||||
that names no org of its own. `since`/`until` bound `when` (RFC 3339).
|
||||
that names no org of its own. `key` narrows to one measurement identity (`key(run)`);
|
||||
`since`/`until` bound `when` (RFC 3339).
|
||||
"""
|
||||
q = {
|
||||
"id": id,
|
||||
"key": key,
|
||||
"benchmark": benchmark,
|
||||
"system": system,
|
||||
"study": study,
|
||||
@@ -253,6 +257,37 @@ class Research:
|
||||
project = project or (None if org else self.project)
|
||||
return _one(self.runs(id=id, org=org, project=project).data, "run", id)
|
||||
|
||||
def measure(
|
||||
self,
|
||||
identity: Run | Mapping[str, Any],
|
||||
fn: Callable[[], Run | Mapping[str, Any]],
|
||||
force: bool = False,
|
||||
) -> Run:
|
||||
"""The run that measures `identity`: the org's complete one when it holds it, else `fn`'s.
|
||||
|
||||
`identity` is the run before it is measured: its id, its benchmark and whichever of
|
||||
`IDENTITY` fix the measurement. Unless `force`, the org's runs are asked for a complete
|
||||
one under `key(identity)` and the latest to end is returned; `fn` is not called and
|
||||
nothing is spent. Otherwise `fn()` measures: it returns the results (measures,
|
||||
questions, answered, ended and the rest) as a mapping laid over the identity, or the
|
||||
whole `Run`. The run is recorded and returned as the server reads it back.
|
||||
|
||||
A partial run is not a hit. `fn` may not change an identity field: a result filed
|
||||
under a configuration it did not measure is the false hit this exists to prevent.
|
||||
"""
|
||||
want = key(identity)
|
||||
if not force:
|
||||
# checked here too: a server that predates ?key= answers every run, and none is a hit
|
||||
held = [r for r in self.runs(key=want, completion="complete").data if key(r) == want]
|
||||
if held:
|
||||
return max(held, key=lambda r: (r.ended, r.when))
|
||||
got = fn()
|
||||
run = Run.load(dump(got) if isinstance(got, Run) else {**dump(identity), **dump(got)})
|
||||
if key(run) != want:
|
||||
raise Invalid(f"measure: {run.id} changed its measurement identity while it was measured")
|
||||
self.record(runs=[run])
|
||||
return _one(self.runs(id=run.id, key=want, project=self.project).data, "run", run.id)
|
||||
|
||||
def studies(self, id: str | None = None, project: str | None = None) -> list[Study]:
|
||||
out = self.http.get("/v1/research/studies", {"id": id, "project": project})
|
||||
return [Study.load(x) for x in out.get("data") or []]
|
||||
|
||||
@@ -18,6 +18,8 @@ import math
|
||||
import types
|
||||
import base64
|
||||
import typing
|
||||
import decimal
|
||||
import hashlib
|
||||
import binascii
|
||||
import datetime
|
||||
import subprocess
|
||||
@@ -47,6 +49,8 @@ __all__ = [
|
||||
"Totals",
|
||||
"Summary",
|
||||
"completion",
|
||||
"IDENTITY",
|
||||
"key",
|
||||
"load",
|
||||
"dump",
|
||||
"token",
|
||||
@@ -184,6 +188,58 @@ def completion(run: Run) -> str:
|
||||
return "complete" if run.ended.strip() else "unknown"
|
||||
|
||||
|
||||
# What decides a run's number, in the order `key` hashes it. Who ran it, when, at which commit
|
||||
# and what it found are not here: two executions of one configuration are one measurement.
|
||||
IDENTITY = (
|
||||
"benchmark",
|
||||
"split",
|
||||
"system",
|
||||
"version",
|
||||
"k",
|
||||
"temperature",
|
||||
"max_tokens",
|
||||
"embedder",
|
||||
"reader",
|
||||
"prompt_digest",
|
||||
"dataset_digest",
|
||||
"store_digest",
|
||||
"facts_digest",
|
||||
)
|
||||
|
||||
|
||||
def key(run: Run | Mapping[str, Any]) -> str:
|
||||
"""A run's measurement identity, the `key` the server derives on read and `?key=` filters on.
|
||||
|
||||
The hex SHA-256 of the `IDENTITY` fields in order, each as text followed by one NUL byte.
|
||||
Absent is the empty string; `k` and `max_tokens` are decimal integers; `temperature` is the
|
||||
shortest decimal that reads back as the same float, never in exponent form (0, 0.7,
|
||||
0.0000001). hanzo-inc/cloud `apps/research` TestKeyVector holds the vectors this package's
|
||||
tests hold.
|
||||
"""
|
||||
d = run.dump() if isinstance(run, Wire) else run
|
||||
h = hashlib.sha256()
|
||||
for f in IDENTITY:
|
||||
h.update(_identity(f, d.get(f)).encode())
|
||||
h.update(b"\0")
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _identity(f: str, v: Any) -> str:
|
||||
if v is None:
|
||||
return ""
|
||||
if f in ("k", "max_tokens"):
|
||||
if isinstance(v, bool) or not isinstance(v, int):
|
||||
raise TypeError(f"{f} must be an int, got {v!r}")
|
||||
return str(v)
|
||||
if f == "temperature":
|
||||
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
||||
raise TypeError(f"temperature must be a number, got {v!r}")
|
||||
return format(decimal.Decimal(repr(float(v))).normalize(), "f")
|
||||
if not isinstance(v, str):
|
||||
raise TypeError(f"{f} must be a str, got {v!r}")
|
||||
return v
|
||||
|
||||
|
||||
@dataclass
|
||||
class Study(Wire):
|
||||
"""The question a set of runs was run to answer, and the answer it reached."""
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "hanzo-research"
|
||||
version = "0.2.1"
|
||||
version = "0.3.0"
|
||||
description = "File measurements to Hanzo's /v1/research and /v1/benchmark with their provenance: harness commit, model weights, host, device."
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
|
||||
@@ -19,6 +19,8 @@ import urllib.parse
|
||||
from typing import Any
|
||||
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
|
||||
|
||||
from hanzo_research.wire import key
|
||||
|
||||
CATALOG = [
|
||||
{
|
||||
"id": "gpqa_diamond",
|
||||
@@ -38,7 +40,7 @@ CATALOG = [
|
||||
]
|
||||
KEYS = {"benchmarks": "benchmarks", "runs": "runs", "studies": "studies", "papers": "papers"}
|
||||
NEEDS = {"benchmarks": ("id",), "runs": ("id", "benchmark"), "studies": ("id",), "papers": ("id",)}
|
||||
DERIVED = ("project", "canonical", "completion", "visibility", "trainable", "publishable")
|
||||
DERIVED = ("project", "canonical", "completion", "key", "visibility", "trainable", "publishable")
|
||||
GAPS = (
|
||||
"embedder",
|
||||
"facts_digest",
|
||||
@@ -145,6 +147,7 @@ class Fake:
|
||||
r["canonical"] = True
|
||||
if kind == "runs":
|
||||
r["completion"] = completion(r)
|
||||
r["key"] = key(r)
|
||||
out.append(r)
|
||||
return sorted(out, key=lambda r: (r["project"], r["id"]))
|
||||
|
||||
@@ -199,7 +202,7 @@ class Fake:
|
||||
rows = self.canonical(who["org"], "runs")
|
||||
if q.get("completion") and q["completion"] not in ("complete", "partial", "inconsistent", "unknown"):
|
||||
raise Problem(422, "completion must be complete, partial, inconsistent, or unknown")
|
||||
for f in ("project", "id", "benchmark", "system", "study", "completion"):
|
||||
for f in ("project", "id", "key", "benchmark", "system", "study", "completion"):
|
||||
if q.get(f):
|
||||
rows = [r for r in rows if r.get(f) == q[f]]
|
||||
if q.get("since"):
|
||||
@@ -545,12 +548,33 @@ class Fake:
|
||||
if name == "runs" and method == "POST":
|
||||
if who is None:
|
||||
raise Problem(401, "sign in")
|
||||
return 202, {
|
||||
"status": "queued",
|
||||
"org": who["org"],
|
||||
"by": who["user"],
|
||||
"benchmarks": (body or {}).get("benchmarks"),
|
||||
}
|
||||
if not (body or {}).get("model"):
|
||||
raise Problem(400, "run needs a model id")
|
||||
native = {c["id"]: c["native"] for c in CATALOG}
|
||||
data = []
|
||||
for b in body.get("benchmarks") or []:
|
||||
if b not in native:
|
||||
raise Problem(422, f"unknown benchmarks: {b}")
|
||||
held = [a for a in self.attempt_rows if a["benchmark"] == b and a["model"] == body["model"]]
|
||||
if held:
|
||||
n, ok = len(held), sum(1 for a in held if a["correct"])
|
||||
data.append(
|
||||
{
|
||||
"benchmark": b,
|
||||
"latest": {"run": held[-1]["run"], "score": 100 * ok / n, "n": n},
|
||||
"note": "measured",
|
||||
}
|
||||
)
|
||||
else:
|
||||
data.append(
|
||||
{
|
||||
"benchmark": b,
|
||||
"latest": None,
|
||||
"harness": "hanzoai/enso-bench" if native[b] else "",
|
||||
"note": "nothing was queued",
|
||||
}
|
||||
)
|
||||
return 200, {"org": who["org"], "by": who["user"], "model": body["model"], "data": data}
|
||||
raise Problem(404, "Not Found")
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""`Arena` against the fake: catalog, board, claims, presets, and the paid run behind confirm."""
|
||||
"""`Arena` against the fake: catalog, board, claims, presets, and a run request that spends nothing."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -69,15 +69,16 @@ def test_org_admin_needs_a_superadmin_token(server):
|
||||
arena(server, org="admin").post_claims([claim()])
|
||||
|
||||
|
||||
def test_a_paid_run_needs_confirm(server):
|
||||
a = arena(server)
|
||||
n = len(server.fake.requests)
|
||||
with pytest.raises(Invalid, match="paid execution"):
|
||||
a.post_run(["gpqa_diamond"], model="zen5-pro")
|
||||
assert len(server.fake.requests) == n
|
||||
adm = a.post_run(["gpqa_diamond"], model="zen5-pro", confirm=True)
|
||||
assert adm.status == "queued" and adm.org == "hanzo"
|
||||
assert server.fake.requests[-1]["body"] == {"benchmarks": ["gpqa_diamond"], "model": "zen5-pro"}
|
||||
def test_a_run_request_answers_what_is_held(server):
|
||||
server.fake.attempt_rows += [
|
||||
{"benchmark": "gpqa_diamond", "model": "zen5-pro", "run": "r1", "correct": c} for c in (True, True, False, True)
|
||||
]
|
||||
held = arena(server).post_run(["gpqa_diamond", "livecodebench"], model="zen5-pro")
|
||||
assert held.org == "hanzo" and held.model == "zen5-pro"
|
||||
assert server.fake.requests[-1]["body"] == {"benchmarks": ["gpqa_diamond", "livecodebench"], "model": "zen5-pro"}
|
||||
got, none = held.data
|
||||
assert got.latest.run == "r1" and got.latest.score == 75 and got.latest.n == 4
|
||||
assert none.latest is None and none.harness == "hanzoai/enso-bench" and "nothing was queued" in none.note
|
||||
|
||||
|
||||
def test_presets_echo(server):
|
||||
@@ -98,9 +99,3 @@ def test_a_claim_names_its_score(server):
|
||||
with pytest.raises(Invalid, match="needs a score"):
|
||||
a.post_claims([row])
|
||||
assert len(server.fake.requests) == n
|
||||
|
||||
|
||||
def test_confirm_is_true_or_nothing(server):
|
||||
for truthy in ("0", 1, "yes"):
|
||||
with pytest.raises(Invalid, match="paid execution"):
|
||||
arena(server).post_run(["gpqa_diamond"], model="m", confirm=truthy)
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""`key` and `measure` against the fake: the key is the server's byte for byte, a held complete run
|
||||
is returned without spending, a miss measures and records, and `force` measures anyway."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from conftest import HANZO
|
||||
from hanzo_research import IDENTITY, Run, Invalid, Research, key
|
||||
|
||||
# hanzo-inc/cloud apps/research key_test.go TestKeyVector: the same identities, the same keys.
|
||||
VECTORS = [
|
||||
(
|
||||
Run(
|
||||
benchmark="locomo",
|
||||
split="all",
|
||||
system="enso-flash",
|
||||
version="1.2.0",
|
||||
k=20,
|
||||
temperature=0.0,
|
||||
max_tokens=512,
|
||||
embedder="text-embedding-3-small",
|
||||
reader="zen6",
|
||||
prompt_digest="9f2c",
|
||||
dataset_digest="79fa87e90f040813",
|
||||
),
|
||||
"098c0e3fe9a62a04ee9b814f62f4812b058cdadc6c72521b5a0112a5440258da",
|
||||
),
|
||||
(
|
||||
{"benchmark": "coding.repair", "split": "held", "system": "hanzo-dev", "temperature": 0.7},
|
||||
"8304fbac139f449f61526f5d58e8c429df235230da789367159c7ab98609a552",
|
||||
),
|
||||
]
|
||||
|
||||
IDENT = {**VECTORS[0][0].dump(), "id": "locomo-all-k20-enso-flash"}
|
||||
DONE = {
|
||||
"questions": 282,
|
||||
"answered": 282,
|
||||
"ended": "2026-10-15T00:00:00Z",
|
||||
"measures": [{"category": "all", "metric": "f1", "value": 0.45}],
|
||||
}
|
||||
|
||||
|
||||
def client(server):
|
||||
return Research(base=server.url, api_key=HANZO, repo=".")
|
||||
|
||||
|
||||
class Spend:
|
||||
"""A measurement that counts how often it was paid for."""
|
||||
|
||||
def __init__(self, out=DONE):
|
||||
self.out, self.calls = out, 0
|
||||
|
||||
def __call__(self):
|
||||
self.calls += 1
|
||||
return dict(self.out)
|
||||
|
||||
|
||||
def posts(server):
|
||||
return [q for q in server.fake.requests if q["method"] == "POST" and q["path"] == "/v1/research/runs"]
|
||||
|
||||
|
||||
def test_the_key_is_the_servers():
|
||||
for identity, want in VECTORS:
|
||||
assert key(identity) == want
|
||||
assert IDENTITY[0] == "benchmark" and IDENTITY[-1] == "facts_digest" and len(IDENTITY) == 13
|
||||
|
||||
|
||||
def test_the_key_reads_numbers_as_go_writes_them():
|
||||
for given, same in ((0, 0.0), (1.0, 1), (100.0, 1e2), (0.7, 0.70), (1e-07, 0.0000001)):
|
||||
assert key({"temperature": given}) == key({"temperature": same})
|
||||
assert key({"temperature": 0.0}) != key({}) and key({"k": 0}) != key({})
|
||||
assert key({"split": "allenso", "system": ""}) != key({"split": "all", "system": "enso"})
|
||||
with pytest.raises(TypeError):
|
||||
key({"k": 20.0})
|
||||
|
||||
|
||||
def test_a_miss_measures_and_records(server):
|
||||
fn = Spend()
|
||||
run = client(server).measure(IDENT, fn)
|
||||
assert fn.calls == 1 and len(posts(server)) == 1
|
||||
assert key(run) == VECTORS[0][1] and run.completion == "complete" and run.measure("all", "f1").value == 0.45
|
||||
|
||||
|
||||
def test_a_held_run_is_returned_without_spending(server):
|
||||
r = client(server)
|
||||
first = r.measure(IDENT, Spend())
|
||||
fn = Spend()
|
||||
again = r.measure({**IDENT, "id": "a-rerun-under-another-name", "commit": "later"}, fn)
|
||||
assert fn.calls == 0 and len(posts(server)) == 1
|
||||
assert again.id == first.id and key(again) == key(first)
|
||||
assert server.fake.requests[-1]["query"] == {"key": VECTORS[0][1], "completion": "complete"}
|
||||
|
||||
|
||||
def test_force_measures_anyway(server):
|
||||
r = client(server)
|
||||
r.measure(IDENT, Spend())
|
||||
fn = Spend({**DONE, "ended": "2026-10-16T00:00:00Z"})
|
||||
r.measure(IDENT, fn, force=True)
|
||||
assert fn.calls == 1 and len(posts(server)) == 2
|
||||
|
||||
|
||||
def test_a_partial_run_or_another_configuration_is_not_a_hit(server):
|
||||
r = client(server)
|
||||
r.measure(IDENT, Spend({**DONE, "answered": 57}))
|
||||
fn = Spend()
|
||||
r.measure(IDENT, fn)
|
||||
r.measure({**IDENT, "id": "k10", "k": 10}, fn)
|
||||
assert fn.calls == 2
|
||||
|
||||
|
||||
def test_fn_may_not_move_the_identity(server):
|
||||
with pytest.raises(Invalid, match="changed its measurement identity"):
|
||||
client(server).measure(IDENT, Spend({**DONE, "reader": "zen6-flash"}))
|
||||
assert posts(server) == []
|
||||
|
||||
|
||||
def test_a_server_that_ignores_the_key_is_not_a_hit(server):
|
||||
r = client(server)
|
||||
r.measure({**IDENT, "id": "another", "k": 10}, Spend())
|
||||
real = server.fake.runs
|
||||
server.fake.runs = lambda who, q: real(who, {k: v for k, v in q.items() if k != "key"})
|
||||
fn = Spend()
|
||||
r.measure(IDENT, fn)
|
||||
assert fn.calls == 1
|
||||
Reference in New Issue
Block a user