zeekay 395bfde4e6 decision/stability: the same request three times, Kai and Jev ...

4,200 /v1/decisions bodies, Hanzo's own support-ticket states with one
typed question each, keyed by the SHA-256 of the body. Each was sent to
each model three times on 2026-10-14; a request is identical when its
three answers are the same JSON. results/ keeps every answer's digest:
the kai-1.2 release on CPU, four replicas, 4,200 of 4,200 identical;
typesafe/jev-1.13 through OpenRouter, 1,858 of 4,200.

run.py asks any server, spreading repeats over replicas as the recorded
Kai run did; score.py counts the recorded runs or a new answer file.

2026-10-15 10:43:30 -07:00

306 KiB

Raw Permalink History

View Raw