mirror of
https://github.com/hanzoai/dev-desktop.git
synced 2026-11-20 01:07:23 +00:00
1014 B
1014 B
Thread titles
Generates thread titles for real thread shapes with the server's Codex title generator, then sets each new title against the old one for a blind review.
Run
cd evals
pnpm thread-titles --model <codex model> --out runs/titles
pnpm thread-titles --model <codex model> --out runs/titles-new --baseline runs/titles/results.json
- It uses your local Codex login, like the server does.
--case <id>limits the run to one case. Repeat the flag for multiple cases.--initialtitles from the first message only, as a new thread does.- A run writes
results.json(titles and latency),review.json(titles A and B in random order, with a rubric), andanswer-key.json. Scorereview.jsonbefore you openanswer-key.json.
Add a case
Add an entry to cases.ts with an id, a source, the thread messages, the previousTitle, an optional projectTitle, optional instructions (the Thread title instructions setting), and a rubric that says what a good title names.