hanzoai/dev
2
0
Fork 0
mirror of https://github.com/hanzoai/dev.git synced 2026-10-17 15:34:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

kai: an answer that does not read or apply is said again once; an escalated attempt with no edit goes to the generator once

Browse Source 
Each call is an action in the ledger (stage first, repair or fallback) and the trace line
lists every call. The generator is never an Anthropic model. An answer of no edits is not
said again. The edit is tested as before.
This commit is contained in:
hanzo-dev
2026-10-14 20:20:21 -07:00
parent e6a813c380
commit 6397df24cf
3 changed files with 280 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+18
View File
@@ -180,6 +180,24 @@ and the tools they run. Search, saved actions and the shell classifier are hanzo
- **Generation** is Zen or enso-flash over an OpenAI-compatible endpoint (`kai.toml` `[generate]`),
called when needed, streamed so the first token and the last are both timed; a paid call is a
line in the spend file with the model the server says served it, and the budget stops it.
- **The edit's path from a model** (`zen.rs`, `edit.rs`, `flow.rs`). At Hanzo's API an edit is asked for
as a strict JSON schema (`edits`, at least one, each `file`, `line`, `old`, `new`). `openai/gpt-6-sol`
decodes under it; `enso-flash` is asked and may still think aloud before the JSON (700 to 900 tokens,
so an answer that does not reason has 2,048 of room), but with at least one edit required it stopped
answering `{"edits":[]}` (6 of 13 calls without, 0 of 19 with). A call that reasons has one cap of 16,384
for reasoning and answer together, at `reasoning_effort` low: a cap of 3,072 at medium lost five answers
of ten to reasoning alone (finish `length`, 3,072 of 3,072 completion tokens reasoning, no text), where
medium reasoned 221 to 3,106 tokens and low 58 to 1,001. Every call that reached the model is kept under
`generate.calls` with its finish reason and usage, reasoning tokens apart; a call without text, or ended by
the server with an error, is kept with its error, billed in the spend file, and asked again. `edit::parse`
says which miss it is: no edits, JSON that does not read or is no edit, or no JSON. An answer refused for
how it is written (it does not read, or its old text is not in the file) goes back to the model once, with
the reason and the file's own lines where the edit said it stood (`stage: repair`); an escalated attempt
that still has no edit goes to the generator once over the same prompt (`stage: fallback`, never an
Anthropic model). Each is an action of its own in the ledger, the trace line lists every call
(`generations`), and the edit is tested as ever: a parsed answer is not a solve. An answer of no edits is
not said again (it costs the attempt, as before); `{"edits":[]}` from a model that names the file it would
need to see is a context failure, not an output one.
- **Kai's transport** is `decide::Transport`: HTTP `POST /v1/decisions` today. The local engine
over ZAP on a Unix socket is the other implementation, and needs the engine to serve a `decisions`
method (the engine's ZAP module and `hanzo-zap` 0.2.1 are stubs; `zapwire` is the working Rust
crates/hanzo-kai/src/flow.rs
+146 -83
View File
@@ -30,6 +30,11 @@
//! configured, does `escalate` end the flow for a person to look. A run that passes the failing
//! command is not done until the tests around it ([`crate::wide`]) show no new failure.
//!
//! An answer refused for how it is written goes back to the model once to say again, and an
//! escalated attempt that still has no edit goes to the generator once; each call is an action in
//! the ledger and a line of the trace's `generations`. Neither is an attempt: the tests judge the
//! edit that results, as for any other.
//!
//! Every step is timed, and the trace carries the times with Kai's answers.
use crate::decide;
@@ -220,7 +225,8 @@ pub struct Attempt {
pub escalated: bool,
/// Who moved the next attempt to the escalation model: `kai`, `failed twice` or `recover`.
pub switch: Option<String>,
pub generation: Option<Generation>,
/// Every call the attempt made, in order: the model's, its repair, the fallback's.
pub generations: Vec<Generation>,
/// The end of the tests' output after the edit.
pub after: String,
/// Step -> milliseconds.
@@ -306,7 +312,7 @@ impl Report {
));
let times: Vec<String> = a.ms.iter().map(|(k, v)| format!("{k} {v:.0}")).collect();
out.push_str(&format!(" ms: {}\n", times.join(", ")));
if let Some(g) = &a.generation {
for g in &a.generations {
out.push_str(&format!(
" model {}{}: first token {:.0} ms, {:.0} ms, {} in, {} out\n",
g.served,
@@ -648,6 +654,15 @@ fn prompt(command: &str, goal: &str, failure: &str, code: &str, last: Option<&At
out
}
/// What a refused answer is asked again with: the prompt it was given, the answer, why it was
/// refused and where the file stands. The reply is the edit, in the same schema.
fn repair(user: &str, answer: &str, why: &str, near: &str) -> String {
format!(
"{user}\nYour answer was:\n{}\n\nIt was refused: {why}. The files are unchanged.\n{near}\nSay the same edit again so that it reads and applies. The reply is the edit and nothing else.\n",
decide::cut(answer, 2000)
)
}
/// A short stable id for a run.
fn task_id() -> String {
ledger::digest(format!("{}{}", crate::trace::now(), std::process::id()).as_bytes())
@@ -1249,90 +1264,133 @@ pub fn run(root: &Path, options: &Options, setup: &Setup) -> Result<Report, Stri
escalated: escalation.is_some(),
switch: None,
kai_applied: false,
generation: None,
generations: Vec::new(),
after: String::new(),
ms: IndexMap::new(),
};
let mut g = ledger.action(&revision, Operation::Generate { model: model.clone(), effort: attempt.effort.clone() }, source, if escalation.is_some() { "recover: the escalation model writes the attempt" } else if n == 1 { "the first edit" } else { "recover: another attempt" });
g.evidence = saw.iter().map(|(f, d)| format!("{f}@{d}")).collect();
g.expected = "one JSON edit that applies".into();
g.advance(Status::Attempted);
let begun = Instant::now();
let generated = setup.zen.generate(&model, SYSTEM, &user, effort);
g.ms = ms(begun);
ms_of.insert("generate".into(), g.ms);
let answer = match generated {
Ok(x) => {
g.result = json!({"model": x.model, "served": x.served, "cached": x.cached, "escalated": escalation.is_some(), "first_ms": x.first, "total_ms": x.total, "prompt_tokens": x.prompt, "completion_tokens": x.completion});
let text = x.text.clone();
attempt.generation = Some(x);
g.advance(Status::Completed);
Ok(text)
}
Err(why) => {
g.fail(&why);
g.result = json!({"error": why});
Err(why)
}
};
ledger.record(&g);
// An answer refused for how it is written (it does not read as an edit, or its text is not
// in the file) goes back to the model once, with the reason and the file where it was
// meant, to say again. An escalated attempt that still has no edit goes to the generator
// once, over the same prompt. Each call is an action of its own.
let fallback = escalation.and(setup.zen.fallback()).map(str::to_string);
let mut asking = (
model.clone(),
effort,
user.clone(),
if escalation.is_some() { "recover: the escalation model writes the attempt" } else if n == 1 { "the first edit" } else { "recover: another attempt" },
"first",
);
let (g, e, after, model) = loop {
let (model, effort, user, why, stage) = asking.clone();
let mut g = ledger.action(&revision, Operation::Generate { model: model.clone(), effort: format!("{effort:?}").to_lowercase() }, source, why);
g.evidence = saw.iter().map(|(f, d)| format!("{f}@{d}")).collect();
g.expected = "one JSON edit that applies".into();
g.advance(Status::Attempted);
let begun = Instant::now();
let generated = setup.zen.generate(&model, SYSTEM, &user, effort);
g.ms = ms(begun);
*ms_of.entry("generate".into()).or_insert(0.0) += g.ms;
let answer = match generated {
Ok(x) => {
g.result = json!({
"model": x.model, "served": x.served, "cached": x.cached, "escalated": escalation.is_some(), "stage": stage,
"first_ms": x.first, "total_ms": x.total, "prompt_tokens": x.prompt, "completion_tokens": x.completion,
"reasoning_tokens": x.reasoning, "finish": x.finish,
});
let text = x.text.clone();
attempt.generations.push(x);
g.advance(Status::Completed);
Ok(text)
}
Err(why) => {
g.fail(&why);
g.result = json!({"error": why, "stage": stage});
Err(why)
}
};
ledger.record(&g);
// The edit is made against the revision and the files the model saw.
let mut e = ledger.action(&revision, Operation::Edit { edits: Vec::new() }, source, "make the model's edit");
e.preconditions = std::iter::once(Precondition::Revision { revision: revision.clone() })
.chain(saw.iter().map(|(f, d)| Precondition::Digest { file: f.clone(), digest: d.clone() }))
.collect();
e.authorization = "auto: a source file of the tree, never a test".into();
e.expected = "the edit applies and the tests then pass".into();
e.advance(Status::Attempted);
let begun = Instant::now();
let held = e.preconditions.iter().find_map(|p| match p {
Precondition::Revision { revision: r } if *r != ledger::revision_over(root, &touched, &origin) => Some(format!("the workspace is no longer at revision {r}")),
Precondition::Digest { file, digest } if ledger::digest(&std::fs::read(root.join(file)).unwrap_or_default()) != *digest => {
Some(format!("{file} changed since the model read it"))
}
_ => None,
});
let applied = match (&answer, held) {
(_, Some(why)) => Err(format!("a precondition does not hold: {why}")),
(Ok(text), None) => edit::parse(text).and_then(|edits| edit::apply(root, &edits)),
(Err(why), None) => Err(why.clone()),
};
let after = match &applied {
Ok(made) => {
let files = made.files();
for (f, d) in made.originals() {
origin.entry(f).or_insert(d);
// The edit is made against the revision and the files the model saw.
let mut e = ledger.action(&revision, Operation::Edit { edits: Vec::new() }, source, "make the model's edit");
e.preconditions = std::iter::once(Precondition::Revision { revision: revision.clone() })
.chain(saw.iter().map(|(f, d)| Precondition::Digest { file: f.clone(), digest: d.clone() }))
.collect();
e.authorization = "auto: a source file of the tree, never a test".into();
e.expected = "the edit applies and the tests then pass".into();
e.advance(Status::Attempted);
let begun = Instant::now();
let held = e.preconditions.iter().find_map(|p| match p {
Precondition::Revision { revision: r } if *r != ledger::revision_over(root, &touched, &origin) => Some(format!("the workspace is no longer at revision {r}")),
Precondition::Digest { file, digest } if ledger::digest(&std::fs::read(root.join(file)).unwrap_or_default()) != *digest => {
Some(format!("{file} changed since the model read it"))
}
let rev = ledger::revision_over(root, &files.iter().chain(touched.iter()).cloned().collect::<Vec<_>>(), &origin);
// The same edit again is no new state: recover chooses another action.
if tried.contains(&rev) {
edit::revert(root, made);
Err("the edit repeats an earlier attempt's: the workspace would be in a state already tried".to_string())
} else {
Ok((made.clone(), files, rev))
_ => None,
});
let applied = match (&answer, held) {
(_, Some(why)) => Err(format!("a precondition does not hold: {why}")),
(Ok(text), None) => edit::parse(text).and_then(|edits| edit::apply(root, &edits)),
(Err(why), None) => Err(why.clone()),
};
let after = match &applied {
Ok(made) => {
let files = made.files();
for (f, d) in made.originals() {
origin.entry(f).or_insert(d);
}
let rev = ledger::revision_over(root, &files.iter().chain(touched.iter()).cloned().collect::<Vec<_>>(), &origin);
// The same edit again is no new state: recover chooses another action.
if tried.contains(&rev) {
edit::revert(root, made);
Err("the edit repeats an earlier attempt's: the workspace would be in a state already tried".to_string())
} else {
Ok((made.clone(), files, rev))
}
}
Err(why) => Err(why.clone()),
};
e.ms = ms(begun);
*ms_of.entry("apply".into()).or_insert(0.0) += e.ms;
match &after {
Ok((made, files, rev)) => {
e.advance(Status::Applied);
e.operation = Operation::Edit { edits: made.summary.clone() };
e.result = json!({"files": files, "revision": rev, "model": model, "stage": stage});
attempt.applied = true;
attempt.refused = None;
attempt.edits = made.summary.clone();
}
Err(why) => {
e.advance(Status::Rejected);
e.result = json!({"error": why, "model": model, "stage": stage});
attempt.refused = Some(why.clone());
}
}
Err(why) => Err(why.clone()),
ledger.record(&e);
let Err(why) = &after else {
break (g, e, after, model);
};
asking = if let (true, Ok(text)) = (stage == "first" && edit::spelling(why), &answer) {
(
model.clone(),
Effort::Low,
repair(&user, text, why, &edit::near(root, text)),
"repair: the answer is said again so that it reads and applies",
"repair",
)
} else if let (Some(generator), "first" | "repair") = (&fallback, stage) {
(
generator.clone(),
Effort::Low,
user.clone(),
"fallback: the escalation model gave no edit that applies; the generator writes it",
"fallback",
)
} else {
break (g, e, after, model);
};
};
e.ms = ms(begun);
ms_of.insert("apply".into(), e.ms);
attempt.model.clone_from(&model);
let mut test = Value::Null;
match &after {
Ok((made, files, rev)) => {
e.advance(Status::Applied);
e.operation = Operation::Edit { edits: made.summary.clone() };
e.result = json!({"files": files, "revision": rev, "model": model});
attempt.applied = true;
attempt.edits = made.summary.clone();
}
Err(why) => {
e.advance(Status::Rejected);
e.result = json!({"error": why, "model": model});
attempt.refused = Some(why.clone());
}
}
ledger.record(&e);
// The tests are the verifier: only their record settles the outcome.
let mut after_rev = revision.clone();
@@ -1618,17 +1676,22 @@ fn line(order: &Order, turn: Option<&Decision>, attempt: &Attempt, run: &str, wh
step.applied = attempt.kai_applied;
ops.insert("turn".to_string(), step);
}
let generation = attempt.generation.as_ref().map(|g| {
json!({"model": g.model, "served": g.served, "cached": g.cached, "first_ms": g.first, "total_ms": g.total, "prompt_tokens": g.prompt, "completion_tokens": g.completion})
});
let generations: Vec<Value> = attempt
.generations
.iter()
.map(|g| {
json!({"model": g.model, "served": g.served, "cached": g.cached, "first_ms": g.first, "total_ms": g.total, "prompt_tokens": g.prompt,
"completion_tokens": g.completion, "reasoning_tokens": g.reasoning, "finish": g.finish})
})
.collect();
Line {
time: crate::trace::now(),
request: format!("{run}#{}", attempt.n),
family: "dev.flow".into(),
sku: SKU.into(),
ms: attempt.ms.values().sum(),
prompt_tokens: attempt.generation.as_ref().map_or(0, |g| g.prompt as i64),
completion_tokens: attempt.generation.as_ref().map_or(0, |g| g.completion as i64),
prompt_tokens: attempt.generations.iter().map(|g| g.prompt as i64).sum(),
completion_tokens: attempt.generations.iter().map(|g| g.completion as i64).sum(),
outcome: crate::trace::Outcome {
status: i64::from(!attempt.passed),
finish: attempt.taken.clone(),
@@ -1646,7 +1709,7 @@ fn line(order: &Order, turn: Option<&Decision>, attempt: &Attempt, run: &str, wh
"refused": attempt.refused,
"recover": why,
"ms": attempt.ms,
"generation": generation,
"generations": generations,
}),
..Default::default()
},
crates/hanzo-kai/src/flow_tests.rs
+116 -4
View File
@@ -199,7 +199,7 @@ async fn the_model_writes_the_edit_when_search_finds_none_and_the_flow_is_done()
for step in ["bundle", "generate", "apply", "test"] {
assert!(a.ms.contains_key(step), "{step}: {:?}", a.ms);
}
assert_eq!(a.generation.as_ref().unwrap().prompt, 100);
assert_eq!(a.generations[0].prompt, 100);
let sent = asked_zen(&zen).await;
assert_eq!(sent.len(), 1);
@@ -212,7 +212,7 @@ async fn the_model_writes_the_edit_when_search_finds_none_and_the_flow_is_done()
assert_eq!(line["family"], "dev.flow");
assert_eq!(line["sku"], "flow.fix@2");
assert_eq!(line["outcome"]["finish"], "done");
assert_eq!(line["outcome"]["result"]["generation"]["prompt_tokens"], 100);
assert_eq!(line["outcome"]["result"]["generations"][0]["prompt_tokens"], 100);
// A run that passes needs no inference: the ledger says so, and Kai was not asked about it.
let led = ledger(repo.path());
@@ -604,6 +604,118 @@ async fn the_escalation_model_spent_the_flow_ends_for_a_person() {
assert_eq!(read(repo.path()), BUGGY, "the tree is as it was");
}
#[tokio::test(flavor = "multi_thread")]
async fn an_answer_that_does_not_read_is_said_again_once_in_the_same_attempt() {
let repo = repo();
let kai = honest().await;
// A bare quote inside `new`: the JSON does not parse. The repair sees why and the old answer.
let broken = "{\"edits\":[{\"file\":\"src/add.py\",\"line\":2,\"old\":\"a ^ b\",\"new\":\"a + b\" \"}]}";
let zen = testing::zen(move |prompt| if prompt.contains("It was refused") { fix() } else { broken.to_string() }).await;
let report = run_flow(repo.path(), &kai.uri(), &zen, options(3, Control::Kai)).await;
assert_eq!(report.outcome, "done", "{}", report.summary());
assert_eq!(report.attempts.len(), 1, "the repair is no attempt of its own");
assert_eq!(report.attempts[0].generations.len(), 2);
assert_eq!(report.attempts[0].refused, None);
let sent = asked_zen(&zen).await;
assert_eq!(sent.len(), 2);
assert!(sent[1].contains("the answer's JSON does not parse") && sent[1].contains("Your answer was:"), "{}", sent[1]);
let led = ledger(repo.path());
let stages: Vec<&str> = ops(&led, "generate").iter().map(|l| l["result"]["stage"].as_str().unwrap()).collect();
assert_eq!(stages, ["first", "repair"]);
let edits: Vec<&str> = ops(&led, "edit").iter().map(|l| l["status"].as_str().unwrap()).collect();
assert_eq!(edits[0], "rejected");
assert!(edits[1..].iter().all(|s| *s == "applied"), "{edits:?}");
assert_eq!(read(repo.path()), FIXED);
}
#[tokio::test(flavor = "multi_thread")]
async fn a_text_that_is_not_in_the_file_is_said_again_with_the_file_where_it_was_meant() {
let repo = repo();
let kai = honest().await;
let zen = testing::zen(|prompt| {
if prompt.contains("It was refused") {
assert!(prompt.contains("src/add.py as it stands") && prompt.contains(" 2 "), "{prompt}");
fix()
} else {
edit_json("return a xor b", "return a + b")
}
})
.await;
let report = run_flow(repo.path(), &kai.uri(), &zen, options(3, Control::Kai)).await;
assert_eq!(report.outcome, "done", "{}", report.summary());
assert_eq!(report.attempts.len(), 1);
assert_eq!(asked_zen(&zen).await.len(), 2);
}
#[tokio::test(flavor = "multi_thread")]
async fn an_answer_that_says_no_edit_is_not_said_again() {
let repo = repo();
let kai = honest().await;
let zen = testing::zen(|prompt| if prompt.contains("Your previous edit") { fix() } else { "{\"edits\":[]}".to_string() }).await;
let report = run_flow(repo.path(), &kai.uri(), &zen, options(3, Control::Kai)).await;
assert_eq!(report.outcome, "done", "{}", report.summary());
assert_eq!(report.attempts.len(), 2, "no edit costs the attempt, as before");
assert_eq!(report.attempts[0].generations.len(), 1);
}
#[tokio::test(flavor = "multi_thread")]
async fn an_escalated_attempt_with_no_edit_goes_to_the_generator_once() {
let repo = repo();
let kai = kai(0, |_| ("escalate", 0.9)).await;
// The escalation model has nothing to say; the generator, asked again over the same prompt, fixes it.
let n = std::sync::atomic::AtomicUsize::new(0);
let zen = testing::zen_model(move |model, _| match model {
"zen-strong" => "{\"edits\":[]}".to_string(),
_ if n.fetch_add(1, std::sync::atomic::Ordering::SeqCst) == 0 => edit_json("a ^ b", "a | b"),
_ => fix(),
})
.await;
let report = run_escalating(repo.path(), &kai.uri(), &zen, options(3, Control::Kai)).await;
assert_eq!(report.outcome, "done", "{}", report.summary());
assert_eq!(testing::models(&zen).await, ["zen-test", "zen-strong", "zen-test"]);
assert_eq!(report.attempts.len(), 2);
assert_eq!(report.attempts[1].generations.len(), 2);
assert_eq!(report.attempts[1].model, "zen-test", "the edit that passed was the generator's");
assert!(report.attempts[1].escalated);
let led = ledger(repo.path());
let stages: Vec<(&str, &str)> = ops(&led, "generate")
.iter()
.map(|l| (l["operation"]["model"].as_str().unwrap(), l["result"]["stage"].as_str().unwrap()))
.collect();
assert_eq!(stages, [("zen-test", "first"), ("zen-strong", "first"), ("zen-test", "fallback")]);
assert_eq!(read(repo.path()), FIXED);
}
#[tokio::test(flavor = "multi_thread")]
async fn the_fallback_is_one_call_and_an_edit_that_applied_is_not_a_failure_to_write() {
let repo = repo();
let kai = kai(0, |_| ("escalate", 0.9)).await;
// The escalation model's edit is valid and wrong: it is tested and reverted, and no fallback is made.
let zen = testing::zen_model(|model, _| edit_json("a ^ b", if model == "zen-strong" { "a | c" } else { "a | b" })).await;
let report = run_escalating(repo.path(), &kai.uri(), &zen, options(2, Control::Kai)).await;
assert_eq!(report.outcome, "escalate");
assert_eq!(testing::models(&zen).await, ["zen-test", "zen-strong"]);
// The escalation model answering nothing at all, and the generator nothing either: one fallback, then the flow ends.
let repo = repo_with(BUGGY);
let zen = testing::zen(|_| "{\"edits\":[]}".to_string()).await;
let report = run_escalating(repo.path(), &kai.uri(), &zen, options(3, Control::Kai)).await;
assert_eq!(report.outcome, "escalate");
// The generator is asked once more over the escalated attempt's prompt; here that is the prompt it
// answered before, so the kept call answers and no fourth request is sent.
let led = ledger(repo.path());
let calls: Vec<(&str, &str, bool)> = ops(&led, "generate")
.iter()
.map(|l| (l["operation"]["model"].as_str().unwrap(), l["result"]["stage"].as_str().unwrap(), l["result"]["cached"].as_bool().unwrap()))
.collect();
assert_eq!(
calls,
[("zen-test", "first", false), ("zen-test", "first", false), ("zen-strong", "first", false), ("zen-test", "fallback", true)]
);
assert_eq!(testing::models(&zen).await, ["zen-test", "zen-test", "zen-strong"]);
assert_eq!(read(repo.path()), BUGGY);
}
#[tokio::test(flavor = "multi_thread")]
async fn rule_control_escalates_after_two_failures_and_asks_kai_nothing() {
let repo = repo();
@@ -627,12 +739,12 @@ async fn a_kept_call_answers_a_rerun_and_the_ledger_says_cached() {
let setup = setup_with(repo.path(), &kai.uri(), &zen, None, calls.path());
let report = run_with(setup, repo.path().to_path_buf(), options(3, Control::Kai)).await.unwrap();
assert_eq!(report.outcome, "done");
seen.push((report.attempts[0].generation.as_ref().unwrap().cached, ops(&ledger(repo.path()), "generate").last().unwrap()["result"]["cached"].clone()));
seen.push((report.attempts[0].generations[0].cached, ops(&ledger(repo.path()), "generate").last().unwrap()["result"]["cached"].clone()));
}
assert_eq!(seen, [(false, json!(false)), (true, json!(true))]);
assert_eq!(asked_zen(&zen).await.len(), 1, "the second run made no call");
let lines = testing::lines(&trace(repo.path()), 2);
assert_eq!(lines[1]["outcome"]["result"]["generation"]["cached"], true);
assert_eq!(lines[1]["outcome"]["result"]["generations"][0]["cached"], true);
}
/// A `pytest` that runs every `test_*.py` under the paths it is given with python3, naming those