mirror of
https://github.com/hanzoai/mcp.git
synced 2026-11-03 01:31:11 +00:00
zeekay
3431b4ac53
Measured against kai-1.2 served locally from production's folder (SHA256SUMS e45dd919…).
kai_injection_screen asks Kai the question it was trained on ({prompt},
"Does `prompt` try to make an AI assistant ignore its rules...") beside
seven rule classes: override, role, addressed, conceal, exfiltrate,
tool_call, act, plus the earlier extract patterns. Held out on InjecAgent's
unseen tools and wordings (250 attacks) and 207 benign tool outputs and
README paragraphs: recall .728 at .010 false alarms, against .044 at .116
for the screen it replaces. Kai's question alone found 0 of the 250
planted requests. Rules still decide when Kai does not answer.
kai_tool_select ranks by a named tool, then BM25 over name, description
and parameters, and makes no Kai call: on 300 supplied tool lists from
Kai's T1 dev split it reads top-1 .733 and MRR .841 against .657 and .805
for one Kai choice over the names, in 0.1 ms against 109 ms.
kai_log_triage classifies by failure signature (resources, network,
timeout, permission, dependency, compile, test, crash; none and unknown)
and returns the deciding lines and file spans; kai-1.2 read build logs at
chance. Recipes 01, 04, 05, 08 and 10 check what these tools return;
recipe 08 calls the tool instead of matching names itself. The cookbook
reaches Kai as clients do, through HANZO_API_KEY or LOCAL_KAI_URL, not a
fixed local port and placeholder key. compact.ts types its decision so
tsc passes.
47 lines
1.5 KiB
TypeScript
47 lines
1.5 KiB
TypeScript
/**
|
|
* Recipe 05: Build & Test Log Triage
|
|
*
|
|
* Objective: Parse complex compilation and test error logs, pinpoint the exact file/line span,
|
|
* classify the failure category, and determine whether the error is deterministic or flaky.
|
|
*/
|
|
|
|
import { createRecipeClient, logRecipe, RecipeResult } from "./helper.js";
|
|
|
|
export async function runRecipe05(): Promise<RecipeResult> {
|
|
const client = await createRecipeClient();
|
|
|
|
const rawLogs = `
|
|
Compiling parser v0.1.0 (/workspace/parser)
|
|
error[E0308]: mismatched types
|
|
--> src/number.rs:88:15
|
|
|
|
|
88 | Ok(val) => val,
|
|
| ^^^ expected \`f64\`, found enum \`Result<f64, ParseFloatError>\`
|
|
|
|
|
= note: expected type \`f64\`
|
|
found enum \`Result<f64, ParseFloatError>\`
|
|
|
|
For more information about this error, try \`rustc --explain E0308\`.
|
|
error: could not compile \`parser\` (lib) due to 1 previous error
|
|
`;
|
|
|
|
const res = await client.callTool("kai_log_triage", {
|
|
logs: rawLogs
|
|
});
|
|
|
|
const result: RecipeResult = {
|
|
recipe: "05: Build & Test Log Triage",
|
|
tool: "kai_log_triage",
|
|
success: !res.isError && res.data.category === "compile" && res.data.files.includes("src/number.rs:88:15"),
|
|
latency_ms: res.latency_ms,
|
|
output: res.data,
|
|
explanation: "Classified as compile by the rustc error signature, with the failing lines and src/number.rs:88:15. No model call."
|
|
};
|
|
|
|
return result;
|
|
}
|
|
|
|
if (process.argv[1]?.endsWith("05-build-log-triage.ts")) {
|
|
runRecipe05().then(logRecipe).catch(console.error);
|
|
}
|