CARGO_PROFILE_RELEASE_LTO "false" is cargo's thin local LTO, not none: 95 of hanzo-dev's 215 seconds. The measurements, the pool's contention and what the heavy class costs are in "The release run".
26 KiB
Hanzo Dev
The dev command: a coding agent for the terminal.
Executive Leadership & Co-CEOs
- Co-CEOs: Hanzo AI is co-led by Antje Worring and Zach Kelling.
- Antje Worring: Co-CEO (LinkedIn · GitHub: zooqueen) — corporate leadership, operations, ecosystem partnerships, and strategic execution.
- Zach Kelling: Co-CEO & Founding CTO (LinkedIn · GitHub: zeekay) — foundational architecture, engineering, systems, and open-source models.
- Corporate Address:
1 Ferry Building, San Francisco, CA 94105(NEVER use1828 Golden Gate). - Web:
https://dev.chat·https://hanzo.ai
Shape
Hanzo owns the front door; upstream owns the engines.
crates/hanzo-devis the binary. Itsmain.rsis the entire command surface, and every subcommand hands off to a library. Nothing is generated and no upstream source is compiled through aninclude!.crates/hanzo-configpicks the provider and the product home.crates/hanzo-tuiis the Hanzo account behind /login, /logout, /auth and /status.crates/hanzo-kaiis Kai.upstream/codexis hanzoai/codex, the engines with Hanzo's commits on them.
The loop the cloud drives
The reasoning half of dev as a wasm module, with its Go host and the wasm2go
build, lives in hanzo-inc/dev. Nothing here depends on it.
Upstream
upstream/codex is a submodule of hanzoai/codex, and the gitlink is the pin. That
repository's main is ours: openai/codex's tree with Hanzo's commits on it. Its
upstream remote is openai/codex. Crates are consumed straight from the checkout by
ordinary path dependency, and the checkout is pristine: nothing rewrites it at build
time, so git -C upstream/codex status --porcelain is empty after make build and
make test.
A bump is a merge, and its conflicts are git's:
- In
upstream/codex, onmain(current with hanzoai/codex):git fetch upstream && git merge upstream/main. Resolve conflicts in git. - Read what the merge brought that the delta has not seen: new literals that say
Codex (
git diff ORIG_HEAD -G'"[^"]*Codex' -- 'codex-rs/*.rs') and newenv!("CARGO_PKG_VERSION")reads (make testfails on one, naming the file). Fix them as commits onmain. make build testhere, pushmainto hanzoai/codex, then commit the gitlink.
make bump prints those steps. The root Cargo.toml's [workspace.dependencies] and
[patch] tables mirror the fork's codex-rs/Cargo.toml (paths rewritten from the root),
because patches apply only from the root workspace; refresh them on a bump. Three
things in the root are Dev's own: the hanzo-* crates in [workspace.dependencies],
the rusqlite patch, and no v8 patch (Dev links no V8, and a patch nothing in the
graph uses warns on every build). The fork
builds and tests on its own (cargo build in codex-rs, make test-upstream here) and
keeps its own codex-rs/Cargo.lock, as upstream does: it names no path outside its tree. Measured on
the move from 11da6b9e to 3ee47c105 (288 upstream commits, 1,608 files), with the whole
delta committed on the old base: the merge stopped on 5 files (one doc comment the
delta rewords, two test fixtures and two strings upstream rewrote), and with upstream's
side taken there, 16 files differed from what the old build-time rules made of the new
tree: the strings and version reads upstream added, which step 2 finds. The first merge,
3ee47c105 to e0aa0434f4 (996 upstream commits), stopped on 39 files; step 2 then found
24 files of new Codex strings and 10 version reads. The next, to 0ed34eb9ce (28 commits),
stopped on one file and brought one string.
The delta (git log --no-merges upstream/main..main in upstream/codex):
codex-rs/hanzo, one crate: Dev's names and the TUI's defaults,API_BASE,DEFAULT_MODEL,is_hanzo, the model catalog,model::pick(the/model <name>match), the controller interface the loop consults (controller::{Controller, Handle, Verdict, Card, Request, Shortlist}) andversion. Upstream crates read these; Dev's crates depend on this one, never the reverse.- Two hooks, each a no-op until a binary fills it, so with nothing registered the engines
behave as upstream's:
codex_app_server::register(Extension)adds an extension to every thread (process-global: the TUI and exec start the app server in process), andcodex_tui::register(Account { login, logout, show })puts an account behind /login, /auth, /logout and /status (without one, /login and /auth are not offered). hanzo-dev'smainregistershanzo_kai::installandhanzo_tui::accountbefore anything starts. tui:,cli:,config:,core:,words:the edits that were anchored strings: Dev's words and colors,/model <name>, the publishedmcp_cmd,plugin_cmdanddoctor(they belong upstream as a pull request),.hanzohomes, the controller's calls in the loop, Hanzo's catalog and reviewers.core:the repeated tool-cycle guard (core/src/session/cycle.rs, checked inrun_turnbefore each sampling request): a step that repeats the one before it, the same calls (name, namespace, arguments; never the call id) answered the same (exec'sChunk IDandWall timeaside), fails the turn withunsupported operation: repeated identical tool-use cycle detected; refusing to continue after duplicate tool calls:and each call as JSON ({"type":"function_call","name":…,"namespace":…,"arguments":"…"}, joined by,). Indev exec --jsonthat isturn.failed, and the exit is non-zero. A user message between steps, a step without calls and compaction start the comparison over, soexec resumecarries on in the thread. hanzo-inc/cloud's coding loop (apps/coding/loop.go) reads the mark. It lives in core because only the loop holds a step's calls and their results together; no extension hook sees a tool's output or can end a turn.- Rate limits never surface.
hanzo::pacekeeps, per origin, what the provider'sx-ratelimit-{limit,remaining,reset}-{requests,tokens}and a 429/503'sRetry-Aftersaid; codex-api's request wrapper (run_with_request_telemetry) waitspace::delaybefore each request and feeds every response back topace::observe, so a spent window holds the next request until it resets and a low one (a quarter of the limit) is spread over what is left. What still comes back as a 429 (not a spent plan or quota),slow_downor a 503/overload is waited out in core's one retry path (responses_retry.rsslow_down):Retry-After, else full-jitter backoff (1 s doubling to 60 s), with no attempt cap while the turn lives. The wait is the turn's status (Session::notify_retry_wait, no error details) and indev exec --jsonit isturn.waiting {"message":"waiting for rate limit, retrying in 12s"}: every retried notice is, anderroris only what ends a turn. Upstream ends a turn on an overloaded model; Dev waits (the owner's rule).crates/hanzo-dev/tests/pace.rsholds it. - Reasoning effort is one table,
hanzo::effort(auto,instant,low,medium,high,max, in /model picker order; rename there). Every Hanzo model starts atauto, andautois sent as no effort, so api.hanzo.ai's router decides; the others are sent as written. A storednoneorminimal(the picker once savednoneas every Hanzo model's default) is rewritten toinstantat startup (hanzo-configupgrade_effort, top level and profiles); upstream'sxhighis shown as Max.crates/hanzo-dev/tests/effort.rsholds it. brand:Dev's name in every user-facing string literal of non-test code.version:the version, below.
Dev reads only Hanzo's directories. Its home is ~/.hanzo/dev (or DEV_HOME), a project's
settings live in <repo>/.hanzo/dev/, and system ones in /etc/hanzo/dev/; a .codex beside
them belongs to another product and is not read. Its server runs inside the process: upstream's
shared background server is installed from a package directory Dev does not ship
(hanzo::BACKGROUND_SERVER). A profile that names no provider is set to Hanzo at startup.
crates/hanzo-dev/tests/launch.rs launches a bare dev in a terminal and holds all three.
The version is the release's, set by hand in the root Cargo.toml's [workspace.package],
in npm/package.json and in the tag, and release.yml refuses a tag the two manifests do
not both say. hanzo-dev alone takes it: its main hands its own CARGO_PKG_VERSION to
hanzo::version::set, and every upstream crate asks hanzo::version::get() at run time.
Dev's libraries stay 0.0.0. So a release recompiles hanzo-dev and the link, nothing
beneath.
crates/hanzo-dev/tests/cli.rs and launch.rs hold --version, the doctor and the TUI
header to the version the binary was built as, and fail on any upstream read of its own.
Never edit upstream by hand
A change upstream hard-codes is a commit on hanzoai/codex main, made in upstream/codex
and pushed there before the gitlink here moves; never a build-time rewrite, never an
uncommitted edit in the checkout. Anything that need not live inside upstream's crates
goes in a hanzo-* crate here.
The Code fork
just-every/code is reference, never a build input (its crates carry their own codex-core); its
clone lives at ~/work/just-every/code, and what was ported from it is recorded in migration/.
Working here
make build the dev binary
make test the Hanzo suite
make test-upstream hanzoai/codex's own suite against the pinned checkout
make bump how to move upstream/codex forward
make help everything
make build is the check that must pass. Fix warnings as well as errors.
The release cache
Each release build runs rustc through sccache, and hanzoai/ci's bin/cache (pinned
by commit as CI_SHA in release.yml) carries the objects, the registry's crate
files and the git dependencies to the next run through the org's ci-cache bucket
in cloud's /v1/s3. The IAM token comes from the KMS client pair inside the restore
and save steps and is never exported, so no build script sees it.
- Name
<sha16 of rustc -vV + sccache --version>-<target>; key<sha32 of git ls-tree of the upstream/codex pin and rust-toolchain.toml, the root Cargo.toml and Cargo.lock without the release's version (its line in[workspace.package]and hanzo-dev's line in the lock), and the RUSTFLAGS/CARGO_*/ZIG*/LLVM_*/XWIN_*/MACOS_SDK_* env>.crates/is not in the key, because a release rebuilds Dev's crates and no later one can use what it built. So a Dev-only release restores its own key exactly and saves nothing; one that moves the pin or the lock misses, restoreslatestfor its name, recompiles what moved and saves the new key. The windows archive also carries the pruned LLVM (llvm) and cargo-xwin's Windows SDK (xwin), which cost 2.2 and 2 minutes to fetch and unpack on every run. - Two traps that cost a cold build each.
cargo-zigbuildnames its linker wrapper by a hash of its own path, cargo hands that name to every rustc asCARGO_TARGET_*_LINKER, and sccache hashes it into every object: the workflow installs the prebuilt binary at~/.cargo/bin, wherecargo installput it, and from anywhere else the build found no object at all (24 minutes, 0 hits of 1,631). And sccache leavesCARGO_BUILD_JOBS(with the jobserver and the registry tokens) out of its hash, so the job count changes freely; any otherCARGO_*variable, flag or path moves every object. - Only main and tags write, and only a key the store does not hold. Before a save the directory is cut to what that build touched: sccache bumps an object's mtime on a hit, and a crate file stays only if cargo unpacked it.
- What compiles every release anyway: proc-macro and build-script crates, which sccache
cannot cache, the crates the commit changed and those above them, hanzo-dev (it carries
the version) and the link. Before the version became a run-time value the stamp
changed every first-party crate's version, which is in both
-C metadataandCARGO_PKG_VERSION, so all ~150 recompiled on every release; that is why Dev's libraries stay0.0.0. - Measured on dgx (its own target, two jobs, a busy host): cold, 1,046 crates in 59.6
minutes; the same release from an empty target dir with a warm cache, 6.7 minutes and
not one Rust miss (1,109 hits), hanzo-dev's own compile and the link ~4 of them; a
new version alone rebuilds hanzo-dev and nothing else (measured then with the version
passed as
DEV_VERSION, which reached the same one crate). sccache --show-statsprints at the end of every Build step, pass or fail.
The release run
release.yml, on a pushed tag: version (a shallow checkout of the tag, about 15 s)
opens the draft; five build jobs on xl-linux-amd64 (a 32 vCPU / 56 GiB node each:
cloud apps/platform/job.go, universe infra/aws/pool.tf) compile and upload into
it; publish checks all 28 assets, publishes, then Homebrew and npm. A branch run
(workflow_dispatch from anything but the default branch) builds into a draft and
never publishes; delete that draft by hand.
Measured 2026-11-08 against v0.7.10's 109 minutes (builds on the 8-CPU ordinary pool, five to a node; each number a run on the xl pool, pin and lock unmoved):
| before | now | |
|---|---|---|
version |
4.1 min (3.9 of it the clone) | 11-20 s |
| a build job's setup (checkout, Rust, cache restore, Zig, SDK) | 3-4 min | 50 s |
Build, warm, hanzo-config and hanzo-dev changed |
58-87 min | 4.6-6.8 min |
| Package and upload | 1-3 min | 30-35 s |
publish (assets 26-32 s, npm 80 s as a dry run: the upload is not in it) |
6.9 min | 2.0 min |
| Build, engine moved (codex-core and 28 crates above it) | 58-87 min | 18 min |
| Build, nothing cached | 59.6 min on dgx | 23-27 min (20-23 with lto = "off") |
Every call to GitHub goes through scripts/ghapi.sh: a non-2xx is never parsed, a
rate-limited call (403 or 429 with retry-after or x-ratelimit-remaining: 0) is
waited out up to 10 minutes and asked again, a 5xx is asked again after 2, 8, 18 and
32 s, and anything else fails at once naming the status and GitHub's message. The token
is the account's 5,000 calls an hour, shared with every tool on it, and it ran dry
during 0.7.11's upload (jq was handed the error object and died). A release makes
about 66 calls: 3 in version, 33 in the builds (a list, then one upload per file),
30 in publish (a list, 28 downloads, the publish). version reads the free
/rate_limit first and stops, or waits for a reset under 10 minutes away, if less
than 100 are left.
What a warm Build still waits on, in order: the walk through ~1,400 units, 90-140 s,
which is build scripts and proc macros (sccache cannot cache them) and the chain from
aws-lc-sys; then hanzo-dev, one crate, 190-240 s (collector 15 s, LLVM 175 s, link
4 s). Memory peaked at 6.7-14 GiB of 56 across every run, so cores, not memory,
set the pace. The engine moved is core 245 s, then app-server 140 s, tui 240 s,
cli, then hanzo-dev again: a serial chain no node size shortens.
Not in the workflow because they change the shipped binary or the pool: lto = "off"
(comment in release.yml), and the pool itself. The xl pool is shared with
hanzo-inc/cloud's gates and hanzo.ai's gates and deploys. In six runs on it, all five
builds started inside two minutes twice; in three the last one waited 4 to 11 minutes
for a node; in the sixth four of five were still waiting after 15, behind 33 pending
pods of other jobs, when I cancelled it. The ordinary pool's heavy-linux-amd64
starts in a minute but ran the same warm Build in 16.5 minutes (hanzo-dev alone 653 s
against 190), so it is no way round.
Review and memory models
Automatic approval review and memory extraction and consolidation name a model
of their own. Upstream's ids are OpenAI's; api.hanzo.ai serves none of them. On
the Hanzo provider (hanzo::is_hanzo, base URL hanzo::API_BASE) all three are
DEFAULT_MODEL. The reviewer takes its preferred id only when the catalog serves
it, else the parent's model, and off OpenAI only a listed entry is served: the
bundled OpenAI entries ride along unlisted in every catalog, for lookups. So a LAN
engine with no catalog of its own is reviewed by the model it runs. Guardian v2's
sync reviewer runs on that same choice; its CodexResponsesHeaders model only
scopes an OpenAI-backend header. crates/hanzo-dev/tests/review.rs holds both.
Controller
Kai is Dev's native controller, not an advisory service: it returns a validated operation or an
explicit abstention, never advice. crates/hanzo-kai is the loop (flow), the ledger (ledger)
and the tools they run. Search, saved actions and the shell classifier are hanzo-mcp's library
(hanzoai/mcp, crate hanzo-trigram today); Dev links it and keeps the seams as traits
(find::Finder, decide::Transport).
- Task state. One per run: objective, workspace revision, pending actions, verification
status. The loop owns it and each action's lifecycle. An expected outcome is never accepted
because something predicted it: only the verifier's record (exit status, the tests, the diff)
settles it.
beneficialis never set at run time, only by a benchmark from a measured gain. - Action.
Action { id, task, revision, evidence, operation, preconditions, authorization, expected, verification, source }plus status, milliseconds and result.operationis typed (search,decide,mutate,generate,edit,test,revert);sourceisrule,kai,modelorrecover. A saved typed operation is an action; a plan of actions is a workflow; a reusable validated result is the cache. Kai selects and binds an action's inputs and never writes shell. Status: eligible, attempted, completed, validated, selected, applied, verified, or ended early as abstained, rejected, timed_out, failed, escalated; each action records the path. - Order of cheapness, per step. An action that applies (one candidate, preconditions hold)
runs with no Kai inference. Else Kai's structured decision, all of a state's questions in one
request (
fix.site@1: site, action, effort;fix.turn@1: next, done, correct, grounded, complete, risk). Else low-effort generation, then full-effort generation, thenrecover. Every transition is a ledger line with its reason (fast,fallback); no step waits on Kai when an action already applies. Each program has its own outage cooldown: one capability's timeout never disables another. - Nothing waits that need not. The first state's first call is begun the moment the state is
ranked, beside the search (
flow::Ahead,zen::Zen::start), against that state's revision. The search settling the task, or a kept step moving the state, cancels it; when the search settles nothing the call has had the search's time to answer, and the ledger'sgeneratesays how long (ahead_ms). A cancelled call closes its stream (the server stops writing it) and leaves acancelledline in the spend file, its prompt counted at three characters a token at the ceiling rate; the ledger has acancelline, never agenerateone. An answer asked ahead is tried before any new call at its revision, and with one queued Kai is not askedfix.turn@1. - Revision-bound. Preconditions are checked immediately before execution; an edit invalidates
every pending decision made against the old revision. A revision is the content of the files the
run touched, and a revert returns to
base. A cached result is keyed on what it depends on (files, symbols, tool versions, environment; uncommitted changes count), never on a commit. - Recover proves progress. A failed verification records the failure's fingerprint (paths, line
numbers and ids normalized), the prior action, the resulting diff and the outcome. An edit that
leaves fewer tests failing is progress and stays; the state is looked at again. The same
fingerprint with no change of state makes
recoverpick a different action or escalate; it never replays the loop. - Search is one engine, tgrep (
hanzo-trigram, in process). It puts the lines that spell the failure's messages and names into the ranking;locateranks every function (kai-gateway's learned preflight weights,locate::testshold its features to the gateway's). - Fast path. The action
mutatetries generic token transforms (comparison and operator flips, constants off by one, arguments swapped, a condition inverted, areturndropped) at the best sites against the one visible test command, eight at once, capped in time; it names no repository, file or task. It solves single faults of that kind and nothing else. Past its 2.5 s cap it goes on while the call asked beside it has not answered, to 8 s (mutate::LONGEST): that wait is spent either way, and on a loaded host the cap cut 34 of 39 unsettled dev-split searches short of their candidates, the fix in the top four sites in 9 of them. A visible test too slow for the cap (its time x 1.5 > the cap) is narrowed to the failing tests it names (mutate::screen) instead of the search being skipped: lark's 18 s file runs its failing tests in 1.7 s, and the candidate picked is verified with the whole command in the tree as ever. A fast file narrowed ran 5% faster (two measurements, 13 and 19 tasks), not worth a second command there. - The tests around the failing ones (
wide) run beside the visible test after an edit, and a failing visible test stops them. What failed among them before any edit is read only when they fail after it, with the files the run touched put back as they were for that read and then as they are (flow::before, ledgerlazy): a run that passes shows nothing new, so on most tasks it is never read, and it never competes with the search for the CPU. - A test run leaves nothing. Files a run in the tree writes that git neither tracks nor ignores
and that were not there before it are removed after it, with the directories they emptied
(
tools::Litter, ledgersweep): they are the run's, never the edit's. What.gitignorenames is left alone, and so is a tree outside git. Hypothesis keeps its example database in the run's own temporary directory (HYPOTHESIS_STORAGE_DIRECTORY, unless the caller set it), so no run replays another's examples and none lands in the tree. - Jobs (not built): long work runs in the background and reports events into the same ledger. Start, progress, completion and verified outcome are four states; a finished process is not a finished task.
- Ledger.
kai/ledger.jsonlbeside the trace, replayable: stable ids for every decision, fallback, execution and verification so time and cost are attributable; credentials and sensitive output redacted. - Generation is Zen or enso-flash over an OpenAI-compatible endpoint (
kai.toml[generate]), called when needed, streamed so the first token and the last are both timed; a paid call is a line in the spend file with the model the server says served it, and the budget stops it. - The edit's path from a model (
zen.rs,edit.rs,flow.rs). At Hanzo's API an edit is asked for as a strict JSON schema (edits, at least one, eachfile,line,old,new).openai/gpt-6-soldecodes under it;enso-flashis asked and may still think aloud before the JSON (700 to 900 tokens, so an answer that does not reason has 2,048 of room), but with at least one edit required it stopped answering{"edits":[]}(6 of 13 calls without, 0 of 19 with). A call that reasons has one cap of 16,384 for reasoning and answer together, atreasoning_effortlow: a cap of 3,072 at medium lost five answers of ten to reasoning alone (finishlength, 3,072 of 3,072 completion tokens reasoning, no text), where medium reasoned 221 to 3,106 tokens and low 58 to 1,001. Every call that reached the model is kept undergenerate.callswith its finish reason and usage, reasoning tokens apart; a call without text, or ended by the server with an error, is kept with its error, billed in the spend file, and asked again.edit::parsesays which miss it is: no edits, JSON that does not read or is no edit, or no JSON. An answer refused for how it is written (it does not read, or its old text is not in the file) goes back to the model once, with the reason and the file's own lines where the edit said it stood (stage: repair); an escalated attempt that still has no edit goes to the generator once over the same prompt (stage: fallback, never an Anthropic model). Each is an action of its own in the ledger, the trace line lists every call (generations), and the edit is tested as ever: a parsed answer is not a solve. An answer of no edits is not said again (it costs the attempt, as before);{"edits":[]}from a model that names the file it would need to see is a context failure, not an output one. - Kai's transport is
decide::Transport: HTTPPOST /v1/decisionstoday. The local engine over ZAP on a Unix socket is the other implementation, and needs the engine to serve adecisionsmethod (the engine's ZAP module andhanzo-zap0.2.1 are stubs;zapwireis the working Rust ZAP wire).
House rules
- Hanzo service APIs use
https://api.hanzo.ai/v1/exclusively. - Keep the upstream interaction model and animation; the branding is ours.
- Keep Linux, macOS, and Windows working.
- Upstream project names appear only in
NOTICE— never in the README, the repository description, the CLI, or the interface.
Runtime & Model Fixes (2026-10-07)
- Model metadata: a model the fork's catalog (
hanzo::catalog) knows (enso-autoamong them) gets its window and modalities from the catalog, so it raises no "Unknown model ... fallback metadata" warning (hanzoai/codexcore:commit, models-manager): upstream marks fallback metadataModelOrigin::Synthetic, and a slug Hanzo's catalog knows resolvesBundled. - Sandbox permissions: a justification with no escalation asked for runs at the default
permissions instead of failing the shell call (
core:commit, core/src/tools/handlers). - ChatGPT legacy plugins 401: featured plugins are asked for only with a ChatGPT sign-in
(
auth.uses_codex_backend();core:commit, core-plugins). - Auth Symlink Guarantee:
hanzo-config::ensure_parent_and_symlinkensures~/.hanzo/auth.jsonis initialized with{}if absent before symlinking, preventing dangling~/.hanzo/dev/auth.json. - Token Refresh Command:
DEFAULT_CONFIGincludes dynamic token refresh configuration[model_providers.hanzo.auth] command = "hanzo auth token", refresh_interval_ms = 600000to prevent 60-minute JWT expiration mid-session.