hanzo.ai/e2e/coverage-manifest.md

App-surface coverage — what each test proves, and what it found

Eighteen tests over five specs, driving the LIVE app surface (/chat, /dev, the customize directory) on a running dev server rather than the export. They live in the chromium project, so — like the nine specs beside them — they cannot gate a build; e2e/gates remains the only thing that does.

BASE_URL=http://localhost:3000 pnpm exec playwright test --project=chromium \
  e2e/agentic-coding.spec.ts e2e/directory.spec.ts e2e/chat-send.spec.ts \
  e2e/mobile-app.spec.ts e2e/headings.spec.ts --workers=1

--workers=1 is not a preference. Two workers share one dev server, which compiles a room on first request; under that load the modal assertions expire and report a broken panel. The same tests pass in isolation and fail together at --workers=2, which is a fact about the server, not the app.

---

The result

16 passed · 1 skipped · 1 failed, in 49s against a stable tree.

The single failure is a REAL FINDING, not a spec defect, and it is named below — four rooms ship no heading. The skip is honest too: the run board is the signed-in answer at /dev, and no signed-in browser session is currently obtainable.

The stable tree is forge/main served from this worktree on :3100. Against the shared :3000 the same suite reports more failures, and mostly not because anything is broken — see the next section. :3000 is still the right target for the board tests specifically: the gateway echoes Access-Control-Allow-Origin for http://localhost:3000 and no other origin, so a browser on any other port reads an empty registry however healthy it is.

---

The target was rebuilt underneath this work

/dev and /chat were being rewritten continuously. Measured from the shared checkout's own log, in order: the sixth rung renamed Code → Dev; the rail cut to two rooms; the Customize rung deleted and its taxonomy moved inside Settings; the rail restored to six; /dev made state-dependent (product signed out, board signed in); the hanzodev landing deleted; /chat rebuilt as a team channel with a rich composer; /dev returned to a marketing page with an install command and an FAQ. Plus two dev-server restarts mid-run.

At one point the suite went from 13 green to 15 red in a single commit with nothing broken — /dev had simply begun answering signed-out readers with the product instead of the board.

That is why nothing here asserts the surface's shape. Every label-shaped anchor tried was outlived within the hour, and each is recorded at its call site in e2e/app.ts so it is not tried again:

tried	died when
the rung labels (Code, Customize)	renamed, then deleted
[data-slot="sidebar"] as the rung container	the rungs moved to a separate unnamed column
[data-slot="composer"] / composer-send	/chat became a channel; both slots gone
the model combobox as the settle signal	it ships in the HTML and DISAPPEARS on hydration
Sign in as the settle signal	a later shell stopped drawing it

What replaced them is structural: rooms are found by aria-current, the directory by the taxonomy it CONTAINS rather than the door that opens it, the composer by ROLE, and a room is settled when it has hydrated and stopped changing.

---

AGENTIC CODING — e2e/agentic-coding.spec.ts

Test	Proves
the workspace mounts and the board agrees with the registry	the room renders, the gui stylesheet arrived, and every row on the board names a run the gateway actually returned — the empty state is asserted only when the registry is genuinely empty	PASS
the rail carries a reader to every room, and says where they are	every rung arrives at a real room, lands somewhere distinct, and is marked current on arrival	PASS
opening a run shows its transcript and offers Pause, Resume and Stop	a run seeded through the real API appears by name, opens, and offers all three steer verbs	SKIP — the board is the signed-in answer, and no signed-in session is obtainable
the board never claims zero while it is still reading	the export never ships No runs yet as its first word — unknown must not render as zero	PASS

The board test is an AGREEMENT test, the house pattern from catalog-agreement.spec.ts. An empty board is CORRECT when the registry is empty and a DEFECT when it is not, so asserting either state unconditionally would be wrong half the time.

The registry is read from the PAGE, not from the test process — same origin, same credential, same question. Reading it from Node held a board that could see nothing against a registry the test could see perfectly, and reported nine missing rows on a room behaving correctly.

Seeding works and is real. POST /v1/agents/sessions answers 201 and POST …/{id}/events 201, so the transcript test runs against a run this suite created. Two things about it are worth knowing:

• The row is titled by AGENT, not by title (agent || title || id), so a fixture needs a unique agent name. Seeded as bugfixer it is indistinguishable from every other bugfixer run.
• The fixture cannot be cleaned up. No DELETE (405), and stop answers 200 while leaving the session running — steering takes effect between turns and a seeded run has no worker to take one. Measured: eight stopped this way were all still running afterwards.

---

CONNECTORS → APPS — e2e/directory.spec.ts

Test	Proves
the taxonomy is the ruled four, in order and unbroken	Apps · Channels · Plugins · Skills appear in that sequence with nothing wedged between them, and the panel lands on Apps	PASS
Apps rows are real, counted honestly, and each one opens a page	the catalog renders, the tally equals the rows drawn, every row links in the one published shape, search narrows, and a query matching nothing says so instead of reporting zero	PASS
an Apps row opens a page this site really serves	a row's href answers 200 with a real body — not a catalogue of dead ends	PASS
Channels, Plugins and Skills say they have not asked, never that there are none	each states its condition, carries the verbatim "it has not asked" sentence, offers the console, and renders no tally at all	PASS
Settings offers the personal agent under Preferences	the Preferences group exists, Personal agent is offered, and selecting it says nothing is bound yet	PASS

The rows are read off the rendered surface rather than an imported fixture. That was forced — @hanzo/app/lib/integrations reaches lib/data/pricing.json through the model-count module, and a JSON import with no type attribute is a hard error under Playwright's loader — and it is the stronger claim anyway: a fixture proves the array, not the page.

---

SKILLS + CHAT — e2e/chat-send.spec.ts

Test	Proves
the composer takes a draft and the room answers in its own words	a draft arms the send, and the send ends in one of the room's honest endings — the reader's own turn, a consent screen, a streamed reply, or a worded refusal — never in silence	PASS
the model surface names something real	the gateway serves a real catalog of named models; if the room offers a picker it must open onto real rows	PASS

Most outcomes are correct, so most are accepted. A spec demanding a model reply would fail the day the account runs out of credit — and the day /chat became a team channel, which has now happened — reporting a broken composer both times. What the test refuses is SILENCE: a draft that disappears with no turn, no reply and no error.

The free lane's refusal is not hypothetical:

POST /v1/chat/public {"model":"free"} -> 402
{"code":"public_allowance_spent","message":"You've used today's free
 messages. They reset at midnight UTC — sign in at https://hanzo.ai…"}

That is a page doing its job, so the test passes on it and records the wording. The refusal is matched by its RETRY CONTROL rather than data-slot="failure" — the room draws its own box rather than using @hanzo/ui's Failure.

free is not in the gateway's 86-model catalog. Recorded as an annotation, not asserted: the public lane may route an alias the catalog does not publish.

---

HEADINGS — e2e/headings.spec.ts

Test	Proves
/chat ships exactly one heading	the shipped bytes carry one non-empty <h1>	PASS
/dev ships exactly one heading	as above	PASS
every room a rail can reach names itself	every room the rail reaches ships exactly one <h1>	FAIL — real, see below

This is its own spec, and small, because it is the claim four other tests kept failing FOR. Using the heading as a precondition made one missing element report as a broken board, a broken rail and a broken composer.

---

MOBILE, 390px — e2e/mobile-app.spec.ts

Test	Proves
both rooms fit, at rest and with the rail open	nothing on /chat or /dev reaches past a 390px viewport once the room settles	PASS
the composer stays on screen and takes a draft at 390	the composer's box is inside the viewport and takes a draft	PASS
a phone can still reach the rail, the rooms and both app actions	opening the drawer exposes the rooms and Settings	PASS
the modals fit the phone they open on	the directory and settings panels fit 390px on every tab	PASS

Overflow is measured on ELEMENTS, never scrollWidth: app/globals.css states overflow-x: clip on html and body, so documentElement.scrollWidth > clientWidth is unobservable and a gate written that way cannot fail.

It is measured at REST, not at first paint. /chat's send button is 42px wide and reaches x=439 in a 390px viewport — 49px off the right edge — from first paint until about two seconds in, then reflows to fit. That is a real if brief defect and it is recorded here rather than asserted: holding the hydration frame makes the gate fire on timing rather than layout. What is held is the state a reader is left in.

---

The findings

1. Four rooms ship no heading at all

/inbox ships 0 · /work ships 0 · /board ships 0 · /drive ships 0

/chat and /dev are correct. The other four rooms have no <h1>, so a crawler cannot tell them apart, a screen reader lands on nothing, and a visitor gets no answer to "which page is this". This repo has shipped exactly this before and wrote down why it goes unnoticed: "Nothing catches this — the export is valid, the build is green."

2. THE PUBLISH GATE WENT RED, AND A RED GATE FREEZES THE SITE

Not inferred — MEASURED. Commit f367b670 was checked out into its own worktree, built with next build --webpack, and the existing gate run against the real out/:

playwright test --project=gates e2e/gates/workspace.spec.ts
4 failed

All four tests of e2e/gates/workspace.spec.ts, which is unchanged since before this redesign. Read off the export at that commit:

the gate requires	out/dev.html	out/chat.html
exactly one <h1>	1 ✓	0 ✗
its own heading	Code with Hanzo anywhere absent	What’s up absent
data-slot="sidebar"	0 occurrences	1 ✓
data-slot="composer"	—	0 occurrences

pnpm gates is a STEP IN THE PUBLISHING JOB, and this repo's own notes record the cost: "four gates went red and hanzo.ai published nothing for a day while pushes kept succeeding." The gate has to move with the design, in the same commit. Re-measure it before the next deploy — the surface has moved several times since — in a scratch worktree, never in the shared one:

git worktree add /tmp/gc HEAD && cd /tmp/gc
ln -s <repo>/node_modules . && cp <repo>/.env.local .
./node_modules/.bin/next build --webpack
./node_modules/.bin/playwright test --project=gates e2e/gates/workspace.spec.ts

3. Two mobile regressions, since fixed

Both were measured at a7da7e70 against forge/main and both are now GONE — recorded because the bisect is the useful part, not the bug.

• /chat overflowed a phone by 6px: a 180px row carrying Sign in reaching to 396 in a 390px viewport. Absent on forge/main, absent again today.
• Customize and Settings were unreachable on a phone — absent from the DOM, drawer open or shut — and the room rungs went with them.

4. Not a regression, but it reads exactly like one

The Settings rung sat in the BOTTOM-LEFT corner, which is where next dev mounts its <nextjs-portal> badge. document.elementFromPoint at the rung's centre returned NEXTJS-PORTAL, so the press landed on the badge and the panel never opened — silently, no console error. Customize, one rung higher, was unaffected, which is what made the pair look like a broken handler. Dev-only: the export has no portal. quiet() hides it so the measurement is of the product.

---

What the harness had to learn

Each cost a wrong answer before it was written down. They live in e2e/app.ts.

• gui buttons are not disabled, they are aria-disabled. Playwright's actionability check reports "element is not enabled" and click() burns its full timeout instead of failing fast.
• fill() does not reach a gui-controlled field. It is driven through onChangeText; a fill sets the DOM value while the React draft stays empty. Type the keys, then confirm the draft landed.
• No named control is a safe settle signal. A room is settled when it has hydrated and its control count has stopped changing.
• Two shells disagree about aria-current. One marks every rung ("true"/"false"), the other only the active one. Reading the marked elements alone returned six rooms on one and one on the other; reading each marked element's SIBLING GROUP works on both.
• aria-current also marks in-page tabs. On /dev the marked controls were Chat, Dev, Build, CLI — two rooms and two tabs, identical in markup. Only pressing one separates them, so the rail test classifies by outcome.
• Wait for the RIGHT thing. The rail test's first wait — "the rail is back" — was true instantly, so the URL was read before the transition finished and every room was recorded as the previous one, one place out all the way down.
• A dev server with a second writer on the tree reloads under the test. Seven full navigations in fifteen seconds, which wipes a draft mid-word and undoes a router.push. go(), type(), compose() and panel() retry.

---

What is genuinely blocked

Signing in. The gateway refuses the IAM session cookie outright — 403 a validated principal is required with a perfectly good __Host-hanzo_session — and the local PKCE path answers invalid redirect_uri for every URI tried, so hanzo-console's registered set is not reachable from here.

That now blocks more than it used to. /dev answers by reader state, so the run board, the transcript and the steer controls are all behind sign-in and a signed-out suite can only reach the product page. Closing this is the single highest-value next step for this coverage: everything the agentic-coding brief asks for lives on the other side of it.

The signed-out surface still reads the registry, because the dev server carries NEXT_PUBLIC_HANZO_WIDGET_KEY — a local, short-lived token. token() reads it from the environment at run time and no spec writes it down.