zeekay
1b367e2b62
The tree was a debranded snapshot of upstream taken around 2026-02-20, and upstream has moved about five months since. This replaces it with upstream master, mechanically re-derived: bin/debrand does the rename, and the pipeline and squashed migrations are restored from the shipped tree. Not a merge. The previous import replaced the tree wholesale in a single commit, so there is no shared content history to merge against. Reproducing the rename on the new tree is what actually converges. What survives: .hanzo/workflows, the Dockerfiles and bin/docker* entrypoints, compose files, LLM.md, and insights/migrations. The squashed initial migration is load-bearing — the live database records it by name, so upstream's ~918 individual migrations cannot come back. Nine products are dropped as unreferenced. Twenty-one more that we do not run are kept anyway because the core app still imports them; unpicking those is a refactor and does not belong in a rebase. This base does not build yet, and the reasons are not in the rename: upstream now wants Python 3.13.13 against our 3.12.12 and Django 5.2 against our 4.2, and the dependency manifests still name upstream packages that we republish under our own names. Those are decisions, not renames.
4.7 KiB
name, description
| name | description |
|---|---|
| extending-personinsights-test-harness | When and how to add scenarios, chaos events, and invariants to the personinsights e2e test harness (rust/personinsights-test-harness). Use after fixing a bug or regression in the personinsights leader path (leader, router, writer, replica, coordination protocol) so the fix gets a permanent regression scenario; when adding a new failure mode to test (crashes, drains, zombies, lag, failover); or when a new correctness property needs asserting during runs. Trigger terms: personinsights gate, chaos scenario, test harness, leader path regression, handoff bug, eviction, writer lag, acked write. |
Extending the personinsights test harness
The harness (rust/personinsights-test-harness/, see its README for usage) spawns a
real personinsights stack and asserts one invariant three ways: every acked write
is visible afterwards — live via read-your-write probers, at end-of-run via
strong reads, and durably via Postgres at the acked version. It has caught real
bugs (eviction data loss, unordered lifecycle shutdown, coordinator failover
stalls) precisely because chaos scenarios run against those assertions.
When to add a scenario
Add one whenever you fix a personinsights bug whose trigger the harness can reproduce — a crash, drain, restart, lag, election, or eviction condition. The fix's PR should show the scenario red before the fix and green after; that run becomes the permanent regression test. Also add one when introducing a new failure mode (a new process to disrupt, a new timing window) or a new correctness property (a new journal check).
Don't add scenarios for behavior already covered by an existing flag combination — compose existing flags instead — and don't add unit-testable logic here: the harness is for whole-stack behavior only.
Where things live
src/cli.rs— every scenario is driven byGateArgsflags.src/scenarios/gate.rs—ChaosEventenum,chaos_timeline()(offset-sorted events, paired stop/resume events), the dispatch loop, and verification ordering (probers → regressions → strong reads → Postgres quiesce).src/stack/mod.rs— stack primitives: spawn/kill/restart leaders (SIGKILL, SIGTERM drain, SIGSTOP/SIGCONT zombie), writer crash/pause, coordinator kill, etcd lease revocation, per-service env. New disruptions are new methods here.src/state.rs— the acked-write journal and its verifiers. New invariants go here, and their decision tables get unit tests: a false-negative verifier looks identical to a healthy stack, so e2e runs cannot reveal it.README.md— every scenario gets a runnable example; scenarios that stay red because of a known unfixed defect go in the "Known defects" section with their observable signature and fix direction.
Adding a chaos scenario
- Add the stack primitive if the disruption is new (prefer process signals and etcd operations; never disrupt the shared docker containers — they also serve the dev stack).
- Add the
GateArgsflag, theChaosEventvariant, and itschaos_timeline()entry. Timed events take an offset; paired disruptions (pause/resume) schedule both entries. State-triggered events (like--kill-handoff-target) poll etcd after their triggering event instead. - Guard invalid combinations in
gate.rswith an earlybail!(for example, coordinator kill requires two routers). - Rehearse locally until deterministic — three consecutive green runs is the bar (see the README for build and docker prerequisites). If timing makes a scenario a lottery, make it deterministic (the coordinator election is forced at bring-up for exactly this reason) rather than accepting flakes.
- Only failed-but-unacked requests are tolerable during chaos; the invariant
machinery needs no changes for a new disruption unless you're adding a new
property, in which case journal it in
state.rswith unit tests. - Add the scenario to the
personinsights-gateCI job in.github/workflows/ci-rust.ymlonce rehearsed — extend an existing composite run if compatible (keep runs to roughly three disruption kinds so failures stay diagnosable), or add a step. Expected-red scenarios (documenting a known defect) stay out of CI until the defect is fixed.
Validating a fix with the harness
Run the harness binary against binaries built from the fix branch:
cargo build -p personinsights-replica -p personinsights-router -p personinsights-leader -p personinsights-writer
personinsights-test-harness gate <scenario flags> --bin-dir <fix-worktree>/rust/target/debug
Show the same command red on the base branch and green on the fix in the PR's testing section.