dataset: settled waits for the slot to clear, not just for the status
CI/CD / containment (push) Successful in 1m37s
Hanzo CI/CD / cicd (push) Successful in 39m10s
CI/CD / gate (push) Successful in 39m10s
CI/CD / image (push) Canceled after 0s
CI/CD / rollout (push) Canceled after 0s
CI/CD / reach (push) Canceled after 0s
CI/CD / fanout (push) Canceled after 0s
CI/CD / receipt (push) Canceled after 0s

TestADisposalIsIdempotent failed the release gate on a 409 — "a materialisation
of this dataset is running" — for a materialisation that had already finished.

The two facts land at different instants. plane.run publishes the version with
p.record and gives the tenant's scan slot back from a DEFERRED p.release, so
between the status going terminal and the slot clearing describe answers
status: ready, running: true. settled returned on the status alone, inside that
window, and dispose refuses a held slot — so the next call 409'd on a job that
was over.

The CI log says exactly this: the FAIL is printed BEFORE the 'dataset
materialised' line for the same dataset, and that line is logged AFTER
p.record — so the version was published, the test read it, disposed, and was
refused by a slot whose owner had not yet returned.

It is microseconds wide, which is why it passes on an idle machine and failed on
a loaded runner. running is already on the wire for this (typed.go view), so
settled now waits on the state the refusal is made from instead of a proxy for
it. No production behaviour changes: a client polling describe was always told
running: true, and always had to wait for it.

30 consecutive runs green under CPU contention.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
2026-08-06 02:07:31 -07:00
parent 0d7011e2b3
commit 41a0e5a3b5
+21 -4
View File
@@ -181,8 +181,22 @@ func declared(name string) riskDatasetSpec {
}
}
// settled waits for a materialisation to reach a terminal state, then returns the
// version as `describe` reports it.
// settled waits for a materialisation to reach a terminal state AND for the job
// holding the tenant's scan slot to let go, then returns the version as
// `describe` reports it.
//
// Both halves are load-bearing, because they are not the same instant. The job
// publishes the version (plane.record) and only then returns, and the slot is
// given back by a DEFERRED plane.release — so between the status going terminal
// and the slot clearing there is a window where describe answers
// `status: ready, running: true`. Waiting on the status alone returns inside that
// window, and anything the caller does next that refuses a running job — dispose
// is the one that does — fails as a 409 that reads like a real conflict. It is
// timing, so it passes on an idle laptop and fails on a loaded CI runner, which
// is the worst way for a gate to be wrong.
//
// `running` is on the wire for exactly this (typed.go view), so this waits on the
// state the refusal is actually made from rather than a proxy for it.
func settled(t *testing.T, app *zip.App, org, name string) riskDataset {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
@@ -200,10 +214,13 @@ func settled(t *testing.T, app *zip.App, org, name string) riskDataset {
}
switch v.Items[0].Status {
case statusReady, statusRefused:
return v.Items[0]
if !v.Items[0].Running {
return v.Items[0]
}
}
if time.Now().After(deadline) {
t.Fatalf("materialisation of %s never settled (status %q)", name, v.Items[0].Status)
t.Fatalf("materialisation of %s never settled (status %q, running %t)",
name, v.Items[0].Status, v.Items[0].Running)
}
time.Sleep(5 * time.Millisecond)
}