dataset: settled waits for the slot to clear, not just for the status
CI/CD / containment (push) Successful in 1m37s
Hanzo CI/CD / cicd (push) Successful in 39m10s
CI/CD / gate (push) Successful in 39m10s
CI/CD / image (push) Canceled after 0s
CI/CD / rollout (push) Canceled after 0s
CI/CD / reach (push) Canceled after 0s
CI/CD / fanout (push) Canceled after 0s
CI/CD / receipt (push) Canceled after 0s
CI/CD / containment (push) Successful in 1m37s
Hanzo CI/CD / cicd (push) Successful in 39m10s
CI/CD / gate (push) Successful in 39m10s
CI/CD / image (push) Canceled after 0s
CI/CD / rollout (push) Canceled after 0s
CI/CD / reach (push) Canceled after 0s
CI/CD / fanout (push) Canceled after 0s
CI/CD / receipt (push) Canceled after 0s
TestADisposalIsIdempotent failed the release gate on a 409 — "a materialisation of this dataset is running" — for a materialisation that had already finished. The two facts land at different instants. plane.run publishes the version with p.record and gives the tenant's scan slot back from a DEFERRED p.release, so between the status going terminal and the slot clearing describe answers status: ready, running: true. settled returned on the status alone, inside that window, and dispose refuses a held slot — so the next call 409'd on a job that was over. The CI log says exactly this: the FAIL is printed BEFORE the 'dataset materialised' line for the same dataset, and that line is logged AFTER p.record — so the version was published, the test read it, disposed, and was refused by a slot whose owner had not yet returned. It is microseconds wide, which is why it passes on an idle machine and failed on a loaded runner. running is already on the wire for this (typed.go view), so settled now waits on the state the refusal is made from instead of a proxy for it. No production behaviour changes: a client polling describe was always told running: true, and always had to wait for it. 30 consecutive runs green under CPU contention. Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
@@ -181,8 +181,22 @@ func declared(name string) riskDatasetSpec {
|
||||
}
|
||||
}
|
||||
|
||||
// settled waits for a materialisation to reach a terminal state, then returns the
|
||||
// version as `describe` reports it.
|
||||
// settled waits for a materialisation to reach a terminal state AND for the job
|
||||
// holding the tenant's scan slot to let go, then returns the version as
|
||||
// `describe` reports it.
|
||||
//
|
||||
// Both halves are load-bearing, because they are not the same instant. The job
|
||||
// publishes the version (plane.record) and only then returns, and the slot is
|
||||
// given back by a DEFERRED plane.release — so between the status going terminal
|
||||
// and the slot clearing there is a window where describe answers
|
||||
// `status: ready, running: true`. Waiting on the status alone returns inside that
|
||||
// window, and anything the caller does next that refuses a running job — dispose
|
||||
// is the one that does — fails as a 409 that reads like a real conflict. It is
|
||||
// timing, so it passes on an idle laptop and fails on a loaded CI runner, which
|
||||
// is the worst way for a gate to be wrong.
|
||||
//
|
||||
// `running` is on the wire for exactly this (typed.go view), so this waits on the
|
||||
// state the refusal is actually made from rather than a proxy for it.
|
||||
func settled(t *testing.T, app *zip.App, org, name string) riskDataset {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
@@ -200,10 +214,13 @@ func settled(t *testing.T, app *zip.App, org, name string) riskDataset {
|
||||
}
|
||||
switch v.Items[0].Status {
|
||||
case statusReady, statusRefused:
|
||||
return v.Items[0]
|
||||
if !v.Items[0].Running {
|
||||
return v.Items[0]
|
||||
}
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("materialisation of %s never settled (status %q)", name, v.Items[0].Status)
|
||||
t.Fatalf("materialisation of %s never settled (status %q, running %t)",
|
||||
name, v.Items[0].Status, v.Items[0].Running)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user