mirror of
https://github.com/hanzoai/forge.git
synced 2026-10-30 15:00:39 +00:00
zeekayandHanzo Dev
5643098eea
A job whose `runs-on` names a label that no registered runner carries is not
slow, it is unroutable — and nothing in the scheduler said so. It stayed
Waiting. CancelAbandonedJobs eventually swept it, but that decides by elapsed
time: ABANDONED_JOB_TIMEOUT defaults to a day and is checked every six hours, so
such a job occupied the queue for up to thirty hours before being quietly
cancelled.
Whether a runner carrying `macos-13` exists is knowable when the job is created
and does not become more true by waiting thirty hours to ask.
Left to time alone they accumulate. One repository whose workflow named a matrix
of platforms we own no runners for leaked two permanently-unclaimable jobs per
push; a day's worth built up, and a release build queued behind them on a
saturated fleet. They also lie while they wait: the run looks like it is about to
start, right up until it is cancelled a day later.
So FailUnsatisfiableJobs sweeps every five minutes and fails jobs no registered
runner can match, on their own run, where whoever wrote the label can see it.
Two things it deliberately does not do:
* It tests REGISTRATION, not liveness. An offline runner is coming back and
must keep its queue; only a label nothing on the instance claims at all is
unsatisfiable. Failing queues during a fleet rollout would be the worse bug.
* It ignores jobs with no labels at all. Those are unrouted, which is a
different fault, and not something this reaper has evidence about.
UNSATISFIABLE_JOB_GRACE (15m) covers the window where a job is created before its
runner registers. The write goes through FailWaitingJobs, conditioned on the row
still being (task_id = 0, waiting), so a runner claiming the job in the same
moment simply wins and the job is absent from the returned slice.
Tests cover both directions, since a reaper that eats routable work would be
worse than the backlog it prevents: unroutable fails; routable keeps waiting no
matter how old; an offline-but-registered runner keeps its jobs; the grace period
protects young jobs; unlabelled jobs are untouched; a claimed job survives.
Each was checked against a mutant — satisfiable forced true, the grace ignored,
and the claim guard dropped — and each mutant is caught by a distinct test.
Co-authored-by: Hanzo Dev <[email protected]>