mirror of
https://github.com/hanzoai/tasks.git
synced 2026-10-04 02:04:03 +00:00
zeekayandHanzo Dev
0daf1a06ed
The sibling fix made the SCHEDULER report. It does not cover the incident.
StartWorkflow SUCCEEDED all 4489 times -- cron-reconcile's actionCount is the
proof -- so the sweeper was healthy and silent by correctness. The fault was
one layer down: cloud's JobWorkflow fires RunJobActivity, which re-reads the
entry's ConfigMap at fire time, and the cloud ServiceAccount had no RBAC to
read ConfigMaps. Every fire's ACTIVITY failed, for 11 days, and all that layer
did about it was e.emit(Event{Kind:"activity.failed"}) into the SSE broker:
ephemeral, no subscriber attached, nothing written anywhere. Six nightly
backups produced nothing, every dashboard read green, and it was found only by
hand-reading the engine's SQLite.
Guarantee provided: DURABLE evidence, plus throttled log lines. Both, not
either. A "fail/<ns>/<fingerprint>" row per failing identity is upserted on
every failure and deleted on the first success. It needs no subscriber and
outlives the process, so ConsecutiveFailures can say 4489 where an in-memory
counter says 1. View.FailureStreaks(ns) hands the rows back worst-first, so
nobody opens SQLite again; the keyspace is bounded by how much is broken now,
not by how often it failed.
Identity is the recurring SHAPE of the work -- (workflowType, activityType,
taskQueue, scheduleId) -- never the run. Every cron fire mints a fresh runId,
so a run-keyed counter would have read "attempt 1 of 10" forty thousand times
and never once said "this has been dead for 11 days". Schedule-started runs now
carry a ScheduleId search attribute (sweeper + TriggerSchedule) so a report can
name the dead cron entry instead of saying "some JobWorkflow is broken"; it
makes ScheduleId a queryable execution filter at the same time.
Three levels, because the distinction is the whole point. WARN "activity
failed" for a failure that will be retried -- normal operation, must not page
anyone. ERROR "activity failing persistently" once the streak has run
unbroken for 15 minutes -- the alertable condition that was invisible.
INFO "activity recovered" when it ends. Same three for a workflow failing with
no activity to blame, which is the hole an activity-only report would leave (a
decider that faults on episode 0 never schedules anything). Persistence is
measured by AGE, not count: a nightly backup reaches a count of 2 in two days
yet is unambiguously broken, while one run burning its 10 default attempts
(~5.5 min of backoff) must not trip it. The persistent transition always
reports whatever the throttle says -- waiting for failure #60 would be the
original bug wearing a different hat.
Throttle reuses the sibling's idiom rather than inventing a second one:
scheduleFailTracker generalises to failTracker (cadence fixed at construction),
sweepFailThrottle to failReportThrottle, and shouldReport is now the single
rule for the whole package. One tracker instance serves schedules and
activities. Two counters exist on purpose and do different jobs: the in-memory
tracker is the throttle authority (a store that is itself failing cannot starve
it into reporting every occurrence, and a restart makes a still-broken activity
re-alert once after a deploy), while the durable row is the evidence and always
supplies the count on the line. Measured: 180 failures produce 4 lines; stub
shouldReport to always-report and it is 180.
Retry and isolation semantics are UNCHANGED -- which branch is taken, when, and
how many times is decided exactly as before, and every emit() stays.
TestFailureReportingPreservesRetrySemantics pins the attempt count, terminal
status, ACTIVITY_TASK_FAILED event count and dispatch count.
Ten tests, all failing without the reporting calls and all on assertions
("rows: []", "records: []" -- the exact silence), covering: durability with no
subscriber, survival across a simulated restart with the count continuing,
naming the schedule end-to-end from a real sweep, the throttle, the
WARN-vs-ERROR distinction, recovery, standalone activities, workflow-only
failures, retry semantics, and per-org isolation of streaks.
Co-authored-by: Hanzo Dev <[email protected]>