zeekayandHanzo Dev
2b86b2ed46
The sweeper had two silent paths. runScheduler discarded the sweep error outright (`_ = e.sweepSchedules()`), so a sweep that failed wholesale -- cron dead for every org at once -- looked exactly like a quiet night. And inside sweepSchedules a schedule whose StartWorkflow failed hit a bare `continue`: correct isolation, zero output. That silence cost 11 days in production. cloud's clients/cron schedules fire a JobWorkflow whose RunJobActivity re-reads the entry's ConfigMap at fire time. The cloud ServiceAccount lacked RBAC to read ConfigMaps, so the activity failed on every single fire. The engine kept firing perfectly (cron-reconcile reached actionCount 4489) and emitted NOTHING, so every dashboard read healthy while all six nightly backups produced nothing for 11 days. It was found only by reading the engine's own durable records. Isolation is unchanged -- one broken schedule still must not stop the sweep. What changes is that it now says so, through the logger the package already uses (slog, from EmbedConfig.Logger; slog.Default() when unset). Everything needed to act goes on the line: org, namespace, scheduleId, workflowType, taskQueue, consecutiveFailures, error. Throttled, because a failed fire is deliberately not re-anchored: the entry stays due and retries on every 5s tick, so unthrottled this is ~17k identical lines per day per broken schedule, which buries the signal as thoroughly as silence did. First failure alerts, then one line per 60 sweeps (~5 min, so any 15-minute window still holds 3+ lines). The counter clears on success -- recovery is logged, and a relapse alerts immediately -- and on DeleteSchedule. Deliberately NOT "log on error-text change": StartWorkflow errors can embed the freshly minted random runId, which would flap every sweep and restore the very flood the throttle prevents. Tests fail without the fix (all five report `records: []`, the exact silence) and also catch a naive unthrottled implementation (60 lines vs 2). Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Tasks
Durable workflow execution engine for AI agent orchestration.
Introduction
Hanzo Tasks is a durable execution engine that powers AI agent orchestration in the Hanzo ecosystem. It enables developers to build scalable, fault-tolerant workflows that automatically handle intermittent failures and retry failed operations.
Tasks provides the backbone for:
- Playground spaces -- each space maps to a Tasks namespace
- Agent execution -- each agent runs as a Tasks worker
- Durable cron and batch jobs -- reliable scheduled and bulk operations
MIT licensed. See LICENSE.
Getting Started
Build
make tasksd
Run
./tasksd start
Or with a config file:
./tasksd --config-file config/development-sqlite.yaml --allow-no-auth start
Build from source
go build ./cmd/tasksd/
./tasksd start
Module
github.com/hanzoai/tasks
Integration
Hanzo Tasks integrates with the broader Hanzo ecosystem:
- Playground connects to the Tasks server via the durable-execution SDK
- Base embeds Tasks for durable cron/batch execution
- Each playground space = a Tasks namespace
- Each agent = a Tasks worker
Repository
This repository contains the source code of the Hanzo Tasks server.
The wire is luxfi/zap (binary, native, on _tasks._tcp:9999).
There is no gRPC and no go.temporal.io anywhere in the build.
To drive Tasks from Go, depend on pkg/sdk — the native
ZAP client. To embed the server in a Go process (e.g. a Hanzo Base
app), depend on pkg/tasks and call tasks.Embed().
Browser clients hit the JSON shim at /v1/tasks/* and the realtime
stream at /v1/tasks/events (Server-Sent Events). AI agents hit the
MCP surface at /v1/tasks/mcp (JSON-RPC 2.0).
Contributing
See CONTRIBUTING.md for development setup and guidelines.