TASKSD_STORE=sql keeps every shard's rows in one table, tasks_kv, on the
hanzoai/sql server TASKSD_SQL_ADDR names (its ZAP listener), so a process running
the engine keeps nothing on disk and any process can take its place. A shard is
(principal, ns), the same value it always was; its rows are a fileKV in its own
SQLite file or an sqlKV in the table, behind one rows interface. Keys are bytea
compared as bytes, so a scan walks keys in exactly a shard file's order whatever
the server's collation; each value is sealed (AES-256-GCM) under the key a shard
file of the same tenant and namespace derives from the master, bound to its shard
and key, so the server holds only ciphertext. A shared store refuses to open
without a master key.
Copy moves every shard from one manager to another a page at a time, leaves the
destination holding the source's rows and no others, and proves each shard by
count and a SHA-256 over its rows in key order. SQLite stays the default; nothing
changes for a deployment that sets nothing.
Proved against a hanzoai/sql test server: scans equal a file shard's for UTF-8,
DEL and prefix-boundary keys; the server holds no plaintext; a crashed engine
recovers a workflow mid-activity from the table into a fresh data directory, the
activity running exactly once.
Co-authored-by: Hanzo Dev <[email protected]>
WorkflowExecutionRetentionTtl (default 720h) was stored and read nowhere,
so finished runs and every key they own stayed forever.
runPurge, beside the cron sweeper, now runs every 5 s across every tenant.
Each pass reads up to 512 of a namespace's run rows, workflow runs then
standalone activities, from a cursor at purge/<ns>, and removes each
terminal run whose close time plus the retention has passed: wf, wfh,
wfact, wfopen and the idem and sctrig entries naming it, or act, ahist and
the aidem entries naming it. Every delete goes through the replicated
delete path, the row last, so a pass cut short leaves the row to find the
rest by. The row is re-read under the run lock first; an open run is never
touched. A retention that does not parse purges nothing and is reported.
Each pass ends in Shard.Reclaim, and a default-mode shard is rebuilt at
the end of a walk, when it is smallest, so the file shrinks.
On a 1.71 GiB shard of 200k expired runs and 10 open ones, 392 passes
removed them (p99 197 ms a pass), the file went to 132 KiB, and Embed took
5 ms and 16 MB peak RSS.
Co-authored-by: Hanzo Dev <[email protected]>
Recover listed every run in a namespace and decoded each one, finished runs
included, to skip all but the few still open, so a restart cost more with
every run a shard had ever held.
wfopen/<ns>/<workflowId>/<runId> now names each workflow run that is not
terminal. It is put before a run's first row and deleted after its terminal
row, so a process stopped between the two leaves an entry that recovery
drops, never an open run the index misses. Recover lists it and re-drives
what it names.
A shard written before the index is walked once, a page at a time, decoding
only each row's status and times; wfidx/<ns> records that the walk is done.
On a 1.71 GiB shard of 200k finished runs and 10 open ones, Embed went from
10.4 s and 1.25 GB peak RSS to 7.7 s and 85 MB on the boot that builds the
index, and 6.0 s and 21 MB after it. What remains is the envelope
decrypting the whole file at open.
Co-authored-by: Hanzo Dev <[email protected]>
Shard.Scan reads at most limit rows of a prefix after a given key, so a
long walk holds the shard's one connection a page at a time; List is Scan
with no limit.
Shard.Reclaim returns free pages to the filesystem. A shard in incremental
auto-vacuum releases up to 4096 per call. A shard in SQLite's default mode
can only be rebuilt with VACUUM, so it is rebuilt when the caller says it is
done deleting and a quarter of it is free; the rebuild moves it to
incremental mode, so it happens once. incremental_vacuum yields a row per
page released, so it is drained as a query.
Co-authored-by: Hanzo Dev <[email protected]>
The sweep stamps TemporalScheduledStartTime on every run it starts: the nominal
tick (RFC 3339, UTC), not the moment the sweep got to it, so a tick fired late
after downtime and the same tick fired by a second engine carry one value. A
manual trigger is not a tick and carries none. ScheduledStart reads it back, and
View.DescribeWorkflow lets an in-process host read its own run, so a consumer
that makes one thing per tick (a cron Job) can tell a fire from a trigger.
Co-authored-by: Hanzo Dev <[email protected]>
A registration that is not a typed operation is spelled Raw now, with the method
named, so opting out of the typed model reads as what it is.
The zip this pins also states the wire explicitly, which matters more than the
version gap suggests: JSON semantics follow the toolchain, so the older pin
answered with different bodies than the code was written against once the
builder moved.
Co-authored-by: Hanzo Dev <[email protected]>
The create-schedule handler kept the action's workflow type and queue
and dropped its input, so every scheduled run started with none. The
forge schedules each cron job with its name as the input; with the name
gone every GitCronJob failed looking up the job "", and mirrors and
every other forge cron stopped. The request decode is now one function,
scheduleFromSDK, and it carries the input.
Co-authored-by: Hanzo Dev <[email protected]>
The tasks client and the SDK client dial the Tasks server and are never
started, so they bind no socket. Responses and server-pushed deliveries
arrive on the connection the client opened; with discovery off nothing
dials back.
TestZAPClientsListenNowhere asks the OS which sockets the process
listens on: the server adds one, both clients add none.
Co-authored-by: Hanzo Dev <[email protected]>
A composer had one way to mount the engine: take HTTPHandler and adapt it,
which puts sixty-odd operations behind one opaque wildcard the composing
router cannot see into. Surface registers them as real routes, so
app.Use(engine.Surface()) puts every address in the composed router and in
every projection that reads it.
Both renderings reach the same functions -- a call in, an answer out --
so the two transports cannot describe one operation differently, and
HTTPHandler is unchanged for callers that have it.
The wire is held by a test rather than by intent.
TestSurfaceAnswersWhatTheMuxAnswers drives the same requests through both
and compares status, body, Content-Type and Location. It replaced a probe
that printed the two answers and asserted nothing, and the first thing it
found was real: http.ServeMux tidies a path with an empty or dot segment
and redirects to the tidied form before matching, so /namespaces//workflows
was a redirect through the mux and a 404 here. The wildcard answers that
redirect now, with the body http.Redirect writes and only for the methods
it writes one for.
tasks generated a DEK per shard, wrapped it under a KEK derived from the
master, and stored the wrapped blob beside the database — so it also needed an
upgrade path to rewrap, a principal vocabulary to derive against, and tests
for all of it. hanzoai/cek derives the key from the master and the namespace
instead. Nothing is stored, so there is nothing to wrap, rewrap, or upgrade.
dek.go, principal.go and upgrade.go are gone with the tests that pinned them:
they proved properties of a scheme that no longer exists, not properties the
store still has to hold.
That also unpins hanzoai/sqlite, which deleted the DEK API in v0.5.0. tasks
now builds against it, which is what proves the migration is complete rather
than merely compiling.
Databases written under the old scheme will not open under the new
derivation. There are no customers and no data to preserve.
Co-authored-by: Hanzo Dev <[email protected]>
Minting is issuing authority. IAM mints — it signs the token, and that
signature is where the authority is created. tasksd verifies one and
restates it as headers a transport that cannot carry a token can read.
Nothing here creates authority, so naming both operations "mint" put the
issuer's job and this service's under one word, and a word that covers
two operations is a word that stops distinguishing them.
Every place that named the header write "mint" now names it "write",
matching the leaf it already cites: hanzoai/authz names its header list
"once by the party that WRITES their values". The four ctx accessors said
"minted" for a value that is neither issued nor a header — they return
what the validated claims RESOLVED to, which is the word the org scoping
already used for Claims.EffectiveOrg.
"mint" stays where it is the right word: the test helper signing a JWT,
the `owner` claim IAM stamps at sign time, the HMAC task tokens dispatch
issues, and money mint/burn in CONTRACT.md.
LLM.md also cited edge.Inject, which no longer exists. The call is
edge.Apply — the name it took when the four readers collapsed onto one
leaf, precisely because writing headers is not injecting authority.
Co-authored-by: Hanzo Dev <[email protected]>
authz v1.10.26: the `owner` claim carries the org of the APPLICATION a token was
minted through, so reading it as the subject's org made a person's durable state —
engine.As(Org(...)) scopes the store by it — follow whichever app they signed in
through. It is the first entry of the signed membership set now.
A token stating no org gets none, with no fallback, so tasksd refuses it when
identity is required rather than scoping a namespace to an app-selected value.
Also edge.Inject→Apply and edge.Mint→Render: minting is issuing authority and only
IAM does that.
Co-authored-by: Hanzo Dev <[email protected]>
TASKSD_WHITELABEL_ISSUERS adds the brands this tasksd accepts tokens from beyond
the primary, each of which signs its own. Adding one is configuration.
The sharp edge is the empty entry: splitting an UNSET env var yields one empty
string, and an empty trusted issuer would match a token that carries no `iss` at
all. Entries are trimmed and dropped, pinned by
TestIssuerAllowlistDropsEmptyEntries — which builds the verifier through the same
code path production uses, so the guard is tested where it runs rather than in
isolation.
Co-authored-by: Hanzo Dev <[email protected]>
pkg/auth held 190 lines of JWKS cache and JWT verification — a fourth independent
reading of what an IAM token means — and it had two defects the shared reader does
not:
it verified against EVERY key in the published set and took the first that
worked, so a token naming kid A was accepted on a signature from key B. Naming a
key is what `kid` is for.
its issuer comparison was CONDITIONAL on the configured issuer being non-empty,
so an unset TASKSD_JWT_ISSUER did not fail closed — it accepted tokens from any
issuer.
It also stripped only the four names it reads. Every other minted header —
X-User-IsAdmin and X-User-Permissions (the platform and money signals),
X-User-Owner, X-Billing-Account-Id, X-Scope, X-Workspace-Id, and the retired names
— passed through from the client untouched. tasksd terminates identity itself (its
ingress routes straight to the Service), so nothing upstream was deleting them
either. Restoring the four-name strip turns
TestRequireIdentity_StripsTheWholeEstateSet red on seven forged headers.
And it had NO machine predicate at all, so an admin-org client_credentials token
arrived as a person. It now reads the membership set, like the rest of the estate.
What remains is the part that is genuinely tasksd's: which environment variables
the policy comes from. The audience stays optional and that is deliberate rather
than an oversight — IAM sets `aud` to the requesting CLIENT (RFC 8707), and tasksd
is not a client id, so it pins the issuer and the signing keys instead.
Both transports now answer "which org does this act in" with the same function,
Claims.EffectiveOrg(""), rather than the HTTP path reading `owner` and the ZAP path
reading it separately.
Cost, measured: 342 linked packages and an 18.3 MB binary, DOWN from 345 and
18.6 MB — dropping go-jose more than paid for the leaf, and tasks links ZERO zip
packages because the edge names no transport. go-jose is gone from the module
entirely; the tests now mint with golang-jwt, the library IAM signs with.
Rebased onto 12 upstream commits: kept their ServeGated(addr string) signature,
their engine.As(Org(...)) rename, their dependency versions, and the LLM.md fact
about explicit project/user isolation. Their only change to the deleted jwt.go was
a comment naming hanzo.id, which the replacement already says.
Co-authored-by: Hanzo Dev <[email protected]>
An older binary addressed a shard by org alone (<data>/<org>/<ns>.db).
This one reads three levels down, so the first boot after the upgrade finds
nothing, creates empty shards beside the full ones, enumerates zero tenants
and never fires a schedule again — silently, with every byte still on disk
one directory up.
store.Upgrade inserts the two sentinel levels uniformly, including the old
empty-org dir (_ -> _/_/_). A rename within one filesystem: no page
rewritten, no re-encryption. Journals move before their database, so a run
that stops half way never leaves a database without the WAL holding its
last commits. Idempotent — a moved shard leaves nothing at its old path —
so a deploy can run it unconditionally ahead of the daemon.
Two refusals, both loud and both non-destructive:
A master key is set. Pre-upgrade shards are plaintext and moving one does
not encrypt it; the next boot would wrap a fresh DEK around a file it
then failed to decrypt. Move first, encrypt second.
A destination is occupied (ErrOccupied). The only way there is booting
this binary before the move ran, which creates an EMPTY shard where the
full one belongs — and anything started after that boot is in the empty
one. Both files are kept, the other tenants still move, and the operator
picks which is authoritative.
upgrade and migrate stay separate axes: upgrade moves EVERY shard forward
in time (layout), migrate moves ONE shard across space (nodes).
Also: refuse a socket path over sockaddr_un.sun_path by name. The kernel
answers "bind: invalid argument" and names neither the path nor the limit,
and a socket under a deployment's data directory sits close to it.
Co-authored-by: Hanzo Dev <[email protected]>
main's cron regression tests manufactured a failing action by naming an
unregistered namespace. This branch creates a namespace on first use, so
registration stopped being evidence of anything: a cron with a typo in its
namespace now starts a run into a namespace nobody serves, and says nothing.
Report the fault instead of the proxy. A fire is a failure when it does not
reach a worker — either the start errored, or the start succeeded and no
worker is subscribed to (namespace, taskQueue). Same line, same throttle,
same recovery, one reporter (engine.reportFire). It also catches what the
old guard never could: a schedule whose worker died, and a task queue
renamed out from under an entry.
Reachability is asked AFTER the start, because enqueueing prunes
subscriptions whose peer the transport can no longer reach — a worker that
died without unsubscribing is gone from the registry by the time we look.
Anchoring follows the start, not the report: an errored fire stays due and
retries, a started-but-unreached fire re-anchors, because the run exists and
re-firing every tick would spawn a run every 5s.
Co-authored-by: Hanzo Dev <[email protected]>
The sibling fix made the SCHEDULER report. It does not cover the incident.
StartWorkflow SUCCEEDED all 4489 times -- cron-reconcile's actionCount is the
proof -- so the sweeper was healthy and silent by correctness. The fault was
one layer down: cloud's JobWorkflow fires RunJobActivity, which re-reads the
entry's ConfigMap at fire time, and the cloud ServiceAccount had no RBAC to
read ConfigMaps. Every fire's ACTIVITY failed, for 11 days, and all that layer
did about it was e.emit(Event{Kind:"activity.failed"}) into the SSE broker:
ephemeral, no subscriber attached, nothing written anywhere. Six nightly
backups produced nothing, every dashboard read green, and it was found only by
hand-reading the engine's SQLite.
Guarantee provided: DURABLE evidence, plus throttled log lines. Both, not
either. A "fail/<ns>/<fingerprint>" row per failing identity is upserted on
every failure and deleted on the first success. It needs no subscriber and
outlives the process, so ConsecutiveFailures can say 4489 where an in-memory
counter says 1. View.FailureStreaks(ns) hands the rows back worst-first, so
nobody opens SQLite again; the keyspace is bounded by how much is broken now,
not by how often it failed.
Identity is the recurring SHAPE of the work -- (workflowType, activityType,
taskQueue, scheduleId) -- never the run. Every cron fire mints a fresh runId,
so a run-keyed counter would have read "attempt 1 of 10" forty thousand times
and never once said "this has been dead for 11 days". Schedule-started runs now
carry a ScheduleId search attribute (sweeper + TriggerSchedule) so a report can
name the dead cron entry instead of saying "some JobWorkflow is broken"; it
makes ScheduleId a queryable execution filter at the same time.
Three levels, because the distinction is the whole point. WARN "activity
failed" for a failure that will be retried -- normal operation, must not page
anyone. ERROR "activity failing persistently" once the streak has run
unbroken for 15 minutes -- the alertable condition that was invisible.
INFO "activity recovered" when it ends. Same three for a workflow failing with
no activity to blame, which is the hole an activity-only report would leave (a
decider that faults on episode 0 never schedules anything). Persistence is
measured by AGE, not count: a nightly backup reaches a count of 2 in two days
yet is unambiguously broken, while one run burning its 10 default attempts
(~5.5 min of backoff) must not trip it. The persistent transition always
reports whatever the throttle says -- waiting for failure #60 would be the
original bug wearing a different hat.
Throttle reuses the sibling's idiom rather than inventing a second one:
scheduleFailTracker generalises to failTracker (cadence fixed at construction),
sweepFailThrottle to failReportThrottle, and shouldReport is now the single
rule for the whole package. One tracker instance serves schedules and
activities. Two counters exist on purpose and do different jobs: the in-memory
tracker is the throttle authority (a store that is itself failing cannot starve
it into reporting every occurrence, and a restart makes a still-broken activity
re-alert once after a deploy), while the durable row is the evidence and always
supplies the count on the line. Measured: 180 failures produce 4 lines; stub
shouldReport to always-report and it is 180.
Retry and isolation semantics are UNCHANGED -- which branch is taken, when, and
how many times is decided exactly as before, and every emit() stays.
TestFailureReportingPreservesRetrySemantics pins the attempt count, terminal
status, ACTIVITY_TASK_FAILED event count and dispatch count.
Ten tests, all failing without the reporting calls and all on assertions
("rows: []", "records: []" -- the exact silence), covering: durability with no
subscriber, survival across a simulated restart with the count continuing,
naming the schedule end-to-end from a real sweep, the throttle, the
WARN-vs-ERROR distinction, recovery, standalone activities, workflow-only
failures, retry semantics, and per-org isolation of streaks.
Co-authored-by: Hanzo Dev <[email protected]>
The claims struct comment credited a third-party identity server. Hanzo IAM
is original work; the vendor-derived server was hanzoai/iam-v1, retired.
The claim set described (owner/sub/email) is unchanged.
The sweeper had two silent paths. runScheduler discarded the sweep error
outright (`_ = e.sweepSchedules()`), so a sweep that failed wholesale --
cron dead for every org at once -- looked exactly like a quiet night. And
inside sweepSchedules a schedule whose StartWorkflow failed hit a bare
`continue`: correct isolation, zero output.
That silence cost 11 days in production. cloud's clients/cron schedules
fire a JobWorkflow whose RunJobActivity re-reads the entry's ConfigMap at
fire time. The cloud ServiceAccount lacked RBAC to read ConfigMaps, so the
activity failed on every single fire. The engine kept firing perfectly
(cron-reconcile reached actionCount 4489) and emitted NOTHING, so every
dashboard read healthy while all six nightly backups produced nothing for
11 days. It was found only by reading the engine's own durable records.
Isolation is unchanged -- one broken schedule still must not stop the
sweep. What changes is that it now says so, through the logger the package
already uses (slog, from EmbedConfig.Logger; slog.Default() when unset).
Everything needed to act goes on the line: org, namespace, scheduleId,
workflowType, taskQueue, consecutiveFailures, error.
Throttled, because a failed fire is deliberately not re-anchored: the entry
stays due and retries on every 5s tick, so unthrottled this is ~17k
identical lines per day per broken schedule, which buries the signal as
thoroughly as silence did. First failure alerts, then one line per 60
sweeps (~5 min, so any 15-minute window still holds 3+ lines). The counter
clears on success -- recovery is logged, and a relapse alerts immediately
-- and on DeleteSchedule. Deliberately NOT "log on error-text change":
StartWorkflow errors can embed the freshly minted random runId, which would
flap every sweep and restore the very flood the throttle prevents.
Tests fail without the fix (all five report `records: []`, the exact
silence) and also catch a naive unthrottled implementation (60 lines vs 2).
Co-authored-by: Hanzo Dev <[email protected]>
A dispatch token held its tenant as the canonical string, so delivering
its result had to parse one back and decide what to do when it did not
parse. Falling back to the root tenant there would have written one
tenant's result into another's shard.
The dispatcher runs in-process with the engine, so it holds the
Principal itself. There is nothing to encode, nothing to parse, and no
fallback to get wrong. The encoding stays where it is actually needed:
on a replication frame and a migration job, which leave the process.
Co-authored-by: Hanzo Dev <[email protected]>
The engine took a PORT and the SDK an Options.HostPort, so neither end
could use a unix socket for traffic that never leaves the host. The
field name was not the only obstacle: luxfi/zap bound net.Listen("tcp",
":port") and reached a peer with net.DialTimeout("tcp", addr), so a path
was refused at the dialer with "missing port in address" and could not
be bound at all. That is fixed upstream (luxfi/zap Address + Network);
this pins the commit until it is tagged.
EmbedConfig.Address, Embedded.Address, ServeGated(addr) and
client.Options.Address each take ONE address, because a port is a detail
of one transport rather than a thing every listener has. A path binds or
dials a unix socket; anything else is host:port.
TestDialRequiresHostPort became TestDialRequiresAddress: Dial needs an
address, and asserts a socket path is refused for being absent rather
than for being portless. TestE2E_UnixSocket runs a real workflow over a
socket end to end, and TestE2E_UnixSocketRebind proves the socket is
unlinked on stop so a restart binds cleanly.
Co-authored-by: Hanzo Dev <[email protected]>
The store addressed a shard by (org, namespace) and only the root org's
`default` namespace existed, so anything else had to be declared before
it could be used and there was no way to isolate a project or a user.
Shards also had no way to be keyed, so tasks was the one Hanzo store
writing tenant state in the clear.
Both are the same missing value, so both are one change.
Principal is the tenant that owns a shard: an org, optionally narrowed
to a project and a user. It serves the two places tenancy is decided, so
they cannot disagree — the shard's directory
(<data>/<org>/<project>/<user>/<ns>.db, each unset leg written "_") and
the key its DEK is wrapped under. The fixed-depth encoding is injective,
so a project and a user of the same name are different tenants, and the
KEK chain master -> org -> project -> user gives each of them a key
contained in the one it narrows.
Namespaces come into existence on first use (engine.namespace), the way
their shards already did. ValidName is the one rule every name that
becomes a path segment passes through, so a namespace can no longer
escape or alias a directory.
EmbedConfig.MasterKey opens every shard encrypted: each file carries a
DEK of its own, wrapped for its principal and bound to a random file id
in a sidecar beside it, so rotating the master rewraps sidecars and
never rewrites ciphertext, and moving the data directory is harmless.
A keyed shard is ciphertext on every build — hanzoai/sqlite v0.4.0 uses
the live libsqlcipher codec when one is linked and its pure-Go SQLCipher
envelope when one is not — so there is nothing here to branch on. On the
envelope a shard is sealed at checkpoint/close rather than per commit,
which is what Shard.Checkpoint and the manager's sweep bound.
replication.Frame and migration.Job carry the canonical principal string
instead of an org, so a frame lands in the directory its principal names.
Co-authored-by: Hanzo Dev <[email protected]>
The bare /.well-known/jwks form is only answered by the hanzo.id-worker
shim, which is being retired; IAM's discovery document advertises
/v1/iam/.well-known/jwks on both the public host and in-cluster.
A host subsystem (cloud functions fleet executor) enqueues standalone
activities on an org queue and polls them without an HTTP loopback hop,
mirroring the schedule passthroughs already on View.
Brings CancelActivityForOrg + ActivitiesPageForOrg onto main and makes the v1.51.2
and v1.51.3 release tags reachable from main's history (they were cut off v1.51.1 as
a detached patch line so cloud's pinned tree stayed valid). Additive methods, no
behavior change to the engine.
ActivitiesForOrg returns only the first hash-ordered 100 rows; a busy org past ~100
lifetime activities then sees a random subset — live jobs hidden, online workers
dropped. ActivitiesPageForOrg exposes the cursor ListActivities already carries so an
in-process host can walk a namespace to completion. Additive; org-scoped tenancy is
the caller's to enforce. (v1.51.3 = v1.51.1 + CancelActivityForOrg + this.)
Mirrors ActivitiesForOrg's org-scoped read with a matching mutator so a host
binary (cloud's clients/visor fleet) can cancel a queued/running gpu-jobs
activity it just listed, without a second HTTP hop into its own tasks surface.
Tenancy is the caller's to enforce (validated X-Org-Id). Enables the cloud
POST /v1/fleet/jobs/:id/cancel endpoint.
The client now has one durable path (ZAP) and a local goroutine-timer
fallback for dev. Removes the tasksURL/httpClient submit path and the
duplicated ZAP opcode/field-offset constants; New(zapAddr, handler) loses
its tasksURL argument. embed.go decodes a schedule's interval spec from the
wire alongside cron. Tests boot an embedded Tasks server on a free ZAP port
and assert Now()/Add() persist workflows and schedule specs over ZAP.
go build + go test ./pkg/tasks/ green.
The quasar-driver package move left FrameKey() defined as a method on the
imported replication.Frame → 'cannot define new methods on non-local type'
broke the build. Make it a free function (same shape as frameID); no callers
change (it had none).
hanzoai/base transitively pulled github.com/luxfi/consensus solely through
pkg/tasks → replication (quasar.go, the only consensus importer). That made
base — and every base consumer — vulnerable to the recurring luxfi/consensus
force-moved-tag / vanished-geth supply-chain rot, for a driver base never uses.
Make Quasar replication optional by isolating it:
- Add Kind() + a unified Stats(accepted,rejected,timeouts) to the Replicator
interface so embed.go introspects drivers through the interface, never a
concrete type.
- Move quasar.go (+ its test) into a new pkg/tasks/replication/quasar package
that imports the interface package; the consensus dependency now lives ONLY
there. Only cmd/tasksd imports it.
- pkg/tasks → replication now has ZERO consensus in its dep graph
(`go list -deps ./pkg/tasks | grep consensus` = 0), so base sheds it via Go
1.17+ module-graph pruning.
Also unbreak tasks' own consensus: v1.22.85 → v1.36.3 (geth v1.16.73 was a
vanished revision; v1.36.3 floors the valid unified luxfi stack). Non-quasar
packages build + test green CGO-off; the quasar package builds in CI (its
consensus→secp256k1 CGO can't compile in this sandbox — environmental).
store.bootNS was a write-only set (never read) guarded by a value-typed mu.
withOrg() shares the map by reference but COPIES the mutex, so N org-scoped
store views each locked a DIFFERENT mutex while writing the ONE shared map —
zero mutual exclusion. Under concurrent cross-org traffic the runtime fataled
with 'concurrent map writes' (store.go:100, via shardFor on every put/get),
crashing the whole embedded tasksd — i.e. the entire cloud process / api.hanzo.ai.
Fix is a decomplection: the set was dead state, so delete it (and its now-unused
mu + sync import + the per-op lock dance in shardFor). No map, no race. Net -11
lines in store.go. Adds TestStore_ConcurrentCrossOrgNoRace which reproduces the
exact crash on the old code (fatal → FAIL) and passes on the fix (verified under
-race).
Co-authored-by: Hanzo Dev <[email protected]>
The IAM identity model is org/project/user; tasks carried only org+user.
IAMClaims gains `project`; RequireIdentity mints X-Project-Id (strip-list
⊇ mint-list unchanged — the header is stripped on every inbound request
and set only from a validated JWT); WithIdentity gains the project param
(in-process embedders pass their validated project through); ProjectID(ctx)
accessor. Convention: a project maps onto a tasks NAMESPACE inside the
org's shard.
Co-authored-by: hanzo-dev <[email protected]>
* fix(engine): make the cron sweeper actually fire on the sharded store + org-scoped View
The 5s sweeper called store.list("sc/") — a cross-namespace, cross-org
scan the per-(org,ns) shard router refuses ("missing namespace in
prefix"). runScheduler discarded the error, so durable schedules NEVER
fired on the sharded store: every CreateSchedule (automations cron,
base durable cron, tasksd schedules) registered and then sat idle.
- store.listAllOrgs + store.Manager.ListOrgs: enumerate every org's
shards for root sweeps.
- sweepSchedules: two-phase (collect due → fire) so starting a workflow
never nests a query inside the shard List iteration (single-conn
SQLite deadlock); fires AT MOST ONE action per schedule per sweep and
re-anchors at now — recovering after downtime runs the schedule, it
does not storm the missed backlog; per-schedule error isolation.
- nextScheduleFire honors Spec.Interval (what base's durable client
writes), not only CronString.
- Embedded.View(org): exported in-process, org-scoped control-plane
handle (RegisterNamespace/CRUD/Trigger schedules) for host binaries
embedding the engine — cloud's platform cron mounts on this.
Tests: sweep fires (root + org shard, no cross-shard leak), backlog
clamped to 1, interval specs fire, broken entry doesn't poison the
sweep, View CRUD e2e. go test -race ./pkg/tasks/... green.
* ci: replace legacy Temporal workflow suite with canonical hanzo.yml + cicd.yml
19 upstream-era workflows (sharded run-tests matrices, promote-* image
lanes, flaky-test reporting, version-info triggers) targeted the pre-rip
fork layout and have been red on every run since the ZAP-native rewrite
(yq-less runners, deleted directories). One way: root hanzo.yml declares
the image + the -race test gate; cicd.yml imports hanzoai/ci build.yml on
the arc pool.
* ci: scope test gate to the packages that exist post-rip
* feat(embed): View.ListWorkflows — hosts observe their schedules' runs
---------
Co-authored-by: hanzo-dev <[email protected]>
luxfi/zap exposes no disconnect hook, so when a worker dies its dispatcher
subscription leaks. Round-robin then keeps picking the dead peer and the
server-push Send is silently dropped (the task is lost, never retried) — a
social orchestrator that restarts would stall on its 'main'/provider queues.
Make delivery self-healing: on a failed Send, prune the dead peer's subscription
and redeliver to the next live subscriber; queue only when none remain.
drainLocked now drains the whole backlog to a fresh subscriber rather than one
task per Subscribe.
Adds TestDispatcher_PrunesDeadPeerOnSendFailure (dead peer pruned, task reaches
the live peer, second enqueue skips the dead peer). Full pkg/tasks suite green.
Motivated by the social→Hanzo Tasks migration: verified live that a durable
workflow runs to COMPLETED over the gated cloud:9999 listener with a real
hanzo-social IAM token; delivery only stalled after leaked subscriptions
accumulated from worker restarts.
Co-authored-by: Hanzo Dev <[email protected]>
The cloud embedded engine links luxfi/zap v1.2.0 via MVS, whose NewBuilder stamps
wire version 2; the TS SDK only accepted version 1 -> 'unsupported version 2'
against cloud's gated :9999. The generic object encoding is byte-identical across
v1/v2 (only the platformvm TxKind schema differs, which this SDK never uses) and
v1.2.0's Parse accepts v1, so: keep emitting v1 (both engines accept it), accept
BOTH 1 and 2 on read. Verified live against cloud v1.786.102.
Also (drop-in compat, from the social migration):
- export TokenProvider from the package root.
- proxyActivities<A>: open the generic so proxyActivities<SomeActivityClass>()
type-checks; ProxyActivityOptions accepts cancellationType (no-op).
- engine.moreCurrentRun: a CONTINUED_AS_NEW run ranks below any other run of the
same workflowId, so the completed tail of a continue-as-new chain resolves over
its closed predecessor on a same-second StartTime (DescribeWorkflow by id).
The social-orchestrator migration off upstream Temporal onto the embedded
Hanzo Tasks engine needs five workflow primitives its digest/sendEmail/post
workflows depend on. Implemented server-side (event-sourced, replay-safe) plus
the @hanzoai/tasks TS SDK (the worker social runs):
- Signal → running workflow: engine.signalWorkflow now re-dispatches a workflow
task after appending WORKFLOW_EXECUTION_SIGNALED, so the stateless worker
replays the extended history and fires the handler (digest/sendEmail are
entirely signal-driven). TS decider read the wrong history attr key
("signal" vs "signalName") — fixed.
- continueAsNew (kind=4): closes the run CONTINUED_AS_NEW + starts a successor
under the same workflowId carrying input/search-attrs/memo. TS continueAsNew
emits the command instead of throwing ContinueAsNewNotSupported.
- startChild (kind=5): detached (ABANDON) child, seq-keyed + idempotent; retires
the 0x006D 501 for the command path. TS startChild/executeChild added.
- Typed search attributes: carried on StartWorkflow/SignalWithStart wire, stored
on the execution, visibility-queryable (already read by fieldValue). TS adds
TypedSearchAttributes/defineSearchAttributeKey/SearchAttributeType.
- workflowIdConflictPolicy USE_EXISTING/TERMINATE_EXISTING/FAIL.
- ActivityFailure wraps ApplicationFailure (with .type) so workflow error
matching (err.cause.type === 'refresh_token') works.
Also fixes a pre-existing wire defect: the server registered
SignalWithStartWorkflow at 0x0069 while every client sends 0x0066 (schema +
Go/TS SDK) — signalWithStart 404'd against the real engine. Swapped with the
HTTP-only GetWorkflowHistory (no ZAP client).
DescribeWorkflow(workflowId, "") now resolves the OPEN run (Temporal's current
execution) so a continueAsNew successor is the one signalled/described.
Tests: engine unit (search attrs, conflict policy, continueAsNew, startChild)
+ TS e2e vs real tasksd (signals-across-continueAsNew, startChild, search attrs
run to COMPLETED). go test -race ./pkg/... green; TS vitest green.
Enables consolidating the standalone tasksd onto an in-process embedded engine
(cloud) by exposing it cluster-wide under mandatory identity gating, and by
fixing two gaps that made identity-gated ZAP durable execution impossible.
Embedded.ServeGated(ctx, port, *auth.Validator): starts a SECOND ZAP listener
that exposes the SAME engine to out-of-process callers with RequireIdentity --
every request must carry an auth_token that validates against IAM, org-scoped to
the token owner. The loopback listener from Embed is unchanged (in-process,
trusted). Server-push is routed to the listener holding the worker's
subscription, so gated and loopback workers both receive delivery. A nil
validator is refused (never expose durable execution ungated).
SDK: Options.Token TokenSource attaches auth_token to every JSON-body RPC
(client lifecycle + worker Subscribe) at the transport; binary respond frames
stay task_token-gated. This is the missing client half -- the frontend already
read req["auth_token"], but no SDK ever set it, so a RequireIdentity frontend
was unreachable (every caller 401'd). StaticToken helper for fixed bearers;
client_credentials callers return a fresh token per call so long-lived workers
never present an expired identity.
Org-scoped completion: the respond handlers ran on rootEn (unscoped) while
org-scoped runs live in an org-prefixed store partition, so an org-scoped
workflow was accepted (RUNNING) but never advanced. Thread orgID through the
dispatcher's pending tasks and apply completion via en.WithOrg(orgID). orgID==""
(loopback/unscoped) makes WithOrg a no-op -- the in-process path is unchanged.
RegisterNamespace over ZAP: accept the SDK's flat wire ({"name",...}); the nested
{"namespaceInfo":{...}} unmarshal never matched what the client sends, so an
org-scoped caller could not register the namespace ExecuteWorkflow requires.
Test: TestE2E_ServeGated_DurableWorkflow_RequiresIdentity drives a real durable
workflow (schedules an activity, returns its result) to COMPLETED over the gated
listener with a JWKS-signed token, and asserts an unauthenticated caller is
refused. Full suite green.
Co-authored-by: Hanzo Dev <[email protected]>
ClaimNextActivity is the NAT-safe pull primitive an outbound worker
(hanzo gpu connect) polls: it atomically claims the oldest SCHEDULED
activity in a namespace/taskQueue for a worker identity, transitioning it
STARTED under a lease (per-(org,ns) claim lock ⇒ no double-claim). Expired
leases are reaped on the claim hot path (self-healing when a worker dies)
and by the background scheduler sweep — requeued (attempt++) or FAILED once
MaximumAttempts is spent. Heartbeat refreshes the lease; unclaimed presence
records (the fleet registry) carry no lease and are never reaped.
Surface: POST /v1/tasks/namespaces/{ns}/activities/claim (200 claimed |
204 empty) and Embedded.ActivitiesForOrg(org, ns) — the thin org-scoped read
the cloud fleet surface uses.
Tests (-race clean): claim transition+lease, taskQueue filter, oldest-first,
reap requeue+attempt-bump, exhausted-attempts fail, heartbeat-keeps-lease,
presence-untouched, and the HTTP 200/204 claim route.
Enables mounting the Tasks server natively inside the unified hanzoai/cloud
binary (HIP-0106) without a JWT re-validation hop or a duplicate sqlite driver.
- auth.WithIdentity injects an already-validated identity (org/user/email)
into the handler context — the in-process twin of RequireIdentity's mint
step, for an embedder that terminates the IAM trust boundary at its gateway
before the request reaches Tasks.
- store opens through github.com/hanzoai/sqlite (the one Hanzo sqlite driver)
instead of blank-importing modernc directly, so a Tasks store embedded in
the cloud binary shares the single "sqlite" driver registration rather than
double-registering it and panicking at init. WAL/pragmas move to explicit
schemaSQL so they apply on both cgo and !cgo backends.
White-box (pkg/tasks/durable_test.go):
- exactly-once activity dispatch keyed by (run,seq): one record/event/dispatch
when the same seq is scheduled twice, plus a concurrent -race variant.
- crash-recovery invariants: a fresh engine over the same store re-dispatches
only the non-terminal activity (the COMPLETED seq is never re-run) and
re-enqueues a workflow task.
Full-stack (pkg/tasks/durable_e2e_test.go, real Embed+client+worker):
- activity retry+backoff: fails twice then succeeds; asserts 3 attempts,
COMPLETED, and growing backoff gaps.
- retry exhaustion: MaximumAttempts=3 always-fail → workflow FAILED after 3.
- crash-recovery: activity in-flight on worker A, restart onto the same store,
worker B completes it exactly once → workflow COMPLETED (no double execution).
go test -race ./pkg/tasks/... ./pkg/sdk/... green.
Replace the coroutine-push activity path with deterministic, history-backed
replay. One way: the old push-result mechanism is removed.
Server (pkg/tasks):
- durable.go: seq-keyed activity records (wfact/ns/wf/run/seq) — the
exactly-once anchor. applyScheduleActivity is idempotent per (run,seq);
completeWorkflowActivity records the terminal outcome and schedules a new
workflow task so the worker replays and advances. Per-run keyedMutex
serializes advancing a run. Server-side retry+backoff per RetryPolicy
(temporal.RetryPolicy, finite default bound). Recover() rebuilds in-flight
dispatch from the store at boot (called from Embed).
- workflow tasks now carry the run's full event-sourced history.
- dispatch.go: ScheduleActivity/CompleteActivityTask(push) -> DispatchActivity
(seq-bound token) + ResolveActivityToken. workflowPeer/pendingActivity and
OpcodeDeliverActivityResult retired.
- respondWorkflowHandler applies ScheduleActivity{seq} commands under the run
lock; respondActivity* resolve the token and advance the run.
Worker (pkg/sdk/worker):
- workerEnv is now a replay decider: ExecuteActivity resolves from history by
seq (recorded result -> no re-dispatch; else emit ScheduleActivity{seq} and
end the episode via a blockedFuture). Stateless between episodes.
- removed OnActivityResult/pendingActivities push machinery.
Client (pkg/sdk/client): drop OnActivityResult + ScheduleActivity from
WorkerTransport (dead in the replay model).
Tests: e2e (exactly-once, real workers) green; new worker replay determinism
tests (unresolved->schedule+block, resolved->recorded result, 2-activity
determinism). Full 'go test -race ./pkg/tasks/... ./pkg/sdk/...' green.
Architecture
One SQLite shard per (org, namespace) under <DataDir>/<org>/<ns>.db
with WAL mode, single writer, 5s busy timeout. Replaces the prior
shared memdb/zapdb store. Shard manager opens/caches/idle-evicts;
prefix scans, namespace isolation, and cross-restart durability all
proven by tests.
Replication (post-quantum, native)
New pkg/tasks/replication/ exposes a Replicator interface with two
drivers:
local — single-node passthrough (default for embedded/dev)
quasar — wraps github.com/luxfi/consensus NewPQ (BLS + Corona
dual-threshold). On every WAL-affecting op the shard
builds a Frame, calls Propose, awaits Decision, applies
on Accept. Followers receive accepted frames via the
same Subscribe path so every node converges.
Sticky task-queue routing
New pkg/tasks/routing/ — rendezvous-style smallest-hash leader
election keyed by SHA-256(shard|taskQueue|seed). OnChange fan-out
re-hashes on cluster membership change. Dispatcher consults Router
before accepting subscriptions and forwards non-leader requests.
Namespace migration
New pkg/tasks/migration/ — lock → checkpoint → copy → unlock with
a job lifecycle (pending|locking|copying|replaying|released|failed).
CLI: `tasksd migrate --org <id> --namespace <name> --to <node>`.
HTTP: POST /v1/tasks/namespaces/{ns}/migrate.
New env / flags
--data / TASKSD_DATA_DIR shard root (./tasks-data)
--replicator / TASKSD_REPLICATOR local | quasar (default local)
--node-id / TASKSD_NODE_ID stable id (required for quasar)
--validators / TASKSD_VALIDATORS CSV host:port
--validator-key / TASKSD_VALIDATOR_KEY (reserved for PQ key path)
TASKSD_STORE sqlite | memory (default sqlite)
New routes
GET /v1/tasks/cluster
GET /v1/tasks/cluster/health
POST /v1/tasks/namespaces/{ns}/migrate
UI bundle (synced)
Cluster page with status / validators / shards / leader-map.
NamespaceDetail "Migrate to node" button gated by
canWriteNamespace + cluster.enabled. MigrationDialog enforces
typed-name confirmation. Sidebar Cluster nav hidden when running
single-node.
Tests
+33 backend tests (store: 11, replication: 10, routing: 5,
migration: 3, cluster routes: 2, store factory: 2). All packages
PASS with -race; go vet clean.
+13 UI tests (effectiveHealth: 6, canSubmitMigration: 4,
useCluster: 3). 169 total UI tests, 1 pre-existing skip.
Migration v3.8.x → v3.9.0
Fresh-data startup. zapdb backend dropped. Operators drain old
cluster, snapshot for archival, deploy v3.9.0 with new --data
directory. To enable HA, scale to 3+ replicas with --replicator
quasar and matching --node-id / --validators.
Security (red-team pass)
- ZAP wire handlers were org-blind: every opcode bypassed WithOrg
and used the unscoped engine. Patched: zapHandlers takes a
JWTValidator; per-request engine is scoped to claims.Owner.
TASKSD_REQUIRE_IDENTITY enforces validation in production.
- /v1/tasks/mcp tools/call ran on the unscoped engine — patched
to use auth.OrgID(r.Context()).
- /v1/tasks/events SSE fan-out was global; events now stamp OrgID
via engine.emit(ev) and the SSE handler filters per-request org.
- AddSearchAttribute rejected reserved/system names (WorkflowId,
ExecutionStatus, TaskQueue, *StartTime, Temporal*).
- Path-traversal guards on every {ns}/{id} URL slot (validIdent +
validPathSegment).
- dispatcher.queries map is now bounded (4096 entries, 5m TTL,
drop-oldest on overflow) — closes a DoS via silent worker.
Durability
- newStoreFromEnv selects memdb (default) or zapdb via
TASKSD_STORE=zapdb. Survives restart; covered by
TestStore_ZapdbPersistsAcrossOpen.
Inverse-of-options
- RevokeIdentity, DeleteNexusEndpoint, RemoveSearchAttribute,
DeprecateNamespace, DeleteDeploymentVersion, UnsetCurrent.
- Schedule full update + trigger + matching-times.
13 new HTTP routes
GET /v1/tasks/settings
DELETE /v1/tasks/namespaces/{ns}
POST /v1/tasks/namespaces/{ns}/schedules/{id} (full update)
POST /v1/tasks/namespaces/{ns}/schedules/{id}/trigger
GET /v1/tasks/namespaces/{ns}/schedules/{id}/matching-times
POST /v1/tasks/namespaces/{ns}/deployments/{name}/set-current
DELETE /v1/tasks/namespaces/{ns}/deployments/{name}/versions/{buildId}
GET /v1/tasks/namespaces/{ns}/archival
GET /v1/tasks/namespaces/{ns}/workflows/{id}/executions
GET /v1/tasks/namespaces/{ns}/task-queues/{q}/partitions
GET /v1/tasks/namespaces/{ns}/workers/{identity}
DELETE /v1/tasks/namespaces/{ns}/nexus/{id}
DELETE /v1/tasks/namespaces/{ns}/identities/{email}
DELETE /v1/tasks/namespaces/{ns}/search-attributes/{name}
Tests
go test ./pkg/tasks/... -race -count=1 → PASS, +22 new tests
(cross-org keyspace isolation, query non-widening, reserved attrs,
pendingQuery bounds, TTL eviction, Identifier validation, persistence
cross-restart). go vet clean.
UI bundle (synced)
Canonical Hanzo H mark in HanzoMark.tsx replaces stylized H rectangle.
useSettings() hook gates write actions when namespaceWriteDisabled,
archival/heartbeats/advanced-visibility flags. New inverse buttons
(worker stop, identity revoke, search-attr delete, deployment unset,
batch terminate confirm).
UI's cross-namespace Nexus list page (no ns selector) hit /v1/tasks/nexus
and got 404. Backend now serves a top-level GET that scans the nx/ key
prefix across namespaces. Read-only — namespace-scoped routes remain the
only mutation surface.
Bundles UI fix for EventDetail (handles flat events response + string
eventId param).
Closes the remaining 501 surfaces and routes worker-bound RPCs through
the existing ZAP dispatcher (no polling).
Cancel handshake
Running → CANCELING via OpcodeDeliverCancelRequest (0x00B3) push to
the workflow's task-queue subscribers. Worker acks via existing
RespondWorkflowTaskCompleted with a Canceled command (Kind=3) →
CANCELED. 60s sweep auto-acks if no subscriber.
Query push
Allocates a query token, picks a subscriber via the existing
round-robin dispatcher, sends OpcodeDeliverQuery (0x00B4),
blocks on a per-token channel with ctx timeout.
Worker replies via OpcodeRespondQuery (0x00C4).
HTTP returns 503 when no workers are subscribed, 504 on timeout.
Visibility grammar
Replaces the AND-only parser with full grammar:
OR / NOT / parens / IN / NOT IN / BETWEEN. Existing AND-only
callers round-trip unchanged.
Admin endpoints (no longer 501)
POST /v1/tasks/namespaces/{ns}/search-attributes (+ GET list)
POST /v1/tasks/namespaces/{ns}/metadata
POST /v1/tasks/namespaces/{ns}/batches/{id}/terminate
POST /v1/tasks/namespaces/{ns}/workflows/{id}/metadata
UI bundle
Embedded SPA refreshed to the React 19 + hanzogui port with
route-level code-splitting (main entry 67KB / 14KB gz).
Tests
go test ./pkg/tasks/... -race -count=1 → 36 tests, 0 failures.
Updates LLM.md and the HTTPHandler doc-comment to reflect that tasksd
self-validates IAM JWTs (no gateway dependency required). Records the
pre-v3.5.0 history so future readers know why the middleware comment
used to mention hanzoai/gateway.
tasksd is now self-sufficient multi-tenant: no gateway dependency
required. Every inbound request has client-supplied X-Org-Id /
X-User-Id / X-User-Email stripped unconditionally; if Authorization:
Bearer <jwt> is present, the token is validated against the configured
JWKS (TASKSD_JWKS_URL), checked against issuer/audience, and identity
headers are minted from validated claims.
Closes the spoofable-identity hole on tasks-api.hanzo.ai: any request
with a self-set X-Org-Id previously passed through and got that org's
scoped engine.
Env:
TASKSD_JWKS_URL e.g. https://hanzo.id/.well-known/jwks
TASKSD_JWT_ISSUER e.g. https://hanzo.id
TASKSD_JWT_AUDIENCE optional
TASKSD_REQUIRE_IDENTITY=true rejects unauthenticated requests
zeekayandHanzo Dev