go.mod asks for go 1.27.1 and the builder stage was golang:1.26.8-alpine,
whose GOTOOLCHAIN=local fails that build; the publish gate refuses it
before it starts. The stage now uses golang:1.27.1-alpine and sets
GOTOOLCHAIN=auto, so a later go.mod bump fetches its Go instead of
failing.
Co-authored-by: Hanzo Dev <[email protected]>
WorkflowExecutionRetentionTtl (default 720h) was stored and read nowhere,
so finished runs and every key they own stayed forever.
runPurge, beside the cron sweeper, now runs every 5 s across every tenant.
Each pass reads up to 512 of a namespace's run rows, workflow runs then
standalone activities, from a cursor at purge/<ns>, and removes each
terminal run whose close time plus the retention has passed: wf, wfh,
wfact, wfopen and the idem and sctrig entries naming it, or act, ahist and
the aidem entries naming it. Every delete goes through the replicated
delete path, the row last, so a pass cut short leaves the row to find the
rest by. The row is re-read under the run lock first; an open run is never
touched. A retention that does not parse purges nothing and is reported.
Each pass ends in Shard.Reclaim, and a default-mode shard is rebuilt at
the end of a walk, when it is smallest, so the file shrinks.
On a 1.71 GiB shard of 200k expired runs and 10 open ones, 392 passes
removed them (p99 197 ms a pass), the file went to 132 KiB, and Embed took
5 ms and 16 MB peak RSS.
Co-authored-by: Hanzo Dev <[email protected]>
Recover listed every run in a namespace and decoded each one, finished runs
included, to skip all but the few still open, so a restart cost more with
every run a shard had ever held.
wfopen/<ns>/<workflowId>/<runId> now names each workflow run that is not
terminal. It is put before a run's first row and deleted after its terminal
row, so a process stopped between the two leaves an entry that recovery
drops, never an open run the index misses. Recover lists it and re-drives
what it names.
A shard written before the index is walked once, a page at a time, decoding
only each row's status and times; wfidx/<ns> records that the walk is done.
On a 1.71 GiB shard of 200k finished runs and 10 open ones, Embed went from
10.4 s and 1.25 GB peak RSS to 7.7 s and 85 MB on the boot that builds the
index, and 6.0 s and 21 MB after it. What remains is the envelope
decrypting the whole file at open.
Co-authored-by: Hanzo Dev <[email protected]>
Shard.Scan reads at most limit rows of a prefix after a given key, so a
long walk holds the shard's one connection a page at a time; List is Scan
with no limit.
Shard.Reclaim returns free pages to the filesystem. A shard in incremental
auto-vacuum releases up to 4096 per call. A shard in SQLite's default mode
can only be rebuilt with VACUUM, so it is rebuilt when the caller says it is
done deleting and a quarter of it is free; the rebuild moves it to
incremental mode, so it happens once. incremental_vacuum yields a row per
page released, so it is drained as a query.
Co-authored-by: Hanzo Dev <[email protected]>
The sweep stamps TemporalScheduledStartTime on every run it starts: the nominal
tick (RFC 3339, UTC), not the moment the sweep got to it, so a tick fired late
after downtime and the same tick fired by a second engine carry one value. A
manual trigger is not a tick and carries none. ScheduledStart reads it back, and
View.DescribeWorkflow lets an in-process host read its own run, so a consumer
that makes one thing per tick (a cron Job) can tell a fire from a trigger.
Co-authored-by: Hanzo Dev <[email protected]>
An adapted net/http handler was handed a request with an empty
RequestURI, so anything that routes or logs on the target saw "/" and
answered for a path nobody asked for. The pin carries the fix.
Co-authored-by: Hanzo Dev <[email protected]>
A registration that is not a typed operation is spelled Raw now, with the method
named, so opting out of the typed model reads as what it is.
The zip this pins also states the wire explicitly, which matters more than the
version gap suggests: JSON semantics follow the toolchain, so the older pin
answered with different bodies than the code was written against once the
builder moved.
Co-authored-by: Hanzo Dev <[email protected]>
1.27 is the released toolchain and the tree already compiles clean on it, so
the go directive names it. Nothing here depends on a 1.27 language feature
yet; this is the floor moving, which is what makes generic methods, the wider
type inference and promoted-field struct literals available to the code that
wants them next.
Co-authored-by: Hanzo Dev <[email protected]>
The tasks client and the SDK client dial the Tasks server and are never
started, so they bind no socket. Responses and server-pushed deliveries
arrive on the connection the client opened; with discovery off nothing
dials back.
TestZAPClientsListenNowhere asks the OS which sockets the process
listens on: the server adds one, both clients add none.
Co-authored-by: Hanzo Dev <[email protected]>
One toolchain across the estate. The tree carried 1.26.0, .1, .3, .4 and .5
at once, so which compiler built a binary depended on which repo it came
from.
Co-authored-by: Hanzo Dev <[email protected]>
The forge reads .github/workflows natively and github.com reads nothing else,
so one directory serves both planes instead of two conventions that disagree
about which one is live. The reusable in hanzoai/ci already lives there, so a
caller naming that path now resolves to a file that exists.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci keeps the reusable at .github/workflows/build.yml — the one path
both planes resolve, since github.com reads a reusable only from there and the
forge reads it too. The v1 tag moved with it, so a caller still naming
.hanzo/workflows/build.yml asks for a blob that is not in the tree:
InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object does not exist
The caller itself stays at .hanzo/workflows/cicd.yml on purpose: github.com has
no runner for these labels, so a caller placed there could never be scheduled.
Co-authored-by: Hanzo Dev <[email protected]>
A version written into a manifest is written into every manifest that wants it,
which is how eslint came to exist in a hundred and four forms across this estate
and how a package came to resolve twice at run time. The catalog moves the number
out of the manifest: a manifest names the package, the workspace names the
version.
Four are held below their latest because the newer release cannot be used, and
the reason is recorded rather than the version alone: @noble/hashes 2 renamed
every subpath the web3 stack imports, @splinetool/runtime 2 depends on a package
that was never published, @dynamic-labs 5.4.1 the same, and react-resizable-panels
3 points its types condition at the bundle.
The social card asked for Inter first. That is not a face this estate
licenses, ships or ever drew, so a renderer with Inter installed painted
the wrong face and one without it silently fell through to system-ui.
Zen leads now; the same tail follows it.
A composer had one way to mount the engine: take HTTPHandler and adapt it,
which puts sixty-odd operations behind one opaque wildcard the composing
router cannot see into. Surface registers them as real routes, so
app.Use(engine.Surface()) puts every address in the composed router and in
every projection that reads it.
Both renderings reach the same functions -- a call in, an answer out --
so the two transports cannot describe one operation differently, and
HTTPHandler is unchanged for callers that have it.
The wire is held by a test rather than by intent.
TestSurfaceAnswersWhatTheMuxAnswers drives the same requests through both
and compares status, body, Content-Type and Location. It replaced a probe
that printed the two answers and asserted nothing, and the first thing it
found was real: http.ServeMux tidies a path with an empty or dot segment
and redirects to the tidied form before matching, so /namespaces//workflows
was a redirect through the mux and a 404 here. The wildcard answers that
redirect now, with the body http.Redirect writes and only for the methods
it writes one for.
tasks generated a DEK per shard, wrapped it under a KEK derived from the
master, and stored the wrapped blob beside the database — so it also needed an
upgrade path to rewrap, a principal vocabulary to derive against, and tests
for all of it. hanzoai/cek derives the key from the master and the namespace
instead. Nothing is stored, so there is nothing to wrap, rewrap, or upgrade.
dek.go, principal.go and upgrade.go are gone with the tests that pinned them:
they proved properties of a scheme that no longer exists, not properties the
store still has to hold.
That also unpins hanzoai/sqlite, which deleted the DEK API in v0.5.0. tasks
now builds against it, which is what proves the migration is complete rather
than merely compiling.
Databases written under the old scheme will not open under the new
derivation. There are no customers and no data to preserve.
Co-authored-by: Hanzo Dev <[email protected]>
This caller already lives under .hanzo/workflows; its `uses:` did not. Reusables
are resolved through services/actions.ResolveUses, which enforces the same
directory allowlist on the REFERENCED path:
"uses:" path %q must be under a configured workflow directory
so `hanzoai/ci/.github/workflows/build.yml@v1` stops resolving the moment
WORKFLOW_DIRS narrows to .hanzo/workflows alone. The failure mode is the one
this file already documents: the forge refuses at InsertRun, before any run row
is written, so there is no failed run to look at — pushes and workflow_dispatch
alike silently do nothing.
Safe to point at .hanzo now because the v1 channel tag moved. hanzoai/ci
66d4f21 publishes build.yml at BOTH paths from every tag, byte-identical apart
from its header, since GitHub resolves only .github/workflows and git.hanzo.ai
only .hanzo/workflows. Verified against the live remote: v1 and v2 each serve
.hanzo/workflows/build.yml as blob 1100f49a672b. Same pipeline, same @v1 — only
the path changed.
Co-authored-by: Hanzo Dev <[email protected]>
PlatformSudo read the HOME org, so every real operator was denied: an operator is
anchored in a brand org where they bill and do ordinary work, and holds the
reserved org as a further membership. The anchor and the authority are different
questions.
This is what made the reserved org unreachable in practice — nobody could cut a
release or reach an admin surface — while the predicate read as though it were
working.
Co-authored-by: Hanzo Dev <[email protected]>
Minting is issuing authority. IAM mints — it signs the token, and that
signature is where the authority is created. tasksd verifies one and
restates it as headers a transport that cannot carry a token can read.
Nothing here creates authority, so naming both operations "mint" put the
issuer's job and this service's under one word, and a word that covers
two operations is a word that stops distinguishing them.
Every place that named the header write "mint" now names it "write",
matching the leaf it already cites: hanzoai/authz names its header list
"once by the party that WRITES their values". The four ctx accessors said
"minted" for a value that is neither issued nor a header — they return
what the validated claims RESOLVED to, which is the word the org scoping
already used for Claims.EffectiveOrg.
"mint" stays where it is the right word: the test helper signing a JWT,
the `owner` claim IAM stamps at sign time, the HMAC task tokens dispatch
issues, and money mint/burn in CONTRACT.md.
LLM.md also cited edge.Inject, which no longer exists. The call is
edge.Apply — the name it took when the four readers collapsed onto one
leaf, precisely because writing headers is not injecting authority.
Co-authored-by: Hanzo Dev <[email protected]>
authz v1.10.26: the `owner` claim carries the org of the APPLICATION a token was
minted through, so reading it as the subject's org made a person's durable state —
engine.As(Org(...)) scopes the store by it — follow whichever app they signed in
through. It is the first entry of the signed membership set now.
A token stating no org gets none, with no fallback, so tasksd refuses it when
identity is required rather than scoping a namespace to an app-selected value.
Also edge.Inject→Apply and edge.Mint→Render: minting is issuing authority and only
IAM does that.
Co-authored-by: Hanzo Dev <[email protected]>
TASKSD_WHITELABEL_ISSUERS adds the brands this tasksd accepts tokens from beyond
the primary, each of which signs its own. Adding one is configuration.
The sharp edge is the empty entry: splitting an UNSET env var yields one empty
string, and an empty trusted issuer would match a token that carries no `iss` at
all. Entries are trimmed and dropped, pinned by
TestIssuerAllowlistDropsEmptyEntries — which builds the verifier through the same
code path production uses, so the guard is tested where it runs rather than in
isolation.
Co-authored-by: Hanzo Dev <[email protected]>
pkg/auth held 190 lines of JWKS cache and JWT verification — a fourth independent
reading of what an IAM token means — and it had two defects the shared reader does
not:
it verified against EVERY key in the published set and took the first that
worked, so a token naming kid A was accepted on a signature from key B. Naming a
key is what `kid` is for.
its issuer comparison was CONDITIONAL on the configured issuer being non-empty,
so an unset TASKSD_JWT_ISSUER did not fail closed — it accepted tokens from any
issuer.
It also stripped only the four names it reads. Every other minted header —
X-User-IsAdmin and X-User-Permissions (the platform and money signals),
X-User-Owner, X-Billing-Account-Id, X-Scope, X-Workspace-Id, and the retired names
— passed through from the client untouched. tasksd terminates identity itself (its
ingress routes straight to the Service), so nothing upstream was deleting them
either. Restoring the four-name strip turns
TestRequireIdentity_StripsTheWholeEstateSet red on seven forged headers.
And it had NO machine predicate at all, so an admin-org client_credentials token
arrived as a person. It now reads the membership set, like the rest of the estate.
What remains is the part that is genuinely tasksd's: which environment variables
the policy comes from. The audience stays optional and that is deliberate rather
than an oversight — IAM sets `aud` to the requesting CLIENT (RFC 8707), and tasksd
is not a client id, so it pins the issuer and the signing keys instead.
Both transports now answer "which org does this act in" with the same function,
Claims.EffectiveOrg(""), rather than the HTTP path reading `owner` and the ZAP path
reading it separately.
Cost, measured: 342 linked packages and an 18.3 MB binary, DOWN from 345 and
18.6 MB — dropping go-jose more than paid for the leaf, and tasks links ZERO zip
packages because the edge names no transport. go-jose is gone from the module
entirely; the tests now mint with golang-jwt, the library IAM signs with.
Rebased onto 12 upstream commits: kept their ServeGated(addr string) signature,
their engine.As(Org(...)) rename, their dependency versions, and the LLM.md fact
about explicit project/user isolation. Their only change to the deleted jwt.go was
a comment naming hanzo.id, which the replacement already says.
Co-authored-by: Hanzo Dev <[email protected]>
An older binary addressed a shard by org alone (<data>/<org>/<ns>.db).
This one reads three levels down, so the first boot after the upgrade finds
nothing, creates empty shards beside the full ones, enumerates zero tenants
and never fires a schedule again — silently, with every byte still on disk
one directory up.
store.Upgrade inserts the two sentinel levels uniformly, including the old
empty-org dir (_ -> _/_/_). A rename within one filesystem: no page
rewritten, no re-encryption. Journals move before their database, so a run
that stops half way never leaves a database without the WAL holding its
last commits. Idempotent — a moved shard leaves nothing at its old path —
so a deploy can run it unconditionally ahead of the daemon.
Two refusals, both loud and both non-destructive:
A master key is set. Pre-upgrade shards are plaintext and moving one does
not encrypt it; the next boot would wrap a fresh DEK around a file it
then failed to decrypt. Move first, encrypt second.
A destination is occupied (ErrOccupied). The only way there is booting
this binary before the move ran, which creates an EMPTY shard where the
full one belongs — and anything started after that boot is in the empty
one. Both files are kept, the other tenants still move, and the operator
picks which is authoritative.
upgrade and migrate stay separate axes: upgrade moves EVERY shard forward
in time (layout), migrate moves ONE shard across space (nodes).
Also: refuse a socket path over sockaddr_un.sun_path by name. The kernel
answers "bind: invalid argument" and names neither the path nor the limit,
and a socket under a deployment's data directory sits close to it.
Co-authored-by: Hanzo Dev <[email protected]>
main's cron regression tests manufactured a failing action by naming an
unregistered namespace. This branch creates a namespace on first use, so
registration stopped being evidence of anything: a cron with a typo in its
namespace now starts a run into a namespace nobody serves, and says nothing.
Report the fault instead of the proxy. A fire is a failure when it does not
reach a worker — either the start errored, or the start succeeded and no
worker is subscribed to (namespace, taskQueue). Same line, same throttle,
same recovery, one reporter (engine.reportFire). It also catches what the
old guard never could: a schedule whose worker died, and a task queue
renamed out from under an entry.
Reachability is asked AFTER the start, because enqueueing prunes
subscriptions whose peer the transport can no longer reach — a worker that
died without unsubscribing is gone from the registry by the time we look.
Anchoring follows the start, not the report: an errored fire stays due and
retries, a started-but-unreached fire re-anchors, because the run exists and
re-firing every tick would spawn a run every 5s.
Co-authored-by: Hanzo Dev <[email protected]>
The sibling fix made the SCHEDULER report. It does not cover the incident.
StartWorkflow SUCCEEDED all 4489 times -- cron-reconcile's actionCount is the
proof -- so the sweeper was healthy and silent by correctness. The fault was
one layer down: cloud's JobWorkflow fires RunJobActivity, which re-reads the
entry's ConfigMap at fire time, and the cloud ServiceAccount had no RBAC to
read ConfigMaps. Every fire's ACTIVITY failed, for 11 days, and all that layer
did about it was e.emit(Event{Kind:"activity.failed"}) into the SSE broker:
ephemeral, no subscriber attached, nothing written anywhere. Six nightly
backups produced nothing, every dashboard read green, and it was found only by
hand-reading the engine's SQLite.
Guarantee provided: DURABLE evidence, plus throttled log lines. Both, not
either. A "fail/<ns>/<fingerprint>" row per failing identity is upserted on
every failure and deleted on the first success. It needs no subscriber and
outlives the process, so ConsecutiveFailures can say 4489 where an in-memory
counter says 1. View.FailureStreaks(ns) hands the rows back worst-first, so
nobody opens SQLite again; the keyspace is bounded by how much is broken now,
not by how often it failed.
Identity is the recurring SHAPE of the work -- (workflowType, activityType,
taskQueue, scheduleId) -- never the run. Every cron fire mints a fresh runId,
so a run-keyed counter would have read "attempt 1 of 10" forty thousand times
and never once said "this has been dead for 11 days". Schedule-started runs now
carry a ScheduleId search attribute (sweeper + TriggerSchedule) so a report can
name the dead cron entry instead of saying "some JobWorkflow is broken"; it
makes ScheduleId a queryable execution filter at the same time.
Three levels, because the distinction is the whole point. WARN "activity
failed" for a failure that will be retried -- normal operation, must not page
anyone. ERROR "activity failing persistently" once the streak has run
unbroken for 15 minutes -- the alertable condition that was invisible.
INFO "activity recovered" when it ends. Same three for a workflow failing with
no activity to blame, which is the hole an activity-only report would leave (a
decider that faults on episode 0 never schedules anything). Persistence is
measured by AGE, not count: a nightly backup reaches a count of 2 in two days
yet is unambiguously broken, while one run burning its 10 default attempts
(~5.5 min of backoff) must not trip it. The persistent transition always
reports whatever the throttle says -- waiting for failure #60 would be the
original bug wearing a different hat.
Throttle reuses the sibling's idiom rather than inventing a second one:
scheduleFailTracker generalises to failTracker (cadence fixed at construction),
sweepFailThrottle to failReportThrottle, and shouldReport is now the single
rule for the whole package. One tracker instance serves schedules and
activities. Two counters exist on purpose and do different jobs: the in-memory
tracker is the throttle authority (a store that is itself failing cannot starve
it into reporting every occurrence, and a restart makes a still-broken activity
re-alert once after a deploy), while the durable row is the evidence and always
supplies the count on the line. Measured: 180 failures produce 4 lines; stub
shouldReport to always-report and it is 180.
Retry and isolation semantics are UNCHANGED -- which branch is taken, when, and
how many times is decided exactly as before, and every emit() stays.
TestFailureReportingPreservesRetrySemantics pins the attempt count, terminal
status, ACTIVITY_TASK_FAILED event count and dispatch count.
Ten tests, all failing without the reporting calls and all on assertions
("rows: []", "records: []" -- the exact silence), covering: durability with no
subscriber, survival across a simulated restart with the count continuing,
naming the schedule end-to-end from a real sweep, the throttle, the
WARN-vs-ERROR distinction, recovery, standalone activities, workflow-only
failures, retry semantics, and per-org isolation of streaks.
Co-authored-by: Hanzo Dev <[email protected]>
The sweeper had two silent paths. runScheduler discarded the sweep error
outright (`_ = e.sweepSchedules()`), so a sweep that failed wholesale --
cron dead for every org at once -- looked exactly like a quiet night. And
inside sweepSchedules a schedule whose StartWorkflow failed hit a bare
`continue`: correct isolation, zero output.
That silence cost 11 days in production. cloud's clients/cron schedules
fire a JobWorkflow whose RunJobActivity re-reads the entry's ConfigMap at
fire time. The cloud ServiceAccount lacked RBAC to read ConfigMaps, so the
activity failed on every single fire. The engine kept firing perfectly
(cron-reconcile reached actionCount 4489) and emitted NOTHING, so every
dashboard read healthy while all six nightly backups produced nothing for
11 days. It was found only by reading the engine's own durable records.
Isolation is unchanged -- one broken schedule still must not stop the
sweep. What changes is that it now says so, through the logger the package
already uses (slog, from EmbedConfig.Logger; slog.Default() when unset).
Everything needed to act goes on the line: org, namespace, scheduleId,
workflowType, taskQueue, consecutiveFailures, error.
Throttled, because a failed fire is deliberately not re-anchored: the entry
stays due and retries on every 5s tick, so unthrottled this is ~17k
identical lines per day per broken schedule, which buries the signal as
thoroughly as silence did. First failure alerts, then one line per 60
sweeps (~5 min, so any 15-minute window still holds 3+ lines). The counter
clears on success -- recovery is logged, and a relapse alerts immediately
-- and on DeleteSchedule. Deliberately NOT "log on error-text change":
StartWorkflow errors can embed the freshly minted random runId, which would
flap every sweep and restore the very flood the throttle prevents.
Tests fail without the fix (all five report `records: []`, the exact
silence) and also catch a naive unthrottled implementation (60 lines vs 2).
Co-authored-by: Hanzo Dev <[email protected]>
A dispatch token held its tenant as the canonical string, so delivering
its result had to parse one back and decide what to do when it did not
parse. Falling back to the root tenant there would have written one
tenant's result into another's shard.
The dispatcher runs in-process with the engine, so it holds the
Principal itself. There is nothing to encode, nothing to parse, and no
fallback to get wrong. The encoding stays where it is actually needed:
on a replication frame and a migration job, which leave the process.
Co-authored-by: Hanzo Dev <[email protected]>
The engine took a PORT and the SDK an Options.HostPort, so neither end
could use a unix socket for traffic that never leaves the host. The
field name was not the only obstacle: luxfi/zap bound net.Listen("tcp",
":port") and reached a peer with net.DialTimeout("tcp", addr), so a path
was refused at the dialer with "missing port in address" and could not
be bound at all. That is fixed upstream (luxfi/zap Address + Network);
this pins the commit until it is tagged.
EmbedConfig.Address, Embedded.Address, ServeGated(addr) and
client.Options.Address each take ONE address, because a port is a detail
of one transport rather than a thing every listener has. A path binds or
dials a unix socket; anything else is host:port.
TestDialRequiresHostPort became TestDialRequiresAddress: Dial needs an
address, and asserts a socket path is refused for being absent rather
than for being portless. TestE2E_UnixSocket runs a real workflow over a
socket end to end, and TestE2E_UnixSocketRebind proves the socket is
unlinked on stop so a restart binds cleanly.
Co-authored-by: Hanzo Dev <[email protected]>
The store addressed a shard by (org, namespace) and only the root org's
`default` namespace existed, so anything else had to be declared before
it could be used and there was no way to isolate a project or a user.
Shards also had no way to be keyed, so tasks was the one Hanzo store
writing tenant state in the clear.
Both are the same missing value, so both are one change.
Principal is the tenant that owns a shard: an org, optionally narrowed
to a project and a user. It serves the two places tenancy is decided, so
they cannot disagree — the shard's directory
(<data>/<org>/<project>/<user>/<ns>.db, each unset leg written "_") and
the key its DEK is wrapped under. The fixed-depth encoding is injective,
so a project and a user of the same name are different tenants, and the
KEK chain master -> org -> project -> user gives each of them a key
contained in the one it narrows.
Namespaces come into existence on first use (engine.namespace), the way
their shards already did. ValidName is the one rule every name that
becomes a path segment passes through, so a namespace can no longer
escape or alias a directory.
EmbedConfig.MasterKey opens every shard encrypted: each file carries a
DEK of its own, wrapped for its principal and bound to a random file id
in a sidecar beside it, so rotating the master rewraps sidecars and
never rewrites ciphertext, and moving the data directory is harmless.
A keyed shard is ciphertext on every build — hanzoai/sqlite v0.4.0 uses
the live libsqlcipher codec when one is linked and its pure-Go SQLCipher
envelope when one is not — so there is nothing here to branch on. On the
envelope a shard is sealed at checkpoint/close rather than per commit,
which is what Shard.Checkpoint and the manager's sweep bound.
replication.Frame and migration.Job carry the canonical principal string
instead of an org, so a frame lands in the directory its principal names.
Co-authored-by: Hanzo Dev <[email protected]>
go.mod declares `go 1.26.5` while the builder pinned golang:1.26.4. The golang
images set GOTOOLCHAIN=local, so the base can never fetch a newer toolchain to
satisfy the directive and the build dies immediately:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
A release build that fails in under a second did not fail to compile — it
failed to acquire a toolchain.
Found by sweeping every hanzoai repo with a go.mod against its Dockerfile base:
25 declare go >= 1.26.5, and 16 pin an explicit 1.26.4 that cannot satisfy it.
Repos on a floating `1.26` / `1.26-alpine` are already fine — both now resolve
to 1.26.5 (verified by digest: golang:1.26-alpine and golang:1.26.5-alpine are
the same sha256:0178a641).
Patch bump only, image variant preserved, and the target tag verified to exist
before the change.
Co-authored-by: Hanzo Dev <[email protected]>
Superseded: the Hanzo GitHub App pushes a webhook, so the forge tracks GitHub
without a per-repo workflow. This file called git.hanzo.ai/api/v1/.../mirror-sync
— a Gitea API for a system we no longer drive — and would sit inert in every repo.
One mechanism, in one place, instead of ~350 copies of a cron.
git.hanzo.ai is canonical and reads .hanzo/workflows, so cicd.yml moves there
by changing directory and nothing else — same GitHub Actions syntax, same
runs-on, byte-identical file. .github/workflows keeps exactly one workflow,
sync.yml, which carries pushed refs to the forge.
hanzo.yml still holds all the config, so the build and test lanes are
unchanged; it just points at the new path. LLM.md documents the chain and
drops a stale reference to build-and-publish.yml, a workflow this repo no
longer has.
Co-authored-by: Hanzo Dev <[email protected]>
The bare /.well-known/jwks form is only answered by the hanzo.id-worker
shim, which is being retired; IAM's discovery document advertises
/v1/iam/.well-known/jwks on both the public host and in-cluster.
A host subsystem (cloud functions fleet executor) enqueues standalone
activities on an org queue and polls them without an HTTP loopback hop,
mirroring the schedule passthroughs already on View.
Read is bounded now (cloud fleet queue recency-sorts + caps terminal + walks all
pages via ActivitiesPageForOrg v1.51.3), but storage isn't: terminal standalone
activities have no retention sweeper. Queued as v1.51.4 fast-follow. Surfaced in the
hanzoai/cloud per-GPU-queue review; issues are disabled on this repo so tracked here.
Brings CancelActivityForOrg + ActivitiesPageForOrg onto main and makes the v1.51.2
and v1.51.3 release tags reachable from main's history (they were cut off v1.51.1 as
a detached patch line so cloud's pinned tree stayed valid). Additive methods, no
behavior change to the engine.
ActivitiesForOrg returns only the first hash-ordered 100 rows; a busy org past ~100
lifetime activities then sees a random subset — live jobs hidden, online workers
dropped. ActivitiesPageForOrg exposes the cursor ListActivities already carries so an
in-process host can walk a namespace to completion. Additive; org-scoped tenancy is
the caller's to enforce. (v1.51.3 = v1.51.1 + CancelActivityForOrg + this.)
Mirrors ActivitiesForOrg's org-scoped read with a matching mutator so a host
binary (cloud's clients/visor fleet) can cancel a queued/running gpu-jobs
activity it just listed, without a second HTTP hop into its own tasks surface.
Tenancy is the caller's to enforce (validated X-Org-Id). Enables the cloud
POST /v1/fleet/jobs/:id/cancel endpoint.
The client now has one durable path (ZAP) and a local goroutine-timer
fallback for dev. Removes the tasksURL/httpClient submit path and the
duplicated ZAP opcode/field-offset constants; New(zapAddr, handler) loses
its tasksURL argument. embed.go decodes a schedule's interval spec from the
wire alongside cron. Tests boot an embedded Tasks server on a free ZAP port
and assert Now()/Add() persist workflows and schedule specs over ZAP.
go build + go test ./pkg/tasks/ green.
The quasar-driver package move left FrameKey() defined as a method on the
imported replication.Frame → 'cannot define new methods on non-local type'
broke the build. Make it a free function (same shape as frameID); no callers
change (it had none).
hanzoai/base transitively pulled github.com/luxfi/consensus solely through
pkg/tasks → replication (quasar.go, the only consensus importer). That made
base — and every base consumer — vulnerable to the recurring luxfi/consensus
force-moved-tag / vanished-geth supply-chain rot, for a driver base never uses.
Make Quasar replication optional by isolating it:
- Add Kind() + a unified Stats(accepted,rejected,timeouts) to the Replicator
interface so embed.go introspects drivers through the interface, never a
concrete type.
- Move quasar.go (+ its test) into a new pkg/tasks/replication/quasar package
that imports the interface package; the consensus dependency now lives ONLY
there. Only cmd/tasksd imports it.
- pkg/tasks → replication now has ZERO consensus in its dep graph
(`go list -deps ./pkg/tasks | grep consensus` = 0), so base sheds it via Go
1.17+ module-graph pruning.
Also unbreak tasks' own consensus: v1.22.85 → v1.36.3 (geth v1.16.73 was a
vanished revision; v1.36.3 floors the valid unified luxfi stack). Non-quasar
packages build + test green CGO-off; the quasar package builds in CI (its
consensus→secp256k1 CGO can't compile in this sandbox — environmental).
store.bootNS was a write-only set (never read) guarded by a value-typed mu.
withOrg() shares the map by reference but COPIES the mutex, so N org-scoped
store views each locked a DIFFERENT mutex while writing the ONE shared map —
zero mutual exclusion. Under concurrent cross-org traffic the runtime fataled
with 'concurrent map writes' (store.go:100, via shardFor on every put/get),
crashing the whole embedded tasksd — i.e. the entire cloud process / api.hanzo.ai.
Fix is a decomplection: the set was dead state, so delete it (and its now-unused
mu + sync import + the per-op lock dance in shardFor). No map, no race. Net -11
lines in store.go. Adds TestStore_ConcurrentCrossOrgNoRace which reproduces the
exact crash on the old code (fatal → FAIL) and passes on the fix (verified under
-race).
Co-authored-by: Hanzo Dev <[email protected]>
The IAM identity model is org/project/user; tasks carried only org+user.
IAMClaims gains `project`; RequireIdentity mints X-Project-Id (strip-list
⊇ mint-list unchanged — the header is stripped on every inbound request
and set only from a validated JWT); WithIdentity gains the project param
(in-process embedders pass their validated project through); ProjectID(ctx)
accessor. Convention: a project maps onto a tasks NAMESPACE inside the
org's shard.
Co-authored-by: hanzo-dev <[email protected]>
The canonical CI image build failed at go mod download: the alpine build
stage has no git and no auth for the private module. Same dual-mount
(gh_token / GIT_AUTH_TOKEN) pattern as hanzoai/iam's Dockerfile; public
modules stay on proxy+sumdb (retag-proof).
Co-authored-by: hanzo-dev <[email protected]>
* fix(engine): make the cron sweeper actually fire on the sharded store + org-scoped View
The 5s sweeper called store.list("sc/") — a cross-namespace, cross-org
scan the per-(org,ns) shard router refuses ("missing namespace in
prefix"). runScheduler discarded the error, so durable schedules NEVER
fired on the sharded store: every CreateSchedule (automations cron,
base durable cron, tasksd schedules) registered and then sat idle.
- store.listAllOrgs + store.Manager.ListOrgs: enumerate every org's
shards for root sweeps.
- sweepSchedules: two-phase (collect due → fire) so starting a workflow
never nests a query inside the shard List iteration (single-conn
SQLite deadlock); fires AT MOST ONE action per schedule per sweep and
re-anchors at now — recovering after downtime runs the schedule, it
does not storm the missed backlog; per-schedule error isolation.
- nextScheduleFire honors Spec.Interval (what base's durable client
writes), not only CronString.
- Embedded.View(org): exported in-process, org-scoped control-plane
handle (RegisterNamespace/CRUD/Trigger schedules) for host binaries
embedding the engine — cloud's platform cron mounts on this.
Tests: sweep fires (root + org shard, no cross-shard leak), backlog
clamped to 1, interval specs fire, broken entry doesn't poison the
sweep, View CRUD e2e. go test -race ./pkg/tasks/... green.
* ci: replace legacy Temporal workflow suite with canonical hanzo.yml + cicd.yml
19 upstream-era workflows (sharded run-tests matrices, promote-* image
lanes, flaky-test reporting, version-info triggers) targeted the pre-rip
fork layout and have been red on every run since the ZAP-native rewrite
(yq-less runners, deleted directories). One way: root hanzo.yml declares
the image + the -race test gate; cicd.yml imports hanzoai/ci build.yml on
the arc pool.
* ci: scope test gate to the packages that exist post-rip
* feat(embed): View.ListWorkflows — hosts observe their schedules' runs
---------
Co-authored-by: hanzo-dev <[email protected]>
Complete the documented 2026-04-26 native-ZAP migration whose deletion
was left incomplete. The engine (cmd/tasksd), SDK (pkg/sdk) and
in-process server (pkg/tasks) are already 100% ZAP-native with zero
real go.temporal.io / google.golang.org/grpc / protobuf imports; the
only residual gRPC+Temporal footprint was dead porting scaffolding:
- rm proto/ — 55 vendored Temporal .proto (4 gRPC service defs +
message shapes), zero generated .pb.go, zero Go references.
- rm cmd/tools/ — 4 Temporal-porting tools (genrpcwrappers,
genroutingkeyextractor, getproto, check-dependencies); the only
files importing google.golang.org/grpc + go.temporal.io/{api,server}.
- go mod tidy — drops go.temporal.io/api, go.temporal.io/server,
google.golang.org/grpc, grpc-gateway entirely (zero grpc/protobuf
direct or transitive in go.mod).
- CI: remove orphaned proto jobs (linters.yml lint-protos/lint-api,
check-release-dependencies.yml) that ran deleted tools / missing
make targets.
- LLM.md: scrub stale gRPC transport claims -> ZAP :9999 / HTTP :7243.
Transport is luxfi/zap on :9999. go build ./... , go vet, go test all green.
Co-authored-by: Hanzo Dev <[email protected]>
luxfi/zap exposes no disconnect hook, so when a worker dies its dispatcher
subscription leaks. Round-robin then keeps picking the dead peer and the
server-push Send is silently dropped (the task is lost, never retried) — a
social orchestrator that restarts would stall on its 'main'/provider queues.
Make delivery self-healing: on a failed Send, prune the dead peer's subscription
and redeliver to the next live subscriber; queue only when none remain.
drainLocked now drains the whole backlog to a fresh subscriber rather than one
task per Subscribe.
Adds TestDispatcher_PrunesDeadPeerOnSendFailure (dead peer pruned, task reaches
the live peer, second enqueue skips the dead peer). Full pkg/tasks suite green.
Motivated by the social→Hanzo Tasks migration: verified live that a durable
workflow runs to COMPLETED over the gated cloud:9999 listener with a real
hanzo-social IAM token; delivery only stalled after leaked subscriptions
accumulated from worker restarts.
Co-authored-by: Hanzo Dev <[email protected]>
The cloud embedded engine links luxfi/zap v1.2.0 via MVS, whose NewBuilder stamps
wire version 2; the TS SDK only accepted version 1 -> 'unsupported version 2'
against cloud's gated :9999. The generic object encoding is byte-identical across
v1/v2 (only the platformvm TxKind schema differs, which this SDK never uses) and
v1.2.0's Parse accepts v1, so: keep emitting v1 (both engines accept it), accept
BOTH 1 and 2 on read. Verified live against cloud v1.786.102.
Also (drop-in compat, from the social migration):
- export TokenProvider from the package root.
- proxyActivities<A>: open the generic so proxyActivities<SomeActivityClass>()
type-checks; ProxyActivityOptions accepts cancellationType (no-op).
- engine.moreCurrentRun: a CONTINUED_AS_NEW run ranks below any other run of the
same workflowId, so the completed tail of a continue-as-new chain resolves over
its closed predecessor on a same-second StartTime (DescribeWorkflow by id).
The gated engine (cloud ServeGated :9999) requires an auth_token on every
JSON-body RPC; the TS SDK had no way to send one. Add a TokenProvider that
rides client lifecycle + worker Subscribe as auth_token (mirrors the Go SDK's
Options.Token), injected at the ZAP transport for JSON bodies only — worker
Respond/Heartbeat field-object frames are HMAC-authenticated and untouched.
Threaded through Connection/TasksClient/Worker options. Proven: unit test
(auth_token rides JSON RPCs, absent without a source) + gated e2e vs a real
RequireIdentity tasksd with a local RS256 JWKS (anonymous -> 401; IAM-bearer
client+worker run a durable workflow to COMPLETED with org-scoped namespace).
The social-orchestrator migration off upstream Temporal onto the embedded
Hanzo Tasks engine needs five workflow primitives its digest/sendEmail/post
workflows depend on. Implemented server-side (event-sourced, replay-safe) plus
the @hanzoai/tasks TS SDK (the worker social runs):
- Signal → running workflow: engine.signalWorkflow now re-dispatches a workflow
task after appending WORKFLOW_EXECUTION_SIGNALED, so the stateless worker
replays the extended history and fires the handler (digest/sendEmail are
entirely signal-driven). TS decider read the wrong history attr key
("signal" vs "signalName") — fixed.
- continueAsNew (kind=4): closes the run CONTINUED_AS_NEW + starts a successor
under the same workflowId carrying input/search-attrs/memo. TS continueAsNew
emits the command instead of throwing ContinueAsNewNotSupported.
- startChild (kind=5): detached (ABANDON) child, seq-keyed + idempotent; retires
the 0x006D 501 for the command path. TS startChild/executeChild added.
- Typed search attributes: carried on StartWorkflow/SignalWithStart wire, stored
on the execution, visibility-queryable (already read by fieldValue). TS adds
TypedSearchAttributes/defineSearchAttributeKey/SearchAttributeType.
- workflowIdConflictPolicy USE_EXISTING/TERMINATE_EXISTING/FAIL.
- ActivityFailure wraps ApplicationFailure (with .type) so workflow error
matching (err.cause.type === 'refresh_token') works.
Also fixes a pre-existing wire defect: the server registered
SignalWithStartWorkflow at 0x0069 while every client sends 0x0066 (schema +
Go/TS SDK) — signalWithStart 404'd against the real engine. Swapped with the
HTTP-only GetWorkflowHistory (no ZAP client).
DescribeWorkflow(workflowId, "") now resolves the OPEN run (Temporal's current
execution) so a continueAsNew successor is the one signalled/described.
Tests: engine unit (search attrs, conflict policy, continueAsNew, startChild)
+ TS e2e vs real tasksd (signals-across-continueAsNew, startChild, search attrs
run to COMPLETED). go test -race ./pkg/... green; TS vitest green.
Enables consolidating the standalone tasksd onto an in-process embedded engine
(cloud) by exposing it cluster-wide under mandatory identity gating, and by
fixing two gaps that made identity-gated ZAP durable execution impossible.
Embedded.ServeGated(ctx, port, *auth.Validator): starts a SECOND ZAP listener
that exposes the SAME engine to out-of-process callers with RequireIdentity --
every request must carry an auth_token that validates against IAM, org-scoped to
the token owner. The loopback listener from Embed is unchanged (in-process,
trusted). Server-push is routed to the listener holding the worker's
subscription, so gated and loopback workers both receive delivery. A nil
validator is refused (never expose durable execution ungated).
SDK: Options.Token TokenSource attaches auth_token to every JSON-body RPC
(client lifecycle + worker Subscribe) at the transport; binary respond frames
stay task_token-gated. This is the missing client half -- the frontend already
read req["auth_token"], but no SDK ever set it, so a RequireIdentity frontend
was unreachable (every caller 401'd). StaticToken helper for fixed bearers;
client_credentials callers return a fresh token per call so long-lived workers
never present an expired identity.
Org-scoped completion: the respond handlers ran on rootEn (unscoped) while
org-scoped runs live in an org-prefixed store partition, so an org-scoped
workflow was accepted (RUNNING) but never advanced. Thread orgID through the
dispatcher's pending tasks and apply completion via en.WithOrg(orgID). orgID==""
(loopback/unscoped) makes WithOrg a no-op -- the in-process path is unchanged.
RegisterNamespace over ZAP: accept the SDK's flat wire ({"name",...}); the nested
{"namespaceInfo":{...}} unmarshal never matched what the client sends, so an
org-scoped caller could not register the namespace ExecuteWorkflow requires.
Test: TestE2E_ServeGated_DurableWorkflow_RequiresIdentity drives a real durable
workflow (schedules an activity, returns its result) to COMPLETED over the gated
listener with a JWKS-signed token, and asserts an unauthenticated caller is
refused. Full suite green.
Co-authored-by: Hanzo Dev <[email protected]>
Native TS client + worker over the Hanzo Tasks ZAP wire, byte-compatible with pkg/sdk (Go) and luxfi/zap v0.2.0. Drop-in for @temporalio/* + nestjs-temporal-core so the social-orchestrator migration is a dependency swap.
Includes: byte-exact ZAP message codec + TCP node (handshake, reqID correlation, Send-push dispatch); full client (start/signal/signalWithStart/query/describe/cancel/terminate/list, namespaces, health); activity worker (subscribe/dispatch/heartbeat/respond); event-sourced workflow replay decider (proxyActivities, durable activity-backed timers, condition, defineSignal/setHandler); temporal failure envelope serde; @temporalio compat (Connection/Client/NativeConnection/Worker/ApplicationFailure).
Tests: 25 vitest specs green (byte-layout golden, decider unit, full durable loop through a faithful ZAP mock) + a live integration suite that passes against a running tasksd. tsc --noEmit clean.
Co-authored-by: Hanzo Dev <[email protected]>
ClaimNextActivity is the NAT-safe pull primitive an outbound worker
(hanzo gpu connect) polls: it atomically claims the oldest SCHEDULED
activity in a namespace/taskQueue for a worker identity, transitioning it
STARTED under a lease (per-(org,ns) claim lock ⇒ no double-claim). Expired
leases are reaped on the claim hot path (self-healing when a worker dies)
and by the background scheduler sweep — requeued (attempt++) or FAILED once
MaximumAttempts is spent. Heartbeat refreshes the lease; unclaimed presence
records (the fleet registry) carry no lease and are never reaped.
Surface: POST /v1/tasks/namespaces/{ns}/activities/claim (200 claimed |
204 empty) and Embedded.ActivitiesForOrg(org, ns) — the thin org-scoped read
the cloud fleet surface uses.
Tests (-race clean): claim transition+lease, taskQueue filter, oldest-first,
reap requeue+attempt-bump, exhausted-attempts fail, heartbeat-keeps-lease,
presence-untouched, and the HTTP 200/204 claim route.
Enables mounting the Tasks server natively inside the unified hanzoai/cloud
binary (HIP-0106) without a JWT re-validation hop or a duplicate sqlite driver.
- auth.WithIdentity injects an already-validated identity (org/user/email)
into the handler context — the in-process twin of RequireIdentity's mint
step, for an embedder that terminates the IAM trust boundary at its gateway
before the request reaches Tasks.
- store opens through github.com/hanzoai/sqlite (the one Hanzo sqlite driver)
instead of blank-importing modernc directly, so a Tasks store embedded in
the cloud binary shares the single "sqlite" driver registration rather than
double-registering it and panicking at init. WAL/pragmas move to explicit
schemaSQL so they apply on both cgo and !cgo backends.
White-box (pkg/tasks/durable_test.go):
- exactly-once activity dispatch keyed by (run,seq): one record/event/dispatch
when the same seq is scheduled twice, plus a concurrent -race variant.
- crash-recovery invariants: a fresh engine over the same store re-dispatches
only the non-terminal activity (the COMPLETED seq is never re-run) and
re-enqueues a workflow task.
Full-stack (pkg/tasks/durable_e2e_test.go, real Embed+client+worker):
- activity retry+backoff: fails twice then succeeds; asserts 3 attempts,
COMPLETED, and growing backoff gaps.
- retry exhaustion: MaximumAttempts=3 always-fail → workflow FAILED after 3.
- crash-recovery: activity in-flight on worker A, restart onto the same store,
worker B completes it exactly once → workflow COMPLETED (no double execution).
go test -race ./pkg/tasks/... ./pkg/sdk/... green.
Replace the coroutine-push activity path with deterministic, history-backed
replay. One way: the old push-result mechanism is removed.
Server (pkg/tasks):
- durable.go: seq-keyed activity records (wfact/ns/wf/run/seq) — the
exactly-once anchor. applyScheduleActivity is idempotent per (run,seq);
completeWorkflowActivity records the terminal outcome and schedules a new
workflow task so the worker replays and advances. Per-run keyedMutex
serializes advancing a run. Server-side retry+backoff per RetryPolicy
(temporal.RetryPolicy, finite default bound). Recover() rebuilds in-flight
dispatch from the store at boot (called from Embed).
- workflow tasks now carry the run's full event-sourced history.
- dispatch.go: ScheduleActivity/CompleteActivityTask(push) -> DispatchActivity
(seq-bound token) + ResolveActivityToken. workflowPeer/pendingActivity and
OpcodeDeliverActivityResult retired.
- respondWorkflowHandler applies ScheduleActivity{seq} commands under the run
lock; respondActivity* resolve the token and advance the run.
Worker (pkg/sdk/worker):
- workerEnv is now a replay decider: ExecuteActivity resolves from history by
seq (recorded result -> no re-dispatch; else emit ScheduleActivity{seq} and
end the episode via a blockedFuture). Stateless between episodes.
- removed OnActivityResult/pendingActivities push machinery.
Client (pkg/sdk/client): drop OnActivityResult + ScheduleActivity from
WorkerTransport (dead in the replay model).
Tests: e2e (exactly-once, real workers) green; new worker replay determinism
tests (unresolved->schedule+block, resolved->recorded result, 2-activity
determinism). Full 'go test -race ./pkg/tasks/... ./pkg/sdk/...' green.
Correct the stale 'handlers return 501' record: the native engine is
Phase-1-complete and green. The real gap is deterministic replay /
durability. Specify the event-sourced model (history as source of truth,
workflow-task decision episodes, seq-keyed exactly-once activity
dispatch, server-side retry+backoff, single-replica crash-recovery,
multi-replica dispatch lease) and the 2a/2b/2c phasing. Flag the
origin/main temporal re-merge regression as a coordination hazard.
Replaces non-canonical scale-set / org-prefixed labels with the
existing labels every arcd host registers with. Matches evo for
amd64 and spark for arm64. No new labels added.
CONTRACT.md is the single durable async substrate spec for every Hanzo +
Liquidity Go service. Covers: when to use tasks vs sync, task definition
shape, worker registration, client API, ZAP transport, per-tenant
namespace isolation, failure modes, observability.
Replaces ad-hoc per-service taskqueue wrappers and the lingering
go.temporal.io/sdk imports in Liquidity ats/bd/ta.
* docs(llm): explicit Upstream attribution line
LICENSE-attested or repo-attested upstream is now called out at the
top of LLM.md alongside the project description, so downstream
audits (universe/docs/PRODUCTS.md) can rely on a single canonical
location for the OSS lineage statement.
* build(tasks): GOEXPERIMENT=jsonv2 in Dockerfile per SCALE_STANDARD
Per SCALE_STANDARD.md §2 — every Go production Dockerfile that emits
JSON to a client builds with GOEXPERIMENT=jsonv2. Verified on
hanzoai/zip's json_bench_test.go:
- Edge POST roundtrip: -12% time, -23% allocs
- Marshal-only: -22% time
- Unmarshal-only: -19% time, -25% allocs
The conn-memory regression test is deferred to a follow-up that
migrates tasksd's net/http listener to zip per §9 step-1; net/http
has a fixed 3-goroutine-per-request overhead that violates §3's 1:1
goroutine-to-conn invariant.
Per SCALE_STANDARD.md §2 — every Go production Dockerfile that emits
JSON to a client builds with GOEXPERIMENT=jsonv2. Verified on
hanzoai/zip's json_bench_test.go:
- Edge POST roundtrip: -12% time, -23% allocs
- Marshal-only: -22% time
- Unmarshal-only: -19% time, -25% allocs
The conn-memory regression test is deferred to a follow-up that
migrates tasksd's net/http listener to zip per §9 step-1; net/http
has a fixed 3-goroutine-per-request overhead that violates §3's 1:1
goroutine-to-conn invariant.
LICENSE-attested or repo-attested upstream is now called out at the
top of LLM.md alongside the project description, so downstream
audits (universe/docs/PRODUCTS.md) can rely on a single canonical
location for the OSS lineage statement.
Architecture
One SQLite shard per (org, namespace) under <DataDir>/<org>/<ns>.db
with WAL mode, single writer, 5s busy timeout. Replaces the prior
shared memdb/zapdb store. Shard manager opens/caches/idle-evicts;
prefix scans, namespace isolation, and cross-restart durability all
proven by tests.
Replication (post-quantum, native)
New pkg/tasks/replication/ exposes a Replicator interface with two
drivers:
local — single-node passthrough (default for embedded/dev)
quasar — wraps github.com/luxfi/consensus NewPQ (BLS + Corona
dual-threshold). On every WAL-affecting op the shard
builds a Frame, calls Propose, awaits Decision, applies
on Accept. Followers receive accepted frames via the
same Subscribe path so every node converges.
Sticky task-queue routing
New pkg/tasks/routing/ — rendezvous-style smallest-hash leader
election keyed by SHA-256(shard|taskQueue|seed). OnChange fan-out
re-hashes on cluster membership change. Dispatcher consults Router
before accepting subscriptions and forwards non-leader requests.
Namespace migration
New pkg/tasks/migration/ — lock → checkpoint → copy → unlock with
a job lifecycle (pending|locking|copying|replaying|released|failed).
CLI: `tasksd migrate --org <id> --namespace <name> --to <node>`.
HTTP: POST /v1/tasks/namespaces/{ns}/migrate.
New env / flags
--data / TASKSD_DATA_DIR shard root (./tasks-data)
--replicator / TASKSD_REPLICATOR local | quasar (default local)
--node-id / TASKSD_NODE_ID stable id (required for quasar)
--validators / TASKSD_VALIDATORS CSV host:port
--validator-key / TASKSD_VALIDATOR_KEY (reserved for PQ key path)
TASKSD_STORE sqlite | memory (default sqlite)
New routes
GET /v1/tasks/cluster
GET /v1/tasks/cluster/health
POST /v1/tasks/namespaces/{ns}/migrate
UI bundle (synced)
Cluster page with status / validators / shards / leader-map.
NamespaceDetail "Migrate to node" button gated by
canWriteNamespace + cluster.enabled. MigrationDialog enforces
typed-name confirmation. Sidebar Cluster nav hidden when running
single-node.
Tests
+33 backend tests (store: 11, replication: 10, routing: 5,
migration: 3, cluster routes: 2, store factory: 2). All packages
PASS with -race; go vet clean.
+13 UI tests (effectiveHealth: 6, canSubmitMigration: 4,
useCluster: 3). 169 total UI tests, 1 pre-existing skip.
Migration v3.8.x → v3.9.0
Fresh-data startup. zapdb backend dropped. Operators drain old
cluster, snapshot for archival, deploy v3.9.0 with new --data
directory. To enable HA, scale to 3+ replicas with --replicator
quasar and matching --node-id / --validators.
Bundles the wire-shape fix (Deployment: name + versions). Front-end was
reading seriesName/buildIds from the v3.7.x type which v3.8.0 renamed.
1 Deployment list page rendered blank with TypeError; now renders
correctly.
zeekayandHanzo Dev
