tasks generated a DEK per shard, wrapped it under a KEK derived from the
master, and stored the wrapped blob beside the database — so it also needed an
upgrade path to rewrap, a principal vocabulary to derive against, and tests
for all of it. hanzoai/cek derives the key from the master and the namespace
instead. Nothing is stored, so there is nothing to wrap, rewrap, or upgrade.
dek.go, principal.go and upgrade.go are gone with the tests that pinned them:
they proved properties of a scheme that no longer exists, not properties the
store still has to hold.
That also unpins hanzoai/sqlite, which deleted the DEK API in v0.5.0. tasks
now builds against it, which is what proves the migration is complete rather
than merely compiling.
Databases written under the old scheme will not open under the new
derivation. There are no customers and no data to preserve.
Co-authored-by: Hanzo Dev <[email protected]>
TASKSD_WHITELABEL_ISSUERS adds the brands this tasksd accepts tokens from beyond
the primary, each of which signs its own. Adding one is configuration.
The sharp edge is the empty entry: splitting an UNSET env var yields one empty
string, and an empty trusted issuer would match a token that carries no `iss` at
all. Entries are trimmed and dropped, pinned by
TestIssuerAllowlistDropsEmptyEntries — which builds the verifier through the same
code path production uses, so the guard is tested where it runs rather than in
isolation.
Co-authored-by: Hanzo Dev <[email protected]>
pkg/auth held 190 lines of JWKS cache and JWT verification — a fourth independent
reading of what an IAM token means — and it had two defects the shared reader does
not:
it verified against EVERY key in the published set and took the first that
worked, so a token naming kid A was accepted on a signature from key B. Naming a
key is what `kid` is for.
its issuer comparison was CONDITIONAL on the configured issuer being non-empty,
so an unset TASKSD_JWT_ISSUER did not fail closed — it accepted tokens from any
issuer.
It also stripped only the four names it reads. Every other minted header —
X-User-IsAdmin and X-User-Permissions (the platform and money signals),
X-User-Owner, X-Billing-Account-Id, X-Scope, X-Workspace-Id, and the retired names
— passed through from the client untouched. tasksd terminates identity itself (its
ingress routes straight to the Service), so nothing upstream was deleting them
either. Restoring the four-name strip turns
TestRequireIdentity_StripsTheWholeEstateSet red on seven forged headers.
And it had NO machine predicate at all, so an admin-org client_credentials token
arrived as a person. It now reads the membership set, like the rest of the estate.
What remains is the part that is genuinely tasksd's: which environment variables
the policy comes from. The audience stays optional and that is deliberate rather
than an oversight — IAM sets `aud` to the requesting CLIENT (RFC 8707), and tasksd
is not a client id, so it pins the issuer and the signing keys instead.
Both transports now answer "which org does this act in" with the same function,
Claims.EffectiveOrg(""), rather than the HTTP path reading `owner` and the ZAP path
reading it separately.
Cost, measured: 342 linked packages and an 18.3 MB binary, DOWN from 345 and
18.6 MB — dropping go-jose more than paid for the leaf, and tasks links ZERO zip
packages because the edge names no transport. go-jose is gone from the module
entirely; the tests now mint with golang-jwt, the library IAM signs with.
Rebased onto 12 upstream commits: kept their ServeGated(addr string) signature,
their engine.As(Org(...)) rename, their dependency versions, and the LLM.md fact
about explicit project/user isolation. Their only change to the deleted jwt.go was
a comment naming hanzo.id, which the replacement already says.
Co-authored-by: Hanzo Dev <[email protected]>
An older binary addressed a shard by org alone (<data>/<org>/<ns>.db).
This one reads three levels down, so the first boot after the upgrade finds
nothing, creates empty shards beside the full ones, enumerates zero tenants
and never fires a schedule again — silently, with every byte still on disk
one directory up.
store.Upgrade inserts the two sentinel levels uniformly, including the old
empty-org dir (_ -> _/_/_). A rename within one filesystem: no page
rewritten, no re-encryption. Journals move before their database, so a run
that stops half way never leaves a database without the WAL holding its
last commits. Idempotent — a moved shard leaves nothing at its old path —
so a deploy can run it unconditionally ahead of the daemon.
Two refusals, both loud and both non-destructive:
A master key is set. Pre-upgrade shards are plaintext and moving one does
not encrypt it; the next boot would wrap a fresh DEK around a file it
then failed to decrypt. Move first, encrypt second.
A destination is occupied (ErrOccupied). The only way there is booting
this binary before the move ran, which creates an EMPTY shard where the
full one belongs — and anything started after that boot is in the empty
one. Both files are kept, the other tenants still move, and the operator
picks which is authoritative.
upgrade and migrate stay separate axes: upgrade moves EVERY shard forward
in time (layout), migrate moves ONE shard across space (nodes).
Also: refuse a socket path over sockaddr_un.sun_path by name. The kernel
answers "bind: invalid argument" and names neither the path nor the limit,
and a socket under a deployment's data directory sits close to it.
Co-authored-by: Hanzo Dev <[email protected]>
The engine took a PORT and the SDK an Options.HostPort, so neither end
could use a unix socket for traffic that never leaves the host. The
field name was not the only obstacle: luxfi/zap bound net.Listen("tcp",
":port") and reached a peer with net.DialTimeout("tcp", addr), so a path
was refused at the dialer with "missing port in address" and could not
be bound at all. That is fixed upstream (luxfi/zap Address + Network);
this pins the commit until it is tagged.
EmbedConfig.Address, Embedded.Address, ServeGated(addr) and
client.Options.Address each take ONE address, because a port is a detail
of one transport rather than a thing every listener has. A path binds or
dials a unix socket; anything else is host:port.
TestDialRequiresHostPort became TestDialRequiresAddress: Dial needs an
address, and asserts a socket path is refused for being absent rather
than for being portless. TestE2E_UnixSocket runs a real workflow over a
socket end to end, and TestE2E_UnixSocketRebind proves the socket is
unlinked on stop so a restart binds cleanly.
Co-authored-by: Hanzo Dev <[email protected]>
The store addressed a shard by (org, namespace) and only the root org's
`default` namespace existed, so anything else had to be declared before
it could be used and there was no way to isolate a project or a user.
Shards also had no way to be keyed, so tasks was the one Hanzo store
writing tenant state in the clear.
Both are the same missing value, so both are one change.
Principal is the tenant that owns a shard: an org, optionally narrowed
to a project and a user. It serves the two places tenancy is decided, so
they cannot disagree — the shard's directory
(<data>/<org>/<project>/<user>/<ns>.db, each unset leg written "_") and
the key its DEK is wrapped under. The fixed-depth encoding is injective,
so a project and a user of the same name are different tenants, and the
KEK chain master -> org -> project -> user gives each of them a key
contained in the one it narrows.
Namespaces come into existence on first use (engine.namespace), the way
their shards already did. ValidName is the one rule every name that
becomes a path segment passes through, so a namespace can no longer
escape or alias a directory.
EmbedConfig.MasterKey opens every shard encrypted: each file carries a
DEK of its own, wrapped for its principal and bound to a random file id
in a sidecar beside it, so rotating the master rewraps sidecars and
never rewrites ciphertext, and moving the data directory is harmless.
A keyed shard is ciphertext on every build — hanzoai/sqlite v0.4.0 uses
the live libsqlcipher codec when one is linked and its pure-Go SQLCipher
envelope when one is not — so there is nothing here to branch on. On the
envelope a shard is sealed at checkpoint/close rather than per commit,
which is what Shard.Checkpoint and the manager's sweep bound.
replication.Frame and migration.Job carry the canonical principal string
instead of an org, so a frame lands in the directory its principal names.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/base transitively pulled github.com/luxfi/consensus solely through
pkg/tasks → replication (quasar.go, the only consensus importer). That made
base — and every base consumer — vulnerable to the recurring luxfi/consensus
force-moved-tag / vanished-geth supply-chain rot, for a driver base never uses.
Make Quasar replication optional by isolating it:
- Add Kind() + a unified Stats(accepted,rejected,timeouts) to the Replicator
interface so embed.go introspects drivers through the interface, never a
concrete type.
- Move quasar.go (+ its test) into a new pkg/tasks/replication/quasar package
that imports the interface package; the consensus dependency now lives ONLY
there. Only cmd/tasksd imports it.
- pkg/tasks → replication now has ZERO consensus in its dep graph
(`go list -deps ./pkg/tasks | grep consensus` = 0), so base sheds it via Go
1.17+ module-graph pruning.
Also unbreak tasks' own consensus: v1.22.85 → v1.36.3 (geth v1.16.73 was a
vanished revision; v1.36.3 floors the valid unified luxfi stack). Non-quasar
packages build + test green CGO-off; the quasar package builds in CI (its
consensus→secp256k1 CGO can't compile in this sandbox — environmental).
Architecture
One SQLite shard per (org, namespace) under <DataDir>/<org>/<ns>.db
with WAL mode, single writer, 5s busy timeout. Replaces the prior
shared memdb/zapdb store. Shard manager opens/caches/idle-evicts;
prefix scans, namespace isolation, and cross-restart durability all
proven by tests.
Replication (post-quantum, native)
New pkg/tasks/replication/ exposes a Replicator interface with two
drivers:
local — single-node passthrough (default for embedded/dev)
quasar — wraps github.com/luxfi/consensus NewPQ (BLS + Corona
dual-threshold). On every WAL-affecting op the shard
builds a Frame, calls Propose, awaits Decision, applies
on Accept. Followers receive accepted frames via the
same Subscribe path so every node converges.
Sticky task-queue routing
New pkg/tasks/routing/ — rendezvous-style smallest-hash leader
election keyed by SHA-256(shard|taskQueue|seed). OnChange fan-out
re-hashes on cluster membership change. Dispatcher consults Router
before accepting subscriptions and forwards non-leader requests.
Namespace migration
New pkg/tasks/migration/ — lock → checkpoint → copy → unlock with
a job lifecycle (pending|locking|copying|replaying|released|failed).
CLI: `tasksd migrate --org <id> --namespace <name> --to <node>`.
HTTP: POST /v1/tasks/namespaces/{ns}/migrate.
New env / flags
--data / TASKSD_DATA_DIR shard root (./tasks-data)
--replicator / TASKSD_REPLICATOR local | quasar (default local)
--node-id / TASKSD_NODE_ID stable id (required for quasar)
--validators / TASKSD_VALIDATORS CSV host:port
--validator-key / TASKSD_VALIDATOR_KEY (reserved for PQ key path)
TASKSD_STORE sqlite | memory (default sqlite)
New routes
GET /v1/tasks/cluster
GET /v1/tasks/cluster/health
POST /v1/tasks/namespaces/{ns}/migrate
UI bundle (synced)
Cluster page with status / validators / shards / leader-map.
NamespaceDetail "Migrate to node" button gated by
canWriteNamespace + cluster.enabled. MigrationDialog enforces
typed-name confirmation. Sidebar Cluster nav hidden when running
single-node.
Tests
+33 backend tests (store: 11, replication: 10, routing: 5,
migration: 3, cluster routes: 2, store factory: 2). All packages
PASS with -race; go vet clean.
+13 UI tests (effectiveHealth: 6, canSubmitMigration: 4,
useCluster: 3). 169 total UI tests, 1 pre-existing skip.
Migration v3.8.x → v3.9.0
Fresh-data startup. zapdb backend dropped. Operators drain old
cluster, snapshot for archival, deploy v3.9.0 with new --data
directory. To enable HA, scale to 3+ replicas with --replicator
quasar and matching --node-id / --validators.
Security (red-team pass)
- ZAP wire handlers were org-blind: every opcode bypassed WithOrg
and used the unscoped engine. Patched: zapHandlers takes a
JWTValidator; per-request engine is scoped to claims.Owner.
TASKSD_REQUIRE_IDENTITY enforces validation in production.
- /v1/tasks/mcp tools/call ran on the unscoped engine — patched
to use auth.OrgID(r.Context()).
- /v1/tasks/events SSE fan-out was global; events now stamp OrgID
via engine.emit(ev) and the SSE handler filters per-request org.
- AddSearchAttribute rejected reserved/system names (WorkflowId,
ExecutionStatus, TaskQueue, *StartTime, Temporal*).
- Path-traversal guards on every {ns}/{id} URL slot (validIdent +
validPathSegment).
- dispatcher.queries map is now bounded (4096 entries, 5m TTL,
drop-oldest on overflow) — closes a DoS via silent worker.
Durability
- newStoreFromEnv selects memdb (default) or zapdb via
TASKSD_STORE=zapdb. Survives restart; covered by
TestStore_ZapdbPersistsAcrossOpen.
Inverse-of-options
- RevokeIdentity, DeleteNexusEndpoint, RemoveSearchAttribute,
DeprecateNamespace, DeleteDeploymentVersion, UnsetCurrent.
- Schedule full update + trigger + matching-times.
13 new HTTP routes
GET /v1/tasks/settings
DELETE /v1/tasks/namespaces/{ns}
POST /v1/tasks/namespaces/{ns}/schedules/{id} (full update)
POST /v1/tasks/namespaces/{ns}/schedules/{id}/trigger
GET /v1/tasks/namespaces/{ns}/schedules/{id}/matching-times
POST /v1/tasks/namespaces/{ns}/deployments/{name}/set-current
DELETE /v1/tasks/namespaces/{ns}/deployments/{name}/versions/{buildId}
GET /v1/tasks/namespaces/{ns}/archival
GET /v1/tasks/namespaces/{ns}/workflows/{id}/executions
GET /v1/tasks/namespaces/{ns}/task-queues/{q}/partitions
GET /v1/tasks/namespaces/{ns}/workers/{identity}
DELETE /v1/tasks/namespaces/{ns}/nexus/{id}
DELETE /v1/tasks/namespaces/{ns}/identities/{email}
DELETE /v1/tasks/namespaces/{ns}/search-attributes/{name}
Tests
go test ./pkg/tasks/... -race -count=1 → PASS, +22 new tests
(cross-org keyspace isolation, query non-widening, reserved attrs,
pendingQuery bounds, TTL eviction, Identifier validation, persistence
cross-restart). go vet clean.
UI bundle (synced)
Canonical Hanzo H mark in HanzoMark.tsx replaces stylized H rectangle.
useSettings() hook gates write actions when namespaceWriteDisabled,
archival/heartbeats/advanced-visibility flags. New inverse buttons
(worker stop, identity revoke, search-attr delete, deployment unset,
batch terminate confirm).
tasksd is now self-sufficient multi-tenant: no gateway dependency
required. Every inbound request has client-supplied X-Org-Id /
X-User-Id / X-User-Email stripped unconditionally; if Authorization:
Bearer <jwt> is present, the token is validated against the configured
JWKS (TASKSD_JWKS_URL), checked against issuer/audience, and identity
headers are minted from validated claims.
Closes the spoofable-identity hole on tasks-api.hanzo.ai: any request
with a self-set X-Org-Id previously passed through and got that org's
scoped engine.
Env:
TASKSD_JWKS_URL e.g. https://hanzo.id/.well-known/jwks
TASKSD_JWT_ISSUER e.g. https://hanzo.id
TASKSD_JWT_AUDIENCE optional
TASKSD_REQUIRE_IDENTITY=true rejects unauthenticated requests
/healthz, /v1/tasks/health, /_/tasks/* stay public — probes and SPA
shell run before any session exists. /v1/tasks/, /v1/tasks/mcp,
/v1/tasks/events go through auth.RequireIdentity, gated by
TASKSD_REQUIRE_IDENTITY (default false).
Two new surfaces on top of the engine, both wire-compatible with the
canonical ZAP transport (their opcodes are reserved for native ZAP
adoption — HTTP today, ZAP tomorrow without breaking clients).
pkg/tasks/mcp.go — Model Context Protocol JSON-RPC 2.0 endpoint at
POST /v1/tasks/mcp. Methods: initialize, tools/list, tools/call, ping.
Tool catalogue:
list_namespaces / describe_namespace
list_workflows / describe_workflow / start_workflow /
signal_workflow / cancel_workflow / terminate_workflow
list_schedules / create_schedule / pause_schedule
Every tool maps onto an engine method with the SDK's snake_case wire
shape. AI agents can now drive Hanzo Tasks end-to-end via stdio MCP
clients (Claude Desktop, agent, Cursor, etc.) by pointing them
at /v1/tasks/mcp.
pkg/tasks/events.go — bounded fan-out broker + GET /v1/tasks/events
SSE stream. Every state-changing engine method publishes an Event:
namespace.registered
workflow.started / workflow.canceled / workflow.terminated /
workflow.signaled
schedule.created (more transitions in v1.41)
Slow subscribers drop the oldest event rather than backing up the
publisher. Hello-frame on connect, ctx.Done() unsubscribes on disconnect.
cmd/tasksd mounts both at fixed paths next to the existing /v1/tasks/*
shim. Binary 7.6 MB, integration tests still 4/4 PASS via ZAP.
Engine (pkg/tasks):
- store.go — KV layer over luxfi/database memdb (snapshot-and-iterate
list to avoid RWMutex upgrade deadlock)
- types.go — wire types: Namespace, WorkflowExecution, Schedule,
BatchOperation, Deployment, NexusEndpoint, Identity
- engine.go — domain ops: register/list/describe ns; start/list/
describe/cancel/terminate/signal workflow; create/list/
describe/delete/pause/unpause schedule; start/list/
describe batch; create/list deployment; create/list
nexus endpoint; grant/list identity; cron sweeper goroutine
- embed.go — every ZAP opcode (0x0060-0x0090) + every HTTP route
dispatches through the same model funcs; no drift
HTTP surface (browser-only JSON shim, ZAP is the canonical wire):
GET /v1/tasks/namespaces
POST /v1/tasks/namespaces
GET /v1/tasks/namespaces/{ns}
GET /v1/tasks/namespaces/{ns}/workflows
POST /v1/tasks/namespaces/{ns}/workflows
GET /v1/tasks/namespaces/{ns}/workflows/{id}
POST /v1/tasks/namespaces/{ns}/workflows/{id}/cancel
POST /v1/tasks/namespaces/{ns}/workflows/{id}/terminate
POST /v1/tasks/namespaces/{ns}/workflows/{id}/signal
GET /v1/tasks/namespaces/{ns}/schedules
POST /v1/tasks/namespaces/{ns}/schedules
GET /v1/tasks/namespaces/{ns}/schedules/{id}
DELETE /v1/tasks/namespaces/{ns}/schedules/{id}
POST /v1/tasks/namespaces/{ns}/schedules/{id}/pause
POST /v1/tasks/namespaces/{ns}/schedules/{id}/unpause
GET /v1/tasks/namespaces/{ns}/batches
POST /v1/tasks/namespaces/{ns}/batches
GET /v1/tasks/namespaces/{ns}/batches/{id}
GET /v1/tasks/namespaces/{ns}/deployments
POST /v1/tasks/namespaces/{ns}/deployments
GET /v1/tasks/namespaces/{ns}/nexus
POST /v1/tasks/namespaces/{ns}/nexus
GET /v1/tasks/namespaces/{ns}/identities
POST /v1/tasks/namespaces/{ns}/identities
UI (full Temporal Cloud parity):
- @hanzo/ui primitives in src/components/ui/{button,card,tabs,badge,
alert,dialog,input,label,skeleton,dropdown-menu}.tsx — same shadcn
pattern @hanzo/ui itself uses; CSS tokens from @hanzo/ui design system
- App.tsx: sidebar layout (Workflows/Schedules/Batches/Deployments/
Namespaces/Nexus/Support) with top bar (UTC clock, theme toggle,
account chip, ns indicator, embedded badge)
- pages/Namespaces.tsx + pages/NamespaceDetail.tsx — list + detail w/
Overview+Identities tabs, Region/Retention/APS cards, Connect
dropdown showing ZAP+HTTP endpoints with copy buttons, Active badge
- pages/Workflows.tsx — list + Start Workflow dialog (real POST)
- pages/WorkflowDetail.tsx — execution details
- pages/Schedules.tsx — list w/ cron / interval describe
- pages/Batches.tsx — list + Start Batch dialog (operation enum)
- pages/Deployments.tsx — list of worker version series
- pages/Nexus.tsx — list of cross-ns endpoints
- pages/Support.tsx — 4 cards linking to docs / GitHub / health
- ThemeToggle (light/dark, persisted), UTCClock (live ticker),
AccountChip (popover w/ identity + sign out)
Binary: 7.5 MB. Workflow execution semantics (deterministic replay of
user code) still pending — engine records state and tracks history len
on signals, but does NOT run user-supplied workflow goroutines yet.
Worker SDK runtime is the next phase.
The browser can only speak HTTP/JSON; ZAP is binary. Both transports now
dispatch to the same model functions in pkg/tasks so they cannot drift.
- pkg/tasks/embed.go: extract per-opcode model() funcs; ZAP handlers and
HTTPHandler() both call them. opcodes 0x0060/0x0065/0x0072/0x0081/
0x0082 return real JSON (empty/default state); the rest still return
501 until the native engine lands.
- cmd/tasksd: mount srv.HTTPHandler() at /v1/tasks/namespaces[/...] so
the embedded React UI's useSWR fetches succeed.
UI verified end-to-end with Playwright (headless Chromium):
/ → 302 → /_/tasks/namespaces (clean redirect)
Namespaces tab: "default" registered, retention 720h
Workflows tab: "No running or recent workflows" empty state
Schedules tab: "No schedules defined" empty state
console.error count: 0
failed requests: 0
Binary 6.99 MB (was 6.65; +340 KB for the shim and model funcs).
The Vite bundle is built with React Router basename="/_/tasks", so
the bundle only renders under that prefix. Serving it at "/" loaded
the HTML but the router refused to render — blank page in browser.
GET / → 302 → /_/tasks/
GET /_/tasks/ → 200 (UI)
GET /_/tasks/foo → 200 (SPA route)
GET /healthz → 200
GET /v1/tasks/health → 200
The plain `start` command on tasksd does not run sqlite/v1 schema migrations,
so a fresh /data PVC crashes with "no such table: cluster_metadata_info".
Add a new `embedded-sqlite` command that wraps temporaltest/embedded.LiteServer,
which calls sqlite.SetupSchema() before binding the frontend.
Also expose the HTTP API by default on port 7234 (gRPC stays on 7233). Without
HTTPPort the frontend is gRPC-only and clients that POST workflows/schedules
over HTTP get connection refused.
Image v2.0.2.
zeekayandHanzo Dev