tasks/LLM.md

Hanzo Tasks

Durable workflow execution engine for AI agent orchestration.

Upstream: Temporal (MIT). Pre-v3.0 history is the Temporal fork; v3.0+ is native ZAP-only with the upstream gRPC/proto/runtime fully ripped out. Public Go module pinned at v1.x forever (semver-import-versioning policy).

Module

github.com/hanzoai/tasks

Versioning policy (2026-04-30)

• Stay on v1.x.x forever. Per global CLAUDE.md "NEVER bump Go packages above v1.x.x."
• Major version bump in semver-import-versioning would require module-path suffix (/v2, /v3); we don't do that.
• Old v2.x and v3.x tags (transition era) were deleted on 2026-04-30. Old v1.x tags (v1.0.0–v1.42.0) are upstream-Temporal-era artifacts and remain in git history for blame.
• New Hanzo-Tasks releases continue from v1.43.0 and bump minor for features, patch for fixes.
• v1.51.2 added Embedded.CancelActivityForOrg; v1.51.3 added Embedded.ActivitiesPageForOrg — the org-scoped cancel + paginated read cloud's fleet queue surface (hanzoai/cloud clients/visor) depends on.
• v1.52.3 added View.FailureStreaks(ns) + the FailureStreak type — the durable "what is broken right now" read a host polls instead of hand-decoding SQLite.

Recovery and retention (v1.53.6)

• Recover reads only open runs. wfopen/<ns>/<workflowId>/<runId> is put before a run's first row and deleted after its terminal row (openRun/closeRun, durable.go). Recover lists it, so boot cost follows what is in flight, not how much history a shard holds. A shard written before the index is walked once (indexOpenRuns, paged, minimal decode) and marked wfidx/<ns>. Recover still covers the root tenant only, as before.
• Retention is enforced. runPurge (every 5 s, retention.go) walks each namespace's wf/ then act/ rows, 512 per pass per shard, from a cursor at purge/<ns>, across every tenant. A terminal run whose close time plus WorkflowExecutionRetentionTtl (default 720h) has passed is removed with every key it owns: wf, wfh, wfact, wfopen, the idem and sctrig entries naming it; for standalone activities act, ahist, aidem. All through the replicated delete path, row last. The row is re-read under the run lock first; an open run is never touched. A retention that does not parse purges nothing and logs tasks: retention purge failed (throttled).
• The file shrinks. Each pass ends in Shard.Reclaim. A default-mode shard is rebuilt once, at the end of a walk, with VACUUM when at least a quarter of it is free; that switches it to incremental auto-vacuum, and from then on each pass releases up to 4096 pages. The smaller file reaches disk at the next seal.
• Measured (200k terminal runs + 10 open, 1.71 GiB encrypted shard, pure-Go envelope, 8 vCPU): Embed 10.4–12.0 s / VmHWM 1.14–1.25 GB before; 7.7–8.7 s / 85 MB on the first boot (backfill walk); 6.0–6.2 s / 21 MB after. What is left is the envelope decrypting the whole file at open (~3.5 s/GiB, single-threaded). The purge drained the 200k runs in 392 passes (p99 197 ms), the shard went to 132 KiB, and Embed took 5 ms / 16 MB.
• Not here: the envelope's minute seal (hanzoai/sqlite Checkpoint) re-encrypts the whole file while holding the shard's one connection, about 9 s per minute at 1.7 GiB, whether or not anything changed.

Cron sweeper observability (v1.52.2)

The sweeper reports what it cannot return to a caller. Three slog messages off EmbedConfig.Logger (slog.Default() when unset), all from pkg/tasks/engine.go:

Message	Level	Meaning
tasks: cron schedule action failed	ERROR	One schedule's fire did not reach a worker. Carries org, namespace, scheduleId, workflowType, taskQueue, consecutiveFailures, error.
tasks: cron schedule action recovered	INFO	A previously failing schedule fired and reached a worker. Carries afterConsecutiveFailures.
tasks: schedule sweep failed, no schedule fired this tick	ERROR	The whole sweep failed — cron is dead for every org at once.

A fire fails in one of two ways, and both produce the same line because the operator's question is the same: the start itself errored, or the start succeeded and no worker is subscribed to (namespace, taskQueue) — the run exists and nothing will ever execute it. The error field says which. The second half catches a cron naming a namespace nobody serves (namespaces are created on first use, so a typo is no longer a start error), a schedule whose worker died, and a task queue renamed out from under an entry. The check is made AFTER the start, because enqueueing prunes subscriptions whose peer the transport can no longer reach.

Alert on the first; consecutiveFailures is the "how long has this been dead" number. Per-schedule isolation is unchanged — a broken entry is still skipped and retried next sweep; only the silence is gone.

Anchoring follows the start, not the report: a fire that ERRORED is not re-anchored (it stays due and retries every tick), a fire that STARTED is re-anchored even when it went unreached — the run exists, and re-firing on the strength of a report would spawn one every 5s.

Throttle: first failure, then one line per failReportThrottle (60) sweeps. Unthrottled a broken 5s-tick schedule is ~17k lines/day. The counter clears on a reached fire and on DeleteSchedule, so a relapse alerts immediately.

Why this exists: cloud's clients/cron fired a JobWorkflow whose activity re-read the entry's ConfigMap; the cloud ServiceAccount had no RBAC to read ConfigMaps, so every fire failed for 11 days across six nightly backups while the engine emitted nothing and every dashboard read healthy. It was found only by hand-reading the engine's durable records.

Activity / workflow failure observability (v1.52.3)

The sweeper fix above covers the SCHEDULER. It does not cover the incident: StartWorkflow succeeded all 4489 times, so the scheduler was healthy and correctly silent — the fault was one layer down, in the ACTIVITY. That layer's only output was e.emit(Event{Kind:"activity.failed"}) into the SSE broker: ephemeral, dropped when nobody is subscribed, and nobody was.

Guarantee: durable evidence, plus throttled log lines. Both, not either.

Durable — fail/<ns>/<fingerprint>, read via View.FailureStreaks(ns)

One FailureStreak row per failing identity, upserted on every failure and deleted on the first success. It needs no subscriber and survives restart, so ConsecutiveFailures can say 4489 where an in-memory counter would have said 1. A non-empty listing IS the list of what is broken; the keyspace is bounded by how much is broken now, not by how often it failed (so it does not have the unbounded-growth problem noted under Known gaps).

Identity is the recurring shape of the work — (workflowType, activityType, taskQueue, scheduleId) — never the run: every cron fire mints a fresh runId, so a run-keyed counter can never say "dead for 11 days". Each row carries org, namespace, workflowType, activityType, taskQueue, scheduleId, consecutiveFailures, attempt, retrying, persistent, firstFailureTime, lastFailureTime, lastError, and lastWorkflowId/lastRunId to go read.

ActivityType empty ⇒ the workflow itself failed (a decider that faults before scheduling anything — the hole an activity-only report would leave).

Log lines — pkg/tasks/failure.go

Message	Level	Meaning
tasks: activity failed	WARN	An activity attempt failed. retrying says whether the engine will try again.
tasks: activity failing persistently	ERROR	Alert on this. Failing continuously for ≥ failPersistentAfter (15 min).
tasks: activity recovered	INFO	Streak ended. Carries afterConsecutiveFailures.
tasks: workflow failed / … failing persistently / … recovered	WARN / ERROR / INFO	Same three, for a workflow failing with no activity to blame.

WARN vs ERROR is the distinction the incident turned on: "failed once and will retry" is normal operation and must not page anyone; "has failed every attempt for a long time" is the outage. Persistence is measured by age, not count — a nightly backup reaches a count of only 2 in two days yet is unambiguously broken, while one run burning its 10 default attempts (≈5.5 min of backoff) must not trip it. The persistent transition always reports, whatever the throttle says; waiting for failure #60 would be the original bug wearing a different hat.

Throttle

shouldReport is the one rule for the whole package (schedules included): first failure, then every failReportThrottle (60). Measured: 180 failures → 4 lines; stubbing the throttle off → 180. The throttle bounds LINES only — the durable count stays exact.

Two counters on purpose: the in-memory failTracker is the throttle authority (a store that is itself failing cannot starve it into reporting every occurrence, and a restart makes a still-broken activity re-alert once after a deploy), while the durable row is the evidence and always supplies the count on the line.

Not changed

Retry and isolation semantics are identical — which branch is taken, when, and how many times. TestFailureReportingPreservesRetrySemantics pins the attempt count, terminal status, history events and dispatch count. The SSE emit calls all remain.

Schedule-started runs now carry a ScheduleId search attribute (sweeper and TriggerSchedule), which is what lets a failure name the dead cron entry — and makes ScheduleId = "nightly-backup" a queryable filter over executions.

Quick Start

go build ./cmd/tasksd/
./tasksd start

Integration

• Playground connects to the Hanzo Tasks server via the durable-execution SDK
• Base embeds Tasks for durable cron/batch execution
• Each playground space = a Tasks namespace
• Each agent = a Tasks worker

Rebrand Notes (2026-03-19, updated 2026-04-26)

• Upstream server packages replaced with github.com/hanzoai/tasks in all Go files, go.mod, go.sum
• cmd/server renamed to cmd/tasksd, binary name is tasksd
• Docker images: ghcr.io/hanzoai/tasks
• 2026-04-26: dropped compat. temporal/→tasks/, temporaltest/→tasktests/, tests/ (upstream-compat suite, 3.7 MB) deleted. No backward-compat shim remains. tests/testutils/ retained (cert/TLS/IO helpers used by real unit tests).

Production Deployment (2026-03-19)

Live at tasks.hanzo.ai

• Cluster: hanzo-k8s (do-sfo3-hanzo-k8s), namespace hanzo
• Server: ghcr.io/hanzoai/tasks:latest -- ZAP on 9999, HTTP on 7243. --zap takes an ADDRESS: :9999 for TCP, or a path to bind a unix socket for service-to-service traffic that never leaves the host. A socket path must fit sockaddr_un.sun_path — 107 bytes plus its NUL — which a socket nested under a deployment's data directory can exceed; Embed and ServeGated refuse an oversized one by name, because the kernel's own answer is bind: invalid argument and nothing more.
• UI: ghcr.io/hanzoai/tasks-ui:latest -- port 8080
• Database: PostgreSQL at sql.hanzo.svc:5432 (databases: tasks, tasks_visibility)
• Secrets: KMS-managed via tasks-secrets (POSTGRES_PASSWORD, TASKS_AUTH_SECRET, IAM_CLIENT_SECRET)

Domains

URL	Service	Protocol
tasks.hanzo.ai	tasks-ui (Web UI)	HTTPS
tasks-api.hanzo.ai	tasks (ZAP)	ZAP+TLS

IAM Integration

• Provider: hanzo.id (OIDC)
• Client ID: app-tasks
• Callback: https://tasks.hanzo.ai/auth/sso/callback
• Scopes: openid, profile, email
• JWKS URI: https://hanzo.id/v1/iam/.well-known/jwks (in-cluster: http://iam.hanzo.svc/v1/iam/.well-known/jwks)
• Discovery: https://hanzo.id/.well-known/openid-configuration
• Registration script: scripts/register-iam.sh

Auth Flow (two layers)

1. UI (OIDC login): Tasks UI (tasks-ui container) handles the browser OIDC flow. User visits tasks.hanzo.ai, gets redirected to hanzo.id for login, callback returns JWT tokens. Configured via TASKS_AUTH_* env vars on the tasks-ui container.
2. Server (JWT validation): Tasks server validates the JWT bearer token on every ZAP/HTTP request using JWKS keys fetched from hanzo.id. Configured via the wire-protocol env vars below (legacy names preserved for upstream config compatibility). These env vars feed into the embedded config template at common/config/config_template_embedded.yaml -> global.authorization.

Embedded Config Template Auth Env Vars (server)

Canonical TASKS_* env var names. The binary reads these directly via the embedded config template parser.

Env Var	Purpose	Value in K8s
TASKS_JWT_KEY_SOURCE1	JWKS URI for key fetching	http://iam.hanzo.svc/v1/iam/.well-known/jwks
TASKS_JWT_KEY_REFRESH	Key refresh interval	5m
TASKS_AUTH_AUTHORIZER	Authorizer type	default
TASKS_AUTH_CLAIM_MAPPER	Claim mapper type	default
TASKS_JWT_PERMISSIONS_CLAIM	JWT claim for permissions	permissions

UI Auth Env Vars (tasks-ui container)

Env Var	Purpose	Value in K8s
TASKS_AUTH_ENABLED	Enable OIDC login	true
TASKS_AUTH_PROVIDER_URL	OIDC issuer	https://hanzo.id
TASKS_AUTH_CLIENT_ID	OIDC client ID	app-tasks
TASKS_AUTH_CLIENT_SECRET	OIDC client secret	(from tasks-secrets)
TASKS_AUTH_CALLBACK_URL	OIDC callback	https://tasks.hanzo.ai/auth/sso/callback
TASKS_AUTH_SCOPES	OIDC scopes	openid,profile,email

Namespace-to-Org Mapping

Tasks namespaces map 1:1 to Hanzo orgs. Users see only namespaces matching their IAM org memberships. The JWT permissions claim carries namespace:role pairs (e.g., hanzo:admin, lux:read). The default claim mapper parses these into the internal permission model.

K8s Manifests

• Canonical source: k8s/ in this repo
• Also mirrored in: universe/infra/k8s/tasks/
• Apply with: kubectl apply -k k8s/
• DB init (one-time): kubectl apply -f k8s/init-db.yaml

CI/CD

One way, and it runs on our own stack:

push  ->  github.com/hanzoai/tasks        (a mirror)
          .github/workflows/sync.yml       carries refs onward
  ->  git.hanzo.ai/hanzoai/tasks           CANONICAL
          .hanzo/workflows/cicd.yml        tests + builds ghcr.io/hanzoai/tasks
  ->  hanzoai/cloud go.mod                 pins the tag this repo cuts

git.hanzo.ai is canonical; GitHub is a mirror. .github/workflows/ holds exactly one file, sync.yml, and its only job is getting refs to the forge. Every build, check and deploy is a workflow under .hanzo/workflows/, which the forge reads. .hanzo/workflows uses GitHub Actions syntax, so a workflow moves between the two by changing directory and nothing else.

.hanzo/workflows/cicd.yml is a thin trigger; all config lives in the repo-root hanzo.yml — go test -race ./pkg/... ./cmd/... plus go vet, and the image ghcr.io/hanzoai/tasks from the root Dockerfile. hanzo.yml declares no deploy: key, so cicd.yml never deploys: it publishes an image and stops.

What this repo is actually load-bearing for is the Go module tag, which hanzoai/cloud go.mod pins. Tags are cut by hand; CI only stamps image tags off a tag that already exists.

The tasks image has no live consumer today. tasks.hanzo.ai is served by cloud's embedded clients/tasks/ui, not by this image, and there is no App CR for it in hanzoai/universe.

Observability

• OTEL traces: otel-collector.hanzo.svc:4318
• Insights analytics: insights-capture.hanzo.svc:3000
• Dynamic config: /etc/tasks/dynamic-config/dynamic-config.yaml (ConfigMap)

Native ZAP-only (2026-04-26)

The temporal fork is GONE. Zero go.temporal.io/*. Zero google.golang.org/grpc. Zero protobuf on the wire. The whole binary is:

cmd/tasksd       # 100 lines: signal handling, ZAP node, HTTP server
pkg/tasks/       # in-process server (zap.Node + opcode dispatch)
pkg/sdk/         # client + worker + workflow + activity + converter
ui/              # embedded React SPA (Vite bundle)
schema/tasks.zap # canonical wire schema

Build proof: go build ./cmd/tasksd → 10 MB native binary, 208 deps. Boot proof: tasksd --zap :9999 --http :7243 listens on both, serves /healthz, /v1/tasks/health, /_/tasks/* (UI), responds to ZAP opcodes 0x0050–0x00A5 from pkg/sdk/client.

The native engine is built and GREEN (see "## Durable Engine" below for the true current state and the Phase-2 event-sourced design). Only one opcode still returns 501: OpcodeStartChildWorkflow (0x006D). Everything else — ExecuteWorkflow, activity dispatch, signal, cancel, terminate, query, reset, schedules, namespaces, deployments — is real and tested.

What was deleted (2026-04-26)

• tasks/ (renamed temporal/, the fork's runtime)
• tasktests/ (renamed temporaltest/)
• service/ (frontend / history / matching / worker — 817 files)
• chasm/ (component state machine framework — 169 files)
• client/ (legacy gRPC clients — 40 files)
• api/ (local mirror of temporal protos — 114 files)
• proto/, tools/ (codegen for temporal protos)
• cmd/tools/ (genrpcwrappers, genrpcserverinterceptors, getproto, etc.)
• components/ (nexusoperations, callbacks, dummy state machines)
• docker/ (pre-native server containerization)
• schema/{cassandra,elasticsearch,mysql,postgresql,sqlite}/ (DB driver schemas — embedded SQLite returns via pkg/tasks when persistence lands)
• 40 tainted common/ subdirs + 7 top-level common files (util.go, rpc.go, rpc_mock.go, client_cache.go, daemon.go, constants.go)
• tests/ (upstream-compat suite, 3.7 MB)

What remains in common/

Pure stdlib utilities that survive the rip: aggregate, auth, build, channel, circuitbreaker, clock, collection, contextutil, convert, debug, definition, effect, finalizer, future, goro, health, masker, number, pingable, pprof, predicates, quotas, resolver, routing, schedules, shuffle, stream_batcher, tasks, tasktoken, timer, util, versioninfo. These are candidates for further pruning once the native engine settles.

Auth: IAM only, through the estate's one reader

tasksd validates Authorization: Bearer <jwt> itself — no gateway required — and the strip+write pattern is the trust boundary. What it does NOT do any more is hold its own reading of that contract.

The reader is hanzoai/authz + hanzoai/authz/edge. pkg/auth is down to the part that is genuinely tasksd's: which environment variables the policy comes from. It had 190 lines of JWKS cache and JWT verification, and two defects a shared reader does not have:

• it verified against every key in the published set and took the first that worked, so a token naming kid A was accepted on a signature from key B. Naming a key is what kid is for.
• its issuer comparison was conditional on the configured issuer being non-empty, so an unset TASKSD_JWT_ISSUER accepted tokens from any issuer instead of failing closed.

It also stripped only the four names it reads. Every other written header — X-User-IsAdmin and X-User-Permissions (the platform and money signals), X-User-Owner, X-Billing-Account-Id, X-Scope, X-Workspace-Id, and the retired names — passed through from the client untouched, and since tasksd's ingress routes straight to the Service, nothing upstream was deleting them either. It now strips authz.Headers ∪ authz.Retired and writes through edge.Apply, so the strip list IS the write list.

And it gains a machine predicate it never had: a client_credentials token carries no membership set, so it is not a person and holds neither admin scope.

Per-org store scoping is engine.As(Org(auth.OrgID(ctx))), resolved through Claims.EffectiveOrg("") on BOTH transports — the one function that answers "which org does this act in". tasksd honours no org switch, so that resolves the home org, which is the first entry of the SIGNED MEMBERSHIP SET. It is deliberately NOT the owner claim: IAM stamps the org of the APPLICATION a token was minted through there, so reading it would make a person's namespace follow whichever app they signed in through. A token with no membership set states no org and is refused. The frontend scopes to the ORG; a host wanting project- or user-level isolation asks for it explicitly via Embedded.View(Principal{Org, Project, User}).

TASKSD_REQUIRE_IDENTITY=true (production) rejects requests without a validated JWT. Default false keeps the embedded/dev path working.

Cost, measured: 342 linked packages and an 18.3 MB binary, DOWN from 345 and 18.6 MB — dropping go-jose more than paid for the leaf. (The "10 MB, 208 deps" figure earlier in this document was already stale before this change.)

Production env (do-sfo3-hanzo-k8s/hanzo):

TASKSD_JWKS_URL=http://iam.hanzo.svc/v1/iam/.well-known/jwks
TASKSD_JWT_ISSUER=https://hanzo.id
TASKSD_REQUIRE_IDENTITY=true

History — pre-v3.5.0 the middleware trusted client-supplied X-* headers on the assumption that hanzoai/gateway sat between ingress and tasksd. The actual topology routed direct to the Service, leaving X-Org-Id spoofable on tasks-api.hanzo.ai. v3.5.0 made tasksd self-sufficient; this change makes it self-sufficient without being a fourth opinion about what an IAM token means.

Durable Engine — true state + Phase-2 event-sourcing design

Authoritative architecture note (CTO). Supersedes the stale "handlers return a 501" text above. Read this before touching pkg/tasks or the workflow runtime in pkg/sdk.

Where the engine actually is (corrected)

The native engine is NOT a shell of 501 stubs. It is Phase-1-complete and green (go test -race ./pkg/tasks/... ./pkg/sdk/... passes). Built and tested today:

• ExecuteWorkflow — engine.startWorkflowWithRequestID: persists a WorkflowExecution (SQLite), writes WORKFLOW_EXECUTION_STARTED to an append-only history, mints a run id, enqueues a workflow task to a subscribed worker. Idempotent on (ns, workflowId, requestID) via the idem/ keyspace.
• Append-only history — engine.appendHistory + GetWorkflowHistory (paginated, reversible). HistoryEvent{EventId, EventTime, EventType, Attributes}, EventId monotonic per (ns, wf, run).
• Signal / Cancel (two-phase CANCEL_REQUESTED→CANCELED handshake with a sweep) / Terminate / Query (server pushes the query to a worker and awaits its response) / Reset (truncate history at an eventId).
• Standalone activities (activities.go): first-class activity records with ACTIVITY_TASK_SCHEDULED/STARTED/COMPLETED/FAILED history.
• Worker dispatch (dispatch.go): workers Subscribe per (ns, queue, kind); server pushes tasks (no polling); HMAC task tokens; round-robin; pending queues drain on subscribe; activity results pushed back to the workflow's peer.
• Schedules (cron + interval), namespaces, deployments (worker versioning), nexus endpoints, identities, visibility query parser, per-tenant store scoping, SSE event stream, replication scaffolding (replication/ quasar + local), migration tool.

Persistence = per-(principal, namespace) SQLite shard (pkg/tasks/store), WAL, single-writer, kv + history + idem + meta tables. Survives restart (TestStore_PersistsAcrossOpen).

A Principal is the tenant that owns a shard — an org, optionally narrowed to a project. Shards and namespaces are created on first use; nothing is declared up front.

Storage and encryption — hanzoai/cek + hanzoai/namespace

Tasks no longer has a storage layout or a key schedule of its own. It uses the estate's:

namespace   names the tenant, and where its file lives
cek         turns the process master + that name into the file's key,
            and opens it
sqlite      opens a file under a raw key, knowing nothing about who owns it
kms         holds the master

<data>/orgs/<org>/<ns>.db
<data>/orgs/<org>/projects/<project>/<ns>.db
<data>/orgs/_platform/<ns>.db          the root (unscoped) tenant

The key is DERIVED (HKDF of master + namespace + namespace-name), not generated, wrapped or stored. So there is no DEK, no .dek sidecar, no unwrap step, no rewrap step, no per-file key material to lose, and no migration path to maintain. A shard is born encrypted or it does not exist; losing the master loses the data, which is the property you want from encryption at rest and the reason the master lives in KMS.

EmbedConfig.MasterKey (32 bytes, KMS_MASTER_KEY base64 for tasksd) is installed into cek at store.New. A nil master no longer means plaintext — there is no such posture. It means "I have no key to state": embedded in a process that already installed one (cloud resolves it through credz at boot) that key is used, so one deployment has one key; otherwise a random master is minted for this process and nothing it writes outlives the run, which is the honest shape of a keyless deployment and why a developer needs no configuration.

A shard is ciphertext on every build: hanzoai/sqlite uses the live libsqlcipher codec when one is linked and its pure-Go SQLCipher envelope when one is not. On the envelope the file is written when it is SEALED (checkpoint/close), not per commit — which is what the manager's sweep bounds, and why ListShards/ListPrincipals read the resident set as well as the directory: a shard created moments ago has no file yet, and that is exactly the tenant the cron sweeper most needs to see.

The org is folded to its storage identity once, at the door (store.Org, via the one injective slugger namespace.Sanitize), and never again when that identity is turned into a place. Principal.Org downstream is therefore the STORED name — which is what lets ListPrincipals read a slug back off disk and reach the same file. Folding twice would re-suffix it into a different, empty database, since Sanitize is injective but deliberately not idempotent. A name Sanitize refuses is kept raw so Valid rejects it, because folding it would yield the empty org — and an empty org leg is the ROOT principal, so a name too hostile to store would otherwise have become the deployment's own tenant.

Known gap — no user-scoped shard

Principal still carries a User leg (it is the wire encoding, and ParsePrincipal/String round-trip it), but a principal that narrows to a user cannot open a shard: Principal.Namespace returns store.ErrUserLeg. namespace.Key refuses a user namespace on purpose — that layout has no place for one, "and inventing one silently is how a second convention starts". The old <org>/<project>/<user>/ tree was tasks' own; sharing the estate's layout costs that depth until hanzoai/namespace grows a user layout. Scope to the org or its project.

tasksd upgrade is gone with the layout it moved shards into. The only subcommand is migrate, which moves ONE shard across NODES.

What is genuinely missing (the real gap)

The whole system is explicitly Phase 1: coroutine-push, not replay (see pkg/sdk/worker/worker.go:19, workflow/version.go:31, workflow/local_activity.go:62). A workflow runs as ONE live worker goroutine that blocks on activity results pushed back over the wire. Consequences:

1. No deterministic replay. Workflow-driven activities (worker OpcodeScheduleActivity 0x006B) write NOTHING to the workflow's history — only START/terminal/signal/cancel land. The history cannot reconstruct a run, so it cannot be replayed.
2. No crash-recovery of workflows. If the worker (or tasksd) dies mid-run, the live goroutine is lost; nothing replays history to resume. The dispatcher's pending/inflight state is in-memory only.
3. No timers (workflow Sleep/NewTimer has no durable server side), no activity retry/backoff (RetryPolicy is stored, never enforced), no child workflows (OpcodeStartChildWorkflow → 501).

Target architecture — event-sourced durable execution

One model, the Temporal core done minimally. History is the only source of truth. A workflow task is one decision episode. The worker is stateless between episodes — it rebuilds state by replaying history.

The loop:

1. ExecuteWorkflow persists the run + WORKFLOW_EXECUTION_STARTED, marks a workflow task pending, enqueues it.
2. Server delivers the workflow task carrying the full history (WorkflowTask.History — the wire slot already exists; today it smuggles raw input, dispatch.go:431 "Phase 2 will parse an event-sourced history").
3. Worker replays from event 0, re-running the registered function. Each workflow primitive consults history by a deterministic command sequence number (seq, incremented per command in program order):
   • ExecuteActivity at seq=k: if ActivityTaskCompleted{seq=k} is in history → return its recorded result (NO re-dispatch); if ActivityTaskFailed{seq=k} (retries exhausted) → return the failure; else emit ScheduleActivity{seq=k} and the future stays unresolved.
   • NewTimer/Sleep at seq=k: TimerFired{seq=k} → resolved; else emit StartTimer{seq=k, fireAt}.
   • Signals: WorkflowExecutionSignaled events replay into the signal channels in history order.
4. When the function can make no more progress (all pending futures unresolved) the episode ends; the worker returns the batch of NEW commands via RespondWorkflowTaskCompleted (the commandsEnvelope already models ScheduleActivity; add StartTimer, StartChild, plus the existing CompleteWorkflow/FailWorkflow).
5. Server applies commands — for each: append the corresponding *_SCHEDULED/*_STARTED history event and act (dispatch activity, arm timer, start child, terminal-transition). Command application is idempotent on (run, seq).
6. When an activity completes/fails, a timer fires, a signal or cancel arrives, or a child completes → append the result event (ActivityTaskCompleted{seq}, TimerFired{seq}, …) and schedule a new workflow task. Back to step 2.

Determinism invariant: same history ⇒ same command sequence. Enforced by (a) the workflow function being pure w.r.t. inputs + history, (b) every non-deterministic input (activity result, timer fire, time, version marker, side-effect) sourced from a history event keyed by seq, (c) the decider dropping commands whose seq already has a *_SCHEDULED/*_STARTED event (no double-schedule on replay).

Exactly-once activity dispatch: an activity is identified by (ns, run, seq). Scheduling is idempotent on that key — replay or a recovery re-dispatch of the same seq returns the same activity record and never runs the user activity twice once it is terminal. Server-side retry: on ActivityTaskFailed, if attempt < RetryPolicy.MaximumAttempts and the error is retryable, re-dispatch after min(MaximumInterval, InitialInterval · BackoffCoefficient^(attempt-1)), recording each attempt; on exhaustion write the terminal ActivityTaskFailed{seq} and schedule a workflow task.

Crash-recovery (single replica): everything above is in SQLite. On tasksd boot, a recovery pass scans RUNNING workflows and rebuilds the in-memory dispatcher from history: any workflow whose latest event needs a decision (started, or a result landed with no following WorkflowTaskCompleted) → re-enqueue a workflow task; any activity Scheduled/Started with no terminal event → re-dispatch (idempotent per (run, seq)); any TimerStarted with no TimerFired → re-arm to its fireAt. Because dispatch is seq-keyed and the decider dedups on replay, recovery causes no lost and no double execution.

Multi-replica (later): the SQLite store already replicates via replication/ (quasar). Add a per-(ns, run) dispatch lease in the replicated meta table so exactly one replica owns advancing a given run at a time (mirror the exactly-once lease pattern used elsewhere in the fleet). Until then tasksd runs single-writer (replicas=1).

History event vocabulary (add to the existing family)

WORKFLOW_TASK_SCHEDULED/STARTED/COMPLETED/FAILED, ACTIVITY_TASK_SCHEDULED/STARTED/COMPLETED/FAILED/TIMED_OUT, TIMER_STARTED/FIRED/CANCELED, CHILD_WORKFLOW_EXECUTION_STARTED/COMPLETED/FAILED. Each carries its seq in Attributes. These are JSON-native, owned by Hanzo, and persisted in the history table alongside the existing kv records.

Phasing (what ships when)

• Phase 2a (first coherent slice, this lane): durable activity core. Workflow-driven activity lifecycle written to the workflow's history; exactly-once dispatch keyed by (run, seq); server-side activity retry+backoff per RetryPolicy; crash-recovery re-dispatch of in-flight activities; history-backed ExecuteActivity replay (recorded result returned on re-run) + workflow-task-driven advance. Tests (-race): determinism/replay, retry+backoff timing, exactly-once dispatch, crash-recovery (kill mid-activity → resume, no double-run).
• Phase 2b: durable timers (StartTimer/TimerFired + timer wheel), signals/cancel folded into replay, child workflows (retire the 0x006D 501), query consistency against replayed state.
• Phase 2c: multi-replica dispatch lease over the replicated store.

Scope discipline (agent orchestration, not Temporal parity)

Workflows = agent-session / subagent trees. Activities = agent runs + tool calls. Signals = control commands (pause/resume/message). Timers = schedules/retries. Child workflows = subagents. Skip search attributes v2, advanced visibility, multi-namespace federation, Nexus cross-cluster — unless trivial. Once ExecuteWorkflow + child + signal + cancel are real, task #31 (agent sessions) wires the live TaskController in one line and #32 (converge agent-scheduler + vm-metering tickers onto durable cron) unblocks.

KNOWN HAZARD — origin/main regression (coordination)

origin/main was force-pushed on 2026-07-02 to re-merge ~494 upstream Temporal commits (merge: upstream/main, pin go.temporal.io/server v1.32.0-157.0), re-adding proto/, config/, develop/, docker/ and the go.temporal.io/* deps — while the LLM.md still claims "runtime fully ripped out." The native engine (pkg/tasks) is temporal-free and identical across the divergence; the re-add is dependency/tooling baggage the engine does not use. This contradicts the "temporal fork is GONE" doctrine and must be resolved by a deliberate decision (re-rip vs. keep as fork base) — do not let it silently become the base. feat/tasks-durable- engine branches off this commit and keeps all new work inside the temporal-free pkg/tasks + pkg/sdk/workflow zones so it rebases clean whichever way that decision lands.