`ct list-changed` diffs against ct 2.3.1's default target branch, master, which
this repo does not have — git exits 128 and the step takes the job with it. It
set `steps.list-changed.outputs.changed`, and no step references it: the two
that follow are `ct lint --all` and `ct install --all`, already scoped by
--chart-dirs. So the step's only effect was to fail.
`branches: [master]` came across from .github/workflows; this repo's default
branch is main, so the lane has never produced a run. `runs-on: ubuntu-latest`
is the second half of the same problem — this fleet declares no ubuntu-* label
on purpose, and an unmatched label does not fail, it queues until the timeout.
workflow_dispatch so a lane gated behind a paths filter can be exercised without
touching the chart.
All three came across from .github/workflows unadapted, and all three ask for
`runs-on: ubuntu-latest`, which this fleet declares no label for on purpose — an
unmatched label does not fail, it queues until the job timeout. Two of them also
trigger on `branches: [master]`, a branch this repo does not have, so they have
never produced a single run.
helm_ci lints and installs the chart, which is real coverage and portable: it now
fires on main and asks for hanzo-build-linux-amd64, plus workflow_dispatch so the
lane behind a paths filter can be exercised on demand.
integration_test's unit-test job is `go vet ./...` + `go test ./...`, which is
already what hanzo.yml's `test:` block runs on every push — the same assertion
written twice. Its integration job needs sudo, a kind cluster and a real CSI
mount, none of which a job container here provides; those tests run nowhere today
and deleting the file says so instead of implying otherwise.
helm_release publishes through three GitHub-only surfaces — a `release:
published` event the forge never emits, a GitHub `environment:` protection, and
stefanprodan/helm-gh-pages. Our charts are served from oci.hanzo.ai/charts.
The official golang images set GOTOOLCHAIN=local, so a builder stage older
than the governing go.mod directive does not upgrade — it dies:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
go.mod here is go 1.26.5, and the two dev builders were on golang:1.24-alpine
and golang:1.25-alpine.
Two independent halves:
1. pin the dev builders to golang:1.26.5-alpine;
2. set ENV GOTOOLCHAIN=auto in every Go builder stage — including the two
images hanzo.yml actually publishes (s3-csi, s3-mount). Those sit on the
floating golang:1.26-alpine tag, which satisfies the directive today and
silently stops satisfying it the day go.mod crosses 1.27. GOTOOLCHAIN=auto
makes that a checksum-verified download (sum.golang.org) rather than a
failed release build. Their pins are left alone; only the ENV is added.
Separately, and NOT fixed here because it is a different bug: both .dev
builders `git clone https://github.com/s3/s3`, which is HTTP 404 — a stale
pre-rebrand URL. `make container-csi` / `container-mount` therefore cannot
succeed regardless of the Go pin. They are stale duplicates of
cmd/s3-csi-driver/Dockerfile and cmd/s3-mount/Dockerfile, which build the
same binaries correctly against github.com/hanzoai/s3 and are the ones
hanzo.yml and the integration workflow use. Deleting them and dropping the
Makefile targets is the DRY fix, but that is a deliberate call, not a
side effect of a toolchain patch.
Co-authored-by: Hanzo Dev <[email protected]>
The image build dies at `go mod download`:
verifying github.com/hanzoai/[email protected]: checksum mismatch
downloaded: h1:jyY5FGgOwXUT5gjuxYKAnvjmLNF3Mzchk6hfN0hqqec=
go.sum: h1:/hb8vroa+abcz+FzhU3PsT+bVg2lBNMioi4CGc36uTU=
SECURITY ERROR
hanzoai/s3 v1.0.5 was re-cut after publication, so nothing reachable still
serves what this lockfile expects. Reproduced independently here with a clean
module cache against the PUBLIC proxy and GOSUMDB=sum.golang.org: the same
mismatch, same two hashes. This build has no GOPRIVATE at all — GOPROXY is
"https://proxy.golang.org,direct" — so there is no fetch path left that agrees
with the pin.
Rolling FORWARD rather than rewriting go.sum to the re-cut hash. Adopting the
new bytes under the old version number would make this repo disagree with the
signed database and would have to be redone the next time that tag moves.
Checked which releases actually verify, clean cache each time:
v1.0.5 MISMATCH
v1.0.13 OK
v1.0.14 OK
v1.0.15 OK
v1.0.15 is the newest and is a patch-level move within v1.0.x — no major bump.
It pulled one transitive upgrade, google/uuid v1.6.0 -> v1.6.1-0.20241114…
Verified: `go get` + `go mod tidy` exit 0 (so the whole graph verifies),
`go build ./...` and `go vet ./...` are both clean for linux/amd64 with
CGO_ENABLED=0 — the configuration the image actually builds — and a pristine
`go mod download` reports zero mismatches. The local host cannot build cgo
targets (its SDK is missing stdlib.h/errno.h), which is unrelated and is why the
target triple is used for verification.
Worth stating: this is the fourth re-cut first-party tag hit today (luxfi
consensus and vm, hanzoai orm, now hanzoai s3). Re-cutting a published version
breaks every consumer that pinned it, silently, until they next resolve.
Co-authored-by: Hanzo Dev <[email protected]>
This repo had seven GitHub workflows. Three of them -- dev.yaml, release.yaml
and versioned_release.yaml -- built THE SAME two images, ghcr.io/hanzoai/s3-csi
and ghcr.io/hanzoai/s3-mount. Three builders on one commit is three chances to
publish something nobody asked for, and no single place that says what this repo
produces.
Now there is one: the repo-root hanzo.yml, which platform.hanzo.ai reads and the
hanzoai/ci reusable gates from, exactly as hanzoai/cloud does it.
.github/workflows is gone entirely. The pipelines that carry their own weight --
the kind integration suite and the two helm chart jobs -- moved to
.hanzo/workflows unchanged; they keep runs-on: ubuntu-latest because our
git-runner registers that label itself
(ubuntu-latest:docker://catthehacker/ubuntu:act-24.04), so the same line now
means our runner instead of GitHub's. prepare_release.yaml is dropped: bumping a
dependency and cutting a tag is the platform's job, not a workflow's.
Co-authored-by: hanzo-dev <[email protected]>
The driver's storage client has been ZAP-native for a while, but it still found
its endpoint by calling ToGrpcAddress() -- naming a protocol that is not on that
port. hanzoai/s3 now exposes ToFilerZapAddress() (hanzoai/s3#19), so the driver
asks for what it actually wants.
No gRPC name remains anywhere in this driver's storage path. The kubelet-facing
side stays CSI gRPC, because that is the Kubernetes contract between kubelet and
any CSI driver, not a choice this repo gets to make.
Both sides move to s3 v1.0.5, which carries the new method.
Co-authored-by: hanzo-dev <[email protected]>
The driver's filer client is already ZAP-native and dials the right port -- it
uses ToGrpcAddress() (httpPort+10000 = 18888), and 18888 is confirmed to be the
filer's ZAP listener, not gRPC: a plain gRPC handshake against it is reset
("error reading server preface: connection reset by peer"), and that is true
even for the fork's own filer.meta.tail. So the transport was never the problem.
What was left is a version skew across that transport. The driver built against
one s3 revision while the test deployed the 4.34.6 image, and the ZAP wire
library moved underneath them -- aligning the two also moves zap-proto/go from
v1.8.1 to v1.8.2.
Both sides now come from the same release, v1.0.4, which is published both as a
module tag and as an image tag. That also keeps the dependency on the only
lineage the module system can resolve: github.com/hanzoai/s3 declares no /vN
path, so its v2025.x/v2026.x tags are unusable, while v1.0.4 is exact -- no
pseudo-version needed.
Co-authored-by: hanzo-dev <[email protected]>
With the pod finally starting, the driver reaches the filer and fails there:
GRPC error: error creating volume:
filer mkdir /buckets/pvc-...: transport: connection closed
The driver was compiled against github.com/hanzoai/s3 v1.0.2 while the server
it dials reports itself as 4.34 -- so the client and the filer are far apart on
a transport this fork has been actively changing. v1.0.2 is also the build the
fleet reverted for breaking the S3 write path, so it is the wrong thing to
target twice over.
Pin to a pseudo-version at current main. That keeps the dependency on the legal
v1 lineage -- github.com/hanzoai/s3 declares no /vN module path, so its
v2025.x/v2026.x tags cannot be resolved by the module system at all, while
v1.0.4-0.20260728034919-7c7bb588e41e can.
Builds clean against the new API with no source changes; unit tests pass.
Co-authored-by: hanzo-dev <[email protected]>
Reproduced outside CI, with the exact args and image the job uses:
invalid value "tcp://10.96.11.22:8333" for environment variable s3.port:
strconv.ParseInt: parsing "tcp://10.96.11.22:8333": invalid syntax
The binary reads S3_PORT as its "s3.port" setting. Kubernetes injects
S3_PORT=tcp://<clusterIP>:<port> for the Service named "s3" into every pod in
that Service's namespace -- and this deployment is a Service named "s3" in a
namespace named "s3". So the server was being handed a URL where it wanted an
int, and exited 2 before it ever started. Production never hit it because it
passes -s3.port=9000 explicitly, and an explicit flag is not overridden by env.
enableServiceLinks: false removes the whole class of collision rather than the
one variable that happened to bite; -s3.port=8333 is set explicitly to match
the Service port and mirror how production runs it.
Also print the HEAD of the container log on failure. The reason is on line 1
and is followed by hundreds of lines of usage text, so --tail=200 hid the cause
twice in a row.
Co-authored-by: hanzo-dev <[email protected]>
With diagnostics in place the crash is unambiguous: the container prints its
full usage text and exits 2, the Go flag package's response to an unknown flag.
Of everything the manifest passes, -dir, -s3, -filer, -volume.max and
-master.volumeSizeLimitMB all appear in that usage. -master.peers does not, and
neither does any other peer flag — the fork dropped it. Production does not pass
it either.
This is the same shape as the driver bug this repo exists to fix: config written
against upstream SeaweedFS, run against the Hanzo fork, failing on an identifier
the fork renamed or removed.
Co-authored-by: hanzo-dev <[email protected]>
The integration job has been failing with
error: timed out waiting for the condition
s3-78ff9f58b-rw4dr 0/1 CrashLoopBackOff 4 (91s ago)
and nothing else. The container's own stderr was never printed, so every
diagnosis had to be a guess — the image runs fine outside kind with these exact
args, so the reason is environmental and only the crashed container knows it.
On a readiness failure, dump describe, the current log, the PREVIOUS log (the
actual crash, since the container has already restarted by then) and recent
events. Costs nothing on the happy path.
Co-authored-by: hanzo-dev <[email protected]>
The integration job has been failing with the s3 pod in CrashLoopBackOff. It
deploys ghcr.io/hanzoai/s3:v1.0.2, and that build is already known bad — the
fleet reverted it:
Revert "s3: roll to v1.0.2" — v1.0.2 breaks the S3 write path
Production runs 4.34.6, which is the build serving the filer right now. All
three references in this workflow (pull, kind load, and the deployment itself)
now use it, so the suite tests the driver against a filer that works instead of
failing for a reason that has nothing to do with the driver.
A test pinned to a build production has rejected can only produce noise.
Co-authored-by: hanzo-dev <[email protected]>
The integration test deployed ghcr.io/hanzoai/s3:latest and the pod went
CrashLoopBackOff — it pulled, so the tag exists, it just does not come up.
:latest was my guess when I repointed this off the upstream image. Production
runs 4.34.6, which is the build we know boots and serves the filer, so the test
now pins that. A floating tag is the wrong thing for a test to depend on
anyway: it makes the suite fail for reasons that have nothing to do with the
commit under test.
Co-authored-by: hanzo-dev <[email protected]>
The integration job died in 25 seconds, before kind ever started:
go build -o _output/s3-csi ./cmd/s3-csi
stat cmd/s3-csi: directory not found
It is cmd/s3-csi-driver. The Dockerfile paths were corrected earlier; this
go build line was the same stale path in a different place and was missed.
Everything after it — create cluster, load images, deploy, the crash-recovery
tests — was skipped, so the job's real coverage never ran at all. Reproduced
locally: the old path errors, the new one builds both binaries.
Co-authored-by: hanzo-dev <[email protected]>
* fix(docker): clone the fork that exists, at a commit that exists
The s3-mount image has never built. A global rename turned the upstream
seaweedfs reference into `github.com/s3/s3`, which is not a repository, so the
build died on `git clone ... exit code 128` before compiling anything.
Three more things were wrong behind it, each of which would have failed the next
step in turn:
clone github.com/s3/s3 -> github.com/hanzoai/s3
path /go/src/github.com/s3/s3 -> /go/src/github.com/hanzoai/s3
dir cd weed -> cd s3 (the fork renamed it)
commit 19ce7c0b6fdb -> 6dda4e4d6bca
That commit is the real one. 19ce7c0b6fdb is an UPSTREAM seaweedfs hash and does
not exist in our fork — the API 422s on it — so the checkout could never have
succeeded even with the URL fixed. The ARG stays so a build can still pin.
SEAWEEDFS_COMMIT is renamed S3_COMMIT while we are here; nothing consumed it yet.
* chore: no seaweed, no weed — it is s3
369 occurrences across 33 files, plus nine files whose names carried it. All of
it gone: manifests, helm chart, nomad jobs, workflows, docs and code.
Two of these were not cosmetic.
The driver spawned a binary called "weed":
DefaultWeedBinary = "weed"
Our fork builds it as `s3` — the running filer pod has /usr/bin/s3 and no weed
at all — so every mount would have failed with an executable that is not there,
however well the image built.
And the crash-recovery integration test killed processes matching `pgrep -x
weed`. Nothing is named that any more, so it would have found no pids, killed
nothing, and passed while testing exactly nothing. It now matches `s3`. The test
already had a guard that fails when there is nothing to kill, which is the only
reason this would have been noticed at all.
The rest follows the same rename the Dockerfile got: SEAWEEDFS_COMMIT is
S3_COMMIT, and the release/sync workflows that patch that ARG were updated with
it or they would silently stop pinning anything.
No upstream import paths were touched — go.mod depends on github.com/hanzoai/s3,
not on anything upstream, so nothing here was load-bearing for the build.
Verified: go build clean, all three test packages pass.
---------
Co-authored-by: hanzo-dev <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
The s3-mount image has never built. A global rename turned the upstream
seaweedfs reference into `github.com/s3/s3`, which is not a repository, so the
build died on `git clone ... exit code 128` before compiling anything.
Three more things were wrong behind it, each of which would have failed the next
step in turn:
clone github.com/s3/s3 -> github.com/hanzoai/s3
path /go/src/github.com/s3/s3 -> /go/src/github.com/hanzoai/s3
dir cd weed -> cd s3 (the fork renamed it)
commit 19ce7c0b6fdb -> 6dda4e4d6bca
That commit is the real one. 19ce7c0b6fdb is an UPSTREAM seaweedfs hash and does
not exist in our fork — the API 422s on it — so the checkout could never have
succeeded even with the URL fixed. The ARG stays so a build can still pin.
SEAWEEDFS_COMMIT is renamed S3_COMMIT while we are here; nothing consumed it yet.
Co-authored-by: hanzo-dev <[email protected]>
The repo was renamed to s3-csi in its workflows but not in its tree, so every
job failed on something that was never there:
build lstat cmd/s3-csi: no such file or directory (it is cmd/s3-csi-driver)
helm stat deploy/helm/s3-csi/Chart.yaml: not found (it is seaweedfs-csi-driver)
The Dockerfile paths now name the real directory, and the chart is renamed to
s3-csi — which the workflows already expected and which removes the last
seaweedfs name from the chart.
The integration test was worse than broken: it pulled chrislusf/s3:latest, the
UPSTREAM protobuf filer, to exercise a driver whose entire purpose is that it
speaks ZAP instead. That test could not have passed, and if it ever had it would
have been testing the wrong thing. It now runs against ghcr.io/hanzoai/s3.
release.yaml and versioned_release.yaml carried the same Docker Hub login as
chrislusf that dev.yaml did; all three now publish to ghcr.io/hanzoai.
Co-authored-by: hanzo-dev <[email protected]>
Every build of this driver has failed:
Error response from daemon: Get "https://registry-1.docker.io/v2/":
unauthorized: incorrect username or password
because the workflow came across with upstream's publishing intact — it logs
into Docker Hub as `chrislusf` and pushes chrislusf/s3-csi and chrislusf/s3-mount.
We do not have that token and should not want it.
Images now go to ghcr.io/hanzoai/{s3-csi,s3-mount} using GITHUB_TOKEN, which
needs no secret provisioned: the repo is public and can push to its own org's
registry. The job gains packages: write for it.
This is load-bearing. The cluster currently runs chrislusf/seaweedfs-csi-driver
because no Hanzo image was ever published, and that driver speaks protobuf to a
filer that is now ZAP-native — so every PVC on the seaweedfs-storage class fails
with `unknown service filer_pb.SeaweedFiler`. Publishing this driver is what
makes that storage class work at all.
Co-authored-by: hanzo-dev <[email protected]>
v1.0.1 predates s3/wire/filer, and v2026.4.0 is not Go-resolvable (the module
path has no /v2026), so @latest silently resolves to v1.0.1 — which has no ZAP
wire at all. Pin the main pseudo-version that actually contains it.
go.sum had no hanzoai/s3 entries because the module was built behind a local
replace; dropping the replace without tidying left the image build failing on a
missing go.sum entry while it compiled fine on my machine.
Co-authored-by: Hanzo Dev <[email protected]>
go.mod requires go >= 1.26.4; the builder image was still 1.25, so the image
build failed on GOTOOLCHAIN=local while the module compiled fine locally.
Co-authored-by: Hanzo Dev <[email protected]>
The content rename left the directories behind, and collapsed
./cmd/seaweedfs-csi-driver/main.go to ./cmd/s3-csi/main.go — a path that never
existed. Directories and build paths now agree.
Co-authored-by: Hanzo Dev <[email protected]>
Fork of seaweedfs/seaweedfs-csi-driver, renamed seaweedfs -> s3 throughout and
ported off protobuf entirely.
WHY a fork was unavoidable: hanzoai/s3 serves HanzoFiler over the native ZAP
transport on the port still named "grpc" (18888) — s3/command/filer.go binds it
with transport.ListenStream and says it "replaces the legacy gRPC HanzoFiler
server: the whole filer (28 unary + 5 streaming RPCs) now answers over ZAP, no
gRPC". RegisterHanzoFilerServer exists nowhere in that tree. So the upstream
driver's gRPC client cannot connect at all; it fails as
`Unimplemented: unknown service filer_pb.SeaweedFiler`, which LOOKS like a
service-name mismatch but is a gRPC client hitting a non-gRPC port. Renaming the
service in the client would not have helped.
pkg/filer is the only place this driver touches storage, and it speaks ZAP:
Mkdir -> CreateEntry (CreateVolume)
Remove -> DeleteEntry (DeleteVolume)
Exists -> LookupDirectoryEntry (ValidateVolumeCapabilities)
One shared, concurrency-safe client multiplexes over a single ZAP connection,
dials lazily so controller start does not depend on filer readiness, and redials
exactly once on a transport fault (covers a filer restart closing an idle conn
without becoming a retry storm against a struggling backend).
pkg/driver/volume.go Quota() likewise moves to s3/wire/mount — the fork generates
no mount_pb at all. Dead gRPC plumbing (WithFilerClient, AdjustedUrl,
grpcDialOption) is deleted rather than left inert.
The kubelet-facing side stays CSI gRPC: that is the Kubernetes contract between
kubelet and any CSI driver, not a Hanzo choice.
Zero protobuf call sites remain. Whole module compiles clean.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay
