6.0 KiB
Hanzo Storage - S3-Compatible Object Storage
Category: Hanzo Ecosystem
Related Skills: hanzo/hanzo-platform.md, hanzo/hanzo-kms.md, hanzo/hanzo-universe.md
Overview
Hanzo Storage is a high-performance, S3-compatible object storage server for AI workloads. Derived from SeaweedFS and optimized for the Hanzo ecosystem. Written in Go. Live in production on Hanzo K8s clusters.
Why Hanzo Storage?
- S3 API Compatible: Drop-in replacement for Amazon S3; works with any S3 client or SDK
- Built for AI: Optimized for large model artifacts, training datasets, and data pipelines
- Erasure Coding: Configurable data redundancy across drives and nodes
- Server-Side Encryption: SSE-S3 and SSE-KMS for data at rest and in transit
- Multi-Tenancy: Isolated namespaces and access boundaries for teams and services
- CLI: the
s3binary's own shell — 153 commands covering fs, cluster, ec, volume, and mq
Tech Stack
- Language: Go (module:
github.com/hanzoai/s3) - Go Version: 1.26.5
- License: Apache-2.0
OSS Base
Derived from SeaweedFS (Apache-2.0), attribution retained in LICENSE. Repo: hanzoai/s3. go.mod names zero MinIO modules; the checksum libraries are our Apache-2.0 forks under hanzos3/ (crc64nvme, md5-simd, go).
There is no separate CLI or console repo. mc and the MinIO console are not part of this stack — use the s3 shell for administration.
When to use
- Storing model artifacts, checkpoints, and training datasets
- Serving static assets and large files via S3 API
- Providing S3-compatible object storage for Hanzo services
- Replacing Amazon S3 in self-hosted or hybrid deployments
- Bucket lifecycle management (expiration, tiering)
Hard requirements
- Go 1.26+ to build from source
- Docker for container deployment
- NVMe/SSD storage recommended for performance
Quick reference
| Item | Value |
|---|---|
| Repo | github.com/hanzoai/s3 |
| Module | github.com/hanzoai/s3 |
| Go Version | 1.26.5 |
| License | Apache-2.0 (SeaweedFS-derived) |
| S3 API Port | 9000 |
| Filer / Master / Volume | 8888 / 9333 / 8080 |
| Docker Image | ghcr.io/hanzoai/s3:v1.0.14 (pin a semver; never :latest) |
| CLI | the s3 shell (in-binary) |
One-file quickstart
Docker
docker run -p 9000:9000 -v s3-data:/data ghcr.io/hanzoai/s3:v1.0.14 \
server -dir=/data -filer -s3 -s3.port=9000
Build from source
git clone https://github.com/hanzoai/s3.git
cd s3 && go build -o s3 .
./s3 server -dir=/data -filer -s3 -s3.port=9000
Administer with the s3 shell
The shell is the CLI. There is no mc.
s3 shell
> s3.bucket.create -name my-bucket
> s3.bucket.list
> s3.accesskey.create -user alice
> fs.ls /buckets/my-bucket
> cluster.status
fs.* walks the filer namespace, s3.* manages buckets and access keys,
cluster.* and ec.* cover topology and erasure coding.
Docker Compose
# compose.yml
services:
s3:
image: ghcr.io/hanzoai/s3:v1.0.14
ports:
- "9000:9000"
command: server -dir=/data -ip.bind=0.0.0.0 -filer -s3 -s3.port=9000
environment:
AWS_ACCESS_KEY_ID: "${S3_ACCESS_KEY}"
AWS_SECRET_ACCESS_KEY: "${S3_SECRET_KEY}"
volumes:
- s3-data:/data
volumes:
s3-data:
Core Concepts
Architecture
S3 clients (any SDK) ──> s3 gateway (:9000)
│
filer (:8888) ──> master (:9333) ──> volume (:8080)
Master tracks topology, volume stores chunks, filer holds the namespace, and the S3 gateway maps buckets onto filer paths.
SDK Compatibility
Standard AWS SDKs (aws-sdk-go, boto3, @aws-sdk/client-s3) work without
modification — prefer them. For Go, github.com/hanzos3/go is our Apache-2.0
client.
Key Internal Packages
s3/
main.go # Entry point (calls cmd.Main)
cmd/ # CLI commands and server logic
internal/
auth/ # Authentication
bucket/ # Bucket management
config/ # Configuration
crypto/ # Encryption (SSE-S3, SSE-KMS)
event/ # Event notification
grid/ # Internal RPC grid
hash/ # Content hashing
http/ # HTTP server
jwt/ # JWT handling
kms/ # KMS integration
s3select/ # S3 Select queries
store/ # Object store backend
helm/ # Helm charts
workers/ # Background workers
buildscripts/ # Build automation
dockerscripts/ # Docker entry scripts
Environment Variables
# S3 credentials (standard AWS names)
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
# Admin identities
S3_ADMIN_USER=...
S3_ADMIN_PASSWORD=...
S3_ADMIN_READONLY_USER=...
S3_ADMIN_READONLY_PASSWORD=...
# Optional
S3_BUCKET=...
S3_EXTERNAL_URL=...
Credentials come from KMS, never from literals in a manifest.
Key Features
- Erasure Coding:
ec.encode/ec.rebuild/ec.balance/ec.scrub - Bucket Policies:
s3.bucket.access,s3.bucket.owner,s3.anonymous.* - Object Lifecycle:
s3.bucket.lifecycle(expiration and tiering rules) - Quotas and Locking:
s3.bucket.quota,s3.bucket.lock - Versioning: Object versioning with delete markers
- Remote Tiering:
remote.mount/remote.cacheagainst a backing store
Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
Reaching for mc |
Habit from the MinIO era | Use s3 shell; mc is not part of this stack |
| Build fails | Go version mismatch | Requires Go 1.26+ |
| S3 API refuses a bucket | Bucket has no filer path yet | s3.bucket.create -name <bucket> in the shell |
| Volume full | volumeSizeLimitMB reached |
Raise -master.volumeSizeLimitMB or add volume servers |
Related Skills
hanzo/hanzo-platform.md- PaaS deployment platformhanzo/hanzo-kms.md- Secret management (SSE-KMS integration)hanzo/hanzo-universe.md- Production K8s infrastructurehanzo/hanzo-pubsub.md- Event notification targets (NATS)
Category: Hanzo Ecosystem Related: s3, storage, object-storage, seaweedfs, ai-data Prerequisites: Go 1.26+, Docker