Files
skills/public/hanzo-storage.md

6.0 KiB

Hanzo Storage - S3-Compatible Object Storage

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-platform.md, hanzo/hanzo-kms.md, hanzo/hanzo-universe.md

Overview

Hanzo Storage is a high-performance, S3-compatible object storage server for AI workloads. Derived from SeaweedFS and optimized for the Hanzo ecosystem. Written in Go. Live in production on Hanzo K8s clusters.

Why Hanzo Storage?

  • S3 API Compatible: Drop-in replacement for Amazon S3; works with any S3 client or SDK
  • Built for AI: Optimized for large model artifacts, training datasets, and data pipelines
  • Erasure Coding: Configurable data redundancy across drives and nodes
  • Server-Side Encryption: SSE-S3 and SSE-KMS for data at rest and in transit
  • Multi-Tenancy: Isolated namespaces and access boundaries for teams and services
  • CLI: the s3 binary's own shell — 153 commands covering fs, cluster, ec, volume, and mq

Tech Stack

  • Language: Go (module: github.com/hanzoai/s3)
  • Go Version: 1.26.5
  • License: Apache-2.0

OSS Base

Derived from SeaweedFS (Apache-2.0), attribution retained in LICENSE. Repo: hanzoai/s3. go.mod names zero MinIO modules; the checksum libraries are our Apache-2.0 forks under hanzos3/ (crc64nvme, md5-simd, go).

There is no separate CLI or console repo. mc and the MinIO console are not part of this stack — use the s3 shell for administration.

When to use

  • Storing model artifacts, checkpoints, and training datasets
  • Serving static assets and large files via S3 API
  • Providing S3-compatible object storage for Hanzo services
  • Replacing Amazon S3 in self-hosted or hybrid deployments
  • Bucket lifecycle management (expiration, tiering)

Hard requirements

  1. Go 1.26+ to build from source
  2. Docker for container deployment
  3. NVMe/SSD storage recommended for performance

Quick reference

Item Value
Repo github.com/hanzoai/s3
Module github.com/hanzoai/s3
Go Version 1.26.5
License Apache-2.0 (SeaweedFS-derived)
S3 API Port 9000
Filer / Master / Volume 8888 / 9333 / 8080
Docker Image ghcr.io/hanzoai/s3:v1.0.14 (pin a semver; never :latest)
CLI the s3 shell (in-binary)

One-file quickstart

Docker

docker run -p 9000:9000 -v s3-data:/data ghcr.io/hanzoai/s3:v1.0.14 \
  server -dir=/data -filer -s3 -s3.port=9000

Build from source

git clone https://github.com/hanzoai/s3.git
cd s3 && go build -o s3 .
./s3 server -dir=/data -filer -s3 -s3.port=9000

Administer with the s3 shell

The shell is the CLI. There is no mc.

s3 shell
> s3.bucket.create -name my-bucket
> s3.bucket.list
> s3.accesskey.create -user alice
> fs.ls /buckets/my-bucket
> cluster.status

fs.* walks the filer namespace, s3.* manages buckets and access keys, cluster.* and ec.* cover topology and erasure coding.

Docker Compose

# compose.yml
services:
  s3:
    image: ghcr.io/hanzoai/s3:v1.0.14
    ports:
      - "9000:9000"
    command: server -dir=/data -ip.bind=0.0.0.0 -filer -s3 -s3.port=9000
    environment:
      AWS_ACCESS_KEY_ID: "${S3_ACCESS_KEY}"
      AWS_SECRET_ACCESS_KEY: "${S3_SECRET_KEY}"
    volumes:
      - s3-data:/data

volumes:
  s3-data:

Core Concepts

Architecture

S3 clients (any SDK) ──> s3 gateway (:9000)
                              │
                          filer (:8888) ──> master (:9333) ──> volume (:8080)

Master tracks topology, volume stores chunks, filer holds the namespace, and the S3 gateway maps buckets onto filer paths.

SDK Compatibility

Standard AWS SDKs (aws-sdk-go, boto3, @aws-sdk/client-s3) work without modification — prefer them. For Go, github.com/hanzos3/go is our Apache-2.0 client.

Key Internal Packages

s3/
 main.go # Entry point (calls cmd.Main)
 cmd/ # CLI commands and server logic
 internal/
 auth/ # Authentication
 bucket/ # Bucket management
 config/ # Configuration
 crypto/ # Encryption (SSE-S3, SSE-KMS)
 event/ # Event notification
 grid/ # Internal RPC grid
 hash/ # Content hashing
 http/ # HTTP server
 jwt/ # JWT handling
 kms/ # KMS integration
 s3select/ # S3 Select queries
 store/ # Object store backend
 helm/ # Helm charts
 workers/ # Background workers
 buildscripts/ # Build automation
 dockerscripts/ # Docker entry scripts

Environment Variables

# S3 credentials (standard AWS names)
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

# Admin identities
S3_ADMIN_USER=...
S3_ADMIN_PASSWORD=...
S3_ADMIN_READONLY_USER=...
S3_ADMIN_READONLY_PASSWORD=...

# Optional
S3_BUCKET=...
S3_EXTERNAL_URL=...

Credentials come from KMS, never from literals in a manifest.

Key Features

  • Erasure Coding: ec.encode / ec.rebuild / ec.balance / ec.scrub
  • Bucket Policies: s3.bucket.access, s3.bucket.owner, s3.anonymous.*
  • Object Lifecycle: s3.bucket.lifecycle (expiration and tiering rules)
  • Quotas and Locking: s3.bucket.quota, s3.bucket.lock
  • Versioning: Object versioning with delete markers
  • Remote Tiering: remote.mount / remote.cache against a backing store

Troubleshooting

Issue Cause Solution
Reaching for mc Habit from the MinIO era Use s3 shell; mc is not part of this stack
Build fails Go version mismatch Requires Go 1.26+
S3 API refuses a bucket Bucket has no filer path yet s3.bucket.create -name <bucket> in the shell
Volume full volumeSizeLimitMB reached Raise -master.volumeSizeLimitMB or add volume servers
  • hanzo/hanzo-platform.md - PaaS deployment platform
  • hanzo/hanzo-kms.md - Secret management (SSE-KMS integration)
  • hanzo/hanzo-universe.md - Production K8s infrastructure
  • hanzo/hanzo-pubsub.md - Event notification targets (NATS)

Category: Hanzo Ecosystem Related: s3, storage, object-storage, seaweedfs, ai-data Prerequisites: Go 1.26+, Docker