mirror of https://github.com/hanzoai/s3.git synced 2026-11-02 13:11:24 +00:00

Files

T

zooqueen dd74d3e6d6 Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

hanzoai/s3 was an AGPL-3.0 MinIO fork whose network-copyleft blocks embedding
and reselling as a managed service. Replace it wholesale with a fork of
SeaweedFS (github.com/seaweedfs/seaweedfs, Apache-2.0, Go) which ships a
built-in S3 gateway + Filer.

- License: AGPL-3.0 -> Apache-2.0 (upstream LICENSE kept verbatim, incl.
  "Copyright 2025 Chris Lu"); NOTICE added crediting SeaweedFS. No AGPL/BUSL/
  SSPL/GPL or enterprise ("ee") files remain.
- Module: go.mod path set to github.com/hanzoai/s3 with a self-replace
  (replace github.com/seaweedfs/seaweedfs => ./) so the ~1800 internal Go files
  keep their upstream import paths with zero churn. Internal identifiers and
  import paths are intentionally NOT renamed.
- CLI rebrand: the binary is built as `s3` (was `weed`); root usage, per-command
  help/usage/flag strings, shell autocomplete program name, and the product name
  in CLI output now read "Hanzo S3" / `s3`. Users never see `weed`.
  (e.g. `s3 server`, `s3 filer`, `s3 master`, `s3 volume`, `s3 s3`, `s3 mount`.)
- Build: `make` (or `go build -o s3 ./weed`) builds the `s3` binary green;
  `cmd/weed-db` and `cmd/weed-sql` also build; `go vet ./weed/command/` clean.
- README/LLM.md rewritten for Hanzo S3; AGENTS.md/CLAUDE.md symlinks preserved.
- Dropped a 33MB prebuilt Mach-O `stress` binary upstream had committed; `.gitignore`
  updated to ignore the `s3` build output.

2026-06-18 23:41:25 -07:00

3.4 KiB

Raw Permalink Blame History

marp, theme, paginate

marp	theme	paginate
true	default	true

SeaweedFS Production Setup

A walkthrough of the recommended deployment topology

Storage layer
File access layer
Backend operations
Erasure coding for durability

Storage Layer

3 Masters + Volume Servers

Masters (run 3)
- Form a Raft quorum; tolerate 1 failure
- Track volume locations and cluster topology
- Lightweight; not on the data path for reads/writes after lookup
Volume Servers (scale out)
- Hold the actual needle data on local disks
- Add more servers for capacity and throughput
- Heartbeat to masters; clients talk to them directly for I/O

Rule of thumb: start with 3 masters, then grow volume servers with data.

File Access Layer

Filer + DB for metadata

Filer serves POSIX-like directory and file metadata
Metadata store options: PostgreSQL, MySQL, Redis, Cassandra, TiKV, etc.
Run 3 filers for HA
Filers are stateless relative to each other — state lives in the DB

S3 Servers

Expose the S3 API on top of the filer
Scale the count based on throughput requirements
Can be co-located with filers or run independently

Backend Operations

1 Admin Server + a few Workers

Admin server (1 is enough)
- Coordinates background jobs: balancing, EC encoding, vacuum, replication fixes
- Not on the data path — a restart does not affect object store reads/writes
- No HA required for normal operation
Workers (a few)
- Execute long-running tasks issued by the admin server
- Horizontally scalable based on maintenance workload

Keeps housekeeping off the hot path.

Production Topology Summary

Component	Count	Role
Master	3	Raft quorum, topology, volume map
Volume Server	N (scale out)	Stores data needles
Filer	3	Metadata gateway
Metadata DB	HA cluster	Persists filer metadata
S3 Server	N (by throughput)	S3 API frontend
Admin	1	Background job coordinator
Worker	A few	Executes admin-scheduled tasks

Erasure Coding for Durability

Prefer many volume servers so EC shards spread across failure domains.

Example: 8 volume servers, EC 5+3

Each volume becomes 5 data shards + 3 parity shards
Shards are placed on 8 distinct volume servers
Tolerates up to 3 simultaneous volume server failures
Storage overhead: 1.6x vs 3x for full replication

Volume -> [D1][D2][D3][D4][D5] + [P1][P2][P3]
           |   |   |   |   |     |   |   |
          VS1 VS2 VS3 VS4 VS5  VS6 VS7 VS8

More volume servers = more EC layouts available (e.g. 10+4, 6+3).

Getting Started Checklist

Provision 3 master nodes (small boxes are fine)
Provision volume servers sized to your data footprint
Stand up an HA metadata DB, then 3 filers pointing at it
Add S3 servers sized to required throughput
Run 1 admin + a few workers for background maintenance
Once you have enough volume servers, enable EC (e.g. 5+3 on 8 servers)

Scale each layer independently as usage grows.

3.4 KiB Raw Permalink Blame History