mirror of https://github.com/hanzoai/s3.git synced 2026-10-30 14:59:09 +00:00

Files

T

History

zooqueen dd74d3e6d6 Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

hanzoai/s3 was an AGPL-3.0 MinIO fork whose network-copyleft blocks embedding
and reselling as a managed service. Replace it wholesale with a fork of
SeaweedFS (github.com/seaweedfs/seaweedfs, Apache-2.0, Go) which ships a
built-in S3 gateway + Filer.

- License: AGPL-3.0 -> Apache-2.0 (upstream LICENSE kept verbatim, incl.
  "Copyright 2025 Chris Lu"); NOTICE added crediting SeaweedFS. No AGPL/BUSL/
  SSPL/GPL or enterprise ("ee") files remain.
- Module: go.mod path set to github.com/hanzoai/s3 with a self-replace
  (replace github.com/seaweedfs/seaweedfs => ./) so the ~1800 internal Go files
  keep their upstream import paths with zero churn. Internal identifiers and
  import paths are intentionally NOT renamed.
- CLI rebrand: the binary is built as `s3` (was `weed`); root usage, per-command
  help/usage/flag strings, shell autocomplete program name, and the product name
  in CLI output now read "Hanzo S3" / `s3`. Users never see `weed`.
  (e.g. `s3 server`, `s3 filer`, `s3 master`, `s3 volume`, `s3 s3`, `s3 mount`.)
- Build: `make` (or `go build -o s3 ./weed`) builds the `s3` binary green;
  `cmd/weed-db` and `cmd/weed-sql` also build; `go vet ./weed/command/` clean.
- README/LLM.md rewritten for Hanzo S3; AGENTS.md/CLAUDE.md symlinks preserved.
- Dropped a 33MB prebuilt Mach-O `stress` binary upstream had committed; `.gitignore`
  updated to ignore the `s3` build output.

2026-06-18 23:41:25 -07:00

append_rows.py

Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

2026-06-18 23:41:25 -07:00

Dockerfile.writer

Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

2026-06-18 23:41:25 -07:00

doris_catalog_test.go

Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

2026-06-18 23:41:25 -07:00

README.md

Replace AGPL MinIO fork with Apache-2.0 SeaweedFS fork; rebrand CLI weed->s3

2026-06-18 23:41:25 -07:00

README.md

Apache Doris Iceberg Catalog Integration Test

This directory contains an Apache Doris integration smoke test for SeaweedFS's Iceberg REST Catalog implementation.

What It Tests

TestDorisIcebergCatalog verifies the Doris path end to end:

Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
Creates a SeaweedFS table bucket.
Creates an Iceberg namespace and an empty table through the SeaweedFS REST catalog OAuth flow.
Creates a second table and populates it with three rows by running a PyIceberg writer container (Dockerfile.writer + append_rows.py) before Doris bootstraps, so the snapshot is part of Doris's first scan.
Starts apache/doris:doris-all-in-one-2.1.0 and waits for the FE MySQL query port and at least one alive BE.
Connects to Doris over the MySQL protocol and registers a Doris EXTERNAL CATALOG of type=iceberg and iceberg.catalog.type=rest that points at the SeaweedFS REST endpoint, using credential = key:secret for the OAuth2 client-credentials flow.
Runs subtests against the SeaweedFS-backed Iceberg tables:
- BasicSelect: Doris is alive and answering SQL.
- CatalogVisible: SHOW CATALOGS lists the SeaweedFS-backed catalog.
- DatabaseVisible: the seeded namespace is exposed as a Doris database.
- TableVisible: the seeded table appears under SHOW TABLES FROM ....
- CountEmptyTable: catalog-to-table resolution and a scan of an empty table.
- ColumnProjection: SELECT id, label succeeds and the response columns are id, label. Failure here means Doris could not parse the schema returned by the SeaweedFS catalog.
- ReadWrittenDataCount and ReadWrittenDataValues: Doris reads back the three PyIceberg-appended rows and the values match. This exercises the actual data path (parquet reads via S3), not just metadata.

The PyIceberg writer image is built on demand via Docker layer caching. The first build pulls python:3.11-slim and pip-installs PyIceberg + PyArrow (~1-2 min in CI); subsequent invocations are cheap.

Running Locally

Build or install weed, then run:

cd test/s3tables/catalog_doris
go test -v -timeout 25m .

The test requires Docker. The GitHub Actions job runs on ubuntu-22.04 and executes the test for pull requests.

Configuration

The test uses these fixed credentials for the local SeaweedFS IAM config:

S3 access key: AKIAIOSFODNN7EXAMPLE
S3 secret key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Region: us-west-2
Warehouse bucket: iceberg-tables

Doris ports:

The all-in-one container exposes 9030 (FE MySQL/SQL) and 8030 (FE HTTP). Only 9030 is mapped to a host port (allocated dynamically) so the Go MySQL client can connect from the test process.
The Iceberg REST endpoint and the S3 endpoint are reached from inside the Doris container via host.docker.internal, matching the Trino and Dremio test paths.

Troubleshooting

Ensure Docker is running: docker version
Ensure weed is built or available on PATH
The Doris all-in-one image is large (~1 GB) and the FE+BE need ~30-60 seconds to register before queries succeed; the test waits up to 4 minutes for the query port and 60 seconds for at least one alive BE.
If queries hang on metadata, run REFRESH CATALOG iceberg_catalog from a Doris MySQL client and retry.
Container logs are printed in the failure message; you can also check docker logs <seaweed-doris-...> while the test is running.