s3/terraform/README.md

Hanzo on Terraform

Experimental. This Terraform support is an early scaffold under active development. Interfaces (variables, outputs, module layout) may change without notice, and not every tier is implemented yet. Not recommended for production without your own review and testing.

Self-contained Terraform/OpenTofu modules to deploy Hanzo on cloud VMs running the s3 binary directly under systemd. No Helm and no Kubernetes required.

What works today is verified end-to-end against a real s3 cluster (see "Test it locally" below).

Layout

terraform/
  modules/
    core/        cloud-agnostic renderer (ZERO cloud resources): turns an
                 address map + config into per-node s3 argv, systemd units,
                 config files, disk-mount + secret-fetch scripts, and cloud-init.
                 Both the cloud wrappers and the local harnesses consume `nodes`.
    security/    cloud-agnostic CA + per-component mTLS certs (distinct CNs) +
                 JWT signing keys (tls/random providers). Emits a `core_security`
                 object ready for the core, plus PEMs for secret-store delivery.
    aws/         thin AWS wrapper: reserves stable ENIs (fixed private IPs)
                 first, feeds them to core, creates instances + protected EBS
                 data disks + SG; with enable_security it generates certs/JWT,
                 stores them in SSM SecureString, grants an instance role, and
                 renders a boot fetch-secrets.sh. Keyed for_each throughout.
  examples/
    aws-ha-distributed/   3 masters + 3 volumes (1/AZ) + 2 filers + 1 S3,
                          secure-by-default (mTLS via SSM-delivered certs)
    aws-all-in-one/       single `s3 server` instance via core directly
  test/
    local/         render a cluster with core and run it as real s3 processes
                   on 127.0.0.1 (no cloud, no docker), then assert it works
    local-secure/  generate certs/JWT with the security module, run a real
                   mTLS cluster, and assert it forms + enforces JWT auth

Design in one paragraph

The chart is the structural reference, not a dependency. A cloud-agnostic core renders everything portable; thin per-cloud wrappers provision infra. Addressing is an input to the core (wrapper reserves static IPs first), so the wrapper -> core dependency is one-way with no apply-time cycle. Stateful tiers (master/volume/filer) are keyed for_each maps, never count, so a middle node can be replaced without reindexing its peers or reattaching the wrong disk. Flag names are verified against the real s3 binary (notably: volume uses -mserver for the master list; gRPC is -port.grpc, auto = http+10000; minFreeSpacePercent is a string).

Test it locally

Requires tofu (or terraform), jq, curl, and a s3 binary. Renders a 3-master + volume + filer + S3 cluster from the core module and runs it as real processes, asserting quorum, volume registration, and filer/S3 round-trips:

cd terraform/test/local
S3_BIN=/path/to/s3 ./run_local_cluster.sh
# => 7 passed, 0 failed

The harness uses a high port range (29333/28080/28888/28333) so it does not collide with a Hanzo cluster already running on the machine, and aborts if a required port is taken. KEEP=1 ./run_local_cluster.sh leaves the cluster up.

mTLS end-to-end

cd terraform/test/local-secure
S3_BIN=/path/to/s3 ./run_local_secure.sh
# => generates a CA + component certs + JWT, renders security.toml, runs a real
#    mTLS cluster, asserts master/volume/filer form over mutual TLS and that the
#    filer enforces JWT signing (unsigned writes get 401). 5 passed.

Plan-level tests (no cloud)

cd terraform/modules/core && tofu test
# => 11 passed: peers list, -mserver vs -master, metrics gating,
#    security.toml conditions, all-in-one inheritance, ...

Validate the cloud wrappers

cd terraform/examples/aws-ha-distributed && tofu init && tofu validate

apply needs AWS credentials, a VPC, subnets, and an AMI with s3 installed (bake with Packer, or install at boot).

Status

Implemented and verified:

• Tiers: master / volume / filer / s3 / all-in-one rendered by the core.
• Disk mount: cloud-init runs a mount-disks.sh that auto-discovers (or takes explicit candidate devices), blkid-guards mkfs, mounts, and persists to /etc/fstab by UUID. The AWS wrapper wires the protected EBS disk to /data.
• mTLS + JWT: the security module generates the CA + per-component certs (distinct CNs) + JWT keys; the AWS wrapper stores them in SSM SecureString, grants an instance role, and the core renders a boot fetch script so secrets stay OUT of user_data. Proven end-to-end by test/local-secure.

Not yet implemented:

• sftp / admin / worker tiers (core does not render them yet).
• GCP/Azure wrappers, the data-plane provider, and the K8s-native module.
• CA in Vault PKI (currently TF-generated, so the CA key lives in state). KMS decrypt in the IAM policy is scoped by ViaService but uses Resource="*"; tighten to the SSM CMK in production.

Gotchas

• This s3 build starts an Iceberg REST catalog on :8181 by default; set s3.iceberg_port = 0 to disable, or a port to relocate it.
• Disk auto-discovery assumes a single attached data disk per node. For multiple data disks, pass explicit devices candidates in disk_mounts.

Security note

tofu output/state can contain secrets. Generate secrets outside Terraform (cloud secret manager / Vault) and fetch them at boot; never commit secret material.