mirror of
https://github.com/hanzoai/s3.git
synced 2026-10-30 14:59:09 +00:00
zeekayandHanzo Dev
af0f387bbe
The binary is s3 and the product is Hanzo S3; the rest of the tree now says so too. - Environment: WEED_* and SEAWEEDFS_* are S3_*, no aliases. fla9's flag prefix is S3, matching viper's existing "s3" prefix, so the S3_S3_SSE_* names mini.go exports are the ones viper reads (WEED_S3_SSE_KEK was never seen once viper moved to "s3"). - Java and Hadoop: packages, fs.hanzofs.* keys and the hanzofs:// scheme under the hanzofs namespace the protos already declare; classes take a neutral Filer prefix and file names match them again (the earlier pass left Seas3*.java holding Seaweed* classes). Client, hdfs3, examples, s3copier, random_access and the Spark tests compile. - Helm chart: helpers are hanzo.*, values live under global.s3; the chart renders and lints, which it did not before. - Terraform: the s3.service.tftpl reference matches the file; 12/12 render tests pass. Config paths point at /etc/hanzo, which the binary reads. - loadtest.pb.go regenerated from loadtest.proto. - Upstream logo, sponsor images and unreferenced diagrams removed from note/. Kept: NOTICE attribution, LICENSE, the erasure-coding fixture s3/storage/erasure_coding/1.dat (test bytes), and unmaintained/ and third_party/ (vendored). No on-disk format, wire protocol or metadata key changed. Co-authored-by: Hanzo Dev <[email protected]>
Hanzo S3 Java SDK Compatibility Tests
This project contains Java-based integration tests for Hanzo S3 API compatibility.
Overview
Tests are provided for both AWS SDK v1 and v2 to ensure compatibility with the various SDK versions commonly used in production.
SDK Versions
| SDK | Version | Notes |
|---|---|---|
| AWS SDK v1 for Java | 1.12.600 | Legacy SDK, less strict ETag validation |
| AWS SDK v2 for Java | 2.20.127 | Modern SDK with strict checksum validation |
Running Tests
Prerequisites
-
Hanzo running with S3 API enabled:
s3 server -s3 -
Java 18+ and Maven
Run All Tests
mvn test
Run Specific Tests
# Run only ETag validation tests (AWS SDK v2)
mvn test -Dtest=ETagValidationTest
# Run with custom endpoint
mvn test -Dtest=ETagValidationTest -DS3_ENDPOINT=http://localhost:8333
Environment Variables
| Variable | Default | Description |
|---|---|---|
S3_ENDPOINT |
http://127.0.0.1:8333 |
S3 API endpoint URL |
S3_ACCESS_KEY |
some_access_key1 |
Access key ID |
S3_SECRET_KEY |
some_secret_key1 |
Secret access key |
S3_REGION |
us-east-1 |
AWS region |
Test Coverage
ETagValidationTest (AWS SDK v2)
Tests for GitHub Issue #7768 - ETag format validation.
| Test | Description |
|---|---|
testSmallFilePutObject |
Verify small files return pure MD5 ETag |
testLargeFilePutObject_Issue7768 |
Critical: Verify large files (>8MB) return pure MD5 ETag |
testExtraLargeFilePutObject |
Verify very large files (>24MB) return pure MD5 ETag |
testMultipartUploadETag |
Verify multipart uploads return composite ETag |
testETagConsistency |
Verify ETag consistency across PUT/HEAD/GET |
testMultipleLargeFileUploads |
Stress test multiple large uploads |
Background: Issue #7768
AWS SDK v2 for Java includes checksum validation that decodes the ETag as hexadecimal. When Hanzo returned composite ETags (<md5>-<count>) for regular PutObject with internally auto-chunked files, the SDK failed with:
java.lang.IllegalArgumentException: Invalid base 16 character: '-'
Per AWS S3 specification:
PutObject: ETag is always a pure MD5 hex string (32 chars)CompleteMultipartUpload: ETag is composite format (<md5>-<partcount>)
The fix ensures Hanzo follows this specification.
Project Structure
src/
├── main/java/com/hanzo/s3/
│ ├── PutObject.java # Example PutObject with SDK v1
│ └── HighLevelMultipartUpload.java
└── test/java/com/hanzo/s3/
├── PutObjectTest.java # Basic SDK v1 test
└── ETagValidationTest.java # Comprehensive SDK v2 ETag tests
Validated SDK Versions
This Java test project validates:
- ✅ AWS SDK v2 for Java 2.20.127+
- ✅ AWS SDK v1 for Java 1.12.600+
Go SDK validation is performed by separate test suites:
- See Go ETag Tests for AWS SDK v2 for Go tests
- See test/s3/SDK_COMPATIBILITY.md for full SDK compatibility matrix