mirror of https://github.com/hanzoai/s3.git synced 2026-10-30 14:59:09 +00:00

Files

T

zeekay 8f5d0b9333 rebrand: rename weed→s3, strip all SeaweedFS/weed branding → Hanzo S3

Full internal rename — build-verified, every module green (root + s3-rdma-sidecar
+ telemetry/server + all test modules):
- Dirs: weed/→s3/ (main tree); cmd/weed-{db,sql}→cmd/s3-{db,sql};
  seaweed-volume→s3-volume; seaweedfs-rdma-sidecar→s3-rdma-sidecar;
  s3/mount/weedfs_*.go→s3fs_*.go (~113 path renames).
- Imports: github.com/hanzoai/s3/weed/ → /s3/ across 1700+ files.
- gRPC proto services Seaweed/SeaweedFiler/SeaweedMessaging/... → Hanzo*
  (8 services) via full protoc regeneration (correct descriptors, no sed-corruption).
- Brand SeaweedFS/Seaweed/seaweed → Hanzo/hanzo; internal X-Seaweedfs-* headers
  → X-Hanzo-*; .seaweedfs config dir → .s3.
- CLI is `s3`: go install github.com/hanzoai/s3/s3@latest | make install | Docker.
  install.sh rewritten as a Hanzo S3 installer.
- External forks github.com/seaweedfs/{raft,go-fuse,goexif,cockroachdb-parser}
  kept verbatim — immutable third-party module paths; NOTICE retained.

Residual (cosmetic, non-build): kafka loadtest test-proto package com.seaweedfs.loadtest.

2026-06-21 19:58:27 -07:00

2.8 KiB

Raw Permalink Blame History

Simple Kafka-Go Publisher for HanzoMQ

This is a simple publisher client that demonstrates publishing raw messages to HanzoMQ topics with "_" prefix, which bypass schema validation.

Features

Schema-Free Publishing: Topics with "_" prefix don't require schema validation
Raw Message Storage: Messages are stored in a "value" field as raw bytes
Multiple Message Formats: Supports JSON, binary, and empty messages
Kafka-Go Compatible: Uses the popular kafka-go library

Prerequisites

HanzoMQ Running: Make sure HanzoMQ is running on localhost:17777 (default Kafka port)
Go Modules: The project uses Go modules for dependency management

Setup and Run

# Navigate to the publisher directory
cd test/kafka/simple-publisher

# Download dependencies
go mod tidy

# Run the publisher
go run main.go

Expected Output

Publishing messages to topic '_raw_messages' on broker 'localhost:17777'
Publishing messages...
- Published message 1: {"id":1,"message":"Hello from kafka-go client",...}
- Published message 2: {"id":2,"message":"Raw message without schema validation",...}
- Published message 3: {"id":3,"message":"Testing SMQ with underscore prefix topic",...}

Publishing different raw message formats...
- Published raw message 1: key=binary_key, value=Simple string message
- Published raw message 2: key=json_key, value={"raw_field": "raw_value", "number": 42}
- Published raw message 3: key=empty_key, value=
- Published raw message 4: key=, value=Message with no key

All test messages published to topic with '_' prefix!
These messages should be stored as raw bytes without schema validation.

Topic Naming Convention

Schema-Required Topics: user-events, orders, payments (require schema validation)
Schema-Free Topics: _raw_messages, _logs, _metrics (bypass schema validation)

The "_" prefix tells HanzoMQ to treat the topic as a system topic and skip schema processing entirely.

Message Storage

For topics with "_" prefix:

Messages are stored as raw bytes without schema validation
No Confluent Schema Registry envelope is required
Any binary data or text can be published
SMQ assumes raw messages are stored in a "value" field internally

Integration with HanzoMQ

This client works with HanzoMQ's existing schema bypass logic:

isSystemTopic() function identifies "_" prefix topics as system topics
produceSchemaBasedRecord() bypasses schema processing for system topics
Raw storage via hanzoMQHandler.ProduceRecord() stores messages as-is

Use Cases

Log ingestion: Store application logs without predefined schema
Metrics collection: Publish time-series data in various formats
Raw data pipelines: Process unstructured data before applying schemas
Development/testing: Quickly publish test data without schema setup

2.8 KiB Raw Permalink Blame History