mirror of
https://github.com/hanzoai/kafka.git
synced 2026-10-30 14:48:25 +00:00
zeekayandHanzo Dev
b08db2f70a
Serve is now Connect (dial the bus, ensure the offset bucket, watch tails, bound partitions, start admin) followed by a TCP listener and Accept on it, with the same behaviour as before. A host that owns the Kafka port itself calls Connect, which fails as soon as the bus refuses rather than after a startup timer, and then Accept on a unix socket it relays to. Metadata and FindCoordinator keep advertising Config.BrokerHost:BrokerPort, never the listener's own address. Accept closes a listener that a Shutdown before it never saw, and returns the error when its listener is closed by anything but Shutdown instead of retrying a dead socket. Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Kafka
Kafka wire protocol adapter for Hanzo PubSub (NATS JetStream). Accepts standard Kafka clients and translates requests to NATS underneath, allowing any Kafka producer/consumer to work against Hanzo infrastructure without code changes.
Features
- Topic creation, produce, and consume via the Kafka binary protocol
- Cluster mode with Raft consensus and Serf membership
- Compression support
- Multiple log segments per partition
- Compatible with standard Kafka CLI tools and client libraries
Quick Start
Start a single-node broker on port 9092:
go run main.go --bootstrap --node-id 1
Test with any Kafka client:
# Create topic
kafka-topics.sh --create --topic events --bootstrap-server localhost:9092
# Produce
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events
# Consume
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events --from-beginning
Cluster Mode
Cluster mode uses Raft for distributed state and HashiCorp Serf for membership management.
# Bootstrap first node
go run main.go --bootstrap --node-id 1 --serf-addr 127.0.0.1:3331
# Join additional nodes
go run main.go --node-id 2 --broker-port 9093 --raft-addr localhost:2222 --serf-addr 127.0.0.1:3332 --serf-join "127.0.0.1:3331"
go run main.go --node-id 3 --broker-port 9094 --raft-addr localhost:2223 --serf-addr 127.0.0.1:3333 --serf-join "127.0.0.1:3331"
Kubernetes
In Hanzo infrastructure, Stream runs as a deployment in the hanzo namespace and connects to NATS via --pubsub-url nats://pubsub.hanzo.svc:4222.
Two instances are typically deployed:
| Deployment | Service | Purpose |
|---|---|---|
insights-kafka |
insights-kafka:9092 |
Dedicated to Insights pipeline |
stream |
stream:9092 |
General purpose |
Running Tests
Tests use Kafka CLI tools for end-to-end verification:
export KAFKA_BIN_DIR=/path/to/kafka_2.13-3.9.0/bin
go test -v ./...
Credits
License
MIT