Files
skills/public/hanzo-pubsub.md

7.6 KiB

Hanzo PubSub - Event Streaming and Message Queue

Category: Hanzo Ecosystem Related Skills: hanzo/hanzo-storage.md, hanzo/hanzo-platform.md, hanzo/hanzo-universe.md

Overview

Hanzo PubSub is a high-performance messaging system for pub/sub, persistent streams, and exactly-once delivery. It is a fork of NATS Server with Hanzo branding and a bundled Kafka-compatible Redpanda sidecar. Written in Go. Provides the messaging backbone for Hanzo infrastructure.

Why Hanzo PubSub?

  • Pub/Sub Messaging: Subject-based routing with wildcard subscriptions
  • JetStream: Persistent streams with configurable retention and replay
  • Consumer Groups: Scalable consumption with automatic load balancing
  • Exactly-Once Delivery: Deduplication and acknowledgment semantics
  • Built-in Key-Value Store: Distributed KV on top of JetStream
  • Built-in Object Store: Large object storage via JetStream
  • Kafka Compatible: Bundled Redpanda sidecar speaks Kafka wire protocol
  • Clustering: Horizontal scaling with automatic failover via Raft

Tech Stack

  • Language: Go (module: github.com/nats-io/nats-server/v2)
  • Go Version: 1.26
  • License: Apache 2.0
  • Kafka Sidecar: Redpanda v25.1.9

OSS Base

Fork of NATS Server. Repo: hanzoai/pubsub.

When to use

  • Inter-service messaging in Hanzo infrastructure
  • Durable event streaming with replay capability
  • Work queues with load-balanced consumers
  • Real-time notifications and event-driven architectures
  • Kafka-compatible streaming (via Redpanda sidecar)
  • Lightweight distributed key-value or object storage

Hard requirements

  1. Go 1.26+ to build from source
  2. Docker for container deployment
  3. Persistent volume for JetStream data durability

Quick reference

Item Value
Repo github.com/hanzoai/pubsub
Module github.com/nats-io/nats-server/v2 (upstream path retained)
Go Version 1.26
License Apache 2.0
Client Port 4222
Monitoring Port 8222
Cluster Port 6222
WebSocket Port 5222
Kafka Port 9092 (Redpanda sidecar)
Docker Image hanzoai/pubsub:latest
Binary pubsub (built from main.go)
Config /pubsub/conf/server.conf

One-file quickstart

Docker

docker run -d --name hanzo-pubsub \
 -p 4222:4222 \
 -p 8222:8222 \
 hanzoai/pubsub:latest

Docker Compose

# compose.yml
services:
 pubsub:
 image: hanzoai/pubsub:latest
 ports:
 - "4222:4222" # Client connections
 - "8222:8222" # HTTP monitoring
 - "6222:6222" # Cluster routing
 volumes:
 - pubsub-data:/data
 command: ["--jetstream", "--store_dir=/data"]

 # Optional: Kafka-compatible sidecar
 kafka:
 build: kafka/
 ports:
 - "9092:9092" # Kafka protocol
 - "8081:8081" # Schema registry
 - "8082:8082" # HTTP proxy
 - "9644:9644" # Admin API

volumes:
 pubsub-data:

Build from source

git clone https://github.com/hanzoai/pubsub.git
cd pubsub
go build -o pubsub .
./pubsub --jetstream --store_dir /tmp/pubsub-data

Core Concepts

Architecture

┌──────────────────────────────────────────────────┐
│ Cluster │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Node 1 │<-->│ Node 2 │<-->│ Node 3 │ │
│ │ :4222 │ │ :4222 │ │ :4222 │ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │ │ │ │
│ └──────────────┼──────────────┘ │
│ │ │
│ ┌───────┴───────┐ │
│ │ JetStream │ │
│ │ (Streams, │ │
│ │ KV, ObjStore)│ │
│ └───────────────┘ │
└──────────────────────────────────────────────────┘

 ┌───────────────────────┐
 │ Redpanda Sidecar │
 │ (Kafka :9092) │
 │ (Schema Reg :8081) │
 └───────────────────────┘

Server Configuration

Default config at docker/nats-server.conf:

# Client port
port: 4222

# HTTP monitoring
monitor_port: 8222

# Clustering
cluster {
 port: 6222
 authorization {
 user: ruser
 password: T0pS3cr3t
 timeout: 2
 }
 routes = []
}

Port Allocation

Port Protocol Purpose
4222 TCP Client connections (NATS protocol)
8222 HTTP Monitoring and management
6222 TCP Cluster routing between nodes
5222 WebSocket WebSocket client connections
9092 TCP Kafka protocol (Redpanda sidecar)
8081 HTTP Schema registry (Redpanda)
8082 HTTP HTTP proxy (Redpanda)
9644 HTTP Redpanda admin API

Directory Structure

pubsub/
 main.go # Entry point (configures and runs server)
 server/ # Core server implementation
 conf/ # Configuration parser (lexer, parser)
 internal/
 antithesis/ # Deterministic testing
 fastrand/ # Fast random number generation
 ldap/ # LDAP authentication
 ocsp/ # OCSP stapling
 testhelper/ # Test utilities
 kafka/
 Dockerfile # Redpanda (Kafka-compatible) sidecar
 logger/ # Logging
 docker/
 nats-server.conf # Default server configuration
 Dockerfile.nightly # Nightly build
 test/ # Integration tests
 scripts/ # Build and CI scripts

Client SDKs

Language Package
Go github.com/nats-io/nats.go
Python nats-py (pip install nats-py)
TypeScript nats (npm install nats)
Rust nats (cargo add nats)

All NATS clients work directly with Hanzo PubSub (wire-compatible).

Usage Examples

Basic Pub/Sub (Go):

nc, _ := nats.Connect("nats://localhost:4222")
defer nc.Close()

// Subscribe
nc.Subscribe("events.>", func(msg *nats.Msg) {
 fmt.Printf("Received: %s\n", string(msg.Data))
})

// Publish
nc.Publish("events.user.created", []byte(`{"user_id":"123"}`))

JetStream Persistent Streams (Go):

nc, _ := nats.Connect("nats://localhost:4222")
js, _ := nc.JetStream()

// Create stream
js.AddStream(&nats.StreamConfig{
 Name: "ORDERS",
 Subjects: []string{"orders.*"},
})

// Publish
js.Publish("orders.new", []byte(`{"order_id":"abc123"}`))

// Durable consumer
sub, _ := js.PullSubscribe("orders.*", "processor")
msgs, _ := sub.Fetch(10)
for _, msg := range msgs {
 msg.Ack()
}

Performance

  • Throughput: 10M+ messages/second
  • Latency: Sub-millisecond publish latency
  • Connections: 100K+ concurrent connections per node

Troubleshooting

Issue Cause Solution
JetStream not enabled Missing flag Add --jetstream to command
Cluster routes not connecting Auth mismatch Verify ruser:T0pS3cr3t matches all nodes
Kafka clients can't connect Redpanda not running Start Kafka sidecar container
Data lost on restart No persistent volume Mount volume at --store_dir path
  • hanzo/hanzo-storage.md - S3-compatible object storage (uses PubSub for event notifications)
  • hanzo/hanzo-platform.md - PaaS deployment
  • hanzo/hanzo-universe.md - Production K8s infrastructure
  • hanzo/hanzo-database.md - PostgreSQL database

Last Updated: 2026-03-13 Category: Hanzo Ecosystem Related: pubsub, messaging, nats, kafka, streaming, events Prerequisites: Go 1.26+, Docker