mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
hanzoskills.com/llms.txt now opens with the install line, the API reference index and the company index, so an agent that lands on any one of the three can find the rest. robots.txt claimed 537 documents while llms.txt counts 920. The number is gone from robots.txt; llms.txt derives it. public/ is regenerated from scripts/site.py, which it had fallen behind: the copy-prompt menu and agent bar were in the generator and not in the committed pages. tests/site.mjs passes against the result. Co-authored-by: Hanzo Dev <[email protected]>
311 lines
21 KiB
HTML
311 lines
21 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>hanzo-stream</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="alternate" type="text/markdown" href="/hanzo-stream.md">
|
|
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/hanzo-stream">
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/hanzo-stream.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>hanzo-stream</h1><p class="lede">Hanzo Stream is a stateless Kafka wire protocol gateway that translates standard Kafka client requests into NATS JetStream operations against Hanzo PubSub.</p><h1>Hanzo Stream - Kafka Wire Protocol Gateway</h1>
|
|
<p><strong>Category</strong>: Hanzo Ecosystem <strong>Related Skills</strong>: <code>hanzo/hanzo-pubsub.md</code>, <code>hanzo/hanzo-kv.md</code>, <code>hanzo/hanzo-universe.md</code></p>
|
|
<h2>Overview</h2>
|
|
<p>Hanzo Stream is a <strong>stateless Kafka wire protocol gateway</strong> that translates standard Kafka client requests into NATS JetStream operations against Hanzo PubSub. Go codebase (module <code>github.com/hanzoai/stream</code>), ships as a single <code>hanzo-stream</code> binary. Any Kafka producer, consumer, or CLI tool connects on port 9092 and works without code changes. All storage and replication is delegated to Hanzo PubSub (NATS JetStream). License: MIT.</p>
|
|
<h3>Why Hanzo Stream?</h3>
|
|
<ul><li><strong>Zero code changes</strong>: Standard Kafka clients work out of the box</li><li><strong>Stateless</strong>: No local storage, no Raft, no Serf -- pure protocol translation</li><li><strong>Hanzo PubSub backed</strong>: All durability and replication via NATS JetStream</li><li><strong>Compression</strong>: GZIP, Snappy, LZ4, ZSTD codec support</li><li><strong>Lightweight</strong>: Single Go binary, ~140KB source, minimal dependencies</li></ul>
|
|
<h3>Tech Stack</h3>
|
|
<ul><li><strong>Language</strong>: Go 1.26</li><li><strong>CLI</strong>: spf13/cobra</li><li><strong>Backend</strong>: nats-io/nats.go (NATS JetStream client)</li><li><strong>Compression</strong>: klauspost/compress (zstd), pierrec/lz4, eapache/go-xerial-snappy</li><li><strong>Image</strong>: <code>ghcr.io/hanzoai/stream</code> (alpine base)</li><li><strong>CI</strong>: GitHub Actions (push to GHCR on main)</li></ul>
|
|
<h3>OSS Base</h3>
|
|
<p>Repo: <code>hanzoai/stream</code>. Default branch: <code>main</code>. Based on <a href="https://github.com/cefboud/monkafka">MonKafka</a>.</p>
|
|
<h2>When to use</h2>
|
|
<ul><li>Kafka clients need to produce/consume against Hanzo PubSub without migration</li><li>Bridging existing Kafka-based pipelines (analytics, logs) to NATS JetStream</li><li>Insights pipeline ingestion via Kafka protocol</li><li>Any scenario where Kafka wire compatibility is needed but NATS is the backend</li></ul>
|
|
<h2>Hard requirements</h2>
|
|
<ol><li><strong>Hanzo PubSub</strong> (NATS with JetStream enabled) running and accessible</li><li><strong>Port 9092</strong> available for Kafka listener</li><li><strong>Port 9093</strong> available for admin HTTP endpoint (optional, disable with <code>--admin-port 0</code>)</li></ol>
|
|
<h2>Quick reference</h2>
|
|
<p>| Item | Value | |------|-------| | Kafka Port | 9092 | | Admin Port | 9093 | | Go Module | <code>github.com/hanzoai/stream</code> | | Go Version | 1.26 | | Binary | <code>hanzo-stream</code> | | Image | <code>ghcr.io/hanzoai/stream</code> | | PubSub Default | <code>nats://localhost:4222</code> | | License | MIT | | Repo | <code>github.com/hanzoai/stream</code> | | Default Branch | <code>main</code> |</p>
|
|
<h2>One-file quickstart</h2>
|
|
<h3>Run locally</h3>
|
|
<pre><code class="lang-bash"># Start NATS with JetStream
|
|
nats-server --jetstream
|
|
|
|
# Start Hanzo Stream
|
|
go run main.go --pubsub-url nats://localhost:4222 --port 9092
|
|
|
|
# Use standard Kafka CLI tools
|
|
kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
|
|
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
|
|
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning</code></pre>
|
|
<h3>Docker</h3>
|
|
<pre><code class="lang-bash">docker run -d --name hanzo-stream \
|
|
-p 9092:9092 -p 9093:9093 \
|
|
ghcr.io/hanzoai/stream:latest \
|
|
--pubsub-url nats://pubsub:4222 --host 0.0.0.0</code></pre>
|
|
<h2>Core Concepts</h2>
|
|
<h3>Architecture</h3>
|
|
<pre><code>Kafka Client ──TCP :9092──> Hanzo Stream (protocol translation) ──> Hanzo PubSub (NATS JetStream)</code></pre>
|
|
<p><strong>Stateless gateway</strong>: All state lives in Hanzo PubSub. Multiple Stream instances share the same PubSub cluster.</p>
|
|
<h3>Kafka-to-PubSub Mapping</h3>
|
|
<p>| Kafka Concept | PubSub Equivalent | |---|---| | Topic <code>foo</code>, Partition N | Stream <code>kafka-foo-N</code>, Subject <code>kafka.foo.N</code> | | Produce | <code>Publish("kafka.foo.0", recordBatchBytes)</code> -- seq = offset+1 | | Fetch at offset | <code>GetMsg(streamName, offset+1)</code> (PubSub 1-based, Kafka 0-based) | | Consumer group offsets | KV bucket <code>kafka-consumer-offsets</code>, key <code>{group}.{topic}.{partition}</code> | | Create topic (N parts) | N calls to <code>AddStream()</code> | | Metadata | <code>StreamInfo()</code> per partition stream |</p>
|
|
<h3>Offset Translation (Critical)</h3>
|
|
<pre><code>Kafka offset 0 <-> PubSub sequence 1
|
|
Kafka offset N <-> PubSub sequence N+1
|
|
Produce: seq = Publish(); return seq - 1
|
|
Fetch: msg = GetMsg(offset + 1)</code></pre>
|
|
<h3>Supported Kafka API Keys</h3>
|
|
<p>| API Key | Name | Implementation | |---------|------|----------------| | 0 | Produce | Hand-written decoder (performance) | | 1 | Fetch | Hand-written decoder (performance) | | 3 | Metadata | Reflection-based serde | | 8 | OffsetCommit | KV-backed | | 9 | OffsetFetch | KV-backed | | 10 | FindCoordinator | Static response | | 11 | JoinGroup | Static response | | 18 | ApiVersions | Supported versions list | | 19 | CreateTopics | Creates JetStream streams | | 32 | DescribeConfigs | Static response |</p>
|
|
<h3>CLI Flags</h3>
|
|
<p>| Flag | Default | Description | |------|---------|-------------| | <code>--pubsub-url</code> | <code>nats://localhost:4222</code> | Hanzo PubSub server URL | | <code>--pubsub-creds</code> | - | PubSub credentials file | | <code>--port</code> | 9092 | Kafka listener port | | <code>--admin-port</code> | 9093 | Admin HTTP port (0 to disable) | | <code>--host</code> | localhost | Advertised hostname | | <code>--node-id</code> | 1 | Broker node ID | | <code>--replicas</code> | 1 | JetStream stream replica count | | <code>--storage</code> | file | JetStream storage type: file or memory |</p>
|
|
<h3>Directory Structure</h3>
|
|
<pre><code>stream/
|
|
main.go # CLI entry point (cobra)
|
|
Dockerfile # Multi-stage build (golang:1.26-alpine -> alpine:3.20)
|
|
go.mod # github.com/hanzoai/stream
|
|
pubsub/ # NATS JetStream client wrapper
|
|
client.go # Connection + stream context
|
|
streams.go # Stream CRUD, publish, get message, list topics
|
|
consumer.go # KV-based consumer offset management
|
|
protocol/ # Kafka wire protocol handlers
|
|
broker.go # TCP server, connection handling
|
|
dispatcher.go # API key -> handler routing
|
|
produce.go # Produce (API key 0)
|
|
fetch.go # Fetch (API key 1)
|
|
metadata.go # Metadata (API key 3)
|
|
create_topic.go # CreateTopics (API key 19)
|
|
responses.go # ListOffsets, OffsetCommit/Fetch, JoinGroup, etc.
|
|
find_coordinator.go
|
|
describe_configs.go
|
|
api_versions.go
|
|
admin.go # Admin HTTP endpoints
|
|
types.go # Request/response struct definitions
|
|
recordbatch.go # Kafka record batch parsing
|
|
error.go # Kafka error codes
|
|
serde/ # Kafka protocol serialization (reflection-based)
|
|
compress/ # GZIP, Snappy, LZ4, ZSTD codecs
|
|
logging/ # Simple log levels
|
|
utils/ # Time utilities
|
|
types/ # Shared types (Config, Request, Record, RecordBatch)
|
|
test/ # E2E and cluster tests
|
|
.github/workflows/ # CI (GHCR push)</code></pre>
|
|
<h3>Deployment (hanzo-k8s)</h3>
|
|
<p>| Deployment | Service | Purpose | |-----------|---------|---------| | <code>insights-kafka</code> | <code>insights-kafka:9092</code> | Dedicated to Insights pipeline | | <code>stream</code> | <code>stream:9092</code> | General purpose |</p>
|
|
<p>Both connect to <code>pubsub.hanzo.svc:4222</code> (NATS with JetStream, 20Gi PVC).</p>
|
|
<h3>Testing</h3>
|
|
<pre><code class="lang-bash"># Requires Kafka CLI tools
|
|
export KAFKA_BIN_DIR=/path/to/kafka_2.13-3.9.0/bin
|
|
go test -v ./...</code></pre>
|
|
<ul><li><code>test/e2e/</code> -- E2E tests using Kafka CLI binaries</li><li><code>test/cluster/</code> -- Multi-instance tests (two gateways sharing same PubSub)</li></ul>
|
|
<h2>Troubleshooting</h2>
|
|
<p>| Issue | Cause | Solution | |-------|-------|----------| | Connection refused on 9092 | Stream not running or wrong host | Check <code>hanzo-stream</code> process, verify <code>--host</code> matches advertised address | | Produce errors | PubSub not reachable | Verify <code>--pubsub-url</code> connectivity, check NATS is running with <code>--jetstream</code> | | Offset mismatch | Off-by-one in custom client | Remember: Kafka offset N = PubSub sequence N+1 | | Consumer group errors | Limited group support | Only basic offset commit/fetch; no full rebalancing protocol | | Topic not found | JetStream stream not created | Create topic first via <code>kafka-topics.sh --create</code> |</p>
|
|
<h2>Related Skills</h2>
|
|
<ul><li><code>hanzo/hanzo-pubsub.md</code> - NATS JetStream (underlying message broker)</li><li><code>hanzo/hanzo-kv.md</code> - Valkey/Redis (sibling infrastructure service)</li><li><code>hanzo/hanzo-universe.md</code> - K8s infrastructure where Stream runs</li><li><code>hanzo/hanzo-insights.md</code> - Analytics pipeline (primary consumer)</li></ul>
|
|
<hr>
|
|
<p><strong>Last Updated</strong>: 2026-03-13 <strong>Category</strong>: Hanzo Ecosystem <strong>Related</strong>: kafka, nats, jetstream, streaming, event-driven, pubsub, message-broker <strong>Prerequisites</strong>: NATS with JetStream, Go 1.26+ or Docker</p>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/hanzo-stream.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |