Files
skills/public/hip-0030.html

460 lines
36 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>hip-0030</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/hip-0030.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/hip-0030">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/hip-0030.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>hip-0030</h1><p class="lede">HIP-30: Event Streaming Standard. Status Draft. Hanzo&#x27;s own standard — read this before implementing against it.</p><h1>HIP-0030: Event Streaming Standard</h1>
<h2>Abstract</h2>
<p>This proposal defines the event streaming standard for the Hanzo ecosystem. Hanzo Stream provides distributed event streaming based on Apache Kafka, serving as the backbone for real-time event ingestion, analytics pipelines, inter-service communication, and billing aggregation. All services producing or consuming asynchronous events MUST use Hanzo Stream as specified in this document.</p>
<p><strong>Repository</strong>: <a href="https://github.com/hanzoai/stream">github.com/hanzoai/stream</a> <strong>Protocol</strong>: Kafka wire protocol (TCP 9092, TLS 9093) <strong>Production</strong>: <code>insights-kafka</code> on <code>the cluster</code> cluster (<code>24.199.76.156</code>)</p>
<h2>Design Philosophy</h2>
<p>This section explains the architectural decisions behind Hanzo Stream. Understanding the <em>why</em> is essential for making correct integration choices.</p>
<h3>Why KRaft Over ZooKeeper</h3>
<p><strong>KRaft</strong> (Kafka Raft) is Kafka&#x27;s built-in consensus protocol, available since Kafka 3.3 and production-ready since Kafka 3.6. It replaces ZooKeeper for metadata management (broker registration, topic configuration, partition leadership).</p>
<p><strong>Before KRaft</strong>, running Kafka meant running two distributed systems: Kafka + ZooKeeper. ZooKeeper was a separate failure domain (session timeouts, snapshot corruption). Most Kafka outages trace back to ZooKeeper.</p>
<p><strong>With KRaft</strong>, Kafka manages its own metadata. Single StatefulSet, no external dependencies. ZooKeeper is deprecated and will be removed in Kafka 4.0.</p>
<h3>How Stream Connects to the Data Pipeline</h3>
<p>The following diagram shows how Hanzo Stream fits into the broader data flow:</p>
<pre><code> ┌──────────────────┐
│ Hanzo Datastore │
│ (Analytics DB) │
└────────▲─────────┘
│ consume
│
┌─────────────┐ ┌──────────────┐ ┌─────────┴─────────┐ ┌──────────────┐
│ Insights │───→│ │───→│ Consumer Groups │ │ Billing │
│ Capture │ │ │ │ │ │ Aggregator │
│ (Rust) │ │ Hanzo │ │ analytics-ingest │ └──────▲───────┘
└─────────────┘ │ Stream │ │ billing-agg │ │
│ (Kafka) │ │ audit-archive │ │ consume
┌─────────────┐ │ │ │ llm-metrics │ ┌──────┴───────┐
│ LLM Gateway │───→│ Topics: │ └───────────────────┘ │ Consumer │
│ (HIP-4) │ │ events_* │ │ Group: │
└─────────────┘ │ llm_usage │ │ billing-agg │
│ billing_* │ └──────────────┘
┌─────────────┐ │ audit_log │
│ IAM │───→│ │
│ (hanzo.id) │ └──────────────┘
└─────────────┘</code></pre>
<p><strong>Data flow summary</strong>:</p>
<ol><li>Producers (Insights capture, LLM Gateway, IAM) write events to Kafka topics</li><li>Kafka retains events in the log for the configured retention period</li><li>Consumer groups read events independently and at their own pace</li><li>Hanzo Datastore ingestion consumer writes analytics events to Hanzo Datastore tables</li><li>Billing aggregator consumes <code>llm_usage</code> events and produces <code>billing_events</code></li><li>Audit archiver consumes <code>audit_log</code> and writes to long-term storage</li></ol>
<h2>Specification</h2>
<h3>Topic Registry</h3>
<p>All Kafka topics MUST be registered in this section. Ad-hoc topic creation is prohibited in production.</p>
<p>| Topic | Partitions | Retention | Key | Producers | Consumers | |-------|-----------|-----------|-----|-----------|-----------| | <code>events_plugin_ingestion</code> | 16 | 7 days | <code>team_id</code> | Insights Capture | Hanzo Datastore Ingestion | | <code>llm_usage</code> | 8 | 14 days | <code>team_id</code> | LLM Gateway | Billing Aggregator, Analytics | | <code>billing_events</code> | 4 | 30 days | <code>org_id</code> | Billing Aggregator | Commerce, Reporting | | <code>audit_log</code> | 4 | 90 days | <code>org_id</code> | IAM, all services | Audit Archiver, Compliance | | <code>agent_rpc_metering</code> | 8 | 7 days | <code>agent_id</code> | Bot Gateway | RPC Billing (HIP-25) | | <code>dead_letter</code> | 1 | 30 days | <code>source_topic</code> | All consumers | Ops team (manual) |</p>
<h3>Topic Naming Convention</h3>
<pre><code>&lt;domain&gt;_&lt;entity&gt;_&lt;action&gt;</code></pre>
<p>Examples: <code>events_plugin_ingestion</code>, <code>llm_usage</code>, <code>billing_events</code>, <code>audit_log</code>.</p>
<p>Topics MUST use snake_case. Topics MUST NOT include environment prefixes (e.g., <code>prod_</code> or <code>stg_</code>). Environment isolation is achieved through separate Kafka clusters, not topic naming.</p>
<h3>Partitioning Strategy</h3>
<p>Partitioning determines parallelism and ordering. The key design principle: <strong>events that must be processed in order MUST share a partition key.</strong></p>
<p>| Topic | Partition Key | Rationale | |-------|--------------|-----------| | <code>events_plugin_ingestion</code> | <code>team_id</code> | All events for a team arrive in order; Hanzo Datastore ingests per-team batches | | <code>llm_usage</code> | <code>team_id</code> | Billing must see all usage for a team in order to compute running totals | | <code>billing_events</code> | <code>org_id</code> | Organization-level billing aggregation | | <code>audit_log</code> | <code>org_id</code> | Audit trail must be ordered per organization | | <code>agent_rpc_metering</code> | <code>agent_id</code> | Per-agent metering must be ordered for accurate tallying |</p>
<p><strong>Partition count guidelines</strong>:</p>
<ul><li>Start with <code>max(expected_consumer_count * 2, 4)</code></li><li>Never reduce partition count (Kafka does not support this)</li><li>Increase partitions only when consumer lag consistently exceeds SLA</li></ul>
<h3>Event Schema</h3>
<p>All events MUST conform to a base envelope. The envelope wraps topic-specific payloads.</p>
<h4>Base Envelope</h4>
<pre><code class="lang-json">{
&quot;schema&quot;: &quot;hanzo.stream.v1&quot;,
&quot;id&quot;: &quot;evt_01HQ3X7K8M2N4P5R6S7T8U9V0W&quot;,
&quot;type&quot;: &quot;llm.usage.completed&quot;,
&quot;source&quot;: &quot;llm-gateway&quot;,
&quot;timestamp&quot;: &quot;2025-01-15T10:30:00.000Z&quot;,
&quot;team_id&quot;: &quot;team_abc123&quot;,
&quot;org_id&quot;: &quot;org_hanzo&quot;,
&quot;data&quot;: { }
}</code></pre>
<p>| Field | Type | Required | Description | |-------|------|----------|-------------| | <code>schema</code> | string | yes | Envelope version. Always <code>hanzo.stream.v1</code> for this spec. | | <code>id</code> | string | yes | Globally unique event ID. ULID or UUID v7 recommended. | | <code>type</code> | string | yes | Dot-delimited event type (e.g., <code>llm.usage.completed</code>). | | <code>source</code> | string | yes | Producing service identifier. | | <code>timestamp</code> | string | yes | ISO 8601 UTC timestamp with millisecond precision. | | <code>team_id</code> | string | no | Team identifier (used as partition key for most topics). | | <code>org_id</code> | string | no | Organization identifier. | | <code>data</code> | object | yes | Topic-specific payload. Schema defined per event type. |</p>
<h4>LLM Usage Event (<code>llm_usage</code>)</h4>
<pre><code class="lang-json">{
&quot;schema&quot;: &quot;hanzo.stream.v1&quot;,
&quot;id&quot;: &quot;evt_01HQ3X7K8M2N4P5R6S7T8U9V0W&quot;,
&quot;type&quot;: &quot;llm.usage.completed&quot;,
&quot;source&quot;: &quot;llm-gateway&quot;,
&quot;timestamp&quot;: &quot;2025-01-15T10:30:00.123Z&quot;,
&quot;team_id&quot;: &quot;team_abc123&quot;,
&quot;org_id&quot;: &quot;org_hanzo&quot;,
&quot;data&quot;: {
&quot;request_id&quot;: &quot;req_xyz789&quot;,
&quot;model&quot;: &quot;zen-72b&quot;,
&quot;provider&quot;: &quot;together&quot;,
&quot;prompt_tokens&quot;: 150,
&quot;completion_tokens&quot;: 230,
&quot;total_tokens&quot;: 380,
&quot;latency_ms&quot;: 1250,
&quot;cost_usd&quot;: 0.0038,
&quot;user_id&quot;: &quot;user_456&quot;,
&quot;api_key_hash&quot;: &quot;sha256:abc...&quot;,
&quot;cache_hit&quot;: false,
&quot;stream&quot;: true
}
}</code></pre>
<h4>Insights Plugin Ingestion Event (<code>events_plugin_ingestion</code>)</h4>
<pre><code class="lang-json">{
&quot;schema&quot;: &quot;hanzo.stream.v1&quot;,
&quot;id&quot;: &quot;evt_01HQ4Y8L9N3O5Q6R7S8T9U0V1X&quot;,
&quot;type&quot;: &quot;insights.event.captured&quot;,
&quot;source&quot;: &quot;insights-capture&quot;,
&quot;timestamp&quot;: &quot;2025-01-15T10:31:00.456Z&quot;,
&quot;team_id&quot;: &quot;team_abc123&quot;,
&quot;data&quot;: {
&quot;event&quot;: &quot;$pageview&quot;,
&quot;distinct_id&quot;: &quot;user_456&quot;,
&quot;properties&quot;: {
&quot;$current_url&quot;: &quot;https://hanzo.ai/dashboard&quot;,
&quot;$browser&quot;: &quot;Chrome&quot;,
&quot;$os&quot;: &quot;macOS&quot;
},
&quot;ip&quot;: &quot;203.0.113.42&quot;,
&quot;session_id&quot;: &quot;sess_abc123&quot;,
&quot;sent_at&quot;: &quot;2025-01-15T10:30:59.800Z&quot;
}
}</code></pre>
<h4>Audit Log Event (<code>audit_log</code>)</h4>
<pre><code class="lang-json">{
&quot;schema&quot;: &quot;hanzo.stream.v1&quot;,
&quot;id&quot;: &quot;evt_01HQ5Z9M0O4P6R7S8T9U0V1W2Y&quot;,
&quot;type&quot;: &quot;iam.auth.login_success&quot;,
&quot;source&quot;: &quot;iam&quot;,
&quot;timestamp&quot;: &quot;2025-01-15T10:32:00.789Z&quot;,
&quot;org_id&quot;: &quot;org_hanzo&quot;,
&quot;data&quot;: {
&quot;user_id&quot;: &quot;user_456&quot;,
&quot;email&quot;: &quot;[email protected]&quot;,
&quot;method&quot;: &quot;oauth2&quot;,
&quot;provider&quot;: &quot;github&quot;,
&quot;ip&quot;: &quot;203.0.113.42&quot;,
&quot;user_agent&quot;: &quot;Mozilla/5.0...&quot;,
&quot;application&quot;: &quot;app-hanzo&quot;
}
}</code></pre>
<h3>Schema Registry</h3>
<p>Event schemas MUST be registered in a schema registry to enable:</p>
<ul><li><strong>Schema evolution</strong>: Add fields without breaking consumers</li><li><strong>Validation</strong>: Reject malformed events at the producer</li><li><strong>Documentation</strong>: Machine-readable schema catalog</li></ul>
<h4>Registry Configuration</h4>
<pre><code class="lang-yaml">schema_registry:
type: json-schema # JSON Schema (not Avro - see rationale below)
url: http://localhost:8081
compatibility: BACKWARD # New schemas must be readable by old consumers</code></pre>
<p><strong>Why JSON Schema over Avro</strong>: Our producers and consumers span multiple languages (Rust capture service, Python billing, Go IAM, TypeScript gateway). JSON Schema is language-agnostic and requires no code generation. Avro has better compression but adds build-time complexity for schema compilation. At our event sizes (&lt; 2KB average), the compression difference is negligible.</p>
<h4>Compatibility Rules</h4>
<ul><li><strong>BACKWARD</strong> compatibility is the default: new schema versions can read data written by the previous version</li><li>Fields MAY be added with defaults</li><li>Fields MUST NOT be removed or renamed (mark as deprecated instead)</li><li>Field types MUST NOT change</li></ul>
<h3>Consumer Groups</h3>
<p>Consumer groups enable parallel processing. Each consumer in a group reads from a disjoint set of partitions.</p>
<p>| Consumer Group | Topic(s) | Consumers | Processing | |---------------|----------|-----------|------------| | <code>analytics-ingest</code> | <code>events_plugin_ingestion</code> | 4 | Write to Hanzo Datastore <code>events</code> table | | <code>billing-agg</code> | <code>llm_usage</code> | 2 | Aggregate usage per team per hour, produce <code>billing_events</code> | | <code>audit-archive</code> | <code>audit_log</code> | 1 | Write to S3-compatible long-term storage | | <code>llm-metrics</code> | <code>llm_usage</code> | 2 | Compute real-time latency/throughput metrics for Prometheus | | <code>rpc-billing</code> | <code>agent_rpc_metering</code> | 2 | Aggregate agent RPC usage for settlement (HIP-25) | | <code>dead-letter-monitor</code> | <code>dead_letter</code> | 1 | Alert on failed events, PagerDuty integration |</p>
<h4>Consumer Group Naming Convention</h4>
<pre><code>&lt;function&gt;-&lt;action&gt;</code></pre>
<p>Examples: <code>analytics-ingest</code>, <code>billing-agg</code>, <code>audit-archive</code>. Kebab-case. MUST NOT include service version or instance identifiers.</p>
<h3>Retention Policies</h3>
<p>| Category | Retention | Rationale | |----------|-----------|-----------| | Analytics events | 7 days | Sufficient for reprocessing; Hanzo Datastore is the durable store | | LLM usage | 14 days | Billing reconciliation window is 14 days | | Billing events | 30 days | Monthly billing cycle + buffer | | Audit log | 90 days | Compliance requirement; also archived to S3 | | Dead letter | 30 days | Ops investigation window |</p>
<p>Retention is configured per topic via <code>retention.ms</code>. After retention expires, segments are deleted. There is no compaction (log compaction is for changelog-style topics, not event streams).</p>
<h3>Producer Configuration</h3>
<pre><code class="lang-yaml">producer:
acks: all # Wait for all in-sync replicas (durability)
retries: 3 # Retry transient failures
retry_backoff_ms: 100 # Backoff between retries
max_in_flight: 5 # Max unacknowledged batches
compression: lz4 # LZ4 for speed; zstd for better ratio if CPU-bound
batch_size: 16384 # 16KB batch size
linger_ms: 5 # Wait up to 5ms to fill batch
idempotence: true # Exactly-once per partition (requires acks=all)</code></pre>
<p><strong>Why <code>acks=all</code></strong>: We cannot lose billing or audit events. The latency cost of waiting for replica acknowledgment (typically &lt; 10ms intra-cluster) is acceptable.</p>
<p><strong>Why LZ4 compression</strong>: LZ4 provides ~2x compression at near-zero CPU cost. For analytics events with repetitive JSON keys, this cuts network and storage usage in half.</p>
<h3>Insights Capture Integration</h3>
<p>The Insights Rust capture service is the highest-throughput producer. It receives HTTP POST requests from frontend SDKs and writes batches to Kafka.</p>
<pre><code>Browser SDK ──HTTP POST──→ Capture (Rust) ──Kafka Produce──→ events_plugin_ingestion
Port 3000 ↓
Batch: 500 events Hanzo Datastore consumer
Flush: 1 second ↓
events table</code></pre>
<p>The capture service MUST:</p>
<ul><li>Batch events (up to 500 or 1 second, whichever comes first)</li><li>Produce with <code>acks=all</code> and idempotence enabled</li><li>Return HTTP 200 immediately after Kafka acknowledgment (not after Hanzo Datastore write)</li><li>Write to <code>dead_letter</code> topic on serialization or validation failure</li></ul>
<h2>Implementation</h2>
<h3>Production Deployment</h3>
<p>Deployed as a Kubernetes StatefulSet on <code>the cluster</code> using <code>bitnami/kafka:3.7</code> in KRaft combined mode (broker + controller in one process).</p>
<pre><code class="lang-yaml"># Key configuration (insights-kafka StatefulSet, namespace: hanzo)
image: bitnami/kafka:3.7
replicas: 1 # Single broker, partition-ready for scale
ports: [9092 (PLAINTEXT), 9093 (TLS), 9094 (CONTROLLER)]
env:
KAFKA_CFG_PROCESS_ROLES: &quot;broker,controller&quot;
KAFKA_CFG_NODE_ID: &quot;0&quot;
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: &quot;0@localhost:9094&quot;
KAFKA_CFG_LISTENERS: &quot;PLAINTEXT://:9092,CONTROLLER://:9094&quot;
KAFKA_CFG_LOG_RETENTION_HOURS: &quot;168&quot; # 7 days default
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: &quot;false&quot; # Explicit topic creation only
resources:
requests: { memory: 1Gi, cpu: 500m }
limits: { memory: 2Gi, cpu: 1000m }
storage: 50Gi (do-block-storage PVC)</code></pre>
<h3>Topic Provisioning</h3>
<p>Topics MUST be created explicitly. Auto-creation is disabled in production.</p>
<pre><code class="lang-bash">BROKER=localhost:9092
# Each topic: --partitions N --replication-factor 1 --config retention.ms=MS
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic events_plugin_ingestion --partitions 16 --config retention.ms=604800000
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic llm_usage --partitions 8 --config retention.ms=1209600000
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic billing_events --partitions 4 --config retention.ms=2592000000
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic audit_log --partitions 4 --config retention.ms=7776000000
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic agent_rpc_metering --partitions 8 --config retention.ms=604800000
kafka-topics.sh --bootstrap-server $BROKER --create \
--topic dead_letter --partitions 1 --config retention.ms=2592000000</code></pre>
<h3>Scaling Path</h3>
<p>The current deployment is a single broker. This is sufficient for our current throughput (&lt; 50 MB/s). The scaling path is:</p>
<ol><li><strong>Single broker</strong> (current): All topics on one node. Replication factor 1. Acceptable for non-critical analytics; billing events are also persisted in SQL.</li><li><strong>Three brokers</strong>: Replication factor 3. Automatic failover. Required when billing events become the sole source of truth.</li><li><strong>Multi-rack</strong>: Spread brokers across availability zones. Required for 99.99% uptime SLA.</li></ol>
<p>Scaling from 1 to 3 brokers requires:</p>
<ul><li>Update StatefulSet <code>replicas: 3</code></li><li>Update <code>KAFKA_CFG_CONTROLLER_QUORUM_VOTERS</code> to include all three nodes</li><li>Reassign partitions with <code>kafka-reassign-partitions.sh</code></li><li>Increase replication factor for existing topics</li></ul>
<h3>Monitoring</h3>
<p>Prometheus metrics are exposed via JMX Exporter. The critical metrics:</p>
<p>| Metric | Alert Threshold | Meaning | |--------|----------------|---------| | <code>kafka_messages_in_per_sec</code> | - | Throughput: messages received per second | | <code>kafka_bytes_in_per_sec</code> | - | Throughput: bytes received per second | | <code>kafka_under_replicated_partitions</code> | &gt; 0 for 5m | Partitions without enough replicas | | <code>kafka_consumer_group_lag</code> | &gt; 100K warn, &gt; 1M critical | Events a consumer is behind |</p>
<p><strong>Consumer lag</strong> is the single most important Kafka metric. It measures how far behind a consumer group is from the latest event. If lag grows continuously, the consumer cannot keep up and events will expire before being processed.</p>
<h2>Security</h2>
<h3>Authentication</h3>
<p>SASL/PLAIN authentication. Each service gets a dedicated credential with scoped topic access. Passwords are injected from KMS via Kubernetes secrets.</p>
<pre><code class="lang-yaml">sasl:
mechanism: PLAIN
credentials:
- { username: insights-capture, topics: [events_plugin_ingestion], ops: [WRITE] }
- { username: llm-gateway, topics: [llm_usage], ops: [WRITE] }
- { username: iam-service, topics: [audit_log], ops: [WRITE] }
- { username: analytics-ingest, topics: [events_plugin_ingestion], ops: [READ] }
- { username: billing-aggregator, topics: [llm_usage, billing_events], ops: [READ, WRITE] }</code></pre>
<h3>Access Control Lists (ACLs)</h3>
<p>Each producer and consumer MUST have minimal permissions:</p>
<p>| Principal | Topic | Operations | Rationale | |-----------|-------|------------|-----------| | <code>insights-capture</code> | <code>events_plugin_ingestion</code> | WRITE | Capture only produces | | <code>llm-gateway</code> | <code>llm_usage</code> | WRITE | Gateway only produces | | <code>analytics-ingest</code> | <code>events_plugin_ingestion</code> | READ | Hanzo Datastore consumer | | <code>billing-aggregator</code> | <code>llm_usage</code> | READ | Reads usage | | <code>billing-aggregator</code> | <code>billing_events</code> | WRITE | Produces billing | | <code>audit-archiver</code> | <code>audit_log</code> | READ | Archives to S3 |</p>
<p><strong>No service gets blanket access to all topics.</strong> The <code>dead_letter</code> topic is writable by all authenticated producers (events that fail processing are redirected here).</p>
<h3>Encryption</h3>
<ul><li><strong>In transit</strong>: TLS 1.3 for all client-to-broker and inter-broker communication on port 9093</li><li><strong>At rest</strong>: Disk encryption via DigitalOcean block storage encryption (transparent)</li><li><strong>Secrets</strong>: Kafka credentials stored in KMS (HIP-5) and injected via Kubernetes secrets</li></ul>
<h3>Network Policies</h3>
<p>Kafka is accessible only from within the <code>hanzo</code> namespace via Kubernetes NetworkPolicy. No external ingress. Services outside the cluster MUST use the LLM Gateway or Insights capture HTTP endpoints, which produce to Kafka internally.</p>
<h2>Operational Runbook</h2>
<h3>Consumer Lag Resolution</h3>
<pre><code class="lang-bash"># Check lag, then scale or reset offsets
kafka-consumer-groups.sh --bootstrap-server insights-kafka-0:9092 \
--group analytics-ingest --describe
kubectl scale deployment analytics-ingest --replicas=8 # if lag growing</code></pre>
<h3>Disaster Recovery</h3>
<p>Single-broker failure causes full outage. Mitigations: persistent volume survives pod restart; producers buffer ~30s in-memory; capture returns HTTP 503 (frontend SDKs retry with backoff); on restart, consumers resume from last committed offset. Multi-broker (Phase 2) with replication factor 3 eliminates single-point failure.</p>
<h2>References</h2>
<ol><li><a href="./hip-0004-llm-gateway-unified-ai-provider-interface.md">HIP-4: LLM Gateway</a></li><li><a href="./hip-0017-analytics-event-standard.md">HIP-17: Analytics Event Standard</a></li><li><a href="./hip-0025-bot-agent-wallet-rpc-billing-protocol.md">HIP-25: Bot Agent Wallet &amp; RPC Billing Protocol</a></li><li><a href="./hip-0028-key-value-store-standard.md">HIP-28: Insights Analytics Platform</a></li><li><a href="https://kafka.apache.org/documentation/">Apache Kafka Documentation</a></li><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-500">KRaft: Apache Kafka Without ZooKeeper</a></li><li><a href="https://strimzi.io/">Kafka on Kubernetes - Strimzi</a></li><li><a href="https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka">ClickHouse Kafka Engine</a></li><li><a href="https://github.com/hanzoai/stream">Hanzo Stream Repository</a></li></ol>
<h2>Copyright</h2>
<p>Copyright and related rights waived via <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0</a>.</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/hip-0030.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div>
</body></html>