Files
skills/public/data-stream-processing.html
zeekay 747dafba10 Register WebMCP tools on every page, and name each page's twin from site.py
Every page now loads /webmcp.js, scripts/webmcp.ts with its types
stripped (npm run webmcp). Where the browser has document.modelContext
it registers two read-only tools: search_skills over the published
skills index, and get_skill, which returns a skill's markdown checked
against the index digest. The markdown alternate link moves from
build.py to site.py, so CI holds every page to it.

The API catalog gains the site itself (llms.txt and the skills index)
and hanzo.ai/agents; the live discovery check now fetches every
catalog link. tests/site.mjs drives the tools through a recording
modelContext, and checks a browser without one loads clean.
2026-10-26 20:57:05 -07:00

753 lines
31 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>data-stream-processing</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/data-stream-processing">
<link rel="alternate" type="text/markdown" href="/data-stream-processing.md">
<script type="module" src="/webmcp.js"></script>
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/data-stream-processing.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>data-stream-processing</h1><p class="lede">Processing real-time event streams (clicks, IoT, logs)</p><h1>Stream Processing</h1>
<p><strong>Scope</strong>: Kafka, event streaming, windowing, stateful processing <strong>Lines</strong>: 368 <strong>Last Updated</strong>: 2025-10-18 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<h2>When to Use This Skill</h2>
<p>Use this skill when:</p>
<ul><li>Processing real-time event streams (clicks, IoT, logs)</li><li>Building event-driven architectures with Kafka</li><li>Implementing windowing aggregations (tumbling, sliding, session)</li><li>Managing stateful stream processing</li><li>Handling exactly-once semantics</li><li>Designing stream joins and enrichment patterns</li><li>Working with Kafka Streams, Flink, or custom consumers</li></ul>
<h2>Core Concepts</h2>
<h3>Stream vs Batch</h3>
<pre><code>Batch Processing
→ Bounded datasets, scheduled runs
→ Use when: Historical analysis, daily reports
→ Tools: Airflow, Spark batch jobs
Stream Processing
→ Unbounded datasets, continuous processing
→ Use when: Real-time alerts, live dashboards
→ Tools: Kafka Streams, Flink, ksqlDB</code></pre>
<h3>Event Time vs Processing Time</h3>
<pre><code>Event Time
→ When event actually occurred
→ Use when: Order matters, late data possible
→ Requires: Timestamps in events, watermarks
Processing Time
→ When event processed by system
→ Use when: Latency critical, order irrelevant
→ Simpler but less accurate for time-based logic</code></pre>
<h3>Windowing Types</h3>
<pre><code>Tumbling Window
→ Fixed, non-overlapping intervals
→ Example: Count events per 5-minute window
→ Use when: Periodic aggregations
Sliding Window
→ Fixed size, overlapping intervals
→ Example: Moving average over last 10 minutes
→ Use when: Continuous metrics
Session Window
→ Dynamic size based on inactivity gap
→ Example: User sessions with 30-min timeout
→ Use when: User behavior analysis
Global Window
→ All events in single window
→ Use when: No time-based grouping needed</code></pre>
<h3>Delivery Guarantees</h3>
<pre><code>At-Most-Once
→ May lose messages
→ Fastest, use for metrics/logs
At-Least-Once
→ May duplicate messages
→ Common default, requires idempotent processing
Exactly-Once
→ No loss, no duplication
→ Kafka transactions, complex but correct</code></pre>
<h2>Patterns</h2>
<h3>Pattern 1: Kafka Producer (Python)</h3>
<pre><code class="lang-python">from kafka import KafkaProducer
from kafka.errors import KafkaError
import json
from typing import Dict, Any
from datetime import datetime
class EventProducer:
def __init__(self, bootstrap_servers: list[str], topic: str):
self.topic = topic
self.producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
value_serializer=lambda v: json.dumps(v).encode(&#x27;utf-8&#x27;),
key_serializer=lambda k: k.encode(&#x27;utf-8&#x27;) if k else None,
acks=&#x27;all&#x27;, # Wait for all replicas (strongest guarantee)
retries=3,
max_in_flight_requests_per_connection=1, # Maintain order
compression_type=&#x27;gzip&#x27;
)
def send_event(self, event: Dict[str, Any], key: str = None) -&gt; bool:
&quot;&quot;&quot;Send event with error handling&quot;&quot;&quot;
# Add metadata
event[&#x27;_timestamp&#x27;] = datetime.utcnow().isoformat()
try:
future = self.producer.send(
self.topic,
key=key,
value=event
)
# Block for &#x27;synchronous&#x27; sends
record_metadata = future.get(timeout=10)
print(f&quot;Sent to {record_metadata.topic} &quot;
f&quot;partition {record_metadata.partition} &quot;
f&quot;offset {record_metadata.offset}&quot;)
return True
except KafkaError as e:
print(f&quot;Failed to send event: {e}&quot;)
return False
def send_batch(self, events: list[Dict[str, Any]]):
&quot;&quot;&quot;Send batch asynchronously&quot;&quot;&quot;
for event in events:
self.producer.send(self.topic, value=event)
self.producer.flush() # Wait for all to complete
def close(self):
self.producer.close()
# Usage
producer = EventProducer([&#x27;localhost:9092&#x27;], &#x27;user-events&#x27;)
event = {
&#x27;user_id&#x27;: &#x27;user123&#x27;,
&#x27;action&#x27;: &#x27;page_view&#x27;,
&#x27;page&#x27;: &#x27;/product/456&#x27;,
&#x27;session_id&#x27;: &#x27;session789&#x27;
}
producer.send_event(event, key=&#x27;user123&#x27;) # Partition by user_id
producer.close()</code></pre>
<h3>Pattern 2: Kafka Consumer with Offset Management</h3>
<pre><code class="lang-python">from kafka import KafkaConsumer, TopicPartition, OffsetAndMetadata
from typing import Callable
import json
class EventConsumer:
def __init__(
self,
bootstrap_servers: list[str],
topic: str,
group_id: str,
auto_commit: bool = False
):
self.consumer = KafkaConsumer(
topic,
bootstrap_servers=bootstrap_servers,
group_id=group_id,
auto_offset_reset=&#x27;earliest&#x27;, # Start from beginning if no offset
enable_auto_commit=auto_commit,
value_deserializer=lambda m: json.loads(m.decode(&#x27;utf-8&#x27;)),
max_poll_records=500,
session_timeout_ms=30000,
heartbeat_interval_ms=10000
)
self.processed_count = 0
def consume(
self,
process_fn: Callable[[dict], bool],
commit_interval: int = 100
):
&quot;&quot;&quot;Consume with manual offset management&quot;&quot;&quot;
buffer = []
try:
for message in self.consumer:
event = message.value
# Process event
success = process_fn(event)
if success:
buffer.append(message)
self.processed_count += 1
# Commit offsets periodically
if len(buffer) &gt;= commit_interval:
self._commit_offsets(buffer)
buffer = []
else:
# Handle failure (DLQ, retry, log)
self._handle_failure(message)
except KeyboardInterrupt:
print(&quot;Shutting down...&quot;)
finally:
if buffer:
self._commit_offsets(buffer)
self.consumer.close()
def _commit_offsets(self, messages: list):
&quot;&quot;&quot;Commit offsets for processed messages&quot;&quot;&quot;
offsets = {}
for msg in messages:
tp = TopicPartition(msg.topic, msg.partition)
offsets[tp] = OffsetAndMetadata(msg.offset + 1, None)
self.consumer.commit(offsets)
print(f&quot;Committed {len(messages)} offsets&quot;)
def _handle_failure(self, message):
&quot;&quot;&quot;Send to dead letter queue&quot;&quot;&quot;
print(f&quot;Failed to process: {message.value}&quot;)
# Could send to DLQ topic here
# Usage
def process_event(event: dict) -&gt; bool:
try:
user_id = event[&#x27;user_id&#x27;]
action = event[&#x27;action&#x27;]
# Business logic
print(f&quot;User {user_id} performed {action}&quot;)
# Could write to database, call API, etc.
return True
except Exception as e:
print(f&quot;Error processing event: {e}&quot;)
return False
consumer = EventConsumer(
[&#x27;localhost:9092&#x27;],
&#x27;user-events&#x27;,
group_id=&#x27;analytics-pipeline&#x27;
)
consumer.consume(process_event, commit_interval=100)</code></pre>
<h3>Pattern 3: Kafka Streams (Windowing &amp; Aggregation)</h3>
<pre><code class="lang-java">// Java - Kafka Streams API
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.*;
import java.time.Duration;
public class EventAggregator {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
// Input stream
KStream&lt;String, UserEvent&gt; events = builder.stream(
&quot;user-events&quot;,
Consumed.with(Serdes.String(), new UserEventSerde())
);
// Tumbling window: Count events per user per 5 minutes
KTable&lt;Windowed&lt;String&gt;, Long&gt; userEventCounts = events
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count();
// Sliding window: Moving average of event values
KTable&lt;Windowed&lt;String&gt;, Double&gt; movingAverage = events
.groupByKey()
.windowedBy(
SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10))
)
.aggregate(
() -&gt; new AggregateState(0.0, 0),
(key, event, aggregate) -&gt; {
aggregate.sum += event.getValue();
aggregate.count += 1;
return aggregate;
},
Materialized.with(Serdes.String(), new AggregateStateSerde())
)
.mapValues(agg -&gt; agg.sum / agg.count);
// Session window: User sessions with 30-minute inactivity gap
KTable&lt;Windowed&lt;String&gt;, Long&gt; sessions = events
.groupByKey()
.windowedBy(
SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30))
)
.count();
// Output to topics
userEventCounts
.toStream()
.map((key, value) -&gt; new KeyValue&lt;&gt;(
key.key() + &quot;@&quot; + key.window().start(),
value
))
.to(&quot;event-counts-per-5min&quot;);
// Build and start
KafkaStreams streams = new KafkaStreams(builder.build(), getConfig());
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
}</code></pre>
<h3>Pattern 4: Stream Joins (Python with Faust)</h3>
<pre><code class="lang-python"># Python - Faust framework (Kafka Streams alternative)
import faust
from datetime import timedelta
app = faust.App(
&#x27;stream-joins&#x27;,
broker=&#x27;kafka://localhost:9092&#x27;,
store=&#x27;rocksdb://&#x27;
)
# Define models
class ClickEvent(faust.Record):
user_id: str
page: str
timestamp: float
class PurchaseEvent(faust.Record):
user_id: str
product_id: str
amount: float
timestamp: float
class EnrichedPurchase(faust.Record):
user_id: str
product_id: str
amount: float
recent_pages: list[str]
# Topics
clicks_topic = app.topic(&#x27;clicks&#x27;, value_type=ClickEvent)
purchases_topic = app.topic(&#x27;purchases&#x27;, value_type=PurchaseEvent)
# Tables for stateful processing
recent_clicks = app.Table(
&#x27;recent_clicks&#x27;,
default=list,
on_window_close=lambda key, values: print(f&quot;Window closed for {key}&quot;)
).tumbling(timedelta(minutes=10), expires=timedelta(hours=1))
# Stream join pattern
@app.agent(purchases_topic)
async def enrich_purchases(purchases):
async for purchase in purchases:
# Get recent clicks for user from state table
clicks = recent_clicks[purchase.user_id].current()
enriched = EnrichedPurchase(
user_id=purchase.user_id,
product_id=purchase.product_id,
amount=purchase.amount,
recent_pages=[c.page for c in clicks] if clicks else []
)
# Send to enriched topic
await app.topic(&#x27;enriched-purchases&#x27;).send(value=enriched)
# Maintain click state
@app.agent(clicks_topic)
async def track_clicks(clicks):
async for click in clicks:
recent_clicks[click.user_id].current().append(click)
# Windowed aggregation
@app.agent(purchases_topic)
async def revenue_per_window(purchases):
async for purchase in purchases.group_by(lambda p: &#x27;all&#x27;):
# Aggregate in 5-minute tumbling windows
window = recent_clicks.current()
# Process aggregation
yield purchase
if __name__ == &#x27;__main__&#x27;:
app.main()</code></pre>
<h3>Pattern 5: Exactly-Once Semantics (Transactional Producer)</h3>
<pre><code class="lang-python">from kafka import KafkaProducer, KafkaConsumer
from kafka.coordinator.assignors.range import RangePartitionAssignor
import json
class ExactlyOnceProcessor:
def __init__(
self,
bootstrap_servers: list[str],
input_topic: str,
output_topic: str,
group_id: str
):
self.input_topic = input_topic
self.output_topic = output_topic
# Transactional producer
self.producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
transactional_id=f&#x27;{group_id}-producer&#x27;,
enable_idempotence=True,
value_serializer=lambda v: json.dumps(v).encode(&#x27;utf-8&#x27;)
)
# Consumer with read_committed isolation
self.consumer = KafkaConsumer(
input_topic,
bootstrap_servers=bootstrap_servers,
group_id=group_id,
enable_auto_commit=False,
isolation_level=&#x27;read_committed&#x27;, # Only read committed messages
value_deserializer=lambda m: json.loads(m.decode(&#x27;utf-8&#x27;))
)
self.producer.init_transactions()
def process(self, transform_fn):
&quot;&quot;&quot;Process with exactly-once semantics&quot;&quot;&quot;
try:
for message in self.consumer:
self.producer.begin_transaction()
try:
# Transform event
input_event = message.value
output_event = transform_fn(input_event)
# Send to output topic
self.producer.send(self.output_topic, value=output_event)
# Commit input offset within transaction
offsets = {
TopicPartition(message.topic, message.partition):
OffsetAndMetadata(message.offset + 1, None)
}
self.producer.send_offsets_to_transaction(
offsets,
self.consumer.config[&#x27;group_id&#x27;]
)
# Commit transaction (atomic)
self.producer.commit_transaction()
except Exception as e:
print(f&quot;Error processing, aborting transaction: {e}&quot;)
self.producer.abort_transaction()
finally:
self.producer.close()
self.consumer.close()
# Usage
processor = ExactlyOnceProcessor(
[&#x27;localhost:9092&#x27;],
&#x27;input-events&#x27;,
&#x27;output-events&#x27;,
&#x27;exactly-once-group&#x27;
)
def transform(event):
# Idempotent transformation
return {
&#x27;user_id&#x27;: event[&#x27;user_id&#x27;],
&#x27;processed_value&#x27;: event[&#x27;value&#x27;] * 2,
&#x27;processed_at&#x27;: datetime.utcnow().isoformat()
}
processor.process(transform)</code></pre>
<h2>Quick Reference</h2>
<h3>Kafka CLI Commands</h3>
<pre><code class="lang-bash"># Create topic
kafka-topics --create --topic my-topic \
--partitions 3 --replication-factor 2 \
--bootstrap-server localhost:9092
# List topics
kafka-topics --list --bootstrap-server localhost:9092
# Describe topic
kafka-topics --describe --topic my-topic \
--bootstrap-server localhost:9092
# Consume from beginning
kafka-console-consumer --topic my-topic \
--from-beginning --bootstrap-server localhost:9092
# Produce messages
kafka-console-producer --topic my-topic \
--bootstrap-server localhost:9092
# Consumer groups
kafka-consumer-groups --list --bootstrap-server localhost:9092
kafka-consumer-groups --describe --group my-group \
--bootstrap-server localhost:9092
# Reset offsets
kafka-consumer-groups --group my-group \
--reset-offsets --to-earliest --topic my-topic \
--execute --bootstrap-server localhost:9092</code></pre>
<h3>Windowing Cheat Sheet</h3>
<pre><code class="lang-python"># Tumbling (Faust)
table.tumbling(timedelta(minutes=5))
# Hopping/Sliding (Faust)
table.hopping(size=timedelta(minutes=10), step=timedelta(minutes=2))
# Kafka Streams (Java)
TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))
SlidingWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10))
SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30))</code></pre>
<h3>Partitioning Strategies</h3>
<pre><code class="lang-python"># By key (ensures same key -&gt; same partition)
producer.send(topic, key=&#x27;user123&#x27;, value=event)
# Round-robin (no key)
producer.send(topic, value=event)
# Custom partitioner
from kafka.partitioner import Murmur2Partitioner
producer = KafkaProducer(partitioner=Murmur2Partitioner())</code></pre>
<h2>Anti-Patterns</h2>
<pre><code>❌ NEVER: Auto-commit without idempotent processing
→ Use manual commits with exactly-once semantics
❌ NEVER: Process events out of order within partition
→ Kafka guarantees order per partition, maintain it
❌ NEVER: Block consumer loop with slow processing
→ Use async processing or separate worker threads
❌ NEVER: Ignore late arriving data
→ Set watermarks and allowed lateness windows
❌ NEVER: Use global state without partitioning
→ State must be partitioned like input data
❌ NEVER: Skip error handling on deserialization
→ Use try-except, send bad records to DLQ
❌ NEVER: Create too many partitions initially
→ Start small, scale up based on throughput needs
❌ NEVER: Use processing time for event-time logic
→ Use event timestamps for accurate windowing
❌ NEVER: Commit offsets before processing completes
→ Commit only after successful processing/write
❌ NEVER: Ignore consumer lag monitoring
→ Monitor lag, alert on growing backlog</code></pre>
<h2>Related Skills</h2>
<ul><li><code>etl-patterns.md</code> - Batch alternatives to stream processing</li><li><code>batch-processing.md</code> - Hybrid batch + stream architectures</li><li><code>pipeline-orchestration.md</code> - Managing stream applications</li><li><code>data-validation.md</code> - Schema validation for streams</li></ul>
<p><strong>Last Updated</strong>: 2025-10-18 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/data-stream-processing.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>