Files
skills/public/data-streaming-aggregations.html
zeekay 747dafba10 Register WebMCP tools on every page, and name each page's twin from site.py
Every page now loads /webmcp.js, scripts/webmcp.ts with its types
stripped (npm run webmcp). Where the browser has document.modelContext
it registers two read-only tools: search_skills over the published
skills index, and get_skill, which returns a skill's markdown checked
against the index digest. The markdown alternate link moves from
build.py to site.py, so CI holds every page to it.

The API catalog gains the site itself (llms.txt and the skills index)
and hanzo.ai/agents; the live discovery check now fetches every
catalog link. tests/site.mjs drives the tools through a recording
modelContext, and checks a browser without one loads clean.
2026-10-26 20:57:05 -07:00

895 lines
35 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>data-streaming-aggregations</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/data-streaming-aggregations">
<link rel="alternate" type="text/markdown" href="/data-streaming-aggregations.md">
<script type="module" src="/webmcp.js"></script>
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/data-streaming-aggregations.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>data-streaming-aggregations</h1><p class="lede">Windowing, sessionization, time-series aggregation, and late data handling for streaming systems</p><h1>Streaming Aggregations</h1>
<p><strong>Scope</strong>: Windowing strategies, sessionization, time-series aggregation, watermarks, late data handling <strong>Lines</strong>: 385 <strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<h2>When to Use This Skill</h2>
<p>Use this skill when:</p>
<ul><li>Aggregating unbounded streams (counts, sums, averages)</li><li>Implementing time-based windows (tumbling, sliding, session)</li><li>Building real-time dashboards and metrics</li><li>Detecting patterns in time-series data</li><li>Handling out-of-order and late-arriving data</li><li>Computing sessionized analytics (user sessions, click streams)</li><li>Building real-time anomaly detection</li><li>Creating streaming leaderboards and rankings</li></ul>
<h2>Core Concepts</h2>
<h3>Window Types</h3>
<pre><code>Tumbling Windows
→ Fixed size, non-overlapping
→ Example: Count events per 5 minutes
→ Use: Periodic reports, batched processing
→ Memory: O(window_size)
Sliding Windows
→ Fixed size, overlapping
→ Example: Moving average over last 10 minutes
→ Use: Continuous metrics, smoothed trends
→ Memory: O(window_size * slide_factor)
Session Windows
→ Dynamic size, gap-based
→ Example: User sessions with 30-min inactivity
→ Use: User behavior, conversation threads
→ Memory: O(active_sessions)
Global Windows
→ Unbounded, single window
→ Example: All-time counts
→ Use: Stateful processing without time bounds
→ Memory: O(cardinality)</code></pre>
<h3>Time Semantics</h3>
<pre><code>Event Time
→ Time when event occurred
→ Requires: Timestamps in data
→ Accurate but complex (late data)
→ Use: Financial, billing, analytics
Processing Time
→ Time when event processed
→ Simple, low latency
→ Inaccurate for time-based logic
→ Use: Monitoring, system metrics
Ingestion Time
→ Time when event entered system
→ Middle ground
→ Use: Approximation when no event time</code></pre>
<h3>Watermarks</h3>
<pre><code>Watermark(T)
→ &quot;All events before T have arrived&quot;
→ Heuristic, not guarantee
→ Triggers window computation
Strategies:
→ Perfect: Wait forever (impractical)
→ Bounded delay: T = max_timestamp - delay
→ Percentile: Allow X% late data
→ Punctuation: Explicit markers in stream</code></pre>
<h3>Late Data Handling</h3>
<pre><code>Strategies:
→ Drop: Ignore late data (simplest)
→ Update: Recompute window (expensive)
→ Side output: Route to separate stream
→ Allowed lateness: Accept within window
Trade-offs:
→ Accuracy vs Latency
→ Completeness vs Timeliness</code></pre>
<h2>Patterns</h2>
<h3>Pattern 1: Tumbling Window Aggregation (Rust/Timely)</h3>
<pre><code class="lang-rust">use timely::dataflow::operators::{ToStream, Map, Inspect};
use differential_dataflow::input::Input;
use differential_dataflow::operators::{Reduce, Consolidate};
use differential_dataflow::operators::arrange::ArrangeByKey;
#[derive(Clone, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)]
struct Event {
user_id: u32,
value: i32,
timestamp: u64, // Event time in milliseconds
}
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = worker.dataflow::&lt;u64, _, _&gt;(|scope| {
let (input, events) = scope.new_collection();
// Tumbling window: 5-minute (300000ms) windows
let window_size = 300_000u64;
events
.map(move |event| {
// Assign to window based on event time
let window_id = event.timestamp / window_size;
((event.user_id, window_id), event.value)
})
.reduce(|_key, values, output| {
let sum: i32 = values.iter()
.map(|(value, diff)| value * diff)
.sum();
let count: i32 = values.iter()
.map(|(_, diff)| diff)
.sum();
output.push(((sum, count), 1));
})
.inspect(|((user_id, window_id), (sum, count))| {
println!(&quot;User {}, Window {}: sum={}, count={}&quot;,
user_id, window_id, sum, count);
});
input
});
// Simulate events
input.insert(Event { user_id: 1, value: 10, timestamp: 100_000 });
input.insert(Event { user_id: 1, value: 20, timestamp: 150_000 });
input.insert(Event { user_id: 1, value: 30, timestamp: 350_000 }); // Next window
input.advance_to(1);
worker.step_while(|| input.time().less_than(&amp;1));
}).expect(&quot;Execution failed&quot;);
}</code></pre>
<h3>Pattern 2: Sliding Window (Python)</h3>
<pre><code class="lang-python">import time
from collections import deque
from dataclasses import dataclass
from typing import Deque, Dict
import threading
@dataclass
class Event:
timestamp: float
user_id: str
value: float
class SlidingWindowAggregator:
&quot;&quot;&quot;Sliding window with event-time semantics&quot;&quot;&quot;
def __init__(self, window_size: float, slide_interval: float):
self.window_size = window_size # Window size in seconds
self.slide_interval = slide_interval # How often to emit
self.buffers: Dict[str, Deque[Event]] = {} # Per-key buffers
self.lock = threading.Lock()
def process(self, event: Event):
&quot;&quot;&quot;Process event and return windows that are ready&quot;&quot;&quot;
with self.lock:
# Initialize buffer for key
if event.user_id not in self.buffers:
self.buffers[event.user_id] = deque()
buffer = self.buffers[event.user_id]
# Add event
buffer.append(event)
# Remove events outside window
cutoff = event.timestamp - self.window_size
while buffer and buffer[0].timestamp &lt; cutoff:
buffer.popleft()
# Compute aggregate
total = sum(e.value for e in buffer)
count = len(buffer)
avg = total / count if count &gt; 0 else 0.0
return {
&#x27;user_id&#x27;: event.user_id,
&#x27;window_start&#x27;: cutoff,
&#x27;window_end&#x27;: event.timestamp,
&#x27;sum&#x27;: total,
&#x27;count&#x27;: count,
&#x27;avg&#x27;: avg
}
# Usage
def main():
aggregator = SlidingWindowAggregator(
window_size=10.0, # 10-second window
slide_interval=2.0 # Emit every 2 seconds
)
# Simulate event stream
events = [
Event(timestamp=1.0, user_id=&#x27;alice&#x27;, value=10),
Event(timestamp=2.0, user_id=&#x27;alice&#x27;, value=20),
Event(timestamp=5.0, user_id=&#x27;alice&#x27;, value=30),
Event(timestamp=12.0, user_id=&#x27;alice&#x27;, value=40), # Outside first window
]
for event in events:
result = aggregator.process(event)
print(f&quot;Window result: {result}&quot;)
if __name__ == &#x27;__main__&#x27;:
main()</code></pre>
<h3>Pattern 3: Session Windows (Go)</h3>
<pre><code class="lang-go">package main
import (
&quot;fmt&quot;
&quot;sort&quot;
&quot;time&quot;
)
type Event struct {
UserID string
Timestamp time.Time
Value int
}
type Session struct {
UserID string
Start time.Time
End time.Time
Events []Event
Sum int
}
type SessionWindowAggregator struct {
inactivityGap time.Duration
sessions map[string]*Session
}
func NewSessionWindowAggregator(gap time.Duration) *SessionWindowAggregator {
return &amp;SessionWindowAggregator{
inactivityGap: gap,
sessions: make(map[string]*Session),
}
}
func (swa *SessionWindowAggregator) Process(event Event) *Session {
session, exists := swa.sessions[event.UserID]
if !exists || event.Timestamp.Sub(session.End) &gt; swa.inactivityGap {
// Start new session
if exists {
// Emit completed session
completed := session
fmt.Printf(&quot;Session complete: %+v\n&quot;, completed)
}
session = &amp;Session{
UserID: event.UserID,
Start: event.Timestamp,
End: event.Timestamp,
Events: []Event{event},
Sum: event.Value,
}
swa.sessions[event.UserID] = session
} else {
// Extend existing session
session.End = event.Timestamp
session.Events = append(session.Events, event)
session.Sum += event.Value
}
return nil
}
func (swa *SessionWindowAggregator) FlushExpired(currentTime time.Time) []*Session {
var expired []*Session
for userID, session := range swa.sessions {
if currentTime.Sub(session.End) &gt; swa.inactivityGap {
expired = append(expired, session)
delete(swa.sessions, userID)
}
}
return expired
}
func main() {
aggregator := NewSessionWindowAggregator(5 * time.Minute)
// Simulate event stream
events := []Event{
{UserID: &quot;alice&quot;, Timestamp: time.Now(), Value: 10},
{UserID: &quot;alice&quot;, Timestamp: time.Now().Add(1 * time.Minute), Value: 20},
{UserID: &quot;alice&quot;, Timestamp: time.Now().Add(10 * time.Minute), Value: 30}, // New session
{UserID: &quot;bob&quot;, Timestamp: time.Now().Add(2 * time.Minute), Value: 15},
}
for _, event := range events {
aggregator.Process(event)
}
// Flush expired sessions
expired := aggregator.FlushExpired(time.Now().Add(20 * time.Minute))
for _, session := range expired {
fmt.Printf(&quot;Expired session: %+v\n&quot;, session)
}
}</code></pre>
<h3>Pattern 4: Watermark-Based Late Data Handling (Python)</h3>
<pre><code class="lang-python">from dataclasses import dataclass
from typing import Dict, List, Optional
import heapq
@dataclass
class Event:
timestamp: int # Event time
key: str
value: int
@dataclass
class Window:
start: int
end: int
key: str
sum: int = 0
count: int = 0
class WatermarkAggregator:
&quot;&quot;&quot;Window aggregator with watermark-based triggering&quot;&quot;&quot;
def __init__(
self,
window_size: int,
allowed_lateness: int,
watermark_delay: int
):
self.window_size = window_size
self.allowed_lateness = allowed_lateness
self.watermark_delay = watermark_delay
self.windows: Dict[tuple, Window] = {} # (key, window_start) -&gt; Window
self.max_timestamp = 0
self.watermark = 0
self.late_events: List[Event] = []
def process(self, event: Event) -&gt; List[Window]:
&quot;&quot;&quot;Process event and emit completed windows&quot;&quot;&quot;
# Update max timestamp and watermark
self.max_timestamp = max(self.max_timestamp, event.timestamp)
self.watermark = self.max_timestamp - self.watermark_delay
# Assign to window
window_start = (event.timestamp // self.window_size) * self.window_size
window_end = window_start + self.window_size
window_key = (event.key, window_start)
# Check if too late
if event.timestamp &lt; self.watermark - self.allowed_lateness:
self.late_events.append(event)
print(f&quot;Late event dropped: {event}&quot;)
return []
# Update window
if window_key not in self.windows:
self.windows[window_key] = Window(
start=window_start,
end=window_end,
key=event.key
)
window = self.windows[window_key]
window.sum += event.value
window.count += 1
# Emit completed windows (watermark passed end + allowed lateness)
return self._emit_completed_windows()
def _emit_completed_windows(self) -&gt; List[Window]:
&quot;&quot;&quot;Emit windows that watermark has passed&quot;&quot;&quot;
completed = []
to_remove = []
for (key, window_start), window in self.windows.items():
# Window complete if watermark &gt; window_end + allowed_lateness
if self.watermark &gt; window.end + self.allowed_lateness:
completed.append(window)
to_remove.append((key, window_start))
# Remove completed windows
for key in to_remove:
del self.windows[key]
return completed
# Usage
def main():
aggregator = WatermarkAggregator(
window_size=10, # 10-second windows
allowed_lateness=5, # Accept events up to 5 seconds late
watermark_delay=3 # Watermark = max_timestamp - 3
)
# Events arriving out of order
events = [
Event(timestamp=5, key=&#x27;user1&#x27;, value=10),
Event(timestamp=15, key=&#x27;user1&#x27;, value=20),
Event(timestamp=8, key=&#x27;user1&#x27;, value=15), # Late but within allowed
Event(timestamp=25, key=&#x27;user1&#x27;, value=30),
Event(timestamp=3, key=&#x27;user1&#x27;, value=5), # Too late, dropped
]
for event in events:
completed = aggregator.process(event)
for window in completed:
print(f&quot;Window complete: {window}&quot;)
if __name__ == &#x27;__main__&#x27;:
main()</code></pre>
<h3>Pattern 5: Time-Series Aggregation with Downsampling</h3>
<pre><code class="lang-python">import numpy as np
from dataclasses import dataclass
from typing import Dict, List
from collections import defaultdict
@dataclass
class TimeSeriesPoint:
timestamp: int
metric: str
value: float
class TimeSeriesAggregator:
&quot;&quot;&quot;Multi-resolution time-series aggregation&quot;&quot;&quot;
def __init__(self):
# Store multiple resolutions
self.raw: Dict[str, List[TimeSeriesPoint]] = defaultdict(list)
self.minute: Dict[str, List[tuple]] = defaultdict(list) # (timestamp, avg, min, max)
self.hour: Dict[str, List[tuple]] = defaultdict(list)
def ingest(self, point: TimeSeriesPoint):
&quot;&quot;&quot;Ingest raw point and update aggregations&quot;&quot;&quot;
self.raw[point.metric].append(point)
# Update minute-level aggregation
minute_bucket = (point.timestamp // 60) * 60
self._update_aggregation(point.metric, minute_bucket, point.value, self.minute)
# Update hour-level aggregation
hour_bucket = (point.timestamp // 3600) * 3600
self._update_aggregation(point.metric, hour_bucket, point.value, self.hour)
def _update_aggregation(
self,
metric: str,
bucket: int,
value: float,
storage: Dict[str, List[tuple]]
):
&quot;&quot;&quot;Update aggregation bucket&quot;&quot;&quot;
buckets = storage[metric]
# Find or create bucket
if not buckets or buckets[-1][0] != bucket:
# New bucket
buckets.append((bucket, value, value, value, 1)) # (time, sum, min, max, count)
else:
# Update existing bucket
old_time, old_sum, old_min, old_max, old_count = buckets[-1]
buckets[-1] = (
old_time,
old_sum + value,
min(old_min, value),
max(old_max, value),
old_count + 1
)
def query(
self,
metric: str,
start: int,
end: int,
resolution: str = &#x27;minute&#x27;
) -&gt; List[tuple]:
&quot;&quot;&quot;Query aggregated data at specified resolution&quot;&quot;&quot;
storage = {
&#x27;minute&#x27;: self.minute,
&#x27;hour&#x27;: self.hour,
&#x27;raw&#x27;: self.raw
}[resolution]
if resolution == &#x27;raw&#x27;:
points = storage[metric]
filtered = [p for p in points if start &lt;= p.timestamp &lt;= end]
return [(p.timestamp, p.value) for p in filtered]
buckets = storage[metric]
filtered = [
(timestamp, total/count, min_val, max_val)
for timestamp, total, min_val, max_val, count in buckets
if start &lt;= timestamp &lt;= end
]
return filtered
# Usage
def main():
agg = TimeSeriesAggregator()
# Ingest high-frequency data
for i in range(7200): # 2 hours of second-level data
point = TimeSeriesPoint(
timestamp=i,
metric=&#x27;cpu_usage&#x27;,
value=50 + 10 * np.sin(i / 60.0) # Simulate pattern
)
agg.ingest(point)
# Query at different resolutions
minute_data = agg.query(&#x27;cpu_usage&#x27;, 0, 3600, resolution=&#x27;minute&#x27;)
print(f&quot;Minute-level aggregation: {len(minute_data)} points&quot;)
hour_data = agg.query(&#x27;cpu_usage&#x27;, 0, 7200, resolution=&#x27;hour&#x27;)
print(f&quot;Hour-level aggregation: {len(hour_data)} points&quot;)
# Each point: (timestamp, avg, min, max)
for timestamp, avg, min_val, max_val in hour_data:
print(f&quot;Hour {timestamp//3600}: avg={avg:.2f}, min={min_val:.2f}, max={max_val:.2f}&quot;)
if __name__ == &#x27;__main__&#x27;:
main()</code></pre>
<h3>Pattern 6: Top-K Streaming Aggregation</h3>
<pre><code class="lang-rust">use std::collections::BinaryHeap;
use std::cmp::Reverse;
#[derive(Debug, Clone, Eq, PartialEq)]
struct Item {
key: String,
count: usize,
}
impl Ord for Item {
fn cmp(&amp;self, other: &amp;Self) -&gt; std::cmp::Ordering {
self.count.cmp(&amp;other.count)
}
}
impl PartialOrd for Item {
fn partial_cmp(&amp;self, other: &amp;Self) -&gt; Option&lt;std::cmp::Ordering&gt; {
Some(self.cmp(other))
}
}
struct TopKAggregator {
k: usize,
counts: std::collections::HashMap&lt;String, usize&gt;,
heap: BinaryHeap&lt;Reverse&lt;Item&gt;&gt;,
}
impl TopKAggregator {
fn new(k: usize) -&gt; Self {
Self {
k,
counts: std::collections::HashMap::new(),
heap: BinaryHeap::new(),
}
}
fn update(&amp;mut self, key: String) {
// Update count
let count = self.counts.entry(key.clone()).or_insert(0);
*count += 1;
// Maintain top-K heap
self.rebuild_heap();
}
fn rebuild_heap(&amp;mut self) {
self.heap.clear();
for (key, count) in &amp;self.counts {
let item = Item {
key: key.clone(),
count: *count,
};
if self.heap.len() &lt; self.k {
self.heap.push(Reverse(item));
} else if let Some(Reverse(min)) = self.heap.peek() {
if item.count &gt; min.count {
self.heap.pop();
self.heap.push(Reverse(item));
}
}
}
}
fn top_k(&amp;self) -&gt; Vec&lt;Item&gt; {
let mut items: Vec&lt;_&gt; = self.heap.iter()
.map(|Reverse(item)| item.clone())
.collect();
items.sort_by(|a, b| b.count.cmp(&amp;a.count));
items
}
}
fn main() {
let mut aggregator = TopKAggregator::new(3);
// Simulate event stream
let events = vec![&quot;apple&quot;, &quot;banana&quot;, &quot;apple&quot;, &quot;cherry&quot;, &quot;apple&quot;, &quot;banana&quot;, &quot;date&quot;];
for event in events {
aggregator.update(event.to_string());
}
println!(&quot;Top 3 items:&quot;);
for item in aggregator.top_k() {
println!(&quot; {}: {}&quot;, item.key, item.count);
}
}</code></pre>
<h2>Quick Reference</h2>
<h3>Window Selection Guide</h3>
<pre><code>Use Tumbling: Periodic reports, non-overlapping batches
Use Sliding: Moving averages, continuous metrics
Use Session: User behavior, conversation analysis
Use Global: Unbounded state, all-time aggregations</code></pre>
<h3>Watermark Strategies</h3>
<pre><code class="lang-python"># Bounded delay
watermark = max_timestamp - fixed_delay
# Percentile-based
watermark = percentile(timestamps, 99) # Allow 1% late
# Heuristic
watermark = max_timestamp - 2 * stddev(inter_arrival_time)</code></pre>
<h3>Time Extraction</h3>
<pre><code class="lang-rust">// Extract event time from data
let event_time = |event: &amp;Event| event.timestamp;
// Assign to window
let window_id = event.timestamp / window_size;
// Session gap check
if current_time - last_event_time &gt; session_gap {
// Start new session
}</code></pre>
<h2>Performance Optimization</h2>
<pre><code>Reduce State Size
→ Compact old windows
→ Use approximate algorithms (HyperLogLog, Count-Min Sketch)
→ Expire inactive keys
Batch Processing
→ Buffer events before aggregating
→ Periodic window evaluation
Incremental Updates
→ Use differential dataflow for efficient re-aggregation
→ Maintain summary statistics (sum, count) instead of raw data</code></pre>
<h2>Anti-Patterns</h2>
<pre><code>❌ NEVER: Use processing time for event-time logic
→ Results depend on processing speed, not actual event timing
❌ NEVER: Wait indefinitely for late data
→ Set allowed lateness bounds
❌ NEVER: Store unbounded state in global windows
→ Use approximate algorithms or periodic cleanup
❌ NEVER: Ignore watermarks
→ Windows never complete, state grows unbounded
❌ NEVER: Use sliding windows with small slide interval on high-volume streams
→ Creates many overlapping windows, high memory usage
❌ NEVER: Recompute entire window on late data
→ Use incremental updates
❌ NEVER: Assume events arrive in order
→ Always design for out-of-order delivery
❌ NEVER: Use session windows without timeouts
→ Sessions never close, memory leak
❌ NEVER: Emit window before watermark passes
→ Incomplete results
❌ NEVER: Drop late data without logging
→ Monitor late data rates for tuning</code></pre>
<h2>Related Skills</h2>
<ul><li><code>timely-dataflow.md</code> - Foundation for windowing with progress tracking</li><li><code>differential-dataflow.md</code> - Incremental window updates</li><li><code>dataflow-coordination.md</code> - Watermarks and coordination</li><li><code>stream-processing.md</code> - High-level stream processing with Kafka</li></ul>
<p><strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/data-streaming-aggregations.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>