mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
Every page now loads /webmcp.js, scripts/webmcp.ts with its types stripped (npm run webmcp). Where the browser has document.modelContext it registers two read-only tools: search_skills over the published skills index, and get_skill, which returns a skill's markdown checked against the index digest. The markdown alternate link moves from build.py to site.py, so CI holds every page to it. The API catalog gains the site itself (llms.txt and the skills index) and hanzo.ai/agents; the live discovery check now fetches every catalog link. tests/site.mjs drives the tools through a recording modelContext, and checks a browser without one loads clean.
895 lines
35 KiB
HTML
895 lines
35 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>data-streaming-aggregations</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/data-streaming-aggregations">
|
|
<link rel="alternate" type="text/markdown" href="/data-streaming-aggregations.md">
|
|
<script type="module" src="/webmcp.js"></script>
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/data-streaming-aggregations.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>data-streaming-aggregations</h1><p class="lede">Windowing, sessionization, time-series aggregation, and late data handling for streaming systems</p><h1>Streaming Aggregations</h1>
|
|
<p><strong>Scope</strong>: Windowing strategies, sessionization, time-series aggregation, watermarks, late data handling <strong>Lines</strong>: 385 <strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
|
|
<h2>When to Use This Skill</h2>
|
|
<p>Use this skill when:</p>
|
|
<ul><li>Aggregating unbounded streams (counts, sums, averages)</li><li>Implementing time-based windows (tumbling, sliding, session)</li><li>Building real-time dashboards and metrics</li><li>Detecting patterns in time-series data</li><li>Handling out-of-order and late-arriving data</li><li>Computing sessionized analytics (user sessions, click streams)</li><li>Building real-time anomaly detection</li><li>Creating streaming leaderboards and rankings</li></ul>
|
|
<h2>Core Concepts</h2>
|
|
<h3>Window Types</h3>
|
|
<pre><code>Tumbling Windows
|
|
→ Fixed size, non-overlapping
|
|
→ Example: Count events per 5 minutes
|
|
→ Use: Periodic reports, batched processing
|
|
→ Memory: O(window_size)
|
|
|
|
Sliding Windows
|
|
→ Fixed size, overlapping
|
|
→ Example: Moving average over last 10 minutes
|
|
→ Use: Continuous metrics, smoothed trends
|
|
→ Memory: O(window_size * slide_factor)
|
|
|
|
Session Windows
|
|
→ Dynamic size, gap-based
|
|
→ Example: User sessions with 30-min inactivity
|
|
→ Use: User behavior, conversation threads
|
|
→ Memory: O(active_sessions)
|
|
|
|
Global Windows
|
|
→ Unbounded, single window
|
|
→ Example: All-time counts
|
|
→ Use: Stateful processing without time bounds
|
|
→ Memory: O(cardinality)</code></pre>
|
|
<h3>Time Semantics</h3>
|
|
<pre><code>Event Time
|
|
→ Time when event occurred
|
|
→ Requires: Timestamps in data
|
|
→ Accurate but complex (late data)
|
|
→ Use: Financial, billing, analytics
|
|
|
|
Processing Time
|
|
→ Time when event processed
|
|
→ Simple, low latency
|
|
→ Inaccurate for time-based logic
|
|
→ Use: Monitoring, system metrics
|
|
|
|
Ingestion Time
|
|
→ Time when event entered system
|
|
→ Middle ground
|
|
→ Use: Approximation when no event time</code></pre>
|
|
<h3>Watermarks</h3>
|
|
<pre><code>Watermark(T)
|
|
→ "All events before T have arrived"
|
|
→ Heuristic, not guarantee
|
|
→ Triggers window computation
|
|
|
|
Strategies:
|
|
→ Perfect: Wait forever (impractical)
|
|
→ Bounded delay: T = max_timestamp - delay
|
|
→ Percentile: Allow X% late data
|
|
→ Punctuation: Explicit markers in stream</code></pre>
|
|
<h3>Late Data Handling</h3>
|
|
<pre><code>Strategies:
|
|
→ Drop: Ignore late data (simplest)
|
|
→ Update: Recompute window (expensive)
|
|
→ Side output: Route to separate stream
|
|
→ Allowed lateness: Accept within window
|
|
|
|
Trade-offs:
|
|
→ Accuracy vs Latency
|
|
→ Completeness vs Timeliness</code></pre>
|
|
<h2>Patterns</h2>
|
|
<h3>Pattern 1: Tumbling Window Aggregation (Rust/Timely)</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::operators::{ToStream, Map, Inspect};
|
|
use differential_dataflow::input::Input;
|
|
use differential_dataflow::operators::{Reduce, Consolidate};
|
|
use differential_dataflow::operators::arrange::ArrangeByKey;
|
|
|
|
#[derive(Clone, Debug, Hash, PartialEq, Eq, PartialOrd, Ord)]
|
|
struct Event {
|
|
user_id: u32,
|
|
value: i32,
|
|
timestamp: u64, // Event time in milliseconds
|
|
}
|
|
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
let mut input = worker.dataflow::<u64, _, _>(|scope| {
|
|
let (input, events) = scope.new_collection();
|
|
|
|
// Tumbling window: 5-minute (300000ms) windows
|
|
let window_size = 300_000u64;
|
|
|
|
events
|
|
.map(move |event| {
|
|
// Assign to window based on event time
|
|
let window_id = event.timestamp / window_size;
|
|
((event.user_id, window_id), event.value)
|
|
})
|
|
.reduce(|_key, values, output| {
|
|
let sum: i32 = values.iter()
|
|
.map(|(value, diff)| value * diff)
|
|
.sum();
|
|
let count: i32 = values.iter()
|
|
.map(|(_, diff)| diff)
|
|
.sum();
|
|
output.push(((sum, count), 1));
|
|
})
|
|
.inspect(|((user_id, window_id), (sum, count))| {
|
|
println!("User {}, Window {}: sum={}, count={}",
|
|
user_id, window_id, sum, count);
|
|
});
|
|
|
|
input
|
|
});
|
|
|
|
// Simulate events
|
|
input.insert(Event { user_id: 1, value: 10, timestamp: 100_000 });
|
|
input.insert(Event { user_id: 1, value: 20, timestamp: 150_000 });
|
|
input.insert(Event { user_id: 1, value: 30, timestamp: 350_000 }); // Next window
|
|
input.advance_to(1);
|
|
worker.step_while(|| input.time().less_than(&1));
|
|
}).expect("Execution failed");
|
|
}</code></pre>
|
|
<h3>Pattern 2: Sliding Window (Python)</h3>
|
|
<pre><code class="lang-python">import time
|
|
from collections import deque
|
|
from dataclasses import dataclass
|
|
from typing import Deque, Dict
|
|
import threading
|
|
|
|
@dataclass
|
|
class Event:
|
|
timestamp: float
|
|
user_id: str
|
|
value: float
|
|
|
|
class SlidingWindowAggregator:
|
|
"""Sliding window with event-time semantics"""
|
|
|
|
def __init__(self, window_size: float, slide_interval: float):
|
|
self.window_size = window_size # Window size in seconds
|
|
self.slide_interval = slide_interval # How often to emit
|
|
self.buffers: Dict[str, Deque[Event]] = {} # Per-key buffers
|
|
self.lock = threading.Lock()
|
|
|
|
def process(self, event: Event):
|
|
"""Process event and return windows that are ready"""
|
|
with self.lock:
|
|
# Initialize buffer for key
|
|
if event.user_id not in self.buffers:
|
|
self.buffers[event.user_id] = deque()
|
|
|
|
buffer = self.buffers[event.user_id]
|
|
|
|
# Add event
|
|
buffer.append(event)
|
|
|
|
# Remove events outside window
|
|
cutoff = event.timestamp - self.window_size
|
|
while buffer and buffer[0].timestamp < cutoff:
|
|
buffer.popleft()
|
|
|
|
# Compute aggregate
|
|
total = sum(e.value for e in buffer)
|
|
count = len(buffer)
|
|
avg = total / count if count > 0 else 0.0
|
|
|
|
return {
|
|
'user_id': event.user_id,
|
|
'window_start': cutoff,
|
|
'window_end': event.timestamp,
|
|
'sum': total,
|
|
'count': count,
|
|
'avg': avg
|
|
}
|
|
|
|
# Usage
|
|
def main():
|
|
aggregator = SlidingWindowAggregator(
|
|
window_size=10.0, # 10-second window
|
|
slide_interval=2.0 # Emit every 2 seconds
|
|
)
|
|
|
|
# Simulate event stream
|
|
events = [
|
|
Event(timestamp=1.0, user_id='alice', value=10),
|
|
Event(timestamp=2.0, user_id='alice', value=20),
|
|
Event(timestamp=5.0, user_id='alice', value=30),
|
|
Event(timestamp=12.0, user_id='alice', value=40), # Outside first window
|
|
]
|
|
|
|
for event in events:
|
|
result = aggregator.process(event)
|
|
print(f"Window result: {result}")
|
|
|
|
if __name__ == '__main__':
|
|
main()</code></pre>
|
|
<h3>Pattern 3: Session Windows (Go)</h3>
|
|
<pre><code class="lang-go">package main
|
|
|
|
import (
|
|
"fmt"
|
|
"sort"
|
|
"time"
|
|
)
|
|
|
|
type Event struct {
|
|
UserID string
|
|
Timestamp time.Time
|
|
Value int
|
|
}
|
|
|
|
type Session struct {
|
|
UserID string
|
|
Start time.Time
|
|
End time.Time
|
|
Events []Event
|
|
Sum int
|
|
}
|
|
|
|
type SessionWindowAggregator struct {
|
|
inactivityGap time.Duration
|
|
sessions map[string]*Session
|
|
}
|
|
|
|
func NewSessionWindowAggregator(gap time.Duration) *SessionWindowAggregator {
|
|
return &SessionWindowAggregator{
|
|
inactivityGap: gap,
|
|
sessions: make(map[string]*Session),
|
|
}
|
|
}
|
|
|
|
func (swa *SessionWindowAggregator) Process(event Event) *Session {
|
|
session, exists := swa.sessions[event.UserID]
|
|
|
|
if !exists || event.Timestamp.Sub(session.End) > swa.inactivityGap {
|
|
// Start new session
|
|
if exists {
|
|
// Emit completed session
|
|
completed := session
|
|
fmt.Printf("Session complete: %+v\n", completed)
|
|
}
|
|
|
|
session = &Session{
|
|
UserID: event.UserID,
|
|
Start: event.Timestamp,
|
|
End: event.Timestamp,
|
|
Events: []Event{event},
|
|
Sum: event.Value,
|
|
}
|
|
swa.sessions[event.UserID] = session
|
|
} else {
|
|
// Extend existing session
|
|
session.End = event.Timestamp
|
|
session.Events = append(session.Events, event)
|
|
session.Sum += event.Value
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func (swa *SessionWindowAggregator) FlushExpired(currentTime time.Time) []*Session {
|
|
var expired []*Session
|
|
|
|
for userID, session := range swa.sessions {
|
|
if currentTime.Sub(session.End) > swa.inactivityGap {
|
|
expired = append(expired, session)
|
|
delete(swa.sessions, userID)
|
|
}
|
|
}
|
|
|
|
return expired
|
|
}
|
|
|
|
func main() {
|
|
aggregator := NewSessionWindowAggregator(5 * time.Minute)
|
|
|
|
// Simulate event stream
|
|
events := []Event{
|
|
{UserID: "alice", Timestamp: time.Now(), Value: 10},
|
|
{UserID: "alice", Timestamp: time.Now().Add(1 * time.Minute), Value: 20},
|
|
{UserID: "alice", Timestamp: time.Now().Add(10 * time.Minute), Value: 30}, // New session
|
|
{UserID: "bob", Timestamp: time.Now().Add(2 * time.Minute), Value: 15},
|
|
}
|
|
|
|
for _, event := range events {
|
|
aggregator.Process(event)
|
|
}
|
|
|
|
// Flush expired sessions
|
|
expired := aggregator.FlushExpired(time.Now().Add(20 * time.Minute))
|
|
for _, session := range expired {
|
|
fmt.Printf("Expired session: %+v\n", session)
|
|
}
|
|
}</code></pre>
|
|
<h3>Pattern 4: Watermark-Based Late Data Handling (Python)</h3>
|
|
<pre><code class="lang-python">from dataclasses import dataclass
|
|
from typing import Dict, List, Optional
|
|
import heapq
|
|
|
|
@dataclass
|
|
class Event:
|
|
timestamp: int # Event time
|
|
key: str
|
|
value: int
|
|
|
|
@dataclass
|
|
class Window:
|
|
start: int
|
|
end: int
|
|
key: str
|
|
sum: int = 0
|
|
count: int = 0
|
|
|
|
class WatermarkAggregator:
|
|
"""Window aggregator with watermark-based triggering"""
|
|
|
|
def __init__(
|
|
self,
|
|
window_size: int,
|
|
allowed_lateness: int,
|
|
watermark_delay: int
|
|
):
|
|
self.window_size = window_size
|
|
self.allowed_lateness = allowed_lateness
|
|
self.watermark_delay = watermark_delay
|
|
|
|
self.windows: Dict[tuple, Window] = {} # (key, window_start) -> Window
|
|
self.max_timestamp = 0
|
|
self.watermark = 0
|
|
self.late_events: List[Event] = []
|
|
|
|
def process(self, event: Event) -> List[Window]:
|
|
"""Process event and emit completed windows"""
|
|
# Update max timestamp and watermark
|
|
self.max_timestamp = max(self.max_timestamp, event.timestamp)
|
|
self.watermark = self.max_timestamp - self.watermark_delay
|
|
|
|
# Assign to window
|
|
window_start = (event.timestamp // self.window_size) * self.window_size
|
|
window_end = window_start + self.window_size
|
|
window_key = (event.key, window_start)
|
|
|
|
# Check if too late
|
|
if event.timestamp < self.watermark - self.allowed_lateness:
|
|
self.late_events.append(event)
|
|
print(f"Late event dropped: {event}")
|
|
return []
|
|
|
|
# Update window
|
|
if window_key not in self.windows:
|
|
self.windows[window_key] = Window(
|
|
start=window_start,
|
|
end=window_end,
|
|
key=event.key
|
|
)
|
|
|
|
window = self.windows[window_key]
|
|
window.sum += event.value
|
|
window.count += 1
|
|
|
|
# Emit completed windows (watermark passed end + allowed lateness)
|
|
return self._emit_completed_windows()
|
|
|
|
def _emit_completed_windows(self) -> List[Window]:
|
|
"""Emit windows that watermark has passed"""
|
|
completed = []
|
|
to_remove = []
|
|
|
|
for (key, window_start), window in self.windows.items():
|
|
# Window complete if watermark > window_end + allowed_lateness
|
|
if self.watermark > window.end + self.allowed_lateness:
|
|
completed.append(window)
|
|
to_remove.append((key, window_start))
|
|
|
|
# Remove completed windows
|
|
for key in to_remove:
|
|
del self.windows[key]
|
|
|
|
return completed
|
|
|
|
# Usage
|
|
def main():
|
|
aggregator = WatermarkAggregator(
|
|
window_size=10, # 10-second windows
|
|
allowed_lateness=5, # Accept events up to 5 seconds late
|
|
watermark_delay=3 # Watermark = max_timestamp - 3
|
|
)
|
|
|
|
# Events arriving out of order
|
|
events = [
|
|
Event(timestamp=5, key='user1', value=10),
|
|
Event(timestamp=15, key='user1', value=20),
|
|
Event(timestamp=8, key='user1', value=15), # Late but within allowed
|
|
Event(timestamp=25, key='user1', value=30),
|
|
Event(timestamp=3, key='user1', value=5), # Too late, dropped
|
|
]
|
|
|
|
for event in events:
|
|
completed = aggregator.process(event)
|
|
for window in completed:
|
|
print(f"Window complete: {window}")
|
|
|
|
if __name__ == '__main__':
|
|
main()</code></pre>
|
|
<h3>Pattern 5: Time-Series Aggregation with Downsampling</h3>
|
|
<pre><code class="lang-python">import numpy as np
|
|
from dataclasses import dataclass
|
|
from typing import Dict, List
|
|
from collections import defaultdict
|
|
|
|
@dataclass
|
|
class TimeSeriesPoint:
|
|
timestamp: int
|
|
metric: str
|
|
value: float
|
|
|
|
class TimeSeriesAggregator:
|
|
"""Multi-resolution time-series aggregation"""
|
|
|
|
def __init__(self):
|
|
# Store multiple resolutions
|
|
self.raw: Dict[str, List[TimeSeriesPoint]] = defaultdict(list)
|
|
self.minute: Dict[str, List[tuple]] = defaultdict(list) # (timestamp, avg, min, max)
|
|
self.hour: Dict[str, List[tuple]] = defaultdict(list)
|
|
|
|
def ingest(self, point: TimeSeriesPoint):
|
|
"""Ingest raw point and update aggregations"""
|
|
self.raw[point.metric].append(point)
|
|
|
|
# Update minute-level aggregation
|
|
minute_bucket = (point.timestamp // 60) * 60
|
|
self._update_aggregation(point.metric, minute_bucket, point.value, self.minute)
|
|
|
|
# Update hour-level aggregation
|
|
hour_bucket = (point.timestamp // 3600) * 3600
|
|
self._update_aggregation(point.metric, hour_bucket, point.value, self.hour)
|
|
|
|
def _update_aggregation(
|
|
self,
|
|
metric: str,
|
|
bucket: int,
|
|
value: float,
|
|
storage: Dict[str, List[tuple]]
|
|
):
|
|
"""Update aggregation bucket"""
|
|
buckets = storage[metric]
|
|
|
|
# Find or create bucket
|
|
if not buckets or buckets[-1][0] != bucket:
|
|
# New bucket
|
|
buckets.append((bucket, value, value, value, 1)) # (time, sum, min, max, count)
|
|
else:
|
|
# Update existing bucket
|
|
old_time, old_sum, old_min, old_max, old_count = buckets[-1]
|
|
buckets[-1] = (
|
|
old_time,
|
|
old_sum + value,
|
|
min(old_min, value),
|
|
max(old_max, value),
|
|
old_count + 1
|
|
)
|
|
|
|
def query(
|
|
self,
|
|
metric: str,
|
|
start: int,
|
|
end: int,
|
|
resolution: str = 'minute'
|
|
) -> List[tuple]:
|
|
"""Query aggregated data at specified resolution"""
|
|
storage = {
|
|
'minute': self.minute,
|
|
'hour': self.hour,
|
|
'raw': self.raw
|
|
}[resolution]
|
|
|
|
if resolution == 'raw':
|
|
points = storage[metric]
|
|
filtered = [p for p in points if start <= p.timestamp <= end]
|
|
return [(p.timestamp, p.value) for p in filtered]
|
|
|
|
buckets = storage[metric]
|
|
filtered = [
|
|
(timestamp, total/count, min_val, max_val)
|
|
for timestamp, total, min_val, max_val, count in buckets
|
|
if start <= timestamp <= end
|
|
]
|
|
|
|
return filtered
|
|
|
|
# Usage
|
|
def main():
|
|
agg = TimeSeriesAggregator()
|
|
|
|
# Ingest high-frequency data
|
|
for i in range(7200): # 2 hours of second-level data
|
|
point = TimeSeriesPoint(
|
|
timestamp=i,
|
|
metric='cpu_usage',
|
|
value=50 + 10 * np.sin(i / 60.0) # Simulate pattern
|
|
)
|
|
agg.ingest(point)
|
|
|
|
# Query at different resolutions
|
|
minute_data = agg.query('cpu_usage', 0, 3600, resolution='minute')
|
|
print(f"Minute-level aggregation: {len(minute_data)} points")
|
|
|
|
hour_data = agg.query('cpu_usage', 0, 7200, resolution='hour')
|
|
print(f"Hour-level aggregation: {len(hour_data)} points")
|
|
|
|
# Each point: (timestamp, avg, min, max)
|
|
for timestamp, avg, min_val, max_val in hour_data:
|
|
print(f"Hour {timestamp//3600}: avg={avg:.2f}, min={min_val:.2f}, max={max_val:.2f}")
|
|
|
|
if __name__ == '__main__':
|
|
main()</code></pre>
|
|
<h3>Pattern 6: Top-K Streaming Aggregation</h3>
|
|
<pre><code class="lang-rust">use std::collections::BinaryHeap;
|
|
use std::cmp::Reverse;
|
|
|
|
#[derive(Debug, Clone, Eq, PartialEq)]
|
|
struct Item {
|
|
key: String,
|
|
count: usize,
|
|
}
|
|
|
|
impl Ord for Item {
|
|
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
|
|
self.count.cmp(&other.count)
|
|
}
|
|
}
|
|
|
|
impl PartialOrd for Item {
|
|
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
|
|
Some(self.cmp(other))
|
|
}
|
|
}
|
|
|
|
struct TopKAggregator {
|
|
k: usize,
|
|
counts: std::collections::HashMap<String, usize>,
|
|
heap: BinaryHeap<Reverse<Item>>,
|
|
}
|
|
|
|
impl TopKAggregator {
|
|
fn new(k: usize) -> Self {
|
|
Self {
|
|
k,
|
|
counts: std::collections::HashMap::new(),
|
|
heap: BinaryHeap::new(),
|
|
}
|
|
}
|
|
|
|
fn update(&mut self, key: String) {
|
|
// Update count
|
|
let count = self.counts.entry(key.clone()).or_insert(0);
|
|
*count += 1;
|
|
|
|
// Maintain top-K heap
|
|
self.rebuild_heap();
|
|
}
|
|
|
|
fn rebuild_heap(&mut self) {
|
|
self.heap.clear();
|
|
|
|
for (key, count) in &self.counts {
|
|
let item = Item {
|
|
key: key.clone(),
|
|
count: *count,
|
|
};
|
|
|
|
if self.heap.len() < self.k {
|
|
self.heap.push(Reverse(item));
|
|
} else if let Some(Reverse(min)) = self.heap.peek() {
|
|
if item.count > min.count {
|
|
self.heap.pop();
|
|
self.heap.push(Reverse(item));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
fn top_k(&self) -> Vec<Item> {
|
|
let mut items: Vec<_> = self.heap.iter()
|
|
.map(|Reverse(item)| item.clone())
|
|
.collect();
|
|
items.sort_by(|a, b| b.count.cmp(&a.count));
|
|
items
|
|
}
|
|
}
|
|
|
|
fn main() {
|
|
let mut aggregator = TopKAggregator::new(3);
|
|
|
|
// Simulate event stream
|
|
let events = vec!["apple", "banana", "apple", "cherry", "apple", "banana", "date"];
|
|
|
|
for event in events {
|
|
aggregator.update(event.to_string());
|
|
}
|
|
|
|
println!("Top 3 items:");
|
|
for item in aggregator.top_k() {
|
|
println!(" {}: {}", item.key, item.count);
|
|
}
|
|
}</code></pre>
|
|
<h2>Quick Reference</h2>
|
|
<h3>Window Selection Guide</h3>
|
|
<pre><code>Use Tumbling: Periodic reports, non-overlapping batches
|
|
Use Sliding: Moving averages, continuous metrics
|
|
Use Session: User behavior, conversation analysis
|
|
Use Global: Unbounded state, all-time aggregations</code></pre>
|
|
<h3>Watermark Strategies</h3>
|
|
<pre><code class="lang-python"># Bounded delay
|
|
watermark = max_timestamp - fixed_delay
|
|
|
|
# Percentile-based
|
|
watermark = percentile(timestamps, 99) # Allow 1% late
|
|
|
|
# Heuristic
|
|
watermark = max_timestamp - 2 * stddev(inter_arrival_time)</code></pre>
|
|
<h3>Time Extraction</h3>
|
|
<pre><code class="lang-rust">// Extract event time from data
|
|
let event_time = |event: &Event| event.timestamp;
|
|
|
|
// Assign to window
|
|
let window_id = event.timestamp / window_size;
|
|
|
|
// Session gap check
|
|
if current_time - last_event_time > session_gap {
|
|
// Start new session
|
|
}</code></pre>
|
|
<h2>Performance Optimization</h2>
|
|
<pre><code>Reduce State Size
|
|
→ Compact old windows
|
|
→ Use approximate algorithms (HyperLogLog, Count-Min Sketch)
|
|
→ Expire inactive keys
|
|
|
|
Batch Processing
|
|
→ Buffer events before aggregating
|
|
→ Periodic window evaluation
|
|
|
|
Incremental Updates
|
|
→ Use differential dataflow for efficient re-aggregation
|
|
→ Maintain summary statistics (sum, count) instead of raw data</code></pre>
|
|
<h2>Anti-Patterns</h2>
|
|
<pre><code>❌ NEVER: Use processing time for event-time logic
|
|
→ Results depend on processing speed, not actual event timing
|
|
|
|
❌ NEVER: Wait indefinitely for late data
|
|
→ Set allowed lateness bounds
|
|
|
|
❌ NEVER: Store unbounded state in global windows
|
|
→ Use approximate algorithms or periodic cleanup
|
|
|
|
❌ NEVER: Ignore watermarks
|
|
→ Windows never complete, state grows unbounded
|
|
|
|
❌ NEVER: Use sliding windows with small slide interval on high-volume streams
|
|
→ Creates many overlapping windows, high memory usage
|
|
|
|
❌ NEVER: Recompute entire window on late data
|
|
→ Use incremental updates
|
|
|
|
❌ NEVER: Assume events arrive in order
|
|
→ Always design for out-of-order delivery
|
|
|
|
❌ NEVER: Use session windows without timeouts
|
|
→ Sessions never close, memory leak
|
|
|
|
❌ NEVER: Emit window before watermark passes
|
|
→ Incomplete results
|
|
|
|
❌ NEVER: Drop late data without logging
|
|
→ Monitor late data rates for tuning</code></pre>
|
|
<h2>Related Skills</h2>
|
|
<ul><li><code>timely-dataflow.md</code> - Foundation for windowing with progress tracking</li><li><code>differential-dataflow.md</code> - Incremental window updates</li><li><code>dataflow-coordination.md</code> - Watermarks and coordination</li><li><code>stream-processing.md</code> - High-level stream processing with Kafka</li></ul>
|
|
<p><strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/data-streaming-aggregations.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |