mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
Every page now loads /webmcp.js, scripts/webmcp.ts with its types stripped (npm run webmcp). Where the browser has document.modelContext it registers two read-only tools: search_skills over the published skills index, and get_skill, which returns a skill's markdown checked against the index digest. The markdown alternate link moves from build.py to site.py, so CI holds every page to it. The API catalog gains the site itself (llms.txt and the skills index) and hanzo.ai/agents; the live discovery check now fetches every catalog link. tests/site.mjs drives the tools through a recording modelContext, and checks a browser without one loads clean.
674 lines
29 KiB
HTML
674 lines
29 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>data-timely-dataflow</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/data-timely-dataflow">
|
|
<link rel="alternate" type="text/markdown" href="/data-timely-dataflow.md">
|
|
<script type="module" src="/webmcp.js"></script>
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/data-timely-dataflow.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>data-timely-dataflow</h1><p class="lede">Timely dataflow framework for low-latency, high-throughput streaming computation with progress tracking</p><h1>Timely Dataflow</h1>
|
|
<p><strong>Scope</strong>: Timely dataflow graphs, progress tracking, dataflow operators, low-latency streaming <strong>Lines</strong>: 380 <strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
|
|
<h2>When to Use This Skill</h2>
|
|
<p>Use this skill when:</p>
|
|
<ul><li>Building low-latency streaming computations with Rust</li><li>Implementing dataflow graphs with explicit progress tracking</li><li>Creating custom streaming operators with timely semantics</li><li>Processing unbounded data streams with strong consistency</li><li>Building foundation for differential dataflow applications</li><li>Designing systems requiring fine-grained progress tracking</li><li>Implementing cyclic dataflow computations (iterative algorithms)</li></ul>
|
|
<h2>Core Concepts</h2>
|
|
<h3>Timely Dataflow Architecture</h3>
|
|
<pre><code>Dataflow Graph
|
|
→ Directed graph of operators and channels
|
|
→ Data flows through edges, computation at nodes
|
|
→ Explicit timestamps on all data
|
|
→ Progress tracking across entire graph
|
|
|
|
Workers
|
|
→ Single-threaded execution units
|
|
→ Shared-nothing architecture
|
|
→ Communication via message passing
|
|
→ Each worker processes partition of data
|
|
|
|
Timestamps
|
|
→ Logical time, not wall-clock time
|
|
→ Partial order (can have multiple dimensions)
|
|
→ Enables reasoning about data completeness
|
|
→ Foundation for coordination-free execution</code></pre>
|
|
<h3>Progress Tracking Protocol</h3>
|
|
<pre><code>Capabilities
|
|
→ Token representing ability to send data at timestamp
|
|
→ Downgrade: Move to later timestamp
|
|
→ Drop: Relinquish ability to send at timestamp
|
|
|
|
Frontier
|
|
→ Set of timestamps that may still be produced
|
|
→ Minimal set of active timestamps
|
|
→ Advances when capabilities downgraded/dropped
|
|
|
|
Notifications
|
|
→ Triggered when frontier advances past timestamp
|
|
→ Enables stateful operators to emit results
|
|
→ Guarantees: All data at timestamp T received</code></pre>
|
|
<h3>Dataflow Operators</h3>
|
|
<pre><code>Unary Operators
|
|
→ Transform single input stream
|
|
→ Examples: map, filter, inspect
|
|
|
|
Binary Operators
|
|
→ Combine two input streams
|
|
→ Examples: concat, join (via state)
|
|
|
|
Feedback/Iteration
|
|
→ Connect output back to input
|
|
→ Creates cycles in dataflow graph
|
|
→ Requires careful timestamp management
|
|
|
|
Stateful Operators
|
|
→ Maintain state indexed by time
|
|
→ Emit results when frontier advances
|
|
→ Examples: aggregations, joins, windows</code></pre>
|
|
<h2>Patterns</h2>
|
|
<h3>Pattern 1: Basic Dataflow Pipeline</h3>
|
|
<pre><code class="lang-rust">// Cargo.toml
|
|
// [dependencies]
|
|
// timely = "0.12"
|
|
|
|
use timely::dataflow::operators::{Inspect, ToStream};
|
|
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
worker.dataflow::<(), _, _>(|scope| {
|
|
// Create input stream
|
|
(0..10)
|
|
.to_stream(scope)
|
|
.inspect(|x| println!("Observed: {}", x));
|
|
});
|
|
}).expect("Execution failed");
|
|
}
|
|
|
|
// Multi-worker execution:
|
|
// cargo run -- -w 2 # 2 workers
|
|
// cargo run -- -w 4 -p 2 # 4 workers, 2 processes</code></pre>
|
|
<h3>Pattern 2: Custom Unary Operator</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
|
|
use timely::dataflow::channels::pact::Pipeline;
|
|
use timely::dataflow::operators::generic::operator::Operator;
|
|
|
|
// Extension trait for custom operator
|
|
trait Multiply<G: Scope> {
|
|
fn multiply(&self, factor: i32) -> Stream<G, i32>;
|
|
}
|
|
|
|
impl<G: Scope> Multiply<G> for Stream<G, i32> {
|
|
fn multiply(&self, factor: i32) -> Stream<G, i32> {
|
|
self.unary(
|
|
Pipeline, // Exchange strategy
|
|
"Multiply", // Operator name
|
|
move |_capability, _info| {
|
|
move |input, output| {
|
|
input.for_each(|time, data| {
|
|
let mut session = output.session(&time);
|
|
for datum in data.iter() {
|
|
session.give(datum * factor);
|
|
}
|
|
});
|
|
}
|
|
},
|
|
)
|
|
}
|
|
}
|
|
|
|
// Usage
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
worker.dataflow::<(), _, _>(|scope| {
|
|
(0..10)
|
|
.to_stream(scope)
|
|
.multiply(5)
|
|
.inspect(|x| println!("Result: {}", x));
|
|
});
|
|
}).expect("Execution failed");
|
|
}</code></pre>
|
|
<h3>Pattern 3: Stateful Aggregation with Progress Tracking</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
|
|
use timely::dataflow::channels::pact::Exchange;
|
|
use timely::dataflow::operators::generic::operator::Operator;
|
|
use std::collections::HashMap;
|
|
|
|
trait Aggregate<G: Scope<Timestamp = u64>> {
|
|
fn count_by_key(&self) -> Stream<G, (String, usize)>;
|
|
}
|
|
|
|
impl<G: Scope<Timestamp = u64>> Aggregate<G> for Stream<G, (String, i32)> {
|
|
fn count_by_key(&self) -> Stream<G, (String, usize)> {
|
|
self.unary_notify(
|
|
Exchange::new(|x: &(String, i32)| x.0.len() as u64),
|
|
"CountByKey",
|
|
vec![], // Initial frontier
|
|
move |input, output, notificator| {
|
|
// State: key -> (count, timestamp)
|
|
let mut state: HashMap<String, (usize, u64)> = HashMap::new();
|
|
|
|
// Process input
|
|
input.for_each(|time, data| {
|
|
// Request notification when this time completes
|
|
notificator.notify_at(time.retain());
|
|
|
|
for (key, _value) in data.iter() {
|
|
let entry = state.entry(key.clone())
|
|
.or_insert((0, time.time().clone()));
|
|
entry.0 += 1;
|
|
entry.1 = time.time().clone();
|
|
}
|
|
});
|
|
|
|
// Emit results when frontier advances
|
|
notificator.for_each(|time, _count, _notify| {
|
|
let mut session = output.session(&time);
|
|
|
|
// Find keys at this timestamp
|
|
let keys: Vec<String> = state.iter()
|
|
.filter(|(_, (_, t))| t == time.time())
|
|
.map(|(k, _)| k.clone())
|
|
.collect();
|
|
|
|
for key in keys {
|
|
if let Some((count, _)) = state.remove(&key) {
|
|
session.give((key, count));
|
|
}
|
|
}
|
|
});
|
|
},
|
|
)
|
|
}
|
|
}
|
|
|
|
// Usage
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
use timely::dataflow::operators::ToStream;
|
|
|
|
worker.dataflow::<u64, _, _>(|scope| {
|
|
vec![
|
|
("apple".to_string(), 1),
|
|
("banana".to_string(), 1),
|
|
("apple".to_string(), 2),
|
|
]
|
|
.to_stream(scope)
|
|
.count_by_key()
|
|
.inspect(|(key, count)| {
|
|
println!("Key: {}, Count: {}", key, count);
|
|
});
|
|
});
|
|
}).expect("Execution failed");
|
|
}</code></pre>
|
|
<h3>Pattern 4: Iterative Computation (Feedback Loop)</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
|
|
use timely::dataflow::operators::{Inspect, ToStream, Map, Filter, Feedback, ConnectLoop};
|
|
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
worker.dataflow::<u64, _, _>(|scope| {
|
|
// Create feedback loop for iterative computation
|
|
let (handle, cycle) = scope.feedback(1); // Summary: Product<u64, u32>
|
|
|
|
// Initial values
|
|
vec![1, 2, 3, 4, 5]
|
|
.to_stream(scope)
|
|
.map(|x| (x, 0)) // (value, iteration)
|
|
.concat(&cycle)
|
|
.inspect(|(val, iter)| {
|
|
println!("Iteration {}: value = {}", iter, val);
|
|
})
|
|
.filter(|(val, iter)| *val < 100 && *iter < 10) // Termination
|
|
.map(|(val, iter)| (val * 2, iter + 1)) // Computation
|
|
.connect_loop(handle);
|
|
});
|
|
}).expect("Execution failed");
|
|
}
|
|
|
|
// Output:
|
|
// Iteration 0: value = 1
|
|
// Iteration 0: value = 2
|
|
// ...
|
|
// Iteration 1: value = 2
|
|
// Iteration 1: value = 4
|
|
// ... (continues until termination)</code></pre>
|
|
<h3>Pattern 5: Multi-Input Operator (Barrier Pattern)</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
|
|
use timely::dataflow::channels::pact::Pipeline;
|
|
use timely::dataflow::operators::generic::builder_rc::OperatorBuilder;
|
|
use std::collections::HashMap;
|
|
|
|
fn barrier_join<G: Scope>(
|
|
stream1: &Stream<G, (String, i32)>,
|
|
stream2: &Stream<G, (String, i32)>,
|
|
) -> Stream<G, (String, i32, i32)> {
|
|
let mut builder = OperatorBuilder::new("BarrierJoin".to_string(), stream1.scope());
|
|
|
|
let mut input1 = builder.new_input(stream1, Pipeline);
|
|
let mut input2 = builder.new_input(stream2, Pipeline);
|
|
let (mut output, stream) = builder.new_output();
|
|
|
|
builder.build(move |_capability| {
|
|
// State indexed by time
|
|
let mut buffer1: HashMap<G::Timestamp, Vec<(String, i32)>> = HashMap::new();
|
|
let mut buffer2: HashMap<G::Timestamp, Vec<(String, i32)>> = HashMap::new();
|
|
|
|
move |_frontiers| {
|
|
// Read input1
|
|
input1.for_each(|time, data| {
|
|
buffer1.entry(time.time().clone())
|
|
.or_insert_with(Vec::new)
|
|
.extend(data.iter().cloned());
|
|
});
|
|
|
|
// Read input2
|
|
input2.for_each(|time, data| {
|
|
buffer2.entry(time.time().clone())
|
|
.or_insert_with(Vec::new)
|
|
.extend(data.iter().cloned());
|
|
});
|
|
|
|
// Process completed times
|
|
let completed: Vec<G::Timestamp> = buffer1.keys()
|
|
.filter(|t| buffer2.contains_key(t))
|
|
.cloned()
|
|
.collect();
|
|
|
|
for time in completed {
|
|
if let (Some(data1), Some(data2)) = (buffer1.remove(&time), buffer2.remove(&time)) {
|
|
let mut session = output.session(&time);
|
|
|
|
// Perform join
|
|
for (key1, val1) in data1 {
|
|
for (key2, val2) in &data2 {
|
|
if key1 == *key2 {
|
|
session.give((key1.clone(), val1, *val2));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
});
|
|
|
|
stream
|
|
}</code></pre>
|
|
<h3>Pattern 6: External Input Source (Probing)</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::operators::{Inspect, Probe};
|
|
use timely::dataflow::operators::input::Handle;
|
|
|
|
fn main() {
|
|
timely::execute_from_args(std::env::args(), |worker| {
|
|
let mut input = Handle::new();
|
|
let probe = worker.dataflow(|scope| {
|
|
input.to_stream(scope)
|
|
.inspect(|x| println!("Saw: {}", x))
|
|
.probe()
|
|
});
|
|
|
|
// External loop feeding data
|
|
for round in 0..10 {
|
|
input.send(round);
|
|
input.advance_to(round + 1);
|
|
|
|
// Wait for computation to complete
|
|
while probe.less_than(input.time()) {
|
|
worker.step();
|
|
}
|
|
}
|
|
}).expect("Execution failed");
|
|
}</code></pre>
|
|
<h2>Architecture Patterns</h2>
|
|
<h3>Timestamp Structure</h3>
|
|
<pre><code>Single Dimension (u64)
|
|
→ Linear time progression
|
|
→ Use for: Simple pipelines, no iteration
|
|
→ Example: Event time in milliseconds
|
|
|
|
Product (Outer, Inner)
|
|
→ Product<u64, u32> for iterative computation
|
|
→ Outer: Input time, Inner: Iteration number
|
|
→ Allows feedback loops with progress guarantees
|
|
|
|
Custom Lattice
|
|
→ Implement PartialOrder trait
|
|
→ Use for: Complex coordination patterns
|
|
→ Example: Vector clocks, causal timestamps</code></pre>
|
|
<h3>Operator Exchange Patterns</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::channels::pact::{Pipeline, Exchange};
|
|
|
|
// Pipeline: No data movement (data stays on same worker)
|
|
stream.unary(Pipeline, "Op", |cap, info| { ... })
|
|
|
|
// Exchange: Partition data by hash of key
|
|
stream.unary(Exchange::new(|x: &T| hash(x)), "Op", |cap, info| { ... })
|
|
|
|
// Broadcast: Send all data to all workers (expensive)
|
|
// Custom Pact: Implement ParallelizationContract trait</code></pre>
|
|
<h2>Performance Optimization</h2>
|
|
<h3>Worker Configuration</h3>
|
|
<pre><code class="lang-bash"># Single machine, multiple threads
|
|
cargo run -- -w 4
|
|
|
|
# Multiple processes (distributed)
|
|
cargo run -- -w 8 -p 2 # 8 workers, 2 processes (4 workers each)
|
|
cargo run -- -w 8 -p 4 # 8 workers, 4 processes (2 workers each)
|
|
|
|
# With specific hosts
|
|
cargo run -- -w 4 -p 2 -h worker1,worker2</code></pre>
|
|
<h3>Batch Size Tuning</h3>
|
|
<pre><code class="lang-rust">// Configure batch sizes for better throughput
|
|
use timely::dataflow::operators::Operator;
|
|
|
|
stream.unary_frontier(
|
|
Pipeline,
|
|
"BatchedOp",
|
|
|_capability, _info| {
|
|
let mut buffer = Vec::new();
|
|
const BATCH_SIZE: usize = 1000;
|
|
|
|
move |input, output| {
|
|
input.for_each(|time, data| {
|
|
buffer.extend(data.iter().cloned());
|
|
|
|
if buffer.len() >= BATCH_SIZE {
|
|
let mut session = output.session(&time);
|
|
for item in buffer.drain(..) {
|
|
session.give(item);
|
|
}
|
|
}
|
|
});
|
|
}
|
|
},
|
|
)</code></pre>
|
|
<h3>Memory Management</h3>
|
|
<pre><code class="lang-rust">// Use capabilities efficiently
|
|
use timely::progress::Timestamp;
|
|
|
|
// Downgrade capabilities when possible
|
|
capability.downgrade(&new_time);
|
|
|
|
// Drop capabilities when done
|
|
drop(capability);
|
|
|
|
// Clean up old state based on frontier
|
|
let frontier = &frontiers[0];
|
|
state.retain(|time, _| !frontier.less_equal(time));</code></pre>
|
|
<h2>Quick Reference</h2>
|
|
<h3>Common Operators</h3>
|
|
<pre><code class="lang-rust">use timely::dataflow::operators::*;
|
|
|
|
// Transform
|
|
stream.map(|x| x * 2)
|
|
stream.flat_map(|x| 0..x)
|
|
stream.filter(|x| *x > 5)
|
|
|
|
// Inspect (side effects)
|
|
stream.inspect(|x| println!("{}", x))
|
|
stream.inspect_time(|t, x| println!("{:?}: {}", t, x))
|
|
|
|
// Combine
|
|
stream1.concat(&stream2)
|
|
stream1.binary(&stream2, ...) // Custom binary operator
|
|
|
|
// Feedback
|
|
let (handle, cycle) = scope.feedback(summary);
|
|
stream.connect_loop(handle)
|
|
|
|
// Control
|
|
stream.probe() // Track progress
|
|
input.advance_to(time) // Advance input frontier</code></pre>
|
|
<h3>Debugging</h3>
|
|
<pre><code class="lang-rust">// Log operator names and structure
|
|
std::env::set_var("TIMELY_WORKER_LOG_ADDR", "localhost:51317");
|
|
|
|
// Progress tracking
|
|
stream.inspect_time(|time, data| {
|
|
eprintln!("Time {:?}: {:?}", time, data);
|
|
})
|
|
|
|
// Frontier inspection
|
|
use timely::dataflow::operators::probe::Handle as ProbeHandle;
|
|
let probe = stream.probe();
|
|
println!("Frontier: {:?}", probe.with_frontier(|f| f.to_vec()));</code></pre>
|
|
<h2>Anti-Patterns</h2>
|
|
<pre><code>❌ NEVER: Hold capabilities indefinitely
|
|
→ Prevents frontier advancement, blocks progress
|
|
|
|
❌ NEVER: Ignore timestamp management in feedback loops
|
|
→ Can create non-monotonic timestamps, breaks progress tracking
|
|
|
|
❌ NEVER: Perform blocking I/O in operators
|
|
→ Single-threaded workers, blocks entire worker
|
|
|
|
❌ NEVER: Accumulate unbounded state without cleanup
|
|
→ Use frontier to determine safe cleanup points
|
|
|
|
❌ NEVER: Use exchange for small data volumes
|
|
→ Overhead of shuffling > benefit, use Pipeline
|
|
|
|
❌ NEVER: Create too many operators
|
|
→ Overhead in progress tracking, combine logic when possible
|
|
|
|
❌ NEVER: Forget to advance input frontiers
|
|
→ Computation stalls waiting for future data
|
|
|
|
❌ NEVER: Use global state across workers
|
|
→ Shared-nothing architecture, use message passing
|
|
|
|
❌ NEVER: Assume data ordering beyond timestamp
|
|
→ Only timestamp order guaranteed, not arrival order
|
|
|
|
❌ NEVER: Mix wall-clock time with logical timestamps
|
|
→ Logical timestamps for correctness, wall-clock for monitoring only</code></pre>
|
|
<h2>Related Skills</h2>
|
|
<ul><li><code>differential-dataflow.md</code> - Incremental computation on top of timely dataflow</li><li><code>dataflow-coordination.md</code> - Advanced coordination patterns and barriers</li><li><code>streaming-aggregations.md</code> - Window-based aggregations using timely semantics</li><li><code>stream-processing.md</code> - Higher-level stream processing frameworks</li></ul>
|
|
<p><strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/data-timely-dataflow.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |