Files
skills/public/data-timely-dataflow.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

674 lines
29 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>data-timely-dataflow</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/data-timely-dataflow.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/data-timely-dataflow">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/data-timely-dataflow.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>data-timely-dataflow</h1><p class="lede">Timely dataflow framework for low-latency, high-throughput streaming computation with progress tracking</p><h1>Timely Dataflow</h1>
<p><strong>Scope</strong>: Timely dataflow graphs, progress tracking, dataflow operators, low-latency streaming <strong>Lines</strong>: 380 <strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<h2>When to Use This Skill</h2>
<p>Use this skill when:</p>
<ul><li>Building low-latency streaming computations with Rust</li><li>Implementing dataflow graphs with explicit progress tracking</li><li>Creating custom streaming operators with timely semantics</li><li>Processing unbounded data streams with strong consistency</li><li>Building foundation for differential dataflow applications</li><li>Designing systems requiring fine-grained progress tracking</li><li>Implementing cyclic dataflow computations (iterative algorithms)</li></ul>
<h2>Core Concepts</h2>
<h3>Timely Dataflow Architecture</h3>
<pre><code>Dataflow Graph
→ Directed graph of operators and channels
→ Data flows through edges, computation at nodes
→ Explicit timestamps on all data
→ Progress tracking across entire graph
Workers
→ Single-threaded execution units
→ Shared-nothing architecture
→ Communication via message passing
→ Each worker processes partition of data
Timestamps
→ Logical time, not wall-clock time
→ Partial order (can have multiple dimensions)
→ Enables reasoning about data completeness
→ Foundation for coordination-free execution</code></pre>
<h3>Progress Tracking Protocol</h3>
<pre><code>Capabilities
→ Token representing ability to send data at timestamp
→ Downgrade: Move to later timestamp
→ Drop: Relinquish ability to send at timestamp
Frontier
→ Set of timestamps that may still be produced
→ Minimal set of active timestamps
→ Advances when capabilities downgraded/dropped
Notifications
→ Triggered when frontier advances past timestamp
→ Enables stateful operators to emit results
→ Guarantees: All data at timestamp T received</code></pre>
<h3>Dataflow Operators</h3>
<pre><code>Unary Operators
→ Transform single input stream
→ Examples: map, filter, inspect
Binary Operators
→ Combine two input streams
→ Examples: concat, join (via state)
Feedback/Iteration
→ Connect output back to input
→ Creates cycles in dataflow graph
→ Requires careful timestamp management
Stateful Operators
→ Maintain state indexed by time
→ Emit results when frontier advances
→ Examples: aggregations, joins, windows</code></pre>
<h2>Patterns</h2>
<h3>Pattern 1: Basic Dataflow Pipeline</h3>
<pre><code class="lang-rust">// Cargo.toml
// [dependencies]
// timely = &quot;0.12&quot;
use timely::dataflow::operators::{Inspect, ToStream};
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
worker.dataflow::&lt;(), _, _&gt;(|scope| {
// Create input stream
(0..10)
.to_stream(scope)
.inspect(|x| println!(&quot;Observed: {}&quot;, x));
});
}).expect(&quot;Execution failed&quot;);
}
// Multi-worker execution:
// cargo run -- -w 2 # 2 workers
// cargo run -- -w 4 -p 2 # 4 workers, 2 processes</code></pre>
<h3>Pattern 2: Custom Unary Operator</h3>
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
use timely::dataflow::channels::pact::Pipeline;
use timely::dataflow::operators::generic::operator::Operator;
// Extension trait for custom operator
trait Multiply&lt;G: Scope&gt; {
fn multiply(&amp;self, factor: i32) -&gt; Stream&lt;G, i32&gt;;
}
impl&lt;G: Scope&gt; Multiply&lt;G&gt; for Stream&lt;G, i32&gt; {
fn multiply(&amp;self, factor: i32) -&gt; Stream&lt;G, i32&gt; {
self.unary(
Pipeline, // Exchange strategy
&quot;Multiply&quot;, // Operator name
move |_capability, _info| {
move |input, output| {
input.for_each(|time, data| {
let mut session = output.session(&amp;time);
for datum in data.iter() {
session.give(datum * factor);
}
});
}
},
)
}
}
// Usage
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
worker.dataflow::&lt;(), _, _&gt;(|scope| {
(0..10)
.to_stream(scope)
.multiply(5)
.inspect(|x| println!(&quot;Result: {}&quot;, x));
});
}).expect(&quot;Execution failed&quot;);
}</code></pre>
<h3>Pattern 3: Stateful Aggregation with Progress Tracking</h3>
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
use timely::dataflow::channels::pact::Exchange;
use timely::dataflow::operators::generic::operator::Operator;
use std::collections::HashMap;
trait Aggregate&lt;G: Scope&lt;Timestamp = u64&gt;&gt; {
fn count_by_key(&amp;self) -&gt; Stream&lt;G, (String, usize)&gt;;
}
impl&lt;G: Scope&lt;Timestamp = u64&gt;&gt; Aggregate&lt;G&gt; for Stream&lt;G, (String, i32)&gt; {
fn count_by_key(&amp;self) -&gt; Stream&lt;G, (String, usize)&gt; {
self.unary_notify(
Exchange::new(|x: &amp;(String, i32)| x.0.len() as u64),
&quot;CountByKey&quot;,
vec![], // Initial frontier
move |input, output, notificator| {
// State: key -&gt; (count, timestamp)
let mut state: HashMap&lt;String, (usize, u64)&gt; = HashMap::new();
// Process input
input.for_each(|time, data| {
// Request notification when this time completes
notificator.notify_at(time.retain());
for (key, _value) in data.iter() {
let entry = state.entry(key.clone())
.or_insert((0, time.time().clone()));
entry.0 += 1;
entry.1 = time.time().clone();
}
});
// Emit results when frontier advances
notificator.for_each(|time, _count, _notify| {
let mut session = output.session(&amp;time);
// Find keys at this timestamp
let keys: Vec&lt;String&gt; = state.iter()
.filter(|(_, (_, t))| t == time.time())
.map(|(k, _)| k.clone())
.collect();
for key in keys {
if let Some((count, _)) = state.remove(&amp;key) {
session.give((key, count));
}
}
});
},
)
}
}
// Usage
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
use timely::dataflow::operators::ToStream;
worker.dataflow::&lt;u64, _, _&gt;(|scope| {
vec![
(&quot;apple&quot;.to_string(), 1),
(&quot;banana&quot;.to_string(), 1),
(&quot;apple&quot;.to_string(), 2),
]
.to_stream(scope)
.count_by_key()
.inspect(|(key, count)| {
println!(&quot;Key: {}, Count: {}&quot;, key, count);
});
});
}).expect(&quot;Execution failed&quot;);
}</code></pre>
<h3>Pattern 4: Iterative Computation (Feedback Loop)</h3>
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
use timely::dataflow::operators::{Inspect, ToStream, Map, Filter, Feedback, ConnectLoop};
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
worker.dataflow::&lt;u64, _, _&gt;(|scope| {
// Create feedback loop for iterative computation
let (handle, cycle) = scope.feedback(1); // Summary: Product&lt;u64, u32&gt;
// Initial values
vec![1, 2, 3, 4, 5]
.to_stream(scope)
.map(|x| (x, 0)) // (value, iteration)
.concat(&amp;cycle)
.inspect(|(val, iter)| {
println!(&quot;Iteration {}: value = {}&quot;, iter, val);
})
.filter(|(val, iter)| *val &lt; 100 &amp;&amp; *iter &lt; 10) // Termination
.map(|(val, iter)| (val * 2, iter + 1)) // Computation
.connect_loop(handle);
});
}).expect(&quot;Execution failed&quot;);
}
// Output:
// Iteration 0: value = 1
// Iteration 0: value = 2
// ...
// Iteration 1: value = 2
// Iteration 1: value = 4
// ... (continues until termination)</code></pre>
<h3>Pattern 5: Multi-Input Operator (Barrier Pattern)</h3>
<pre><code class="lang-rust">use timely::dataflow::{Scope, Stream};
use timely::dataflow::channels::pact::Pipeline;
use timely::dataflow::operators::generic::builder_rc::OperatorBuilder;
use std::collections::HashMap;
fn barrier_join&lt;G: Scope&gt;(
stream1: &amp;Stream&lt;G, (String, i32)&gt;,
stream2: &amp;Stream&lt;G, (String, i32)&gt;,
) -&gt; Stream&lt;G, (String, i32, i32)&gt; {
let mut builder = OperatorBuilder::new(&quot;BarrierJoin&quot;.to_string(), stream1.scope());
let mut input1 = builder.new_input(stream1, Pipeline);
let mut input2 = builder.new_input(stream2, Pipeline);
let (mut output, stream) = builder.new_output();
builder.build(move |_capability| {
// State indexed by time
let mut buffer1: HashMap&lt;G::Timestamp, Vec&lt;(String, i32)&gt;&gt; = HashMap::new();
let mut buffer2: HashMap&lt;G::Timestamp, Vec&lt;(String, i32)&gt;&gt; = HashMap::new();
move |_frontiers| {
// Read input1
input1.for_each(|time, data| {
buffer1.entry(time.time().clone())
.or_insert_with(Vec::new)
.extend(data.iter().cloned());
});
// Read input2
input2.for_each(|time, data| {
buffer2.entry(time.time().clone())
.or_insert_with(Vec::new)
.extend(data.iter().cloned());
});
// Process completed times
let completed: Vec&lt;G::Timestamp&gt; = buffer1.keys()
.filter(|t| buffer2.contains_key(t))
.cloned()
.collect();
for time in completed {
if let (Some(data1), Some(data2)) = (buffer1.remove(&amp;time), buffer2.remove(&amp;time)) {
let mut session = output.session(&amp;time);
// Perform join
for (key1, val1) in data1 {
for (key2, val2) in &amp;data2 {
if key1 == *key2 {
session.give((key1.clone(), val1, *val2));
}
}
}
}
}
}
});
stream
}</code></pre>
<h3>Pattern 6: External Input Source (Probing)</h3>
<pre><code class="lang-rust">use timely::dataflow::operators::{Inspect, Probe};
use timely::dataflow::operators::input::Handle;
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = Handle::new();
let probe = worker.dataflow(|scope| {
input.to_stream(scope)
.inspect(|x| println!(&quot;Saw: {}&quot;, x))
.probe()
});
// External loop feeding data
for round in 0..10 {
input.send(round);
input.advance_to(round + 1);
// Wait for computation to complete
while probe.less_than(input.time()) {
worker.step();
}
}
}).expect(&quot;Execution failed&quot;);
}</code></pre>
<h2>Architecture Patterns</h2>
<h3>Timestamp Structure</h3>
<pre><code>Single Dimension (u64)
→ Linear time progression
→ Use for: Simple pipelines, no iteration
→ Example: Event time in milliseconds
Product (Outer, Inner)
→ Product&lt;u64, u32&gt; for iterative computation
→ Outer: Input time, Inner: Iteration number
→ Allows feedback loops with progress guarantees
Custom Lattice
→ Implement PartialOrder trait
→ Use for: Complex coordination patterns
→ Example: Vector clocks, causal timestamps</code></pre>
<h3>Operator Exchange Patterns</h3>
<pre><code class="lang-rust">use timely::dataflow::channels::pact::{Pipeline, Exchange};
// Pipeline: No data movement (data stays on same worker)
stream.unary(Pipeline, &quot;Op&quot;, |cap, info| { ... })
// Exchange: Partition data by hash of key
stream.unary(Exchange::new(|x: &amp;T| hash(x)), &quot;Op&quot;, |cap, info| { ... })
// Broadcast: Send all data to all workers (expensive)
// Custom Pact: Implement ParallelizationContract trait</code></pre>
<h2>Performance Optimization</h2>
<h3>Worker Configuration</h3>
<pre><code class="lang-bash"># Single machine, multiple threads
cargo run -- -w 4
# Multiple processes (distributed)
cargo run -- -w 8 -p 2 # 8 workers, 2 processes (4 workers each)
cargo run -- -w 8 -p 4 # 8 workers, 4 processes (2 workers each)
# With specific hosts
cargo run -- -w 4 -p 2 -h worker1,worker2</code></pre>
<h3>Batch Size Tuning</h3>
<pre><code class="lang-rust">// Configure batch sizes for better throughput
use timely::dataflow::operators::Operator;
stream.unary_frontier(
Pipeline,
&quot;BatchedOp&quot;,
|_capability, _info| {
let mut buffer = Vec::new();
const BATCH_SIZE: usize = 1000;
move |input, output| {
input.for_each(|time, data| {
buffer.extend(data.iter().cloned());
if buffer.len() &gt;= BATCH_SIZE {
let mut session = output.session(&amp;time);
for item in buffer.drain(..) {
session.give(item);
}
}
});
}
},
)</code></pre>
<h3>Memory Management</h3>
<pre><code class="lang-rust">// Use capabilities efficiently
use timely::progress::Timestamp;
// Downgrade capabilities when possible
capability.downgrade(&amp;new_time);
// Drop capabilities when done
drop(capability);
// Clean up old state based on frontier
let frontier = &amp;frontiers[0];
state.retain(|time, _| !frontier.less_equal(time));</code></pre>
<h2>Quick Reference</h2>
<h3>Common Operators</h3>
<pre><code class="lang-rust">use timely::dataflow::operators::*;
// Transform
stream.map(|x| x * 2)
stream.flat_map(|x| 0..x)
stream.filter(|x| *x &gt; 5)
// Inspect (side effects)
stream.inspect(|x| println!(&quot;{}&quot;, x))
stream.inspect_time(|t, x| println!(&quot;{:?}: {}&quot;, t, x))
// Combine
stream1.concat(&amp;stream2)
stream1.binary(&amp;stream2, ...) // Custom binary operator
// Feedback
let (handle, cycle) = scope.feedback(summary);
stream.connect_loop(handle)
// Control
stream.probe() // Track progress
input.advance_to(time) // Advance input frontier</code></pre>
<h3>Debugging</h3>
<pre><code class="lang-rust">// Log operator names and structure
std::env::set_var(&quot;TIMELY_WORKER_LOG_ADDR&quot;, &quot;localhost:51317&quot;);
// Progress tracking
stream.inspect_time(|time, data| {
eprintln!(&quot;Time {:?}: {:?}&quot;, time, data);
})
// Frontier inspection
use timely::dataflow::operators::probe::Handle as ProbeHandle;
let probe = stream.probe();
println!(&quot;Frontier: {:?}&quot;, probe.with_frontier(|f| f.to_vec()));</code></pre>
<h2>Anti-Patterns</h2>
<pre><code>❌ NEVER: Hold capabilities indefinitely
→ Prevents frontier advancement, blocks progress
❌ NEVER: Ignore timestamp management in feedback loops
→ Can create non-monotonic timestamps, breaks progress tracking
❌ NEVER: Perform blocking I/O in operators
→ Single-threaded workers, blocks entire worker
❌ NEVER: Accumulate unbounded state without cleanup
→ Use frontier to determine safe cleanup points
❌ NEVER: Use exchange for small data volumes
→ Overhead of shuffling &gt; benefit, use Pipeline
❌ NEVER: Create too many operators
→ Overhead in progress tracking, combine logic when possible
❌ NEVER: Forget to advance input frontiers
→ Computation stalls waiting for future data
❌ NEVER: Use global state across workers
→ Shared-nothing architecture, use message passing
❌ NEVER: Assume data ordering beyond timestamp
→ Only timestamp order guaranteed, not arrival order
❌ NEVER: Mix wall-clock time with logical timestamps
→ Logical timestamps for correctness, wall-clock for monitoring only</code></pre>
<h2>Related Skills</h2>
<ul><li><code>differential-dataflow.md</code> - Incremental computation on top of timely dataflow</li><li><code>dataflow-coordination.md</code> - Advanced coordination patterns and barriers</li><li><code>streaming-aggregations.md</code> - Window-based aggregations using timely semantics</li><li><code>stream-processing.md</code> - Higher-level stream processing frameworks</li></ul>
<p><strong>Last Updated</strong>: 2025-10-27 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/data-timely-dataflow.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>