Files
skills/public/apache-iceberg.html
zeekay 747dafba10 Register WebMCP tools on every page, and name each page's twin from site.py
Every page now loads /webmcp.js, scripts/webmcp.ts with its types
stripped (npm run webmcp). Where the browser has document.modelContext
it registers two read-only tools: search_skills over the published
skills index, and get_skill, which returns a skill's markdown checked
against the index digest. The markdown alternate link moves from
build.py to site.py, so CI holds every page to it.

The API catalog gains the site itself (llms.txt and the skills index)
and hanzo.ai/agents; the live discovery check now fetches every
catalog link. tests/site.mjs drives the tools through a recording
modelContext, and checks a browser without one loads clean.
2026-10-26 20:57:05 -07:00

658 lines
34 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>apache-iceberg</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/apache-iceberg">
<link rel="alternate" type="text/markdown" href="/apache-iceberg.md">
<script type="module" src="/webmcp.js"></script>
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/apache-iceberg.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>apache-iceberg</h1><p class="lede">Building data lakes with ACID transaction support and schema evolution</p><h1>Apache Iceberg</h1>
<p><strong>Scope</strong>: Table format, schema/partition evolution, time travel, ACID transactions, catalog integration, performance optimization <strong>Lines</strong>: ~390 <strong>Last Updated</strong>: 2025-10-18</p>
<h2>When to Use This Skill</h2>
<p>Activate this skill when:</p>
<ul><li>Building data lakes with ACID transaction support and schema evolution</li><li>Implementing time travel queries and snapshot-based rollback</li><li>Migrating from Hive tables to modern table formats</li><li>Designing partition strategies that can evolve without data rewrites</li><li>Integrating with Spark, Flink, Trino, or Presto for analytics workloads</li><li>Implementing incremental reads and change data capture (CDC) patterns</li><li>Managing large-scale data with efficient snapshot and metadata management</li><li>Optimizing query performance with hidden partitioning and file layout</li></ul>
<h2>Core Concepts</h2>
<h3>Table Format Architecture</h3>
<p><strong>Three-layer metadata hierarchy</strong>:</p>
<ul><li><strong>Metadata files</strong>: JSON files tracking table schema, partitioning, snapshots</li><li><strong>Manifest lists</strong>: Lists of manifest files for each snapshot</li><li><strong>Manifest files</strong>: Lists of data files with partition information and statistics</li><li><strong>Data files</strong>: Actual data in Parquet, ORC, or Avro format</li></ul>
<p><strong>ACID guarantees</strong>:</p>
<ul><li>Atomic commits via metadata file replacement (optimistic concurrency)</li><li>Isolation through snapshot isolation (readers see consistent snapshot)</li><li>Consistency enforced by schema validation</li><li>Durability through immutable data and metadata files</li></ul>
<p><strong>Hidden partitioning</strong>:</p>
<ul><li>Partition columns don&#x27;t appear in user queries</li><li>Automatically applied based on table metadata</li><li>Evolution without breaking existing queries</li><li>Supports transforms (year, month, day, hour, bucket, truncate) &lt;!-- NOTE: truncate is Iceberg&#x27;s data transform function, not SQL TRUNCATE --&gt;</li></ul>
<h3>Catalog Integration</h3>
<p><strong>Catalog types</strong>:</p>
<ul><li><strong>Hive Metastore</strong>: Legacy, widely compatible, atomic table updates</li><li><strong>AWS Glue</strong>: Managed, serverless, integrated with AWS services</li><li><strong>Nessie</strong>: Git-like versioning, multi-table transactions, branching</li><li><strong>REST Catalog</strong>: HTTP-based, cloud-native, flexible implementation</li><li><strong>JDBC</strong>: Direct database storage for metadata</li></ul>
<p><strong>Catalog configuration</strong>:</p>
<pre><code class="lang-python"># PyIceberg with different catalogs
from pyiceberg.catalog import load_catalog
# Hive Metastore
catalog = load_catalog(&quot;hive&quot;,
uri=&quot;thrift://localhost:9083&quot;,
warehouse=&quot;s3://bucket/warehouse&quot;
)
# AWS Glue
catalog = load_catalog(&quot;glue&quot;,
warehouse=&quot;s3://bucket/warehouse&quot;
)
# REST Catalog
catalog = load_catalog(&quot;rest&quot;,
uri=&quot;https://catalog-api.example.com&quot;,
credential=&quot;token:abc123&quot;
)</code></pre>
<h3>Snapshots and Time Travel</h3>
<p><strong>Snapshot lifecycle</strong>:</p>
<ul><li>Every write operation creates new snapshot</li><li>Snapshots are immutable and point-in-time consistent</li><li>Metadata tracks parent-child snapshot relationships</li><li>Snapshots can be expired to reclaim storage</li></ul>
<p><strong>Time travel queries</strong>:</p>
<ul><li>Query historical data by snapshot ID or timestamp</li><li>Compare data between snapshots for audit/debugging</li><li>Rollback to previous snapshot for error recovery</li><li>Incremental reads between snapshots for CDC</li></ul>
<hr>
<h2>Patterns</h2>
<h3>Table Creation with Schema and Partitioning</h3>
<pre><code class="lang-sql">-- Spark SQL: Create Iceberg table with partitioning
CREATE TABLE catalog.db.events (
event_id BIGINT,
user_id STRING,
event_type STRING,
event_time TIMESTAMP,
properties MAP&lt;STRING, STRING&gt;
)
USING iceberg
PARTITIONED BY (days(event_time), event_type)
TBLPROPERTIES (
&#x27;write.format.default&#x27; = &#x27;parquet&#x27;,
&#x27;write.parquet.compression-codec&#x27; = &#x27;zstd&#x27;,
&#x27;write.metadata.compression-codec&#x27; = &#x27;gzip&#x27;
);</code></pre>
<p><strong>Python PyIceberg</strong>:</p>
<pre><code class="lang-python">from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
NestedField, LongType, StringType, TimestampType, MapType
)
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform, IdentityTransform
catalog = load_catalog(&quot;default&quot;)
schema = Schema(
NestedField(1, &quot;event_id&quot;, LongType(), required=True),
NestedField(2, &quot;user_id&quot;, StringType(), required=True),
NestedField(3, &quot;event_type&quot;, StringType(), required=True),
NestedField(4, &quot;event_time&quot;, TimestampType(), required=True),
NestedField(5, &quot;properties&quot;, MapType(6, StringType(), 7, StringType()))
)
# Hidden partitioning: users query event_time, Iceberg handles partitioning
partition_spec = PartitionSpec(
PartitionField(source_id=4, field_id=1000, transform=DayTransform(), name=&quot;event_day&quot;),
PartitionField(source_id=3, field_id=1001, transform=IdentityTransform(), name=&quot;event_type&quot;)
)
table = catalog.create_table(
identifier=&quot;db.events&quot;,
schema=schema,
partition_spec=partition_spec,
properties={&quot;write.format.default&quot;: &quot;parquet&quot;}
)</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Starting new data lake projects</li><li>Migrating from Hive with schema evolution needs</li><li>Requiring flexible partition strategies</li></ul>
<h3>Schema Evolution</h3>
<pre><code class="lang-sql">-- Add column (backward compatible)
ALTER TABLE catalog.db.events
ADD COLUMN session_id STRING;
-- Drop column (forward compatible)
ALTER TABLE catalog.db.events
DROP COLUMN properties;
-- Rename column (maintains column IDs)
ALTER TABLE catalog.db.events
RENAME COLUMN user_id TO customer_id;
-- Type promotion (safe widening)
ALTER TABLE catalog.db.events
ALTER COLUMN event_id TYPE BIGINT; -- int -&gt; bigint allowed</code></pre>
<p><strong>Python schema updates</strong>:</p>
<pre><code class="lang-python">from pyiceberg.catalog import load_catalog
from pyiceberg.types import StringType, LongType
catalog = load_catalog(&quot;default&quot;)
table = catalog.load_table(&quot;db.events&quot;)
# Add column
with table.update_schema() as update:
update.add_column(&quot;session_id&quot;, StringType())
# Rename column
with table.update_schema() as update:
update.rename_column(&quot;user_id&quot;, &quot;customer_id&quot;)
# Type promotion
with table.update_schema() as update:
update.update_column(&quot;event_id&quot;, field_type=LongType())</code></pre>
<p><strong>Benefits</strong>:</p>
<ul><li>No downtime or data rewrites for schema changes</li><li>Column IDs preserve compatibility across renames</li><li>Safe type promotions (int→long, float→double, decimal scale up)</li></ul>
<h3>Partition Evolution</h3>
<pre><code class="lang-sql">-- Initial partitioning by day
CREATE TABLE catalog.db.logs (
log_time TIMESTAMP,
message STRING
)
USING iceberg
PARTITIONED BY (days(log_time));
-- Later: Add hourly partitioning for recent data
ALTER TABLE catalog.db.logs
REPLACE PARTITION FIELD days(log_time)
WITH hours(log_time);
-- Old data stays partitioned by day, new data by hour
-- Queries work seamlessly across both partition layouts</code></pre>
<p><strong>Partition transform types</strong>:</p>
<pre><code class="lang-python">from pyiceberg.transforms import (
YearTransform, MonthTransform, DayTransform, HourTransform,
BucketTransform, TruncateTransform
)
# Date/time transforms
YearTransform() # Extract year
MonthTransform() # Extract month
DayTransform() # Extract day
HourTransform() # Extract hour
# Distribution transforms
BucketTransform(num_buckets=16) # Hash bucket for uniform distribution
TruncateTransform(width=10) # NOTE: Iceberg transform for trimming strings/numbers to width, not SQL TRUNCATE</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Workload patterns change over time (daily → hourly)</li><li>Need to rebalance data distribution</li><li>Optimizing query performance without data migration</li></ul>
<h3>Time Travel and Snapshot Queries</h3>
<pre><code class="lang-sql">-- Query as of specific timestamp
SELECT * FROM catalog.db.events
FOR SYSTEM_TIME AS OF TIMESTAMP &#x27;2025-10-01 00:00:00&#x27;;
-- Query specific snapshot by ID
SELECT * FROM catalog.db.events
FOR SYSTEM_VERSION AS OF 1234567890;
-- View all snapshots
SELECT * FROM catalog.db.events.snapshots;
-- View snapshot history
SELECT
snapshot_id,
committed_at,
operation,
summary
FROM catalog.db.events.history
ORDER BY committed_at DESC;
-- Rollback to previous snapshot
CALL catalog.system.rollback_to_snapshot(&#x27;db.events&#x27;, 1234567890);
-- Cherry-pick specific snapshot
CALL catalog.system.cherrypick_snapshot(&#x27;db.events&#x27;, 1234567890);</code></pre>
<p><strong>Python time travel</strong>:</p>
<pre><code class="lang-python">from datetime import datetime
from pyiceberg.catalog import load_catalog
catalog = load_catalog(&quot;default&quot;)
table = catalog.load_table(&quot;db.events&quot;)
# Read current snapshot
df = table.scan().to_arrow()
# Read specific snapshot
snapshot = table.snapshot_by_id(1234567890)
df_historical = table.scan(snapshot_id=snapshot.snapshot_id).to_arrow()
# Read as of timestamp
timestamp = datetime(2025, 10, 1, 0, 0, 0)
df_time_travel = table.scan(as_of_timestamp=timestamp).to_arrow()
# List all snapshots
for snapshot in table.snapshots():
print(f&quot;Snapshot {snapshot.snapshot_id} at {snapshot.timestamp_ms}&quot;)</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Auditing data changes over time</li><li>Debugging production issues with historical data</li><li>Implementing CDC by comparing snapshots</li><li>Rolling back accidental data corruption</li></ul>
<h3>Incremental Reads and CDC</h3>
<pre><code class="lang-python">from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import And, EqualTo, GreaterThanOrEqual
catalog = load_catalog(&quot;default&quot;)
table = catalog.load_table(&quot;db.events&quot;)
# Get current and previous snapshots
snapshots = list(table.snapshots())
current_snapshot = snapshots[-1]
previous_snapshot = snapshots[-2]
# Incremental read: only files added between snapshots
scan = table.scan(
snapshot_id=current_snapshot.snapshot_id,
from_snapshot_id=previous_snapshot.snapshot_id
)
# Filter incremental data
incremental_df = scan.filter(
GreaterThanOrEqual(&quot;event_time&quot;, &quot;2025-10-18 00:00:00&quot;)
).to_arrow()
print(f&quot;Incremental records: {len(incremental_df)}&quot;)</code></pre>
<p><strong>Spark incremental reads</strong>:</p>
<pre><code class="lang-scala">// Read changes between two snapshots
val incrementalDF = spark.read
.format(&quot;iceberg&quot;)
.option(&quot;start-snapshot-id&quot;, &quot;1234567890&quot;)
.option(&quot;end-snapshot-id&quot;, &quot;1234567891&quot;)
.load(&quot;catalog.db.events&quot;)
// Read all changes since timestamp
val cdcDF = spark.read
.format(&quot;iceberg&quot;)
.option(&quot;start-timestamp&quot;, &quot;2025-10-18 00:00:00&quot;)
.load(&quot;catalog.db.events&quot;)</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Building incremental ETL pipelines</li><li>Streaming data processing with batch catchup</li><li>Change data capture for downstream systems</li><li>Reducing processing by reading only new data</li></ul>
<h3>ACID Transactions with Spark</h3>
<pre><code class="lang-scala">import org.apache.spark.sql.SparkSession
import org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
val spark = SparkSession.builder()
.appName(&quot;Iceberg ACID&quot;)
.config(&quot;spark.sql.extensions&quot;,
&quot;org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions&quot;)
.config(&quot;spark.sql.catalog.catalog&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;)
.config(&quot;spark.sql.catalog.catalog.type&quot;, &quot;hive&quot;)
.getOrCreate()
// Atomic write: all or nothing
val newEvents = spark.read.json(&quot;s3://bucket/raw/events/&quot;)
newEvents.writeTo(&quot;catalog.db.events&quot;)
.append() // Atomic commit
// Upsert with MERGE INTO (atomic)
spark.sql(&quot;&quot;&quot;
MERGE INTO catalog.db.events t
USING updates s
ON t.event_id = s.event_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
&quot;&quot;&quot;)
// Delete with predicate (atomic)
spark.sql(&quot;&quot;&quot;
DELETE FROM catalog.db.events
WHERE event_time &lt; TIMESTAMP &#x27;2024-01-01&#x27;
&quot;&quot;&quot;)
// Multi-table transaction (via WAP - Write-Audit-Publish)
spark.conf.set(&quot;spark.wap.id&quot;, &quot;staging-branch&quot;)
// Write to staging branch
df.writeTo(&quot;catalog.db.events&quot;).append()
// Validate data
val count = spark.sql(&quot;SELECT COUNT(*) FROM catalog.db.events&quot;).collect()
// Publish if valid (atomic promotion)
if (count.head.getLong(0) &gt; 0) {
spark.sql(&quot;CALL catalog.system.cherrypick_snapshot(&#x27;db.events&#x27;, &#x27;staging-branch&#x27;)&quot;)
}</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Ensuring data consistency across failures</li><li>Implementing upsert/delete operations</li><li>Multi-stage ETL with validation gates</li></ul>
<h3>Table Maintenance and Optimization</h3>
<pre><code class="lang-sql">-- Expire old snapshots (free storage)
CALL catalog.system.expire_snapshots(
table =&gt; &#x27;db.events&#x27;,
older_than =&gt; TIMESTAMP &#x27;2025-09-01 00:00:00&#x27;,
retain_last =&gt; 10
);
-- Remove orphan files (unreferenced data files)
CALL catalog.system.remove_orphan_files(
table =&gt; &#x27;db.events&#x27;,
older_than =&gt; TIMESTAMP &#x27;2025-10-01 00:00:00&#x27;
);
-- Rewrite small files into larger files
CALL catalog.system.rewrite_data_files(
table =&gt; &#x27;db.events&#x27;,
options =&gt; map(&#x27;target-file-size-bytes&#x27;, &#x27;536870912&#x27;) -- 512 MB
);
-- Rewrite manifest files for faster planning
CALL catalog.system.rewrite_manifests(&#x27;db.events&#x27;);
-- Compact data files in partition
CALL catalog.system.rewrite_data_files(
table =&gt; &#x27;db.events&#x27;,
where =&gt; &#x27;event_date = DATE &quot;2025-10-18&quot;&#x27;
);</code></pre>
<p><strong>Python maintenance</strong>:</p>
<pre><code class="lang-python">from pyiceberg.catalog import load_catalog
from datetime import datetime, timedelta
catalog = load_catalog(&quot;default&quot;)
table = catalog.load_table(&quot;db.events&quot;)
# Expire snapshots older than 30 days
expire_before = datetime.now() - timedelta(days=30)
table.expire_snapshots(
older_than=expire_before,
retain_last=10 # Keep at least 10 snapshots
)
# Get table statistics
metadata = table.metadata
print(f&quot;Snapshots: {len(metadata.snapshots)}&quot;)
print(f&quot;Schema version: {metadata.current_schema_id}&quot;)
print(f&quot;Partition spec: {metadata.default_spec_id}&quot;)</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Regular maintenance (weekly/monthly)</li><li>After bulk deletes or updates</li><li>When query planning is slow (too many small files)</li><li>Storage cost optimization</li></ul>
<h3>Integration with Flink for Streaming</h3>
<pre><code class="lang-java">import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
TableEnvironment tEnv = TableEnvironment.create(env);
// Configure Iceberg catalog
tEnv.executeSql(
&quot;CREATE CATALOG iceberg_catalog WITH (&quot; +
&quot; &#x27;type&#x27;=&#x27;iceberg&#x27;,&quot; +
&quot; &#x27;catalog-type&#x27;=&#x27;hive&#x27;,&quot; +
&quot; &#x27;uri&#x27;=&#x27;thrift://localhost:9083&#x27;,&quot; +
&quot; &#x27;warehouse&#x27;=&#x27;s3://bucket/warehouse&#x27;&quot; +
&quot;)&quot;
);
tEnv.useCatalog(&quot;iceberg_catalog&quot;);
// Create Iceberg table from Flink
tEnv.executeSql(
&quot;CREATE TABLE events (&quot; +
&quot; event_id BIGINT,&quot; +
&quot; user_id STRING,&quot; +
&quot; event_time TIMESTAMP(6),&quot; +
&quot; PRIMARY KEY (event_id) NOT ENFORCED&quot; +
&quot;) PARTITIONED BY (event_type)&quot; +
&quot; WITH (&#x27;write.format.default&#x27;=&#x27;parquet&#x27;)&quot;
);
// Stream write to Iceberg (micro-batch commits)
tEnv.executeSql(
&quot;INSERT INTO events &quot; +
&quot;SELECT event_id, user_id, event_time FROM kafka_source&quot;
);
// Stream read from Iceberg
tEnv.executeSql(
&quot;SELECT * FROM events &quot; +
&quot;/*+ OPTIONS(&#x27;streaming&#x27;=&#x27;true&#x27;, &#x27;monitor-interval&#x27;=&#x27;10s&#x27;) */&quot;
);</code></pre>
<p><strong>When to use</strong>:</p>
<ul><li>Real-time streaming ingestion to data lake</li><li>Streaming analytics on Iceberg tables</li><li>Combining batch and streaming workloads</li></ul>
<hr>
<h2>Quick Reference</h2>
<h3>Core Commands</h3>
<p>| Operation | Spark SQL | Python PyIceberg | |-----------|-----------|------------------| | Create table | <code>CREATE TABLE ... USING iceberg</code> | <code>catalog.create_table()</code> | | Add column | <code>ALTER TABLE ... ADD COLUMN</code> | <code>table.update_schema().add_column()</code> | | Rename column | <code>ALTER TABLE ... RENAME COLUMN</code> | <code>table.update_schema().rename_column()</code> | | Time travel | <code>FOR SYSTEM_TIME AS OF</code> | <code>table.scan(as_of_timestamp=...)</code> | | Expire snapshots | <code>CALL system.expire_snapshots()</code> | <code>table.expire_snapshots()</code> | | Rewrite files | <code>CALL system.rewrite_data_files()</code> | N/A (use Spark) |</p>
<h3>Partition Transforms</h3>
<pre><code>Transform | Example | Use Case
------------------|----------------------------|-----------------------
years(ts) | PARTITIONED BY (years(ts)) | Long-term archives
months(ts) | PARTITIONED BY (months(ts))| Monthly aggregations
days(ts) | PARTITIONED BY (days(ts)) | Daily pipelines
hours(ts) | PARTITIONED BY (hours(ts)) | Streaming ingestion
bucket(N, id) | PARTITIONED BY (bucket(16, id)) | Uniform distribution
truncate(W, str) | PARTITIONED BY (truncate(10, str)) | String prefixes &lt;!-- NOTE: Iceberg transform function --&gt;</code></pre>
<h3>Configuration Best Practices</h3>
<pre><code class="lang-properties"># Write performance
write.format.default=parquet
write.parquet.compression-codec=zstd
write.target-file-size-bytes=536870912 # 512 MB
# Metadata optimization
write.metadata.compression-codec=gzip
write.metadata.metrics.default=truncate(16) # NOTE: Iceberg&#x27;s truncate transform for metrics, not SQL TRUNCATE
# Commit behavior
commit.retry.num-retries=4
commit.retry.min-wait-ms=100
# Snapshot expiration
history.expire.max-snapshot-age-ms=432000000 # 5 days</code></pre>
<h3>Catalog URLs</h3>
<pre><code>Catalog Type | URI Format | Example
-------------|------------------------------------------|------------------
Hive | thrift://host:port | thrift://localhost:9083
Glue | (use AWS config) | N/A
REST | https://catalog-host/path | https://api.example.com
JDBC | jdbc:postgresql://host:port/db | jdbc:postgresql://localhost:5432/iceberg
Nessie | https://nessie-host/api/v1 | https://nessie.example.com/api/v1</code></pre>
<hr>
<h2>Anti-Patterns</h2>
<p>❌ <strong>Partitioning by high-cardinality columns</strong>: Creating too many partitions (user_id, session_id) ✅ Use bucketing for high-cardinality: <code>PARTITIONED BY (bucket(256, user_id))</code></p>
<p>❌ <strong>Manually managing partition columns in queries</strong>: <code>WHERE date_partition = &#x27;2025-10-18&#x27;</code> ✅ Use hidden partitioning: <code>WHERE event_time &gt;= &#x27;2025-10-18&#x27;</code> (Iceberg handles partitions)</p>
<p>❌ <strong>Never expiring snapshots</strong>: Unbounded metadata growth, slow query planning ✅ Schedule regular snapshot expiration: <code>CALL system.expire_snapshots()</code> weekly</p>
<p>❌ <strong>Creating many small files</strong>: Thousands of tiny files per partition ✅ Rewrite to target file size (512 MB - 1 GB): <code>CALL system.rewrite_data_files()</code></p>
<p>❌ <strong>Using Hive table format for schema evolution</strong>: Requires full table rewrites ✅ Use Iceberg&#x27;s built-in schema evolution: <code>ALTER TABLE ... ADD COLUMN</code></p>
<p>❌ <strong>Querying without partition filters</strong>: Full table scans on large datasets ✅ Always filter on partition columns: <code>WHERE event_time BETWEEN ... AND ...</code></p>
<p>❌ <strong>Ignoring snapshot metadata</strong>: Missing opportunities for incremental processing ✅ Use snapshot IDs for incremental reads: <code>scan(from_snapshot_id=...)</code></p>
<p>❌ <strong>Mixing partition schemes without evolution</strong>: Manual partition column changes ✅ Use partition evolution: <code>REPLACE PARTITION FIELD ... WITH ...</code></p>
<hr>
<h2>Related Skills</h2>
<ul><li><code>spark-optimization.md</code> - Optimizing Spark jobs for Iceberg table reads/writes</li><li><code>delta-lake.md</code> - Alternative lakehouse format, comparison with Iceberg</li><li><code>hudi.md</code> - Another lakehouse format, streaming use cases</li><li><code>parquet-optimization.md</code> - Understanding underlying data file format</li><li><code>aws-glue-catalog.md</code> - Managing Iceberg tables with AWS Glue</li><li><code>dbt-data-modeling.md</code> - Building data models on top of Iceberg tables</li></ul>
<hr>
<p><strong>Last Updated</strong>: 2025-10-18 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/apache-iceberg.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>