Files
skills/public/data-data-validation.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

779 lines
33 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>data-data-validation</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/data-data-validation.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/data-data-validation">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/data-data-validation.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>data-data-validation</h1><p class="lede">Validating data schema before processing</p><h1>Data Validation</h1>
<p><strong>Scope</strong>: Schema validation, data quality checks, anomaly detection <strong>Lines</strong>: 357 <strong>Last Updated</strong>: 2025-10-18 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<h2>When to Use This Skill</h2>
<p>Use this skill when:</p>
<ul><li>Validating data schema before processing</li><li>Implementing data quality gates in pipelines</li><li>Detecting anomalies and outliers in datasets</li><li>Ensuring referential integrity across tables</li><li>Monitoring data freshness and completeness</li><li>Building data quality dashboards</li><li>Working with Great Expectations, Pandera, or custom validators</li></ul>
<h2>Core Concepts</h2>
<h3>Validation Types</h3>
<pre><code>Schema Validation
→ Data types, column presence, nullable constraints
→ Use when: Incoming data format can vary
→ Tools: Pandera, Great Expectations, JSON Schema
Business Rules Validation
→ Domain-specific constraints (age &gt; 0, email format)
→ Use when: Enforcing business logic
→ Tools: Custom validators, dbt tests
Statistical Validation
→ Distribution checks, outliers, anomalies
→ Use when: Data quality monitoring
→ Tools: Great Expectations, custom statistics
Referential Integrity
→ Foreign key constraints, cross-table checks
→ Use when: Multi-table relationships
→ Tools: SQL constraints, dbt tests</code></pre>
<h3>Validation Strategies</h3>
<pre><code>Fail Fast
→ Stop pipeline on first error
→ Use when: Downstream depends on quality
Collect All Errors
→ Continue validation, report all issues
→ Use when: Data profiling, quality reporting
Quarantine
→ Separate valid/invalid records
→ Use when: Partial data processing acceptable
Auto-Repair
→ Fix known issues automatically
→ Use when: Predictable, fixable problems</code></pre>
<h3>Data Quality Dimensions</h3>
<pre><code>Completeness: No missing required fields
Accuracy: Values match real-world entities
Consistency: Same value across systems
Timeliness: Data is fresh/current
Uniqueness: No duplicates where not expected
Validity: Values in expected format/range</code></pre>
<h2>Patterns</h2>
<h3>Pattern 1: Pandera Schema Validation (Python)</h3>
<pre><code class="lang-python">import pandas as pd
import pandera as pa
from pandera import Column, Check, DataFrameSchema
from datetime import datetime
# Define schema with constraints
user_schema = DataFrameSchema(
columns={
&quot;user_id&quot;: Column(
int,
checks=[
Check.greater_than(0),
Check(lambda s: s.is_unique, error=&quot;user_id must be unique&quot;)
],
nullable=False
),
&quot;email&quot;: Column(
str,
checks=Check.str_matches(r&#x27;^[\w\.-]+@[\w\.-]+\.\w+$&#x27;),
nullable=False
),
&quot;age&quot;: Column(
int,
checks=[
Check.in_range(0, 120),
Check(lambda s: s &gt;= 18, error=&quot;Users must be 18+&quot;)
],
nullable=True
),
&quot;created_at&quot;: Column(
pa.DateTime,
checks=Check.less_than_or_equal_to(datetime.now()),
nullable=False
),
&quot;status&quot;: Column(
str,
checks=Check.isin([&#x27;active&#x27;, &#x27;inactive&#x27;, &#x27;suspended&#x27;]),
nullable=False
)
},
strict=True, # No extra columns allowed
coerce=True # Attempt type coercion
)
# Validate DataFrame
def validate_users(df: pd.DataFrame) -&gt; tuple[pd.DataFrame, list]:
&quot;&quot;&quot;Validate user data, return valid rows and errors&quot;&quot;&quot;
errors = []
try:
# Validate entire DataFrame
validated_df = user_schema.validate(df, lazy=True)
return validated_df, errors
except pa.errors.SchemaErrors as e:
# Collect all validation errors
for error in e.failure_cases.itertuples():
errors.append({
&#x27;column&#x27;: error.column,
&#x27;check&#x27;: error.check,
&#x27;index&#x27;: error.index,
&#x27;value&#x27;: error.failure_case
})
# Return rows that passed validation
valid_indices = set(df.index) - set(e.failure_cases[&#x27;index&#x27;])
valid_df = df.loc[list(valid_indices)]
return valid_df, errors
# Usage
df = pd.DataFrame({
&#x27;user_id&#x27;: [1, 2, 3, 2], # Duplicate
&#x27;email&#x27;: [&#x27;[email protected]&#x27;, &#x27;invalid-email&#x27;, &#x27;[email protected]&#x27;, &#x27;[email protected]&#x27;],
&#x27;age&#x27;: [25, 15, 30, 200], # 15 too young, 200 out of range
&#x27;created_at&#x27;: [datetime(2025, 1, 1)] * 4,
&#x27;status&#x27;: [&#x27;active&#x27;, &#x27;pending&#x27;, &#x27;inactive&#x27;, &#x27;active&#x27;] # &#x27;pending&#x27; not in allowed
})
valid_df, errors = validate_users(df)
print(f&quot;Valid rows: {len(valid_df)}&quot;)
print(f&quot;Errors: {errors}&quot;)</code></pre>
<h3>Pattern 2: Great Expectations Integration</h3>
<pre><code class="lang-python">import great_expectations as gx
from great_expectations.data_context import DataContext
from great_expectations.checkpoint import SimpleCheckpoint
class GXValidator:
def __init__(self, context_root_dir: str = None):
&quot;&quot;&quot;Initialize Great Expectations context&quot;&quot;&quot;
if context_root_dir:
self.context = DataContext(context_root_dir)
else:
self.context = gx.get_context()
def create_expectation_suite(self, suite_name: str) -&gt; None:
&quot;&quot;&quot;Create expectation suite for a dataset&quot;&quot;&quot;
suite = self.context.add_expectation_suite(suite_name)
# Define expectations
suite.add_expectation(
gx.expectations.ExpectColumnValuesToNotBeNull(column=&quot;user_id&quot;)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeUnique(column=&quot;user_id&quot;)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeInSet(
column=&quot;status&quot;,
value_set=[&#x27;active&#x27;, &#x27;inactive&#x27;, &#x27;suspended&#x27;]
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToBeBetween(
column=&quot;age&quot;,
min_value=18,
max_value=120
)
)
suite.add_expectation(
gx.expectations.ExpectColumnValuesToMatchRegex(
column=&quot;email&quot;,
regex=r&#x27;^[\w\.-]+@[\w\.-]+\.\w+$&#x27;
)
)
self.context.save_expectation_suite(suite)
def validate_dataframe(
self,
df: pd.DataFrame,
suite_name: str,
datasource_name: str = &quot;pandas_datasource&quot;
) -&gt; dict:
&quot;&quot;&quot;Validate DataFrame against expectation suite&quot;&quot;&quot;
# Add datasource
datasource = self.context.sources.add_pandas(datasource_name)
data_asset = datasource.add_dataframe_asset(name=&quot;df_asset&quot;)
# Create batch request
batch_request = data_asset.build_batch_request(dataframe=df)
# Create and run checkpoint
checkpoint = SimpleCheckpoint(
name=&quot;validation_checkpoint&quot;,
data_context=self.context,
validations=[
{
&quot;batch_request&quot;: batch_request,
&quot;expectation_suite_name&quot;: suite_name
}
]
)
result = checkpoint.run()
return {
&#x27;success&#x27;: result.success,
&#x27;statistics&#x27;: result.statistics,
&#x27;results&#x27;: result.list_validation_results()
}
# Usage
validator = GXValidator()
validator.create_expectation_suite(&quot;user_suite&quot;)
result = validator.validate_dataframe(df, &quot;user_suite&quot;)
print(f&quot;Validation success: {result[&#x27;success&#x27;]}&quot;)
print(f&quot;Statistics: {result[&#x27;statistics&#x27;]}&quot;)</code></pre>
<h3>Pattern 3: Custom Statistical Validation</h3>
<pre><code class="lang-python">import numpy as np
from scipy import stats
from typing import Dict, Any, List
import pandas as pd
class StatisticalValidator:
def __init__(self, baseline_stats: Dict[str, Any] = None):
&quot;&quot;&quot;Initialize with baseline statistics&quot;&quot;&quot;
self.baseline_stats = baseline_stats or {}
def calculate_baseline(self, df: pd.DataFrame) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Calculate baseline statistics for numeric columns&quot;&quot;&quot;
baseline = {}
for col in df.select_dtypes(include=[np.number]).columns:
baseline[col] = {
&#x27;mean&#x27;: df[col].mean(),
&#x27;std&#x27;: df[col].std(),
&#x27;median&#x27;: df[col].median(),
&#x27;q1&#x27;: df[col].quantile(0.25),
&#x27;q3&#x27;: df[col].quantile(0.75),
&#x27;min&#x27;: df[col].min(),
&#x27;max&#x27;: df[col].max(),
&#x27;null_rate&#x27;: df[col].isnull().mean()
}
self.baseline_stats = baseline
return baseline
def detect_outliers_iqr(
self,
df: pd.DataFrame,
column: str,
multiplier: float = 1.5
) -&gt; pd.Series:
&quot;&quot;&quot;Detect outliers using IQR method&quot;&quot;&quot;
Q1 = df[column].quantile(0.25)
Q3 = df[column].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - multiplier * IQR
upper_bound = Q3 + multiplier * IQR
is_outlier = (df[column] &lt; lower_bound) | (df[column] &gt; upper_bound)
return is_outlier
def detect_outliers_zscore(
self,
df: pd.DataFrame,
column: str,
threshold: float = 3.0
) -&gt; pd.Series:
&quot;&quot;&quot;Detect outliers using Z-score method&quot;&quot;&quot;
z_scores = np.abs(stats.zscore(df[column].dropna()))
is_outlier = pd.Series([False] * len(df), index=df.index)
is_outlier[df[column].notna()] = z_scores &gt; threshold
return is_outlier
def validate_distribution(
self,
df: pd.DataFrame,
column: str,
drift_threshold: float = 0.1
) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Check if distribution has drifted from baseline&quot;&quot;&quot;
if column not in self.baseline_stats:
return {&#x27;error&#x27;: &#x27;No baseline for column&#x27;}
baseline = self.baseline_stats[column]
current_mean = df[column].mean()
current_std = df[column].std()
# Calculate drift
mean_drift = abs(current_mean - baseline[&#x27;mean&#x27;]) / baseline[&#x27;mean&#x27;]
std_drift = abs(current_std - baseline[&#x27;std&#x27;]) / baseline[&#x27;std&#x27;]
return {
&#x27;column&#x27;: column,
&#x27;mean_drift&#x27;: mean_drift,
&#x27;std_drift&#x27;: std_drift,
&#x27;drift_detected&#x27;: mean_drift &gt; drift_threshold or std_drift &gt; drift_threshold,
&#x27;baseline_mean&#x27;: baseline[&#x27;mean&#x27;],
&#x27;current_mean&#x27;: current_mean,
&#x27;baseline_std&#x27;: baseline[&#x27;std&#x27;],
&#x27;current_std&#x27;: current_std
}
def validate_completeness(
self,
df: pd.DataFrame,
required_columns: List[str],
max_null_rate: float = 0.05
) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Check data completeness&quot;&quot;&quot;
results = {}
for col in required_columns:
if col not in df.columns:
results[col] = {&#x27;error&#x27;: &#x27;Column missing&#x27;}
continue
null_rate = df[col].isnull().mean()
results[col] = {
&#x27;null_rate&#x27;: null_rate,
&#x27;passes&#x27;: null_rate &lt;= max_null_rate,
&#x27;null_count&#x27;: df[col].isnull().sum(),
&#x27;total_count&#x27;: len(df)
}
return results
# Usage
validator = StatisticalValidator()
# Calculate baseline from historical data
historical_df = pd.DataFrame({
&#x27;revenue&#x27;: np.random.normal(1000, 200, 10000),
&#x27;quantity&#x27;: np.random.poisson(50, 10000)
})
validator.calculate_baseline(historical_df)
# Validate new data
new_df = pd.DataFrame({
&#x27;revenue&#x27;: np.random.normal(1200, 250, 1000), # Drifted
&#x27;quantity&#x27;: np.random.poisson(51, 1000)
})
# Check distribution drift
drift = validator.validate_distribution(new_df, &#x27;revenue&#x27;)
print(f&quot;Distribution drift: {drift}&quot;)
# Detect outliers
outliers = validator.detect_outliers_iqr(new_df, &#x27;revenue&#x27;)
print(f&quot;Outliers detected: {outliers.sum()}&quot;)</code></pre>
<h3>Pattern 4: Cross-Table Validation</h3>
<pre><code class="lang-python">from sqlalchemy import create_engine, text
from typing import List, Dict
class CrossTableValidator:
def __init__(self, connection_string: str):
self.engine = create_engine(connection_string)
def validate_referential_integrity(
self,
child_table: str,
child_column: str,
parent_table: str,
parent_column: str
) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Check foreign key integrity&quot;&quot;&quot;
query = text(f&quot;&quot;&quot;
SELECT
COUNT(*) as orphan_count,
ARRAY_AGG(DISTINCT c.{child_column}) as orphan_values
FROM {child_table} c
LEFT JOIN {parent_table} p ON c.{child_column} = p.{parent_column}
WHERE p.{parent_column} IS NULL
AND c.{child_column} IS NOT NULL
&quot;&quot;&quot;)
with self.engine.connect() as conn:
result = conn.execute(query).fetchone()
return {
&#x27;valid&#x27;: result.orphan_count == 0,
&#x27;orphan_count&#x27;: result.orphan_count,
&#x27;orphan_values&#x27;: result.orphan_values[:10] # Sample
}
def validate_aggregates(
self,
detail_table: str,
summary_table: str,
group_column: str,
agg_column: str,
tolerance: float = 0.01
) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Validate summary table against detail table&quot;&quot;&quot;
query = text(f&quot;&quot;&quot;
WITH detail_agg AS (
SELECT
{group_column},
SUM({agg_column}) as detail_sum
FROM {detail_table}
GROUP BY {group_column}
),
comparison AS (
SELECT
d.{group_column},
d.detail_sum,
s.{agg_column} as summary_sum,
ABS(d.detail_sum - s.{agg_column}) / NULLIF(d.detail_sum, 0) as diff_pct
FROM detail_agg d
FULL OUTER JOIN {summary_table} s
ON d.{group_column} = s.{group_column}
)
SELECT
COUNT(*) as total_groups,
SUM(CASE WHEN diff_pct &gt; :tolerance THEN 1 ELSE 0 END) as mismatches,
ARRAY_AGG(
{group_column} ORDER BY diff_pct DESC
) FILTER (WHERE diff_pct &gt; :tolerance) as mismatch_groups
FROM comparison
&quot;&quot;&quot;)
with self.engine.connect() as conn:
result = conn.execute(query, {&#x27;tolerance&#x27;: tolerance}).fetchone()
return {
&#x27;valid&#x27;: result.mismatches == 0,
&#x27;total_groups&#x27;: result.total_groups,
&#x27;mismatch_count&#x27;: result.mismatches,
&#x27;mismatch_groups&#x27;: result.mismatch_groups[:10]
}
def validate_freshness(
self,
table: str,
timestamp_column: str,
max_age_hours: int = 24
) -&gt; Dict[str, Any]:
&quot;&quot;&quot;Check data freshness&quot;&quot;&quot;
query = text(f&quot;&quot;&quot;
SELECT
MAX({timestamp_column}) as latest_timestamp,
EXTRACT(EPOCH FROM (NOW() - MAX({timestamp_column}))) / 3600 as age_hours
FROM {table}
&quot;&quot;&quot;)
with self.engine.connect() as conn:
result = conn.execute(query).fetchone()
return {
&#x27;valid&#x27;: result.age_hours &lt;= max_age_hours,
&#x27;latest_timestamp&#x27;: result.latest_timestamp,
&#x27;age_hours&#x27;: result.age_hours,
&#x27;max_age_hours&#x27;: max_age_hours
}
# Usage
validator = CrossTableValidator(&quot;postgresql://user:pass@localhost/db&quot;)
# Check foreign keys
fk_result = validator.validate_referential_integrity(
&#x27;orders&#x27;, &#x27;customer_id&#x27;,
&#x27;customers&#x27;, &#x27;id&#x27;
)
print(f&quot;Referential integrity: {fk_result}&quot;)
# Validate aggregates
agg_result = validator.validate_aggregates(
&#x27;order_items&#x27;, &#x27;order_totals&#x27;,
&#x27;order_id&#x27;, &#x27;amount&#x27;
)
print(f&quot;Aggregate validation: {agg_result}&quot;)
# Check freshness
fresh_result = validator.validate_freshness(
&#x27;events&#x27;, &#x27;created_at&#x27;, max_age_hours=2
)
print(f&quot;Data freshness: {fresh_result}&quot;)</code></pre>
<h2>Quick Reference</h2>
<h3>Common Validation Checks</h3>
<pre><code class="lang-python"># Null checks
df[&#x27;column&#x27;].isnull().any()
df[&#x27;column&#x27;].notna().all()
# Uniqueness
df[&#x27;column&#x27;].is_unique
df.duplicated(subset=[&#x27;col1&#x27;, &#x27;col2&#x27;]).any()
# Range checks
df[&#x27;column&#x27;].between(min_val, max_val).all()
(df[&#x27;column&#x27;] &gt;= 0).all()
# Set membership
df[&#x27;column&#x27;].isin(allowed_values).all()
# Regex pattern
df[&#x27;column&#x27;].str.match(r&#x27;^pattern$&#x27;).all()
# Type checks
df[&#x27;column&#x27;].dtype == &#x27;int64&#x27;
pd.api.types.is_numeric_dtype(df[&#x27;column&#x27;])
# Relationships
df[&#x27;col1&#x27;].sum() == df[&#x27;col2&#x27;].sum() # Balance check</code></pre>
<h3>Pandera Quick Syntax</h3>
<pre><code class="lang-python">schema = pa.DataFrameSchema({
&quot;col&quot;: pa.Column(int, checks=pa.Check.greater_than(0)),
&quot;email&quot;: pa.Column(str, checks=pa.Check.str_matches(r&#x27;@&#x27;)),
&quot;status&quot;: pa.Column(str, checks=pa.Check.isin([&#x27;A&#x27;, &#x27;B&#x27;]))
})
validated_df = schema.validate(df)</code></pre>
<h3>Great Expectations Quick Setup</h3>
<pre><code class="lang-bash"># Initialize
great_expectations init
# Create expectation suite
great_expectations suite new
# Edit expectations
great_expectations suite edit my_suite
# Run checkpoint
great_expectations checkpoint run my_checkpoint</code></pre>
<h2>Anti-Patterns</h2>
<pre><code>❌ NEVER: Validate after transformation
→ Validate raw data first, then transform
❌ NEVER: Silently drop invalid records
→ Log, alert, or quarantine for review
❌ NEVER: Use generic error messages
→ Specify column, check, and value that failed
❌ NEVER: Validate only in production
→ Run same validations in dev/staging
❌ NEVER: Skip statistical validation
→ Check distributions, not just schema
❌ NEVER: Ignore data drift
→ Monitor and alert on statistical changes
❌ NEVER: Fail pipeline without retry logic
→ Transient issues may resolve on retry
❌ NEVER: Hardcode validation rules in code
→ Use configuration files or database
❌ NEVER: Validate samples instead of full dataset
→ Sample for profiling, validate all for quality
❌ NEVER: Skip cross-table validation
→ Verify referential integrity and aggregates</code></pre>
<h2>Related Skills</h2>
<ul><li><code>etl-patterns.md</code> - Integrating validation into ETL</li><li><code>batch-processing.md</code> - Adding validation tasks to DAGs</li><li><code>stream-processing.md</code> - Real-time validation patterns</li><li><code>pipeline-orchestration.md</code> - Orchestrating validation workflows</li></ul>
<p><strong>Last Updated</strong>: 2025-10-18 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/data-data-validation.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>