mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
Every page now loads /webmcp.js, scripts/webmcp.ts with its types stripped (npm run webmcp). Where the browser has document.modelContext it registers two read-only tools: search_skills over the published skills index, and get_skill, which returns a skill's markdown checked against the index digest. The markdown alternate link moves from build.py to site.py, so CI holds every page to it. The API catalog gains the site itself (llms.txt and the skills index) and hanzo.ai/agents; the live discovery check now fetches every catalog link. tests/site.mjs drives the tools through a recording modelContext, and checks a browser without one loads clean.
949 lines
42 KiB
HTML
949 lines
42 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>research-data-analysis</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/research-data-analysis">
|
|
<link rel="alternate" type="text/markdown" href="/research-data-analysis.md">
|
|
<script type="module" src="/webmcp.js"></script>
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/research-data-analysis.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>research-data-analysis</h1><p class="lede">Master data analysis techniques including coding, statistical tests, visualization, interpretation, and reporting findings</p><h1>Data Analysis Skill</h1>
|
|
<h2>When to Use This Skill</h2>
|
|
<p>Use this skill when you need to:</p>
|
|
<ul><li>Analyze quantitative or qualitative data</li><li>Code textual data systematically</li><li>Choose appropriate statistical tests</li><li>Visualize data effectively</li><li>Interpret analysis results</li><li>Check analysis assumptions</li><li>Report findings clearly</li><li>Handle complex or messy data</li></ul>
|
|
<h2>Quantitative Data Analysis</h2>
|
|
<h3>1. Descriptive Statistics</h3>
|
|
<p><strong>Comprehensive Description Framework</strong>:</p>
|
|
<pre><code class="lang-python">import pandas as pd
|
|
import numpy as np
|
|
from scipy import stats
|
|
import matplotlib.pyplot as plt
|
|
import seaborn as sns
|
|
|
|
class DescriptiveAnalysis:
|
|
"""Comprehensive descriptive statistics"""
|
|
|
|
def __init__(self, data: pd.DataFrame):
|
|
self.data = data
|
|
|
|
def continuous_summary(self, var: str):
|
|
"""Summarize continuous variable"""
|
|
series = self.data[var].dropna()
|
|
|
|
summary = {
|
|
'n': len(series),
|
|
'missing': self.data[var].isna().sum(),
|
|
'mean': series.mean(),
|
|
'median': series.median(),
|
|
'std': series.std(),
|
|
'min': series.min(),
|
|
'max': series.max(),
|
|
'q25': series.quantile(0.25),
|
|
'q75': series.quantile(0.75),
|
|
'skewness': stats.skew(series),
|
|
'kurtosis': stats.kurtosis(series)
|
|
}
|
|
|
|
# Add confidence interval for mean
|
|
ci = stats.t.interval(0.95, len(series)-1,
|
|
loc=summary['mean'],
|
|
scale=stats.sem(series))
|
|
summary['ci_95_lower'] = ci[0]
|
|
summary['ci_95_upper'] = ci[1]
|
|
|
|
return summary
|
|
|
|
def categorical_summary(self, var: str):
|
|
"""Summarize categorical variable"""
|
|
counts = self.data[var].value_counts()
|
|
proportions = self.data[var].value_counts(normalize=True)
|
|
|
|
summary = pd.DataFrame({
|
|
'Count': counts,
|
|
'Proportion': proportions,
|
|
'Percentage': proportions * 100
|
|
})
|
|
|
|
summary['95% CI Lower'] = summary.apply(
|
|
lambda row: self._proportion_ci(row['Count'], len(self.data))[0],
|
|
axis=1
|
|
)
|
|
summary['95% CI Upper'] = summary.apply(
|
|
lambda row: self._proportion_ci(row['Count'], len(self.data))[1],
|
|
axis=1
|
|
)
|
|
|
|
return summary
|
|
|
|
@staticmethod
|
|
def _proportion_ci(count, n, confidence=0.95):
|
|
"""Wilson score confidence interval for proportion"""
|
|
from statsmodels.stats.proportion import proportion_confint
|
|
return proportion_confint(count, n, alpha=1-confidence, method='wilson')
|
|
|
|
def create_descriptive_table(self, continuous_vars: list,
|
|
categorical_vars: list):
|
|
"""Create Table 1 style descriptive statistics"""
|
|
table = []
|
|
|
|
for var in continuous_vars:
|
|
summary = self.continuous_summary(var)
|
|
table.append({
|
|
'Variable': var,
|
|
'Type': 'Continuous',
|
|
'Summary': f"{summary['mean']:.2f} ± {summary['std']:.2f}",
|
|
'Range': f"[{summary['min']:.2f}, {summary['max']:.2f}]",
|
|
'N': summary['n']
|
|
})
|
|
|
|
for var in categorical_vars:
|
|
summary = self.categorical_summary(var)
|
|
for category, row in summary.iterrows():
|
|
table.append({
|
|
'Variable': f"{var}: {category}",
|
|
'Type': 'Categorical',
|
|
'Summary': f"{row['Count']} ({row['Percentage']:.1f}%)",
|
|
'Range': f"[{row['95% CI Lower']:.3f}, "
|
|
f"{row['95% CI Upper']:.3f}]",
|
|
'N': row['Count']
|
|
})
|
|
|
|
return pd.DataFrame(table)
|
|
|
|
def visualize_distribution(self, var: str, var_type: str = 'continuous'):
|
|
"""Create distribution visualizations"""
|
|
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
|
|
|
|
if var_type == 'continuous':
|
|
# Histogram with KDE
|
|
axes[0].hist(self.data[var].dropna(), bins=30,
|
|
density=True, alpha=0.7, edgecolor='black')
|
|
self.data[var].plot.kde(ax=axes[0], color='red', linewidth=2)
|
|
axes[0].set_xlabel(var)
|
|
axes[0].set_ylabel('Density')
|
|
axes[0].set_title(f'Distribution of {var}')
|
|
|
|
# Q-Q plot for normality
|
|
stats.probplot(self.data[var].dropna(), dist="norm", plot=axes[1])
|
|
axes[1].set_title('Q-Q Plot (Normality Check)')
|
|
|
|
else: # categorical
|
|
# Bar chart
|
|
counts = self.data[var].value_counts()
|
|
axes[0].bar(range(len(counts)), counts.values)
|
|
axes[0].set_xticks(range(len(counts)))
|
|
axes[0].set_xticklabels(counts.index, rotation=45, ha='right')
|
|
axes[0].set_ylabel('Count')
|
|
axes[0].set_title(f'Frequency of {var}')
|
|
|
|
# Proportion with CI
|
|
summary = self.categorical_summary(var)
|
|
x = range(len(summary))
|
|
axes[1].bar(x, summary['Percentage'])
|
|
axes[1].errorbar(x,
|
|
summary['Percentage'],
|
|
yerr=[summary['Percentage'] - summary['95% CI Lower']*100,
|
|
summary['95% CI Upper']*100 - summary['Percentage']],
|
|
fmt='none', color='black', capsize=5)
|
|
axes[1].set_xticks(x)
|
|
axes[1].set_xticklabels(summary.index, rotation=45, ha='right')
|
|
axes[1].set_ylabel('Percentage')
|
|
axes[1].set_title(f'Proportion with 95% CI')
|
|
|
|
plt.tight_layout()
|
|
return fig
|
|
|
|
# Example usage
|
|
data = pd.DataFrame({
|
|
'age': np.random.normal(35, 10, 100),
|
|
'satisfaction': np.random.choice([1, 2, 3, 4, 5], 100),
|
|
'group': np.random.choice(['Control', 'Treatment'], 100)
|
|
})
|
|
|
|
analysis = DescriptiveAnalysis(data)
|
|
print(analysis.continuous_summary('age'))
|
|
print(analysis.categorical_summary('group'))
|
|
table = analysis.create_descriptive_table(
|
|
continuous_vars=['age'],
|
|
categorical_vars=['group', 'satisfaction']
|
|
)
|
|
print(table)</code></pre>
|
|
<h3>2. Inferential Statistics</h3>
|
|
<p><strong>Comprehensive Testing Framework</strong>:</p>
|
|
<pre><code class="lang-python">from scipy import stats
|
|
import pandas as pd
|
|
import numpy as np
|
|
from typing import Tuple, Dict
|
|
|
|
class InferentialAnalysis:
|
|
"""Statistical inference with proper reporting"""
|
|
|
|
@staticmethod
|
|
def t_test(group1, group2, paired: bool = False,
|
|
alpha: float = 0.05) -> Dict:
|
|
"""Conduct t-test with full reporting"""
|
|
g1 = np.array(group1)
|
|
g2 = np.array(group2)
|
|
|
|
# Remove NaN
|
|
if paired:
|
|
mask = ~(np.isnan(g1) | np.isnan(g2))
|
|
g1, g2 = g1[mask], g2[mask]
|
|
t_stat, p_value = stats.ttest_rel(g1, g2)
|
|
df = len(g1) - 1
|
|
# Cohen's d for paired
|
|
diff = g1 - g2
|
|
cohens_d = np.mean(diff) / np.std(diff, ddof=1)
|
|
else:
|
|
g1 = g1[~np.isnan(g1)]
|
|
g2 = g2[~np.isnan(g2)]
|
|
t_stat, p_value = stats.ttest_ind(g1, g2)
|
|
df = len(g1) + len(g2) - 2
|
|
# Cohen's d for independent
|
|
pooled_std = np.sqrt(((len(g1)-1)*np.var(g1, ddof=1) +
|
|
(len(g2)-1)*np.var(g2, ddof=1)) / df)
|
|
cohens_d = (np.mean(g1) - np.mean(g2)) / pooled_std
|
|
|
|
# Effect size interpretation
|
|
if abs(cohens_d) < 0.2:
|
|
effect_size = 'negligible'
|
|
elif abs(cohens_d) < 0.5:
|
|
effect_size = 'small'
|
|
elif abs(cohens_d) < 0.8:
|
|
effect_size = 'medium'
|
|
else:
|
|
effect_size = 'large'
|
|
|
|
return {
|
|
'test': 'paired t-test' if paired else 'independent t-test',
|
|
't_statistic': t_stat,
|
|
'df': df,
|
|
'p_value': p_value,
|
|
'significant': p_value < alpha,
|
|
'mean_1': np.mean(g1),
|
|
'mean_2': np.mean(g2),
|
|
'mean_diff': np.mean(g1) - np.mean(g2),
|
|
'cohens_d': cohens_d,
|
|
'effect_size': effect_size,
|
|
'n_1': len(g1),
|
|
'n_2': len(g2)
|
|
}
|
|
|
|
@staticmethod
|
|
def anova(groups: list, alpha: float = 0.05) -> Dict:
|
|
"""One-way ANOVA with post-hoc tests"""
|
|
# ANOVA
|
|
f_stat, p_value = stats.f_oneway(*groups)
|
|
df_between = len(groups) - 1
|
|
df_within = sum(len(g) for g in groups) - len(groups)
|
|
|
|
# Effect size (eta-squared)
|
|
grand_mean = np.mean([x for g in groups for x in g])
|
|
ss_between = sum(len(g) * (np.mean(g) - grand_mean)**2
|
|
for g in groups)
|
|
ss_total = sum((x - grand_mean)**2 for g in groups for x in g)
|
|
eta_squared = ss_between / ss_total
|
|
|
|
result = {
|
|
'test': 'One-way ANOVA',
|
|
'f_statistic': f_stat,
|
|
'df_between': df_between,
|
|
'df_within': df_within,
|
|
'p_value': p_value,
|
|
'significant': p_value < alpha,
|
|
'eta_squared': eta_squared,
|
|
'n_groups': len(groups)
|
|
}
|
|
|
|
# Post-hoc pairwise comparisons if significant
|
|
if p_value < alpha:
|
|
from itertools import combinations
|
|
pairwise = []
|
|
|
|
for i, j in combinations(range(len(groups)), 2):
|
|
t_result = InferentialAnalysis.t_test(groups[i], groups[j])
|
|
# Bonferroni correction
|
|
adjusted_alpha = alpha / (len(groups) * (len(groups) - 1) / 2)
|
|
pairwise.append({
|
|
'comparison': f'Group {i+1} vs Group {j+1}',
|
|
'p_value': t_result['p_value'],
|
|
'significant_bonferroni': t_result['p_value'] < adjusted_alpha,
|
|
'mean_diff': t_result['mean_diff']
|
|
})
|
|
|
|
result['post_hoc'] = pairwise
|
|
|
|
return result
|
|
|
|
@staticmethod
|
|
def correlation(x, y, method: str = 'pearson',
|
|
alpha: float = 0.05) -> Dict:
|
|
"""Correlation with confidence interval"""
|
|
# Remove NaN
|
|
mask = ~(np.isnan(x) | np.isnan(y))
|
|
x_clean = np.array(x)[mask]
|
|
y_clean = np.array(y)[mask]
|
|
|
|
if method == 'pearson':
|
|
r, p_value = stats.pearsonr(x_clean, y_clean)
|
|
elif method == 'spearman':
|
|
r, p_value = stats.spearmanr(x_clean, y_clean)
|
|
else:
|
|
raise ValueError("method must be 'pearson' or 'spearman'")
|
|
|
|
n = len(x_clean)
|
|
|
|
# Fisher's z transformation for CI
|
|
if method == 'pearson':
|
|
z = 0.5 * np.log((1 + r) / (1 - r))
|
|
se = 1 / np.sqrt(n - 3)
|
|
z_crit = stats.norm.ppf(1 - alpha/2)
|
|
ci_lower = np.tanh(z - z_crit * se)
|
|
ci_upper = np.tanh(z + z_crit * se)
|
|
else:
|
|
ci_lower, ci_upper = None, None
|
|
|
|
# Coefficient of determination
|
|
r_squared = r ** 2
|
|
|
|
return {
|
|
'test': f'{method.capitalize()} correlation',
|
|
'r': r,
|
|
'r_squared': r_squared,
|
|
'p_value': p_value,
|
|
'significant': p_value < alpha,
|
|
'ci_95_lower': ci_lower,
|
|
'ci_95_upper': ci_upper,
|
|
'n': n
|
|
}
|
|
|
|
@staticmethod
|
|
def chi_square(contingency_table, alpha: float = 0.05) -> Dict:
|
|
"""Chi-square test of independence"""
|
|
chi2, p_value, df, expected = stats.chi2_contingency(contingency_table)
|
|
|
|
# Effect size (Cramer's V)
|
|
n = np.sum(contingency_table)
|
|
min_dim = min(contingency_table.shape[0] - 1,
|
|
contingency_table.shape[1] - 1)
|
|
cramers_v = np.sqrt(chi2 / (n * min_dim))
|
|
|
|
return {
|
|
'test': 'Chi-square test of independence',
|
|
'chi2_statistic': chi2,
|
|
'df': df,
|
|
'p_value': p_value,
|
|
'significant': p_value < alpha,
|
|
'cramers_v': cramers_v,
|
|
'expected_frequencies': expected
|
|
}
|
|
|
|
@staticmethod
|
|
def format_apa(result: Dict) -> str:
|
|
"""Format result in APA style"""
|
|
test = result['test']
|
|
|
|
if 't-test' in test:
|
|
return (f"t({result['df']}) = {result['t_statistic']:.2f}, "
|
|
f"p = {result['p_value']:.3f}, d = {result['cohens_d']:.2f}")
|
|
|
|
elif 'ANOVA' in test:
|
|
return (f"F({result['df_between']}, {result['df_within']}) = "
|
|
f"{result['f_statistic']:.2f}, p = {result['p_value']:.3f}, "
|
|
f"η² = {result['eta_squared']:.2f}")
|
|
|
|
elif 'correlation' in test:
|
|
return (f"r({result['n']-2}) = {result['r']:.2f}, "
|
|
f"p = {result['p_value']:.3f}")
|
|
|
|
elif 'Chi-square' in test:
|
|
return (f"χ²({result['df']}) = {result['chi2_statistic']:.2f}, "
|
|
f"p = {result['p_value']:.3f}, V = {result['cramers_v']:.2f}")
|
|
|
|
return str(result)
|
|
|
|
# Example usage
|
|
control = np.random.normal(50, 10, 30)
|
|
treatment = np.random.normal(55, 10, 30)
|
|
|
|
result = InferentialAnalysis.t_test(control, treatment)
|
|
print(InferentialAnalysis.format_apa(result))
|
|
print(f"Interpretation: {result['effect_size']} effect size")</code></pre>
|
|
<h2>Qualitative Data Analysis</h2>
|
|
<h3>1. Coding Process</h3>
|
|
<p><strong>Systematic Coding Framework</strong>:</p>
|
|
<pre><code class="lang-python">from dataclasses import dataclass
|
|
from typing import List, Dict, Optional
|
|
from collections import defaultdict
|
|
import pandas as pd
|
|
|
|
@dataclass
|
|
class Code:
|
|
"""Represent a qualitative code"""
|
|
name: str
|
|
definition: str
|
|
when_to_use: str
|
|
when_not_to_use: str
|
|
examples: List[str]
|
|
|
|
@dataclass
|
|
class CodedSegment:
|
|
"""Represent coded text segment"""
|
|
source: str
|
|
text: str
|
|
codes: List[str]
|
|
line_numbers: tuple
|
|
memo: str = ""
|
|
|
|
class QualitativeCoding:
|
|
"""Systematic qualitative coding"""
|
|
|
|
def __init__(self):
|
|
self.codebook = {}
|
|
self.coded_segments = []
|
|
self.memos = []
|
|
|
|
def add_code(self, code: Code):
|
|
"""Add code to codebook"""
|
|
self.codebook[code.name] = code
|
|
|
|
def apply_code(self, segment: CodedSegment):
|
|
"""Apply codes to text segment"""
|
|
# Validate codes exist
|
|
for code in segment.codes:
|
|
if code not in self.codebook:
|
|
print(f"Warning: Code '{code}' not in codebook")
|
|
|
|
self.coded_segments.append(segment)
|
|
|
|
def add_memo(self, memo: str, codes: Optional[List[str]] = None):
|
|
"""Write analytical memo"""
|
|
self.memos.append({
|
|
'memo': memo,
|
|
'codes': codes,
|
|
'date': pd.Timestamp.now()
|
|
})
|
|
|
|
def get_code_frequency(self):
|
|
"""Count code applications"""
|
|
freq = defaultdict(int)
|
|
for segment in self.coded_segments:
|
|
for code in segment.codes:
|
|
freq[code] += 1
|
|
return dict(freq)
|
|
|
|
def get_coded_segments(self, code: str) -> List[CodedSegment]:
|
|
"""Retrieve all segments with a specific code"""
|
|
return [seg for seg in self.coded_segments if code in seg.codes]
|
|
|
|
def co_occurrence_matrix(self):
|
|
"""Create code co-occurrence matrix"""
|
|
codes = list(self.codebook.keys())
|
|
matrix = pd.DataFrame(0, index=codes, columns=codes)
|
|
|
|
for segment in self.coded_segments:
|
|
seg_codes = segment.codes
|
|
for code1 in seg_codes:
|
|
for code2 in seg_codes:
|
|
if code1 != code2:
|
|
matrix.loc[code1, code2] += 1
|
|
|
|
return matrix
|
|
|
|
def inter_rater_reliability(self, coder2_segments: List[CodedSegment]):
|
|
"""Calculate Cohen's Kappa for inter-rater reliability"""
|
|
# Match segments by source and line numbers
|
|
agreements = 0
|
|
total = 0
|
|
|
|
for seg1 in self.coded_segments:
|
|
# Find matching segment from coder 2
|
|
matching = [s for s in coder2_segments
|
|
if s.source == seg1.source and
|
|
s.line_numbers == seg1.line_numbers]
|
|
|
|
if matching:
|
|
seg2 = matching[0]
|
|
# Check if codes match
|
|
if set(seg1.codes) == set(seg2.codes):
|
|
agreements += 1
|
|
total += 1
|
|
|
|
if total == 0:
|
|
return None
|
|
|
|
observed_agreement = agreements / total
|
|
|
|
# For simplicity, calculate expected agreement assuming random
|
|
# (Full Cohen's Kappa requires more complex calculation)
|
|
code_freq = self.get_code_frequency()
|
|
total_codes = sum(code_freq.values())
|
|
expected_agreement = sum((freq/total_codes)**2
|
|
for freq in code_freq.values())
|
|
|
|
kappa = (observed_agreement - expected_agreement) / (1 - expected_agreement)
|
|
|
|
return {
|
|
'observed_agreement': observed_agreement,
|
|
'expected_agreement': expected_agreement,
|
|
'cohens_kappa': kappa,
|
|
'interpretation': 'substantial' if kappa > 0.6 else
|
|
'moderate' if kappa > 0.4 else 'fair'
|
|
}
|
|
|
|
def generate_codebook_document(self):
|
|
"""Export codebook"""
|
|
doc = "# Qualitative Codebook\n\n"
|
|
|
|
for code_name, code in self.codebook.items():
|
|
doc += f"## {code_name}\n\n"
|
|
doc += f"**Definition**: {code.definition}\n\n"
|
|
doc += f"**When to use**: {code.when_to_use}\n\n"
|
|
doc += f"**When NOT to use**: {code.when_not_to_use}\n\n"
|
|
doc += "**Examples**:\n"
|
|
for ex in code.examples:
|
|
doc += f"- {ex}\n"
|
|
doc += "\n"
|
|
|
|
# Add frequency
|
|
freq = self.get_code_frequency().get(code_name, 0)
|
|
doc += f"*Applied {freq} times*\n\n"
|
|
doc += "---\n\n"
|
|
|
|
return doc
|
|
|
|
# Example usage
|
|
coding = QualitativeCoding()
|
|
|
|
# Build codebook
|
|
coding.add_code(Code(
|
|
name='boundary_work',
|
|
definition='Actions taken to create or maintain work-life boundaries',
|
|
when_to_use='When participant describes specific strategies or behaviors '
|
|
'for separating work and personal life',
|
|
when_not_to_use='When simply mentioning boundaries without action',
|
|
examples=[
|
|
'"I close my laptop at 5pm every day"',
|
|
'"I have a separate room for work"'
|
|
]
|
|
))
|
|
|
|
coding.add_code(Code(
|
|
name='role_conflict',
|
|
definition='Tension between work and personal role demands',
|
|
when_to_use='When participant describes competing demands or identity conflicts',
|
|
when_not_to_use='When mentioning roles without conflict',
|
|
examples=[
|
|
'"I felt torn between meeting and picking up my kids"',
|
|
'"I can\'t be fully present at home because I\'m thinking about work"'
|
|
]
|
|
))
|
|
|
|
# Code data
|
|
coding.apply_code(CodedSegment(
|
|
source='Interview_P001',
|
|
text='I always close my laptop at 5pm, even if work isn\'t done',
|
|
codes=['boundary_work'],
|
|
line_numbers=(45, 47),
|
|
memo='Strong temporal boundary - clock-based'
|
|
))
|
|
|
|
coding.apply_code(CodedSegment(
|
|
source='Interview_P001',
|
|
text='But then I feel guilty about unfinished tasks',
|
|
codes=['role_conflict'],
|
|
line_numbers=(48, 49),
|
|
memo='Guilt suggests incomplete boundary - work identity bleeds through'
|
|
))
|
|
|
|
# Analyze
|
|
print(coding.get_code_frequency())
|
|
print(coding.generate_codebook_document())</code></pre>
|
|
<h3>2. Thematic Development</h3>
|
|
<p><strong>Theme Building Process</strong>:</p>
|
|
<pre><code class="lang-python">class ThematicDevelopment:
|
|
"""Develop and refine themes from codes"""
|
|
|
|
def __init__(self, coding: QualitativeCoding):
|
|
self.coding = coding
|
|
self.themes = {}
|
|
|
|
def create_theme(self, theme_name: str, codes: List[str],
|
|
description: str):
|
|
"""Group codes into theme"""
|
|
self.themes[theme_name] = {
|
|
'codes': codes,
|
|
'description': description,
|
|
'definition': '',
|
|
'subthemes': []
|
|
}
|
|
|
|
def define_theme(self, theme_name: str, definition: str):
|
|
"""Provide clear theme definition"""
|
|
self.themes[theme_name]['definition'] = definition
|
|
|
|
def add_subtheme(self, theme_name: str, subtheme_name: str,
|
|
codes: List[str]):
|
|
"""Add subtheme"""
|
|
self.themes[theme_name]['subthemes'].append({
|
|
'name': subtheme_name,
|
|
'codes': codes
|
|
})
|
|
|
|
def theme_prevalence(self, theme_name: str):
|
|
"""Calculate theme prevalence"""
|
|
theme = self.themes[theme_name]
|
|
segments = []
|
|
|
|
for code in theme['codes']:
|
|
segments.extend(self.coding.get_coded_segments(code))
|
|
|
|
# Get unique sources
|
|
sources = set(seg.source for seg in segments)
|
|
|
|
return {
|
|
'theme': theme_name,
|
|
'n_segments': len(segments),
|
|
'n_sources': len(sources),
|
|
'percentage_sources': (len(sources) /
|
|
len(set(s.source for s in self.coding.coded_segments))) * 100
|
|
}
|
|
|
|
def generate_theme_map(self):
|
|
"""Create visual theme hierarchy"""
|
|
map_str = "# Thematic Map\n\n"
|
|
|
|
for theme_name, theme in self.themes.items():
|
|
map_str += f"## {theme_name}\n"
|
|
map_str += f"{theme['description']}\n\n"
|
|
map_str += f"**Codes**: {', '.join(theme['codes'])}\n\n"
|
|
|
|
if theme['subthemes']:
|
|
map_str += "**Subthemes**:\n"
|
|
for subtheme in theme['subthemes']:
|
|
map_str += f"- {subtheme['name']}: "
|
|
map_str += f"{', '.join(subtheme['codes'])}\n"
|
|
|
|
prev = self.theme_prevalence(theme_name)
|
|
map_str += f"\n*{prev['n_sources']} sources "
|
|
map_str += f"({prev['percentage_sources']:.0f}%), "
|
|
map_str += f"{prev['n_segments']} segments*\n\n"
|
|
|
|
return map_str
|
|
|
|
# Example
|
|
thematic = ThematicDevelopment(coding)
|
|
|
|
thematic.create_theme(
|
|
'Negotiating Boundaries',
|
|
codes=['boundary_work', 'role_conflict'],
|
|
description='The ongoing process of creating and maintaining work-life separation'
|
|
)
|
|
|
|
thematic.define_theme(
|
|
'Negotiating Boundaries',
|
|
'Remote workers actively and continuously construct boundaries between work '
|
|
'and personal life through spatial, temporal, and communicative strategies, '
|
|
'while managing guilt and role conflicts that arise from boundary violations.'
|
|
)
|
|
|
|
print(thematic.generate_theme_map())</code></pre>
|
|
<h2>Data Visualization Best Practices</h2>
|
|
<pre><code class="lang-python">import matplotlib.pyplot as plt
|
|
import seaborn as sns
|
|
|
|
class VisualizationGuide:
|
|
"""Create publication-ready visualizations"""
|
|
|
|
@staticmethod
|
|
def set_publication_style():
|
|
"""Set consistent style for publication"""
|
|
sns.set_style('whitegrid')
|
|
sns.set_context('paper', font_scale=1.2)
|
|
plt.rcParams['font.family'] = 'sans-serif'
|
|
plt.rcParams['font.sans-serif'] = ['Arial']
|
|
plt.rcParams['figure.dpi'] = 300
|
|
|
|
@staticmethod
|
|
def comparison_plot(groups: Dict[str, list], ylabel: str, title: str):
|
|
"""Create comparison plot with error bars"""
|
|
fig, ax = plt.subplots(figsize=(8, 6))
|
|
|
|
x_pos = range(len(groups))
|
|
means = [np.mean(data) for data in groups.values()]
|
|
sems = [stats.sem(data) for data in groups.values()]
|
|
|
|
ax.bar(x_pos, means, yerr=sems, capsize=5,
|
|
color='steelblue', alpha=0.8, edgecolor='black')
|
|
|
|
ax.set_xticks(x_pos)
|
|
ax.set_xticklabels(groups.keys())
|
|
ax.set_ylabel(ylabel)
|
|
ax.set_title(title)
|
|
ax.spines['top'].set_visible(False)
|
|
ax.spines['right'].set_visible(False)
|
|
|
|
return fig
|
|
|
|
@staticmethod
|
|
def scatter_with_regression(x, y, xlabel: str, ylabel: str):
|
|
"""Scatter plot with regression line"""
|
|
fig, ax = plt.subplots(figsize=(8, 6))
|
|
|
|
ax.scatter(x, y, alpha=0.6, s=50)
|
|
|
|
# Add regression line
|
|
z = np.polyfit(x, y, 1)
|
|
p = np.poly1d(z)
|
|
ax.plot(x, p(x), 'r--', linewidth=2)
|
|
|
|
# Add correlation
|
|
r, p_val = stats.pearsonr(x, y)
|
|
ax.text(0.05, 0.95, f'r = {r:.3f}, p = {p_val:.3f}',
|
|
transform=ax.transAxes, verticalalignment='top',
|
|
bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))
|
|
|
|
ax.set_xlabel(xlabel)
|
|
ax.set_ylabel(ylabel)
|
|
ax.spines['top'].set_visible(False)
|
|
ax.spines['right'].set_visible(False)
|
|
|
|
return fig</code></pre>
|
|
<h2>Best Practices</h2>
|
|
<h3>Quantitative Analysis</h3>
|
|
<ul><li>Check assumptions before tests</li><li>Report effect sizes, not just p-values</li><li>Provide confidence intervals</li><li>Correct for multiple comparisons</li><li>Document all analysis decisions</li><li>Report negative results</li><li>Make data and code available</li></ul>
|
|
<h3>Qualitative Analysis</h3>
|
|
<ul><li>Code systematically with clear definitions</li><li>Calculate inter-rater reliability</li><li>Write analytic memos throughout</li><li>Seek negative cases</li><li>Use software for transparency</li><li>Provide thick description</li><li>Maintain audit trail</li></ul>
|
|
<h3>Visualization</h3>
|
|
<ul><li>Choose appropriate chart type</li><li>Label axes clearly</li><li>Include error bars or confidence intervals</li><li>Use colorblind-friendly palettes</li><li>Remove chart junk</li><li>Export at high resolution</li><li>Caption thoroughly</li></ul>
|
|
<h2>Related Skills</h2>
|
|
<ul><li><strong>quantitative-methods</strong>: Quantitative research approaches</li><li><strong>qualitative-methods</strong>: Qualitative research approaches</li><li><strong>research-synthesis</strong>: Analyzing across studies</li><li><strong>data-collection</strong>: Obtaining analyzable data</li><li><strong>research-writing</strong>: Reporting analyses</li></ul>
|
|
<h2>Quick Reference</h2>
|
|
<h3>Test Selection Guide</h3>
|
|
<pre><code>Compare 2 groups: Independent t-test
|
|
Compare 2 timepoints: Paired t-test
|
|
Compare 3+ groups: ANOVA
|
|
Relationship: Correlation
|
|
Prediction: Regression
|
|
Categorical association: Chi-square
|
|
Non-normal data: Non-parametric alternatives</code></pre>
|
|
<h3>Effect Size Interpretation</h3>
|
|
<pre><code>Cohen's d: 0.2 small, 0.5 medium, 0.8 large
|
|
Correlation: 0.1 small, 0.3 medium, 0.5 large
|
|
Eta-squared: 0.01 small, 0.06 medium, 0.14 large</code></pre>
|
|
<h3>Coding Reliability Standards</h3>
|
|
<pre><code>Cohen's Kappa:
|
|
< 0.20: Poor
|
|
0.21-0.40: Fair
|
|
0.41-0.60: Moderate
|
|
0.61-0.80: Substantial
|
|
0.81-1.00: Almost perfect</code></pre>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/research-data-analysis.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |