Files
skills/commands/analyze-test.md
zeekayandHanzo Dev 65c3b27843 160 marketing, growth and PM skills, on our own connectors
Four MIT collections ported — growthOS, openclaw-marketing-skills,
Marketing-Skills and pm-skills — as 160 skills, 12 agents and 45
commands, each naming its origin in its own frontmatter so the tree can
be checked against THIRD_PARTY_NOTICES rather than trusted. Sixteen names
appeared in more than one collection and none of the copies were
identical; the longest wins, which lands on the more developed text where
they genuinely diverge.

A fifth collection is deliberately absent. awesome-growth-hacking-skills
carries NO licence, which leaves its author every right by default — and
it is a list of links rather than a collection of skills, so there is
nothing in it to port and no permission to port it with. It is cited as a
directory to read.

THE CONNECTOR SKILLS ARE REWRITTEN, not ported. They arrived installing
vendor SDKs, running OAuth flows of their own, and ending with long-lived
credentials in JSON files under .agents/ and ~/.growthos/ at mode 600.
That is a secret on a disk with no rotation path, protected from other
users and not from the agent — and one auth implementation per skill.
Hanzo holds these connectors in Go, per user, sealed in KMS, so `connect`
is the one home for the flow and the platform skills defer to it by
provider id: google_ads, meta_ads, x. It is reachable as HTTP or as MCP
tools at api.hanzo.ai/v1/mcp — the same routes, typed once in Go, so a
tool and a curl cannot disagree.

Two had nothing real behind them and now say so. There is no
search_console provider, so that skill names what does answer the same
questions (google_analytics, google_bigquery) and states that the remedy
is a provider in hanzoai/cloud rather than a vendor SDK here. And
getxapi-connect was a wrapper around a paid reseller of X's public data
with the vendor's key in an env var; it is `listen` now, on our own x
connector — a skill named after a vendor makes the vendor the capability
and dies with it.

The skills referenced paths that only existed inside the source repo, so
the shared material they read is ported to reference/ and the paths
rewritten. Three skills need the growthOS APPLICATION rather than its
data; per the estate's own split an application belongs in hanzoai/ads,
social, email or sms as a cloud plugin, so those three say what they need
instead of carrying an import that resolves to nothing.

Third-party product names are out of our prose and kept in the notices,
which is where the licence requires them and the only place they belong.
A skill's persona names a role, not somebody else's product.


Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 01:27:26 -07:00

3.9 KiB

description, argument-hint, source
description argument-hint source
Analyze A/B test results — statistical significance, sample size validation, and ship/extend/stop recommendations <test results as data, screenshot, or description> phuryn/pm-skills

/analyze-test -- A/B Test Analysis

Evaluate experiment results with statistical rigor and translate findings into a clear product decision: ship, extend, or stop.

Invocation

/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)
/analyze-test [upload a CSV of test results]
/analyze-test [screenshot from your experimentation platform]

Workflow

Step 1: Accept Test Data

Accept in any format:

  • Summary statistics (conversion rates, sample sizes per variant)
  • Raw event data (CSV with user_id, variant, converted, timestamp)
  • Screenshot from an experimentation platform (Optimizely, LaunchDarkly, etc.)
  • Description of the experiment and results

Step 2: Validate Test Design

Before analyzing results, check:

  • Was sample size sufficient? (run a power analysis)
  • Was the test run long enough? (capture weekly cycles, minimum 1-2 business cycles)
  • Was randomization clean? (check for sample ratio mismatch)
  • Were there any external factors during the test period?

Flag issues if found — results from a flawed test can be misleading.

Step 3: Analyze Results

Apply the ab-test-analysis skill:

  • Statistical significance: Calculate p-value and confidence interval
  • Effect size: Absolute and relative difference between variants
  • Practical significance: Is the effect large enough to matter for the business?
  • Confidence interval: What's the range of plausible true effects?
  • Segment analysis: If data allows, check for differential effects by user segment

Step 4: Generate Analysis

## A/B Test Analysis: [Test Name]

**Date**: [today]
**Test duration**: [X days/weeks]
**Total sample**: [N users]

### Results Summary
| Variant | Sample | Metric | Rate | 95% CI |
|---------|--------|--------|------|--------|
| Control | [n] | [metric] | [X%] | [X% - Y%] |
| Variant | [n] | [metric] | [X%] | [X% - Y%] |

### Statistical Analysis
- **Relative lift**: [+X%] ([CI range])
- **P-value**: [X]
- **Statistically significant**: [Yes/No] at 95% confidence
- **Minimum detectable effect**: [X%] (what the test was powered to detect)

### Sample Size Check
- **Required sample**: [N] per variant (for [X%] MDE at 80% power)
- **Actual sample**: [N] per variant
- **Verdict**: [Sufficiently powered / Underpowered / Overpowered]

### Decision

**Recommendation: [SHIP / EXTEND / STOP]**

[Clear explanation of why, considering both statistical and practical significance]

### Business Impact Estimate
If shipped to 100% of users:
- **Expected impact**: [metric change per month/quarter]
- **Revenue impact**: [if applicable]
- **Confidence**: [How certain we are about this estimate]

### Caveats
- [Any concerns about the test validity]
- [Segments where results differ]
- [Novelty effects or other biases to consider]

### Follow-Up
- [What to test next based on learnings]
- [Monitoring plan if shipping the variant]

Step 5: Offer Next Steps

  • "Want me to design a follow-up experiment based on these findings?"
  • "Should I run the analysis for specific segments?"
  • "Want me to generate the SQL to monitor this metric post-launch?"

Notes

  • Statistical significance ≠ practical significance — a 0.1% lift can be significant with enough data but not worth shipping
  • Always check for sample ratio mismatch before trusting results
  • Novelty effects can inflate short-term results — recommend monitoring for 2-4 weeks post-launch
  • If the test is underpowered, the right answer is usually "extend" not "no effect"
  • For revenue metrics, use confidence intervals to estimate best-case and worst-case business impact
  • If data is provided as CSV, generate the full analysis using Python with scipy.stats