Analyse a finished A/B test and write an honest results readout with real statistics. Use when asked to read out an A/B test, analyse experiment results, che...
数据分析
Ab Test Readout
试用Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship c...
它能做什么
Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call. Use when asked to analyse experiment results, write an A/B test readout, interpret test data, or decide whether to ship a variant. Produces a clear verdict with the lift and confidence, segment cuts, the risks (peeking, novelty, sample), and a recommendation. Distinct from planning a test — this reads results.
技能文档
A/B Test Readout Skill
The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie.
Working from a brief
Given results (even partial), write the full readout anyway. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample.
Required Inputs
Ask for (if not already provided):
- The hypothesis and the primary metric
- Results — control vs variant: conversions/rate, sample size per arm, duration
- Guardrail metrics (revenue, retention, latency, complaints) that mustn't regress
- Pre-registered decision rule (what would count as a win) if one exists
Output Format
1. Verdict (one line)
Ship / Don't ship / Inconclusive — keep running — with the headline number.
2. The result
| Metric | Control | Variant | Relative lift | Significant? |
|---|---|---|---|---|
| Primary | p / CI | |||
| Guardrail(s) |
State statistical significance (p-value / confidence interval) and practical significance (is the lift big enough to matter given the cost?).
3. Did it really win?
Address the ways A/B tests mislead:
- Sample / power — was the test adequately powered, or under-sampled?
- Peeking — was the call made early, inflating false positives?
- Novelty / primacy — could the effect fade?
- Segments — does the win hold across key segments, or is it driven by one?
4. Segment cuts
Where the effect is strong vs flat vs negative (new vs returning, platform, geography).
5. Recommendation & next step
Ship / iterate / re-run, plus what to monitor post-launch or what the follow-up test should isolate.
Quality Checks
- Distinguishes statistical from practical significance
- Checks guardrail metrics, not just the primary
- Flags peeking, power, novelty, and segment-driven wins
- Recommendation follows from the evidence, with a monitoring/next-test step
- Doesn't declare a winner on an underpowered or peeked result
Anti-Patterns
- "B won by 8%!" with no significance or sample size
- Calling a result early (peeking) and shipping
- Ignoring a guardrail regression because the primary went up
- A statistically significant but practically meaningless lift treated as a win
相关技能
Design statistically rigorous A/B tests for product features, UI changes, onboarding flows, and pricing experiments. Use when asked to set up an experiment,...
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, int...
2 次安装
改版犹豫不决？A/B测试怎么设计才有说服力？从假设定义、样本量计算到结果解读，全流程帮你搞定。支持贝叶斯、序贯检验、多臂老虎机，7步走完一个严谨实验，让你用数据拍板而不是拍脑袋。 触发词：A/B测试、AB测试、对照实验、样本量计算、实验设计、灰度发布、转化率优化、多臂老虎机、epsilon-greedy、UCB、...
4 次安装
Use when the user asks to "design an A/B test", "set up a creative/landing test", "run an incrementality test", or "is this result statistically and practica...
Design and evaluate A/B tests for different price points, discount levels, and bundle combinations to find the highest-converting offer structure.
16 次安装