A/B Test Calculator: Design Experiments and Analyze Significance
What is A/B Testing?
A/B testing randomly splits users or products into a control group (A) and a treatment group (B), then compares key metrics such as conversion rate or click-through rate between the two. It is the standard method for verifying causal effects in product iteration, marketing campaigns and website redesigns; randomization removes confounding and statistical tests control the error rate. Uneven traffic allocation or self-selection destroys experimental validity, so the two groups must be identical in every respect except the treatment.
When to Use It
Use A/B testing whenever a change to a page, feature, price or campaign needs an evidence-based go/no-go decision. The required sample size depends on the baseline conversion rate, the minimum detectable effect (MDE), the significance level α (usually 5%) and power (usually 80%). A lower baseline or a smaller MDE demands far more traffic; underpowered experiments end in false negatives while an overly small MDE makes the experiment run too long, so the plan must balance business need against duration.
How to Use It (Step by Step)
Enter the baseline conversion rate, MDE, α and power on the design page to obtain the required sample size per group, then launch the experiment and collect data. When the experiment ends, enter both groups' sample sizes and conversion counts on the results page to get the lift, p-value and confidence interval for the conversion-rate difference. Review whether the confidence-interval lower bound clears the business threshold before adopting B, and avoid peeking at results before the target sample size is reached.
Key Formulas / Example
The two-proportion Z test uses z = (p̂1 − p̂2) / √(p̄(1−p̄)(1/n1 + 1/n2)), where p̄ is the pooled proportion; B beats A when p < 0.05 and the confidence interval for the difference excludes zero. The per-group sample size follows n = (z_{1−α/2} + z_{1−β})²[p1(1−p1) + p2(1−p2)] / (p1 − p2)². Example: baseline 10%, absolute MDE 2%, α = 0.05, power = 0.8 gives roughly 3,800 users per group.