Formula and method
- p₁ = baseline rate; p₂ = p₁ × (1 + relative increase); p̄ = (p₁ + p₂) ÷ 2
- n per arm ≈ [z₁₋α/₂ × √(2p̄(1−p̄)) + zpower × √(p₁(1−p₁)+p₂(1−p₂))]² ÷ (p₂−p₁)²
- Round n up. Total participants = 2n. Sample collection days = ceiling(2n ÷ total daily eligible participants).
Worked example
Illustrative baseline: 5%. Relative increase to detect: 20%, giving a 6% target. Confidence: 95%. Power: 80%.
The approximation requires 8,158 participants per arm, or 16,316 total. At 1,000 eligible participants per day, collecting the sample takes at least 17 days. Conversion maturity and operational constraints can require a longer experiment.
Choose an effect that would change a decision
Start with the smallest improvement that would make the proposed change worthwhile. A tiny MDE can create a sample requirement your traffic cannot support. A very large MDE can make a test feasible while leaving smaller, still valuable improvements unresolved. The right input is a business decision, not the number that makes the calculator output convenient.
Use a baseline from comparable, mature traffic. Keep the conversion definition and observation window fixed. If the baseline came from a broad website average but the experiment includes only qualified paid visitors, the planning assumptions may be wrong before the first participant is assigned.
Understand confidence, power and the approximation
The confidence setting determines the two-sided significance threshold. Power is the probability of detecting the specified effect under the planning assumptions. An 80% power setting is not an 80% probability that a winning variation is better.
The calculation uses a normal approximation with pooled variance under the null and separate outcome variance under the alternative. It omits the usually negligible far tail of the two-sided power calculation, following the approach documented by statsmodels. Small outcome counts or unusual designs need a different method.
These settings do not correct a biased allocation, tracking failure, repeated observations from the same participant or many comparisons. Predefine one primary outcome and one comparison. Use a suitable multiple-testing or sequential method if the design requires it.
Turn the estimate into a test plan
Record the assignment unit, eligible population, allocation, primary metric, conversion window, start date and planned stopping rule. State what happens if tracking breaks or recruitment is slower than expected. Save those decisions before results are visible.
The time output is only a recruitment estimate. Allow the last participants to finish the conversion window and account for weekly demand patterns. Do not end a test early just because one variant is ahead on Tuesday.
An advertising platform allocating delivery across creatives is not automatically running this experiment. Use a platform experiment with a documented random split or a properly designed landing-page test. For low-volume accounts, fewer test arms and a more material hypothesis are often more practical than dozens of variants.