Power Analysis Calculator — Sample Sizes

The worst A/B test is the one that could never have answered its question — too little traffic for the effect it hunted. Enter your baseline conversion rate and the smallest uplift worth detecting, and this calculator returns the sample size per group your test needs, at the power and confidence you choose.

The Formula and Its Four Honest Dials

Sample size for comparing two proportions is one of applied statistics’ most-used formulas, and every symbol in it is a decision: n = (zα + zβ)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)² per group. The denominator is the effect you refuse to miss, squared — the formula’s tyrant, responsible for the note’s warning that halving the detectable effect quadruples the cost. The bracket is the two rates’ binomial noise. And the z-sum prices your two error tolerances together: zα for the false-positive rate you accept (significance), zβ for the false-negative rate (one minus power). Power of 80% at 95% confidence — the defaults, and the industry’s handshake — means: if the true uplift is exactly the one entered, this test detects it four times in five, while crying wolf on identical variants one time in twenty. Every dial turned toward certainty is paid for in the only currency tests accept: traffic, and the calendar it arrives on.

Worked Example: Detecting 10% → 12%

The defaults size a test that haunts every growth team: baseline conversion 10%, and the uplift worth shipping is 2 points, to 12%. The z-sum at the standard settings is 1.96 + 0.84 = 2.80, squared ≈ 7.85. The noise bracket: 0.10 × 0.90 + 0.12 × 0.88 = 0.09 + 0.1056 = 0.1956. The effect squared: 0.02² = 0.0004. Division: 7.85 × 0.1956 ÷ 0.0004 ≈ 3,839 visitors per group — about 7,700 total — to have an 80% chance of certifying a real 2-point uplift at the 95% level. The note then administers the formula’s cold shower: aim at 11% instead (a 1-point effect) and the requirement leaps to roughly 14,740 per group, nearly quadruple, exactly as the squared denominator promised. Teams that internalise this one example stop asking analytics for miracles at 500 visitors — the arithmetic of evidence has a minimum order size, and this page prints it before the sprint planning, not after the post-mortem.

Power: The Probability Your Test Deserves Its Budget Line

An underpowered test is worse than no test, and the reason deserves its section. Run at 30% power, a real effect still returns “not significant” seven times in ten — and the organisation learns, falsely, that the idea failed, which is misinformation with a lab coat. Worse, the rare underpowered test that does reach significance systematically overstates the effect (it only crossed the line on a lucky upward fluctuation — the winner’s curse), planting inflated expectations the rollout cannot meet. Power analysis is therefore not bureaucratic throat-clearing before the real work; it is the calculation that decides whether the experiment can produce knowledge at all, and its verdict is sometimes “do not run this test at this traffic — accumulate the change into a bigger one, or decide by judgment and spend the traffic learning something learnable.” The calculator’s neutrality includes delivering that verdict without flinching.

The Ethics of the Underpowered Study, Stated Plainly

In medicine, this calculator’s arithmetic carries moral weight the marketing version only borrows: a clinical trial sized too small to detect the effect it hunts exposes its participants to risk and burden while being structurally incapable of producing the knowledge that justified them — which is why ethics boards demand power calculations in protocols, and why the demand generalises. The A/B test that cannot reach its sample size is the commercial echo: it spends users’ experiences and the organisation’s attention on a question it cannot answer, and then — the deeper injury — publishes its shrug as a finding. Sizing a study before running it is therefore not statistical etiquette but a duty owed to everyone whose data the study will consume. This page exists so that the duty takes ninety seconds and one honest conversation.

Sequential Designs and the Modern Escape Hatches

The fixed-sample discipline this formula encodes has principled modern alternatives, and naming them keeps the page honest rather than hidebound. Sequential and group-sequential designs pre-commit a schedule of interim looks with adjusted boundaries, buying the right to stop early for overwhelming effects without inflating the error rates — the peeking problem solved by mathematics instead of by abstinence. Bayesian experimental frameworks replace the power question with posterior precision targets, answering in the currency of credible intervals what this page answers in error rates. Both are legitimate; neither is a licence to improvise mid-test. The fixed design sized here remains the reference instrument — the one whose behaviour under the null is known to the third decimal — and the alternatives are best understood as its negotiated settlements, not its refutations.

Related Tools

More from the Statistics Calculators hub: the AB Test Calculator — run this one’s output through it when the traffic lands — and the Sampling Error Calculator for the survey-side version of the same noise.

Frequently Asked Questions

Why is 80% power the standard choice?

Convention with a rationale: Cohen proposed it as the balance point where a false negative is treated as roughly four times as costly as a false positive (80/20 against 5% alpha) — a value judgment the field adopted as its default handshake, not a theorem. Tests whose misses are dangerous (safety checks, medical screening analogues) belong at 90% or higher, priced by this calculator’s power selector; tests screening cheap ideas can accept 70%. What is not defensible is the unexamined default of whatever power the traffic happened to give — the calculator exists to make the dial a decision.

What is the minimum detectable effect (MDE), then?

The smallest true uplift your test, at its actual sample size, is powered to certify — this formula read backwards. Fix the traffic you have, solve for (p₂ − p₁), and the MDE tells you the resolution of the instrument you are holding: if the MDE is 3 points, a genuinely-good 1-point improvement will usually read “not significant,” and the organisation should know that before interpreting the shrug. Experienced teams quote the MDE alongside every result; the calculator’s two directions (size from effect, effect from size) are the same equation wearing its two hats, and fluency in both is experimental literacy.

Does the baseline rate really matter that much, though?

Enormously, through the noise bracket: binomial variance p(1−p) peaks at 50%, so a test around a 50% baseline needs the most traffic for a given point-effect, while rare-event baselines (1–2% conversions) need vast traffic for relative uplifts but modest traffic for point uplifts that are large relative to the base. Doubling a 1% rate to 2% (one point) needs about 2,300 per group at standard settings; doubling 20% to 40% (twenty points) needs about 80. Same word “doubling,” thirty-fold difference in price — the calculator is the antidote to planning tests in percentages of percentages.

Should the groups be equal in size?

For pure detection efficiency, yes: at fixed total traffic, equal groups minimise the standard error of the difference, which is why the formula and the calculator are built on equal n per group. Unequal splits (90/10 to protect users from a risky variant) are computed by the general formula with each group’s own n, and always cost power relative to the balanced design at the same total — sometimes a price worth paying for risk or revenue reasons, always a price worth seeing stated, which the equal-group benchmark on this page provides as the reference.

How do I turn visitors-per-group into a runtime estimate?

Divide by the daily traffic the variant will actually receive (total daily visitors × the variant’s share), then round up to complete weeks — the worked rhythm of the AB Test page’s FAQ, repeated here because it is where power analyses go to be betrayed by calendars: a test sized for 3,839 per group at 500 variant-visitors a day needs eight days minimum, and eight days means nine or the nearest whole weeks that cover the weekday cycle honestly. Seasonality, campaigns and launches warp the estimate; the defence is the same as everywhere in this craft — size generously, commit the stopping rule in writing, and let the test run its prescribed course.

What if I cannot reach the required sample size?

Then the formula has told you something worth its fee: at your traffic, this effect is not learnable by this method in acceptable time. The honest options are to test a bigger change (bundle improvements until the expected effect clears your MDE), to accept a lower power or confidence consciously via the selectors — eyes open, error rates stated — to use a more sensitive metric or design (before/after variance reduction, CUPED-style adjustments in mature tooling), or to decide without a test and reserve experimentation for questions it can answer. What the arithmetic forbids is running the small test anyway and believing its verdicts: an underpowered shrug is not evidence of no effect, and this page’s numbers are the receipt.