AB Test Calculator
Version B beat version A by 25%. Version B also might have just gotten lucky. Enter each variant’s visitors and conversions, and this calculator runs the two-proportion z-test that separates the two stories — rates, absolute and relative uplift, the z-score, the p-value, and a plain-language verdict you can paste into the launch review.
The Two-Proportion Z-Test, in One Honest Breath
Each variant's conversion rate is a proportion built from coin flips — visitor converts or not — so each carries binomial sampling noise of a known size. The test pools both variants' data into one overall rate (the null hypothesis's world, where A and B are secretly identical), computes the standard error of the difference under that pooling, SE = √(p̂(1 − p̂)(1/nA + 1/nB)), and asks how many standard errors the observed difference spans: z = (pB − pA) ÷ SE. The p-value is then the two-sided normal tail beyond |z| — the probability that pure luck, dealing identical variants, would produce a gap at least this wide in either direction. Every figure the calculator prints derives from those three lines; there is no dashboard magic to audit beyond them, which is exactly how test statistics should feel — inevitable once stated.
Worked Example: A 25% Uplift That Barely Clears the Bar
The defaults stage the classic near-miss: A converts 120 of 1,000 (12.00%), B converts 150 of 1,000 (15.00%) — an uplift of 3.00 points, 25.0% relative, the stuff of triumphant sprint reviews. The pooled rate is 13.5%, the standard error √(0.135 × 0.865 × 0.002) ≈ 0.01528, and z = 0.03 ÷ 0.01528 ≈ 1.963, giving p ≈ 0.0496 — significant at 95%, by a hair's breadth of four ten-thousandths. The example is chosen to teach the fragility inside the triumph: two fewer conversions for B and the result slips over the line into "not significant," which is why professionals treat 0.049 and 0.051 as nearly identical evidence (they are) while rituals treat them as opposite verdicts (they are not). Report the p-value, the effect size and the sample sizes together, as this page prints them, and the reader gets the evidence; report only "significant," and they get a coin flip about a coin flip. The habit costs nothing — the numbers are already on the screen — and it is the single clearest marker separating teams that experiment from teams that merely A/B test.
Significance Is Not a Shipping Decision
The calculator's note in the non-significant case prescribes the honest remedy — more traffic, sized in advance by a power analysis — and its significant case carries a warning for a reason. A significant result says the gap is unlikely under the null; it does not say the gap is large enough to matter (a 0.1-point uplift is significant at a million visitors and irrelevant at any margin), nor that the test was run honestly (a test peeked at daily and stopped at the first p under 0.05 has an effective error rate several times the nominal one — sequential testing is its own discipline, and the peeking FAQ below is this page's conscience), nor that B is better for the segments that pay (aggregate wins can conceal segment losses, Simpson's paradox in a growth team's clothing). The z-test is the trial's verdict on one narrow question — could luck do this? — and shipping decisions deserve the wider court: effect size, cost of being wrong, segment behaviour and the roadmap the slot could serve. Use this page for the narrow question; it answers it exactly, and it refuses — by printing its standard error's ingredients in the open — to let the answer masquerade as broader than it is.
From Fisher's Fields to the Internet's Laboratory
The hypothesis test this page runs was forged in agricultural plots: Fisher's Rothamsted experiments of the 1920s compared crop treatments with exactly the logic you have just applied to landing pages, and the migration of that logic to the internet changed only the sample sizes — from dozens of plots to millions of visitors — and the cycle time, from seasons to sprints. The continuity carries a warning inside it: Fisher's framework assumes the experiment was designed before it was run, with its comparisons and stopping rule committed in advance, because his fields could not be peeked at into significance the way a dashboard can. The internet's gift to experimentation is scale; its curse is the temptation to run Fisher's verdict machinery without Fisher's discipline. This page computes his arithmetic; the FAQs plead for his discipline; the combination is the whole professional inheritance.
Guardrails: Novelty, Segments and the Second Week
Two failure modes ambush tests that pass this page cleanly, and both deserve print. The novelty effect: a redesign's first-week uplift can be the audience's curiosity rather than the design's merit, and it decays — which is one reason the runtime guidance insists on full weekly cycles and why veteran teams watch the effect's stability across the run, not just its final p. The segment trap: an aggregate win can be one segment's triumph papering over another's loss (new visitors love it, loyal customers quietly convert less), and a shipping decision that never disaggregates will eventually be ambushed by its own dashboard. Neither hazard is visible to the z-test, and neither excuses skipping it: the test certifies the aggregate's reality, and the analyst's craft audits its composition. Evidence first, then anatomy — in that order, every time, on every test the organisation runs this year and next.
Related Tools
More from the Statistics Calculators hub: the Power Analysis Calculator to size the test before you run it, and the Hypothesis Testing Calculator for the one-sample questions this page's two-sample test generalises.
Frequently Asked Questions
How long should I run my A/B test, then?
Until it reaches the sample size your power analysis prescribed for the smallest effect worth detecting — and through at least one full weekly cycle of your traffic's behaviour, because Tuesday's visitors are not Sunday's. The two constraints together (enough n, enough calendar) replace the folklore of fixed fortnights: a high-traffic site may satisfy both in days; a low-traffic one may need months, during which the honest answer to "which is winning?" is the calculator's shrug at small samples. Stopping early at a significant peek is the peeking fallacy below; stopping late wastes the slot; the prescription set before launch is the only stopping rule that keeps the p-value honest.
What is the peeking problem, exactly?
Checking the test daily and shipping the moment p dips under 0.05 multiplies your false-positive rate — with continuous peeking it can exceed 20–30% instead of the nominal 5%, because you are giving luck repeated chances to cross the line once. The mathematics is unforgiving: the p-value's guarantee holds only for a single look at a pre-committed sample size (or under sequential methods designed for repeated looks, with adjusted boundaries). Run to the planned n, then look — the calculator will be here, and its verdict will mean what it says.
One-sided or two-sided, and why?
This calculator reports the two-sided p-value — the probability of a gap this wide in either direction — because B being secretly worse is information a shipping decision needs, and because two-sided is the conservative default the field defends. A one-sided test (halving the p-value when the direction matches the prediction) is legitimate only when the direction and the one-sidedness were committed before the data arrived; choosing it after seeing B ahead is a quiet doubling of the false-positive rate. When in doubt, two-sided — the worked example's hair's-breadth verdict is the honest price of the honesty.
My uplift is significant but tiny. Ship it anyway?
Maybe — the decision is economics, not statistics. A 0.2-point conversion uplift across a hundred million annual visitors is a fortune; the same uplift on a landing page seen by thousands is a rounding error that costs a codebase's complexity forever. Multiply the uplift by traffic and margin, subtract the maintenance and the opportunity cost of the test slot, and let that arithmetic decide; the calculator's role ends at certifying the uplift is real. Significance answers "is it there?", never "is it worth it?" — the two questions' confusion funds a thousand shipped trivialities.
What if the variants have very unequal traffic splits?
The pooled formula handles unequal n natively (the 1/nA + 1/nB term prices each variant's noise), so a 90/10 split computes correctly — it just computes a weaker test than the same total traffic split evenly, because the small variant's noise dominates the difference's standard error. Unequal splits are sometimes chosen to limit a risky variant's exposure; the cost is real power, priced by the standard error on this page. Balance is statistically free and operationally brave; imbalance is the reverse — choose knowing the exchange rate.
Can I test three or more variants here, too?
Not honestly in one pass: pairwise z-tests among several variants multiply the false-positive risk (three variants make three comparisons; at 5% each, the family error rate approaches 14%), which is why multi-variant tests use corrections (Bonferroni and kin) or omnibus tests first. A pragmatic path: test the challenger against control here, treat additional variants as separate experiments, and apply a correction to your significance threshold when the comparisons were planned as a family. The calculator's two-variant honesty is a feature; the discipline of not silently extending it is yours.