Hypothesis Testing Calculator

Your sample says one thing; the claim says another. Is the gap evidence, or is it noise? Enter the sample mean, standard deviation and size, the value being claimed, and your significance level — this calculator runs the one-sample test (t or z), and reports the statistic, the p-value and the verdict in plain language that survives a committee.

What the Test Actually Asks, Stripped Down

Strip the ritual to its skeleton and a hypothesis test is one question: if the claim were true, how surprising is my sample? The machinery has three moving parts, and not one of them is optional or decorative. The test statistic measures the gap in noise units — sample mean minus claimed value, divided by the standard error s ÷ √n — so a gap of 5 counts enormously from a precise sample and barely at all from a sloppy one. The p-value converts that statistic, through the t distribution (or the normal, for the z-test), into the probability of a gap at least this large arising from pure sampling luck if the claim held. And the significance level α — chosen in advance, conventionally 0.05 — is the surprise threshold at which you agree to stop believing the claim. Reject, or fail to reject: those are the only verdicts, and the calculator's note on the second one is worded with legal care, because "fail to reject" is the most misread sentence in applied statistics — misread weekly, in boardrooms and journals alike, at prices this page cannot compute for you.

Worked Example: The Filling Machine

The defaults audit a classic: a machine claims to fill to a mean of 100; a sample of 36 fills averages 105 with s = 15. Standard error: 15 ÷ 6 = 2.5. Statistic: t = (105 − 100) ÷ 2.5 = 2.000, on 35 degrees of freedom. The two-sided p-value for t = 2.00 at df 35 is about 0.053 — a whisker above 0.05, so the verdict at the standard level is fail to reject: the machine's overfill, if real, is not yet proven by this sample. Switch the alternative to one-sided ("greater than") and the same data returns p ≈ 0.027 — significant — which is not the test contradicting itself but the two questions differing: the one-sided test spends all its scepticism in one direction, legitimate only when that direction was committed before sampling (the AB Test page's FAQ patrols the same border). And the z-test on identical numbers gives p ≈ 0.0455 — significant two-sided — the reward for pretending σ is known: the t's heavier tails are the honest price of estimating it.

t versus z: The Price of Not Knowing Sigma

The selector between the tests is a selector between two epistemic situations. The z-test applies when the population standard deviation is genuinely known — a calibrated process's long history, a standardised exam's published σ — and it uses the normal distribution, whose tails are thin and verdicts brisk. The t-test applies to the real world's usual case, where σ is estimated from the same sample being tested, and Student's t distribution fattens the tails by exactly the right amount for the sample size (the degrees of freedom, n − 1): at n = 10 the t's scepticism is visibly heavier, at n = 100 the two distributions nearly merge, and at the worked example's 36 they differ just enough to straddle a verdict. Choosing z because it flatters the result is the oldest small sin in testing; the calculator offers both because both are legitimate — under their labelled conditions, which the labels state on the selector itself. When σ is "known" because last year's sample said so, it is not known; it is estimated with extra steps, and the t-test is the honest instrument for it.

The Replication Crisis: This Test's Modern Reckoning

Any honest page on hypothesis testing must file its report from the field's difficult decade: large-scale replication projects in psychology and the social sciences re-ran published significant results and reproduced barely a third to a half of them, and the post-mortems indicted exactly the practices this page's FAQs warn against — peeking and optional stopping, silent multiple comparisons, significance levels shopped after the fact, and publication systems that printed only the p-values under 0.05. The machinery on this page was never the accused; the t-test computes what it computes, correctly, then and now. The crisis's lesson is procedural, and it is the one the worked example's p = 0.053 teaches by accident: treat the test as one disciplined look at pre-committed questions, report exact p-values, and let replication — not ritual — be the arbiter of what is known. A generation of researchers learned to operate this machinery by its thresholds alone; the repair now under way consists of reattaching the thresholds to the reasoning they were always meant to summarise, which is the spirit in which this page prints its verdicts.

Related Tools

More from the Statistics Calculators hub: the AB Test Calculator for the two-sample comparison, the t-statistic Calculator for the statistic alone, and the Critical Value Calculator for the threshold side of the same test.

Frequently Asked Questions

What does the p-value actually mean, precisely?

The probability, computed assuming the null claim is true, of obtaining a sample at least as extreme as yours — a statement about the data given the claim, not about the claim given the data. It is not the probability that the claim is true (that inversion is the most common misreading in all of statistics, and no calculator can perform it without prior probabilities entering through a Bayesian door). Small p says: either the claim is false, or an unlikely sample happened — and the significance level is your pre-agreed exchange rate between those two explanations. Everything else in testing is bookkeeping around that single honest sentence.

Why "fail to reject" instead of "accept the null" outright?

Because the test is structurally incapable of proving the claim — it can only fail to convict it. A small sample with a huge standard error "fails to reject" almost any claim, including absurd ones, which shows the verdict's asymmetry: absence of evidence against μ₀ is not evidence for it. If your decision needs affirmative support for the claimed value (equivalence testing exists for exactly this, with its own machinery), a standard test's shrug is the wrong instrument. The calculator's note is worded accordingly, and the wording is the lesson: precise verdicts prevent expensive misreadings.

What are Type I and Type II errors, then?

The two ways the verdict can be wrong: Type I, rejecting a true claim (a false alarm, probability α by construction), and Type II, failing to reject a false claim (a miss, probability β, whose complement is the test's power). The two errors trade against each other through sample size — the Power Analysis calculator on this hub prices the trade — and which one you fear more is a decision-theoretic question about your costs (a false alarm scraps a good machine; a miss ships underfilled product), not a statistical one. Statistics hands you both dials; judgment sets them.

How large a sample do I need, then?

Enough that the standard error makes the effect you care about span several units: the worked example's t of exactly 2.0 shows a sample sized to the edge of detectability for its gap. Planning forward uses the power machinery (the hub's Power Analysis calculator for proportions; for means, the same algebra with σ and the target gap) — and the retrospective rule is the standard error's √n law: quadrupling n halves the noise, doubling the statistic for the same gap. Tests are bought by the square root, which is why precision is expensive and why the sample size belongs in the planning meeting, not the results meeting.

Does the test assume my data is normally distributed?

The t-test's exactness does; its usefulness mostly does not. What the test truly needs is a roughly normal sampling distribution of the mean, which the central limit theorem supplies for moderate n from almost any well-behaved population — hence the craft rule that t-tests are trustworthy from n ≈ 30 upward for ordinary data, fragile for tiny samples of skewed data, and improvable by transformation or by the nonparametric alternatives (the hub's Mann-Whitney and Wilcoxon pages) when the distribution fights back. The calculator computes the classical test; the judgement about its assumptions is the analyst's signature on the work.

Why do different fields use different significance levels?

Because α prices a false alarm, and false alarms cost differently across fields: 0.05 is social science's convention, particle physics demands five sigma (p ≈ 3 × 10⁻⁷) before announcing a discovery, and quality engineering often works at 0.01 or tighter where a false rejection scraps good production. The calculator offers 0.10, 0.05 and 0.01 as the standard shelf; the level must be chosen before the data is seen (choosing it afterward to manufacture a verdict is the peeking sin in ceremonial dress), and a result reported with its exact p-value — as this page reports it — lets every reader apply their own field's threshold honestly.