Skip to main content

Type 1 and Type 2 Errors, With a Worked Example

Type 1 and type 2 errors with a worked example: at α = .05 and a medium effect, 20 people per group give power of .34, and 64 per group give power of .80.

Type 1 and Type 2 Errors, With a Worked Example

Type 1 and type 2 errors are the two ways a hypothesis test goes wrong. A Type I error (false positive) rejects a null hypothesis that is true; its probability is α, the significance level you choose. A Type II error (false negative) fails to reject a null hypothesis that is false; its probability is β, and power is 1 − β. This guide works both out with numbers.

Key Takeaways

  • A Type I error is a false positive: the test reports an effect and there is none. A Type II error is a false negative: there is an effect and the test misses it.
  • You set α in advance, usually at .05. β follows from α, the sample size, the variability of the data and the size of the true effect.
  • In our worked example (two groups, α = .05, a true effect of half a standard deviation), 20 people per group give power of .34 and 64 per group give .80.
  • Lowering α without adding data raises β. At 64 per group, moving α from .05 to .01 cuts power from .80 to .59.
  • A non-significant result from a low-powered study says little either way. Report α, the planned power with its effect size, and the confidence interval.

What are Type I and Type II errors?

A Type I error is rejecting a null hypothesis that is in fact true, so you report an effect that does not exist. A Type II error is failing to reject a null hypothesis that is in fact false, so a real effect goes unreported. Statistics texts write the names with Roman numerals; “type 1 and type 2” and “Type I and Type II” mean the same thing.

A test ends in one of two decisions, and the null hypothesis is either true or false, which gives four outcomes. OpenStax’s Introductory Statistics 2e lists the same four.

The null hypothesis is true (no effect)The null hypothesis is false (a real effect)
You reject the nullType I error, a false positive. Probability αCorrect decision. Probability 1 − β, the power of the test
You fail to reject the nullCorrect decision. Probability 1 − αType II error, a false negative. Probability β

Each probability is conditional on its column: α is the chance of rejecting when the null is true, β the chance of not rejecting when it is false. Neither is the chance that your conclusion is wrong, because you never know which column you are in. Our guide to writing a hypothesis covers how the null and the alternative are worded, and the glossary has a short definition of the null hypothesis.

The courtroom comparison that most explanations use is as old as the idea. In their 1933 paper on the theory of testing, Neyman and Pearson wrote of “two sources of error” and asked: “Is it more serious to convict an innocent man or to acquit a guilty? That will depend upon the consequences of the error”. If innocence is the null hypothesis, convicting an innocent defendant is the Type I error and acquitting a guilty one is the Type II error.

What are alpha, beta and statistical power?

Alpha (α) is the probability of a Type I error: the long-run share of true null hypotheses a test rejects. Beta (β) is the probability of a Type II error: the share of false null hypotheses it fails to reject, for one specific true effect. Statistical power is 1 − β, the probability that the test detects that effect. You choose α; β follows from the design.

The NIST/SEMATECH e-Handbook of Statistical Methods puts α this way: “A value of α = 0.05 implies that the null hypothesis is rejected 5 % of the time when it is in fact true.” Jacob Cohen called α a policy, “the maximum risk attending such a rejection”. That is all a statistically significant result promises: the procedure rarely rejects true nulls.

Three common readings of these numbers are wrong.

  • α is not the probability that a significant result is a false positive. That share depends on power and on how many of the tested hypotheses are true, as the count further down shows.
  • A p-value is not the probability that the null hypothesis is true. The American Statistical Association’s 2016 statement: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”
  • Failing to reject is not accepting the null. In NIST’s words, a non-significant result may be a sign that “we may not yet have enough data”.

β has no single value. NIST again: it “is not chosen by the user but is determined, as one might expect, by the magnitude of the real discrepancy”. A power figure without an effect size means nothing.

A worked example: one study at three sample sizes

Take a two-group study with α = .05, two-tailed, a standard deviation of 10 points and a true difference of 5 points (d = 0.50). With 20 people per group, power is .34, so β is .66. With 64 per group, power is .80. With 100 per group, power is .94. Nothing changed except the sample size.

Suppose a study-skills course really does raise exam scores by 5 points, and scores have a standard deviation of 10. The arithmetic, using the normal approximation:

  1. Standard error of the difference. SE = SD × √(2 ÷ n). With n = 64 per group, SE = 10 × √(2 ÷ 64) = 1.77 points.
  2. Smallest difference that reaches significance. 1.96 × SE = 3.46 points. A smaller observed difference gives p > .05.
  3. How often the study clears that bar. Observed differences scatter around the true 5 points with the same SE. The bar sits (5 − 3.46) ÷ 1.77 = 0.87 standard errors below 5, and a normal curve has 81% of its area above that point.

So power is about .81 and β about .19. In one line: power ≈ Φ(d × √(n ÷ 2) − 1.96), where Φ is the standard normal cumulative probability. The exact calculation uses the t distribution and gives slightly less.

n per groupSE of the differenceSmallest significant difference (normal approximation, 1.96 × SE)Power (normal approximation)Power (exact t test)β (exact)
203.166.20.35.34.66
302.585.06.49.48.52
641.773.46.81.80.20
1001.412.77.94.94.06

We computed the exact column with the noncentral t distribution in Python and checked it by simulation: in 100,000 simulated studies with 64 per group, 80.4% reached p < .05 when the true difference was 5 points, and 4.96% did when it was zero. Those two percentages are power and α, observed.

The 64 matches a published table. The first example in Cohen’s “A power primer” reads: “To detect a medium difference between two independent sample means (d = .50 in Table 1) at α = .05 requires N = 64 in each group.”

Look at the row for 20 per group. The smallest difference that reaches significance, 6.20 points (6.40 with the exact t test), is larger than the true effect of 5. That study reaches significance only when sampling error pushes the estimate too high, so in a study this underpowered, a significant result overstates the effect.

What changes the risk of each error?

Four things: the α you choose, the sample size, the size of the true effect, and the variability of the measurements. α sets the Type I risk directly. The other three, together with α, set β. Cohen sums it up from the planning side: the sample size needed rises “with an increase in the power desired, a decrease in the ES, and a decrease in α.”

With the data held fixed, the two errors trade against each other. NIST: “the risk β increases as the risk α decreases.” In the example:

Change (64 per group unless stated)Powerβ
Baseline: α = .05, SD = 10, true difference 5 points.80.20
Stricter α = .01.59.41
Noisier measure: SD = 15 (d = 0.33).46.54
Stricter α = .01, with 96 per group.80.20

A stricter α buys fewer false positives at the price of more false negatives, unless you pay with data: holding power at .80 with α = .01 takes 96 per group. Noise costs power the same way a small sample size does. With an SD of 15 the same 5 points is a third of a standard deviation, and power of .80 needs 143 per group. For a small effect (d = 0.20), Cohen’s table asks for 393 per group.

Which is worse, a Type I or a Type II error?

Neither is worse in general. It depends on what each mistake costs in the situation at hand. Research convention treats a false positive as the more serious one, which is why α is usually .05 while β is allowed to be .20. Where a miss is dangerous, as with safety equipment, the Type II error costs more.

OpenStax gives two examples that point in opposite directions. If the null hypothesis is that a climber’s equipment is safe, the Type II error (believing unsafe gear is safe) has “the greater consequence”. If the null is that an accident victim is alive, the Type I error (concluding they are dead) is the worse one.

The .80 convention for power is a judgment of the same kind. Cohen proposed it “for general use”: a much smaller value “would incur too great a risk of a Type II error”, and a much larger one would demand a sample “likely to exceed the investigator’s resources.” With α = .05, he noted, power of .80 gives “a β:α ratio of 4:1 (.20 to .05) of the two kinds of risks.” If a missed effect would cost you more than a false alarm, choose a different ratio and say why.

How common are Type I and Type II errors in published research?

Nobody can count them directly, because the truth of each hypothesis is unknown. What can be measured is power, and it has been low for decades: about .48 for medium effects in a 1960 psychology journal volume, and a median somewhere between 8% and 31% in neuroscience studies. Low power means many Type II errors, and it also makes significant findings less trustworthy.

Cohen reviewed the 1960 volume of the Journal of Abnormal and Social Psychology and found “the mean power to detect medium effect sizes to be .48”, so that “the chance of obtaining a significant result was about that of tossing a head with a fair coin.” Button and colleagues (2013) estimated that “the median statistical power of studies in the neurosciences is between ∼8% and ∼31%”, and added that low power “reduces the likelihood that a statistically significant result reflects a true effect.”

A count shows why. Suppose 1,000 hypotheses are tested at α = .05 and 100 of them are true effects (the 10% is our assumption, for illustration). The 900 true nulls produce 45 false positives. At power of .80, the 100 real effects produce 80 true positives, so 80 of 125 significant results (64%) are real. At power of .34 they produce 34, and only 34 of 79 (43%) are real. The Type I rate was .05 throughout.

The Open Science Collaboration (2015) repeated 100 studies from three psychology journals. Of the 100 originals, 97 had statistically significant results; of their replications, 36% did. Given the average replication power of .92, the authors expected about 89 significant replications if every original effect were true and accurately estimated. They got 35. They did not call the rest Type I errors: “It is also too easy to conclude that a failure to replicate a result means that the original evidence was a false positive.”

What happens when you run many tests?

Each test at α = .05 carries its own 5% chance of a false positive when its null is true, and the chances accumulate. Across 10 independent tests of true null hypotheses, the probability of at least one false positive is 1 − .9510, about 40%. This is the familywise (or experimentwise) error rate, and corrections such as Bonferroni’s exist to hold it down.

Independent tests, all nulls trueChance of at least one false positive at α = .05Bonferroni α per test (.05 ÷ tests)
15.0%.05
522.6%.01
1040.1%.005
1451.2%.0036

By 14 tests a false positive is more likely than not. The Bonferroni method divides α by the number of tests: NIST’s handbook describes making each of g statements at confidence 1 − α/g so that the overall confidence “is at least 1 − α”. Cohen recommended, for studies testing several hypotheses, “α = .01 per hypothesis in order that the experimentwise risk (i.e., the risk of any false rejections) not become too large.”

The correction is paid for in power: in the worked example, 10 tests at .005 each drop power from .80 to .49 at 64 per group. Planning fewer tests and naming the primary one in advance costs nothing. Our guide to correlation vs causation shows how often chance alone produces a large correlation in a small sample.

How do you reduce Type I and Type II errors?

Reduce Type I errors by setting α before you see the data, limiting the number of tests and correcting for the ones you run. Reduce Type II errors by raising power: a larger sample, a more precise measure, or a design that makes the effect larger. Only more information, from a larger sample, a more precise measure or a stronger design, lets you lower both at once: spend some of the extra power on a stricter α.

Every fix has a price. A smaller α or a multiple-test correction lowers power unless the sample grows, and a larger sample costs time, money and participants. A power analysis before data collection tells you what the trade will cost.

A memory aid for which is which: in the story of the boy who cried wolf, the villagers make both errors in numerical order. First they believe in a wolf that is not there (Type I, a false positive). Later they ignore a wolf that is there (Type II, a false negative).

How do you report alpha, power and a non-significant result?

State α and whether tests were two-tailed. Report the power analysis with all of its ingredients: test, α, target power, and the effect size it assumes. For every result, significant or not, give the exact p-value, the effect size and a confidence interval. Never describe a non-significant result as proof of no effect.

The two passages below were written for this guide as illustrations. They describe two different invented studies, one planned at 64 per group and one that ran with 20 per group; both figures come from the worked example above.

Methods (illustration). An a priori power analysis for a two-tailed independent-samples t test (α = .05, power = .80) indicated that 64 participants per group were needed to detect a difference of d = 0.50, the smallest effect we considered educationally meaningful.

Results (illustration). Students who took the course scored 3.8 points higher than the comparison group, a difference that was not statistically significant, t(38) = 1.20, p = .24, d = 0.38, 95% CI for the difference [−2.6, 10.2]. With 20 participants per group, the study had power of .34 to detect d = 0.50, so this result is inconclusive: the interval includes zero and also differences twice the size we planned for.

The second passage gives the estimate, shows how wide the uncertainty is, and says what the study could have detected. Cohen applies the same logic to a well-powered study that comes up non-significant: either the effect is smaller than planned for, “or the investigator has been the victim of the .20 (β) risk of making a Type II error.”

The power analysis goes in the participants part of the methods; our guide on how to write a methodology section shows where. The formatting of t, p and d is covered in how to write a results section. If the study ended up underpowered, say what size of effect it could have detected when you write the limitations of your study.

Where Papero fits

Papero is an AI writing workspace for students and researchers. It is a writing and citation tool, not a statistics package: it does not run tests or compute power. It helps with the sentences around the numbers. Paste a DOI to cite the paper behind a convention such as power of .80, and Review’s Claims check finds sentences that need a citation. The research paper writing page walks through the process from outline to export.

Frequently Asked Questions

What is the difference between a Type I and a Type II error?

A Type I error is a false positive: the test rejects a null hypothesis that is true, so you report an effect that is not there. A Type II error is a false negative: the test fails to reject a null hypothesis that is false, so a real effect is missed. The first has probability α, the second β.

What are alpha and beta errors?

They are other names for Type I and Type II errors. An alpha error is a Type I error, and α is its probability, set by the researcher as the significance level (often .05). A beta error is a Type II error, and β is its probability. Power is 1 − β.

How do you memorize type 1 and type 2 errors?

Use the boy who cried wolf. The villagers first believe in a wolf that is not there, which is a false positive and Type I. Later they ignore a wolf that is there, which is a false negative and Type II. The errors happen in numerical order.

Does a larger sample size reduce Type I errors?

Not by itself. The Type I error rate is α, which you set, and it stays at .05 whether you test 20 people or 2,000. A larger sample reduces Type II errors by raising power. It also lets you afford a stricter α: for a two-group study looking for a medium effect (d = 0.50), 96 per group at α = .01 gives the same power, .80, as 64 per group at α = .05.

Sources

Papero Team

Papero

The Papero team builds an AI writing assistant for students, scholars and researchers, and writes about academic writing from what the product does.

Follow along on Instagram at @paperoai for campaign breakdowns.

Your next paper, with real citations.

Describe your paper and get an outline in a minute. The Free plan needs no card.

Write your paper with the sources attached.