Sampling bias is a systematic difference between a sample and the population it is meant to describe, caused by the way the sample was drawn: some people had a higher or lower chance of ending up in it than the design allows for, and those people differ on what you are measuring. Unlike chance error, it does not shrink as the sample grows. This guide works through the 1936 Literary Digest poll, which missed by 19.6 points with nearly 2.4 million ballots.
Key Takeaways
- The bias comes from the selection process: a list that leaves people out, people choosing whether to take part, or only the survivors being visible.
- Sampling error is the chance gap between one random sample and the population. It gets smaller as the sample gets bigger; bias does not.
- The Literary Digest’s 1936 poll had a 95% margin of error of about 0.06 points and a real error of 19.6 points. A 2012 reanalysis of Gallup data traces most of that error to who sent ballots back, with the mailing list adding the rest.
- In epidemiology, “selection bias” usually means a distorted comparison inside a study, while a biased sample threatens claims about a population. Many sources use the terms loosely, so define yours.
- Probability sampling, a better list, chasing nonresponders and weighting all reduce the bias. Weighting only fixes imbalances on variables you measured.
What is sampling bias?
Sampling bias happens when the method of choosing participants gives some members of the target population a higher or lower chance of selection than the design assumes, and those chances are linked to the outcome. The result is a sample that is wrong in a predictable direction. Because the problem sits in the selection process, analysing the data more carefully afterwards cannot fully remove it.
The OpenStax Introductory Statistics textbook puts it this way: “a sampling bias is created when a sample is collected from a population and some members of the population are not as likely to be chosen as others.” Its parenthetical reminder, that each member “should have an equally likely chance of being chosen,” needs one correction. Good designs often give unequal chances on purpose, as when a stratified sample oversamples a small group. What matters is that every chance is known and above zero, so the analysis can weight people back to their share of the population.
Two conditions have to hold for unequal chances to bias a result: some group is over- or under-represented, and that group differs on the variable you are estimating. If left-handed people are scarce in your sample but handedness has nothing to do with your outcome, the estimate is unharmed.
Sampling bias vs sampling error: what is the difference?
Sampling error is the random gap between a single sample and the population, the reason two random samples of the same size give slightly different answers. It averages out over repeated samples and shrinks as the sample grows. Bias pushes every sample from the same method in the same direction, so a larger sample only measures the wrong value more precisely. Confidence intervals describe the first, never the second.
| Sampling error | Sampling bias | |
|---|---|---|
| What it is | Chance difference between a sample and the population | Systematic difference built into how the sample is chosen |
| Direction | Varies from sample to sample, centred on the true value | The same direction every time the method is used |
| Effect of a larger sample | Shrinks, roughly with the square root of the sample size | Unchanged |
| Measured by | Standard error, margin of error, confidence interval | Not visible in the data; needs outside benchmarks or a design argument |
| Fixed by | More participants | A different selection method, follow-up or weighting |
OpenStax states the sampling-error side plainly: “As a rule, the larger the sample, the smaller the sampling error.” The STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guideline states the bias side: bias is “a systematic deviation of a study’s result from a true value,” typically introduced during the design or implementation of a study, and then it “cannot be remedied later,” unlike random error, which runs “in either direction.” Sampling error is the chance gap between a sample statistic and the population parameter it estimates, and our parameter and statistic guide shows that gap shrinking as samples grow from 25 to 400.
OpenStax adds a third category, nonsampling error, from factors unrelated to sampling, such as a defective counting device. Measurement problems are the most common kind, and the reliability and validity guide covers them.
Worked example: why 2.4 million ballots failed in 1936
The Literary Digest poll is the standard proof that size cannot cure bias. The magazine mailed about 10 million ballots, got nearly 2.4 million back and predicted that Alf Landon would beat Franklin Roosevelt, who then won about 61% of the vote. The poll’s chance error was tiny and its systematic error about 20 points; how to split that error between the mailing list and the people who replied is still argued over.
What happened. According to Lusinchi (2012), the Digest forecast Landon at 54% of the popular vote and Roosevelt at 41%. The mailing list came from “the rosters of clubs and associations, voter registration rolls, occupational records, classified mail-order and city directories,” with the bulk drawn from telephone directories and car registrations. The response rate was roughly 24%.
The size of the chance error. Measured as a share of the two-party vote, the Digest gave Roosevelt 42.9%. With 2.4 million ballots, the usual 95% margin of error is 1.96 × √(0.429 × 0.571 ÷ 2,400,000) ≈ 0.06 percentage points. A simple random sample of only 1,000 voters, centred on Roosevelt’s true 62.5%, would have had a margin of about ±3 points (our arithmetic, assuming perfect random sampling and full response).
The size of the bias. Lusinchi reanalysed a May 1937 Gallup survey that asked people whether they had received and returned a Digest ballot, and weighted the survey to match the Digest’s known 24% response rate and its published tally:
| Figure | Roosevelt’s share of the two-party vote |
|---|---|
| Actual 1936 result | 62.5% |
| Everyone on the Digest’s 10-million list, had all of them replied (estimated) | 56.6% (95% confidence interval 53.1% to 60.1%) |
| Ballots actually returned to the Digest | 42.9% |
| Error from the mailing list (coverage) | 62.5 − 56.6 = 5.9 points |
| Error from who replied (nonresponse) | 56.6 − 42.9 = 13.7 points |
| Total error | 19.6 points |
| 95% margin of error from chance alone, 2.4 million ballots | about 0.06 points |
The total error was more than 300 times the margin of error. No confidence interval computed from the returned ballots could have warned the editors.
Where the explanations disagree. The textbook story blames telephone and car owners, supposedly richer and Republican. The Gallup data contradict it: Lusinchi finds that 53.9% of telephone and car owners favoured Roosevelt, and Landon’s supporters returned their ballots far more often, “nearly one-third versus only one-fifth.” He concludes the “principal culprit” was nonresponse. Squire (1988), working from the same survey without weighting, split the error the other way, about 11 points from the sample and 7 from nonresponse, as Lusinchi reports. Lohr and Brick (2017) summarise the earlier postmortems, Squire’s included, as finding that “the low response rate combined with the flawed sample produced the inaccurate forecast.” The safe reading: coverage and nonresponse both pushed the same way, and a bigger mailing would have repeated both.
Types of sampling bias
The common types differ in where the selection goes wrong: the list you sample from (undercoverage), the people who decide whether to join (self-selection, volunteer and nonresponse bias), the cases still around to be counted (survivorship), or the place you recruit (convenience samples). Health research adds healthy-user bias, where people who take up a treatment differ from those who do not.
| Type | How it arises | Example | First fix to consider |
|---|---|---|---|
| Undercoverage (frame bias) | The sampling frame misses part of the population | A door-to-door Parkinson’s survey of a US county of about 24,000 people found 31 cases, 13 never seen for medical care; a records-based count would have missed over 40% (Catalogue of Bias) | A complete frame, or several frames combined |
| Nonresponse bias | People who reply differ from those who do not | Literary Digest 1936: Landon backers returned ballots more often (Lusinchi) | Follow-up of nonresponders; compare early and late replies |
| Self-selection (voluntary response) | People choose to take part, often because they care about the topic | Digest ballots became a way for Landon supporters to protest the New Deal (Lusinchi) | Invite a random sample instead of an open call |
| Volunteer bias | Volunteers differ from non-volunteers in attitudes, health or education | Volunteers in sexuality research reported more sexual experience and less sexual guilt (Catalogue of Bias) | Raise participation; anonymity; compare with refusers |
| Survivorship bias | Only cases that survived a process are observed | Wald’s wartime memoranda estimated aircraft survival using damage on planes that returned (Casselman) | Find records of the failures too |
| Convenience-sample bias | Participants are whoever is easiest to reach | Students outside one campus library (our convenience sampling guide) | Random sampling within the accessible group, then honest scope |
| Healthy-user bias | People who start or keep taking a treatment are healthier in other ways | Observational studies suggested hormone therapy protected the heart; trials did not confirm it (Catalogue of Bias) | Randomisation; active comparators |
The survivorship row describes Wald’s actual, highly technical work. Casselman, reviewing the memoranda, calls the famous version, in which Wald tells the military to armour the engines, “plausible reconstruction” with “very little evidence for the best bits.” Healthy-user bias distorts a comparison between users and non-users, so it sits closer to selection bias in the epidemiological sense below. For the convenience row, our guide to convenience sampling simulates how far a campus sample drifts. Referral chains bring a bias of their own, since each participant recruits people they know: our guide to snowball sampling simulates how much the first participants, the seeds, decide.
The Catalogue also lists ascertainment bias, where data are “collected in such a way that some members of the target population are more or less likely to be included in the final results than others.” It overlaps with undercoverage, but the typical case is uneven detection: an outcome found more often in people who are screened or examined more often.
Sampling bias vs selection bias
In survey statistics, sampling bias means the sample misrepresents the population, which threatens estimates such as a prevalence or a vote share. In epidemiology, selection bias usually means the way people entered or stayed in a study distorted a comparison between groups, which threatens a causal conclusion even if nobody wants to generalise. Broader definitions file both under selection bias, so state which sense you mean.
STROBE is explicit about the epidemiological sense: “In general, selection bias affects the internal validity of a study. This is different from problems that may arise with the selection of participants for a study in general, which affects the external rather than the internal validity of a study.” The same passage (STROBE’s Box 3) still counts response bias, which distorts prevalence estimates, as a type of selection bias, so even STROBE’s line is not clean. Hernán and Robins define selection bias as “an association created as a result of the process by which individuals are selected into the analysis,” and add that it “can arise in both randomized experiments and observational studies.” The Catalogue of Bias uses a broader definition that covers both: a systematic difference from “the population of interest” leading to error “in an association or outcome.”
A practical way to keep them apart:
- You want a number for a population (how many students skip breakfast, what share of voters back a candidate). Unequal selection chances bias that number. Name the type: undercoverage, nonresponse, self-selection.
- You want a comparison (does a drug lower risk, do night shifts raise it). The worry is that the way people got into, or out of, your analysis created or hid an association. That is selection bias in the STROBE sense, and losses to follow-up are a common source.
Sampling bias examples from real research
Two documented cases beyond the Literary Digest, one new and one expanded from the table above, show how the problem looks in practice: a 2016 polling error that came partly from not correcting a known skew, and a health finding reversed once randomised trials removed self-selection. In both, the warning sign was visible in the methods: a group that the selection process favoured.
State polls in 2016. A committee of the American Association for Public Opinion Research (AAPOR) found that many polls, especially at the state level, “did not adjust their weights to correct for the over-representation of college graduates in their surveys, and the result was over-estimation of support for Clinton.” College graduates answer surveys more often, and in 2016 education tracked the vote closely. Only 18% of final Michigan polls weighted for education, compared with 52% of final national polls. The University of New Hampshire poll showed Clinton 11 points ahead in a state she won by 0.4; its director reported that with an education weight, “our predictions match the final number.”
Hormone therapy and heart disease. This is the healthy-user case from the table, on the selection-bias side of the line drawn above. Women who took hormone therapy in observational studies were, in the words of the Catalogue’s selection bias entry, “more health conscious, more physically active, and had higher socioeconomic status” than those who did not, and the studies suggested a protective effect. Randomised trials did not find it. The confounding variables guide walks through how that reversal played out.
How to spot a biased sample in a paper
Compare the population the paper makes claims about with the list or place the sample actually came from, then follow the numbers from people invited to people analysed. Any group that could not be selected, or that dropped out at a higher rate, is a candidate for bias. Then check whether the authors measured the variables that would let them correct it.
Six questions cover most cases:
- Who is the target population? Find the sentence where the authors generalise.
- What was the sampling frame? A phone list, a clinic register, a website, a class: who is missing from it?
- How were people selected from the frame? At random from a list, or whoever turned up?
- What was the response rate, and is the flow reported? STROBE item 13 asks authors to “Report the numbers of individuals at each stage of the study,” from eligible to analysed. A paper that gives only the final sample size hides where people were lost.
- Do the authors compare respondents with the population or with nonresponders? A table of age, sex and education against census figures is a good sign.
- What did they weight on, and was it related to the outcome? Weighting on age and sex does nothing for an imbalance in education if education drives the result, as the 2016 polls showed.
If the frame or the flow is unclear, read the paper’s population figures as describing only the people the study reached.
How to avoid sampling bias
Start at the design stage: define the target population, find or build a frame that covers it, and select from that frame with known probabilities. Then protect the sample you drew by following up nonresponders. Weighting to population figures can reduce what remains, but only for imbalances you measured, and it rarely removes all of it.
Use probability sampling. Random selection from a complete list gives every member a known chance of inclusion; simple random, stratified, cluster and systematic designs all qualify. A random sample can still suffer nonresponse, so this is necessary but not enough.
Fix the frame. If one list misses a group, combine lists or use one that reaches everyone, such as addresses instead of a phone directory. The Parkinson’s survey in the table went door to door instead of using medical records.
Chase nonresponders. The Catalogue of Bias’s volunteer bias entry notes that “any technique that increases volunteer numbers is likely to reduce bias.” Its non-response entry adds the continuum-of-resistance idea: late responders tend to resemble nonresponders, so comparing early and late replies gives a rough check.
Weight, and know the limits. Weighting gives under-represented groups more influence so the sample matches known population totals. In SurveyMonkey’s 2016 Wisconsin data it cut the error on the margin from 10 points to 3, mainly by correcting education (AAPOR). It can also fall short. Lohr and Brick reweighted the Literary Digest data by how respondents said they voted in 1932. Roosevelt went from 16 states and 161 electoral votes to 26 states and 276, the right winner, but the weighted poll still did “a poor job of predicting the margin of the win.” Their summary: “This partial reduction of bias by adjustment is typical of what can be expected.” Weighting only helps if the people you reached in each group resemble the people you missed in that group. It also leaves the answers themselves untouched: when respondents overreport or follow a leading question, that is response bias, and no weight corrects it.
Plan the size after the method. A sample size calculation controls sampling error. It says nothing about bias, so choose how to select people before deciding how many.
How to write about bias in your limitations section
Name the group that was under- or over-represented, say how it got that way, and state which direction it likely pushed your estimate and roughly how far. Give the response rate and any comparison with population figures, and end with the population your findings describe. A stated direction is useful to readers; a generic “results may not generalise” is not.
STROBE item 19 sets the standard: “Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias.” Its own example, from Galuska and colleagues’ 1999 analysis of a 1996 telephone survey that asked obese adults whether a health professional had advised them to lose weight, does exactly that:
“Since the prevalence of counseling increases with increasing levels of obesity, our estimates may overestimate the true prevalence. Telephone surveys also may overestimate the true prevalence of counseling. Although persons without telephones have similar levels of overweight as persons with telephones, persons without telephones tend to be less educated, a factor associated with lower levels of counseling in our study.”
It names the frame (telephone owners), the group it misses (less educated people), why that matters (education predicts the outcome) and the direction (overestimate). A model for a student survey, with placeholders in brackets:
“We invited all [number] students enrolled in [course] by email; [number] replied ([percent]%). Respondents included more women than the course ([percent]% against [percent]%) and more students in their final year. Because both groups reported higher [outcome] in our data, the overall figure of [percent]% is likely an overestimate for the course. Weighting by sex and year gave [percent]%, but it cannot correct differences between respondents and nonrespondents on variables we did not measure, such as interest in the topic.”
Our guide to the limitations of a study has a full template and more published examples.
Where Papero fits
Papero is an AI writing workspace for students and researchers. It does not draw your sample, check your frame or weight your data; those decisions stay with you. It helps with the reading and writing around them.
Find papers searches more than 250 million papers through OpenAlex and saves any result to your source library; “Get free PDF” downloads a saved paper’s open-access PDF when one exists, and you can upload the rest. Chat answers questions about the PDFs in your library and the document you are writing, citing the passages it used, which helps when you compare how several studies reported response rates or worded their limitations. The Review Claims check finds sentences that need a citation and offers papers to cite. What each plan includes is on the pricing page. The glossary entry on random sampling covers the design side, and the literature review page shows how one source library carries through a review.
Frequently Asked Questions
What is sampling error vs. sampling bias?
Sampling error is the chance difference between one random sample and the population. It varies in direction from sample to sample and shrinks as the sample grows, and a margin of error measures it. Sampling bias is a systematic difference caused by how the sample was chosen. It points the same way every time the method is used, does not shrink with a larger sample and does not show up in the margin of error.
Can you give me an example of bias in research?
The 1936 Literary Digest poll mailed about 10 million ballots, received nearly 2.4 million and predicted that Alf Landon would beat Franklin Roosevelt, who won about 61% of the vote. Its margin of error from chance alone was about 0.06 points, yet it missed by roughly 20. A reanalysis found most of the error came from Landon supporters returning ballots more often than Roosevelt supporters.
What is ascertainment bias?
Ascertainment bias arises when data are collected so that some members of the target population are more or less likely to be included in the results than others. A common form is uneven detection: an outcome is found more often in people who are screened or examined more often, such as patients who see a doctor regularly, so the recorded cases differ from the real ones.
Does a larger sample reduce sampling bias?
No. A larger sample reduces random sampling error, but if the selection method favours certain people, every extra participant repeats the same imbalance. The Literary Digest collected nearly 2.4 million ballots and still missed Roosevelt’s share by about 20 points. Reducing bias needs a better frame, random selection, follow-up of nonresponders or weighting, not more of the same sample.
Is sampling bias the same as selection bias?
Not exactly, and sources differ. In survey statistics, the first term describes a sample that misrepresents its population, which affects estimates such as prevalence. In epidemiology, selection bias usually means the way people entered or left a study distorted a comparison between groups, which affects internal validity. Some sources file both under selection bias, so define the term you use.
Sources
- Casselman, B. (2016, June). The legend of Abraham Wald. Feature Column, American Mathematical Society. Internet Archive capture of 30 May 2018: https://web.archive.org/web/20180530221618/http://www.ams.org/publicoutreach/feature-column/fc-2016-06
- Catalogue of Bias Collaboration, Nunan, D., Bankhead, C., & Aronson, J. K. (2017). Selection bias. In Catalogue of Bias. https://catalogofbias.org/biases/selection-bias/
- Catalogue of Bias Collaboration, Turk, A., Heneghan, C., & Nunan, D. (2019). Non-response bias. In Catalogue of Bias. https://catalogofbias.org/biases/non-response-bias/
- Catalogue of Bias Collaboration, Brassey, J., Mahtani, K. R., Spencer, E. A., & Heneghan, C. (2017). Volunteer bias. In Catalogue of Bias. https://catalogofbias.org/biases/volunteer-bias/
- Catalogue of Bias Collaboration, Wu, A. D., Knight, E., & Yang, X. (2026). Healthy user bias. In Catalogue of Bias. https://catalogofbias.org/biases/healthy-user-bias/
- Catalogue of Bias Collaboration, Spencer, E. A., & Brassey, J. (2017). Ascertainment bias. In Catalogue of Bias. https://catalogofbias.org/biases/ascertainment-bias/
- Hernán, M. A., & Robins, J. M. (2020). Causal inference: What if (Chapter 8, Selection bias). Chapman & Hall/CRC. https://miguelhernan.org/whatifbook
- Illowsky, B., & Dean, S. (2023). Data, sampling, and variation in data and sampling. In Introductory statistics 2e (Section 1.2). OpenStax. https://openstax.org/books/introductory-statistics-2e/pages/1-2-data-sampling-and-variation-in-data-and-sampling
- Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., Saad, L., Witt, E., & Wlezien, C. (2017). An evaluation of 2016 election polls in the U.S. American Association for Public Opinion Research. Internet Archive capture of 6 June 2017: https://web.archive.org/web/20170606125213/http://www.aapor.org/Education-Resources/Reports/An-Evaluation-of-2016-Election-Polls-in-the-U-S.aspx
- Lohr, S. L., & Brick, J. M. (2017). Roosevelt predicted to win: Revisiting the 1936 Literary Digest poll. Statistics, Politics and Policy, 8(1), 65–84. https://doi.org/10.1515/spp-2016-0006
- Lusinchi, D. (2012). “President” Landon and the 1936 Literary Digest poll: Were automobile and telephone owners to blame? Social Science History, 36(1), 23–54. https://doi.org/10.1017/S014555320001035X
- Squire, P. (1988). Why the 1936 Literary Digest poll failed. Public Opinion Quarterly, 52(1), 125–133. https://doi.org/10.1086/269085 (findings as reported by Lusinchi, 2012, and Lohr & Brick, 2017)
- Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and elaboration. PLOS Medicine, 4(10), Article e297. https://doi.org/10.1371/journal.pmed.0040297