Response bias is a systematic error in the answers people give to a survey, interview or questionnaire. The people took part, but what they reported drifts from the truth in a predictable direction, because of how the question was worded, where it sat, who asked it, or the respondent’s wish to look good. This guide gives each of seven types a published example, with a measured size wherever one exists, and the fixes that reduce it.
Key Takeaways
- The bias sits in the answers of people who did respond. Nonresponse bias is a different problem: the people who never answered differ from those who did.
- Sensitive questions produce the largest errors. In a 2011 Mississippi validation study, 30% of voters told interviewers they had voted “no” on an anti-abortion amendment; the official count in their counties was 65.3%. Almost a fifth did not answer, and among those who did, 38% said “no”.
- Small changes in wording and order moved answers by 7 to 21 points in the experiments described below, with no change in what the question asks.
- Self-administration, forced-choice questions and indirect questioning each reduce specific types, and randomized order spreads order effects evenly instead of letting them pile up in one direction. Weighting the sample fixes none of these answer errors.
What is response bias?
Response bias is any systematic difference between what respondents report and what is true, caused by how the answers are collected or given: the wording, the answer options, the order of questions, the presence of an interviewer, what respondents think the study expects, their wish to look good, or faulty memory. Unlike random error, it pushes many answers the same way, so adding respondents does not cancel it.
As Pew Research Center’s questionnaire guide puts it, “Accurate random sampling will be wasted if the information gathered is built on a shaky foundation of ambiguous or biased questions.” Epidemiologists file the same problem under information bias, which the Catalogue of Bias defines as “any systematic difference from the truth that arises in the collection, recall, recording and handling of information in a study.”
Because the error lives in the measure, this kind of bias threatens validity more than reliability: a question that makes people overreport voting does so just as consistently next month, so a test-retest check can look excellent while the answers stay wrong. The guide to reliability and validity explains why consistency and accuracy need separate checks.
The 7 types at a glance, with published examples
The seven types differ in what drives the error: the respondent’s image or reading of the study (social desirability, demand characteristics), a habit of answering (acquiescence, extreme or midpoint responding), the instrument (wording and order), or memory (recall). Each row gives a published example, with its size where one was measured, and the first fix to consider.
| Type | What goes wrong | Published evidence | First fix |
|---|---|---|---|
| Social desirability bias | People overreport what looks good and underreport what looks bad | Mississippi, 2011: 38% of those who answered said they voted “no”; the official count was 65.3% (Rosenfeld et al.) | Self-administration, indirect questioning, record checks |
| Acquiescence (yea-saying) | Agreeing with a statement whatever it says | Pew, 1999: 55% agreed peace is best ensured by military strength; 33% picked that view in a forced choice | Forced-choice or item-specific questions |
| Extreme and midpoint responding | Using the ends or the middle of a scale out of habit or because of the mode | Pew, 2014: 45% of phone and 36% of web respondents put Democratic leaders in the middle of a 0 to 100 scale | One mode for everyone; labelled scale points |
| Question wording | Equivalent words draw different answers | Rugg, 1941: 46% would “not forbid” speeches against democracy, 25% would “allow” them | Split-ballot tests of wording; neutral terms |
| Question order | An earlier question sets the context for a later one | Pew, 2008: 88% dissatisfied with the country right after a Bush approval question, 78% without it | Randomized order; general questions first |
| Demand characteristics | Participants answer as they think the study wants | A systematic review found only 7 field studies, with mixed results (McCambridge et al.) | Hide the hypothesis; measure outcomes separately |
| Recall bias | Memory errors, often different between groups | Daily self-reports of computer use ran high below 3.6 hours a day and low above it (Chang et al., via the Catalogue of Bias) | Prospective data, diaries, records |
Social desirability bias: how large is it?
Social desirability bias is the tendency to give the answer that presents you well: more voting, exercise and charity, less drinking, drug use and prejudice than is true. Its size depends on how embarrassing the true answer is and on who is listening. In the two voting studies below, where answers could be checked against records, self-reports missed the true figure by 17 to about 25 points.
Tourangeau and Yan’s review in Psychological Bulletin concludes that “misreporting about sensitive topics is quite common” and “largely situational.” Three studies put numbers on it.
Validated votes. Rosenfeld, Imai and Shapiro drew a random sample of 2,655 people who had voted in Mississippi’s 2011 election and asked by phone how they voted on the “personhood” amendment, an anti-abortion measure. Asked directly, 30% said “no”; the official count in the 19 sampled counties was 65.3% “no”. Almost 20% said they did not know or refused; among those who answered, 38% said “no”. Even after the authors adjusted for nonresponse, their county-level estimates from the direct question fell about 24 to 25 points short of the official count. The authors trace the direct question’s error to “two sources of bias, namely nonresponse and social desirability.”
Turnout. Enamorado and Imai linked the 2016 American National Election Studies (ANES) to a national voter file of over 180 million records. Respondents put their turnout at 76.0%; the actual rate, from the US Election Project, was 58.8%. Among respondents the file showed as nonvoters, more than 30% said they had voted, and people who were educated and interested in politics overreported more. A bias that varies by group distorts comparisons as well as totals.
Mode of interview. Pew randomly assigned panel members to answer 60 questions either by phone with an interviewer or on the web; 3,003 completed the survey. On the phone, 62% said they were “very satisfied” with their family life; on the web, 44%. Pew notes that most of the largest gaps fell on questions “where social desirability bias could play a role.”
The same study shows the limit of the simple story: Black respondents were more likely on the web than on the phone to say that Black people face “a lot” of discrimination (86% against 71%), the reverse of the pattern among white respondents. Taking the interviewer away changes the answers; only a record check, as in the two voting studies, tells you which answer is true.
Acquiescence bias and agree-disagree questions
Acquiescence bias is the tendency to agree with a statement regardless of its content, also called yea-saying. It inflates agreement with whatever the statement asserts, so an agree-disagree item partly measures its own wording. Pew reports the tendency is stronger among less educated and less informed respondents, and stronger still when an interviewer is present.
Pew asked the same idea two ways in 1999. Offered the statement “The best way to ensure peace is through military strength,” 55% agreed and 42% disagreed. Offered a choice between that statement and “Diplomacy is the best way to ensure peace,” 33% chose military strength and 55% chose diplomacy. The versions ran a month apart, so this is not one randomized experiment, but a 22-point gap is hard to blame on a month of opinion change.
The fixes are forced-choice questions, as in Pew’s second version, and item-specific ones that ask about the attribute directly (“How satisfied are you with your course?”). Balanced keying, which reverses half of a scale’s items so yea-saying cancels out, needs care: the forbid and allow experiment below shows that a reversed item is not always the mirror image of the original.
Extreme and midpoint responding
Extreme responding is a habit of choosing the end points of a scale; midpoint responding is a habit of choosing the middle. Both are response styles that belong partly to the person and partly to the situation, and they can make two groups with the same opinions look different. The mode of interview is one documented trigger.
In Pew’s randomized mode study, 45% of phone respondents placed the Democratic leaders in Congress between 34 and 66 on a 0 to 100 “feeling thermometer,” against 36% on the web. The extremes moved the other way: 19% of phone respondents gave Hillary Clinton a “very unfavorable” rating, against 27% on the web, while her positive ratings barely changed (57% and 53%).
Keep the mode the same for every group you plan to compare, label every scale point, and when comparing groups known to differ in style, report the share of end-point and midpoint answers next to the means.
How question wording and question order change answers
Wording and order effects arise when two versions of a question that ask the same thing get different answers, because a word carries a connotation or an earlier question sets the frame. Split-ballot experiments, where random halves of a sample get different versions, measure them directly. The published differences below run from 7 to 21 points.
Forbid and allow. In an experiment published in 1941, Donald Rugg asked Americans whether the United States should “forbid” public speeches against democracy, or whether it should “allow” them. As Holleman reports Rugg’s figures, 46% answered “No, not forbid,” but only 25% answered “Yes, allow,” a 21-point gap between two logically equivalent positions. Respondents, Holleman writes, “rather answer ‘no, not forbid’ than ‘yes, allow’.” The effect has been replicated in the United States (by Schuman and Presser among others), Germany, Belgium and the Netherlands.
Loaded terms. In a 2005 Pew survey, 51% favored “making it legal for doctors to give terminally ill patients the means to end their lives,” but 44% favored “making it legal for doctors to assist terminally ill patients in committing suicide.”
Order. In a contrast effect, the order makes answers differ more: in December 2008, 88% said they were dissatisfied with the way things were going in the country right after a question on George W. Bush’s job approval, against 78% without it. In an assimilation effect, answers move closer together: 66% said Republican leaders should work with Barack Obama when that question came first, and 81% when it followed a question on whether Democratic leaders should work with Republicans.
Demand characteristics and recall bias
Demand characteristics are cues that tell participants what a study expects, so they answer to fit it. Recall bias is error in remembering past events, often worse in one group than another. Both are textbook threats, and both have thinner direct evidence than their fame suggests.
Demand characteristics. Martin Orne, who introduced the idea, later defined it as “the totality of cues and mutual expectations which inhere in a social context” that shape how participants behave and report (as quoted by McCambridge and colleagues). Their 2012 systematic review screened 3,505 references for field studies designed to measure the effect and found seven, with mixed results and weak designs. They concluded that “unqualified use of the term demand characteristics should be abandoned.” Name the specific cue you worry about, keep the hypothesis from participants, and use a measure other than self-report where you can.
Recall bias. The Catalogue of Bias says it “occurs when participants do not remember previous events or experiences accurately or omit details.” Its information-bias entry cites a study that checked young adults’ daily reports of computer use against monitoring software: people using a computer less than about 3.6 hours a day overestimated, and heavier users underestimated. Recall becomes a comparison problem when one group searches its memory harder: parents of autistic children diagnosed after the 1998 MMR vaccine scare placed the onset of regression shortly after vaccination more often than parents of children diagnosed before it (Andrews and colleagues compared 26 children with regression in a study after the publicity with 86 in one before it; the difference was not statistically significant but ran in the expected direction). This is the central worry in a case-control study, where people report past exposures after the outcome is known.
Response bias vs nonresponse bias vs voluntary response bias
Response bias distorts the answers of people who took part. Nonresponse bias arises when the people who did not take part differ from those who did. Voluntary response bias arises when people choose themselves into a sample, as in a call-in poll, so those with strong views dominate. The first is a measurement problem; the other two are selection problems.
| Response bias | Nonresponse bias | Voluntary response bias | |
|---|---|---|---|
| Who is affected | People who answered | People who were asked but did not answer | Whoever chose to answer an open invitation |
| What goes wrong | Their answers are inaccurate | The responders are unrepresentative | The sample was never drawn at random |
| Typical fix | Better questions, mode, indirect methods | Follow-up, incentives, weighting | Invite a random sample instead |
Response and nonresponse bias often travel together: surveys overstate turnout because nonvoters respond less and because some nonvoters claim to have voted. With each respondent linked to the voter file, Enamorado and Imai found that overreporting “accounts for more than 90 percent of the bias of self-reported turnout, while nonresponse due to attrition plays a smaller role.” Weighting corrects who is in the sample, not what they say: in Mississippi, three adjustments for nonresponse left the direct question’s bias almost unchanged. The sampling bias guide covers nonresponse and self-selection in depth.
How to reduce response bias
Match the fix to the type. Self-administration and indirect questions reduce social desirability; forced-choice and item-specific formats reduce acquiescence; split-ballot pretests and randomized order expose wording and order effects; prospective data and records reduce recall error. Then validate what you can against an outside source, because no single fix is complete.
- Let people answer privately. Web and paper questionnaires remove the interviewer’s audience.
- Ask sensitive questions indirectly. In the randomized response technique, each respondent flips a coin in private and either answers truthfully or gives a fixed answer, so no single answer reveals anything. In Mississippi it “essentially eliminates the bias,” while a list experiment still showed a substantial amount. The price is precision, so indirect methods need larger samples.
- Make the honest answer easy to give. Pew asks about past voting with “did things come up that kept you from voting, or did you happen to vote?”, which offers a respectable reason not to have voted.
- Pretest and randomize. Pew tests new questions with focus groups, cognitive interviews and online pilots, and randomizes lists that have no natural order. Run a split-ballot pilot on the questions your conclusions rest on.
- Collect memories early. The Catalogue of Bias calls a prospective design “the most appropriate way to avoid recall bias.”
- Validate against records. Voter files, medical records and device logs show how far self-reports drift; even a small validation subsample lets you state a direction and a size.
Report the exact wording of key questions, the mode and the order in your methods; the guide to writing a methodology section shows where each detail goes.
How to write about response bias in your limitations section
Name the type, say which way it likely pushed your estimate and, where you have evidence, by roughly how much. Point to the design choices that reduced it and the ones that did not. A stated direction tells readers how to read your numbers; a generic “self-report data may be biased” does not.
A model for a student survey, with placeholders in brackets:
“Alcohol use was self-reported in an online questionnaire. Self-administration reduces the pressure to give socially desirable answers, but [number] of [number] respondents completed it in a supervised computer room where others could see their screens. Because heavy drinking is the less desirable answer, our estimate of [percent]% is more likely to be too low than too high. We did not validate answers against any record, so we cannot say by how much.”
Our guide to the limitations of a study has a full template and published examples.
Where Papero fits
Papero is an AI writing workspace for students and researchers. It does not design questionnaires or find biased answers in your data; those judgments stay with you. It helps with the reading and writing around those judgments: comparing how published studies worded their questions, and checking that the claims in your methods and limitations sections have sources.
Chat answers questions about the PDFs in your library and your document, citing the passages it used, which helps when you compare how several studies worded a sensitive question. In Review, the Claims check finds sentences that need a citation and offers papers to cite, and the source check flags cited papers that are retracted or preprints. For a dissertation built on survey data, the thesis writing page shows how chapter outlines, one source library and version history fit together.
Frequently Asked Questions
What is response bias in simple terms?
It means people’s answers to a survey or interview are wrong in a consistent direction. They took part, but the wording, the question order, an interviewer’s presence or a wish to look good led them away from the truth. Because the error points the same way for many people, a larger sample does not cancel it.
What is social desirability bias?
Social desirability bias is the tendency to answer in the way that makes you look good: overreporting voting, exercise or charity and underreporting drinking, drug use or prejudice. In the 2016 American National Election Studies, respondents reported 76.0% turnout against an actual 58.8%. Private, self-administered questionnaires and indirect questioning methods reduce it.
What is nonresponse bias in simple terms?
Nonresponse bias happens when the people who do not answer a survey differ from those who do in ways related to what it measures. If healthier people are more willing to fill in a health questionnaire, the results look healthier than the population. It concerns who is missing, not whether the answers given are accurate.
What are examples of response bias in survey questions?
Agree-disagree statements draw extra agreement: in a 1999 Pew survey, 55% agreed that military strength best ensures peace, but 33% chose that view in a forced choice. In a 1941 experiment, 46% said the US should “not forbid” speeches against democracy, but only 25% said it should “allow” them. Direct questions about voting or drug use also invite flattering answers.
What are the types of bias in psychology research?
Psychology research meets three broad families of bias. Selection biases concern who takes part, such as volunteer and nonresponse bias. Biases in what participants report include social desirability, acquiescence, demand characteristics and recall bias. Researcher biases concern how data are observed and interpreted, such as observer bias and confirmation bias.
Sources
- Andrews, N., Miller, E., Taylor, B., Lingam, R., Simmons, A., Stowe, J., & Waight, P. (2002). Recall bias, MMR, and autism. Archives of Disease in Childhood, 87(6), 493–494. https://doi.org/10.1136/adc.87.6.493
- Catalogue of Bias Collaboration, Bankhead, C. R., Spencer, E. A., & Nunan, D. (2019). Information bias. In Catalogue of Bias. https://catalogofbias.org/biases/information-bias/
- Catalogue of Bias Collaboration, Spencer, E. A., Brassey, J., & Mahtani, K. (2017). Recall bias. In Catalogue of Bias. https://catalogofbias.org/biases/recall-bias/
- Enamorado, T., & Imai, K. (2019). Validating self-reported turnout by linking public opinion surveys with administrative records. Public Opinion Quarterly, 83(4), 723–748. https://doi.org/10.1093/poq/nfz051
- Holleman, B. C. (2006). The meanings of ‘yes’ and ‘no’: An explanation for the forbid/allow asymmetry. Quality & Quantity, 40(1), 1–38. https://doi.org/10.1007/s11135-005-4479-6
- Keeter, S. (2015). From telephone to the web: The challenge of mode of interview effects in public opinion polls. Pew Research Center. https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/
- McCambridge, J., de Bruin, M., & Witton, J. (2012). The effects of demand characteristics on research participant behaviours in non-laboratory settings: A systematic review. PLOS ONE, 7(6), Article e39116. https://doi.org/10.1371/journal.pone.0039116
- Pew Research Center. (n.d.). Questionnaire design. Internet Archive capture of 8 December 2018: https://web.archive.org/web/20181208024723/http://www.pewresearch.org/methods/u-s-survey-research/questionnaire-design/ (source of the 1999 acquiescence example)
- Pew Research Center. (2021). Writing survey questions. https://www.pewresearch.org/writing-survey-questions/
- Rosenfeld, B., Imai, K., & Shapiro, J. N. (2016). An empirical validation study of popular survey methodologies for sensitive questions. American Journal of Political Science, 60(3), 783–802. https://doi.org/10.1111/ajps.12205
- Rugg, D. (1941). Experiments in wording questions: II. Public Opinion Quarterly, 5(1), 91–92. https://doi.org/10.1086/265467 (figures as reported by Holleman, 2006)
- Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859