Quality Days Recap: Are We Getting Survey Data Quality Wrong?

Rep Data Team · October 29, 2026 · 7 min read

New findings on survey fraud detection, overcleaning, respondent quality and sampling bias

Are we getting survey data quality right? When findings show that only one-third of research buyers feel highly confident that poor-quality respondents are being weeded out, this is a problem for our industry. Another problem is that traditional quality checks don't always distinguish between fraudulent respondents and inattentive participants, both of whom can compromise research findings. There's clearly still some work to do here to set a higher standard for data quality across the industry…

In this blog post, you'll learn:

  • What research buyers want to know about survey data quality, and how quality concerns are shaping their purchasing decisions

  • Why a single failed quality check isn't enough to justify removing a respondent

  • How fraudulent and inattentive respondents behave differently, and why they call for different detection methods

  • How survey design and sampling decisions can introduce bias into research findings

We dug into these issues and plenty more during Rep Data Quality Days, our two-part webinar series featuring experts from the Insights Association, Esomar, the Market Research Society (MRS) and the Global Data Quality Initiative (GDQ). Surprising findings were shared in the presentations and our speakers offered practical ideas for improving the quality of market research data.

Five Quality Days takeaways at a glance

  • Research buyers are demanding greater transparency into data quality.

  • Have proper checks and balances in place to evaluate respondents on a variety of measures.

  • Fraudulent respondents can look remarkably attentive.

  • Survey design decisions can introduce significant bias.

  • No single quality check tells the whole story.

Here's a closer look at some of the research and advice shared during the sessions.

Research buyers want greater transparency into survey data quality

Justifiably, research buyers are concerned about the quality of the data they're receiving, and those concerns are affecting their purchasing decisions. During Quality Days, Anita Watkins, CEO of the Insights Association, and Ray Poynter, chair of Esomar's Professional Standards Committee, shared findings from a GDQ study of 173 research buyers across 29 countries. Just 33% were very or extremely confident that poor-quality respondents were removed before data delivery, and 45% had eliminated a supplier from future consideration because of quality issues. Another 39% had paid more for stronger sample or quality controls.

Buyers also want a better understanding of what happens to their data before they receive it. Anita and Ray shared that 62% wanted more information about sample sources and recruitment methods, while the same percentage wanted details on data cleaning rules and how many respondents were removed. Providing that visibility into sample sourcing and quality checks is already part of how Rep Data works with its clients.

What this means for researchers: Make data quality expectations part of the conversation before fieldwork begins, including what suppliers will report back and how quality issues will be handled.

Watch: What buyers expect from suppliers

Why survey data quality requires more than a single check

Removing poor-quality responses is an essential part of research, but there's a risk in being too aggressive. Some respondents who fail an individual quality check may actually be providing perfectly usable data. Research presented by Chris Stevens, a GDQ contributor, and Debrah Harding, Managing Director of MRS, found that 55% of respondents who straight-lined through a survey grid triggered no other quality flags. Another analysis showed that 40 to 50% of participants flagged for questionable open-ended responses were flagged by only one of five reviewers. As Stevens put it, "One flag does not make a poor participant."

Straightlining, where a respondent selects the same answer across a series of questions, unusual open-ended responses and attention-check failures can all warrant a closer look.

What this means for researchers: Consider multiple quality signals before excluding respondents, and assess whether your cleaning decisions could introduce bias into the sample.

Watch: The risks of overcleaning

Why survey fraud and respondent inattention require different detection methods

Fraudulent respondents and inattentive respondents can both compromise research findings, but they don't necessarily behave the same way. Researchers need to account for those differences when choosing quality checks and evaluating the results. Original Rep Data research presented by our very own Steven Snell, PhD, Chief Research Officer, looked at almost 12,000 survey completes and found that conditional on being allowed into the survey, 76% of suspicious and fraudulent respondents passed subsequent attentiveness checks. On the other hand, approximately 60% of inattentive respondents appeared to be from well-qualified, non-suspicious respondents.

Traditional survey quality checks don't always catch the problems they're meant to catch. Someone deliberately trying to cheat a survey may give perfectly reasonable, consistent answers, while a genuine respondent might rush through a few questions or leave something incomplete. AI agents are making things even trickier, since they can complete surveys in ways that look remarkably human.

What this means for researchers: Use different approaches to identify fraud and inattention, combining device intelligence and digital fingerprinting before the survey, behavioral checks during participation and response analysis afterward.

Watch: Detecting fraud vs. inattention

How survey design and sampling decisions affect research quality

Something as simple as reversing the direction of a response scale can change research results quite a bit. During Quality Days, John Bremer, CEO and founder of Signals and Standards Research Consulting, shared an example where changing the scale direction shifted top-two-box scores by 8 to 10 percentage points. Quota management, sampling frames, incentives, weighting and even how questions appear on mobile devices can also influence who participates, how people respond and ultimately the conclusions researchers draw from the data.

Researchers put a lot of effort into checking respondent quality, but even the best sample and fraud detection tools can only do so much if the survey itself isn't well designed.

What this means for researchers: Review survey design and sampling decisions before fieldwork begins, paying attention to how question wording, response scales and sampling methods could affect the results.

Watch: How survey design introduces bias

Improving survey data quality with a layered approach

Check out our Quality Days recordings for some things to think about before your next project. It's worth taking some time reviewing how respondents are screened, when responses are flagged for removal and whether survey design or sampling decisions could be introducing bias. At Rep Data, we've built our approach around multiple layers of survey fraud detection, behavioral assessments and data validation, giving researchers more information about respondent quality before making decisions about which data to keep. Ultimately, better data quality isn't necessarily about removing more respondents. It's about making better-informed decisions about which data to trust.

There's more to dig into from Quality Days, including the research behind these findings and advice from our speakers on improving survey data quality. Watch both sessions on demand for the full presentations and audience Q&A: Day 1: From Buyer Trust to Better Quality Decisions and Day 2: Survey Design and Data Quality in the Age of AI. And if you're reviewing your own sample sourcing or fraud detection processes, talk with the Rep Data team about how we can help.

Frequently asked questions about survey data quality

What should I ask my sample provider about data quality?

You can start with asking where respondents come from and how they're verified. What fraud detection tools are used, how quality is monitored during fieldwork and what happens when a respondent is flagged? You should also know what information you'll receive about respondents who were removed and why.

Can someone pass attention checks and still be a fraudulent respondent?

Absolutely. Fraudsters who know what they're doing can give thoughtful, consistent answers and easily pass traditional attention checks. A respondent who appears attentive isn't necessarily legitimate, which is why researchers need other ways to identify suspicious activity.

Does one failed quality check mean a respondent should be removed?

This isn't necessarily true. Someone might straight-line through a grid because they genuinely feel the same way about every statement, or an unusual open-ended answer might also be perfectly valid. It's worth looking at other indicators before deciding whether that respondent's data should be excluded.

When should survey data quality checks happen?

Before, during and after the survey! Some suspicious activity can be identified before a respondent even starts answering questions, while other problems become apparent through participation patterns or responses. Checking at different stages of the research process helps researchers catch potential quality issues early and make better decisions about which data to trust.

How do I know if my survey design is affecting data quality?

It is important to test your questionnaire before launching it. Are any questions confusing? Does the survey work well on a phone? Are respondents getting tired of answering repetitive grids? It's also worth looking at where people drop out or start rushing through questions once you are in field, since those patterns may point to problems with the questionnaire itself.