What new research-on-research says about fraud in survey data

There’s a myth that continues to hang around in the sample industry. It goes something like this: if you start with a “good” panel, you don’t need to worry very much about fraud. The panel is double opt-in. It’s by invitation only. It appears to have all the right things that should lead to data quality.

We wanted to see if those claims really hold up under scrutiny, so we recently ran a research-on-research study to evaluate fraud across six different online sources. We used the exact same survey for respondents from each source, applied census-balanced demographic quotas, and then tested all responses using the full suite of the fraud detection tools available in Rep Data’s Research Defender.

What we found was that fraud is not a niche problem. It is not something that comes only from low-quality sources or among bad actors in obscure corners of the internet. It is everywhere.

29 percent. That’s the average share of total respondents flagged as suspicious or fraudulent. And that number was consistent across all the online sources we included in the study.

One of the biggest surprises from the study came from our hyperactivity metric. This measure looks at how many surveys a person attempts in a 24-hour period. We flagged respondents who had attempted more than 70 surveys in that window. Every source had some of these users, with respondents attempting 200 or more surveys in all panels. One source had a respondent who tried to take more than 1,100 surveys in a day. This is not an edge case. These are not isolated blips. These are symptoms of a system that lets repeat offenders roam freely.

We also flagged 12 to 14 percent of respondents across sources for digital manipulation—things like duplicate IP addresses, VPN usage, and device spoofing. These kinds of behaviors are harder to catch with traditional checks. But they matter, because they often signal professional fraud.

Open-ended response failures were also consistent across sources. One to three percent of respondents were flagged for nonsensical or auto-generated answers. That may not sound like much, but these patterns add up, especially in longer studies or those involving qualitative follow-up.

One way to combat this is with a layered approach to fighting fraud. You can start by seeking a provider that pulls from a curated set of sources to avoid single-source bias and ensure feasibility. Make sure to screen every respondent through a comprehensive fraud detection solution before they enter the survey—and throughout the survey process. Look for things like flagging of suspicious open-ended responses and the ability to uncover those respondents who are attempting dozens of surveys in a short amount of time. Lastly, apply a mix of the latest technology, machine learning and more, and human expertise to clean up low-quality data after collection. Rep's Research Defender is the most advanced tool for gaining a complete understanding of your data quality, and was used to detect the fraud in this recent study.

No one can eliminate fraud entirely, but you can reduce its impact dramatically. The key is not to assume that quality comes baked into any specific sources you are utilizing. You have to measure it, track it and block it using the right tools and approaches. And this isn’t a one and done, it is an ongoing battle that requires building safeguards into every step of the process.

The fraud is there. The only question is whether you’re finding it. 

Discover more about how Research Defender can help! Want to learn more about this recent research-on-research study from Rep Data? Sign up for our webinar.