A quick diagnostic checklist for research and insights teams. If three or more sound familiar, keep reading.
Fraud rarely announces itself. With roughly a third of survey attempts now fraudulent and AI generated answers passing standard quality checks, the signs are subtle, statistical, and easy to rationalize away. Here are twelve worth taking seriously.
1. Your open ends got better. Suspiciously better. Fluent, on topic, well-structured answers at scale are now more likely to be a language model than an unusually articulate panel. The old tells (gibberish, copy paste) are gone; eloquence is the new red flag.
2. Completion times cluster too tightly. Real humans are messy: some race, some wander off and return. When a large share of completes land in a narrow time band, automation or scripted farms are the likelier explanation.
3. Straight lining has gotten smarter. Instead of all 5s, you see plausible variation that never quite contradicts itself. Sophisticated fraud mimics attentiveness; check whether grid answers correlate too perfectly with each other.
4. Incidence rates do not match reality. When 30% of your general population sample claims to own a boat, manage enterprise IT budgets, or have a rare condition, fraudsters are qualifying into your highest paying screeners.
5. Demographics shift between waves. A tracker whose respondent profile drifts wave to wave without a real world reason often means the fraudulent share of your panel is changing underneath you.
6. Geography and IP do not line up. Respondents claiming one location while submitting from another, or clusters of completes from data center IP ranges, point to farms and proxies.
7. Your cleaning rate keeps creeping up. If you removed 5% of completes two years ago and remove 15% now, the question is not whether fraud is rising; it is how much is still getting through, since cleanup only catches what your rules can see.
8. Trap questions stopped trapping. When attention check failure rates fall while everything else looks worse, the fraud has learned your checks. AI assisted respondents pass traps designed for careless humans.
9. Surprising findings keep failing to replicate. Fraud injects noise that masquerades as insight. If your interesting subgroup differences evaporate on re fielding, contamination is a prime suspect.
10. The same “person” keeps coming back. Matching response patterns, device fingerprints, or open end phrasing across supposedly different respondents means duplicates or a persona farm.
11. Your panel provider cannot answer the identity question. Ask directly: what share of these respondents passed identity verification, and by what method? If the answer is a quality score rather than a verification method, identity is unverified.
12. A client asked, and you got defensive. The clearest sign of all. If “how do we know these are real people” produces discomfort instead of a document, your process has a proof gap regardless of how clean the data actually is.
What to do with your count
0 to 2 signs: Stay vigilant; your exposure is likely moderate. Your next move is proof: being able to demonstrate integrity, not just maintain it.
3 to 5 signs: You have a live problem. Take our Survey Fraud Risk Scorecard to locate exactly where fraud is entering: at identity, at submission, or after.
6 or more: Your datasets are materially contaminated, and cleaning alone will not fix it, because cleaning only catches what your rules already know to look for. The fix is structural: verify identity before anyone answers, screen at submission, and keep a tamper evident record after.
That three layer structure is what BallotHut does: KYC verified identities, CAPTCHA plus 80M+ address database screening at submission, and an immutable post submission record on the XRP Ledger. The fastest way to see the difference is a paid pilot on one of your own studies.
Tracy Wehringer CMO, BallotHut