Market Research: Foundations to Practice · Survey Data Quality and Fraud Prevention
Data Cleaning and Exclusion Criteria
Deciding which responses to exclude after collection requires transparent, pre-defined criteria, not ad hoc judgment calls made after seeing which responses are inconvenient for the emerging findings. This chapter covers what a defensible process looks like.
Data cleaning criteria, such as minimum completion time thresholds, straight-lining detection rules, and attention check requirements, should be defined and documented before a researcher looks at the actual substantive results, since defining exclusion rules after seeing which responses are inconvenient for the emerging findings introduces a serious risk of researchers unconsciously cherry-picking exclusion criteria that happen to remove responses contradicting a preferred, expected conclusion.
- Data cleaning criteria should be defined and documented before viewing results, to avoid unconsciously cherry-picking exclusions that favor a preferred conclusion.
- A defensible process combines multiple checks; a response failing several independent signals is a stronger exclusion case than one failing a single weak signal.
- Transparently reporting the cleaning rate and specific removal reasons is a necessary part of methodology documentation, not an optional detail.
- Both under-cleaning (leaving fraud in) and over-cleaning (removing valid responses) are real risks; criteria should be proportionate and defensible, not maximally aggressive.