Online providers of paid participants yield high data quality and reliability in psychological experiments
the verdict
CONTESTED
the evidence cuts both ways
confidence 35/100
While online participant providers can yield reliable data when using strict screening filters and platforms like Prolific or CloudResearch, overall data quality is heavily contested due to prevalent issues with inattentive respondents, speeders, and non-naivety.
Evidence for · 3
Data quality of platforms and panels for online behavioral research
2021 · cited by 1,422
The study finds that specific platforms like Prolific and CloudResearch can yield high data quality when proper attention and screening filters are utilized.
Evidence against · 4
A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)
2021 · cited by 80
The guide highlights frequent data quality issues on crowdsourcing platforms, such as speeders, cheaters, and deliberate falsification.
See more details
More for · 2
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA
2023 · cited by 1,347
Results indicate that Prolific and CloudResearch participants frequently pass attention checks, follow instructions carefully, and provide meaningful responses.
Can Crowdsourcing Platforms Be Useful for Educational Research?
2024 · cited by 8
Data from problem-solving tasks show that Prolific participants perform comparably to traditional in-person student samples.
More against · 3
Evaluating mobile-based data collection for crowdsourcing behavioral research
2025 · cited by 11
Research points out that participants highly motivated by income or who frequently use platforms may provide lower-quality, inattentive responses.
Caution when Crowdsourcing: Prolific as a Superior Platform Compared with MTurk
2025 · cited by 3
A direct comparison found that only a minority of participants on platforms like MTurk and Prolific produced high-quality data, emphasizing that data quality is often questionable.
Beyond online participant crowdsourcing: The benefits and opportunities of big team addiction science.
2022 · cited by 0
Concerns regarding participant non-naivety, low motivation, and careless responding can severely compromise the reliability of online participant crowdsourcing.