Brief overview
It is generally thought that pay rate does not affect data quality on Mechanical Turk. For example (Buhrmester, Kwang, & Gosling, 2011) showed that whether Workers are paid 5 cents or one dollar for a survey study, the internal reliability of the surveys does not change. They did show however that fewer Workers will take the surveys that pay less. We recently replicated these findings for both US and India-based Workers (Litman et al, 2014). Here we show that low pay rates have two effects on Workers: 1) Workers are more likely to return a HIT before completing it and 2) Workers spend less time answering questions. We examined 30 MTurk studies that were run over the last 6 months. The findings show that 36% of the dropout rate variance is explained by the length and pay rate of a survey. These results show that low pay rates do more than just slow down the rate at which Workers take HITs. Low pay rates may also negatively impact the representativeness of data due to high participant dropout, and they may also decrease how much attention participants pay to each question. Based on these findings we recommend against low paying HIT We also recommend against overly long surveys, unless Workers are appropriately compensated. To minimize dropout and to maximize time on task, compensation for HITs should not be below $4 per hour and should be closer to $6 per hour or more.
Introduction
Dropout on Mechanical Turk occurs when Workers do not complete a started HIT. Although sometimes a Worker cannot complete a HIT due to technical difficulties, the most common reason for dropout is that a Worker Returns the HIT. An important question that has not been previously addressed in the MTurk literature is whether there are specific factors that may increase the likelihood with which HITs are returned.
In a previous post we discussed how to monitor dropout rate in real time on TurkPrime. Here we address a more complex issue – what factors contribute to dropout rate and what are the ways to minimize dropout. To examine the factors that may be associated with dropout rate we examined 30 studies that we ran on TurkPrime. Payment, completion time, the number of questions in the survey, the number of seconds it takes to answer a survey question, and pay rate per hour, was examined as possible predictors of dropout rate.
Method
Thirty studies that were run over a period of 6 months on TurkPrime in 2014 were examined. This study only examined Hits that utilized survey instruments, and did not examine non-survey tasks, or surveys that included open-ended questions. The number of unique participants in each study ranged between 20 and 3991 (mean = 530, SD = 927). Payments ranged between 20 cents and $1.25; corresponding to a pay rate per hour that ranged between $1.88 and $8.57 (mean = $4.50, SD $1.7). The total number of questions per survey ranged between 28 and 176 (mean = 65, SD = 42).
Results
Completion rate ranged between 69% and 100%. Average completion rate was 89.7% (SD = 8.7). Completion rate did not correlate with payment, r (28) = -.025, p = .896; but completion rate did positively correlate with pay rate per hour, r (28) = .39, p = .039. Completion rate also negatively correlated with the number of survey questions in a HIT, r (28) = -.414, p = .026. Completion rate also positively correlated with the average time that it took participants to answer survey questions, r (28) = .5, p = .020.
In the next analysis all three variables that were shown to correlate with completion rate were entered as simultaneous predictors of completion rate in OLS regression. The results showed that 36% of completion rate can be predicted from pay rate per hour (Beta = .25), number of questions in a HIT (Beta = -.3), and average time to answer a question (Beta = .25), F(3,27) = 3.9, p = .022. However, partial correlations were not significant for any of the three predictors, indicating that there is a single underlying process that accounts for completion rate.
Discussion
Three factors were shown to be predictive of completion rate. Payrate per hour positively predicted completion rates, and so did the length of the HIT. This indicates that when workers see that a HIT is taking too long they are more likely to return it. Requesters’ estimates of how long it takes to complete a HIT varies in accuracy. If a hit that pays 50 cents is advertised at taking 5 minutes, but in reality takes ten minutes, Workers will Return that HIT when they become aware of the discrepancy between the advertised and the actual HIT length. The number of questions in a survey is also a significant predictor of dropout, indicating that Workers are less likely to complete very long tasks.
Perhaps the most surprising results was the positive correlation between the average time it takes to answer a question and completion rate. This correlation indicates that HITs with a low dropout rate are also the ones on which Workers take more time per question. This finding suggests that Hits that increase the likelihood of dropout also increase the likelihood that Workers will spend less time on the HITs questions, possibly because they are becoming more impatient and less interested in the task. These findings suggest that long and low paying HITs have a dual effect on Workers: some workers dropout and other workers start speeding through the study.
Previous studies have reported that low paying tasks do not affect data quality but only slow down the speed with which Workers take a HIT. The data presented here paint a different picture, and suggest that low paying tasks can decrease data quality in two ways. First, low paying tasks increase drop out, which may have a systematic effect on the representativeness of the data. Second, low paying tasks make Workers more likely to speed through the HIT. This is likely to affect how much cognitive resources a worker allocates to a task, and may influence the likelihood that a worker will devote their energy to the task.
Conclusion
Overall, increasing the pay rate of a HIT is likely to minimize dropout and to increase how much attention your Workers allocate to your study. To minimize dropout, compensation for HITs should not be below $4 per hour and should be closer to $6 per hour or more.
