It is important to consider how many highly experienced workers there are on Mechanical Turk. As discussed in previous posts, there is a population pool of active workers in the thousands, but this is far from exhaustible. A small group of workers take a very large number of HITs posted to MTurk, and these workers are very experienced and have seen measures commonly used in the social and behavioral sciences. Research has shown that when participants are repeatedly exposed to the same measures, this can have negative effects on data collection, changing the way workers perform, creating treatment effects, giving participants insight into the purpose of some studies, and in some cases impact effect sizes of experimental manipulations. This issue is referred to as non-naivete (Chandler, 2014; Chandler, 2016).
Best recruitment practices: working with issues of non-naivete on MTurk
Topics: amazon mechanical turk, approval rating, experience, exposure, HIT, mechanical turk, mturk, naivete, non-naive, primepanels, qualification, recruitment, requester, workers
The internet has the reputation of being a place where people can hide in anonymity, and present as being very different people than who they actually are. Is this a problem on Mechanical Turk? Is the self-reported information provided by Mechanical Turk workers reliable? These are important questions which have been addressed with several different methods. Researchers have examined a) consistency of responding to the same questions over time and across studies b) the validity of responses, or the degree to which the items capture responses that represent the truth from participants. It turns out that there are certain situations in which MTurk workers are likely to lie, but they are who they say they are in almost all cases.
Topics: amazon mechanical turk, anonymous, demographics, HIT, mturk, panels, qualification, turkprime panels, unique worker, worker groups, workers
Hundreds of academic papers are published each year using data collected through Mechanical Turk. Researchers have gravitated to Mechanical Turk primarily because it provides high quality data quickly and affordably. However, Mechanical Turk has strengths and weaknesses as a platform for data collection. While Mechanical Turk has revolutionized data collection, it is by no means a perfect platform. Some of the major strengths and limitations of MTurk are summarized below.
Topics: amazon mechanical turk, demographics, exclude workers, google form mechanical turk, HIT, mechanical turk, mturk, mturk api, panels, qualification, study, turkprime panels, unique worker, worker groups, workers
How to Create a Universal Exclude Worker List
Problem:
Requesters may observe that some workers, even those with high Approval ratings, may not perform to their expectations on a study. Sometimes this may result in rejecting their work which affects the Worker approval rating. But, often the work is not acceptable for research but is not worthy of rejection, or, it may simply be the policy of the research lab to approve all assignments for IRB or some ethical standard they may follow.
Topics: amazon mechanical turk, block worker, exclude, exclude workers, mturk, mturk api, qualification, turkprime
Problem:
You are running a longitudinal study and have identified 1000 workers who you want to allow to take your second phase studies. How do you easily group those workers for easy access.
Or you want to exclude certain workers from taking a number of your studies and wish to group them for easy exclusion in future studies. How can you do that?
Topics: amazon mechanical turk, exclude, exclude workers, include workers, Longitudinal, qualification, turkprime, worker groups
Studies with Panels for just $0.15 - 0.75 / complete
Now you can run Mechanical Turk studies using your own Requester account and specify over two dozen demographic traits!. The traits include gender, ethnicity, age, marital status and sexual orientation. But it does not stop there! The available options also include occupation, medical and health history, cell phone use and much more.
Topics: amazon mechanical turk, demographics, mechanical turk, mturk, qualification, traits, turkprime, turkprime panels
Survey Groups Ensure Unique Workers Across Multiple Concurrent Studies
Problem
Have you ever wanted to run multiple studies simultaneously and make sure that each worker only takes a single study? On MTurk, there is no simple way to block workers who completed one study from accepting and completing another study being run at the same time....until now.
(Note: TurkPrime's exclude feature excludes workers who have completed one study from taking another subsequent study, but not if both studies are being run at the same time.)
Topics: amazon mechanical turk, exclude, exclude workers, follow up, qualification
Problem:
How can you increase Amazon Mechanical Turk HIT Worker participation rates and speed completion of a HIT? This is particularly an issue with HITs that have a large number of required participants or have Qualifications that limit the number of qualified Workers
Topics: amazon mechanical turk, completion rate, demographics, HIT, increase participation, qualification, speed, turkprime
Creating Mechanical Turk Custom Panels with TurkPrime Worker Groups
Problem:
Suppose you need to run a group of HITs open only to participants who are women under 50. You previously ran a HIT and know the Worker IDs that you want to reach, but have no way to email them and limit your survey to only them. How can you proceed?
Topics: amazon mechanical turk, mechanical turk, mturk api, panels, qualification, worker groups
System Qualification Enhancements - US State Qualifications
Amazon just announced that their Worker Qualification now supports US State locations. It is currently available through their API and is also available through their Web Interface.
Topics: amazon mechanical turk, mturk, mturk api, qualification