Administering Panels on Amazon Mechanical Turk: A Guide to Within-Subjects Experiments. Click here for latest version.

Transcription

1 Administering Panels on Amazon Mechanical Turk: A Guide to Within-Subjects Experiments Kyle A. Dropp 1 March 7, 2014 Click here for latest version. Comments Welcome! This document provides step-by-step instructions for implementing multiple, complex treatments across survey waves. Data analysis will be conducted in R, surveys will be programmed in Qualtrics, and cases will be collected via Amazon Mechanical Turk (AMT), though the instructions are applicable to other samples. 2 Note: this tutorial assumes very limited knowledge or familiarity with Qualtrics, Amazon Mechanical Turk, HTML or server applications, but more advanced applications require basic knowledge of php, sql and MySQL. All supporting materials are available at kyleadropp.com/panels 1 Assistant Professor, Department of Government, Dartmouth College kyle.dropp@dartmouth.edu I thank Solomon Messing for providing information on MTurkR /06/24/streamline-your-mechanical-turk-workflow-with-mturkr/ 2 These instructions can be utilized for implementing panels via Survey Sampling International, Inc. (SSI), for instance. 1

2 Contents 1. Why Panels? 3 2. Wave Programming Survey in Qualtrics Login Uploading Survey Obtaining Worker ID Foreign Direct Investment Experiment Confirmation Code Programming in Amazon Mechanical Turk Fielding Wave Evaluating Results from Wave Wave Upload Wave Branch Logic and Wave 2 Treatment Managing Wave 2 from R Install MTurkR Enter Amazon Mechanical Turk credentials Invite Wave 1 respondents to Wave Compensating Respondents Data Analysis 15 2 Materials: kyleadropp.com/panels

3 1. Why Panels? Panels are a powerful tool that can identify changes in opinion (or behavior) over time, inference inferences made from single-shot, between-subjects experiments, increase the precision of statistical estimates and address important methodological questions such as the rate of experimental decay. This tutorial provides step-by-step instructions for implementing multiple, complex treatments across survey waves. 2. Wave Programming Survey in Qualtrics. Prior to creating our HIT in Amazon Mechanical Turk (AMT), we will program a survey in Qualtrics, a top notch interface for programming surveys Login. Login to your Qualtrics account (e.g., com/ for Stanford University, princeton.qualtrics.com for Princeton University, or tuck. qualtrics.com/ for Dartmouth College). University affiliates should have free access to an account, but see your department or university administrator if you have difficulty accessing Qualtrics Uploading Survey. Surveys for Waves 1 and 2 are pre-programmed for this tutorial. You can simply upload them to Qualtrics. On the Edit Survey tab, select Advanced Options on the far right, then select Import Survey, and browse to the file panel wave1 tutorial.qsf in the supporting materials folder. Choose the file and then select Import. This survey is the first wave of the study and includes a basic experiment on how Americans preferences toward foreign direct investment vary based on the foreign country making the investment. The foregoing subsections will describe key segments of the uploaded survey Obtaining Worker ID. Each worker on AMT has an ID called a Worker ID that we will obtain to prevent duplicate entries and to link individual responses across multiple waves. On the Edit Survey tab, select Survey Flow. There is an empty embedded data variable called MID in your Amazon Mechanical Turk survey flow that obtains each AMT Worker ID. Basically, when AMT workers enter the survey, they will have an MID on their url and this embedded data captures this value. For Survey Sampling International (SSI), the respondent identifier is typically psid. 3 Materials: kyleadropp.com/panels

4 Foreign Direct Investment Experiment. Respondents view a fake newspaper article describing a battery making company that has been acquired. By random draw, the company purchasing the battery maker is Japanese, German, Chinese, or American. After viewing the article, respondents state whether they favor or oppose the acquisition. In a subsequent question battery, they state whether they believe the acquisition will harm national security, lead to job losses, or harm American culture and values. See below for a picture of the newspaper article in the Japanese condition: The two figures below, contained in the Survey Flow, indicate that the country ( rand- Country ) is randomized between one of four options and the first option displays the Germany treatment. The other randomizations ( randcountry = 2, 3, or 4) display the Chinese, German, and American treatments, respectively. 4 Materials: kyleadropp.com/panels

5 In Wave 2, the respondent will view a similar article but will be randomly assigned to another value of randcountry. That is, the respondent who sees the Germany treatment in Wave 1 will have an equal probability of viewing the American, Chinese, or Japanese treatment in Wave 2. This design creates a powerful between-subjects and within-subjects analysis. Following the main post-test dependent variable, they answer a simple question battery to identify mechanisms for support or opposition to the acquisition Confirmation Code. We generate a confirmation code between 5,000,000 and 9,999,999 and ask respondents to enter the code into the AMT interface to receive payment. A corresponding descriptive text question at the end of the survey provides the confirmation code generated from the web service. See the figure below. 5 Materials: kyleadropp.com/panels

6 2.2. Programming in Amazon Mechanical Turk. Login to Amazon Mechanical Turk as a Requester. If you are not familiar with AMT, there are many good tutorials. 3 Please note jobs, also known as HITs, can be setup with the R package mturkr. 4 Enter the Create tab in Amazon Mechanical Turk, select New Project, then select Survey, and finally click Create Project. Here are the details I suggest entering for this project. Enter the title Answer a short survey and description Answer a short, fun survey and keywords surveys. Set the reward to $0.20, assign 100 HITs, allot 1 hour to complete, have the HIT expire in 6 hours, and have the results automatically approved in 6 hours. 3 Welcome.html Materials: kyleadropp.com/panels

7 Select the Advanced tab on the bottom right, select Worker requirements, click Customize Worker Requirements.. in the drop down menu, specify Location as United States, HIT Approval rate greater than or equal to 95, and Number of HITs Approved greater than or equal to 100. Select Design Layout to move to the next pane, click OK when prompted regarding Master Workers. You will now add the content to the HIT. This will include a brief description of the survey, a link to the survey in Qualtrics, and a confirmation code for the AMT respondent to enter upon completion of the Qualtrics survey. 7 Materials: kyleadropp.com/panels

8 Open the AMT wave1.html file in the supporting materials folder and click Source on the right side of the Design Layout page. Paste the entire HTML file into the body. This file includes code with a brief description of the project, commands to extract the AMT respondent s unique Worker ID, and a confirmation code. Modify lines 1, 4, and 5 that pertain to survey length and eligibility. In line 28, modify the quoted href portion to the full Qualtrics link for the survey. You can find the link to your survey in Qualtrics by clicking on the Distribute Survey tab in Qualtrics. 5 If you click Source again, the body should look like this (figure below) and contain a warning message regarding JavaScript. You can ignore this warning message. Click Preview to preview the HIT and click Finish to finish the programming portion of the HIT. 5 Thanks to these scholars for the HTML code - participants_from_previous_studies_on_amazon_mechanical_turk_and_qualtrics 8 Materials: kyleadropp.com/panels

9 2.3. Fielding Wave 1. Now click the Create tab in AMT, find your Project, select New Batch, click Next, and then click Publish HITs. You may need to add funds to your account. Your HIT is now live and you are collecting data! You can review the status of your project by clicking the Manage tab. There are a number of methods for reviewing and approving completed HITs. From the least to most automated, you can select the Results tab on a given project and review the confirmation codes for completed HITs, you can merge a.csv of the results from Amazon with your Qualtrics.csv file (on the confcode variable) and check whether the confirmation code is correct, or you can use the package mturkr to batch approve respondents who have provided the correct confirmation code. 9 Materials: kyleadropp.com/panels

10 Evaluating Results from Wave 1. After you have finished Wave 1 data collection, login to Qualtrics, select View Results, click Download Data, scroll down and click the highlighted CSV to download a.csv. The key variables in this file are the Worker ID ( MID ) and the Wave 1 treatment assignment ( randcountry ), the confirmation code ( confcode ), support for the acquisition ( Q16 ), and post-test mechanism questions (e.g., Q25 1 through Q25 5). Using R, 6 I randomized the Wave 2 treatment assignment ( randcountry2 ). This file has been saved as wave2 assignment.csv online. ## Read file with completed responses from Dropbox public folder df0 = read.csv(" df1 = read.csv(" names(df1)=names(df0) ## Add Randomization to Wave 2 df1$randcountry2 = rep(na,dim(df1)[1] ##create null variable for(i in 1:nrow(df1)){ ##loop through Wave 1 rows hline1 = df1$randcountry[i] ## identify Wave 1 treatment conds = 1:4 ## list of possible Wave 2 treatments df1$randcountry2[i] =sample(conds[conds!=hline1],1)} ## Sample from non-wave 1 treatments df1$randcountry2 write.csv(df1[order(df1$randcountry2),],"wave2_assignment.csv",row.names=f) # specify location In an ideal world, we would wait a few weeks to administer Wave 2 of this panel. For this tutorial, I will administer it less than 24 hours after Wave 1. Let s get to it. 6 R can be downloaded here Materials: kyleadropp.com/panels

11 3. Wave 2 This section will provide instructions for administering Wave 2. I typically use MySQL databases for this wave, but this tutorial does not use any servers and should be accessible to anyone with basic knowledge of R and Qualtrics. See the Appendix (or message me) for methods for administering Wave 2 treatments using MySQL Upload Wave 2. Enter Qualtrics, and Upload the survey panel tutorial wave2.qsf in the same way you uploaded the first survey at the beginning of the tutorial (i.e., edit survey, advanced options, import survey). This survey contains the same questions and question blocks as Wave Branch Logic and Wave 2 Treatment. On the Edit Survey tab, select Survey Flow. Scroll down to see the new branch logic. Open the file wave2 assignment.csv, which is sorted by Wave 2 treatment assignment ( randcountry2 ) and also includes the AMT Worker ID. I have copied the AMT Worker IDs associated with each unique treatment into the logic of the Survey Flow. 11 Materials: kyleadropp.com/panels

12 We have assigned embedded data variables to each of the four possible Wave 2 treatment assignments. Now, when respondents enter the survey we will link their MID to their Wave 2 treatment group. Next, we use branching logic to assign respondents to an appropriate treatment Managing Wave 2 from R Install MTurkR. library(devtools) ## install.packages( MTurkR ) if you don t have package installed require("mturkr") Enter Amazon Mechanical Turk credentials. You must provide two unique identifiers, your AWS Access Key ID and your AWS Secret Access Key, to control AMT from the command line. ## Sign into Amazon Mechanical Turk with AWS Access Key ID ( xxxx ) ## and AWS Secret Access Key ( yyyy ) credentials(c("xxxx","yyyy")) AccountBalance() Here is how you acquire your AWS Access Key ID. First, go to select Security Credentials in the top right, sign into your account, click Continue to Security 12 Materials: kyleadropp.com/panels

13 Credentials, click Access Keys tab, and you will see your Access Key ID. Copy this into the xxxx portion of the credentials command. This is first value. Now, to find your AWS Secret Access Key, select Security Credentials in the yellow box below your Access Key ID, select Access Credentials, and click Show under Secret Access Key. This is the AWS Secret Access Key. Copy this into the yyyy portion of the credentials command Invite Wave 1 respondents to Wave 2. Now, you will contact workers to invite them to the Wave 2 panel. Use the ContactWorker() command to send messages to each respondent, specify a bonus, and add a subject to the . A typical subject line is Thanks for completing my HIT!, a typical body is I will pay a $0.20 bonus if you complete a brief follow-up study. The survey can be completed at xxxxx Please note the MID is the unique Amazon Mechanical Turk ID for the respondent. Here is the appropriate R code (with the same data frame df1). See the Distribute Survey link in Qualtrics to provide to correct Qualtrics survey link. Use the paste command to append an MID identifier to each respondent who responds to your invitation. df0 = read.csv(" df = read.csv(" names(df)=names(df0) df = df[as.character(df$mid)!="",] a <- "Complete a brief question follow-up survey for $0.20 bonus!" b <- paste("the survey can be completed at ", " df$mid,sep="") c = as.character(df$mid) d <- ContactWorker(subjects=a,msgs=b,workers=c) Your screen will look like this when you run the command. 13 Materials: kyleadropp.com/panels

14 Respondents now will take Wave 2 of the survey. Download the results from the View Results tab after a sufficient number of individuals have responded. Typically, I have received Wave 2 response rates of 70%+. The Wave 2 data file is saved as data wave2.csv Compensating Respondents. After the worker has submitted the job, or HIT, you must pay him or her using the GrantBonus() command. First, in the manage tab in AMT, select Results on your latest batch of HITs, click Download, then click here on the Download Batch Results page to download an individual level file with completed HITs. The file includes the AssignmentId column, along with a WorkerId column. Merge this dataset with the Wave 2 results file, on the MID variable to determine which respondents successfully entered Wave 2. Then, using AMT Worker IDs and Assignment Ids, send a bonus to the workers. wave1amt = read.csv(" wave2data = read.csv(" merge1 = merge(wave2data,wave1amt,by.y="workerid",by.x="mid") dim(merge1) 14 Materials: kyleadropp.com/panels

15 a1 <- as.character(merge1$mid) b1 <- as.character(merge1$assignmentid) c1 <- ".20" d1 <- "Thanks for your great work on my HIT! I really appreciate it!" ##GrantBonus(workers=a1,assignments=b1,amounts=c1,reasons=d1) 4. Data Analysis You now have two separate.csvs containing completed responses to Wave 1, and a second file containing responses to Wave 2. Merge the two files based on the respondent s Amazon Mechanical Turk ID and then start your between-subjects and within-subjects analysis. Here is code to merge the files and examine the treatment assignments. A figure below demonstrates that the Wave 1 treatment assignments differ from Wave 2 assignments. df0 = read.csv(" wave1data = read.csv(" names(wave1data)=names(df0) wave2data = read.csv(" dim(wave1data) dim(wave2data) names(wave1data)=paste(names(wave1data),"_w1",sep="") names(wave2data)=paste(names(wave2data),"_w2",sep="") waves12 = merge(wave1data,wave2data,by.x="mid_w1",by.y="mid_w2") cbind(waves12$headline_w1[11:20],waves12$headline_w2[11:20]) 15 Materials: kyleadropp.com/panels

16 16 Materials: kyleadropp.com/panels