The Promises and Perils of Pushing Respondents to the Web in Mixed-Mode Surveys

Transcription

1 The Promises and Perils of Pushing Respondents to the Web in Mixed-Mode Surveys For 9th Colloque Francophone sur les sondages Gatineau, Quebec, Canada 11 au 14 Octobre 2016 By Don A. Dillman, Regents Professor Department of Sociology and the Social and Economic Sciences Research Center Washington State University, Pullman, WA

2 A brief look to the past In 1989 an iconic American movie, Field of Dreams, was created around the idea that if a baseball field were to be built in a rural Iowa corn field, They (the players and people) will come. In the late 1990 s a similar idea developed about conducting surveys by with responses sent over the Internet. Hopefully, thepeople would not only come, but the cost of surveying would decline dramatically, making it a surveyor s field of dreams. This reality has yet to be fully realized. 2

3 Three parts to this presentation Part I. The decline of telephone and difficulty with finding alternatives. Part II. A decade of research aimed at developing an effective web-push methodology. Part III. Specific challenges associated with the promises and perils of web-push surveying. 3

4 My specific focus My focus in this talk is on probability sampling of the general public. In this context I am also focused on simultaneously reducing four types of error: coverage, sampling, nonresponse and measurement. By web-push I am referring to initially asking people to respond only over the web, while reserving alternative response modes until later in the data collection process. Mixed-mode refers to using more than one survey mode as contact and/or response mode. 4

5 Part I. The Decline of Telephone and difficulty with finding alternatives 5

6 Around 2005 it became clear that the voice telephone era of RDD household surveys was coming to an end The presenting problems for RDD surveys were: Low response rates. Need to combine individual (cell) and household (land) lines. Need for brevity. Need to devote questions to correcting for area code transportability, ownership of cell/landline access, not driving a vehicle, age of respondent. People answered phone only once, so that it is becoming a one chance to persuade (in 5 seconds or less) methodology However, cultural change is the biggest problem; the telephone is no longer used for ordinary conversations. Most telephone communication is asynchronous; texts and s. 6

7 The hoped for contact/web response replacement, is not yet working well. Coverage for household populations remains limited (<80%), some on slow connections. Individual internet skills were limited among some adults, especially the less educated, older and lower income individuals. No sample frame for household selection comparable to telephone RDD. only contact produces response rates similar to phone. 7

8 The hoped for contact web response replacement, is not yet working well (2). Lack of TRUST on multiple levels; sponsorship, misrepresentation of purpose, threat of malware, and consequences of making a mistake. We are moving from age of desktops, to laptops, (and now) to devices, which are not optimally designed for answering surveys screen size affects when and how they get used. The purse and pocket problem combined with the smartphone response situation of people being on the move from one task to another makes getting lengthy and thoughtful survey responses very difficult. 8

9 I am uncomfortable with some of the assessments and proposed solutions Response rates no longer matter (Higher the response, the better our chance of going after non-responders) All response modes are facing declining response rates (less true for mail than telephone or the Internet) Repair the telephone (Too many problems: coverage, brevity, extra questions, minimal opportunity to persuade, CULTURE moving away from voice exchanges) Give households the Internet and rely on web-only panels (Too many people don t trust or are uncomfortable with computers and internet; results are biased) Give people a choice of modes (Most choose mail) 9

10 Postal addresses have emerged as sample frame with most complete household coverage Early focus (2007) was on drawing area-probability samples for personal interviewing. I am struck by the high coverage rates for households 98 % in the U.S. Postal Service Delivery Sequence File available as addresses without names. If this could work for personal interview samples, why not mail surveys. Reason for my interest in mail surveys was: I could deliver incentives with the survey request (much more effective that post payments) I could deliver follow-up contacts in different forms to offer additional arguments for responding. Research done in the 90 s at the Census Bureau convinced me that we didn t need names for mail methods to be effective. I had not seen evidence of a precipitous decline in mail response rates as happened for telephone. I could try to push people to the web instead of asking for a paper response. 10

11 In order to replace telephone with a webpush methodology We needed a better sample frame than random addresses. Postal mail appeared to be the only realistic choice. Using postal addresses would require mixing survey modes: a different contact mode than response mode. And might involve obtaining responses from more than one mode (e.g. mail or telephone) Mixing modes could introduce measurement issues Potential cost savings provided a powerful motivation for finding ways to make this work. Acceptance by the survey methodology community could be challenge because of losing interviewer assistance, our standard way of eliciting answers to survey questions. 11

12 Part II. A decade of research aimed at developing an effective webpush methodology 12

13 The Prototype (Lewiston-Clarkston) Experiment Address-based sample. Push-to-web with mail contact (send mail response option in third contact). Undo old mail strategies, e.g. name personalization, envelopes, use of postcard. Token cash incentives with response request Unified mode construction to reduce measurement differences. Tailor to population with graphic design and color. Reasonable length: At least 20 minutes of questions My background model was, What I thought could be applied to the American Community Survey 13

14 Conceptually, I was trying to undo earlier mail procedures ( ) and use the visual design and mixed-mode concepts introduced in (2000 and 2009) versions of book. Each revision was substantial (see% below). New concepts are coming at us faster and faster ~95% 2009 ~75% 2014 ~65% 14

15 2007 Lewiston-Clarkston Study 51 numbered questions, 90 responses, 10 pages (paper) a 20 minute (if it were telephone) conversation Four contacts Pre-notice letter. Questionnaire (or web request). Thank-you post card. Replacement questionnaire (adjusted by treatment). $5 token cash incentive included with mail questionnaire or web contact (Later studies would change number and nature of contacts) 15

16 We compared four treatments 1. Mail preference with web mention: Send mail questionnaire and mention web with initial request 2. Push-to-mail: Send mail questionnaire but withhold mention of web for about two weeks 3. Push-to-web: Web invitation with no mail questionnaire, but explain that mail questionnaire will be sent in about two weeks 4. Equal preference: It is your choice! 16

17 Larger plain envelopes (even for web-only request) to get them opened Normal business stationery more likely to be ignored. Used a return label showing the photo from survey cover and the survey title to increase familiarity. Dillman 17

18 Correspondence tailored to city address All letters used WSU stationary (legitimation). $5 affixed to letter (to get it read) Photo of questionnaire cover used to tie different elements together (interest). 18

19 A picture of the Lewiston and Clarkston region identified the survey area (confluence of the Snake and Clearwater Rivers) 19

20 Additional pictures identified the survey area Photos taken of local landmarks, artwork, and symbols to make survey recognizable and visually attractive. 20

21 Mail Questionnaire tailored with broad survey topic, who should respond, and back-page pictures Use of tailored images to help connect respondents to survey and to place an emphasis on study area instead of on survey source. 90 responses requested 21

22 Web questionnaire was similarly tailored Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images. 22

23 Web survey followed unified mode construction principles to link modes Question 2 Similar design format to paper survey, and use of familiar image in upper left-hand corner of the screen. Emphasized respondents over sponsors (using other communications to do that). 23

24 Second example of unifying web presentation with mail presentation Question 13 The small image would change every ten questions. Trying to show all of the same images as the paper survey, as well as keep interest and familiarity. 24

25 Mail (on left) and web (on right) unified to control measurement 25

26 Initial withholding of mail drove 41% to the web; paper follow-up added 14%. Offering choice drove ~80% to mail. Treatments Web (%) Paper(%) Total (%) Mail preference with web mention Push-to-Mail (web in third contact) Push-to-web Mail questionnaire sent in 3rd of 4 contacts Equal preference (choice)

27 High response rates are desirable but we need also to focus on nonresponse error It does not help much to improve response rates if our respondents are different from non-respondents on variables important to the study objectives Thus, we need to compare respondent characteristics on web vs. mail within the different treatment groups. If respondents are different from non-respondents in ways that affect results of the data analysis, then we have a significant problem 27

28 In the push-to-web treatment, web and mail respondents were quite different * * * * * * * * p.05 28

29 But the complete push-to-web group was quite similar to the complete push-to-mail treatment Push-to-web * Push-to-mail * p.05 29

30 Initial success encouraged us to do four more large-scale tests to evaluate the webpush methodology Lewiston, ID-Clarkston, WA Mail Test 2005 Lewiston, ID-Clarkston, WA Survey 2007 Washington Community Survey 2008 Washington Economic Survey 2009 WA, PA, AL Tri-state Electricity Survey2011 WA and NE Water Management Survey2012 Needed multiple topics and populations Research goal was to refine through subtraction and addition experiments, e.g. token cash incentives, other state populations, questionnaire design, respondent selection, location of sponsor, etc.) ~ 28 additional experiment treatments, with successful panels being carried forward. For summarized detail, see Chapter 11 of Dillman, D., Smyth, J.D., Christian, L.M Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition. (John Wiley, Hoboken, NJ) 30

31 Tests 2-5; Changes in burden, topic, and procedures (e.g. no pre-notice, no postcard reminder) Questionnaire minutes to complete. (equivalent to ten 8 ½ x 11 pages of paper. Up to 140 individual questions General Implementation Protocol : specifics varied somewhat across studies. Week 1: Postal request that includes $4- $5 token cash incentive with request to respond by web. Week 2: A thank you/reminder letter (not postcard). Week 5: Another letter request with a mail questionnaire and, in some cases, an additional smaller token cash incentive Week 6: A final thankyou/reminder. 31

32 2008 Washington Community Survey (WCS): first statewide test Example of the mail version: 32

33 A $5 cash incentive (WCS) more effective for web-push than paper-only (+13.3*) 39, ,3 (+17.9*) , $5 Internet Internet w/o $5 $5 Mail Mail w/o $5 Mail * p.05 33

34 Push-to-web had lower total response, but 2/3 of responses came over the web. For push-to-mail, a late web request had very small effect % 3, % 40 15, ,9 57% 56, , $5 Mail-only; 0 $5 Push-to-web w/card $5 Push-to-mail $5 Mail-only 34

35 Introducing a paper questionnaire in third contact brought in different kinds of respondents than did the initial push-to-web effort. (Messer and Dillman, 2011) * * * * Education (HS or less) Age (65+) * * # in HH (2 or less) Married (% Yes) Employed (% Income Yes) ($25/year or less) Web Mail follow-up * p.05 35

36 The combined push-to-web group (web plus mail respondents) was demographically similar to the mail-only treatment Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes) Income ($25/year or less) Push-to-web Mail-only 36

37 Complete push-to-web group was more representative than the web-only respondents(comparison to U.S. Census Bureau s American Community Survey) Education (HS or less)children in HH # in HH (2 or less) Married (% Yes) Employed (% Yes) Income ($25/year or less) Web-only Push-to-web ACS 37

38 2009 statewide economic survey, tailored to state with map and pictures 11 X 8.5 booklet with personalized images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey. (Messer and Dillman, 2012) 38

39 2011 Three state Electricity Survey, with tailored covers Examples of the mail covers: 39

40 2012 water management survey: tailored to state by pictures and sponsorship Examples of the mail covers: 40

41 Response rates for 10 statewide web-push (mean, 43%, with 62% over the web) vs. mail out/mail back (mean, 53%) surveys, (Dillman, Smyth and Christian, 2014, Chapter 11). 41

42 Does mail item non-response negate gains in response rates? If mail higher item non-response rates then perhaps that cancels out the benefits of gaining the additional responses in web-push designs. Thus, it was important to evaluate this. (See Messer, Edwards and Dillman, Survey Practice, 2012). 42

43 Web vs. mail item non-response in the pushto-web groups for LCS, WCS, and WES Item nonresponse rates lower for web , ,9 6,2 6, ,7 2, LCS 2008 WCS 2009 WES 43

44 BUT, push-to-web and mail-only groups have similar overall item nonresponse rates; Late mail respondents in push-to-web groups are probably less capable respondents(older with less education). Push-toweb ,0 8,1 6 5, , LCS 4,2 4, WCS 2009 WES 44

45 Summary of Major findings Postal-only response = 53% (38% to 71%) Push-to-web response = 43%(31% to 55%). 3. Offering initial choice pushes most people (~80%) to mail. 4. Web-push produces about 60% of responses over the web. 5. Token cash incentives with request significantly improves web and overall response rates. 45

46 Summary of Major findings (2) 1. Significant differences exist between web and mail responders (education, age, income, marital status). 2. Web+mail treatment respondents are quite similar to all mail-only treatment respondents. 3. Item non-response to mail is a less of a problem than we had anticipated when visual design principles are applied. 4. Response decreases when surveying in distant states. 5. Web-push costs slightly more per respondent than mail-only because of lower response rates. 46

47 Web-push methods being used for many national surveys 2015 Japanese Census. American Community Survey (3.5million households per year) Screen and collect data for U.S. child education survey. Screen and collect household data for U.S. child health survey Australian Census 2020 U.S. Census 47

48 Part III. Additional challenges associated with the promises and perils of web-push surveys 48

49 Promises and Perils of Web-Push In the long run, we need to push respondents to the web, but we need to do it with eyes wide-open. There are significant promises to be realized. There are also some perils in going this direction that we need to contemplate 49

50 12 ways Web-push exhibits promise 1 Address-based sampling provides superior coverage. 2 Response rates to web-push are dramatically higher than RDD telephone. 3 Cost reductions seem likely. 4 Quicker responses from most people seems likely. 5 A better fit with society s do-it-myself culture. 50

51 Web-push exhibits promise (2) 6 Reduction in socially desirable answers (vs. interviews) 7 Reduction in extremeness of scalar responses. 8 More effective branching (vs. mail) 9 Initial mail contact increases feelings of trust. 10 Pre-incentives improve likelihood of response. 11 Two-phase data collection may be collapsed into one phase (screen, plus longer questionnaire). 12 Better fit with organizational preferences and pressures. 51

52 Web-push perils calling for research and innovation 1 How to overcome irritation of not providing immediate choice. 2 Pre-incentives needed for getting people to transfer across modes (i.e. mail contact to web response). 3 Decisions on communication access have switched from organizational control to individual preference and behavior. 4 Pocket/Purse devices are dominant (and perhaps only) interface some people have with computers. 5 Web response is not sufficient for most surveys bias towards higher education, income and younger households with kids. 6 Mail follow-up to get balance faces constraints branching plus need for unified mode construction. 52

53 Additional web-push perils calling for attention (2) 7 Mixing of response modes are likely to create measurement differences, if allowed to ask questions in different ways. 8 Switching to mail in 3rd or 4th contact is not being handled well, e.g. still emphasizing web, minimal incentive and/or follow-up. 9 Not developing coordinated or mutually supportive communications focused on the process of responding 53

54 Additional web-push perils calling for attention (3) 10 Not treating mail and web as help-mates. Respondents can use mail to think through answers before entering them on web, but different construction makes that difficult. 11 Are we on a slippery slope because of do it yourself software? Let s use only people contactable by Let s only obtain web responses Increase sample size because of low response Treating large sample sizes as if they have sampling error. Use of weighting to create data for underrepresented categories. A catch, release, and catch-again mentality develops. 54

55 Some additional steps Introducing more effective mail follow-up to the initial webpush strategy, e.g. multiple contacts with additional incentives to get those communications read. Dealing with unified design between mail follow-up and smartphones that are requiring redesign of laptop/desktop surveys. Dealing with branching issues in visual design of paper surveys. Is there a role for telephone follow-up despite social desirability and aural vs. visual design effects? There is a lot more that can and needs to be done. 55

56 Selected references Smyth, J.D., Dillman, D.A., Christian, L.M., & O Neill, A. (2010). Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century. American Behavioral Scientist 53: Dillman, D.A., Smyth, J.D., Christian, L.M Internet, Phone, Mail and Mixed-Mode Surveys; The Tailored Design Method 4th edition. (John Wiley Co Hoboken, NJ) Messer, Benjamin L. and Don A. Dillman Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures. Public Opinion Quarterly 75(3): Millar, Morgan M. and Don A. Dillman Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly 75 (2): Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3):

57 Selected references, page 2 6. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012). Determinants of Web & Mail Item Nonresponse in Address-Based Samples of the General Public. Survey Practice, April: Messer, Benjamin L Pushing households to the web: Results from Web+Mail experiments using address based samples of the general public and mail contact procedures. Ph.D. Dissertation. Washington State University, Pullman. 8. Edwards, Michelle L Measuring Public Perceptions of Water Governance in Nebraska and Washington. Ph.D. Dissertation. Washington State University, Pullman. 57

58 Selected references page 3 Dillman, Don A. and Michelle L. Edwards. In Press. Chapter 17. Designing a Mixed-Mode Survey. Sage Handbook of Survey Methodology. Sage Publications Wolf, Joye, Smith and Fu. Thousand Oaks. CA Dillman, Don A., Feng Hao, Morgan M. Millar. In Press. Chapter 15. Improving the Effectiveness of Online Data Collection by Mixing Survey Modes. In Fielding, Nigel, Raymond M. Lee and Grant Blank. The Sage handbook of Online Research Methods, 2nd edition. Sage Publications, London. Harter, Rachel, Battaglia, Michael P., Buskirk, Trent D., Dillman, Don A., English, Ned, Mansour Fahimi, Frankel, Martin R., Kennel, Timothy, McMichael, Joseph, McPhee, Cameron Brook, Montaquila, Jill, Yancey, Tracie, Zukerberg, Andrew L Addressbase Sampling. American Association for Public Opinion Research Task Force Report 133 pp. t_1_7_16_clean-copy-final.pdf 58

59 Thank you! For additional information contact: 133 Wilson-Short Hall Washington State University Pullman, WA