Web+Mail as a Mixed-Mode Solution to General Public Survey Challenges in the United States

Transcription

1 Web+Mail as a Mixed-Mode Solution to General Public Survey Challenges in the United States Don A. Dillman and Benjamin Messer* Social & Economic Sciences Research Center Washington State University Pullman, WA *Regents Professor and Graduate Research Assistant, Department of Sociology and the Social and Economic Sciences Research Center (SESRC) at Washington State University (WSU). For additional information contact dillman@wsu.edu Support for this research was provided by USDA-National Agricultural Statistical Service and the NSF Center for Science and Engineering Statistics, under a Cooperative Agreement to the SESRC at WSU. Additional support was provided by the SESRC. Copyright: Don A. Dillman and Benjamin Messer,

2 Background and Purpose The U.S. is facing enormous challenges in conducting general public surveys. Telephone coverage and response rates are in sharp decline. The web lacks adequate coverage for the general public. Mail contact is now our best sample frame In this presentation we will explain why and report on several research studies we have conducted in order to find solutions 2

3 In the U.S., landline telephone coverage has been declining RDD response rates for surveys of general public households are now around 10%! (Pew Research Center, 2012, AAPOR) 3

4 Landlines per 100 US residents have been declining Landline and cell phones per 100 people Source: FCC, 2010 Landline while cell phone penetration has been increasing. Combining them for RDD surveys is possible, but the geography (residential location) is less knowable. Dillman and Messer Cell 4

5 Household Internet access has been increasing... PEW, 2010 May be faster and cheaper than other survey modes but lack of regular use means having reliable connections with only about 2/3 of all U.S. households.5

6 But the Internet also has significant limitations Sampling & Contacting households: No address frame available for general public A significant portion (22-33%) of households lacks Internet access or does not use it regularly How people access Internet discourages responding to surveys 25% of smartphone owners use Internet mostly from their phone (PEW, 2011) 32% of smartphone owners do not have Internet access from another device (e.g. laptop, desktop, tablet) Nonresponse bias: Demographic digital divide in the US in regards to Internet access and literacy. % with broadband home access. 100 Broadband in HH

7 The U.S. Postal Service maintains an address-based sample (ABS) frame with high coverage of US households. Using ABS and postal mail contacts may overcome coverage and contact problems with telephone and Internet. XYZ Survey Center SURVEY REQUEST 7

8 The Web+Mail Design Mail sampled residential postal addresses a web survey request, push respondents to use the Internet to respond. XYZ Survey Center WEB SURVEY REQUEST 8

9 About 2 weeks after the web request, non-respondents are mailed a paper XYZ Survey Center questionnaire PAPER SURVEY REQUEST XYZ Survey Center 9

10 During the past five years our research team has conducted several studies to test web+mail methods and procedures as a potential alternative to RDD 1) 2007 Lewiston/Clarkston Quality of Life Survey (LCS) Smyth, Dillman, Christian, & O Neill, American Behavioral Scientist. 2&3) 2008 Washington Community Survey (WCS) & 2009 Washington Economic Survey (WES) Messer & Dillman, Technical Report. Messer & Dillman, Public Opinion Quarterly. 4) 2011 Tri-State Electricity Survey (TSES) in Washington, Pennsylvania, & Alabama Messer Dissertation in process. 10

11 Questions Answered by Study 1: 2007 LCS Which mixed-mode design obtains the most respondents via web? The most overall? Web+Mail: 1) Prenotice, 2) $5 Web request, 3) Reminder, 4) Mail follow-up Mail+Web: 1) Prenotice, 2) $5 Mail request, 3) Reminder, 4) Web follow-up Web-Mail Choice: 1) Prenotice, 2) Web or mail request, 3) Reminder, 4) Web or mail request Are web respondents different from mail respondents? 11

12 Results from 2007 LCS suggest Web+Mail (55% RR) most effective at obtaining web responses (3/4s via web) Mail+Web the least effective (1% via web) Mail+Web (71% RR) obtains higher overall response rate Web+Mail obtained the lowest Web-Mail Choice (63%) falls in between (1/4 via web) Web+Mail Web-Mail Choice Mail+Web Web Mail 12

13 Web+Mail respondents who respond by each survey mode are quite different types of people * * * * * * Female Age (65+) Education (College degree) Married (% Yes) Employed (% Yes) Income ($25/year or less) * p.05 Web Mail follow-up 13

14 However, web+mail respondents are similar to mail+web respondents * * Female Age (65+) Education (College degree) Married (% Yes) Employed (% Yes) Income ($25/year or less) * p.05 Web+Mail Mail+Web 14

15 2007 LCS Limitations Conducted in a local rural region: Not sure if methods will be effective in more urban, diverse population. Unable to test respondent representativeness. Used but did not test a $5 incentive sent with the web request. 15

16 Questions answer by Study 2: 2008 WCS (Washington Community Survey) Can Web+Mail be used effectively in a more urban and diverse statewide population? What are the effects of the $5 incentive on web & mail response? 16

17 Similar response rate trends as in the 2007 LCS; we can push 2/3 of responses to web, but lose overall response (46% vs. 57%) $5 Web+Mail w/card $5 Mail+Web $5 Mail-only Web Mail 17

18 The $5 incentive was very effective at increasing response rates, especially for web+mail * * $5 Internet Internet w/o $5 Web+Mail 0 $5 Mail Mail w/o $5 * p Mail

19 Demographic trends similar to 2007 LCS: Web respondents quite different than mail follow-up respondents * Education (HS or less) * Age (65+) * # in HH (2 or less) * * Married (% Yes) Employed (% Yes) * Income ($25/year or less) * p.05 Web Mail follow-up 19

20 But combined Web+Mail respondents demographically similar to mail-only respondents Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes) Income ($25/year or less) Web+Mail Mail-only 20

21 Web+Mail respondents also more representative of Washington s population (vs. web alone) as measured by U.S. Census American Community Survey (ACS) Education (HS or less) Children in HH # in HH (2 or less) Married (% Yes) Employed (% Yes) Income ($25/year or less) Web-only Web+Mail ACS 21

22 2008 WCS Limitations Did not test offering non-respondents a second incentive in the third contact. Did not use a special mail contact (i.e. Priority Mail). 22

23 Questions answered from Study 3: 2009 WES (Washington Economic Survey) Will sending the mail follow-up with a second $5 incentive and in a Priority Mail (PM) envelope increase response rates? Can we push more people to the web? 23

24 PM+$5 increased response rates, particularly for mail-only. (However, effect was due entirely to incentive and not the Priority Mail.) * * Web+Mail PM+$5 Web+Mail Mail-only PM+$5 Mail-only * p.05 Web Mail 24

25 2008 WCS & 2009 WES Limitations Conducted for local statewide population in same state as the sponsor. It also had higher than average Internet penetration and levels of SES (vs. U.S.). Web+Mail design informed respondents that mail would be sent later and provided respondents with the 2 nd incentive with the mail follow-up less web encouragement 25

26 Questions answer from Study 4: 2011 TSES (Tri-State Electricity Survey) Can web+mail be used effectively in 1) More distant states? 2) States with lower SES and Internet access? Is 2web+mail more effective than web+mail methods? Withholding the mention of the mail follow-up and offering web with the 2 nd incentive more web encouragement 26

27 Web+Mail less effective in more distant states, and especially in states with lower SES & Internet access Alabama: lower Internet penetration and SES, also distant Pennsylvania: demographically similar but distant Washington: control population Web+Mail Mail-only Web+Mail Mail-only Web+Mail Mail-only Alabama Pennsylvania Washington Web Mail 27

28 2Web+Mail Design more effective in more distant state (no difference in WA) Web+Mail: 1) $5 Web request, 2) reminder, 3) $2 Mail follow-up, 4) Reminder 2Web+Mail: 1) $5 Web request, 2) Reminder, 3) $2 Web request, 4) Mail follow-up Web+Mail Web+Mail 2Web+Mail Web+Mail Washington Pennsylvania Web Mail Follow-up 28

29 Tri-State Conclusions A potential backlash? A very small number of respondents called to request a paper questionnaire WA: 13 PA: 7 2web+mail may be the best design for increasing web response rates, particularly in more distant populations. In WA, the web+mail design performed even better than in the 2008 & 2009 statewide studies (i.e. WCS/WES). However, in PA and AL, only about 1/3 of web+mail respondents chose web, and total web+mail response rates are significantly lower than in WA. 29

30 Web vs. Mail Data Quality Item nonresponse rates lower for web LCS 2008 WCS 2009 WES Web Mail Follow-up Messer, Edwards, & Dillman, Survey Practice. 30

31 But, Web+Mail and Mail-only item nonresponse rates are similar LCS 2008 WCS 2009 WES Web Mail-only 31

32 Web+Mail Limitations Web+mail may not be faster than mail-only: 2008 WCS response times /27 6/29 7/01 7/03 7/05 7/07 7/09 7/11 7/13 7/15 7/17 7/19 7/21 7/23 7/25 7/27 7/29 7/31 8/02 8/04 8/06 8/08 8/11 8/13 8/15 8/18 8/20 1) $5 Mail-only 2) $5 Mail+Web 3) Mail+Web 4) $5 Web+Mail 5) $5 Web+Mail WC 6) Web+Mail WC 32

33 2009 WES Response Times ) $5 Web+Mail 11) $5 Web+Mail PM 12) $5 Web+Mail PM+$5 13) $5 Mail-only 14) $5 Mail-only PM 15) $5 Mail-only PM+$5 33

34 Or less expensive than mail-only. Average WCS & WES costs/respondent $45.00 $40.00 $35.00 $30.00 $25.00 $20.00 $15.00 $10.00 $5.00 $0.00 $39.05 $30.26 Cost/Respondent Web+Mail Mail-only 34

35 Conclusions In a world of declining telephone coverage and response, a web+mail methodology could provide an alternative: Reasonably high response rates in regional and statewide surveys Majority respond via web Demographics characteristics similar to using mail alone However, web+mail is also limited: More expensive than mail-only Faster than mail-only for a short time May not be as effective in more distant populations Mail may seem outdated or archaic but it is even more effective than web+mail in many respects Should be used to deliver incentives, etc. 35

36 Appendix The questionnaires used for each of these studies were 12 page booklets, requiring minutes to answer. All paper questionnaires were printed in color with graphics tailored to the state or region being surveyed. Similar graphics were used on the web pages to enhance recognition and connectivity to mailings. 36

37 Appendix: 2007 Mail LCS Questionnaire Mail version 37: 37

38 Appendix: 2007 LCS Web Questionnaire Web version: 38

39 Appendix: 2008 WCS Mail Questionnaire Mail version: 39

40 Appendix: 2008 WCS Web Questionnaire Web version: 40

41 Appendix: 2009 WES Mail Questionnaire Mail version: 41

42 Appendix: 2009 WES Web Questionnaire Web version: 42

43 Appendix: 2011 TSES Mail Questionnaires Mail Covers: 43

44 Citations LANDLINE/CELL PHONE: Blumberg, S.J. & Luke, J.V. (2008). U.S. Households with Wireless or Landline Telephone Service. National Center for Health Statistics (NCHS). Federal Communications Commission (FCC). (2011). Trends in Telephone Service. INTERNET ACCESS/SMARTPHONE: Raine, L. (2010). Internet, broadband, & cell phone statistics. Pew Internet & American Life Project (PEW). Smith, A. (2011). Smartphone Adoption & Usage. Pew Internet & American Life Project (PEW). MIXED MODE SURVEY METHODS Dillman, D.A., Smyth, J., & Christian, L.M. (2009). Internet, Mail, & Mixed Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley & Sons. De Leeuw, E.D., Hox, J.J., & Dillman, D.A. (2010). International Handbook of Survey Methodology. New York, NY: Taylor & Francis Group. Marsden, P.V. & Wright, J.D., eds. (2010). Handbook of Survey Research. Bingley, UK: Emerald Press. 44

45 Citations, cont. WEB+MAIL METHODS Smyth, J.D., Dillman, D.A., Christian, L.M., & O Neill, A. (2010). Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century. American Behavioral Scientist, 53: Messer, B.L. & Dillman, D.A. (2010). Using Address Based Sampling to Survey the General Public by Mail vs. 'Web plus Mail. Technical Report 10-13, Social and Economic Sciences Research Center, Pullman, WA: Washington State University. Messer, Benjamin L. and Don A. Dillman Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures. Public Opinion Quarterly 75(3): Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012). Determinants of Web & Mail Item Nonresponse in Address-Based Samples of the General Public. Survey Practice, April: Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012). Determinants of Item Nonresponse to Web and Mail Respondents in Three Address-Based Mixed-Mode Surveys of the General Public. Technical Report , Social and Economic Sciences Research Center, Pullman, WA: Washington State University. Messer, Benjamin L. In Process. Pushing households to the web: Results from Web+Mail experiments using address based samples of the general public and mail contact procedures. Dissertation. 45

46 For Additional Information For additional information on these studies contact Don Dillman at: Web page information is at: Postal address: Don A. Dillman, Ph.D. 133 Wilson Hall Washington State University Pullman, WA United States of America 46