Text Interviews on Mobile Devices. Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Andrew L. Hupp, H. Yanna Yan

Transcription

1 Text Interviews on Mobile Devices Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Andrew L. Hupp, H. Yanna Yan

2 Acknowledgment NSF grants SES and SES Awarded to Conrad and Schober, respectively 2

3 Communication is changing Increasingly, people communicate when, where and in ways that are convenient and practical for the task: talking in real time (with or without video) exchanging recorded spoken messages exchanging written messages (through or text messaging, or on social media sites) in more than one mode at a time (e.g., with images, audio or video in text messages) Smart phones make multiple modes available on single, handheld device 3

4 Texting is ubiquitous Texting is primary method of daily communication in many parts of the world The most basic mobile phones allow text messaging In US, 169 billion text messages sent per month (CTIA, 2015) Globally, 561 billion text messages sent per month (CTIA and Nielsen, 2015) Texting is worth exploring as survey mode for recruiting into an online or telephone survey (see Couper, Antoun & Mavletova, this session) for collecting data 4

5 What do we mean by texting? Sending and receiving character-based messages limited to 160 characters in SMS, unlimited in internetbased apps created via alphanumeric vs. virtual (alphabetic) keypads displayed as individual or threaded messages that alert user via tone or vibration or visual notification that may require sender and recipient to subscribe to same service (e.g., WhatsApp)? 5

6 Property Voice Text Synchrony Fully synchronous Less or asynchronous Medium Auditory Visual Language Spoken/heard Written/read Conversational structure Turn-by-turn, with potential for simultaneous speech Turn-by-turn, rarely but possibly out-ofsequence Persistence of turn No Yes Persistence of entire conversation No Yes, threaded Nonverbal cues of emotional state and intentions Always present: speech always has pitch and timing Intermittent evidence from content of texts; no additional evidence between texts Character of multitasking Often simultaneous, especially when hands free, unless other task involves talking Switching often required between texting and other tasks Impact of environmental conditions Potential (auditory) interference from ambient noise Potential (visual) interference from visual glare Impact of nearby others Others may hear speaker s side of conversation; potential audio interference from others talk Others unlikely to see conversation on screen, though possible Required connection to network (cellular or wifi) Must be continuous Can be intermittent 6

7 Contact via Text Message Text messages can be used to invite sample members Including pre-notification and reminder For online (both desktop and mobile) and mixedmode surveys 7

8 Example: Text pre-notification Holland et al. (2014) prenotified ~2500 US soldiers about upcoming online survey using one of four strategies 1. No pre-notifcation 2. Letter only 3. Text only 4. Text and Letter Pre-notification followed by invitation Results: Text (alone or with letter) led to: 1. Highest completion rate 2. Fastest completion before first reminder 3. Diminished effect once the researchers reminded respondents (by and phone) 8

9 Persistence of Text Invitations Message remains in text inbox (feature phones) or messages app (smart phones) until deleted Eliminates non-contact which, in voice, occurs when potential Rs do not answer phone Allows potential Rs to respond when convenient 9

10 Text Interviews: Coverage Error If some mobile owners do not text, and so cannot be reached for a text interview, estimates derived from these interviews may be biased Unknown to what extent text users represent all mobile users 10

11 Texting and Coverage Error (2) Full sample of mobile owners (n = 1127) A Mobile owners who text (n = 962) B Coverage deviation A-B % (SE) % (SE) Est.(SE) Age (>34 years old) 62.4% (1.4) 57.0% (1.6) 5.4%* (0.4) Sex (=Male) 56.2% (1.5) 55.3% (1.6) 0.9% (0.6) Education (=College grad) 36.4% (1.4) 38.0% (1.6) 1.7%* (0.6) Race (=White) 70.5% (1.4) 70.1% (1.6) 0.5% (0.5) Ethnicity (=Hispanic) 16.1% (1.1) 16.4% (1.2) 0.4% (0.4) Income (=$30K or more) 71.8% (1.3) 72.9% (1.4) 1.1% (0.6) Source: Pew s Spring Tracking Survey conducted from April-May

12 Texting and Coverage Error (3) As proportion of texters among mobile owners increases, potential coverage bias will be reduced Texting rates among US mobile owners, from 2007 to 2013, increased from 58% to 81% (Pew, 2013) 12

13 Percent of U.S. cell owners who text Texting and Coverage Error (4) (r 2 =.95) Actual growth Linear growth If current trends continue, most cell owners are likely to become text users (in one format or another) In this case, coverage error may not be a concern for most topics 13

14 Nonresponse (Error) Virtually no evidence about nonresponse error Evidence on response rates in text interviews is sparse and highly variable (especially between countries) comes primarily from opt-in panels and convenience samples rather than probability samples Awareness that an invitation to a particular text survey is on its way seems to increase response rate compared to general willingness to take part in a text interview Being invited by a human interviewer increases response rates (in the one study that tested this) 14

15 Study Features Automated? Marlar, McGeeney & Chattopadhyay, 2014 Schober et al., 2014, in prep Gallup Daily Tracking Study (US) GeoPoll panel (Tunesia) Participation Awareness of Particular Text Survey? Response Rate (%) # Qs Completion Rate (%) Y N Y N Cooke, Nielsen, & Strong, 2003 Schober et al., 2015 online panelists (UK) Convenience sample of iphone users Y Y N Y Y Y

16 Text vs. Voice: Schober et al., 2015: Experimental Design 4 Modes on iphone: Human Voice Human Text (SMS) Automated Voice Automated Text (SMS) 16

17 Response Rate* Response Rates* Across All Modes 80% Schober et al., % 60% 50% 40% 30% Voice Text 20% 10% 0% Automated Human Higher response rate in text could be due to (1) persistence of invitation (no noncontact), (2) ability to respond when convenient, (3) more time to decide *AAPOR RR1: # complete interviews / # invitations 17

18 Breakoffs 16% 14% 12% 10% 8% 6% 4% 2% 0% Breakoffs Across All Modes Automated Schober et al., 2015 Human More breakoffs in Text could be due to (1) less social presence (no voice) or (2) asynchronous character (no need to answer quickly) Substantially higher breakoff rates in Automated than Human modes likely due to absence of human interviewer Rs who broke off and Rs who completed equivalent demographically except slightly more females broke off Voice Text 18

19 Measurement Error: Data Quality Schober et al., (2015) Compared satisficing and disclosure between text and voice interviews automated and human-administered interviews Satisficing: Rounded numerical answers: divisible by 10 Straightlining: same answer (e.g., strongly favor ) for 7 of 8 statements about diet e.g. avoid red meat, limit fast food Disclosure: Undesirable responses, e.g., less than one day a week when asked How often do you exercise? 19

20 Text Respondent 20

21 Human Text Interviewer Interface 21

22 Text Interaction between Human Interviewer and Respondent 22

23 Possible effects of texting on quality of Satisficing answers More: Rs may import least effort strategies from usual texting practice Truncated, abbreviated interaction Less: Reduced time pressure Rs can answer when they have time to think about answer and, possibly consult records Disclosure: More: reduced social presence; no interviewer face or voice Less: permanent record of messages 23

24 Text vs. Voice: Satisficing 24

25 Text vs. Voice: Disclosure 25

26 Measurement Error: Interface Design Even to design an interface that works on the smallest screens and within the constraints of SMS, there is still a range of design considerations Single- vs. multi-character responses threaded questions and answers vs. single message displays physical numeric vs. virtual alphanumeric keyboard Must spelling be correct? 26

27 Effort Rs may be more likely to choose response options that require fewer keystrokes Each keystroke takes additional effort This is probably why in daily communication, texters frequently abbreviate (c u l8r, omg, lol, imho ) In interacting with interfaces (generally), users tend to minimize effort 27

28 Sensitivity Rs may be more likely to select more sensitive response options when they are singlecharacter than when they must articulate (type) full answer Single-character labels are arbitrary rather than meaningful may be less value laden than multiple-character Single-character responding may allow respondents to hide behind the character 28

29 Single vs. multiple character responding Schober et al. (2014, in prep) Rs randomly assigned to either single- or multi-character interviews multi-character Rs instructed to key in full responses (open-ended) single-character respondents required to answer with single character (i.e. 1, 2, 3 ) 12 closed-form survey questions Texted in French by an automated system to Rs in Tunisia (Geopoll) 0.5 dinars (roughly $0.30 USD) as incentive 6 Qs more and 6 Qs less likely to be sensitive for Tunisian Rs e.g., How often do you attend religious services? vs. Which of these recreational activities do you most prefer: Response options varied in length 29

30 Results: Effort Rs in multicharacter condition more likely to select shortest response option and less likely to select longest response option than Rs in singlecharacter condition Average number (LS means) of shortest response option selected Average number (LS means) of longest response option selected p < p < single-character condition multiple-character condition 0.0 single-character condition multiple-character condition 30

31 Results: Sensitivity findings Overall, we did not consistently observe more socially undesirable responding in the single-character condition Average number (LS means) of most socially undesirable answers selected 1.32 P = single-character condition multiple-character condition 31

32 Sensitivity cont d: Bribe item BUT Rs more likely to report having paid a bribe with a single-character response than by typing out oui ( yes ), which is no shorter than non Has your household paid a bribe to any of the following in the past 6 months: university, gov official, traffic police, tax official? (%) p = single-character condition multi-character condition 0.0 Yes No 32

33 Efficiency of Text Interviews While the interface matters, there are clear data quality advantages for text over voice interviews What is the cost of these advantages (Conrad et al., 2013, in prep)? How long does it take to complete text cases from when they are released? How long does it take to complete text interviews? 33

34 % of completed cases Cumulative Completion of Cases Automated Voice Automated Text Human Voice Human Text Days after initial invitation In automated text Rs could reply at any time in order to start interview 34

35 Median Hours Time in Field per case Automated Human In automated text, system available 24/7; responded immediately (median response time = 3 seconds) after R texted answer 1.10 Voice Text 35

36 Interview Duration and Median Number of Turns per Question Voice Human Question number Questions Subsequent turns Automated Question number Text Human Question number Automated Question number Questions Subsequent turns Time (min)! So Rs may be using the time between turns productively Could involve checking records and thinking about answer before answering 36

37 Summary Precontact: Text pre-contact seems to increase participation in web and mobile web data collection more than pre-contact in other modes Interview: Coverage: No difference between texters and non-texting mobile users on most demographic variables Nonresponse: Response rate higher in text than voice interviews; response rates seem to be higher when sample members expect text invitation and when it is sent by human interviewer Measurement: Texting leads to less satisficing and more disclosure than voice interviews; as in other modes, text Rs minimize effort by selecting shorter response options (fewer characters) when need to enter full option Efficiency: Although interviews take longer by text than voice, field period is shorter for text than voice surveys: automated text particularly cost-effective 37

38 Conclusions and Discussion Texting appears to be viable both for contacting and interviewing As interview mode, reductions in measurement error may depend on Rs experience and comfort with texting whether implemented as SMS or in multimodal texting app, on mobile or nonmobile devices Is texting suitable for only brief interviews? What incentives would be required for participation in longer interviews? Multi-modal texting: Integration of dictation and text-to-speech can allow users who do not read or are visually impaired to be interviewed via text Ability to attach video (e.g., Fuchs, 2007) could become commonplace for at least some populations 38

39 Final Word Many potential survey respondents use text messaging as a central part of their daily communication Not including text messaging in a study's design may well require an active choice as texting becomes more ubiquitous 39

40 Thank You! 40