Mobility Enabled: Effects of Mobile Devices on Survey Response and Substantive Measures

Transcription

1 Mobility Enabled: Effects of Mobile Devices on Survey Response and Substantive Measures Frances M. Barlas, Randall K. Thomas, and Patricia Graham GfK Custom Research Presented at CASRO Digital Research Conference February 11-12, 2015 Nashville, TN. GfK

2 Online Surveys are Increasingly Mobile About 25% to 33% of respondents are completing our online surveys on mobile devices. 22% on Smartphones 10% on Tablets GfK

3 Mobile Respondents Do They Matter? Demographic differences across device preference: Respondents = younger, lower education and income, Hispanic and African American Respondents = more middle-aged, higher education and income Mobile respondents also: Take longer to complete surveys Have higher survey breakoff rates And tend to have higher rates of suboptimal response, such as item nonresponse, failing trap questions GfK

4 Screen Size Diversity of Screen Real Estate 221 unique screen resolutions 18.6% Designing our online surveys for mobile respondents first will help to: Increase data quality Increase response and completion rates Improved survey experience and panel retention Decrease sampling and coverage error GfK

5 Research on Research This study used GfK s KnowledgePanel, the largest probability-based online panel in the US with approximately 55,000 members and average cooperation rates of 60%. Three sample groups, all with access to a Desktop/Laptop and GfK

6 Research on Research To control self-selection bias, participants were randomly assigned to completion device. Invited Responded Used assigned device 10,672 7,837 4,555 1,497 Desktop/laptop respondents 1,483 Smartphone respondents 1,575 Tablet respondents GfK

7 What Researchers Can Control Two Factors Survey Software Survey Design GfK

8 Experimental Assignment Survey Software Template 12% Traditional template 88% Responsive template GfK

9 Traditional Template Mobile Devices First Look GfK

10 Traditional Template Mobile Devices First Look After Zooming After zooming in, question is still difficult to read. It does not automatically reformat to fit the screen, but requires scrolling left to right to read the question. GfK

11 Responsive Template Mobile Devices Smartphone Tablet The responsive template automatically resizes the survey text to fit the detected screen size. GfK

12 Experimental Assignment Mobile Friendliness 50% Mobile unfriendly 50% Mobile friendly GfK

13 Example: Mobile Friendly vs. Unfriendly Mobile Unfriendly longer question and response options GfK

14 Example: Mobile Friendly vs. Unfriendly Mobile Friendly shorter question and response options GfK

15 Results GfK

16 Demographics Statistically significant differences in demographics across devices tended to confirm accidental mobile demographics. GfK

17 Minutes Median Survey Completion Times 30 Desktop/Laptop Tablet Smartphone Traditional, Mobile Unfriendly Traditional, Mobile Friendly Responsive, Mobile Unfriendly Responsive, Mobile Friendly GfK

18 Survey Breakoff Rates 10% Breakoff rates dropped by 50% when responsive and mobile friendly design used 0% 5% 3% 4% 5% 2% 3% 5% 1% 2% 4% Traditional, Mobile Unfriendly Traditional, Mobile Friendly Responsive, Mobile Unfriendly Responsive, Mobile Friendly Desktop/Laptop Tablet Smartphone GfK

19 Survey Location Over 90% of Tablet respondents completed the survey at home. Almost 20% of Smartphone respondents completed the survey at the office or somewhere else outside the home. GfK

20 Device Effects Device effects occur when responses to a question on one device differ significantly from responses when answered on a different device. Most studies have a difficult time teasing apart device effects from sample differences because they rely on accidental mobile respondents. In this study, random assignment controlled for selfselection effects. Due to differential nonresponse, covariates used to further equate respondent groups across devices. GfK

21 Device Effects were present when measuring Survey Experience: GfK

22 Device Effects were present when measuring Technology: GfK

23 No device effects across range of topics: GfK

24 Device Effects No interactions with template version or mobile friendliness of questionnaire when present, device effects consist across these experimental conditions. No clear pattern of device effects by response format assessed single item with varying scale lengths, vertical vs. horizontal response presentation, grids. GfK

25 Yes/No Grids vs. Multiple Response Format GfK

26 Yes/No Grids vs. Multiple Response Format GfK

27 Conclusions GfK

28 Conclusions Mobile respondents are not all the same. Demographic differences due to differential nonresponse tended to support demographics of accidental mobile respondents. Improving survey display for mobile respondents helps, but it is not enough we also need to approach questionnaire design with a mobile first mentality. We saw decreased completion times and breakoff rates with responsive template and mobile friendly design. Smartphone respondents appear to be more mobile than both desktop/laptop and tablet respondents. Tablet respondents most likely complete at home. GfK

29 Conclusions Device effects appeared to be more driven by survey content than response format. On the whole, device usage does not appear to affect responses, though be careful when measuring technology-related issues or assessing survey experience. Overall, device effects appeared to be minimal, though we caution this was a study with relatively high survey completion rates and with experimental and statistical controls. To minimize device effects it is essential to minimize breakoff and nonresponse bias. GfK

30 Thank you! Frances M. Barlas GfK