SLIDER BARS IN MULTI-DEVICE WEB SURVEYS

Transcription

1 SLIDER BARS IN MULTI-DEVICE WEB SURVEYS A N G E L I C A M. M A I N E R I ( T I L B U R G U N I V E R S I T Y ) A. M. M A I N E R U V T. N L I V A N O B I S O N ( U N I V E R S I T Á D I T R E N T O ) R U U D L U I J K X ( T I L B U R G U N I V E R S I T Y & U N I V E R S I T Á D I T R E N T O ) ESRA Conference 2017 Lisbon Thu, July 20 th SESSION: Adapting online surveys for mobile devices (2)

2 INTRODUCTION & RESEARCH QUESTION Growth of mobile access to Web surveys Web surveys new possibilities, interactive object Consequences on data quality? Are there differences in the occurrence of satisficing behaviors in the use of slider bars according to the device employed? Relevance o Contribution to literature on consequences of unintended mobile access o Experimental design: explore peculiar features of slider bars SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 2

3 THEORETICAL BACKGROUND Pc vs mobile Input method: mouse vs touchscreen Environment Mavletova, 2013; Toninelli and Revilla, 2016 Technical problems Funke, 2016 (Software optimasation) Pc/Tablet vs smartphone Screen size Data quality? Mixed evidence growing literature SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 3

4 DATA & METHODS Data: Web surveys collected on students of University of Trento Students satisfaction survey (2015) Use of time and space (2016) Survey Students satisfaction Use of time Population All students All students Period of data collection Feb/Apr 2015 May/June 2016 Paradata (UAS) Yes Yes Question on device in use No Yes Questions on dominant hand and screen orientation No Yes Software Lime survey Lime survey Size of population Opened the survey (% of population) 7768 (43.6%) 6024 (37.7%) Completed the survey (% of population) 6346 (35.6%) 4423 (27.7%) Table 1 Overview of datasets employed We are deeply grateful to Enzo Loner for the technical support in setting up the surveys and the experiments. SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 4

5 EXPERIMENTS 1. Numeric labels (Students satisfaction) 2. Initial position of the slider (Use of time) Same layout as above, handle on left or right; 8 couples of adjectives SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 5

6 RESULTS: DESCRIPTIVES Device used - only respondents with cookies enabled Device used to access (UAS) Measured at beginning of survey Satisfaction survey (2015) Use of time (2016) % N % N Self-reported device Measured at the end Use of time (2016) % N Desktop/Laptop Smartphone Tablet Total Discrepant cases Different device recorded at the beginning and reported at the end SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 6

7 RESULTS: COMPLETION RATES Completion rates by device SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 7

8 RESULTS: DESCRIPTIVES (1) Average scores of perception of the website (left- n=6343) and the city (rightn=4207) by device SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 8

9 RESULTS: ROUNDING (1) OLS Regression model (N=6343) Number of scores ending with 0 or 5 selected Constant 1.75*** (0.06) [H1] rounding occurs more often when numeric labels are visible Rounding = selection of scores ending with 0 or 5 Group: Numeric labels 0.66*** (0.04) Female (0.04) Age: ref (0.06) (0.07) (0.08) ** (0.08) Master student -0.12** (0.05) R * p<0.1; ** p<0.05; *** p< Source: Students' satisfaction (2015) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 9

10 RESULTS: ROUNDING (2) Effect of numeric labels weaker for [H2a] smartphone, because of screen/font size OR [H2b] smartphone and tablet, because of input methods SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 10

11 RESULTS: ANCHORING (1) Likelihood to select extremes higher for [H3a] smartphone respondents OR [H3b] smartphone and tablet SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 11

12 RESULTS: ANCHORING (2) Puzzled situation look at anchoring to central scores If significant and positive effect only on Students satisfaction -> position of the handle SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 12

13 RESULTS: POSITION OF THE HANDLE (1) [H4] respondents with the handle on the left select lower scores (closer to left extreme) than the ones with the handle on the right. AVERAGE SCORES OF PERCEPTION OF THE CITY (N=4207) BY EXPERIMENTAL CONDITION SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 13

14 RESULTS: POSITION OF THE HANDLE (2) The effect of the initial position of the handle is weaker [H5a] for smartphone respondents OR [H5b] for smartphone and tablet respondents OLS Regression model (N=4207) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 14

15 RESULTS: POSITION OF THE HANDLE (2) The effect of the initial position of the handle is weaker Stronger [H5a] for smartphone respondents OR [H5b] for smartphone and tablet respondents OLS Regression model (N=4207) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 15

16 DISCUSSION - Numeric labels weaker effect of smartphone - Initial position of the handle stronger effect on smartphone - Anchoring unclear Higher satisficing on smartphone Mixed evidence on satisficing - Tablet? Looks similar to pc but - Completion outcomes underestimation of satisficing effects? SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 16

17 CONCLUSIONS - Careful in design choices with multi-device surveys - Completion outcomes: how to retain respondents on smartphones? - Future research SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 17

18 Thank you! Questions? SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 18

19 REFERENCES Antoun, C. (2015). Effects of mobile versus pc web on survey response quality: a crossover experiment in a probability web panel. Paper presented at the Annual Conference of the Americal Association for Public Opinion Research, Hollywood, FL. Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education, 53(2), Buskirk, T. D., & Andrus, C. (2012). Online surveys aren't just for computers anymore! Exploring potential mode effects between smartphone and computer-based online surveys. Paper presented at the AAPOR Annual Conference, Orlando, FL, USA, May 17 20, Buskirk, T. D., & Andrus, C. (2014). Making Mobile Browser Surveys Smarter: Results from a Randomized Experiment Comparing Online Surveys Completed via Computer or Smartphone. Field Methods, 26(4), doi: / x Callegaro, M. (2013). Do You Know Which Device Your Respondent Has Used to Take Your Online Survey? Survey Practice, 3(6). Couper, M. P., Tourangeau, R., Conrad, F. G., & Singer, E. (2006). Evaluating the effectiveness of visual analog scales: A web experiment. Social Science Computer Review, 24(2), De Bruijne, M., & Wijnant, A. (2014). Improving Response Rates and Questionnaire Design for Mobile Web Surveys. Public Opinion Quarterly, 78(4), Funke, F. (2016). A Web Experiment Showing Negative Effects of Slider Scales Compared to Visual Analogue Scales and Radio Button Scales. Social Science Computer Review, 34(2), doi: / Heerwegh, D., & Loosveldt, G. (2008). Face-to-Face versus Web Surveying in a High-Internet-Coverage Population: Differences in Response Quality. Public Opinion Quarterly, 72(5), doi: /poq/nfn045 Klausch, T., Hox, J. J., & Schouten, B. (2013). Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions. Sociological Methods & Research, 42(3), doi: / SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 19

20 REFERENCES Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3), Lugtig, P., & Toepoel, V. (2016). The Use of PCs, Smartphones, and Tablets in a Probability-Based Panel Survey Effects on Survey Measurement Error. Social Science Computer Review, 34(1), doi: / Mavletova, A. (2013). Data Quality in PC and Mobile Web Surveys. Social Science Computer Review, 31(6), doi: / Peterson, G., Mechling, J., LaFrance, J., Swinehart, J., & Ham, G. (2013). Solving the unintentional mobile challenge. Paper presented at the CASRO Online Research Conference, March. PEW. (2013). Smartphone Ownership 2013 Update. Peytchev, A., & Hill, C. A. (2010). Experiments in Mobile Web Survey Design: Similarities to Other Modes and Unique Considerations. Social Science Computer Review, 28(3), doi: / Roßmann, J., & Gummer, T. (2014). PARSEUAS: Stata module to extract detailed information from user agent strings: Statistical Software Components. Retrieved from Struminskaya, B., Weyandt, K., & Bosnjak, M. (2015). The Effects of Questionnaire Completion Using Mobile Devices on Data Quality. Evidence from a Probability-based General Population Panel. The Collection of Survey Data using Mixed Devices, 261. Toepoel, V., & Lugtig, P. (2015). Online surveys are mixed-device surveys. Issues associated with the use of different (mobile) devices in web surveys. Methods, Data, Analyses, 9(2), Toninelli, D., & Revilla, M. (2016). Smartphones vs PCs: Does the Device Affect the Web Survey Experience and the Measurement Error for Sensitive Topics?-A Replication of the Mavletova & Couper s 2013 Experiment. Paper presented at the Survey Research Methods. SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 20

21 RESULTS: DESCRIPTIVES Average scores of perception of the website (left- n=6343) and the city (right- n=4207) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 21

22 RESULTS: ANCHORING BY POSITION OF THE HANDLE Predicted values (OLS regression) of number of extreme scores selected by device, experimental group (and controlled for gender, age and type of study) N=4207 SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 22

23 RESULTS: POSITION OF THE HANDLE (3) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 23

24 RESULTS: POSITION OF THE HANDLE (3) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 24

25 RESULTS: DOMINANT HAND AVERAGE SCORES OF PERCEPTION OF THE CITY (N=4196) BY DOMINANT HAND SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 25

26 RESULTS: SCREEN ORIENTATION AVERAGE SCORES OF PERCEPTION OF THE CITY (N=1725) BY SCREEN ORIENTATION SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 26

27 THEORETICAL BACKGROUND (EXTRA) Overall, no lower data quality on mobiles vs pc Mavletova, 2013; Buskirk and Andrus, 2012; Peterson et al., 2013; Antoun, 2015 Worse completion outcomes on smartphones De Bruijne and Wijnant, 2013; Mavletova and Couper, 2015 (but ) Higher measurement error among smartphone users Lugtig and Toepoel 2016; Struminskaya et al Depend on indicators, research set up, software/optmisation Slider Technical issues (Funke, 2016) Mistakes (Antoun, 2015) SLIDER BARS IN MULTI-DEVICE WEB SURVEYS 27