Towards a Web-based multimode data collection system for household surveys in Statistics Portugal: a report and discussion of the first experiences Pedro Lima Maria José Fernandes Wiesbaden 4-5 September 2014 1
Summary 1. Web data collecting context 2. Introducing the CAWI mode in the National Health Survey (INS 2014) - The INS2014 main features - Pros and cons of collecting INS data with CAWI 3. The main challenges of implementing the INS on the Web and the respective decisions taken 4. The main challenges of implementing a mixed mode CAPI-CAWI data collection and the respective decisions taken 5. Evaluation 2
Web data collecting context Encouraging results e-census/2011: 33% of census 2011 dwellings (50% of population) answered by web 97% of web satisfaction questionnaire respondents reported their willingness to answer by web in future surveys Information and Communication Technology Survey/2013: 62% of households reported to have an internet connection 58% of households contacted public administration through their websites 42% of these fill and sent online forms (mainly tax forms) 3
Internal motivations Modernize the data collecting process Web data collecting context Offer to respondents more comfortable and less intrusive ways to answer surveys Reach out population segments that are usually difficult to interview in CAPI mode Need of cost rationalization of the official statistical production system The commitment to introduce a web tool for a household survey in 2014 4
Introducing the CAWI mode in the National Health Survey (INS 2014) Determinants for choosing the 2014 web survey Internal discussion (main departments involved) No longitudinal surveys Survey with a data collection period for a sequential multimode data collection INS emerged as the only consensual household survey to have a CAWI mode in 2014 5
Introducing the CAWI mode in the National Health Survey (INS 2014) Additional favorable factors for choosing INS Sensitive and personal issues High CAPI costs Length (250 questions = EHIS + National needs) - too long for face-to-face contact - web answer more convenient for respondents 6
Introducing the CAWI mode in the National Health Survey (INS 2014) Risks of implementing the CAWI mode in the INS 2014 Length - break offs No proxy interviews - impossible to be 100% sure that the eligible individual is the real respondent; Very short period for development and implementation No prior in-house web experience in household surveys 7
Introducing the CAWI mode in the National Health Survey (INS 2014) Total Sample 22.538 dwellings Target population: individuals aged 15 and more Selection method: last birthday Sept. 10-30 (21 days) 50% CAWI 50% CAPI Sept. 15 - Dec. 15 Response Non-Response Response Non-Response Criteria for questionnaire data transition CAWI no answer -> CAPI blank CAWI no eligible dwelling -> CAPI blank CAWI eligible individual less than 18 years old -> CAPI blank CAWI incomplete -> CAPI blank CAWI prefer CAPI -> CAPI household data pre-filled CAPI Oct. 8 - Dec. 15 Response Non-Response 8
The main challenges of implementing the INS questionnaire on the Web Log in and authentication Advanced letter with user ID and password 0. Login screen 1. Terms of services acceptance 2. Provide email 3. Email confirmation (code sent to respondent) 9
The main challenges of implementing the INS questionnaire on the Web Log in and authentication 10
The main challenges of implementing the INS questionnaire on the Web Log in and authentication 11
The main challenges of implementing the INS questionnaire on the Web Questionnaire-related topics Multimode questionnaire design approach: unimode design (CAPI main mode) How to emulate the CAPI behaviour for: Don t know option Access to auxiliary questions (alternative questions for the same variable) Access to instructions (about questions) Questionnaire design: Layout template: mainly a paging design (one question per screen) 12
Layout template - I 13
Layout template - II 14
Navigation menu 15
Help tab 16
The main challenges of implementing a multimode data collection Organization and case-management 1st - CAWI data collecting management 2nd - Sizing up the CAPI field team to collect data from CAWI sample non-response 17
The main challenges of implementing a multimode data collection 1st - CAWI data collecting management Data collection management tools Two web applications i. manage the total sample, the mode allocation, the transition mode CAWI -> CAPI and the CAPI field team ii. monitor web data collection status i. questionnaire status (non-answer/incomplete/submitted) ii. Incomplete questionnaires (fulfillment degree, break-off question) Client-side dwelling paradata i. n.º of sessions per dwelling ii. session s date and duration iii. break-off reasons 18
The main challenges of implementing a multimode data collection 1st - CAWI data collecting management Contact strategy (reminders) i. For: incomplete questionnaires and non-responses ii. When: 1 and 2 weeks after the CAWI started iii. By: letter, email Web BackOffice (help respondents and answer their doubts) i. Phone line support ii. Help form answer 19
The main challenges of implementing a multimode data collection 2nd - Sizing up the CAPI field team to collect data from CAWI sample non-response Large enough and geographically well placed field team of interviewers to collect data that comes from the CAWI non-respondents sample within the collection period. Risks: the CAWI response rate is very low the sample CAWI->CAPI is too spread-out geographically The field teams were dimensioned to accommodate these possible scenarios 20
Evaluation We had A short period for development and testing, so we had to Be pragmatic and take decisions under uncertainty We are aware of risks of Potential high number of CAWI break-offs Higher number of Don t Knows answers in CAWI mode Mode sensitive bias for some questions We have to Find a proper solution to handle the informed consent for minors on the web Improve the web questionnaire layout and performance Assess mode-effects in the data processing Evaluate the CAWI paradata (server-side and client-side) 21
Thank You pedro.lima@ine.pt mjose.fernandes@ine.pt 22