Web Surveys: The Good, The Bad, & The Ugly

Transcription

1 Web Surveys: The Good, The Bad, & The Ugly June 5, 2006 Carrie Bibik Senior Associate, PRA Inc. PRA Inc., 2006

2 Outline Introduction to use of Web surveys Two examples of Web surveys Survey of Undergraduate Students Canadian College Student Survey The Good, The Bad, & The Ugly Best practices for using Web surveys in evaluation 1

3 Introduction Web surveys are becoming increasingly popular Ability of Web surveys to achieve higher response rates is debatable Encounter many of the same problems as traditional survey approaches (mail and telephone) Distinct advantages and disadvantages to Web surveys 2

4 Example 1 Survey of Undergraduate Students Canadian Undergraduate Survey Consortium

5 Background Conducted annually since 1994 Three-year cycle: all undergraduates, first-year students, graduating students Traditionally paper-based Sample 1,000 students per university 28 universities participated in 2005 survey Web pilot with six universities 22 paper-based 27 universities participated in 2006 survey 13 Web-based 14 paper-based 4

6 Web survey methodology To participate in Web survey, universities had to provide addresses for sample of students PRA ed students a personalized invitation, including unique link to survey 5

7 Definition and analysis of completes Had to answer 100 of 143 questions (or 70%) to be considered complete Dropped 2 to 12 paper questionnaires Dropped 30 to 308 Web questionnaires Results suggest higher incidence of incompletes among Web surveys However, this may simply reflect that respondents are unlikely to return an incomplete paper survey 6

8 Response rates Based on: # of completes / original sample size Does not account for: Undeliverable mailing/ ing Addition of replacement students Removal of ineligible students Per-university adjustments to response rates for Web survey, would involve: Removal of 3 to 99 undeliverable s Addition of 0 to 10 replacement students Removal of 0 to 5 ineligible students 7

9 Response rates 2006 survey Paper Range from 21.2% to 71.6% Less than 40%: n=4 40% to 49.9%: n=5 50% or higher: n=4 Web Range from 37.0% to 58.2% Less than 40%: n=1 40% to 49.9%: n=6 50% or higher: n=7 8

10 Comparison of Web-based and paperbased response rates 2005 and 2006 (across similar universities) Grad All Univ Web Paper Web Paper % 29.7% 55.1% 31.9% % 50.7% 49.8% 28.1% % 40.7% 47.7% 52.6% 4* 41.9% 54.9% 47.0% 34.8% % 40.5% 38.5% 48.1% % 53.4% 35.8% 40.1% 9

11 Influences on response rates Methodology Paper Web Universities administered themselves Assume conducted two reminders to non-responders Financial disincentive to conduct multiple reminders PRA administered Conducted minimum of two reminders to nonresponders Conducted an additional reminder for some universities with lower response rates Type of incentives offered by each university Other research being conducted at university 10

12 Example 2 Canadian College Student Survey Canada Millennium Scholarship Foundation

13 Background Conducted annually since 2002 Traditionally paper-based, college administered with students in class Colleges aim to achieve target number of completes (n=450) Web-based survey first offered in colleges participated in paper-based 14 Web-based 12

14 Methodology Unique invitation PRA administered using student addresses Open invitation College administered using various modes ing generic Web address to students Taking students to computer lab Making in-class announcements Placing notices around campus/on college Web site 13

15 Results Number of completions Paper: 108 to 452 Web: 71 to 947 Differences in completion rates reflect: Methodology used Number of students notified (some notified entire eligible student population and others notified a targeted sample) Follow-up efforts of college Size of college Success by college compared to previous years varies 14

16 Findings

17 The Good Pre-testable Requires shorter fielding period Can monitor completion rates No need for data entry Data readily available Minimal cleaning and coding required: Can pre-program skip logic Can pre-program valid response ranges Can use drop-down menus Respondents can ask questions via 16

18 The Bad & The Ugly Not appropriate for all types of respondents Respondents may not regularly check Dependent on quality of lists Subject to typos, closed accounts, etc. Invitation may be blocked or sent to junk mail text may not display/function properly Questionnaire may not work well with some Internet browsers Web surveys tend to not work well with dial-up Cannot track respondents if using open Web site 17

19 The Bad & The Ugly (continued) Respondents may begin to resent reminders May receive more negative feedback from respondents Respondents may not provide as much information in open-ended questions Can be comparability issues between responses from paper-based and Web-based surveys Typically, Web surveys require respondents to answer each question on a page before advancing to the next page Raises issue of whether to include a no response option or to make questions non-mandatory Privacy concerns may arise if data are stored in the US 18

20 Best practices for using Web surveys in evaluation: set-up stage Understand target sample s experience with and usage of Internet and Use logos/electronic signatures to validate survey Catchy subject line for Keep surveys to less than 20 minutes in length Keep questions simple and minimize reference to previous questions Finalize questionnaire in paper format before transferring to Web format Included address for a contact on each page of survey Sign confidentiality/non-disclosure/privacy agreements Password protection/encrypt personal information transferred via 19

21 Best practices for using Web surveys in evaluation: fielding stage Can send notification letters in mail to inform sample of survey Can use incentives to encourage participation Use single Web methodology when surveying multiple groups Send reminders at intervals of about one week Limit number of reminders to two or three Identify an IT contact to help address technical issues 20

22 Carrie Bibik Senior Associate, PRA Inc. Winnipeg, Manitoba Ph: