FACE-TO-FACE VERSUS WEB SURVEYING IN A HIGH-INTERNET-COVERAGE POPULATION DIFFERENCES IN RESPONSE QUALITY

Transcription

1 Public Opinion Quarterly, Vol. 72, No , pp FACE-TO-FACE VERSUS WEB SURVEYING IN A HIGH-INTERNET-COVERAGE POPULATION DIFFERENCES IN RESPONSE QUALITY DIRK HEERWEGH GEERT LOOSVELDT Abstract The current study experimentally investigates the differences in data quality between a face-to-face and a web survey. Based on satisficing theory, it was hypothesized that web survey respondents would be more likely to satisfice for a multitude of reasons, thereby producing data of lower quality. The data show support for the hypothesis. Web survey respondents were shown to produce a higher don t know response rate, to differentiate less on rating scales, and to produce more item nonresponse than face-to-face survey respondents. Introduction A general concern of survey researchers is to obtain high response rates while keeping the financial costs in check. As early as in 1967, Hochstim asked these crucial questions: Could the inexpensive mail technique be used to gather the bulk of the responses, at the same time identifying the hard-toget part of the public for follow-up by telephone or personal calls? Would such an approach achieve a high response of good quality at a cost saving? (Hochstim 1967, p. 976). Today, these questions seem at least as relevant as back then, given that survey nonresponse rates appear to be rising (Goyder and McKenzie Leiper 1985; Hox and de Leeuw 1994; Curtin, Presser, and Singer 2005; Voogt and Saris 2005). Combining different survey modes to increase survey response rates while keeping the financial costs down has been practiced DIRK HEERWEGH AND GEERT LOOSVELDT are with are with the Center for Sociological Research (CeSO), Katholieke Universiteit Leuven, Parkstraat 45 Box 3601, 3000 Leuven, Belgium. The authors would like to thank Ingrid Willems of University of Antwerp, Belgium, and the anonymous reviewers for their valuable comments on an earlier version of this paper. The first author gratefully acknowledges support for this research from a postdoctoral research grant of the KULeuven Research Fund (BOF) and from a postdoctoral research grant of the FWO Flanders (Fonds voor Wetenschappelijk Onderzoek Vlaanderen). Errors remaining and views expressed are the authors. Address correspondence to Dirk Heerwegh; dirk.heerwegh@soc.kuleuven.be. doi: /poq/nfn045 Advance Access publication October 31, 2008 C The Author Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please journals.permissions@oxfordjournals.org

2 Response Quality in Web Surveys 837 at least since Hochstim s study. A recent revival of this strategy is notable, due in part to a relatively new survey mode, the web survey. Web surveys are an appealing candidate for mixed-mode survey designs because they are cost and time efficient (Couper 2000; Dillman 2000). Combining different survey modes could lead to mode effects (Dillman 2000). Several studies have investigated differences between web surveys and other approaches. Some studies have concentrated on the response rate differences between web and mail surveys (e.g., Kaplowitz, Hadlock, and Levine 2004), while others have also looked into possible differences in respondent profile and data quality (e.g., Bates 2001; Kwak and Radler 2002). Several studies have made a comparison between web and telephone surveys (Dillman et al. 2001; Fricker et al. 2005). The current study compares a web survey to a face-to-face survey. These two modes could be employed as the start and end point of a sequential mixedmode design. Given a population with a sufficiently high Internet coverage, a web survey could be deployed to collect as much data as possible, after which other, more expensive survey modes could be used to follow up the nonrespondents. A face-to-face survey could be the final stage of such a survey design. Even though such a sequential design would probably be the most practical, it obscures the distinction between mode effects and differences in sample composition by survey mode. Since the current study is interested in mode effects, a simultaneous mode experiment is conducted. A sample is drawn from a single, high-internet-coverage sampling frame, after which the sample units are randomly distributed across the survey modes. This experiment considers the possibility that face-to-face respondents are more likely to exert the necessary cognitive effort to answer the survey questions than the web survey respondents. The higher degree of satisficing among the web respondents would lead to lower data quality than that obtained in the face-to-face survey. Theoretical Background and Expectations One theoretical approach to characterize and explain response errors in surveys is satisficing theory (Krosnick 1991; see also Cannell, Miller, and Oksenberg 1981). The basic assumption of this theory is that optimal question-answering takes a considerable amount of cognitive work. A respondent must interpret the meaning and intent of each question, retrieve relevant information from his or her memory, integrate that information into a summary judgment, and then report that judgment taking into account the provided response alternatives (Tourangeau, Rips, and Rasinski 2000). While many respondents may perform these steps, other respondents might take cognitive shortcuts to reduce the required effort (and exhibit satisficing behavior). The satisficing probability is thought to be influenced by three factors: motivation, ability, and task difficulty. Respondent motivation and ability are

3 838 Heerwegh and Loosveldt inversely related to the satisficing probability. Lower motivation and lower ability increase the satisficing likelihood. Higher task difficulty boosts the satisficing propensity (Krosnick 1991). Holbrook et al. (2003) link satisficing theory with survey mode. In their comparison between face-to-face and telephone surveys, they point out that these modes differ on a number of dimensions. According to Holbrook et al. (2003, pp. 83 4), face-to-face surveys allow for nonverbal communication between the respondent and the interviewer, which could motivate respondents to properly respond to each of the questions (see Holbrook, Green, and Krosnick 2003). Similar to telephone surveys, web surveys preclude nonverbal communication. For this reason, a higher satisficing probability in the web than in the face-to-face survey could be expected. Holbrook et al. (2003, pp. 84 5) argue that telephone respondents might be engaged in different activities while answering the questions (multitasking). The added distraction due to multitasking could induce more satisficing. Perhaps multitasking is even more common in web surveys. Respondents might have a number of programs running concurrent with the web browser opened for the purpose of answering the web survey (e.g., an program) and/or could be accessing different web sites at the same time. This also leads to the expectation of a higher satisficing likelihood in a web than in a face-to-face survey. A third difference between the two survey modes considered in this study is the cognitive demand placed on the respondents (Tourangeau, Rips, and Rasinski 2000, p. 302). To a certain extent, this is influenced by the distinction between aural versus visual communication (Dillman 2000), which constitutes a key difference between a face-to-face and a web survey. In a face-to-face survey, the survey questions are read to the respondent who can answer them orally. This is less demanding than a web survey, which requires some level of computer literacy apart from the general ability to read, recognize numbers, know how to use response formats etc. (see also Bowling 2005). The higher task difficulty of the web survey once again increases the likelihood of satisficing. The general hypothesis of a web survey inducing more satisficing than a faceto-face survey translates into the expectation that web survey respondents will use the don t know (DK) response alternative more frequently, and that they will differentiate less on rating scales than face-to-face respondents (Holbrook, Green, and Krosnick 2003). Nondifferentiation refers to the respondent s limited use of the available response alternatives on rating scales (see e.g., McCarty and Shrum 2000). In addition, a higher item nonresponse rate is expected in the web than in the face-to-face survey. Item nonresponse is defined here as not answering (skipping) an applicable survey question. Method and Data The experiment was embedded in a social survey conducted at the Katholieke Universiteit Leuven (Belgium). The survey intended to measure respondents

4 Response Quality in Web Surveys 839 attitudes toward immigrants and toward asylum seekers as well as respondent characteristics thought to affect these attitudes. Depending on the skip pattern, 180 to 235 survey questions were asked. 1 The questionnaire was designed according to the unimode construction principles (Dillman 2000). To avoid differences in primacy versus recency effects due to visual versus aural presentation of the response options (Tourangeau, Rips, and Rasinski 2000, pp ), show cards were used in the face-to-face survey to make these stimuli visual in both survey modes. To prevent differences in questionnaire context effects across both modes (Dillman 1991, pp ; Tarnai and Dillman 1992), web survey respondents only saw one question at a time. 2 The sample was randomly drawn from the official database of freshmen of the Katholieke Universiteit Leuven (sample n = 3,255). 3 The database contained a number of variables such as name, address, address, major, and date of birth. Prior to the fieldwork, 3,000 sample cases were randomly selected and assigned to the web survey. The remaining 255 cases were assigned to a face-to-face survey condition. Second year sociology students (n = 38) would each conduct six face-to-face interviews. 4 The web survey was launched on March 1, 2005, and closed on March 25, During that period of time, three contacts were made: one initial invitation (March 1) and two reminder s (March 7 and March 16) sent to all sample units who had not completed the survey at that time. 5 The face-to-face survey (interviewers using a paper-and-pencil questionnaire) was also fielded from March 1 to March 25, Prior to the fieldwork, all the interviewers attended two 2-hour training sessions. The first session focused on gaining cooperation (including an explanation of the concepts of tailoring and maintaining interaction as described by Groves and Couper 1998; and the compliance heuristics as described by Cialdini 1984). The second session focused on correctly administering the survey questionnaire. One week prior to the fieldwork start, an advance letter containing the same information 1. Item batteries contribute the number of items to this count (e.g., an item battery of eight items counts as eight survey questions). 2. Up to eight items of item battery questions were visible at once however. 3. A freshman is defined as a student enrolling for the first time at the University. The total number of freshmen equaled 5,461 (i.e., the sample frame). 4. Interviewers were required to report whether they knew any of the sample cases (first year students) assigned to them so that they could be substituted for sample cases they did not know. None of the sample cases had to be substituted for this reason. The number of face-to-face sample cases (n = 255) is slightly above the maximum number of face-to-face interviews (228) because of anticipated nonresponse that were substituted with reserve sample cases. Substituted sample cases are not included in the current analyses. 5. The web survey included an experiment on personalization and use of photographs on the login page. Except for the personalization treatment to increase the web survey response rate, none of these treatments had any significant effects and did not influence the responses to the questions used in the analyses to be presented. Therefore, the experimental conditions were collapsed for the analyses.

5 840 Heerwegh and Loosveldt Table 1. Outcome of Fielded Face-to-Face Addresses (Originally Sampled Cases Only) Outcome Number Percentage Interview No contact Refusal Does not meet selection criterion (e.g., no longer a registered student) Total as the invitation sent to the web survey sample cases was sent to the face-to-face sample cases. Results WEB SURVEY RESPONSE Of the 3,000 cases assigned to the web survey, 1,888 (62.9 percent) logged on to it. Of these 1,888 respondents, 1,575 reached and submitted the final page of the web survey. This means the break-off rate equals 16.6 percent and the response rate for the total web survey sample is 52.5 percent. 6 FACE-TO-FACE SURVEY RESPONSE Of the 255 sampled cases, 228 addresses were randomly selected (38 interviewers 6 interviews each). Due to noncontact, refusal or because the sample case did not meet the selection criterion, 19 additional addresses randomly selected from the remaining addresses (n = 27) were fielded. The current analyses do not take these substituted cases into account and are restricted to the 228 originally sampled cases. Table 1 shows the outcomes for the 228 originally sampled and fielded addresses. The table shows that the response rate is high; over 90 percent of all fielded addresses led to an interview (92.8 percent if ineligibles are omitted; AAPOR 2004 Response Rate 1). This high percentage could be attributed to the fact that a (first year) student population was approached by other (second year) students who served as interviewers (which might have led to feelings of sympathy for a fellow student who had to do the interview as an assignment). 6. The response rate is defined as the percentage of invited sample units reaching and submitting the final question of the web survey. This calculation adheres to AAPOR s definition 1 of the response rate (AAPOR 2004).

6 Response Quality in Web Surveys 841 NONRESPONSE ANALYSIS Within the context of a mode experiment, it is important to be able to distinguish between mode effects and differences between the eventual sample compositions due to nonresponse. The markedly different response rate in the two survey modes raises the possibility that the final sample composition differs across modes. The face-to-face survey respondents (n = 206) do not differ significantly in terms of gender (p =.992), age (p =.994), or major (p = 1.0) from the population sampled. The web survey respondents (n = 1,575) do differ significantly from the population in terms of gender (overrepresentation of female respondents, p =.005), but not in terms of age (p =.117) or major (p =.115). 7 These three variables are used as control variables in subsequent analyses on mode differences. The survey also asked if the respondent was a native of Belgium. Since the survey topic was views of immigrants and asylum seekers, country of origin serves as a fourth control variable. The percentage of natives did not significantly differ between the two samples (face-to-face: 4.37 percent, web: 3.43 percent; χ 2 (1) = 0.470; p =.493; n = 1,781). DATA QUALITY While it was expected that the face-to-face survey would take somewhat longer than the web survey because interviewers had to read questions aloud, record responses, use probing techniques, etc., the web survey was completed much faster (M = 31 39,sd= 17 15, n = 1,553) than the face-to-face survey (M = 47 45,sd= 14 16, n = 206; Kruskal Wallis χ 2 (1) = ; p <.0001), although the questionnaires were nearly identical. 8 The data suggest that the web survey respondents might have paid less attention to the questions, which would imply a lower level of data quality in the web survey. DK responses: To avoid loss of substantive data as a result of offering an explicit DK response alternative (Krosnick et al. 2002), only 12 survey questions explicitly offered this response option. These were three item batteries (two consisting of five items and one of two items), all asking the respondents to estimate their parents views of immigrants. In the web survey, the DK response option was visibly offered as the last response alternative. In the face-to-face survey, the interviewer read out the question and informed the respondent at this time that he or she could also say I don t know. In line with common face-to-face survey practice, the DK response option was not provided on the 7. In the sample, age varied between 16 and 21 years. 8. While the web survey automatically recorded timestamps, in the face-to-face survey, interviewers were instructed to note time at the beginning of the interview and note time again at the end of the interview. Twenty-two web survey respondents were excluded from this analysis since they completed the survey in more than 3 hours, which was considered to be due to an interruption of the web survey.

7 842 Heerwegh and Loosveldt show cards. 9 Based on these 12 survey questions, a DK rate was calculated for each respondent. 10 The average DK rate is 2.6 times higher in the web than that in the face-to-face survey (.174 versus.068). This difference could partly be due to the fact that an interviewer can provide further survey instructions or use probing techniques to elicit a response, but the available data do not allow assessing this hypothesis. 11 As expected, controlling for gender, age, country of origin, and major, survey mode has a significant effect on the DK response rate (see table 2). Nondifferentiation: The survey questionnaire included a number of item batteries using rating scales. For each of nine such item batteries, a response differentiation index P d (rho) was calculated (McCarty and Shrum 2000, p. 278). 12 This rate refers to the number of different scale points used by a respondent. The average of these nine rates is used here as the dependent variable. A higher P d value indicates more differentiation and reflects that the respondent used more of the response options. Only respondents answering all these questions are included in this analysis. 13 Although the difference is small, the web survey had a lower average differentiation rate (.613) than the face-toface survey (.632). Controlling for gender, age, ethnic origin, and major, the web survey has a significantly lower degree of differentiation on rating scales (table 2). It is possible to further explore this difference between the two survey modes by investigating in more detail how the rating scales are used in both survey modes. A visual survey mode could lead to increased use of the middle categories of a rating scale as compared to a more aural survey mode (the 9. Presenting the DK option on the show cards would increase mode comparability, but this would make the comparison rather artificial at the expense of the face-to-face mode. 10. The DK rate was calculated by dividing the number of DK responses to these questions by 12 (since there were no skips, all 12 questions applied to all respondents). 11. During the interaction with the respondent, interviewers can detect satisficing behavior and can attempt to reduce that behavior (e.g., reducing task difficulty by explaining the question). Interviewers were trained to provide assistance in case the respondent experienced difficulties while answering the questions. However, this assistance was limited to repeating the question and clarifying terms if needed. The interviewer was not allowed to rephrase or otherwise alter the questions. 12. These item batteries were the following: reasons to skip classes (6 items, 4 response options); frequenting types of establishments while going out (5 items, 5 response options); feelings of insecurity while going out (5 items, 5 response options); self-evaluation on a number of psychological traits (e.g., honesty, selfishness, etc.; 18 items, 9 response options); attitude toward immigrants (7 items, 5 response options); attitude toward refugees (7 items, 5 response options); behavior during vacations abroad (e.g., interacting with local population; 11 items, 4 response options); scale on social domination orientation (14 items, 5 response options); scale on conformism (11 items, 7 response options). 13. This decreases the number of usable cases in the analysis. Not all item batteries using a rating scale were included in this analysis because this would have resulted in even smaller numbers of usable cases.

8 Response Quality in Web Surveys 843 Table 2. Regression Analysis on the DK, the Response Differentiation, and the Item Nonresponse Rate Controlling for Gender, Age, Ethnic Origin, and Major DK rate (n = 1,780) Response differentiation (n = 1,156) Item nonresponse (n = 1,780) Standardized b (t) Standardized b (t) Standardized b (t) Variable R-square Survey mode 0.14 (5.92) 0.11 ( 3.90) 0.10 (4.36) (1 = web) Gender (1 = male) 0.08 ( 3.24) 0.04 ( 1.22) 0.01 (0.26) Age 0.03 (1.05) 0.01 ( 0.42) 0.01 (0.27) Ethnic origin 0.04 ( 1.63) 0.03 ( 0.86) 0.07 ( 2.87) (1 = autochthon) Major (ref. cat. = Social Sciences) Theology and 0.02 ( 0.88) 0.02 (0.61) 0.01 ( 0.49) canon Law Philosophy 0.02 (0.76) 0.04 (1.23) 0.01 ( 0.44) Law 0.03 (0.79) 0.09 (2.30) 0.01 (0.43) Economics and 0.03 (1.05) 0.08 (1.93) 0.03 (0.90) Applied Economics Arts 0.04 (1.10) 0.12 (2.63) 0.00 ( 0.01) Psychology and 0.03 (0.71) 0.15 (3.49) 0.01 ( 0.16) Educational Sciences Science 0.05 (1.53) 0.06 (1.69) 0.07 (2.15) Engineering 0.06 (1.76) 0.05 (1.16) 0.04 (1.15) Applied Bioscience 0.03 (0.97) 0.02 (0.44) 0.03 (1.12) and Engineering Medicine 0.05 (1.54) 0.10 (2.42) 0.02 (0.73) Pharmaceutical 0.01 (0.42) 0.02 (0.76) 0.09 (3.44) Sciences Kinesiology and Rehabilitation Sciences 0.06 (1.90) 0.03 ( 0.77) 0.06 (1.82) p <.05; p <.01; p <.001. face-to-face survey; see Dillman et al. 2001), and an aural survey mode could lead to increased use of extreme response categories (Tarnai and Dillman 1992, p. 117). Contrary to prediction, in five item batteries using a five-point rating scale and accounting for a total of 38 items, the middle category was selected significantly more often in the web than in the face-to-face survey. However, the two survey modes do not differ significantly regarding use of scale extremes

9 844 Heerwegh and Loosveldt Table 3. Use of Left Extreme, Middle, and Right Extreme Response Category on Five-Point Rating Scales (Average Rates across Five Item Batteries) Survey mode Left extreme Middle Right extreme Face-to-face (n = 206) Web (n = 1,575) p.79 < NOTE. Wilcoxon s two-sample nonparametric test p-values are reported. (see table 3). This suggests that the lower degree of differentiation on rating scales in the web survey is primarily due to an increased selection of the middle response alternatives. Item nonresponse: One last criterion of data quality considered here is the rate at which respondents left applicable questions unanswered. The web survey technically allowed leaving all questions blank except for four filter questions which were equipped with a prompt (cf. DeRouvray and Couper 2002). Taking into account the skip pattern and excluding the survey questions that could not be left unanswered in the web survey, an item nonresponse rate was calculated for each respondent. The average item nonresponse rates were small, but the web did yield a considerably higher average rate (.020) than did the face-to-face survey (.0001). As expected, controlling for gender, age, ethnic origin, and major, the web survey has a significantly higher item nonresponse rate (table 2). Discussion and Conclusion This experiment investigated the differences in data quality between a faceto-face and a web survey. As expected, the web survey elicited more DK responses, more nondifferentiation on rating scales, and a higher item nonresponse rate. This suggests that web survey respondents are more prone to satisficing than are face-to-face survey respondents, which is in line with the general hypothesis. Because this experiment was fielded in a student population, more research is needed to evaluate the generalizability of the obtained results. It can be speculated though that the finding of less satisficing in a face-to-face than in a web survey will generally be retrieved in other experiments targeting other study populations. As long as web surveys preclude nonverbal communication, foster multitasking, and add difficulty to the task of responding because of maintained predominant reliance on visual rather than aural cues, the satisficing likelihood will probably remain higher in web than in face-to-face surveys, regardless of the specific study population. This does not imply that the characteristics of the population involved in the mixed-mode survey would be irrelevant.

10 Response Quality in Web Surveys 845 The discrepancy in data quality between survey modes can be expected to increase as the average ability and motivation levels in the study population drop. Satisficing may increase in both survey modes as these levels decrease, but the added difficulty of a web survey both cognitively and with respect to familiarity with the Internet may cause the satisficing propensity to rise more rapidly in this survey mode than in a face-to-face survey. The current experiment targeted relatively young and highly educated people who are presumably comfortable with using new technologies such as the Internet. Other samples, which could be more representative of the general population, might include older people who may be less computer literate and hence have more difficulty with web surveys (Couper and Rowe 1996). In such samples, the mode differences might be larger than those in the current experiment, but the direction of the effects would probably remain the same (i.e., more satisficing in the web than in the face-to-face mode). As explained, the practical combination of a web and a face-to-face survey should probably be seen in the context of a sequential mixed-mode survey design, in which the web survey comes first and after which other survey modes are deployed to gather the responses of the remainder of the sample. The last stage in such a design could be a face-to-face survey. The current study wished to point out some differences between these survey modes so that researchers contemplating on using such a mixed-mode design would be aware of the risks that are involved. Notwithstanding its limitations, the current experiment suggests that important differences in data quality between a web and a face-to-face survey exist. This indicates that the methodological strategy of mixing these two survey modes should be further explored and refined, and also that the web survey design in itself should be further improved. For instance, methodologies to increase the web survey response rate, to decrease the DK responding tendency and nondifferentiation on rating scales, and to lower the item nonresponse rate in web surveys should be further explored. This constitutes an important research agenda for research on web survey and mixed-mode survey methodology. References AAPOR Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 3rd ed. Lenexa, KS: AAPOR. Bates, Nancy Internet versus Mail as a Data Collection Methodology from a High-Coverage Population. Paper presented at the AAPOR Annual Conference, Montreal, Canada, May 17 20, Bowling, Ann Mode of Questionnaire Administration Can Have Serious Effects on Data Quality. Journal of Public Health 27: Cannell, Charles F., Peter V. Miller, and Lois Oksenberg Research on Interviewing Techniques. In Sociological Methodology, ed. Leinhardt Samuel, pp San Francisco: Jossey-Bass. Cialdini, Robert B Influence. How and Why People Agree to Things. New York: William Morrow and Company.

11 846 Heerwegh and Loosveldt Couper, Mick P Web Surveys. A Review of Issues and Approaches. Public Opinion Quarterly 64: Couper, Mick P., and Benjamin Rowe Evaluation of a Computer-Assisted Self-Interview Component in a Computer-Assisted Personal Interview Survey. Public Opinion Quarterly 60: Curtin, Richard, Stanley Presser, and Eleanor Singer Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opinion Quarterly 69: DeRouvray, Cristel, and Mick P. Couper Designing a Strategy for Reducing No Opinion Responses in Web-Based Surveys. Social Science Computer Review 20:3 9. Dillman, Don A The Design and Administration of Mail Surveys. Annual Review of Sociology 17: Dillman, Don A Mail and Internet Surveys: The Tailored Design Method.New York: Wiley. Dillman, Don A., Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, and Jodi Berck Response Rate and Measurement Differences in Mixed Mode Surveys Using Mail, Telephone, Interactive Voice Response and the Internet. Draft paper retrieved February 28, 2006 from Fricker, Scott, Mirta Galesic, Roger Tourangeau, and Ting Yan An Experimental Comparison of Web and Telephone Surveys. Public Opinion Quarterly 69: Goyder, John, and Jean McKenzie Leiper The Decline in Survey Response: A Social Values Interpretation. Sociology 19: Groves, Robert M., and Mick P. Couper Nonresponse in Household Interview Surveys. New York: Wiley. Hochstim, Joseph R A Critical Comparison of These Strategies of Collecting Data from Households. Journal of the American Statistical Association 62: Holbrook, Allyson L., Melanie C. Green, and Jon A. Krosnick Telephone versus Faceto-Face Interviewing of National Probability Samples with Long Questionnaires. Comparisons of Respondent Satisficing and Social Desirability Response Bias. Public Opinion Quarterly 67: Hox, Joop J., and Edith D. de Leeuw A Comparison of Nonresponse in Mail, Telephone, and Face-to-Face Surveys: Applying Multilevel Models to Meta-analysis. Quality and Quantity 28: Kaplowitz, Michael D., Timothy D. Hadlock, and Ralph Levine A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly 68: Krosnick, Jon A Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology 5: Krosnick, Jon A., Allyson L. Holbrook, Matthew K. Berent, Richard T. Carson, W. M. Hanemann, Raymond J. Kopp, Robert C. Mitchell, Stanley Presser, Paul A. Ruud, V. K. Smith, Wendy R. Moody, Melanie C. Green, and Michael Conaway The Impact of No Opinion Response Options on Data Quality. Non-attitude Reduction or an Invitation to Satisfice? Public Opinion Quarterly 66: Kwak, Nojin, and Barry Radler A Comparison between Mail and Web Surveys: Response Pattern, Respondent Profile, and Data Quality. Journal of Official Statistics 18: McCarty, John A., and L. J. Shrum The Measurement of Personal Values in Survey Research. A Test of Alternative Rating Procedures. Public Opinion Quarterly 64: Tarnai, John, and Don A. Dillman Questionnaire Context as a Source of Response Differences in Mail and Telephone Surveys. In Context Effects in Social and Psychological Research, eds. N. Schwarz, and S. Sudman, pp New York: Springer. Tourangeau, Roger, Lance J. Rips, and Kenneth A. Rasinski The Psychology of Survey Response. Cambridge: Cambridge University Press. Voogt, Robert J. J., and Willem E. Saris Mixed Mode Designs: Finding the Balance between Nonresponse Bias and Mode Effects. Journal of Official Statistics 21: