Constructing a Qualtrics survey Before you = start

Transcription

1 Data Science Tutorials Constructing a Qualtrics survey Before you = start César Zamudio Assistant Professor of Marketing & Entrepreneurship Kent State University

2 Before you start Qualtrics is an extremely powerful, Web-based tool that will allow you to: Design a Web-based survey; Give you the opportunity to distribute such survey massively, as you will have a Web link to your survey; Allow you to perform basic analysis of the survey results; and Allow you to produce basic graphics of the survey results. HOWEVER: Qualtrics won t guarantee you determine an adequate sampling frame; Qualtrics will not review the structure of your survey; Qualtrics will not review the spelling and grammar of your survey; Qualtrics will not interpret the survey results for you; and Qualtrics will not tell you what graph best suits which type of data In other words, Qualtrics DOES NOT SUBSTITUTE FOR your knowledge of how to design, test, and distribute a survey, analyze, interpret and communicate its results. As such, it is an extremely powerful, Web-based tool. It is your responsibility to use it adequately, professionally, and ethically. With great power comes great responsibility Consequently, before starting the Qualtrics tutorial proper, let s begin by pointing out some good habits and basic design issues when it comes to constructing your survey. Again, these are independent of Qualtrics. Yes even comics make the point of this slide clear!

3 First habit: Define research objectives Research objectives are narrow, specific issues that marketing research can help understand. Generally, each research objective should be answered with one or a couple questions per research method. We have three methods available, which are generally used in this order: Secondary data search: Gathering of information to solve a research objective through the use of databases, Internet searches, review of trade and academic articles, and so forth, to paint a broad picture of our current situation; Primary data Focus groups and interviews: Gathering of information by talking personally to one (interview) or multiple (focus groups) individuals. Analysis of the data is qualitative, and sampling is generally not probabilistic or random. This helps asses consumers needs and wants in a broad way; Primary data Surveys: Gathering of information by distributing surveys personally or via the Internet, with the possibility of a researcher administering them. Analysis of the data is quantitative and samples can be probabilistic or random. This helps zoom in into consumers specific needs and wants better, and generalize to the population. In order to plan ahead, match your research objectives to the research methods we have. For example, suppose you have four research objectives. Those may be matched to methods like so: RO1: Determine consumers current and last year s willingness to pay RO2: Determine consumers reasons for consuming salty snacks RO3: Determine consumers reasons for stopping to exercise RO3: Determine consumers set of considered salty snack options Secondary data Focus groups/interviews Surveys As can be seen, here the four objectives require the use of different methods. RO2 is so broad that will be addressed with all methods, in turn; RO3 is quite sensitive, so only FG/interviews may be helpful to get consumers talking about this issue; RO1 would be assessed quantitatively through surveys only; and RO3 with both FGs and surveys. A structure such as the above clarifies which method will be used to solve which objective and in what order.

4 Second habit: Structure appropriately A logical flow of information will always help respondents better understand and answer your survey. A flow such as the following is adequate, in a generic example of a survey about a product you re interested in: Greeting Broad questions about product category Specific questions about each product in the category Psychographics questions (hobbies & lifestyle) Demographics (optional) questions Debriefing and thanks

5 Data Science Tutorials = Constructing a Qualtrics survey César Zamudio Assistant Professor of Marketing & Entrepreneurship Kent State University

6 First things first: Qualtrics account Before learning how to put together Web surveys using Qualtrics, you need to obtain a Qualtrics account. To do so 2 3 Step 1: Visit Step 2: Input your credentials: your KSU username only (NOT your full ) and KSU password. Step 3: Click Log In and follow the steps to confirm you re a student. Do not

7 1 To use Qualtrics, you can use its multiple tabs and options. The Qualtrics environment Tabs: These tabs present the activities you have available, and range from looking at a list of your surveys and their status, to distributing a survey, view results, etc. Options: What I call Options is a set of buttons within each tab. They allow you to perform tasks related to that tab. For example, in the My Surveys tab, you will be able to create a survey, it, etc. Tabs can be useful for navigation but, once you have created one or several surveys, it is best to use what I will call Survey Buttons to navigate through each survey. As you can see, there is one set of Survey Buttons per survey. To create your first survey: Step 1: Click Create Survey in the Options panel. Options Survey Buttons Tabs

8 Different ways to create a survey There are a few ways you can start a survey, and the choice is yours: Quick Survey Builder: Start a new survey from scratch. Ideal option for class. Create from Copy: Creates a new survey based on a previous one. Useful if you want to back up existing surveys before extensive modifications or if you fear the possibility of overwriting and losing information on the existing survey. Survey Library: Start from an existing Qualtrics template. Many are available. 1 You re meant to construct your own survey from scratch. This will allow you to customize it extensively. Step 1: Click on Quick Survey Builder. Step 2: Assign a name to your survey. You may create a folder to keep your surveys well-organized. Click Create Survey to proceed to Edit Survey environment. 2 Happens all the time in Qualtrics. Watch out!

9 The Edit Survey environment When creating a survey, or clicking on its name in the Qualtrics environment, you reach the Edit Survey environment. This environment will allow you to design your survey through its multiple options here are the most relevant: Look and Feel: Change colors, themes and fonts for your survey. Ideally done as the last step in the survey design. Survey Options: Change multiple survey details, such as allowing respondents to back-track and change answers, customize end-of-survey messages, allow respondents to answer only if invited, etc. Rarely used. Survey Flow: Extremely important option. Allows you to modify the order in which consumers see questions and to show entire groups of questions randomly to conduct survey experiments. Preview Survey: Allows to preview the survey and answer it as if you were a respondent. Useful to test whether the options and the logic of the question work. By logic, we mean how the Web survey reacts to respondents questions. For example, suppose your survey is meant only for females. By introducing Skip Logic on a question on gender, you will be able to send male respondents to the end of the survey, filtering them out. More on this in the next tutorial!

10 Edit Survey environment areas Getting ahead of ourselves a bit: here is how your Edit Survey environment will look when you add the first question to your survey. Notice there is now a column of tools to the right. To more easily understand Qualtrics, it is useful to think of the Edit Survey environment as consisting of four main areas: 1. Tabs and options: Use to move from survey to survey, or to view, modify, or alter the survey in general. 2. Advanced Options: Use to add Blocks, which can hold multiple Items. A survey consists of multiple Blocks, each of which holds multiple Items. Each Item is a question or picture in the survey. 3. Main window: Use to create new Items or to modify the text inside them. 4. Toolbar: Use to MODIFY Items as a WHOLE: adding or removing scale points, changing the item to be multiple choice or not, copying an item, etc.

11 The block metaphor To construct a survey, you need multiple sections, as outlined in the Before you start slides. Each section of your survey corresponds to a Block on Qualtrics. Consequently, if your survey design consists of 5 sections, your Qualtrics survey should have 5 blocks. Suppose, now, that your survey, which consists of 5 sections, has 3 questions each. Then, you need 5*3 = 15 Items, or questions. Plus, you need two more blocks and two more items: one for the Greeting, another for the Debriefing. This makes the survey 7 blocks total and 17 items total. An example of such a structure for a new product (a shampoo, NoBald) may be: BLOCK 1: Greeting Item 1: Greeting text BLOCK 2: Questions about product category Item 2: Do you wash your hair using shampoo at least once a week? Item 3: How often do you purchase products from the health and beauty aisle at your grocery store? Item 4: Please indicate your agreement with the following statement: I purchase health and beauty products on impulse. BLOCK 3: Questions about shampoos Item 5: Which three brands of shampoo first come to mind? Item 6: Please rate the following shampoo brands on the attributes shown. Item 7: Please rank the following shampoo brands in order of preference. BLOCK 4: Questions about the new shampoo product Item 8: Above is the logo for a new shampoo product called NoBald. Please indicate your agreement to the following statements regarding this new product. Item 9: How likely would you be to purchase a 20 oz. bottle of NoBald? Item 10: How much would you be willing to pay to purchase a 20 oz. bottle of NoBald? BLOCK 5: Psychographics questions Item 11: What are your hobbies? Check all that apply. Item 12: Which among the following media sites you visit on a daily basis? Check all that apply. Item 13: Please indicate your agreement to the following statements regarding your opinion on current events. BLOCK 6: Demographics questions Item 14: What is your gender? Item 15: What is your age? Item 16: Please indicate your current income. BLOCK 7: Debriefing Item 1: Debriefing text Note: If you need an IRB Consent Form (e.g., for graduate students conducting research) you should place a Block 0 that includes such Consent Form. Then, those who do not accept the Consent should be FILTERED (see slides on Qualtrics Logic)

12 Set up your survey s block structure Qualtrics starts you with one empty block. We need to edit the first one, and add more as needed. Step 1: Select the name of the Block ( Default Question Block ). Step 2: Change the name of the Block to a more appropriate name as in the example earlier. Step 3: Add another block, and repeat Steps until you have added all the blocks needed

13 1 Resulting block structure Once you have added your survey Blocks, your structure should look just like the picture on the left. Each Block is named, but does not contain any Items yet. Note, also, that the structure occupies a lot of space on the screen. Another advantage of the Block structure is that these can be collapsed, allowing you to focus only on the Block(s) you re currently working on. To collapse the blocks: Step 1: Click on the down arrow to the left of each Block s name to minimize them. Now that your Block structure is complete, you now need to start adding items! In the following, we will discuss how to add an item, then details on each item type in Qualtrics as well as how to use them from a technical standpoint. For a primer on measurement, you may want to read my class notes.

14 Adding Items to a Block Step 1: In the Block you want to add an Item to, click Create a New Item. A generic question appears. 2 1 Along with the question, you will see that the Toolbar (column to the right) now appears. This will allow you to modify your item in various ways. Step 2: To modify any text you see in your new Item (e.g. question wording, choices, etc.) click any text you can see. Step 3: To modify how your Item looks and the options available to respondents, use the Toolbar (more on this soon). 3

15 Adding breaks between questions (Page Breaks) For each Block, respondents will see a single Web page with all Items contained within each Block. Sometimes this is not ideal: you may want respondents to see a couple questions at a time. To separate the Items within a Block such that respondents have to click a button to proceed from one group of Items to the other, we use Qualtrics Page Breaks. Step 1: Select the Item after which you want to insert a Page Break. Step 2: In the Toolbar, select Page Break. Once you click Page Break, a page break will appear as shown below. Note that you can add another item, or remove the Page Break, with the plus and minus buttons to the right of the Page Break. 2

16 Respondents screen without page break

17 Respondents screen with page break Page 1 As can be seen, Page Breaks help subdivide your questions as shown to respondents so they do not see too many questions in the same page. For each Page Break you introduce, a button will be generated. The button must be clicked in order to advance in the survey. NOTE: If you want EACH QUESTION to be in a SINGLE PAGE (i.e., a Page Break in between each question), click on Look and Feel in the Options toolbar (the rainbow icon ) and specify 1 question per page! Page 2

18 Changing Item text directly Option 1: Change the text by clicking on existing text and writing new text: Step 1: Click any text you see within an Item. Step 2: A blue box surrounds the text. Type any new text. When finished, click anywhere outside the text. There are some details you must know when editing text in this way: 1 If you copy and paste text from another source, such as another survey, the Internet, etc. the font size and color from the source will TRANSFER to the question. So, you may still need to manually edit the text (discussed in the next slide). Qualtrics is smart: when you write a question, it will automatically insert choices that are most appropriate for your question. For example, notice that, in the shampoo question above, Qualtrics automatically changed the number of choices and the text in those choices to two options: Yes and No! This may not be what you want, so you may need to further make changes using the Toolbar (discussed shortly). THERE IS NO UNDO BUTTON IN QUALTRICS. If you accidentally overwrite or modify incorrectly the text within an Item, you may lose it forever. Be careful or 2

19 1 Changing Item text with Rich Content The Rich Content Editor will allow you to modify the text of a question just as in a text editor such as Microsoft Word. This is useful is you want to highlight portions of the text with bold or italic font, color, and so forth. To edit text using Rich Content: Step 1: Click any text you see within an Item. 2 Step 2: A text editor window appears. Write your text. Then, select any text you want to modify, and use the buttons on the text toolbar to change the text. When finished, click anywhere outside the text box. The text of the question will appear updated as you edited it. Note that the edition of the text in this example clearly is not how a survey question should look like: the Rich Content editor should not be abused. Use it very sparingly, and only whenever you want to highlight relevant information for respondents, as in the example in the next slide.

20 Example of good Rich Editing (Consent Form)

21 Changing choice/scale item text Individual choices and statements within an Item can be modified just as question text. THIS INCLUDES SCALE POINT LABELS ANY TEXT can be edited. Step 1: Click any text you see in a choice or statement. Input text as desired. You may want to either edit using the Rich Text editor, or allow respondents to insert their own text. To do so: Step 2: Click the blue arrow to the right of the text in the choice or statement. Step 3: Click Rich Content Editor to edit using Rich Content. After editing click anywhere outside the text. Step 4: If you want to allow respondents to enter their own text (such as if you need them to potentially specify brands they may consider, but that are not in the list you show them) click Allow Text Entry. A text box will appear that respondents will be able to use. 1 2 Let us now discuss the most common types of Items and how to use them in your survey! 4 3

22 Selecting an Item type As we studied earlier, once you create a new Item, a generic, multiple-choice question will appear. To view the range of different Item types you can use: 1 2 Step 1: If your block is empty, click Create a New Item to display a generic question. Step 2: Click the green dropdown to the right of Create New Item. Step 3: Select any of the Item types to change the current Item into that type. In this tutorial we will cover the types highlighted in green. IMPORTANT NOTE: To add Items above or below, click the plus signs; to delete, click Minus. 4 3

23 Question 1: Text input When to use: When you need respondents to give an open-ended response. It can be short or long. Step 1: Create an Item. Choose or from the Create Item dropdown from the Toolbar to the right. The result should be the following. You can vary the text entry size by using the Toolbar: Step 2: For single line responses, select Single Line from the Toolbar. This is the default option that Qualtrics assigns. Step 3: For responses that may involve a couple lines (e.g., 3), select Multi Line from the Toolbar. Step 4: For longer responses, such as descriptions of an image, short essays, opinions, elicitation of words, etc. select Essay Text Box from the Toolbar

24 Question 1: Text input (example) Single Line Item Multi line Item Essay text box Item

25 Question 2: Number input When to use: When you need respondents to state a number, such as their age, willingness to pay, etc. Step 1: Create an Item. Choose This item should be a Single Line item. That s all you need. However, you still need to specify the bounds of the Item: Step 2: From the Toolbar, select Content Validation. This means that Qualtrics will review the respondents input and may require them to reenter it if it doesn t satisfy a condition in this case, bounds (min and max). A question with validation will have this icon next to it: Step 3: From the list of validations available from Content Type, select the Number validation. Step 4: Specify the minimum bound for the number. For example, people can t respond negative ages, so you may choose 0. Step 5: Specify the maximum bound for the number. For example, for an age question, a reasonable bound is

26 Question 2: Number input (Example) If respondents input a number below or above bounds, they ll get an error message.

27 When to use: When you need respondents to describe a picture. Useful for English checks. Step 1: Create an Item. Choose or from the Create Item dropdown Step 2: Create a second Item immediately below. Choose The result should be: Step 5: Select the Text Entry Item you just created. In the Toolbar to the right, select Essay Text Box. Changing this will cause the text entry area to be bigger. Question 3: Picture + Text 4 from the Toolbar to the right. 3 Step 3: Click this link and follow the steps to upload a graphic. Past graphics will remain in Qualtrics and you can use them later. Step 4: Write the instruction that respondents should follow, for example, Please describe the picture above in two full sentences. Notice that respondents have very little space to answer. To fix this:

28 Question 3: Picture + Text (example) Note that you can construct questions using Picture + Number, Picture + Multiple Choice, etc. Whenever you need a visual stimulus of any sort, use this general approach.

29 Question 4: Multiple choice When to use: When respondents should be given the option to pick one, several, or all items from a group of items. Step 1: Create an Item. Choose from the Toolbar to the right or from the Create Item dropdown. 2 3 Step 2: If you want respondents to select only one option from the group, select Single Answer from the Toolbar. Step 3: If you want respondents to select multiple options from the group, select Multiple Answer from the Toolbar. Step 4: If you want to force respondents to answer before moving on with the survey, select from the toolbar. The question will now have the icon next to it.

30 Question 4: Multiple choice (Options) You can modify some options from Multiple Choice questions. Number of choices: In the Toolbar, you can increase/decrease the number of choices in the group to show respondents: Automatic choices: In the Toolbar, you can select from pre-constructed question types, such as Disagree-Agree, Unlikely-Likely, etc. by clicking on Automatic Choices. Make sure you pick the number of choices first! Position: In the Toolbar, you can choose between Vertical, Horizontal or Column arrangement for the group of choices: Column Horizontal Vertical

31 Question 4: Multiple choice (Examples) Multiple answer, 6 choices specified by the researcher, with text entry and forced response, column layout. Single answer, 5 choices (automatic Disagree-Agree), horizontal layout. Single answer, 7 choices (automatic Unlikely-Likely), forced response, horizontal layout.

32 Step 1: Create an Item. Choose Question 5: Scale When to use: When respondents should answer multiple statements about a similar subject. These groups of statements are known as scales. For example, you may have five statements about attitudes towards a new product that you will later average to determine consumers attitude. Scales are always composed of SINGLE ANSWER Items! from the Toolbar to the right or from the Create Item dropdown. The generic question above will be created. The Statements are that which the respondent will judge or opine on; the Scale points are the range of choices you will give them for each Statement. You can use Automatic Statements and Automatic Scale points just like you did in the Multiple Choice Items (Question 4). Modifying number of Statements: In the Toolbar: Since this is a scale, you may want to add a number of Statements at least three or five. These should all attempt to measure an underlying dimension such as attitude towards a product, brand or service. Modifying number of Choices: In the Toolbar: An adequate number of Scale points should be 5 or 7. This will give enough latitude to the respondent. Note that the odd numbering is chosen so that respondents can have a neutral option. If you wish to force them either way, use an even number of Scale points.

33 Question 5: Scale (Examples) Matrix Table, 5 Statements defined by the researcher, 5 scale points (automatic Disagree-Agree) Matrix Table, 3 Statements defined by the researcher, 7 scale points (automatic Unlikely-Likely)

34 Question 6: Semantic Differential When to use: When respondents should answer multiple statements about a subject, and your objective is to discover its meaning. Ideally, you would have respondents rate a number of subjects for example, three brands along the same dimensions (Statements). Respondents will assign meaning by selecting points between two bipolar adjectives. Step 1: Create an Item. Choose Step 2: Choose The following Item template appears: from the Toolbar to the right. Step 3: Fill in negative adjectives to the left (e.g., unsightly, unhealthy, slow, dirty) from the Toolbar to the right or from the Create Item dropdown. 3 4 Step 4: Fill in positive adjectives to the left. (e.g., attractive, healthy, fast, clean) Step 5: Select the number of Scale points for the semantic differential. An adequate number is 7.

35 Question 6: Semantic Differential (Example) Matrix Table (Semantic Differential), 2 Brands (each a different Item), 4 Adjectives (Statements) per brand

36 Question 7: Ratings question When to use: When respondents should rate multiple brands along multiple dimensions, and your objective is to assess how brands compare in every dimension. Step 1: Create an Item. Choose from the Toolbar to the right or from the Create Item dropdown. Step 2: Choose Text Entry from the Toolbar to the right. You may have to click More below. The standard question changes: Notice that now instead of choices, respondents will have to enter a value (numbers) in each cell to rate each brand along each dimension. Validating numbers: Use the same approach as in Question 2 (Number Input): Select Select from the Toolbar; Content Type from the Toolbar; Specify the minimum at 1, and the maximum at 10:

37 Question 7: Ratings question (Example) Matrix Table (Rating question), 5 Brands, 5 dimensions per brand If any number in the question does not lie within the range, the respondent will receive an error message. You can request Force Validation so respondents have to fill in every cell correctly. Otherwise some may not be filled.

38 Question 8: Rank-ordering When to use: When respondents should rate multiple objects (e.g., brands) on a descending order. Note that you could simply have a rating question and it will give you more information because respondents will assign ratings, which tell you something about the distance between the objects, instead of rankings, which do not. Step 1: Create an Item. Choose A template appears: 3 from the Toolbar to the right or from the Create Item dropdown. Step 2: Select the number of objects the respondent will rank with the Toolbar. Step 3: Fill in the objects by editing the text. 2

39 Question 8: Rank-ordering (Examples) Rank order item, 6 Brands

40 Question 9: Constant sum When to use: When respondents should assign importance or weight to multiple possible objects. For example, you could ask how important for respondents different components of a computer are, or the extent to which students believe that their professors, or them, are responsible for their grades. Respondents will distribute tokens, or points, that represent such importance. Step 1: Create an Item. Choose A template appears: from the Toolbar to the right or from the Create Item dropdown. Step 2: Select the number of objects the respondent will compare with the Toolbar. Step 3: Fill in the objects by editing the text. Step 4: Decide the total amount that respondents must distribute using the Toolbar. Typical amount is 100 points. Step 5: You may want to change the way the constant sum Item is represented using the Toolbar (Choices, Bars or Sliders)

41 Question 9: Constant sum (Examples) Constant sum (computer components), 3 components, 100 total points Constant sum (insurance), 2 objects (agent and company), 100 total points Note the instructions in this question are far more detailed than in the previous one. Also, the orange warning to the left indicates the question may be too long (horizontally) for mobile devices.

42 Data Science Tutorials Questions? = Set up accounts Create HIT Manage and Pay