Reflections on using Qualtrics for Course Evaluations EFS & ALC Summer Jennifer DeSantis August 28, 2013

Transcription

1 Reflections on using Qualtrics for Course Evaluations EFS & ALC Summer 2013 Jennifer DeSantis August 28, 2013 Summary: Using Qualtrics for course evaluations has overall been a success. There are several advantages to Qualtrics relative to paper evaluations, though there are some disadvantages as well. The consensus of the administrative staff and instructors seems to be that the benefits markedly outweigh the disadvantages. Furthermore, the disadvantages we have found of Qualtrics are generally comparable to those inherit in any system of electronic evaluations. Below, I provide a more detailed discussion and suggestions to consider. Step-by-step instructions on how to build the surveys are provided in a separate document. Qualtrics Overview: For each survey, unique links for each student are sent via . o The individual link allows the survey to be completed only once via that link. This does not prevent the student from sending the link and allowing someone else to complete the survey, but it does mitigate this problem by allowing the link to only be used once. o Students receive one per each survey link that is sent to them. o This type of link is the default option when sending surveys via the Qualtrics Mailer (i.e., the Surveys page). Anonymous Responses. There is no way to know who submitted responses. This box must be checked under Survey Options when using individual links. A second way to ensure that students do not complete a survey more than once (a redundancy when using the unique links) is to select the option for Prevent Ballot Box Stuffing in Survey Options. However, this can be circumvented by students who use a different browser, computer, or who clear their browser s cookies. Responses are kept hidden from instructors. The only individuals who can see survey results are the user that created the survey, and collaborators with whom the survey has been shared. Response data are downloaded in one click; no reformatting is necessary. Note: There are other options for survey distribution and anonymity, but my research showed alternatives to be less ideal. For example, the Anonymous Link method allows anyone with the link to access the survey and submit it (i.e., a student could share this link with any number of other individuals who could use it). The Anonymous Link method also records IP addresses, whereas setting the Survey Options to Anonymize Responses does not. 1

2 Qualtrics vs. Paper: Advantages and Disadvantages: Qualtrics is a favorable alternative to paper, from the following perspectives: Instructors: eliminates wasted class time to complete evaluations. Administrative staff: evaluation results can be downloaded in a usable electronic form instantly, through just one click. No calculations or reformatting of the responses are necessary. Disadvantages to using Qualtrics rather than paper course evaluations is: Inability to secure a 100% response rate from the class, and to know which students have not responded. Students may be less confident of their anonymity. There are several opportunities for administrative/technology support staff to make errors when creating and ing the surveys (e.g., mislabeling of links, sending a survey to the wrong group of students). These errors compromise the validity of the responses that come in, and could potentially render the responses useless. On paper, however, there is confidence that responses match the course properly and that students will not encounter any errors when submitting. With paper evaluations, staff errors are more likely to affect how the responses are processed, which can always be redone and rectified if the physical surveys are kept on file. Detailed Discussion of Qualtrics for Course Evaluations: Creating Surveys & Distributing to Students: Evaluation surveys are best created by making copies of templates that were saved in the previous year. It is important that survey options are changed, in addition to any survey text, before making copies of the survey. The interface for making edits to surveys is intuitive and easy to learn. The controls for distributing surveys provide several options and could be confusing to someone who does not know the appropriate distribution method to use. Staff can consult the step-by-step instructional guide created in 2013, which provides this information. Current class lists must be located in order to create panels of s that will be used when distributing surveys to each section. I was added as a member to all EFS & ALC CourseWork sites for this purpose, I copy-pasted each student s address into an Excel spreadsheet according to section. Then, I copy-pasted these groups of s into Qualtrics panels. This worked, but it was time-consuming and very vulnerable to user error. o It would be more efficient and less prone to mistakes if we could find any other electronic locations of current class lists. Note that these lists need to only include addresses (not names). Before survey distributions can be scheduled, surveys must first be activated. 2

3 ACCOUNT LIMITS: Stanford student accounts are limited and cannot have more than two active surveys at a time. o Ken s account, however, is unlimited. He must be relied on to activate all of the surveys for both EFS and ALC. This means that all surveys must be shared with him and he must be involved before the survey mailings can be scheduled. Stanford student accounts cannot have more than twenty surveys at one time that they have created or copied. There are, however, no limits (that I could find) on the number of surveys that are shared with these accounts. o Student accounts will have enough room for all the EFS evaluations, but will not have enough room for all the ALC evaluations. Ken must be relied on to create several ALC surveys. Ken must also share these with the student and any other collaborators (due to sharing permissions). LIKELY AREAS FOR USER ERROR: Any users creating or distributing surveys should be extra mindful of these steps: Survey labeling. Surveys are ideally labeled in three places, and these all should be double-checked for accuracy: in the text of the survey above the instructions; in the body of the message above the survey link; and in the subject line of the distribution . Panels of class lists. Copy-pasting from CW to Excel, then from Excel to Qualtrics is not ideal, though it can work. This is a critical step that must be at least double-checked. Survey Expiration dates. This is a setting for each survey, found under Survey Options. It is important that it is updated to the current year s start and end times for every survey. Tip: to prevent the need to update this in every survey manually, the student should make this change to a copy of the template first, before making multiple survey copies for each section. Sharing Surveys & Monitoring Results: Evaluation surveys can be shared with collaborators. Sharing allows collaborators to view and edit the survey, activate and distribute it to students, and view and download responses. o The only individuals that can see a survey or its results are those with whom the survey has been shared. Instructors cannot see the evaluation results. Permissions to share a survey with others is given only to the account under which that survey was created. o Anyone who wants to share a survey but does not have permission to (because the survey was shared with them by a different user), must first copy the survey. Copies of surveys act like surveys that are created by the user, and are able to be shared by that user. Copies of 3

4 surveys will not include any response data that may have already been collected from the original survey. Therefore, if a survey and its response data must be shared, the user who created the survey must add the collaborators. Downloading & Distributing Response Data: Response data for each course evaluation are downloaded independently of one another. In other words, response data from one evaluation will not include data from any other evaluations. Downloaded responses for each survey includes: each student s exact response to each item, as well as statistics on each item (including number of responses and, where applicable, bar graphs, means, and ranges are also provided). Options include downloading to Word, Excel, PPT, and PDF. o We found that pdf provided the best readability. o However, pdf s show each question on a new page. This means pdf files were over 30 pages. This is more a nuisance than a technical problem; it can also be circumvented by saving the file to Word and then deleting any empty spaces. Summary of Considerations for Future Years: Topic: Issue: Considerations: Class Lists Vulnerable to user error. Not efficient to copy-paste each address from CourseWorks. Maintain an electronic spreadsheet of current class lists with student addresses. Account Limits: Survey Creation & Activation Sharing Permissions Under Stanford student user accounts: No more than twenty surveys can be created at a time. No more than two surveys can be active at a time. Permission to share a survey is provided only to the user that created that survey. Perhaps the student creating the surveys could be given a new Qualtrics account that has the same permissions as Ken s account. Otherwise, Ken will need to continue do some of the leg-work. The student should begin by making a copy of last year s template so she can add collaborators to all the evaluation surveys. 4

5 Student Response Rates There is no way to know who submitted a response. Survey collaborators can monitor the number of responses as they come in for each survey. Class sections can be incentivized to see their grades earlier if they have 100% response rate. Reminder s can be sent to all students every day, and to particular sections. Instructors can make announcements in their courses. 5