End of Summer Usability Study Page 1 Table of Contents: Update Timeline of Events Page 2 Original Project Proposal & What s Changed Page 3 Summary of Usability Survey Results Page 4 Hallway Usability Testing Page 6 Opinion: Where to go from here Page 10 Update Summary: Based on the attached documentation, I propose the following: Ideas to explore/decision items Increased use of notifications protocol to save users from having to check my.evergreen.edu for information Exploration of the idea of Tabs can we implement them in a way that will improve navigation and usability? Reorganizing the content of the front page of my.evergreen.edu according to office, with sub-items in alphabetical order (ala Plaza mockup) Descriptions of link detail (ala Kiwi mockup) Increased or more visible help documentation In this update, the backstory is on pages 1-9 and the good stuff is on page 10
Timeline of Events: 5/6/09: Met with ITCH for approval of Usability Study proposal 5/20/09: Met with My.evergreen.edu Owners for approval of Usability Study proposal 5/20/09: Usability Survey posted on my.evergreen.edu 8/12/09: Met with My.evergreen.edu Owners to discuss results of the Usability Survey and change focus of Study to gathering specific stories 8/20/09: Met with BUG to discuss Usability Survey results and shift to story gathering 8/26/09 : Gathered stories about my.evergreen.edu & hallway tested Kiwi and Plaza mockups Page 2 What s hallway testing? Rather than using an in-house, trained group of testers, just five to six random people, indicative of a crosssection of end users, are brought in to test the software; the name of the technique refers to the fact that the testers should be random people who pass by in the hallway. -From Wikipedia Kiwi Plaza The Kiwi mockup was originally developed by the Web Team. The Plaza mockup was originally developed by Lee Gillentine (with help from Miles Franzoni). What s with the funny mockup names? I had to name the mockups so I could keep track of them in my notes.
Page 3 Original Proposal & What s Changed: From Original Proposal: Scope My.evergreen.edu is the primary method of interaction with the administration for many students. The services and information provided by the site must be easily accessible. Towards this end, a usability study is proposed. The study will assess the current level of usability and facilitate suggestions for improvement. The study will conclude with a reorganization of the front page of my.evergreen.edu based on the suggestions of students participating in the study. A report will be prepared suggesting future projects to improve my.evergreen.edu Purpose The purpose of this usability study is to improve the overall student satisfaction when using my.evergreen.edu. The study has two parts: 1) An online survey given to all students assessing the current usability of my.evergreen.edu and facilitating ideas for the future. 2) As the data from the survey is aggregated, solutions to usability issues will be developed. In the fall, usability testing will be conducted on these solutions through a series of focus groups. Timeline The online survey will be posted towards the end of Spring quarter, and will stay up until the beginning of Fall 2009. This is an ideal window of time for the survey, since most students must visit the site to register for classes and submit evaluations. Posters will also go up around campus advertising the survey. Starting in Summer 2009, data from the survey will be processed as it comes in, and general usability issues with the site will be determined. During this time, improvements to the site in response to usability issues will be developed. In Fall 2009, focus groups of around five students will be held in either an open computer lab in Academic Computing or the Administrative Computing training lab. The students will test the site s improvements and give feedback. What s changed? No posters went up. Instead, email blasts from Art Costantino (5/26/09) and on TescCrier(6/5/09) went out with a link to the Usability Survey At the 8/12/09 meeting with My.evergreen.edu Owners, the study was expanded to explore hallway usability testing and story gathering through individual interviews with users At the end of Fall quarter, feedback from the focus groups and online survey will be used to develop a reorganized and more usable front page for my.evergreen.edu.
Summary of Usability Survey Results: Page 4 Results as of 9/23/09 Out of 231 respondendents, 222 (96%) were students The current results are compared to the results released mid-summer and presented 8/12/09 to the My.evergreen.edu Owners. Hardest to Easiest Tasks (Scale of 1 to 5, mean=3) Value: See how close I am to graudation 2.75 Find job or internship opportunities 2.77 See class schedules 2.95 See my financial aid awards (harder than mid-summer results) 3.27 Register for Classes (eaiser than mid-summer results) 3.28 View holds, unpaid tuition, and academic alerts (harder than mid-summer results) 3.44 Read past self-evaluations (easier than mid-summer results) 3.46 Pay tuition and fees (easier than mid-summer results) 3.56 Vote 3.8 Take a survey 4.06 Most Difficult Task on My.Evergreen (same as midsummer) Percentage: Responses: Navigate the site 36.80% 85 Other 14.72% 34 Info. About Academic Status 15.58% 36 Find job or internship opportunities 10.82% 25 Info. About Financial Status 10.39% 24 None 8.23% 19 Most Common Use of My.Evergeen (same as mid-summer) Percentage: Responses: Registering for classes 41.56% 96 Finding out about my financial aid 24.24% 56 Other 18.18% 42 For information about my academic standing 8.23% 19 Class schedules 4.33% 10 Access of My.Evergreen Percentage: Responses: On Campus 32.03% 74 Off Campus 66.23% 153 Frequency of Access Percentage: Responses: More than once a month 41.13% 95 More than once a week 34.20% 79 Every day 9.52% 22 Only during registration and evals 7.79% 18 To register and find jobs/internships 4.76% 11 Only to find a job or internship oppourtunity 0.00% 0
Page 5 Most to Least Useful Features (Scale of 1 to 3, mean=2) Not Sure Percentage: Not Sure Responses Value: Campus Calendar (more useful than mid-summer results) 2.61 1.30% 3 Access to academic program resources (less useful than mid-summer results) 2.54 5.63% 13 Info. about student activities that interest me 2.11 3.46% 8 The ability to add customizable news feeds (less useful than mid-summer results) 1.7 9.96% 23 My @evergreen.edu email avail. in digest form 1.68 17.75% 41 Student-generated multimedia (more useful than mid-summer results) 1.67 9.09% 21 Percentage: Responses: Keep the site the way it is now 8.23% 19 Most to Least Important Improvements (Sale of 1 to 3, mean=2) Value: Information displayed in a more accessible way 2.75 Better organization of content 2.7 More intuitive menus and language 2.52 A more attractive design 1.55
Hallway Usability Testing: Page 6 Interview methodology: For everyone: The guiding question of the interview was What do you use my.evergreen.edu for? This question provided anecdotal evidence about usability issues and the role that my.evergreen.edu plays for users that could not be gleaned from the survey results. For students: After the guiding question was asked, students would be shown the Plaza and Kiwi mockups in a random order and be asked where they would click to complete a task. Students would then be asked to compare the two mockups, and I would also solicit feature requests. For faculty and staff: The discussion would revolve mainly around the guiding question. Users would also be asked to identify pain-points not just for their roles, but for all roles they had heard about. The mockups would be shown to users, and a compare/contrast discussion would follow. Feature requests would often appear in staff interviews. I think this is because staff spend more time thinking about my.evegreen.edu than students and faculty. In response to an informal interview (detailed below), I also observed staff interacting with students at the Registration & Records, Student Employment, and Financial Aid. Interview form:
Page 7 Interview Demographics: Over summer, fifteen in-person interviews were conducted, and a total of 22 people were interviewed. Most interviews were conduced one-on-one, but some interviews took place across whole offices or with more than one person. Interview Demographics: Pure students: 5 Pure staff: 12 Pure faculty: 1 Stu, staff, fac: 1 Stu, staff: 2 Staff, fac: 1 Total # Students: 8 Total # Staff: 15 Total # Faculty: 3 A bit about roles My.evergreen.edu has three primary roles (student, faculty, and staff), each associated with a view of the site. If a user has more than one role with the school (such as a student who is employed as a staff member), then they see the student view and the staff view of the site. Pure students, staff, or faculty is someone whose primary interaction with my.evergreen.edu is within a single role. A Pure user may have more than one role as recognized by my.evergreen.edu, but only interact with a certain view of the site. (such as a former student who is now a staff member) How many users do you need for successful hallway testing? Jakob Neilsen recommends 3 users from each category if testing three or more groups of users (http://www.useit.com/alertbox /20000319.html)
Page 8 Interview Results: Common uses of my.evergreen.edu for: Staff Faculty -Work certification -Work certification -Leave requests -Look up P.O.S. and budgets -Check class registration/enrollment report -Ordering books -Class status Students -Registering for classes -Finding out ( checking ) financial aid -Other -Information about academic standing -Class schedules (data from survey) Some of the tasks for faculty may be redundant I m not sure of the official terminology for faculty tasks on the site, so some tasks in the list may be repeated under different names Major Themes from Usability Interviews: The tabs on the Kiwi mockup garnered much attention. Multi-role users responded positively to tabs. Students responded positively to the spacing of the Plaza mockup, preferring Plaza over Kiwi in interviews. A large part of interaction with my.evergreen.edu for students and faculty is checking the site for information that is updated. Financial aid status is one of example of information that is checked for updates. Minor Themes/Feature Requests: Students and staff who have contact with students responded positively to the idea of more visible help documentation on the site. A feature request for a Getting Started tab was made in one instance, and a feature request for link details along the lines of some awesome details on the Kiwi mockup was also made. Definition of Major Themes: A major theme is something that came up in more than two interviews. There is a significant amount of bias in the major theme issues relating to the mockups, since all groups were shown the mockups and asked to talk about them. Definition of Minor Themes/Feature Requests: This is essentially anything else that came up in the interview. I have tried to organize these by related topics from the perspective of the user.
Page 9 More visible payment history was requested in a student interview. In a similar vein, a view of paychecks/paystubs was requested by a staff member, while a student view for lab fees was requested by a member of multi-media faculty. A faculty member requested a quicker way to monitor online registration. A member of Academic Advising requested an easier way for students to view enrollment reports. Enrollment reports, in this context, means a list of classes that are still accepting students. This would be helpful for newly admitted students and if a student has not yet registered near the end of their time-ticket. In a similar vein, a request was made from Registration & Records for students to be notified if they have been moved from a waitlist. A member of Academic Advising requested a more visible link to the waitlist preference form. Requests from both the student side and the Admissions office for an easier way to view academic history were made. In both a student and a staff interview, questions about the meaning of a link to student accounts were asked. The student accounts office and Financial Aid expressed concern about student confusion regarding Perkins Loans. Perkins Loans are part of Student Accounts, but since they are a form of aid, many students see them as part of Financial Aid. A staff member requested a link to my.evergreen.edu on the front page of evergreen.edu. In the comments of the survey, a student requested the same thing. Informal Interview Issue In an informal interview with a student (at a party), the student said they [the administration] direct you to my.evergreen.edu but never tell you what to do. They just tell you to go to the site, not what to click on. In response to this story, I spent some time observing staff interact with students and asking staff what they say when they tell students to go to the website. Based on my observations, I could find no evidence to support this usability issue. Good job!
Where to go from here: Page 10 On my end, I am planning on closing down the Usability Survey and posting the results to the Usability Study s blog. Users will still be invited to comment. The following items are already in Administrative Computing s Case Tracker: Faculty notification of Student Registration/Drop Students can view enrollment reports (to find a class with open slots for them to join if they are late registering) My.evergreen.edu-wide Search Function Better CRN Display Academic History Display I would like to suggest that the following item be added to Administrative Computing s Case Tracker: Student notifications around Financial Aid Also, an issue around the visibility of Tuition Due Dates has been addressed. Ideas to explore/decision items Increased use of notifications protocol to save users from having to check my.evergreen.edu for information Exploration of the idea of Tabs can we implement them in a way that will improve navigation and usability? Reorganizing the content of the front page of my.evergreen.edu according to office, with sub-items in alphabetical order (ala Plaza mockup) Descriptions of link detail (ala Kiwi mockup) Increased or more visible help documentation For me, I see the notifications protocol, Tabs, link details, and help documentation as places where it would be a good idea to think about the theming of the site. Most of the above ideas address the most difficult task on my.evergreen.edu (navigation) and the most important improvement (information displayed in a more accessible way). What about focus groups? 55 people indicated that they would be willing to participate in a focus group around the site. If we decide to reorganize or retheme the front page of the website, I think it would be a good idea to convene these focus groups. If we decide to limit the usability work to notifications, then focus groups should not be needed.