Memorandum To: Elizabeth Pass, Associate Professor, School of Writing, Rhetoric and Technical Communication From: Andrew Carnes, WRTC 456 Section 1[ADC] Date: February 2, 2016 Re: Project 1 Competitor Test Report For this project, I performed informal usability testing on the University of Illinois s student union website (http://union.illinois.edu). The University of Illinois s Illini Union can be considered a competitor with JMU s student union, and since I hope to test JMU s student union website (http://info.jmu.edu) later this semester, starting with a test of the Illini Union s website seemed like a good idea. Although both unions serve similar functions, their websites are markedly different. Author Steve Krug suggests testing a competitor because evaluating a competitor website can provide insight into usability issues the websites share and differing approaches to them (Krug, 2010). Participants I recruited three participants for this informal test. Due to time constraints and uncooperative weather, I was unable to recruit three random participants. My three roommates, however, proved to be excellent participants, and, as college students, they fit the profile of an average student union user very well. In addition, since my roommates are all close friends, it was easy to recruit them without offering an incentive. A simple hey, can you look at this website and tell me what you think? was enough to commission their help. Method My first job in conducting an informal test of the website was to come up with the critical tasks users visit the website expecting to perform. I examined the homepage and navigation items and thought about the purpose of a student union to help me think of tasks. I eventually formulated the following list: Find the location of the union Find parking near the union Use the union calendar to find an event Reserve a meeting space Find and interact with the union on social media (e.g., Facebook, Twitter, Pinterest) Navigate the building(s) using a map Browse organizations housed in the union For the sake of keeping my tests short so that I had a manageable amount of data and so that my participants did not grow bored, I decided to test only the most important of these tasks. I reflected on my own experiences as a student union user to help me select the most important tasks. (The four tasks I chose to test were: find the location of the union, find parking near the union, use the union calendar to find an event, and reserve a meeting space.) To ensure my participants had an appropriate context for the tests, I created three scenarios (see Appendix) that gave the participants a motivation to complete the tasks. I performed my informal test in a quiet, normally lit room with a large monitor and normal keyboard and mouse. As I brought each participant in for their round of testing, I asked them to make themselves comfortable and sit in front of the computer. I sat next to each participant as they tested the website. I was sure to sit close enough so that I could see what my participant was doing but not so close that my participant felt uncomfortable. Next, I told my participants that they would be performing a brief usability test on a website (I was sure to stress that I would be testing the website and not them) and that I wanted to hear what they thought about it while they performed three tasks. To help me collect data, I recorded the audio of each session (with each participant s consent, of course) and took notes. When it came time to actually perform the tests, I handed participants one scenario at a time and asked them not to use the website s search function
Carnes February 2, 2016 2 (since that wasn t something I wanted to test). When participants finished reading the scenario and as they began to use the website, I prompted them to tell me what they were thinking and feeling at various intervals. My participants seemed to get more comfortable thinking aloud as they used the website and I feel that I was able to gather lots of useful data. After participants completed each scenario, I returned them to the website s homepage so that they would start fresh with each one. Once the participants had completed all three scenarios, I thanked them and escorted them out. Results and Recommendations Scenario 1 When I gave the first scenario sheet to participants, they all reported they did not have any difficulty understanding the task and felt they had a reasonable amount of context. The first, and perhaps most critical, usability error became apparent when participants began looking for the calendar. In pilot testing my scenarios, I failed to notice that the University of Illinois s student union website has two different calendars that are both billed as a listing of events at Illini union. One of the calendars looks like part of the website (see Figure 1). Figure 1. The First of Two Calendars This calendar appears to be part of the Illini Union website but it wasn t the calendar I wanted participants to use.
Carnes February 2, 2016 3 In the first scenario, I wanted participants to use a calendar to find a specific event that wasn t listed on the calendar. Rather, I expected participants to use another calendar (see Figure 2). Figure 2. The Second of Two Calendars This was the calendar I expected participants to use in the first scenario. Although the calendar in does not appear to be a part of the Illini Union website, it has more events and is laid out in a more traditional calendar format. Two of my three participants went to the first calendar before finding the second calendar. When they realized they couldn t find the event they were looking for in the first calendar and when I told them to find a different calendar, they expressed frustration and annoyance. Most links on the Illini Union website lead to the first calendar, but a link to the second calendar is located at the top of all pages, above the navigation bar (see Figure 3). Figure 3. Top Navigation This navigation bar, including a link to the second calendar, can be found at the top of every page on the Illini Union website. Two of three participants reported that this top calendar link was hard to see, despite the fact that it appears at the top of every page. To remedy this usability issue, the University of Illinois should improve the visibility of the calendar link and find a way to combine the two calendars; having more than one is very confusing to users. In addition, all three participants encountered issues with the functionality of the calendar; when participants clicked an event to view more details, the X button at the top right did not close the event pop-up as expected. All participants reported that the second calendar was overwhelming at first glance; the large number of events made finding the specific one the scenario asked for difficult. Asking users to narrow-down the type of event they re interested in first might prevent this inordinate first-impression.
Carnes February 2, 2016 4 Once participants found the event they were looking for on the right calendar, they experienced more frustration when they searched for the event location. When viewing the event details in a calendar pop-up, nothing indicated location; instead, a number without any sort of label or context was given (see Figure 4). Figure 4. The Event Pop-Up This calendar pop-up displayed more information about events but did not provide a clear location; the blue arrow (added for visibility) points to (what is assumed to be) the room number for the event. Two of three participants reported that they would not know where to find the event (especially because no building was given with the room number). One participant reported that the number was clearly a room number, but this may have been due to the fact that he is a frequent user of JMU s student union calendar (i.e., he may have had domain knowledge the other two participants lacked). In addition, participant 2 remarked that it was helpful that the pop-up provided a contact person but that he did not know how to get in touch with them (i.e., no email address or phone number was provided in the event listing). The next part of the scenario involved finding parking near the student union to attend the event. The participants had less difficulty with this task because a link to the parking page could be found in the top navigation bar and in the footer of every page. Participant 3 remarked that the gateway links in the navigation bar were helpful (e.g., visit, get involved, plan an event ) and that they made finding parking easy (see Figure 5). Figure 5. Illini Union Main Navigation The website s main navigation included gateway links that participants reported were helpful. The parking page included a listing of parking lots, their availability, and a map. Participants had some difficulty finding the union on the map, but all eventually found it. This may have been another issue stemming from visibility. Scenario 2 Scenario 2 asked participants to take on the role of a University of Illinois freshman with little information about the student union and tasked them with finding the address. Participants usually went to the Building Map page (found in the main navigation) first. This page, however, did not prominently display an address or directions to the building. Participants quickly became frustrated with being unable to complete this seemingly simple task, but they all eventually found the address in the footer of the website. Including an address or directions on the building map page could greatly help new website users.
Carnes February 2, 2016 5 Scenario 3 Scenario 3 tasked participants with finding and reserving a meeting space in the Illini Union. Aided by the plan an event navigation gateway, participants quickly found a page that provided information about reserving spaces and from that page found a link to a space reservation form. Participants reported that this scenario was the easiest, but all three participants failed to find and use an online space request form link that was included on the homepage of the site (see Figure 6). Figure 6. Online Space Request Form Homepage Link This link, found on the homepage of the Illini Union site, quickly takes users to a form that allows them to reserve a meeting space in the union; all participants failed to notice this link. Moving this link or improving its visibility might help users to take advantage of this handy short-cut. Discussion Having performed this informal usability test, I believe I was able to gather lots of useful data. The data I gathered from observing my participants and hearing their candid remarks was surprisingly rich, and gathering the data was much easier than expected. From testing just three scenarios with three participants, I believe I was able to generate a respectable list of usability problems and possible solutions to them. In addition, I think I am now prepared to evaluate the JMU University Unions site with information about a competitor. I think I was successful in my testing because I prepared good scenarios, kept my participants comfortable with a casual atmosphere, and frequently prompted my participants to talk as part of the think aloud protocol. Creating audio recordings of the sessions was also helpful because I could listen to the recordings to hear exactly what was said instead of relying solely on hastily scribbled notes. Although I feel my testing went well, there are two things I think I would do differently as part of my next informal test: I would prepare more complex scenarios and questions to ask my participants. The first scenario I gave participants yielded rich data because it tasked participants with multiple objectives as part of the same story. The latter two of the three scenarios, however, seemed too easy and participants quickly completed them. This quick completion time ultimately meant I was able to gather less information about usability issues with the site and the impressions it left on users. I think next time I would prepare more questions to ask participants because it would get them more comfortable talking earlier in the test; I found that participants were more reluctant to talk as they were getting started. In addition, asking more questions could ultimately mean I could gather more data.
Carnes February 2, 2016 6 References Krug, S. (2010). Rocket surgery made easy. Berkeley, CA: New Riders.
Carnes February 2, 2016 7 Appendix Scenario 1 You are a business professional in the Urbana-Champaign area and you have been asked to speak at a chapter meeting of Kappa Phi Lambda Sorority during the week of January 24th. Using the Illini Student Union website, find out when and where the meeting will take place and find a place to park nearby. Scenario 2 You are a freshman at the University of Illinois. You've heard about the student union on campus, but you don't know much about it. You don't know where the union is, but you'd like to find out. Using the University of Illinois's Student Union website, look up information about how to get to the Illini Union. Scenario 3 You are a sophomore at the University of Illinois and have recently taken on the role of secretary for an organization you are a part of. Using the University of Illinois's Student Union website, reserve a space for your next club meeting on February 16th at 6pm.