Waze Usability Test Final Analysis

Transcription

1 Waze Usability Test Final Analysis Introduction We are evaluating a mobile app named Waze, a unique crowd sourced, community based traffic and navigation platform. On Waze, users can join other drivers in the area and share real time traffic and road info, saving users time and gas money on their commutes. In terms of functionality, Waze relies on GPS for geographical navigation and has display screens, which provides turn by turn information and user submitted travel times and route details with GPS support. The application downloads location dependent information over the mobile telephone network. The audience profile of Waze is as follows: the typical user of Waze is someone who drives a vehicle and travels frequently to places both familiar and unfamiliar. This is not limited to demographic or region, although it can be expected that the age limit would be over sixteen years (the legal driving age in the United States). A Wazer (Waze user) wants to find the most time efficient route, avoiding traffic, speed traps, construction, and other common pitfalls of driving. Methods When we planned our usability test, our main goal was to get a better sense of how Waze s target audience interacts the app. We aimed to gain some insight as to why Waze is not nearly as popular as Google Maps or Apple Maps. Seeing as Waze has a very unique interface unlike many of the other navigational apps, we assumed that usability issues could easily play a role in this division between Waze and its competitors. Our main objective while conducting these usability tests was to study how difficult it is for a new user to interact with the Waze interface. Additionally, the post test questionnaire also gave us an idea as to why our target audience prefered Google Maps or Apple Maps over Waze. Our main overarching research question is to better understand why Waze isn t remotely as popular as Google or Apple Maps. What deters people from using Waze? While designing our task list, we had to take into account that the majority of our testers have never used Waze before, and instead, most likely always use Google or Apple Maps. While the pro to this meant that each tester would be a total new user experiencing Waze for the first

2 time, the con was that many of our testers probably already had bias and favoritism towards Google or Apple Maps. Knowing this, we wanted to create a task list that was compiled of actions and features that are unique to Waze, elements that our users would be unfamiliar with. For example, while most people are familiar with navigating to a specific address, they have never had to change their mood on a navigation app or report a police sighting. While we did include the basic navigate to this address task during the test, we wanted to make sure that the majority of our tasks were Waze specific. As we created our task list, one of the things we had to keep in mind was that our testers would not actually be driving. This was one of our biggest limitations: we couldn t test our users solely based on navigation or navigational elements. Instead, we had to come up with tasks that could be performed while the tester was stationary within a building. So while we couldn t have the testers actually report a road hazard or police sighting, we still found ways to incorporate those certain aspects into our usability test. So when we would ask the test subject to report a police sighting, we would stop them before they actually submitted any inaccurate data to the app. After much deliberation, we were able to narrow down our task list to about eight tasks. While there are many things we wanted to ask our participants, we only had time for so much. We roughly assumed that if everyone took around two minutes per task, the usability test would be just under twenty minutes. We wanted to focused on the quality of the tasks, not the quantity. Our tasks covered a wide range of elements and features of Waze, but specifically we wanted to make sure we included both social and navigational related tasks. When we has finished planning for the usability test, we had an excellent mix of diverse questions that would give our test participants some direction of what to do, without being too detailed. Our questions were fairly open ended, so rather than saying go to the home screen, then click on the navigational menu, and exit the route you are currently in we would simply say exit the route you re currently in. We wanted to only tell our subjects what to do, not how to do it. Prior to our first usability test, we also had an introductory script planned out. Because Waze isn t very popular, we wanted to first familiarize our subjects with what the app does and how it works. Many of our subjects had never even heard of Waze, so giving them a short

3 background was crucial. This background included what Waze does, how Waze works, and why Waze is unique. Our introductory script also mapped out an extremely brief synopsis of what our subjects would be doing. We made it clear that we would be giving them a set list of eight tasks to complete and that they should expect the test to take around 15 minutes. We also made it very clear that there were certain rules to the usability test that we had to abide by: we couldn t answer any of their questions and they must speak their thoughts aloud. And finally, our introductory script prompted the participant to tell us if they were uncomfortable or unable to complete a task; we needed them to understand that we were simply testing the app, not their ability to complete tasks. Along with the set task list and the introductory prompt, we created a set list of post test questions to ask our subjects. We expected the majority of our participants to be Google or Apple Maps users, so we really wanted to see how they thought Waze compared to their primary navigational app. After our subjects completed the test and were somewhat familiar with the Waze platformed, we asked them to discuss ways in which they felt Waze measured up to Google or Apple. Specifically, we asked them how the visual aesthetics of Waze were different from Google or Apple, what unique Waze features they liked or disliked the most, and why they would still use Google or Apple over Waze. Additionally, we asked them what tasks they found were the easiest or hardest to complete. Overall, we wanted to get their impressions about Waze and how they felt Waze was weaker (or possibly stronger) than Google or Apple Maps. When performing our usability tests, we relied on basic equipment and technology. All of our subjects already used iphone and ios platforms, so we were able to use one of our iphone s for the participants. We also used an iphone to record video of the participant as they completed the tasks. We wanted to be able to go back and look at the participant s facial expressions, body language and exact words. We performed six different usability tests, and it was sometimes hard to keep all of them straight; that s why the video was crucial: it allowed us to go back and analyze anything we might have missed or forgotten. And finally, we had at least one group member take notes as the test was happening. Our note sheets were well organized grids that let the note taker easily record data. We used a series of abbreviations while taking notes so we wouldn t fall behind while writing: CU stood for comment by user, E for error, I for

4 critical incident. The grid also had spaces for the note taker to record what time each error or comment happened at, what screen the participant was on at the time of the error, and additional notes. These note sheets made sure that we all had uniform data that would be easy to understand for all participants. These note sheets have been scanned are located in the appendix of this assignment. Unfortunately, we did not have the opportunity to choose our test subjects and find a diverse group of participants. While it was easy and convenient to test only other students in the class, it really limited the range of responses we were looking for. Many college students don t have a car on campus or if they do, they don t drive it very often. While working on this project, we assessed our target user base as people who have are familiar with technology, and people who commute everyday or drive for a living. While the people we tested were all very proficient with iphones and navigation technology as a whole, they weren t people who drive everyday for a living. In our scenarios and personas, we defined the typical user as possibly a middle aged handyman that drives from house to house, all over town, for a living. Our data could have possibly been more enriched if we had the ability to test middle aged people as well as people in the workforce. Before our first usability test, we all picked various roles for us to complete during the testing. Drew was the video recorded and filmed all tests; Angela and Blake were note takers for all of the tests; and Maddie was the prompter who read the introductory script, tasks, and asked the post test questions. We all assimilated well to our roles and by our third test, we knew what to do and what to expect. The majority of our usability tests went by without any major issues or errors. Unfortunately, we didn t have the opportunity to choose the setting and environment where these tests took place. One of our first issues happening when we couldn t get any cellular service in the basement of North Quadrangle. One of our first tasks was to download Waze from the App Store and create an account. We didn t take into account the fact that Waze sends a registration code via SMS text message to your phone, which was an issue because we didn t have any cellular service. We ended up giving the participant another one of our phones that already had Waze installed and registered. After this incident, we learned our lesson and adjusted our test

5 procedure to no longer include download Waze from the App Store and set up an account. We figured that this was the least important task and that it was only testing the participant s familiarity with the App Store, rather than testing their ability to actually use Waze. This was the only major issue we came across and other than this, our usability tests went very well. Results When conducting usability tests for Waze, we discovered that the mobile app was not nearly as intuitive as we initially anticipated. Users continuously ran into errors and roadblocks along the way while attempting to navigate around the app. As a navigational app, the interface of Waze should be easy to use, as most people using a navigational feature are operating a motor vehicle and cannot spend a large portion of time staring at a display screen. The biggest failures that we found during our usability test included the display of the app. Numerous testers commented that they found it difficult to find their way around the app because the display was too cluttered. This led them to not be able to perform rudimentary tasks that should be second nature on a navigational device, such as inputting an address and saving it as a favorite for later use. Many respondents found that the display of Waze looked childish almost like it was designed for children. The cartoonish feel ultimately led the buttons within the app to become commonly confused with one another and not easily recognizable. For example, the majority of navigational apps use similar buttons, but during our tests we realized that users who have not used Waze before found the different feel of their buttons to create a complex interface. The number one purpose of a navigational app is to get from point A to point B, however, testers continuously felt lost while using Waze, let alone how to cancel a pre existing route. Even on entering an address, some users would have to spend upwards of 2 minutes simply attempting to get to the navigational page of Waze. On top of those failures, Waze is an app that uses user submitted data, even when the user believes the app is not running. Commonly on an iphone, users will cancel out of an app by double clicking the home button and sliding any apps that they want up, in order to close those apps. With Waze, this method does not work because in the background the app is still running,

6 collecting user data without the user even being aware. When asking users to close the app so there is no data being collected, they struggled. This was a very hard task to accomplish and it was failed numerous times. Even though this feature may create a more up to date app, it could be seen as a breach of an individual's privacy. Despite the numerous failures, there were some successful features in Waze that the testers liked. To start, testers found it easy to navigate to a location within the app. The third task we had testers complete was to enter an address (1300 S. University, Ann Arbor, MI 48104) and start the travel route to get there. All of our testers were able to accomplish this task, and the baseline task completion time was around 1 minute or less. This was the lowest time testers had in completing a task, and the only task all of the testers completed with little to no trouble at all. In fact, one of our testers, User 6, commented that it was easy to do while another tester, User 3, called it an easy task. User 4 appreciated how the navigation screen was very direct and made it clear what to press next after inputting the address. Since Waze is first and foremost a navigation app, this is a positive sign that the app is designed well enough to serve its main purpose. In addition, the majority of testers were able to finish the fifth task in our usability test, Private message any user that appears on the map that is near to you. After testers realized that other icons on the map represented other users, (User 6: Are these all your friends, these little guys? ), it was simple for them to select one to message directly. Testers could also use the feature to report a police sighting with minimal effort. User 3 mentioned That was easy after finishing it, while User 1 commented that It wasn t really intuitive to find this, but I noticed it earlier. This shows that the icon depicting the Report Police is striking and memorable to users, which adds a valuable insight into the user experience. It should be noted that there was one outlier, User 5, who struggled to complete this task, so this could be an area to look into improving. The final success relates to the unique appearance of the Waze app. Although there were comments from some testers about the childish aesthetic, other users liked how Waze looked different from other apps on the marketplace. The design of Waze is ultimately a differentiation point to users, for better or for worse.

7 Our participants were all up to date with the latest technology. They all knew how to operate an iphone, along with having prior knowledge of navigational devices/apps that are in use today. Conclusion As one can see clearly from our results, Waze has a lot of user experience and interaction issues. From the beginning of each interview the user was lost within seconds for various reasons, one of which that we believe to be detrimental to Waze s adoption is its different aesthetic look and feel compared to other navigational apps. In a way, Waze is a caricature of sleek, modern navigation applications such as Google Maps. This is for a variety of reasons. First, its cartoonish features distract the user, especially those who are first time users, because they are displayed sporadically on the map and they have no descriptions, as Waze is a icon heavy app. Second, the map itself is navigated in a difficult way. It keeps with the cartoonish display with big roads, many colors, and covered in users, hazards, police, etc. Most difficult though is the constant orientation changes of the app, rather than the user having to be very intentional about changing the direction of the map, it happens very easily. Finally, Waze is a difficult app to understand and navigate, mainly because of its many features, its icon based display, and its lack of common features in standard places on the app. This was found from our user interviews and experiments; every new user did not find Waze intuitive. Therefore, we recommend that Waze do a very intentional reboot of their user interface. While they will not please everyone with their cartoonish design, it is their brand and should not be altered. However, the inconsistency of the Waze design and interface with various other navigation apps has proved detrimental to the overall experience and impression on initial users. For this reason, Waze is limiting its growth opportunities into at least the demographic that we focused our research, college students, and potentially others as well. There are key aspects of the application such as canceling a route or zooming out on the map that should not have the learning curve that is currently present. We recommend that Waze take our findings under consideration for future updates. Future research on our end could include conducting more user experience interviews and experiments with different target populations such as the professional

8 who would utilize Waze for their job every day. Learning the opinions of the different target markets would give more perspectives for well rounded findings. Reflection It can always be a difficult job at anticipating what a new user to an app will intuitively understand and what they will struggle at accomplishing. When creating our task list we anticipated for learning effects. We attempted to create tasks that operated individually from one another in order to get the most accurate usability test results. Along with that same point, we aimed at creating tasks that were not bent around features that Waze supports. This would lead to data that truly shows how a new user will interact with Waze. One struggle that we had sometimes, especially during the usability tests, was getting our participants to think aloud. Often they would become frustrated with some part of the interface and focus, which is normal, but when they stopped talking and started only worrying about the tasks they were completing we should have intervened and asked them to continue to vocalize their thoughts. Failing to do so left us watching their actions and relying on those explicitly, instead of hearing what their thoughts were during the test. Another aspect of our tests that we could have improved on is our task questions. We didn t really evolve our line of questions throughout the different usability tests. We could have reworded our questions that weren t straight forward, added more tasks to test hypothesis formed from previous tasks, as well as added more tasks based on what the user feedback was following previous tests. If we would have adapted our task list and questions we may have had more findings in later tests that could help our overall understanding. Next time, we would have a larger focus on actionable intelligence. Along the way we would synthesize our key findings after each usability test instead of waiting until the end. We would also interpret our debrief questions at a higher level because this could aid in finding out where our user demographic was coming from and could give us valuable information on the way in which a user interacted with Waze. We would also create a more uniform note taking

9 methodology during each usability test in order to make sure that similar critical incidences were noted throughout the tests. Task 1 Find in App Store and download Task 2 Input home address and save as a favorite Task 3 Navigate to 1300 S. University Ave. Ann Arbor, MI Task 4 Add friends from Facebook Task 5 Private message any user on the map Task 6 Report a police sighting Task 7 Change Waze emotion Task 8 Exit the route + completely close the app User 1 User 2 User 3 User 4 User 5 User 6

10 Key: Success Error Critical Incidence