Usability Study Report

Transcription

1 Usability Study Report University of Washington HCDE 517 D Winter 2015 Mobile website March 15, 2015 UX Researchers Sanny Lin sanny@uw.edu Jaime Park jaimempark@gmail.com Julia Chamberlain chamberlain.julia@gmail.com Heather Larson digitalenvironments@yahoo.com Clients Avvo Inc. 1

2 PAGE TABLE OF CONTENTS 3 Executive Summary 4 Introduction 5 Study Design & Methodology 6 Materials 7 Participants 8 Findings and Recommendations 8 Task 1 Usability findings 10 Task 2 Usability findings 13 Overall user experience 16 Conclusion 16 Further Study 18 Acknowledgements 19 Appendices 19 Appendix A: Detailed Participant Profiles 20 Appendix B: Interview Data & Analysis 24 Appendix C: Affinity Diagram 2

3 Executive Summary The purpose of this study is to evaluate the usability of Avvo Advisor mobile website. Currently, X% of Avvo desktop users purchase legal advice, whereas only X% of mobile users complete a transaction. The company is curious about their mobile users overall experience, usage, pain points, and reasons for lower conversion than desktop. Data was collected through interviews and think-aloud tasks to identify usability issues and to prioritize findings for future redesigns. Nine participants were asked to walk through two tasks using an iphone. All nine participants completed Task 1 (default flow/choose any lawyer) without our assistance. Six out of nine participants completed Task 2 (select a specific lawyer flow) without our assistance. Upon first glance of the site, most participants found it visually appealing and they expected it would be easy to use. During Task 1, many participants were unsure about which category to select and said they desired more information before checking out, likely due to the Package Details link not functioning on the page before checkout. Three participants could not complete Task 2 without our assistance. Every participant was surprised about the horizontal layout and many faced usability issues while interacting with the page displaying specific lawyers. Many participants stated that they would leave the Avvo Advisor mobile site to find out more information about each lawyer from a different website. After the tasks were complete, participants said that the Avvo Advisor mobile site was easy to use overall and that Task 1 was easier than Task 2, but it was more important for them to select their own lawyer than to get advice promptly. They felt comfortable purchasing legal advice on an iphone and receiving legal advice over the phone. However, they weren t completely confident about the legal advice they would receive and they were not very interested in purchasing the service. We recommend providing more specific information about each legal category and each lawyer, making the link to select a specific lawyer more prominent, and altering the lawyers page to be responsive to a mobile screen. If we ran our study over again, we would alternate the order of the two tasks for different participants. We would also like to perform further research on how users find Avvo Advisor, categorize their legal issues, and interact with the site after checkout. 3

4 Introduction PRODUCT: AVVO ADVISOR MOBILE WEBSITE We tested Avvo Advisor s mobile website advisor.avvo.com a service that provides 15-minute legal advice with a top-rated lawyer over the phone for $39. STUDY OBJECTIVE The objective of this report to identify key usability findings and make recommendations to the design and product team for possible improvement. RESEARCH QUESTIONS Our primary research questions are: 1. Do users want to purchase legal advice on their mobile phone? Why or why not? 2. What legal questions do people have prior to using Avvo Advisor? 3. Do users successfully complete purchases? Why or why not? 4. Is there enough information for users to make a purchase decision? 5. What did users like about the experience? Why? 6. What did they not like about the site? Why? 7. What changes, if any, would users want to make to improve the site? TARGET AUDIENCE The target audience for Avvo Advisor is people who are seeking quick, practical legal advice from a lawyer, but are not necessarily looking to hire a lawyer for ongoing legal representation. 4

5 Study Design & Methodology Participants were recruited through the HCDE department, Seattle-area forums, mailing lists, and social networks. Participants were selected based on their answers to a screening survey (See Study Kit for screening survey). They needed to meet the following requirements: Be 18+ Have valid legal question(s) Seek legal advice in WA state Have made a credit card purchase on an iphone within the past 6 months Be comfortable discussing their legal issue Must not work in the legal field Each individual session lasted approximately one hour. During this time, the administrator briefly explained the test session, asked a series of pre-test questions regarding the participant s previous experiences seeking legal advice, facilitated tasks for the user to complete, and then asked a series of post-test questions regarding satisfaction, personal preferences, and confidence in the service. Participants were given two primary tasks to complete: purchase a 15-minute session with any attorney and then purchase a 15-minute session with a chosen attorney. (See Study Kit for interview guide.) Task 1: Default Flow Task 2: Specific Attorney Flow 5

6 At each new Avvo Advisor page within each task, participants were asked: Is this what you expected to see? What do you think you can do from this page? After all tasks were completed, participants were given a post-test interview to rate their overall experience on a 5-point Likert scale for the following measures: Ease of use Confidence Comfort level Likelihood of using the service again Likelihood of recommending service to someone else The test administrator concluded the study by asking the participant a few open-ended questions: What did you like about the site? Why? What did you not like about this site? Why? What would you change to improve the site? Is there anything else you would like to add? MATERIALS The study kit contained the following materials: Consent form An explanation of the participant s rights and permission to record audio and video. Pre-test semi-structured interview Preliminary set of questions regarding the participant s legal question(s) and past experiences of seeking legal advice. Task guide Scenarios and task lists for the two primary tasks, read aloud by the test administrator. After each task, the administrator also asked a series of brief post-task questions. Task support materials Sheet of paper containing non-valid sample credit card information for the participant to input on the checkout page. Post-test questionnaire Eight questions on a five-point Likert scale and three open-ended questions to evaluate the participant s overall experience regarding the mobile site. Test iphone device Participants interacted with the mobile site through either iphone 6 or iphone 5S. Cameras Two video cameras were used to record the study: 6

7 Webcam on a Macbook Pro to record audio and facial expressions. HUE external webcam to record the participant s interactions with the mobile site. Honorarium Each participant received a $10 Starbucks gift card and a single-use URL for a free 15-minute session of legal advice through Avvo Advisor. PARTICIPANTS All participants are current residents of the greater Seattle area. Nine participants were scheduled over four testing days (February 27 March 2) on the UW Seattle campus in a private closed-room environment. All participants fully completed the study. Of the nine participants, six were female, three were male. In addition to meeting the screening requirements, participants ages fell within the range. Seven out of nine participants were between years old. Participants had a range of occupations and legal questions, also some participants had multiple legal questions. None of the participants had ever heard of or used Avvo prior to the study. Full participant profiles can be found in Appendix A. Occupations Student Landscape Curator Assistant UX Professional Production Artist Nurse Legal topics discussed in the participants own words Inheritance Marriage Automotive Creating a will Employment visa status Taxes Citizenship Legal questions (based on Avvo Advisor categories) Family Business Criminal Defense Immigration Divorce / Separation Employment / Labor

8 Findings & Recommendations All participants successfully completed Task 1 (choose any lawyer/default flow); they found the site easy to use and visually appealing. However, eight out of nine respondents were unsure about which legal category to choose and wanted more information in general. Six out of nine participants completed Task 2 (choose a specific lawyer) with no assistance. In contrast to Task 1, they needed help finding the choose a specific lawyer. All participants believed this section of the site should be redesigned for mobile and wanted to see additional lawyer information. We have aggregated the interview notes (see Appendix B & C) and the results of this study are presented in the following sections: Task 1 Usability findings Task completion success rate Detailed usability issues Task 2 Usability findings Task completion success rate Detailed usability issues Overall user experience Likelihood of making a purchase Confidence level TASK 1 USABILITY FINDINGS: Purchase a 15-minute session with any lawyer Task completion success rate All participants successfully completed Task 1. Detailed Usability Issues for Task 1 The following rows of data represent usability issues. They are sorted by the Avvo Advisor page in which they occurred as well as the severity rating (1 for high severity, 3 for low) based on how likely they are to block the user from completing the task. Issues were derived from participant statements and direct observations. Subtask Findings, Evidence, Recommendations, Fixes Severity 1.1: Landing Page All participants completely understood what to do to start Task : Landing page All participants completely understood the next step (enter ZIP code) to get to the next page in the Task 1 flow. 5/9 participants expected to see a list of local lawyers after entering the ZIP code. 2 Mid 8

9 Though participants expected to see a list of local lawyers because of having to enter their ZIP code on the homepage, the next page does not show this. It shows categories of law. P2 stated that the ZIP code was confusing because it was unclear whether it meant the ZIP code of where she lived or where the law applied. Recommendation: Show the [Select a specific lawyer] link earlier in the flow or emphasize it as another option. Show local lawyers within the lawyer page. 1.2: Category page 8/9 participants expressed unsureness about which legal category to choose. 1 High 8/9 participants expressed verbal unsureness about which legal category to choose and 3 participants were specifically unsure about automotive-related legal questions. P3 and P8 both clicked on FAQ in the footer of the legal category page first, then were prompted to return to the category page and select the most fitting category. P3 selected Criminal Defense and P8 selected Employment / Labor, referring to it as L&I. P3 was adverse to selecting the Criminal Defense category, stating: I don't know if I'd want Criminal Defense, seems a little scary because it has a police officer. Most of our participants described their legal questions using different terms than what was on the Avvo Advisor mobile site. We believe that this is a high-severity issue because by the end of the task, 4/9 participants did not believe they would find the right legal information for their question. Recommendation: Provide more specific information on each legal category or give the users the opportunity to search for keywords. 1.2: Category page 2/9 participants had expected to tailor the amount of time purchased. 3 Low P6 and P8 had expected to be able to choose the amount of time they would pay for on this page yet were not able to. Recommendation: We recommend including information about what happens if calls run over 15 minutes. 1.3: Precheckout page 8/9 participants wanted more information. 1 High 9

10 8/9 participants expressed either uncertainty over what was being purchased, attempted to click on Package details, or stated that they desired more information. Recommendation: Explore design options that help users ensure that they are choosing the right legal category while keeping the user flow simple and short. 1.3: Precheckout page 6/9 participants wanted to provide more information. 2 Mid 6/9 participants wanted to provide more information about their legal question before making a purchase, either through inputting their question or selecting a more specific subcategory of law. Recommendation: Allow the user to type their legal question before paying or allow them to select a more specific subcategory of law. 1.3: Precheckout page 5/9 clicked on Package Details but the link did not respond. 1 High Recommendation: Fix the Package Details link. 1.3: Precheckout page 2/9 participants didn t know when they would get their legal call. 3 Low 7/9 participants correctly understood that they would get a call within 15-minutes from any advisor. P6 expected to receive a call within 1-2 business days. Recommendation: Make the wait time of the call more noticeable. 1.4: Checkout page 2/9 participants had slight issues inputting credit card expiration date. 3 Low P1 first entered the expiration date in the incorrect format, then erased it and typed it in again in the correct format. P3 expected to see a dropdown menu with numbers. Recommendation: Change the input from a text box to a dropdown menu of months and years. TASK 2 USABILITY FINDINGS: Select a specific lawyer Task completion success rate 6/9 participants completed task 2 with no assistance. 10

11 Detailed Usability Issues for Task 2 The following rows of data represent usability issues. They are sorted by the page they occurred on and labeled with severity (1 for high severity, 3 for low) based on how likely they are to block the user from completing the task. Issues were derived from participant statements and direct observations. Subtask Findings, Evidence, Recommendations, Fixes Severity 2.1: From landing to lawyer page 3/9 participants needed assistance navigating to the Choose your own lawyer page. 9/9 participants scrolled up and down the landing page to look for a specific lawyer. 3/9 participants expected to be able to click on the lawyer profiles on the landing page. 4/9 participants went through the links in the footer trying to find the page. 5/9 participants did not remember seeing the select a specific lawyer link from Task 1. P7 said that she would've given up looking for specific lawyer link. 1 High Recommendation: Include a secondary [Choose your own lawyer] call-to-action before the pre-checkout page, perhaps even on the landing page. 2.2: Lawyers 9/9 performance and usability issues from non-mobile friendly layout. 1 High 9/9 participants expressed surprise at the horizontal layout of the page. 4/9 participants did not initially understand that they could scroll horizontally. 11

12 P1: Didn't notice that I had to swipe to the left to see the stuff on the right. That wasn't intuitive. P6: If I hadn t looked carefully to notice that the words were cut off at the top of the screen, I would ve just scrolled up and down the page and clicked on the lawyer s profile picture and waited. P9: You don't usually scroll left or right on a phone. Never happened in the previous pages. Recommendation: Redesign the Select a specific lawyer page to fit standard mobile viewports. 2.2: Lawyers 6/9 participants did not understand the pie chart and percentages. 2 Mid 6/9 participants were unsure of the meaning of the pie chart and what it represented. P6: What is this pie chart? I don t know what it means.. 25% of what? success stories? Helpful to see what that pie chart indicates. P7: Not really sure what this is. There is no title for these. Not really sure what they are referring to. Recommendation: Include more information on the percentages of the pie chart or replace them with another way of representing information. 2.2: Lawyers 9/9 participants wanted more information about each lawyer. 1 High 9/9 participants tried to tap on the stars to see detailed written reviews. 7/9 participants would Google/Yelp the lawyer's name to find more information. 12

13 P2: Wouldn't select any of the attorneys. Not enough information. Would copy and paste to find attorney's name on a different site for more information. P6: Can I see the reviews?.. I can t. So I can only see the stars and the number of reviews they have. I want to see the reviews. 2.2: Lawyers Recommendation: If users are going to another site to find detailed review information, this indicates a high incidence of dropping off. We recommend making information and reviews clickable on the Select an Attorney page, provide more information about lawyers, and show lawyer information before purchase in the default flow. 4/9 participants did not change category after clicking on the dropdown for Filter lawyers by practice area A few participants made comments about how they wanted to be able to sort by something else, such as lawyer rating, percentage of experience, or proximity. P2: Says filter by practice but I can't sort by higher rating to lower rating. Recommendation: Allow the users to filter the lawyers by more than practice area. Consider allowing them to sort by a lawyer s reviews, experience, or proximity. 3 Low 2.2: Lawyers 7/9 participants were unsure about when they would receive their call. 2 Mid 2.3: Checkout 7/9 participants were unsure about when they would receive their call. Recommendation: Make the no time guarantee disclaimer at the bottom of this page more noticeable or repeated in a more prominent location. 4/9 participants did not expect Schedule an advice session to lead them to check out. 4/9 participants expected scheduling time slots or seeing who is available instead of checkout. P6: I ve expected to see a scheduling page. Like a Day that would work for you? page instead of payment page. 2 Mid Recommendation: Create a scheduling page for participants to choose when they would be available speaking with the lawyers, indicate that they can schedule an appointment after checkout. The wording of Schedule an advice session could also be changed to not imply a scheduled date or time. OVERALL USER EXPERIENCE Overall, participants thought that the design was clean and easy to use. Participants stated that the service is useful, however, they answered neutrally regarding purchasing the service again or recommending it to a friend. This section details the findings about the overall user experience. Ease of use 13

14 Participants were asked to rate expected and actual ease of use at five stages in the study on a five-point Likert scale ranging from very easy to very difficult. These ratings represented the ease the participants expected upon first hearing the prompt for the task, the actual ease after they completed the task, and then the overall ease of use for the mobile site. When asked to rate the expected ease of use upon hearing the prompt for Task 1, the mean response was 3.8 on the 1-5 scale. The actual ease of use garnered a mean response of 4.8, meaning that Task 1 was easier than expected. When asked to rate the expected ease of use upon hearing the prompt for Task 2, the mean response was 2.9, while the actual ease was only 2.4, meaning that Task 2 was not as easy as expected. In the posttest analysis, 6/9 participants stated that Task 1 was easier than Task 2. Preference to select lawyer Every participant stated that it was more important to them to select a specific lawyer (Task 2) than to get advice promptly (Task 1). Four participants went on to elaborate that they felt more confident in the quality of their legal advice when chose their own lawyer. Comfort level Participants rated on average that they were somewhat comfortable purchasing (3.89) and seeking legal advice over the phone (3.83). Two participants mentioned that they would prefer to seek advice in person and one had concerns that advice over the phone would be scripted. Overall, participants were comfortable shopping for legal advice online and all could understand the appeal of receiving quick advice on the go. Purchase interest At the end of the study, 7/9 participants stated that they would not purchase advice. We also asked participants to rate their purchase interest three times on a scale of 1-5 (5 being high) over the course of the study: right before Task 1, right after Task 1, and right after Task 2. Despite the website being easy to use, our participants responses fell between the neutral to not interested range in purchasing Avvo Advisor again. Our participants stated that their interest increased slightly after Task 2 when they learned that they could select a specific lawyer. 14

15 Confidence in receiving good legal advice After they completed the study, participants were asked: On a scale of 1-5, how confident are you that you will receive good advice from the attorney? With one being not confident and five being very confident. Overall the mean confidence level of is 2.7. Reasons included: Likert Rating Reason 5 (very confident) No participants selected 5 (very confident) 4 P1: Especially when you can see reviews, of other people having good experiences, trust id have a good experience also. 3 P2: Because it wasn't clear that guy dealt with taxes for foreign residents, I wasn't confident 100% P3: Neutral P4: 15-minutes might not be enough and not specific to my question. P7: Not sure if he will get the right advice. 2 P6: Not a lot of information provided that makes me not confident enough to make this purchase. P9: Not trusting because all of the lawyers had less than 5 total reviews. 1 (not confident) P5: Believes she will get a generalist when she wants a specialist. P8: I don t know anything about them. I need more information than what they look like. One recommendation is to emphasize on the category selection page, category confirmation page, and/or checkout page that the user will be matched with a lawyer that is actually experienced in that particular legal field. Recommending Avvo Advisor to a friend Lastly, participants were asked to rate on a scale of 1-5: How likely are you to recommend Avvo Advisor to a friend seeking legal advice? With one being unlikely and five being very likely. Overall, participants were unlikely or neutral on recommending the service, though two participants stated that they wanted to promote it. 15

16 Conclusion The site appeared easy to use and visually appealing to our participants. However, they wanted more information about legal categories, package details, and lawyers. The page displaying lawyers was difficult to find for many participants and it surprised all participants with its horizontal layout. They felt comfortable purchasing the service on a phone and also felt that the entire site was easy. Our participants found Task 1 to be easier than Task 2, yet Task 2 was reported to be more important to them for the ability to select their own lawyer. Our participants were also not completely confident about the legal advice they would receive and were not very interested in purchasing the service. PRIORITIZATION OF ISSUES We recommend the Avvo design team to first address the following Level 1 high-severity issues: On 1.2 Category page: Provide more information about each legal category or allow users to search On 1.3 Pre-checkout page: Fix the Package Details link Provide more information about package details From the landing page to the lawyer page: Make the link to select a specific lawyer more visually noticeable Move the link to a more prominent location earlier in the Task 1 flow On 2.2 Lawyers page: Make the page size responsive to a mobile device Provide more information about each lawyer After Level 1 issues are resolved, we recommend considering the following Level 2 mid-severity issues: On 1.1 Landing page: Show the link to select a specific lawyer on the homepage and display local lawyers On 1.3 Pre-checkout page: Allow the user to input their legal question or specify more information On 2.2 Lawyers page: Make the information in the pie chart easier to understand Make the disclaimer more noticeable of non-guaranteed call time when selecting a lawyer Either allow users to schedule a session on a calendar or change the wording to not imply a set scheduled time If these aspects of the mobile site garner more complaints or errors, we also recommend considering the following Level 3 low-severity issues: On 1.2 Category page: Provide more information on what happens when a call hits 15 minutes Make the time the user has to wait for a call more noticeable On 1.4: Checkout page: Change the expiration date input to be a dropdown menu of months and years On 2.2: Lawyers page: Allow users to sort by a lawyer s reviews, experience, or proximity 16

17 OUR STUDY Opportunities for us to improve as researchers If we were to conduct our study over again, we would alternate the order of our tasks so that odd-numbered participants would complete Task 1 first, and even-numbered participants would complete Task 2 first. This way the participants would not be familiar with a past task flow. We would alternate tasks to make sure that content and not task order influenced a participant s perception of the experience. Alternating the tasks could also generate more comprehensive feedback for both tasks. Another thing we would change in a future study would be to measure the usability of the mobile site using the System Usability Scale (SUS) because a SUS score is a reliable and a valid measure of a system s usability. This scale would provide usability data to compare Avvo Advisor against other mobile sites. It would also identify areas for improvement. Strengths of our study For each team member being relatively new to usability testing, we ran a very strong usability study. Our testing moderators almost always stuck to a consistent script with each participant except when asking contextual follow-up questions. We collected a large amount of data to confidently make sound design recommendations to Avvo. The data we collected was organized in a way that made it easy to review later, perform analysis, and quantify trends among participant statements and actions. FURTHER STUDY We believe that there are a few topics outside the scope of our current study that could be beneficial for Avvo to look into. Searching for Avvo Advisor How do first time users find Avvo Advisor? For example, are users coming to Avvo Advisor via a search engine, Avvo.com, or a different source? This understanding could shape the site s content and the amount of legal information needed for users to purchase legal advice. Refine Avvo s List of Categories We would like to study how users categorize their legal issues. Participants legal categories did not match Avvo Advisor s legal categories. They described their legal issues as inheritance, marriage, automotive, creating a will, employment status, taxes, and citizenship. In trying to find more information about their legal issues, participants selected Avvo Advisor s categories pertaining to family, business, criminal defense, immigration, divorce/separation, and employment/labor. If there was a better match between these groups, users may be more likely to use the site. User Experience Beyond Checkout We would like to investigate the user experience provided by Avvo Advisor after purchase. Were they satisfied with their legal advice? Will they return to Avvo Advisor? How was their overall experience with the mobile site and lawyer phone call? Ultimately, was their legal issue resolved? 17

18 Thank You Avvo We are grateful for your participation in this project. Thank you for meeting with our team, responding to our questions, providing incentives for our participants, and arranging for us to present our findings at Avvo. We learned so much about usability testing. 18

19 Appendix APPENDIX A: Detailed Participant Profiles Participant Gender Age Occupation Legal Issue(s) 1 Female Undergraduate Student Writing a will because of upcoming marriage 2 Female Graduate Student Taxes for non-us residents, past traffic ticket outside county of residence 3 Male Landscape Designer Unlawful traffic ticket 4 Male Assistant When to write a will 5 Female Designer Work visa allowing a second job 6 Female Designer Citizenship, Marriage, Inheritance (Does her husband s family have any claim to her inheritance?) 7 Female Production Artist Writing a will 8 Female Nurse Car accident insurance issue 9 Male Graduate Student Dual citizenship 19