Usability Report Author: Stephen Varnado Version: 1.0 Date: November 24, 2014
2 Table of Contents Executive summary... 3 Introduction... 3 Methodology... 3 Usability test results... 4 Effectiveness ratings by task... 4 Post-test questionnaire ratings...6 Participant comments and observations... 7 Conclusions...8 Recommendations... 8 Appendix... 9 Orientation script... 9 Background questionnaire...9 Task list... 10 Post-test questionnaire...12
3 Executive summary Bookshelf Editor is a software application used to create and organize a visual library of URL links known as a bookshelf. The links in a bookshelf can be graphically organized just like books on actual shelves. Our goal was to build a reliable, efficient, and effective user manual for the Bookshelf Editor application. The manual was developed in a series of iterations with each author contributing half of the content. These halves were combined and the resulting first draft was submitted for expert review. The first draft was also subjected to usability testing with a pool of 5 participants who had no previous Bookshelf Editor experience. We conducted this usability testing in order to gain direct feedback from participants on the usability and effectiveness of the manual. We then used the findings of our tests to revise and improve the manual in order to provide the user with the best overall experience when learning how to use the application. This report contains the methods, results, and recommendations of the usability testing we conducted. We have taken the recommendations listed at the end of this report and applied them to the revised manual. We hope this has resulted in a more effective, reliable, and efficient resource for users of Bookshelf Editor 1.1. Introduction Product Bookshelf Editor 1.1 is a web based tool that can be used to organize and store web links or links to local HTML documents. A user can create a virtual bookshelf, add multiple shelves, add books to the shelves, name the books, and add a link to a web page or HTML document. Once the bookshelf has been created, the user can open it in a web browser for easy access to the links attached to each book. The manual we authored for Bookshelf Editor 1.1 contains basic overview information about the application well as step-by-step instructions for creating and using bookshelves. The manual covers all major features and functions of the application, and includes illustrations, explanations, detailed procedural information, an index, and a glossary. Test objectives Our primary objective in conducting this test was to improve the accuracy, reliability, and usability of the manual, and to uncover opportunities for improvement such as confusing or missing instructions. In order to achieve this objective, our goals were to: Independently test 5 users with no previous experience in Bookshelf Editor. Ask each participant to complete a list of twenty common tasks in Bookshelf Editor using the manual as their only source of information. Ask participants to rate the effectiveness of the manual in helping them to perform each of these tasks. Ask participants to evaluate the overall accuracy, reliability, and usability of the manual, and offer suggestions for improvement. Analyze our test results and use the findings to revise and improve the manual. Methodology We used the following methodology when conducting the usability test: 1. We created an orientation script, task list, post-test questionnaire, and user profile. 2. We recruited five participants to perform the test. The participants' profiles are listed in the table at the bottom of this page. 3. We set up the computer and software for testing to ensure that all applications were working properly and all testing requirements could be achieved. 4. We tested each participant separately and one-at-a-time. 5. We provided each participant with the orientation script and a brief overview of the software being tested. 6. We gave each participant a copy of the manual to use during testing. 7. We gave each participant the task list and instructions, and explained how to use the accompanying six-point scale to rate the effectiveness in the manual in carrying out each task. 8. Each tester was present at all times to answer questions and provide guidance to each participant and record their reactions. 9. We administered a post-test questionnaire to each participant after they completed all of the tasks.
4 Below are the test participant profiles: User Profile Participant Background Participant #1 Participant #2 Participant #3 Participant #4 Participant #5 Age Range 50-59 20-30 60-70 40-50 40-50 How many years of computer experience do you have? How would you describe your proficiency with an Internet Browser? (Beginner, Intermediate, Experienced, or Advanced) Do you have any Bookshelf Editor experience? Do you have any experience using on-line user manuals? How many years of formal education do you have? >1 5 30 10 25 Beginner Experienced Experienced Experienced Advanced No No No No No No Yes Yes No Yes 12+ 12 16 16 16 Usability test results Effectiveness ratings by task Overall, test results showed that participants felt the user manual was both effective and easy to use. On a sixpoint scale from 6 (least effective) to 1 (most effective), the majority of task ratings were a 1 or 2. Only three tasks were rated at 3 and none were rated 4 or 5. As a whole, all twenty common Bookshelf Editor tasks scored an average effectiveness rating of 1.23. Seven tasks were rated the most effective score of 1.0 by all participants. The task with the most difficult rating was task 18, "Open the bookshelf in a browser," with a score of 2.2. The following table shows the effectiveness rating of the manual for each task. The average rating is listed in the far right column. The effectiveness rating is based on a 6-point scale: Manual effectiveness ratings by task
5 Task Description P1 P2 P3 P4 P5 Average 1 Open bookshelf editor. 2 Add a new shelf. 3 Modify the width of the bottom shelf so that it is 80%. 4 Modify the alignment of the bottom shelf, changing it to left justification. 1 1 1 1 1 1 1 1 1 2 1 1.2 1 1 1 1 1 1 1 1 1 2 1 1.2 5 Add this title to the bookshelf: Thanksgiving Recipes. 1 1 1 1 1 1 6 Save the bookshelf as "Thanksgiving Recipes.bsh." 1 1 1 1 1 1 7 Add 4 narrow books to the bottom shelf. 8 Add a space between each of the books on the bottom shelf. 9 Increase the width of the middle space by using the space trackbar. 10 Add 4 wide books to the top shelf. 11 Change the color of these books to Sky Blue. 12 Add these titles to the books on the top shelf. 13 Add these 4 web URLs to the books on the top shelf. 14 View the bookshelf 2 1 1 2 1 1.4 2 1 1 3 2 1.8 1 1 1 2 1 1.2 1 1 1 2 1 1.2 1 1 1 2 1 1.2 1 1 1 2 1 1.2 1 1 1 2 1 1.2 1 1 2 1 1 1.2
6 Post-test questionnaire ratings Post-test questionnaire results also indicated that overall, participants considered the manual to be reliable, accurate,and easy to use. Ease of navigation rated a perfect score of 1.0. The table below shows the accuracy, reliability, and satisfaction ratings of the manual. The average rating is listed in the far right column. These ratings are based on a 6-point scale: [Most accurate/reliable] 1 --- 2 --- 3 --- 4 --- 5 --- 6 [Least accurate/reliable] Accuracy and reliability ratings Question Description P1 P2 P3 P4 P5 Average 1 Overall, how easy was it to understand the information in the manual? 2 How easy was it to navigate the manual? 3 How accurate and reliable was the information in the manual? 4 How would you rate your overall satisfaction with the manual? 1 1 2 1 1 1.2 1 1 1 1 1 1 1 1 2 2 2 1.6 1 1 2 1 3 1.6 Additional information collected from the post-test questionnaire revealed a tie score for the hardest task to perform. Two participants indicated Task 17 was the most difficult (Export the bookshelf to HTML), and two indicated Task 18 (Open the bookshelf in a browser) was the most difficult. The table below shows the additional information collected from the post-test questionnaire: The easiest task to complete The hardest task to complete Suggestions for improvement Post-test questionnaire: additional information
7 Question Description P1 P2 P3 P4 P5 5 What was the easiest task to complete? 6 What was the hardest task to complete? #15 #11 #2 #1 #19 #13 #17 #17 #18 #18 7 What suggestions do you have to improve the manual? none Focus on Toolbar for tasks. Section 4 (Getting started) is confusing. Show bookshelf file location. Show bookshelf file location. Participant comments and observations We made observations and collected both oral and written feedback as the participants progressed through the tasks. After each participant completed all the tasks, we collected written comments from them using the post-test questionnaire. Some participant comments had more to do with the functionality of the application and Windows file directories than the manual itself. Participant 1 Participant #1 required the most help. As a user with very little experience in using computers or computer applications, some of the basic terms that you would assume most users would already know had to be explained. For example, the following terms were not familiar and need an explanation: URL - I explained that the URL is the address of a web page or other HTML document and referred to section 6.4 of the manual to show the graphic of a URL on a web page. Web Browser - Participant was more familiar with terms such as Google rather than web browser. Click, double-click, and right click - Participant needed explanation on how fast to double-click in order for task to work. I also explained the difference between click and right-click. The menu bar vs. the Toolbar - I had to refer him back Section 3.1 of the user manual to review the user interface as well as the following sections explaining the functions of each menu option. Participant #1 found that typing the long URL address took to long and stated that we should highlight the copy/ paste option more. Participants 2 and 3 Participants #2 and #3 began to get impatient with the process and began anticipating how the software should work rather than reading all of the steps in the manual. I had to remind each of them to refer to the manual and follow the steps as that is what was being tested, not the software. Participants #2 and #3 pointed out that the task list jumped around compared to the manual layout. They suggested that the task list used for testing following the sections of the manual to eliminate the need to flip back and forth throughout the manual. Once Participants #2 and #3 figured out that many of the tasks could be done from the Toolbar rather than the menu bar, they felt we should focus on writing the manual from that standpoint because it is easier, you see everything on the screen, and it requires fewer clicks or steps. Participant 4 By Task 3, Participant #4 was confident that whatever he was asked to do would be listed somewhere in the Table of Contents. For Task 4, Participant #4 noted that the bottom shelf was already in left justification so no action is required. This is an issue with the task design rather than the manual. For Task 7, Participant #4 noted that there is no option to add a narrow book, only a default book and a wide book. This is an issue with the task design rather than the manual. The term "default" should have been used in the test instructions. P4 also noted that it is easier to do this task using the Object Palette from the Toolbar. For Task 8, Participant #4 remarked that it's not as easy as it should be in the software. "You have to insert the spaces first, and they only go into the far right position. Then you have to move the spaces to the left. This is a pain. But the manual does show how to do this." For Task 12, Participant #4 noted that the titles of books are cut off when displayed on the shelf.
8 By Task 14, Participant #4 is bored with the test at this point. Participant 5 On Task 5, Participant #5 remarked that he doesn't need the manual to figure out how to do this. "The Title field is right there at the top of the Toolbar." On Task 2, Participant #5 wrote that this section of the manual (section 5.1) should be named adding a new shelf instead of adding a bookshelf. On Task 9, Participant #5 remarked that the menu should be labeled 'Object properties not Book properties inside the application. He also noted that you can't see how wide the space will be until you click OK. On Task 10, Participant #5 immediately added the four wide books from the Toolbar. Then, he went to section 6.1 of the manual to review the instructions, but he had already used that page for Task 7, so was not interested. On Task 11, Participant #5 stated that users should be able to select more than one book at a time in this software. On Task 15, Participant #5 did this task on his own before testing the manual. On Task 18, both Participants #4 and #5 noted that the the manual instructions left out the details of exactly where the bookshelf was located in the file structure. It took both of them a few minutes to find the correct folder. Yet, the user manual cannot show the exact navigation path for all users because of the differences in file structures between various operating systems. Conclusions Based on the results of this usability study, we conclude the following: The usability test showed that overall, the format of the manual was easy to use and participants were able to work through the tasks with relative ease. Test results demonstrated that the participants considered the manual to be an effective help document for the most common tasks performed in Bookshelf Editor 1.1. Most users would rather see tasks presented so that they focus on the Toolbar rather than the menu bar. The Toolbar method is easier to use, especially after the first several tasks. However the authors were following the course guidelines to consistently use the menu bar to describe procedural information for tasks. To the majority of users, the introductory sections look too much like procedural information when they are intended to be informative information only. If a participant acts on the wording and attempts to perform the step shown, some items won't work correctly because a bookshelf has not yet been built or finished. For the most part, participants did not investigate any section of the manual other than those required to complete the tasks in the usability test. For example, participants 4 and 5 made no use of the index or technical support section. Future usability testers may wish to include tasks that require the use of these sections. At least three out of five users had issues with the bookshelf file location in Task 18 due to the idiosyncrasies of Windows file systems and lack of detail in the user manual. However, the authors cannot show the exact file location for every user or support Windows folder systems in detail. The most efficient way to address the issue is to include a tip or warning in the manual. Recommendations After examining the usability test findings, we recommend the following improvements to make the manual more effective and easier to use. These recommendations have been implemented in the revised version of the manual: 1. Since the directions for the manual said to write it from the menu bar perspective, we have chosen not to change the manual to emphasize working from the Toolbar as suggested by participants. However, we suggest this approach be considered in the future. 2. Modify the preface to remove the duplicate word "manual" in the last sentence. 3. Change the name of section 5.1 to "Adding a new shelf," as bookshelf refers to the file as a whole. 4. In Section 3.3.4, Book Properties, the graphic is preceded by "Click the book properties button to open the Object Properties Window." If a user is reading through this section and tries to click on book properties, it is unavailable because the bookshelf is empty. Revise the wording so it doesn't look like an instruction. 5. Introductory sections look like instructions when they should be informative only. Recommend revising the wording so they do not appear to be instructions.
9 6. Regarding section 5.2, participant stated that it would be much easier if you just use the Toolbar and show the menu bar should be as an alternative way of performing the task. Since the recommended practice is to work from the menu bar, we will not make this change but will instead emphasize these Toolbar efficiencies to the user with a "Tip" whenever appropriate. 7. Regarding section 6.5 (Adding a web URL link to a book): participant stated that some users may not be familiar with the term URL or web Browser but would recognize Yahoo, Google, or Internet Explorer. Recommend revising item #1 to say, "Open a web Browser (i.e, Yahoo, Google, Internet Explorer, etc.) to provide additional information to the user. 8. Regarding section 9.1, (Opening a bookshelf in a web browser): include a note that the exact file location will depend on operating system and Windows file virtualization settings. Appendix Orientation script Thank you for participating in this usability test today. My name is Stephen/Linda and I will be administering the test and assisting you through the process. We are here today to test the user manual for a software application called Bookshelf Editor. Bookshelf Editor is an application that is used to create and organize a visual library of web links known as a bookshelf. Your actions and feedback during this testing will be very helpful in improving the user manual that is designed to support the users of Bookshelf Editor. During this usability test, you will perform twenty tasks in Bookshelf Editor to create and use a bookshelf of Thanksgiving recipes on the Internet. After each task you will be asked to rate the effectiveness of the user manual on a scale from one to six in helping you perform the requested action. Please be aware at all times that our only purpose is to evaluate this user manual. We are not testing the bookshelf editor software, and we are definitely not testing you. I want to reassure you that your responses are not being evaluated or judged in any way, and there are no mistakes or wrong answers on your part. Our only goal is to investigate how well the manual helps you to carry out the list of tasks. Because we are testing the effectiveness of this manual, it is very important that you always consult the manual before completing each task no matter how easy the task may seem. Conversely, if you have questions or difficulties regarding Bookshelf Editor, your only source of knowledge will be the manual. Feel free to ask questions at any time, but I cannot help you complete the tasks once you have started. Please try to think out loud as you proceed, giving voice to your thoughts so that I can follow your thinking. I will be taking notes as you move through the steps. After the test you will be asked to complete a post-test questionnaire that includes open-ended questions. Your honest feedback on these questions will prove to be very helpful in creating a better manual. Thank you once again. Please let me know if you have any questions before we get started. Background questionnaire 1. What is your age range? 10-19 20-29 30-39 40-49 50-59 60-69 2. How many years of computer experience do you have? 3. How would you describe your proficiency with Internet browsers? Beginner Intermediate Experienced Advanced 4. Do you have any Bookshelf Editor experience? 5. Do you have any experience using online online user manuals?
10 6. How many years of formal education do you have? Task list Purpose: You will use Bookshelf Editor to create a bookshelf of Thanksgiving recipes that are posted on the Internet. Instructions: Perform these tasks in order. Read the task from the task list, then consult the user manual for assistance. After task completion, rate the effectiveness of the manual in helping you complete the task. Proceed to the next task. Task List: 1. Open bookshelf editor. 2. Add a new shelf. 3. Modify the width of the bottom shelf so that it is 80%. 4. Modify the alignment of the bottom shelf, changing it to left justification. 5. Add this title to the bookshelf: Thanksgiving Recipes. 6. Save the bookshelf as "Thanksgiving Recipes.bsh." 7. Add 4 narrow books to the bottom shelf.
11 8. Add a space between each of the books on the bottom shelf. 9. Increase the width of the middle space by using the space trackbar. 10. Add 4 wide books to the top shelf. 11. Change the color of these books to Sky Blue. 12. Add these titles to the books on the top shelf: 1. Deep Fried Turkey 2. Green Bean Casserole 3. Sweet Potato Pie 4. Chop Suey Add these 4 web URLs to the books on the top shelf: 1. http://www.foodnetwork.com/recipes/alton-brown/deep-fried-turkey-recipe/index.html 2. http://www.food.com/recipe/americas-test-kitchen-green-bean-casserole-421411 3. http://southernfood.about.com/od/sweetpotatodess/r/bl1109d.htm 4. http://allrecipes.com/recipe/chop-suey/ 14. View the bookshelf statistics.
12 15. Delete the Chop Suey book. 16. Save the bookshelf. 17. Export the bookshelf to HTML. 18. Open the bookshelf in a browser. 19. Click the recipe links on the top shelf books. 20. Exit the bookshelf file. Post-test questionnaire 1. Overall, how easy was it to understand the information in the manual? [Easiest] 1 --- 2 --- 3 --- 4 --- 5 --- 6 [Most difficult] 2. How easy was it to navigate the manual? [Easiest] 1 --- 2 --- 3 --- 4 --- 5 --- 6 [Most difficult] 3. How accurate and reliable was the information in the manual? [Most accurate] 1 --- 2 --- 3 --- 4 --- 5 --- 6 [Least accurate] 4. How would you rate your overall satisfaction with the manual?
13 [Most satisfied] 1 --- 2 --- 3 --- 4 --- 5 --- 6 [Least satisfied] 5. What was the easiest task to complete and why? 6. What was the hardest task to complete and why? 7. What suggestions do you have to improve the manual?
14