Dhuel Fisher. BookFusion Usability Report

Transcription

1 BookFusion Usability Report Dhuel Fisher HCC 729: Human Centered Design Spring

2 Table of Contents BookFusion Usability Report... 1 Table of Contents... 2 Introduction... 3 Competitive analysis... 3 Executive Summary... 5 Methodology... 7 User Analysis... 7 Environmental Analysis... 9 Task Analysis... 9 Hierarchical Task Analysis... 9 Session and Recruitment Details Participants Evaluation Tasks/Scenarios Results Recommendations Purchase book (Task 2) Conclusion Attachments

3 Introduction BookFusion is an ebook platform that aims to redefine the reading experience. It allows users to read, share and have all their ebooks available across all devices. Readers are able to discover new books by accessing the global library of other readers with the capability to easily borrow/loan books to friends and family. With over 30,000+ free ebooks and a tightly integrated social experience, readers are able to find new books based on their social network or simply browsing the store. BookFusion is available on the Web and Android Tablets. At the end of this analysis, I intend to provide my client with useful, tested feedback that when implemented will improve the overall usability and acceptance of the application. The goal of this usability analysis is to provide the reader with in depth knowledge of the different methods used to test and evaluate the system, data gathered, analysis of the data and recommendations suggested to improve the system. Competitive analysis A competitive analysis was conducted to establish what makes BookFusion unique, to compare it with other similar products and to determine what works well, and what doesn t between the systems. Similar software to Bookfusion are: Kindle app This amazon based application allows thousands of books to be read on a personal computer in color, with no Kindle unit required, for e-books purchased from Amazon's store. There are several similarities in the design of the two software including the display of books on the home page, the ability to adjust brightness/layout and the layout of the menu bar. However, the kindle provides a cleaner display when compared to that of my client s. The ability to organize and create personal groups of books is also a functionality that could help users who own a large number of books. 3

4 Nook app The nook is an e-book reader created by Barnes and noble which can be used to access Nook reading material. Once again, we see the similarity between the front pages of the e-book readers. However, one distinct difference that can be seen with the nook is the titles underneath the books, this serves as a secondary means of identification for users. There are even more ebook readers available including Scribd, Oyster, Bluefire, Moon+Reader, FB reader and alkido. One distinguishing feature that these systems have that was not seen on my clients, was the use of their logo on the header bar of the main page. This will help to reinforce the brand. Although each of these software provide similar services, they are limited to the type of content they provide. Bookfusion seeks to enable the reading and sharing of different types of ebooks across different platforms. Scribd Oyster 4

5 Bluefire reader Moon+Reader Suggested changes based on competitive analysis 1. Removal of whitespace behind books (Kindle and Nook) 2. Addition of title text underneath books (Nook) 3. Create custom groups for books (Kindle) 4. Ability to search for words in books ( Kindle) 5. Adding logo to top bar (header bar) of home page (Kindle) 6. Enlarge items of interest (Oyster) 7. Percentage bar of completion underneath book (Moon+Reader) 8. Splitting taskbar between top bar (header) and bottom bar (footer) as seen in Bluefire 9. Allow orientation change while reading book. (This was noticed while using software) Executive Summary The test administrator, Dhuel Fisher conducted contextual inquiries, interviews and Heuristic evaluations of the BookFusion application. Personas were also created to aid in usability testing (see attachment 6 for personas). The contextual inquiries and interviews were conducted with different users who used the system on their own mobile device. The heuristic evaluations were conducted by Dhuel Fisher another student within the HCC 729 class. The sessions captured each participant s navigational choices, task completion rates, comments, overall satisfaction ratings, questions and feedback. The main goal of these contextual inquiries were to see users interact with the system and get a good understanding of what features were intuitive and easy to use or hidden and problematic. The contextual inquiries were conducted with two participants. My first user (User A) was a female student who lives in Florida. The interview was conducted using skype on February 14, 2016 and lasted approximately 30 minutes. My second user (User B) was a female 23 year old UMBC graduate student. The contextual 5

6 inquiry was done in the user s apartment on February 15, 2016 and lasted roughly 30 minutes. Both users have never used this system before but they were familiar with a similar systems. The contextual inquiry identified several problems: - Users disliked books not having correct covers - Back button always leads back to bookshelf - Not all books are able to be enlarged with gestures and enlarge feature covers text when present. - Profile information did not update - There was not a wide variety of books - Commenting and bookmarking features were hidden to some users - Text alignment was off with certain books The contextual inquiries acted as an introductory study to gauge which aspects of the interface should be focused on. After gathering this information, we needed to conduct further research to address these problems and find solutions to them. To do so, interviews were conducted. The interviews were structured interviews and the topics focused mainly on usability and the client s ability to do tasks that client requested. User questions were closed with the opportunity for further explanation. The interviews were conducted with the creator of the system, and two test users. The interviews were conducted in person at the interviewer s apartment on February 21 st 2016 and lasted approximately 15 minutes. Overall, the responses were that the application was fairly easy to use and received an 80% rating from the client, 83% rating from user A and a 94% rating from user B. The main issues identified through the interviews were: - Hard to understand notifications - Difficult to understand filtering - Not enough books due to limited licensing - Difficult to see bookmark feature Another usability test performed was a Heuristic evaluation. A heuristic evaluation is a usability inspection software that helps to identify usability problems in the user interface (UI) design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles (the "heuristics").these evaluations were conducted by two members of the HCC 729 Human Centered Design class. The main purpose of the Usability Action Report is to gather data about the usability of the product or design by a particular group of users for a particular activity or task within a particular environment or context. The heuristic analysis was used to assess the extent of system s functionality, assess the effect of interface on user and to identify specific problems with system. The issues received through the heuristic evaluations were: - The white spaces around book titles - Hidden notifications - Hard to understand book filtering - Not enough books After these tests were performed, both low and medium level prototypes were created, tested and redesigned in an effort to create the best version of the system. The final focal points addressed by the prototypes were: The sort function The notification function The bookmarking functions. 6

7 This document contains the participant feedback, prototypes, observations, heuristic analyses, results from tests and participatory designs and recommendations for improvements. A copy of the scenarios and questionnaires are included in the Attachments section. Methodology User Analysis The target users of this application are the teenage to adult population. Although the system is available to the public and may be used by a wider range of individuals, the largest users of this system are expected to fall within this range. My client also intended for the system to be implemented on a large scale in Jamaican schools as part of a new technology in education program. The population that will be using the system, are expected to have the following characteristics: 1. Intermediate or high literacy ability 2. Above 12 years of age and under 60 years 3. Basic to high understanding of technology 4. Has obtained at least a primary school level of education 5. Average to high level of proficiency in English 6. Owns or has access to a smartphone or tablet 7. Has internet services when purchasing of books is necessary 8. Jamaican students (select group) 9. Are a part of a social network (Facebook or twitter) 10. Interest in reading e-books or digital content. Outside of main target users, the system is useful for all individuals who which to read and share e-books. This includes children, teenagers, adults and the elderly. The Major tasks that the users will want to perform are: 1. Purchasing books 2. Reading books 3. Making bookmarks 4. Tracking progress 5. Sharing books Personas of possible users were also created to aid in the usability testing. 7

8 8

9 Environmental Analysis The system will be used for both learning and leisure. My client hopes that this system will be used in schools by both teachers and students as a way to facilitate learning, sharing and collaboration. This system will also be used greatly outside of educational setting for leisure purposes. With this in mind, the redesign has to be both informative and intuitive so that all readers can quickly and easily interact with the system. The system should also be professional while maintain an inviting presence. One useful feature, could be the ability to customize the display to each user s preference. Task Analysis The main goal that is trying to be supported is the bookmarking a specific page of a book. If a user is able to do this, they will have a good understanding of most features in the system and the locations of most features. The ability to know how to bookmark is important because it allows users to resume reading from their bookmarked position. During one of the observations conducted, the user did not recognize the bookmarking icon and was unable to bookmark a page. The sub goals are to teach the user how to purchase and start reading books which are also an integral part of the system. My client s main focus was on the sign in/ log in process, bookmarking and the system s UI. I chose to go with bookmarking for the task analysis because it also covers other areas. Hierarchical Task Analysis A Hierarchical task analysis is a hierarchical representation of what steps it takes to perform a task for which there is a goal and for which there is some lowest-level "action" or interaction. This analysis allows us to understand the processes necessary to complete specific tasks and analyze how multiple tasks may be connected. Within the Hierarchical task analysis, we assume that the user already has an account and is already signed into the system (See attachment 7 for Diagrammatic HTA) 1. Log into application a. Sign in using Facebook b. Sign in using twitter c. Sign in using BookFusion 2. Purchase a book a. Click menu icon at top left of screen b. Go to store. c. Search for book. d. Click book. e. Click add to library. 3. Read book a. Click menu button b. Go to book shelf. c. Click on book. d. Click read. 4. Bookmark page a. Click the bookmark icon on top right of page. b. Click add 5. Return to Bookmarked page a. Open bookmarked book b. Click on book icon c. Select bookmark Plan 1 for task 1 A. If user uses Facebook do 1a Else if user uses Twitter do 1b Else use 1c Plan 1 for task 2 A. Do 1 B. Do 2 Plan 1 for task 3 A. Do 1 B. If bookshelf is empty or you want a new book Do 2 C. Do 3 and 4 Plan 1 for task 4 D. Do 5 9

10 Session and Recruitment Details The recruited participants for both the contextual inquiry and the interviews were recruited through direct communication. The test administrator Dhuel Fisher ed the attendees informing them of the test logistics and requesting their availability and participation. Participants responded with an appropriate date and time. Each individual contextual inquiry session lasted approximately thirty minutes and each interview lasted approximately fifteen minutes. The test administrator introduced the users to this method by explaining how a contextual inquiry works and the different roles that we would both play. An environment of partnership was created by asking the user several questions while looking out for indicators such as yes but or kind of. Depending on these answers, follow up questions would then be asked to ensure that the test administrator had a good understanding of what the users were trying to say/do. To create the right amount of focus, the test administrator asked the candidate specific questions about aspects that the client was interested in then allowed them to explore the systems for themselves to see what they were interested in. For the interview sessions, the test administrator asked the user to do specific tasks then followed up with asking the user to rate the system on a one to ten ranking. Post-task scenario subjective measures included (See attachment 1 for full list of scenarios and post scenario questions): How would you rate the sign in process? How would you rate the purchasing of books? How difficult was it to place a bookmark? How difficult was it to return to a bookmarked page? How difficult was it to resize text After the last task was completed, the test administrator asked the participant to rate then application on a one to ten ranking scale on two subjective measures including: Difficulty of maneuvering the system Overall graphical layout of the system In addition, the test administrator asked the participants the following overall website questions: What observations did you make about the system that you think could use improvement? Would you recommend this system to a friend? Why? See attachment 1 for subjective and overall questionnaires. After conducting the interviews, it was realized that the ranking system may have resulted in unintentional biasing. To counteract this, several of the questions were changed and a Likert scale was used instead of the previous one to ten ranking. This is the proposed format for future studies (See attachment 2 for updated interview questions). To conduct the Usability Action Report, the two parties Dhuel Fisher and Hwaju Chung, evaluated the interface according to ten heuristics (See attachment 3 for Dhuel Fisher s UAR forms and attachment 4 for Hwaju Chung s UAR forms). After these Usability Action Reports were gathered, they were aggregated and then sorted into different groups based on ratings and relationship (see attachment 5 for categorization of Usability Action Reports). After the above methods were conducted, low fidelity prototypes were created with the adjusted changes recommended by the previous test users. These prototypes were then tested using a think 10

11 aloud method. A participatory design session was also conducted afterwards to further improve upon the system (See attachment 8 for Low fidelity prototypes and attachment 9 for think aloud and participatory design solutions). UAR forms were also created to find any connections or similarities between the solutions (See attachment 10) The three issues addressed in the prototypes were: The categories used to filter books The lack of book options False notifications The low fidelity prototypes provided a lot of useful information but was unable to fully capture all the processes and features of the application. To better asses the applications features, medium fidelity prototypes were created using the information gathered through the low fidelity prototypes (The prototype of the system can be found at (See attachment 11 for test protocol and test plan for medium fidelity prototype). When creating the medium fidelity prototype, the client suggested that the focus be moved from the lack of book options to the bookmarking feature due to limited ability to obtain books and the fact that the bookmarking feature was the main initial concern. The main improvements made on the medium fidelity prototype were 1. Added a sort button to the bookshelf This addition to the prototype was added because one of the users in the think aloud session said that the menu icon to the right of the page was not clear or visible. She then suggested that a possible solution was to have a sort button which would be much more visible and easier to interact with. 2. Changed the color of the bookmarking icon Many users were not able to see the Bookmark icon because it was close in color to the background. My solution was to make it yellow which would draw the user s attention to it. 3. Removed the notification tab that did not have any notifications This problem occurred each time the user logged into the system. A notification would appear but when entered, would be blank. To solve this problem, the notification tab was removed. One other possible solution would be to provide a welcome message with a coupon for the first book the user purchases. I chose to make these changes because they were the most troubling to the users, and by fixing these, it would greatly improve the system. I also decided to add other changes to the prototypes based on other ideas received in the participatory design sessions such as: 1. Changing the sort categories The sort categories in previous prototypes were difficult to understand according to both participants and a new set of categories were chosen. 2. Allowing swiping of books to sort them One participant said that they would like to be able to sort their book in their own way and suggested the ability to swipe the books to different locations. 3. Replacing notification tab with individual group notifications Since the removal of the notification tabs was not replaced with the welcome message, we chose to place notifications in other tabs which would reduce the number of tabs and also help the user to see what category each notification belonged to. Finally, in an effort to gather more data and support my final conclusion, two users were asked to participate in an eye tracking study which looked at the time taken to recognize the bookmark feature and the focus that the original layout received versus the prototyped version (See attachment 11 for eye tracker instructions). 11

12 Participants For the contextual inquiry, two participants both of which were female, were scheduled on February 14 th and February 15 th. These participants were contacted through the test Administrators contacts. For the interviews, two participants both of which were female, were scheduled on February 21 st Participants were chosen based on availability and proximity to test administrator. Participant A is a current university student and participant B is a lawyer. Participants within the low fidelity, participatory design and medium fidelity studies were chosen from within UMBC s ITE building. The two participants chosen for the eye tracker were 2 HCC Masters students from UMBC who were not familiar with the system. Evaluation Tasks/Scenarios Within the interviews, test participants attempted completion of the following tasks (see Attachment 2 for complete test scenarios/tasks: Create an account on the application Purchase a book of their choice Read the book and place a bookmark at desired page Exit book and return to bookmarked page Resize the text Within the low fidelity prototype, the users were asked to Sort the bookshelf Select a specific notification Purchase a book Within the medium fidelity prototype, the users were asked to Sort the bookshelf Select a specific notification Bookmark a page in a book Within the eye tracker study, the user was asked to look for the bookmark feature Contextual inquiries and interviews Task Completion Success Rate Results Both participants successfully completed task 1 (create an account on application). One out of the two participants had trouble completing the second task (purchase a book of their choice). Both participants were able to complete task 3 (Read the book and place a bookmark at desired page). One out of the two participants had trouble completing task 4 (Exit book and return to bookmarked page). One out of the two participants had trouble completing task 5 (Exit book and return to bookmarked page). 12

13 Task Completion Rates Participant Task 1 Task 2 Task 3 Task 4 Task 5 User A - - User B - Success Task Ratings After the completion of each task, participants rated the ease or difficult of completing the task. The 10-point rating scale ranged from 1 to 10 (See attachment 1). Participant Task 1 Task 2 Task 3 Task 4 Task 5 User A User B Overall ranking 18/20 15/20 20/20 17/20 19/20 Liked Most The following comments capture what the participants liked most: Ease of signing in and the easy to read font Liked Least The following comments capture what the participants liked the least: Limited number of books Difficult to leave comments Profile information does not update Hidden bookmarking Improper text alignment of some books Recommendations for Improvement based on preliminary findings Provide tutorial at first login to show user where features are Increase available books Allow zooming gesture for all books Fix notification alert Fix book cover page display Fix text alignment in certain book Prototypes All participants were able to complete all tasks in the low fidelity prototype and medium fidelity prototype. 13

14 The following data was received from the interview and time recordings conducted during the medium fidelity interviews. User A and User B both used the original version of the interface but user C and D used the prototyped version. Participant Pre interview question 1 Pre interview question 2 Pre interview question 3 Overall ranking User A User B User C User D Overall ranking Participant Task 1 completion time (sec) Task 2 completion time (sec) Task 3 completion time (sec) Overall Ranking (sec) User A User B User C User D Overall ranking As seen in the table above, there is a significant difference in the users that used the previous version of the system versus those that used the prototyped version. A summary and explanation of these results can be found in attachment 13. Eye tracker Within the eye tracker test, participant A was shown the original layout of the application and participant B was shown the prototyped version. It is clear from the results, that there is a large difference in the noticeability of the bookmarking icon if its color is changed to yellow. More data and precise timing associated with the eye tracker will be added along with this analysis. There is a possibility of bias present in the eye tracker data because two different individuals were used in between each study. This was done in an effort to create the least bias possible. Because the location of the bookmark feature is the same in both versions and the only change is the color of the notification, a bias would be present since the user would be more likely to look in the general area of the first bookmarked icon thereby making it more likely for them to notice it. 14

15 Participant A Participant B 15

16 TIME TAKEN TO RECOGNIZE BOOKMARK (SEC) PARTICIPANT A PARTICIPANTB As you can see from the above data, there is a significant difference in the time it takes for the user to recognize the yellow bookmarked feature. Recommendations The recommendations section provides recommended changes and justifications driven by the participant success rate, behaviors, and comments. Each recommendation includes a severity rating. These ratings varying between low, medium and high, were given based on the impact that the problem may have on users. Low severity Although the problem exists, the user may not notice or will not be severely bothered by the problem. Medium severity The user may be aware of the problem but the overall functionality or usability of the system is not compromised. High severity There is a usability or functionality problem that must be addressed as soon as possible. The following recommendations will improve the overall ease of use and address the areas where participants experienced problems or found the interface/information architecture unclear. The following recommendations are based on the results from the interviews, prototypes and eye tracking tests conducted. They are ordered based on increasing severity. Hidden notification Whenever the system is started, there is a notification at the side that is empty. Change Justification Severity Remove false notification Replace with a coupon or welcome message Change notification so that they are grouped by respective categories 1. Based on the initial interviews, this notification does not cause severe problems to the overall use of the system, it is a flaw which has the possibility to annoy users. 2. After creating several prototyped solutions to this problem, it was noticed that there could be a slight improvement to the system if this is changed. Low 16

17 Profile information does not update The user used the system for several minutes but their reading time in their profile did not update. Change Justification Severity Better updating of profile information Although this information does update, it does not do so in real time which may bothersome users Low Allow enlarging gesture Not all books are able to be enlarged through the enlarging gesture and requires the user to go into settings to resize. Change Justification Severity Allow all books to be enlarged through enlarging gesture Many users are accustomed to the enlarging feature through their other devices. Consistency across all platforms will increase usability. Low Improper text alignment of some books During contextual inquiry, the user was upset that some books only took up half the page. Change Justification Severity Ensure that all books occupy entire section of screen Although this is not present in all books, this is still a serious issue when it occurs. Medium White space around books There is a white space that appears around book covers and varies between books. Change Justification Severity Remove white space around books or ensure uniformity of white space. Several users were distracted by the difference in white spaces around the books. This visual effect may dissuade users from using the system. Medium 17

18 Sort books by Tag None of the users that interacted with this feature understood how it worked. Change Justification Severity Remove ways in which books are sorted. Add sort by name, Author, genre, date purchased, custom 1. The sort categories and phrasings were difficult for all users to understand. 2. As shown in the prototypes, a change in the name of the sorting tools allowed users to better understand them and make them more likely to be used Medium Show where everything is One user said that the system was easy to use, but they did not know where all the functions were or that they did. Change Justification Severity Provide a short tutorial at the beginning of the system. This will allow all users to know what each function does and where to locate them. High Change color of bookmarking feature During the interviews, prototypes and eye tracker studies, the users were asked to bookmark a book. Change Justification Severity Change the color of the bookmark feature from grey to yellow 1. Many participants were unable to recognize the bookmark feature because of its color. 2. During the eye tracker study, the color of the bookmark feature was changed to yellow and there was large increase in visibility. High Purchase book (Task 2) Task 2 required participants to find a book of their choice and purchase it. Change Justification Severity Add more books to the library Participants from all studies searched for books within the store and was unable to find desired book. By providing more content, it will greatly improve usability. High 18

19 Conclusion After conducting the original set of interviews, a large number of issues were recognized within the system. These issues were then focused by importance and the probability of the user encountering them. Through the low level prototyping and the participatory design, I was able to get feedback directly from users while working with them to improve the system by making changes that they would understand and accept. These ideas were then tested using the medium level prototyping accompanied with eye tracker data. It was seen that most of the suggested changes produced positive results and if implemented, would increase the overall usability of the application. All these collective sources of information influenced the final set of recommendations made. The overall portrayal of the system is that it is simple and easy to use. Having an application that will allows users to read, share and have all their ebooks available across all devices will be a great asset to users. By implementing the recommendations and continuing to work with users, this application has the potential to reach a wide ranging audience. 19

20 Attachments Attachment 1 (initial questionnaire) 20

21 Attachment 2 (Updated interview questionnaire) Attachment 2.1 Attachment

22 Attachment 3 Dhuel Fisher s Usability Action Reports Attachment

23 Attachment

24 Attachment

25 Attachment

26 Attachment

27 Attachment

28 Attachment

29 Attachment

30 Attachment 4 Hwaju Chung s Usability Action Reports Attachment

31 Attachment

32 Attachment

33 Attachment

34 Attachment 5 - Usability Action Report categorization Attachment 5.1 Below are the UAR categorizations generated from the documents in attachment 4. These problems and good aspects were then grouped into categories based on their similarities. These include: positive aspects, negative aspects, 1 rated items, 2 rated items, 3 rated items and related items 34

35 Attachment

36 Attachment

37 Attachment

38 Attachment 6 Personas Attachment

39 Attachment

40 Attachment 7 HTA Attachment 7.1 Diagrammatic HTA 40

41 Attachment 8 Low fidelity prototypes Attachment 8.1 For the first problem which involved a difficult to understand tag, solution A, was to change the dropdown menu to show all books, filter by category, filter by genre and filter by date purchased 41

42 Attachment 8.2 Solution B was to remove the filter by tag choice and keep the remaining two options 42

43 Attachment 8.3 For the second problem which was a lack of book options, solution A was to provide the user with the option to request the book which would make it available in a few days. 43

44 Attachment 8.4 Solution B was to increase to number of possible books which would thereby reduce the number of interactions necessary while providing users with what they wanted. (This is the preferred choice) 44

45 Attachment 8.5 For the third problem which was false notifications, solution A was have a notification which welcomed the user to the software. 45

46 Attachment 8.6 Solution B was to remove the notification tab completely since it is not completely necessary. This can be done by having notification numbers on other tabs. 46

47 Attachment 9 - Redesigned low fidelity prototypes Attachment 9.1 User A Think aloud set 1 searching for books - With the first prototype, the user got confused when the search result showed books but then they were unable to actually get the book. The suggestion was that three notifications be shown. o Your search result yielded 0 results o Would you like to request a new book o Click on book to confirm - Although the second prototype had more books than the original system, the user was still displeased with the number of books available. However, the ease of purchasing a book was much less confusing. 47

48 Attachment 9.2 User A Think aloud set 2 hidden welcome notification - With the first prototype, the user did not like the notification that as a welcome message. They felt like it was a waste of time and would have preferred if it was a coupon or had some form of incentive. - The second prototype with the individual notifications were preferable to the individual notification tabs 48

49 Attachment 9.3 User A Think aloud set 3 Sorting books - Within both prototypes, the user said that the sorting menu button at the top right of the screen was not intuitive. A better way would be to have a sort tab at the top of the bookshelf with a drop down menu. - The sort categories suggested were o Name o Author o Topic o Date purchased - The user would also like to be able to drag and drop their books and sort it in their own way 49

50 Attachment 9.4 User B Think aloud set 1 searching for books - With the first prototype, the user did not understand how request books worked and would have preferred an explanation. They suggested the following prompt o We re sorry, we do not have the book you are looking for but if you see it in the list below, we can get it for you as soon as possible. o Followed by a list of books with an estimated time of arrival. - The second prototype worked well with the user and they said it was not difficult to understand. 50

51 Attachment 9.4 User B Think aloud set 2 hidden welcome notification - In the first prototype, the user did not mind the welcome notification but said it would get annoying if it appeared every time they logged in. - In the second prototype, the user said that they preferred the notifications on the different tab instead of a notification tab, which is consistent with other systems. If the system wants to tell me something important, They can just have a pop up when I open the app 51

52 Attachment 9.5 User B Think aloud set 3 Sorting books - The user was able to find the sorting section but was not happy with the sorting options. They suggested that the books be able to be sorted in the following categories: o Title o Author o Date purchased o Genre 52

53 Attachment 10 UAR for low fidelity prototype Attachment 10.1 No. UE - 01 Problem Name: Not speaking user s language Evidence: This problem lies in the prototype which gave the users an error when the book was not found. Explanation: The error was not clear and did not explain to the user what would happen if they requested a book Severity or Benefit: Rating: 3 Justification (Frequency, Impact, Persistence, Weights): This is difficult to understand and will confuse user Possible solution and/or Trade-offs: Tell user what happens upon request Relationships: UE-02, UE-03 53

54 Attachment 10.2 No. UE - 02 Problem Name: Not speaking user s language Evidence: This problem lies in the prototype which gave the users an error when the book was not found. Explanation: The error was not clear and did not explain to the user how much time it would take to receive their book Severity or Benefit: Rating: 3 Justification (Frequency, Impact, Persistence, Weights): This is difficult to understand and will confuse user Possible solution and/or Trade-offs: Tell user expected wait time Relationships: UE-01, UE-03 54

55 Attachment 10.3 No. UE - 03 Problem Name: Not speaking user s language Evidence: This problem lies in the prototype which gave the users an error when the book was not found. Explanation: The error was not clear and did not explain to the user which book would be requested if there were multiple books Severity or Benefit: Rating: 3 Justification (Frequency, Impact, Persistence, Weights): This is difficult to understand and will confuse user Possible solution and/or Trade-offs: Place notice over specific book or allow user to choose book Relationships: UE-01, UE-02 55

56 Attachment 10.4 No. UE - 04 Name: Did not display enough books Evidence: This problem lies in the prototype which displayed the store results Problem Explanation: The store needs to accurately display all books searched Severity or Benefit: Rating: 2 Justification (Frequency, Impact, Persistence, Weights): Although this is a low fidelity prototype, this affected the user Possible solution and/or Trade-offs: Provide more search options Relationships: 56

57 Attachment 10.5 No. UE - 05 Name: Waste of time notification Evidence: This problem lies in the prototype which displayed the notification Problem Explanation: Users did not like this and felt that it was a waste of time Severity or Benefit: Rating: 1 Justification (Frequency, Impact, Persistence, Weights): Although it has a low impact on the user, it can still be considered annoying Possible solution and/or Trade-offs: Provide incentive or remove completely Relationships: UE-06 57

58 Attachment 10.6 No. UE - 06 Name: Good notification Evidence: Users liked having simple notifications instead of an entire tab Good aspect Explanation: Users liked having simple notifications instead of an entire tab Severity or Benefit: Rating: 0 Justification (Frequency, Impact, Persistence, Weights): Good feature Possible solution and/or Trade-offs: Relationships: UE-05 58

59 Attachment 10.7 No. UE - 07 Name: Menu not intuitive Evidence: This problem lies in the prototype which displayed sorting categories Problem Explanation: Users found that the icon at the top right, was not very intuitive Severity or Benefit: Rating: 2 Justification (Frequency, Impact, Persistence, Weights): Users won t be able to find necessary features Possible solution and/or Trade-offs: Have a tab that says sort Relationships: UE

60 Attachment 10.8 No. UE - 08 Name: Bad naming convention Evidence: This problem lies in the prototype which displayed sorting categories Problem Explanation: Users did not like the naming in the sorting fields Severity or Benefit: Rating: 2 Justification (Frequency, Impact, Persistence, Weights): Users won t be able to sort according to their mental model Possible solution and/or Trade-offs: Arrange sort options by title, genre, date purchased, etc. Relationships: UE

61 Attachment 10.9 No. UE - 09 Name: Too many notifications Evidence: This problem lies in the prototype which displayed the notification Problem Explanation: Users said that it would become annoying if notifications appeared repeatedly Severity or Benefit: Rating: 1 Justification (Frequency, Impact, Persistence, Weights): Although it has a low impact on the user, it can still be considered annoying Possible solution and/or Trade-offs: Have notification appear only once Relationships: UE-06, UE-05 61

62 Attachment 11 Test plan and Test protocol Med. Fid. Prototype Attachment 11.1 Test plan Bookfusion has been tested and evaluated for usability and several prototypes were created to find different ways to improve the system. To gather objective data, we will conduct a usability test that shows the benefits between these interfaces. Overall objectives for the study The goals of this study are to: - Assess the best design between the current system and prototypes - Identify obstacles that users face while interacting with the system - Create a repeatable usability study protocol Research questions In addition, this study will try to answer the following questions: - Does a change in the display affect the length of time for a user to sort their books? - Are specific notifications more accessible for users? - Is the bookmark feature of the books easier to notice when compared to previous versions? At the end of the sessions, we will have qualitative data: - Visibility of bookmark whether or not the bookmarking tool was noticeable - User s think aloud statements We ll also have quantitative data: - Book sorting time The time required to sort books in a particular order - Sorting time Time taken to notice and sort books in specific category - Notification time The time it takes to receive notifications on a specific subject Location and setup These sessions will be conducted in a controlled setting. They will take place in the usability studies lab at UMBC, Maryland. Participants will be given a prototype of the system and asked to complete several tasks. The length of time for certain tasks will be monitored and after completing the required task, a short questionnaire will be provided to the participant for them to complete. The participant will also be asked to think aloud. Recruiting participants Because of the large intended user base of the system, many on campus students are possible users. I would like to recruit 4 participants. The recruited participants will most likely have the following characteristics 62

63 CHARACTERISTIC DESIRED NUMBER OF PARTICIPANTS PARTICIPANT TYPE PILOT 2 REGULAR 4 TOTAL NUMBER OF PARTICIPANTS 4 GENDER MALE 2 FEMALE 2 AGE Methodology This usability study will be somewhat exploratory but will also gather assessment data about the ease of use of BookFusion. Participants will be broken into two groups. Those that use the original layout of the system and those that use the prototyped version. Data will be collected about task completion times, task success rates and overall fondness of the system. We will be using a between-subjects design Because of the nature of the tasks (especially bookmarking), there would be severe bias if we were to perform a within-subject study. In this study, each participant will be given a prototype of the system and asked to perform a series of tasks I will use 5 minutes to explain the session to the participant then begin the test. Each participant will be asked to perform all of the tasks on that one system. Participants will sort their books, choose a store related notification and also bookmark a page. After the test, I will conduct a short 3 minute interview. Pre-test arrangements Have the participant: - Load application on phone or computer Introduction to the session Discuss: - Participant s experience with usability studies - Importance of their involvement I the study - Moderator s role 63

64 - The protocol for the rest of the session. - Thinking aloud Background Interview Discuss the participant s: - Experience with e-book readers - Reasons for using an e-book reader Task (10 minutes) Participants will start using the interface to look at notifications, sort books and bookmark a specific book. Post-test debriefing (3 minutes) - Ask participant about what they liked or disliked about the system - Ask participant about noticeability of bookmarking and sorting feature - Ask participant about possible fixes to system Measures To answer these questions: - Does a change in the display affect the length of time for a user to sort their books? - Are specific notifications more accessible for users? - Is the bookmark feature of the books easier to notice when compared to previous versions? I will collect both performance and preference data during the test sessions: Performance: Number of tasks completed without assistance Length of time to complete tasks Preference Ease of use overall Usefulness of terms and labelling Report contents I will deliver a draft of the final report to my professors that: Briefly summarizes the background of the study including the goals, methodology, logistics and important participant characteristics Presents findings for the original questions to investigate Gives quantitative results and discusses specifics as appropriate to the question and the data Provides visuals of pages of Bokfusion that are relevant to specific questions where they will help reviewers understand what we are talking about Discuss the implications of the results Provides recommendations 64

65 Moderator role I will sit in the room with the participant while conducting the session. I will introduce them to the test and explain our roles. I will conduct the pre-test debrief and then I will tell them the tasks that need to be completed. I will not aid in the completion of any task but I will prompt them to maintain communication while completing them. After completion of all the tasks, I will conduct the follow up interview and thank the participant for their participation. Reviewing, tabulating, and analyzing data Using my notes and answers to interviews, Ill tabulate and analyze the data to answer the key questions with findings and recommendations. Ill provide these results in a final written report which will be given to my professors. Tasks Participants will be broken into two groups and asked to complete the same task but on different versions of the systems. The overall task will be to use the system as they would if they were a regular user. Group one will use the previous version of the system and group two will be asked to use the current version. The list of tasks for both groups are as follows: Task 1: Select a notification relevant to the store Task 2: Sort the bookshelf based on your own sorting preference Task 3: bookmark a specific book after reading to three pages. 65

66 Attachment 11.2 Test protocol Preparing for the session To get ready for the sessions, I will - Set up prototypes for participants Before each session, I will - Set up starting point for participant - Conduct pre-session interview After each session - Conduct post question interviews with participant - Note particular issues - Debrief with participants Session introduction Thank you for agreeing to take part in my study. My name is Dhuel. During the rest of the session, I ll be working from a script to ensure that my instructions to everyone who participates in the study are the same. The main goal of this study is to see the overall usability of the BookFusion application, there are no right or wrong answers. Before we begin the usability test, I will ask you a few questions to determine your familiarity with similar systems During the session, I will ask you to use the BookFusion application to complete a series of tasks. As you try to do these tasks, please try to do whatever you would normally do. Please try to think out loud while you re working. Just tell me whatever is going through your mind and remember that there is no wrong answer. You doing this helps us to understand what works or doesn t work about this application. This entire session should take approximately 15 minutes. Do you have any questions before we begin? 66

67 Pre-session interview Please rate these statements from 1, strongly disagree to 5, strongly agree by circling your choice. 1. I am very familiar with e-book readers Strongly disagree Disagree Neither agree nor disagree Agree Strongly Agree 2. I use e-book readers frequently Strongly Disagree disagree Neither agree nor disagree Agree Strongly Agree 3. E-book readers are a daily part of my life Strongly disagree Disagree Neither agree nor disagree Agree Strongly Agree Task Matrix Let s get started For the session today, I d like you to imagine that you would like to read a book using this BookFusion application. Select a notification related to the store Time: 3 min. Start at application s bookshelf Show me the steps you would take if you were trying to select a notification relevant to the store. Sort your books Time: 2 min. Start at application s bookshelf Show me the steps you would take if you were trying to sort the books Bookmark a book Time: 3 min. Start at application s bookshelf Select the L illustration book, read or skim to page four then place a bookmark on that page. 67

68 Close the session Time 5 min Having used the application, I would like you to answer a few more questions. Closing questions 1. I liked the application. Strongly Disagree disagree Neither agree nor disagree Agree Strongly Agree Why or why not? 2. It was easy to recognize the bookmarking feature. Strongly disagree Disagree Neither agree nor disagree Why or why not? Agree Strongly Agree 3. The notifications were clear Strongly Disagree disagree Neither agree nor disagree Agree Strongly Agree Why or why not? 4. Overall, the system was easy to use Strongly disagree Disagree Neither agree nor disagree Why or why not? Agree Strongly Agree 5. What improvements could be made to the system? 68

69 Data measures Quantitative data The length of time taken to sort bookshelf Time taken to notice and use bookmark feature Qualitative data Think aloud verbal protocol Noticeability of bookmarking tool Preference measures Ratings and rationale concerning: Ease of use overall 69

70 Attachment 12 Eye tracker instructions Attachment 12.1 User A 70

71 Attachment 12.2 User B 71

72 Attachment 13 Summary of prototype results Attachment Research question I want to know if the participants are able to recognize and use the bookmarking function easier using the redesigned prototype when compared to the previous Bookfusion version. 2. What did your participants do to answer this research question? Each participant was required to bookmark a specific book. Half of the participants used the redesigned version while the other half used the previous version. 3. What data did you gather to answer this research question? I measured the number of seconds that it took the participants to complete the bookmarking process. I measured this data using a stopwatch. A Likert scale was also provided which asked the user about the functions visibility. 4. How many participants did you have? What demographic data is relevant to this research question? I conducted this evaluation with 4 participants (2 saw the original design and 2 saw the prototype design). All of the participants were within the age range and have never used the Bookfusion interface before but were familiar with e-book readers. 5. What is/are your independent variable(s)? The design of the system (original vs prototype) 6. What is/are your dependent variable(s)? The amount of time taken to bookmark a page. 7. What result(s) did you find? The participants who used the interface with the yellow bookmarking feature, took less time to complete the task when compared to those that used the original system. 8. What are the implications of these results? I recommended that Bookfusion changes the color of their current bookmarking icon from grey to yellow. 9. What confounding variable(s) might explain these results? The users who used the prototype may have been more familiar with e-book readers which may have led them to more easily recognize the bookmark feature. 72