Web Evaluation Report Guidelines Graduate Students: You are required to conduct a usability test for your final project in this course. Please review the project description and the evaluation rubric on how usability will be factored into your final assignment grade. Here in a nutshell is what you need to do for your Final Project related to usability testing. 1) Read through this document. 2) Conduct ONE observation (Appendix A) 3) Conduct TWO Surveys using the one page evaluation form (Appendix B) 4) Write up a report summary, which should include: a. A summary of survey results b. A summary of observations on the tasks you asked users to complete (Appendix C) c. Major findings and recommendations d. Reflection on your experience in performing the Usability evaluation 5) Upload your report to the Web Portfolio Final Project assignment in Sakai The entire report should not exceed five double-spaced pages. You do not need to include your observation note sheets or the original survey report forms. The Appendix C file may be used as a template for your report. An Overview of Usability Evaluation An important concept that web designers and developers must keep in mind is Usability. Usability is an objective measure of the quality of users experience with your website. Here is an excellent definition by well known Web usability guru Jakob Nielsen: Usability is a quality attribute relating to how easy something is to use. More specifically, it refers to how quickly people can learn to use something, how efficient they are while using it, how memorable it is, how error-prone it is, and how much users like using it. If people can t or won t use a feature, it might as well not exist. (Prioritizing Web Usability, pp. xvi) How usable a website is often depends on the purpose and target audience of a particular website. However, there is general agreement that usable websites are accessible, appealing, consistent, clear, simple, navigable and forgiving of user blunders (Murray & Costanzo, 1999). To ensure good usability of your website and web pages, you need to design it with the users in mind. Considering the User Perspective: A Summary of Design Issues http://www.webaim.org/techniques/userperspective/ Creating accessible websites has never been more important because of the adoption of accessibility standards by the U.S. government. Web designers and developers need to incorporate accessibility into web design to ensure that people with disabilities can successfully access their websites. Page 1 of 12
Adobe Accessibility Resource Center http://www.adobe.com/accessibility Note: In this course you are not required to validate your web page(s) against the existing U.S. standards for accessibility. However if you are designing a web page or website for the general public as part of a library, university, or school website you may be required to meet these official guidelines (known as section 508). Section 508 requires that U.S. government agencies' electronic and information technology is accessible to people with disabilities. The IT Accessibility & Workforce Division, in the U.S. General Services Administration's Office of Government Policy, has been charged with the task of educating Federal employees and building the infrastructure necessary to support Section 508 implementation. Federal employees and the public can access resources for understanding and implementing the requirements of Section 508 from this website: http://www.section508.gov/ How to Conduct a Usability Test After you create a web page how do you know that your design ideas are really working well for your users? The easiest way is to seek feedback from a representative sample of users who are most likely to use your website. This process is called usability testing. There are actually a variety of ways to conduct a usability test, including direct observation, laboratory experiments, interviews, expert reviews, or focus groups. All of these approaches have one thing in common: to get feedback for improvement. No matter what kinds of method(s) you select, it is critical that you carefully take notes and stay neutral while gathering information from users who are testing your website. Do not demonstrate the page before observing your user or lead the user s responses by providing comments or instructions about the site. You are simply there to observe how your test users behave or experience the web page or website to complete a set of common tasks that a typical user might perform when visiting the website. After the usability test is completed, you then analyze the feedback you collect and use it to make changes to the web pages or website that make sense for your audience For more information, see the excellent collection of resources for conducting usability testing at http://usability.gov. Page 2 of 12
APPENDIX A Usability Observation & Evaluation Observation Guide The purpose of this observation guide is to walk you through how you might conduct a simple usability observation with a user. Usability observation is an evaluation method commonly used to gather data from potential or target users while developing a web site. The feedback you receive from testing can help you determine if there are changes or improvements which need to be done before launching your web page(s)/site to the public. To complete this brief user observation, or usability test, you need to find a person whom you can actually observe as he/she interacts with your website. Choose someone who is not an expert if possible just an average user. Before you start actually having the user perform a series of simple tasks read through the following questions and then use them to adjust the sample evaluation script provided on pages 2-3 so it fits your needs. What aspects or items on your web showcase do you want to receive user feedback on? What kinds of simple tasks will you ask the user to perform while you observe him/her? What specific operations or steps do you want the user to attempt for each task? When the user starts to perform the tasks, make sure that you take careful notes as you observe the user interact with your website. Take notes about the user s actions, comments, suggestions, and any places or items on the page(s) that the user has trouble with. Make sure you record the type of computer, operating system, browser, and any other factors that might be important (place, day, time, distractions, etc). You should also record information about the user s level of computer knowledge, prior web experience, or experience with your web page(s). Sometimes it helps to use a tape recorder, or even a video recorder to capture the session to help you remember what happened when you refer to your notes. This observation session should probably take 15-25 minutes. Don t make the tasks difficult. Try not to guide the user, unless they are really confused and ask for help to finish a task. There are no right or wrong answers in this process (from the user s perspective), so don t make them think they are wrong when testing your web page/site like the customer in a store, the user is always right! Page 3 of 12
Sample Web Evaluation Script [Areas you need to fill in to personalize this assignment are in brackets] Thank you for agreeing to join me here today so I can observe you using the website I am creating for my final project in Introduction to Digital Media. Your experience working with the website/pages will help me refine the design and operation of the web page(s) so that they are easier to use and more effective for future users. During this experience, please remember that you are not being tested. Rather, we are working together to help me create a better website/web page(s). If anything I ask you to do seems difficult to you or confusing this is not a reflection on your ability, skill or knowledge; your experience will help me figure out what improvements are needed in my design. In order for me to understand what you are doing and thinking about, I'd like to ask you to do a few things while you try out the web page(s)/website. The first, and most important, is that you should constantly verbalize what you are thinking about while you are doing something -- if you are confused say what confuses you and why. If you like something (such as text or design) please say so. For example, if you were trying to play an audio or video file, or open a link to another page or image you might say: Now I am clicking on the audio link to play this song. If you encounter a problem, or are confused, say something like: This does not make sense to me. I can t get this video to play by clicking on the play button. Basically, I'd just like you to verbalize both the action(s) you are attempting, and what you are thinking while attempting it. Try not to ask me too many questions on how things should work, unless you are really stuck. Just try to perform the task(s) to the best of your ability. Ok are you ready? Do you have any questions before we start? First, I'd like to ask you how much experience you have using computers and the Internet. Would you consider yourself a novice user, an average user, or an advanced user? [You can insert other questions here if you like.] Page 4 of 12
Ok, now let s begin. Please take about 2 or 3 minutes to explore the page(s). Now here are some things I would like you to try [This is where you need to insert simple tasks, or task sequences that fit YOUR web showcase. (either read the tasks aloud or provide the user with a short paragraph on a handout or index card to read through before attempting the task). Give them one task at a time. When they finish Task 1, then give them Task 2, etc.] TASK 1 description. [You need to create the tasks and enter them here] Remember to verbalize what you are doing and thinking while you work. TASK 2 description. [You need to create the tasks and enter them here] Remember to verbalize what you are doing and thinking while you work. TASK 3 description. [You need to create the tasks and enter them here] Remember to verbalize what you are doing and thinking while you work. Thank you for helping me. [For the web showcase you should have at least one task that addresses each of the following: Navigation -- especially if you have more than one page Audio and Video files -- can users play them without asking for guidance, do they work, etc. Site design or readability -- How do users like the layout, color, text, on the page(s)? Is the site readable? Enjoyable? ] [You don t need more than 5 tasks keep them short and simple!] [At the end of each task you might ask the user What problems did you have with this task? (If it was not clear from their think aloud verbalizing while performing the tasks). At the end of ALL tasks you might ask some general closing questions see these on page 7 below. Page 5 of 12
NOTE: Use the following table, or one like it you create yourself, to take your notes when the user is working on the tasks. Task 1 Description: Item Start Time / End Time Errors or problems encountered when completing the task Observation Notes Amount of time to complete tasks is important as it may indicate variations in different users skills. User comments/suggestions Task 2 Description: Items Start Time / End Time Observation Notes Errors or problems encountered when completing the task User comments/suggestions Task 3 Description: Items Start Time / End Time Observation Note Errors or problems encountered when completing the task Page 6 of 12
User comments/suggestions Closing Questions: Now that we have completed the tasks, I have a few questions regarding your experience. Use sample questions like these below AFTER the user has completed the task above. Do you feel the web page/page(s) were easy or difficult to use? When completing the tasks did you feel that you had to stop and think for a while figuring out how to perform the task, or any steps in a task? Did the website work the way you expected? Could you give me an example of the most frustrating or difficult task I asked you to perform? Could you give me an example of the most pleasing or easiest task? What is your overall feeling about the web page or pages? Questions User Answer Page 7 of 12
Close Well, that's it! We're done. Thank you again for helping me figure out what might make my web showcase better. Do you have any questions or final thoughts? Page 8 of 12
APPENDIX B Usability Testing Sample Survey Evaluation Form (Modify this form as you see necessary and use with two different persons, not with the same user who already did the full user test). Usability Scale (Based on System Usability Scale developed by Digital Equipment Corporation, 1986) 1. I think that I would use this page frequently Strongly Strongly disagree Neutral agree 2. I found the page(s) hard to navigate 3. I thought the page and its features were easy to use 4. I think that I would need to be more technically literate to use this page 5. I found the various functions in this page worked well 6. I thought the features of the page(s) were too complex 7. I would imagine that most people would learn to use this page very quickly 8. I found the audios or videos cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going Page 9 of 12
APPENDIX C Usability Report Summary (sample) Your Name: URL Tested: Course: Summary of Users and Usability Tests Participant 1 Age / Gender Internet Experience Profession Thoughts about This User Usability Test Example : Survey/Test Method Date of Test Platform / Browser User Profile Test Context Summary of Observations Task 1 Task 2 Task 3 Description of the task Time spent to complete the task(s) Errors or problems identified by a user Overall user comments Major Findings and Recommendations Major Findings Decision for the findings or issues? or Issues Option 1: Modify?! Identify Solution Option 2: Keep as is?! Provide Rationale for modifying OR keeping. Page 10 of 12
Summary: After filling out the charts, write up a summary of the observations and major findings with the test and survey (everything you have done). Page 11 of 12
Reflection on your experience in performing the Usability evaluation exercise. What did you learn from your users from your observation session and the evaluation forms that helped you improve your final web showcase? Page 12 of 12