Usability Report for Online Writing Portfolio October 30, 2012 WR 305.01 Written By: Kelsey Carper I pledge on my honor that I have not given or received any unauthorized assistance in the completion of this assignment. All work contained herein is my own. All referenced work is cited correctly.
Writing Portfolio Usability Report 2 Table of Contents Introduction 3 Methodology 3 Step 1: Gather Usability Information 3 Step 2: Perform Usability Testing and Gather User Feedback 3 Step 3: Analyze Results 4 Results 4 The Participant Demographic Questionnaire 4 Task- Based Protocols 4 Figure 1: Task Results 4 After- Test Survey and Questionnaire 5 Conclusion and Recommendations 5 Appendices 7 Appendix A 7 Appendix B 9 Appendix C 10 Appendix D 11 Appendix E 13
Writing Portfolio Usability Report 3 Introduction An online Writing Portfolio exhibits ones writing to peers and potential employers. The goal of creating my own Portfolio is to display my work in a user- centered and effective manner. The purpose of this report is to explain the usability testing I conducted to ensure my website is successful. This report begins by outlining my methodology, followed by the results of the testing. I will then discuss these findings and provide recommendations for the improvement of my Portfolio. Methodology To determine the usability and accessibility of my Writing Portfolio, I followed a three- step plan. Step 1: Gather Usability Information I first researched theory and practices of usability through the textbook Letting Go of the Words: Writing Web Content that Works by Janice (Ginny) Redish. I also read articles in class to learn the Principles of Readability and Usability Methods. Through this secondary research I identified three key areas to ensure that my Portfolio is user- centered: Clear Content: The information within the website must be easy to read and comprehend. Easy Navigation: Information on an effective website must be reached in very few steps. Clear Organization: The headings and pages within a website must be clearly labeled and lead the user to the desired information quickly. Step 2: Perform Usability Testing and Gather User Feedback I then conducted a usability test on my Portfolio to gather user feedback. Before the test, the participant completed a questionnaire to record her demographic information and self- identified computer proficiency. She then began the test, performing three tasks that required her to navigate to and from various pages in the site while seeking specific information. Finally, the participant filled out an after- test survey to record her opinions of the website.
Writing Portfolio Usability Report 4 Step 3: Analyze Results In order to analyze the results of the testing, I read the handout An Introduction to Coding and Analyzing Qualitative and Quantitative Data by Dr. Allen Brizee. This article helped me analyze the results of my testing while keeping in mind the three key areas identified in Step 1. Results The Participant Demographic Questionnaire The purpose of the pre- test questionnaire was to gather information about the participant s demographics, self- identified computer proficiency, and expectations of the website. She considered herself at an average level of technical competence, which implies that she has some familiarity with the internet, but might have more trouble than someone with more experience. The participant wrote that the most important features in a user- centered website are ease of navigation, visuals, and simplicity/brevity of [the] overall site. She then added that she expected to see the author s bio, résumé, interests, samples of previous work, and contact info. Task- Based Protocols The purpose of the tasks during the testing was to gauge the usability of the content, navigation, and organization of the Portfolio. During the testing, the time on task, number of mouse clicks, navigation path, and participant s response were recorded. The gathered information was mainly quantitative, and the results are as follows: Task What is the first news story summary written by the portfolio author? Where is the author of the portfolio from originally? Time in Seconds Mouse Clicks Navigation Path 06.1 1 Homepage to News Page 22.2 1 Homepage to About Me What is the author s email address? 07.5 1 Homepage to About Me Figure 1, Task Results Answered Correctly Yes Yes Yes
Writing Portfolio Usability Report 5 After- Test Survey and Questionnaire The purpose of the after- test questionnaire was the gather the participant s feedback. Seven Likert scale questions collected quantitative data while five open- ended questions collected qualitative data. In the Likert scale portion of the survey, the participant gave the following ratings on a 1-5 (low to high) scale: Overall, this web site rates a: 4 The content of this web site rate a: 4 The design of the web site pages rate a: 5 The organization (taxonomy) of the web site rates a: 4 On a Likert scale of Strongly Agree to Strongly Disagree, the participant gave the following ratings: The web site is usable: Strongly Agree This web site makes me feel as if the author/designer is credible: Agree I would feel comfortable hiring this writer: Agree In the qualitative portion of the survey, the participant suggested adding more personal information as well as samples of work to the site, as well as adjusting the placement of the About Me section. She added that the website is simple and easy to navigate, but the pictures could be varied on each page. Conclusion and Recommendations The participant gave both positive and negative feedback on the Portfolio. She praised the design, usability, and navigation of the website, and felt that the designer was credible. Overall, the participant rated the website as good. There were a few issues with the website, however, that need to be addressed. The participant had trouble locating the About Me section of the website, and felt that the information should be located on the homepage. She also thought that adding more writing samples would improve the website. To address these issues, I will relocate the About Me information to the homepage and add other writing samples and my résumé to the Portfolio.
Writing Portfolio Usability Report 6 Thank you for taking the time to read this report. If you have any questions or concerns about the testing or recommendations, please feel free to contact me: Kelsey Carper: kacarper@loyola.edu
Writing Portfolio Usability Report 7 Appendices Appendix A WR305, Writing for the Web Usability Test Documents Introduction This document contains the portfolio usability test informed consent form, participant demographic questionnaire, protocols, and surveys that you will use to assess your website in week seven of the term and help you revise it for the end of the semester. Normally, web writers and designers would develop their own test methods. But in many cases, customized testing resources are augmented by existing models. To simplify our testing process, I am providing these resources. If you feel that they do not work with your site or provide you with information that you need, let me know, and we can discuss altering or expanding them to match your requirements. Usability Testing Script and Protocols Purpose: The purpose of this test is to collect data from participants while they use students online portfolios. Portfolio authors should use this data to help them improve their web sites for a final grade at the end of the term. Under normal testing conditions, designers would collect data from at least 11 participants. Moreover, data would be collected by multiple data recorders. For this test session, however, portfolio authors will collect data individually and work with one participant. The basic tests contained here will form the baseline for data collection that will be augmented by eye tracking tests later in the term. Portfolio authors will use eye tracking data collected using the EyeGuide system to help make final changes to their web sites. Procedure: Please follow the steps outlined below. Your script is indicated in italics: 1. Hand the informed consent form to the participant and explain what the form is and what it says. Ask the participant if s/he has any questions. Then ask the participant to sign the form if they agree to the terms. 2. Hand the demographic questionnaire to the participant and explain what the form is and what it says. Ask the participant if s/he has any questions. Tell the participant that s/he may choose to not answer any questions s/he is not comfortable answering. 3. Read the following:
Writing Portfolio Usability Report 8 a. Thank you for agreeing to spend time with us today. Please remember that you can stop testing at any time for any reason. If you are uncomfortable or want a break for any reason, simply tell me and we will take a break. My name is (FIRST NAME), and I can answer any questions that may occur to you during testing. In addition to walking you through these tests, I am recording your navigation paths, timing how long it takes you to complete tasks, counting mouse clicks, and noting phrases that you use to describe the web site we are testing today. Overall, this test should take about an hour. We may break at any time, or you may choose not to continue the testing if you feel uncomfortable.
Writing Portfolio Usability Report 9 Appendix B Informed Consent Form (This is a sample form used for class and not part of a formal research study) Purpose of Research The purpose of this research is to test the usability of students online portfolios. Students are completing these portfolios as part of coursework for WR305. Specific Procedures to be Used Participants will fill out a demographic survey, complete task- based protocols on the web site to find information, and fill out a post- test questionnaire. Duration of Participation The usability test session should take about an hour. Benefits to the Individual Participants will learn about usability testing and about using a writing student online portfolio. I may contribute to a body of knowledge that supports innovation in web writing and usability research. Risks to the Individual Risks for participating in this study are minimal and are no more than I would encounter in everyday life or in the course of my studies at Loyola University Maryland. Compensation Participants will not receive any compensation for this study. Confidentiality Confidentiality cannot be guaranteed because of the nature of large groups - the class. Participant responses cannot be guaranteed to be anonymous because Allen Brizee cannot guarantee that other classroom participants will not disclose my participation in the study. Furthermore, documents created for the course will be posted on the Internet. Information collected from me during the test will be stored by the tester (the portfolio author). Aggregate results - data compiled from test responses - will be shared with the class and included in usability reports that will be posted online. No information from this course- based study will be published in peer- reviewed venues. Voluntary Nature of Participation While the usability testing is part of the class requirements, I understand that I can stop the testing process at any time without penalty. Participant s Signature and Date
Writing Portfolio Usability Report 10 Appendix C Participant Demographic Questionnaire Age: Gender: Male Female I prefer not to identify Race: 1. On a scale of 1 to 5 (with 1 being the lowest and 5 being the highest), how would you describe your level of technical competence, e.g., levels of comfort or skill using the Internet? 1 2 3 4 5 2. How many hours per week do you spend on the Internet? 1-5 6-11 12-17 18+ 3. Why do you use the Internet? Please circle all that apply. Pleasure (to find out what I would like to know, to watch television shows, play games, etc.) School work or research Facebook or other social media Emailing Banking Other (please explain) 4. What are some features that you think are important in making a web page user- centered? 5. What kinds of information would you expect to see in a writer s online portfolio?
Writing Portfolio Usability Report 11 Appendix D Task- Based Protocols This protocol is a quantitative, task- based process combined with a short questionnaire about the experience. The proctor will ask the participant questions that will require them to find information in the online portfolio, time the participant, count mouse clicks, and track navigation paths. Note: Make sure each computer has the home set to the landing page of the portfolio. Note #2: The proctor is permitted to repeat the task or clarify questions about the task itself but is not to assist the participant or answer questions relating to the navigation of the site as a whole. (i.e., the proctor can help the participant understand the task but not complete it). Proctor: The computer in front of you is displaying the landing page of the web site we are testing today. During this phase of the test, we would like you to use the site to find specific information, which will be given to you shortly. You are not going to be evaluated on your proficiency with navigating the Internet or using the computer, rather, this test is to see how well the site allows you to find particular information. That is, we are testing the site and not you. Task 1: What is the first news story summary written by the portfolio author? Thank you. Please return to the portfolio s homepage. Time: Number of mouse clicks: Navigation path: Did the participant answer the question correctly? Y N Task 2: Where is the author of the portfolio from originally? Thank you. Please return to the portfolio s homepage. Time: Number of mouse clicks: Navigation path: Did the participant answer the question correctly? Y N
Writing Portfolio Usability Report 12 Task 3: What is the author s email address? Thank you. Please return to the portfolio s homepage. Time: Number of mouse clicks: Navigation path: Did the participant answer the question correctly? Y N
Writing Portfolio Usability Report 13 Appendix E After- Test Survey and Questionnaire Please fill out this survey and questionnaire based on your experiences with the portfolio and this testing session. Thank you for your time today. On a scale from 1 (lowest score) to 5 (highest score), please circle your response to the following: 1. Overall, this web site rates a: 1 2 3 4 5 2. The content of this web site rates a: 1 2 3 4 5 3. The design of the web site pages rates a: 1 2 3 4 5 4. The organizations (taxonomy) of the web site rates a: 1 2 3 4 5 Please respond to the Likert scale questions below (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree): 1. This web site is usable: SA A N D SD 2. This web site makes me feel as if the author/designer is credible: SA A N D SD 3. I would feel comfortable hiring this writer: SA A N D SD What did you like about the web site? What did you not like about the web site? What suggestions do you have for improving the web site? What did you think of the testing process? How would you improve the testing process?