assignment #9: usability study and website design group #3 christina carrasquilla sarah hough stacy rempel executive summary This report will examine the usability of the website http://iwantsandy.com and based on the findings will suggest a homepage redesign. The testing will cover the main topics of the website including sign up, service provided, examples, widgets, accolades, 24/7 customer service, and the help guide. Testing was performed on the first week of November 2008 with 3 members on the test crew and 9 users tested. The test lasted around 10-15 minutes per user and was performed at the homes of the test crew. The findings were that users could locate and understand the main functions of the website http://iwantsandy.com such as the sign up, the services provided, the examples, the accolades, and the help guide but were slightly confused with the more advanced functions of the website such as the advertising widgets and 24/7 customer service forum. page 1
table of contents executive summary 1 purpose of the study 3 participants 3 method 3 step one: select a project 3 step two: user profile 3 step three: heuristic evaluation 4 step four: study planning 5 step five: prototyping and scenarios 5 findings and recommendations 5 sign up 5 service provided 5 examples 6 widgets 6 accolades 6 24/7 customer service 7 help guide 7 post questionnaire 7 original homepage 9 suggested homepage redesign 10 conclusion 11 appendix a: participant questionnaire 12 appendix b: consent form 14 appendix c: tasks 15 appendix d: post questionnaire 17 appendix e: script 19 page 2
purpose of the study This site was chosen because of its potential to improve. Overall, we felt that the site provides a valuable service but that it is presented in an unprofessional way. Inconsistent colors, fonts, and verbage throughout the site look unorganized which hurts a site that is trying to help the user get organized. However, the service the site provides is performed very well and with some basic graphic uniformity can be greatly improved. The homepage suggests the main usability goals of the site are to get the users to sign up, to explain the service provided, to provide examples of the service, to provide advertising widgets, to list accolades given to the website, to provide 24/7 customer service and a user help guide. participants This study tested 9 users that were contacted in person. The average profile of the users was: Age- 33-37 Sex- Female Country of Origin- United States Primary Language- English Income Level- $41-45K Highest Education Level- Some College Operating System- Windows XP Web Browser- Internet Explorer Computer Experience- 14.5 years Hours Daily Spent on Computer- 7-8 hours http://iwantsandy.com Experience- None method step one: select a project Choose a website to do a usability study on and familiarize yourself with all pages including identifying the major goals of the site. step two: user profile Build a profile of your end user according to the following categories: 1. Age or date of birth 2. Gender 3. Location 4. Country of Origin 5. Languages Spoken 6. Income level 7. Education level 8. Operating system used/preferred 9. Browser used/preferred 10. Computer Experience 11. Internet/Web/email experience 12. Experience with this site page 3
step three: heuristic evaluation Perform a Heuristic Evaluation by answering the following questions: http://iwantsandy.com usability study Visibility of system status 1. Does the system keep users informed about what is going on, through appropriate feedback within reasonable time? 2. Does the site are tell users Where am I? and Where can I go next? 3. Is each page is branded and does it indicate which section it belongs to? 4. Stress Test Match between system and the real world 5. Does the system speak the users language? User control and freedom 6. Does the site support undo and redo? 7. Is there a home button on every page? 8. Are users forced into certain fonts, colors, screen widths or browser versions? 9. Does the site have animated GIFs? 10. Does the site follow HTML standards and other specifications? Consistency and standards 11. Is the wording in the content and buttons consistent? 12. Do the page titles match the links that lead to the page? 13. Does the site use custom link colors (a color other than blue)? 14. Is the navigation where you expect to find it? 15. Is the search box located where you expect to find it? 16. Are any site features unusable? Error prevention 17. Do you receive any error messages when using the site? 18. Do forms offer a submission confirmation? Recognition rather than recall 19. Does the navigation clearly indicate which page you are using? 20. Are there bread crumbs defining the path to the current page? 21. Are the link labels descriptive? 22. Does the site appear to use linked images? 23. If so, was it obvious the image was linked? Flexibility and efficiency of use 24. Are pages of the site easy to bookmark? 25. Are frames used for the site? 26. Are temporary URL s used? 27. If forms are used, is GET used instead of POST? Aesthetic and minimalist design 28. Do dialogues contain relevant information? 29. Is there extra information on the page that could be eliminated? 30. If users need additional information, are there progressive levels of detail within the site? Help users recognize, diagnose, and recover from errors 31. Are error messages expressed in plain language, indicating the issue and offering a solution? Help and documentation page 4
32. Is documentation available for users when they need help? 33. If so, is it easy to read, focusing on the user s task and concrete steps to be carried out? 34. Is help integrated into the site, making it easy for users to find and use? step four: study planning This usability test will consist of: Welcome User will complete a pre-test questionnaire (appendix a), which will include basic demographic information, along with a consent form (appendix b). Observation Time Users will have an opportunity to explore the site, without interruption. Tasks Users will be asked to complete a set of tasks (appendix c). Post questionnaire A questionnaire will be administered where the users can provide feedback about their experiences (appendix d). step five: prototyping and scenarios A script (appendix e) will be followed by each member of the test crew. findings and recommendations sign up The main purpose of the site is to get users to sign up for an account. The sign up section was easily understood by the users. They were able to quickly locate and click on of the two sign up links. However, the image accompanying on the front page was unclear and user where unsure of what it is displaying and why. During testing, users were asked and indicated the following: Where would you go to sign up for an http://iwantsandy.com account? 89% clicked on the correct link on the first click 100% eventually clicked on the correct link 7 seconds was the average time it took to click on the correct link Does the site lead you to sign up? 89% said yes 11% said no service provided A main focus of the site is to explain the service of the site to its users. The service provided section of the website was also easily understood by the users. They were able to quickly locate and click on the How I Work link. However, once there, some users felt like the language was too cute. During testing, users were asked and indicated the following: Where would you go to learn if the site provides a service you may need? 67% clicked on the correct link on the first click 100% eventually clicked on the correct link 12 seconds was the average time it took to click on the correct link Did you have trouble finding this? 89% said no page 5
11% said yes examples Showing examples of how the service can be used is an important feature to educate users on how to use the site and how it might address their specific needs. The participants were asked a series of questions regarding finding examples of how to use the service. Where would you go to find an example of how to use the services this site provides? 100% clicked on the correct link on the first click 7 seconds was the average time it took to click on the correct link Do you need examples in addition to the service explanation? 67% of the participants wanted examples in addition to the service explanation 33% felt that the explanation alone was sufficient Do the examples provided match your situation? 67% fell within the categories listed 33% of the participants felt the example categories did not match their situation widgets Wikipedia defines a widget as a portable chunk of code that can be installed and executed within any separate HTML-based web page by an end user without requiring additional compilation. They are derived from the idea of code reuse. Widgets often take the form of on-screen tools that draw on Internet databases to show the information on the users widget so a web-browser does not need to be used. http://iwantsandy.com makes widgets available on its site so that calendar reminders can be brought to the user through whichever widget dashboard they may use. Participants were asked questions regarding finding and implementing the widgets offered by http://iwantsandy.com. Where would you go to find this site s advertisement widgets? 33% of participants achieved the successful end state in one click 56% took two clicks 11% took five clicks to find the appropriate page 17 seconds was the average time to complete this task Was this confusing? 89% of the participants found the previous question to be confusing 11% clearly understood what was being asked Overall, it was found that although a few users had heard of a widget, they weren t sure what it was; therefore, were unsure of where it would be logically located Do you know what a widget is? If yes, could you implement one? 44% of participants knew what a widget was 56% did not The users who knew what a widget was, were not all able to implement one accolades Accolades can build trust with potential users and is an important feature to build service credibility and security. Participants were asked questions regarding locating accolades given to http:// iwantsandy.com. Where would you go to read accolades earned by this site? 100% of participants were successful in locating this information in one click 11 seconds was the average completion time page 6
Is seeing accolades necessary for you to sign up for the site? 67% of participants did not feel it was necessary to see accolades in order to sign up 33% felt it was necessary 24/7 customer service Participants reported that they would need live support, but then indicated that they would use a help guide over this resource. If the live customer support was easier to locate, then participants may be more likely to use this feature. During testing, users were asked and indicated the following: Where would you go to contact a customer service representative for help? 22% in one click 22% in two clicks 22% in three clicks 11% in five clicks 11% in 10 clicks 60 seconds was the average time for finding this page Was this hard to find? 78% reported that customer support was difficult to find 22% two stated that it was easy to locate Do you require 24/7 service? 55% answered that they would need the support 44% answered that they would not need 24 hour live support help guide Users would prefer using a help guide in order to find the answers that they are looking for on the site, and it appears that this is relatively easy to find. During the study, users had an easier time with finding the help guide, so this may have contributed to the answers they provided. During testing, users were asked and indicated the following: Where would you go to find the help guide? 100% clicked on the correct link on the first click 7 seconds was the average time for finding this page 9 clicks was the average of to get to this page Was this confusing? 0% of the participants reported that the help guide was difficult to find Are you more likely to use a help guide or a live person? 100% reported they would more likely use the help guide than 24 hour live support post questionnaire Following the main usability questions, participants were asked and indicated the following: The questions were easily understood 78% 11% Neither or 11% The homepage, http://iwantsandy.com, contained adequate initial information 78% 22% The colors used were appropriate for the site page 7
11% Strongly 56% 22% Neither or 11% The navigation was grouped in a logical manner 56% 22% Neither or 22% The pages contained sufficient information 89% 11% Topic and page headers were easy to understand 78% 22% Terms used throughout the site were understandable 33% 11% Neither or 56% There were an adequate number of images 67% 11% Neither or 22% Images were clear and sharp 78% 22% The site was visually appealing 89% 11% The pages loaded quickly 89% 11% http://iwantsandy.com usability study On a scale of 1-10, 10 being the best, how would you rate this site based on what you experienced today? 11% Rated 4 11% Rated 5 22% Rated 6 44% Rated 7 11% Rated 8 Additional comments Thought the top utility links were too obscure and didn t like that the screen shot on the front page was too small to really see an example of use. Interesting concept, but a little confusing - could be better done. I don t like that so much of it is based on text messaging. This site is really good service idea and I m going to start using it. Front image (table in the middle) is hard to see so not sure what is shows, sandbox and buzz are not familiar terms. page 8
Needs more than the one image, sandbox and community terms don t make any sense, how I work isn t the best description, overall the site is too cute and uses cutesy terms and explanations. Names of sandbox and buzz are confusing terms. I could do this service myself. Interesting concept. I liked the look and feel of the site. I want to start using the site, but I keep feeling there is a catch somewhere. original homepage page 9
suggested homepage redesign Life moves too fast for your calendar. sign up for free log in [ Home ] [ How I work ] [ Examples ] [ Extras ] [ News ] [ Help] Hi! I m Sandy, your new assistant. I'll remember the details so you can focus on what's important. Kick off your morning knowing what's on for the day. Keep your day moving forward with reminders right when you need them. Books to read, people to call, your cousin's birthday I'll remember it all. Stay organized with everyone in your life effortlessly. All without leaving your email inbox. Everyone deserves an assistant... Sign up for free! Already a client? Log in About us Blog Buzz Policy Terms of use Privacy A Values of n, Inc product Powered by Stikkit Copyright 2006-2008 Really can't stand Comic Sans? Sans Avec the sans page 10
conclusion Based on the usability study of the website http://iwantsandy.com we find that users could locate and understand the main functions of the website such as the sign up, the services provided, the examples, the accolades, and the help guide but were slightly confused with the more advanced functions of the website such as the advertising widgets and 24/7 customer service forum. Most of the problems were due to confusing terms such as sandbox for widgets (although most completely unaware of what a widget is) and buzz for accolades, and community for 24/7 customer service (especially when there is also a help guide). Our recommendations for a follow up study would be to implement our suggested changes and retest with a much larger number of participants. page 10
appendix a: participant questionnaire The following answers will be confidential and are for demographic purposes only. 1. Age: 18-22 23-27 28-32 33-37 38-42 43-47 48-52 2. Sex: Male Female 3. Country of origin: 4. Primary language spoken: 5. Income level: $20,000 - $25,000 $26,000 - $30,000 $31,000 - $35,000 $36,000 - $40,000 $41,000 - $45,000 $46,000 - $50,000 $51,000 - $55,000 $56,000 - $60,000 6. Highest education level completed: High School Some College Bachelor s Degree Master s Degree Doctoral Degree 7. Which operating system do you use? Windows 2000 Windows XP Windows Vista Mac OS Classic Mac OS X Linux 8. Which web browser do you prefer? Internet Explorer Safari http://iwantsandy.com usability study page 12
Firefox Opera 9. How many years have you been using a personal computer? 10. On average, how much time do you spend on a computer each day? Not at all 0-2 hours a day 3-4 hours a day 5-6 hours a day 7-8 hours a day 9-10 hours a day 11. How much experience do you have with the iwantsandy.com website? None Little Moderate Experienced page 13
appendix b: consent form iwantsandy.com Usability Testing Thank you for participating in our usability testing http://iwantsandy.com. Your usability test will be recorded by webcam for informational purposes only. Your feedback and participation will beneficial to the future growth of this site. This video will not be viewed by anyone outside of the evaluation team. I understand that a video recording will be made of my usability testing. I grant the usability test group permission to use these recordings for the purposes mentioned above, and waive my right to review or inspect the tapes prior to their dissemination and distribution. Please print name: Signature: Date: page 14
appendix c: tasks Task 1: Sign Up Script: Where would you go to sign up for an http://iwantsandy.com account? Start state: http://iwantsandy.com Success Criteria (end state): http://iwantsandy.com/signup Notes: Number of clicks to complete: Time to complete: Script: Does the site lead you to sign up? Task 2: Service Provided Script: Where would you go to learn if this site provides a service you may need? Start state: http://iwantsandy.com/signup Success Criteria (end state): http://iwantsandy.com/how_i_work Notes: Number of clicks to complete: Time to complete: Script: Did you have any trouble finding this? Task 3: Examples Script: Where would you go to find an example of how to use the services this site provides? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/examples Notes: Number of clicks to complete: Time to complete: Script: Do you need examples in addition to the service explanation? Script: Do the examples provided match your situation? Task 4: Implement Advertisement Widgets Script: Where would you go to find this site s advertisement widgets? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/sandbox#widgetbox http://iwantsandy.com usability study page 15
Notes: Number of clicks to complete: Time to complete: Script: Was this confusing? Script: Do you know what a widget is? If yes, could you implement one? Task 5: Site Accolades Script: Where would you go to read accolades earned by this site? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/buzz Notes: Number of clicks to complete: Time to complete: Script: Is seeing accolades necessary for you to sign up for the site? Task 6: 24/7 Online Customer Service Script: Where would you go to contact a customer service representative for help? Start state: http://iwantsandy.com/ Success Criteria (end state): http://getsatisfaction.com/iwantsandy Notes: Number of clicks to complete: Time to complete: Script: Was this hard to find? Script: Do you require 24/7 service? Task 7: Help Guide Script: Where would you go to find the help guide? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/help?context=%2fhelp%3fcontext%3d %252Fbuzz Notes: Number of clicks to complete: Script: Was this confusing? Script: Are you more likely to use a help guide or a live person? http://iwantsandy.com usability study page 16
appendix d: post questionnaire Please answer the following questions based on your experience using http://iwantsandy.com. Task Questions/Issues: 1. The questions were easily understood. Site Design: 2. The homepage, iwantsandy.com contained adequate initial information. 3. The colors used were appropriate for the site. 4. The navigation information was grouped in a logical manner. 5. The pages contained sufficient information. 6. Topic and page headers were easy to understand. page 17
7. Terms used throughout the site were understandable. 8. There were an adequate number of images. 9. Images were clear and sharp. 10. The site was visually appealing. 11. The pages loaded quickly. Overall Impression Of The Site On a scale of 1 to 10, 10 being the best, how would you rate this site based on what you have experienced today? 1 2 3 4 5 6 7 8 9 10 Comments/Suggestions All comments are welcome. page 18
appendix e: script INTRODUCTION Before testing begins, the testing crew will briefly introduce themselves and explain how testing will work and how long the test will take. Test Crew 1: Hi, We are Christina, Sarah and Stacy and we are doing a group project to test a websites usability. We have a few brief questions to ask in order for us to analyze the current site s usability and then suggest improvements. It will only take 10 minutes and you do not need any previous knowledge about the site or usability. Test Crew 2: In order to evaluate the site s navigation, we will ask you where you would click to complete a task based on the functions that the site provides. Test Crew 3: In other words, where would you click to do (blank)? SCENARIO You are trying to find information on the website http://iwantsandy.com. TASKS The testing crew will have a computer on hand that will be open to the website http://iwantsandy. com. Each member of the testing crew will present a task and ask the participant where they would click to find the information. Task 1: Sign Up Script: Where would you go to sign up for an http://iwantsandy.com account? Start state: http://iwantsandy.com Success Criteria (end state): http://iwantsandy.com/signup Notes: Number of clicks to complete: Time to complete: Script: Does the site lead you to sign up? Task 2: Service Provided Script: Where would you go to learn if this site provides a service you may need? Start state: http://iwantsandy.com/signup Success Criteria (end state): http://iwantsandy.com/how_i_work Notes: Number of clicks to complete: Time to complete: Script: Did you have any trouble finding this? page 19
Task 3: Examples Script: Where would you go to find an example of how to use the services this site provides? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/examples Notes: Number of clicks to complete: Time to complete: Script: Do you need examples in addition to the service explanation? Script: Do the examples provided match your situation? Task 4: Implement Advertisement Widgets Script: Where would you go to find this site s advertisement widgets? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/sandbox#widgetbox Notes: Number of clicks to complete: Time to complete: Script: Was this confusing? Script: Do you know what a widget is? If yes, could you implement one? Task 5: Site Accolades Script: Where would you go to read accolades earned by this site? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/buzz Notes: Number of clicks to complete: Time to complete: Script: Is seeing accolades necessary for you to sign up for the site? Task 6: 24/7 Online Customer Service Script: Where would you go to contact a customer service representative for help? Start state: http://iwantsandy.com/ Success Criteria (end state): http://getsatisfaction.com/iwantsandy page 20
Notes: Number of clicks to complete: Time to complete: Script: Was this hard to find? Script: Do you require 24/7 service? Task 7: Help Guide Script: Where would you go to find the help guide? Start state: http://iwantsandy.com/ Success Criteria (end state): http://iwantsandy.com/help?context=%2fhelp%3fcontext%3d %252Fbuzz Notes: Number of clicks to complete: Script: Was this confusing? Script: Are you more likely to use a help guide or a live person? CONCLUSION Test Crew: Thank you for participating in our usability survey. If you have any further questions, please contact us. page 21