Memorandum. The report also includes appendices that include the information used to create the usability test and the artifacts generated from it.

Transcription

1 Memorandum Date: April 28, 2013 To: : From: Subjet: Stephanie Coleman, Southern Polytehni State University Dr. Carol Barnum, Southern Polytehni State University 900ers Projet Team: Ann Beebe, Niole MLaughlin, and Wendy Stengel Usability report on Attahed is the 900ers projet team s final report on the web site usability study. This report provides the findings from the usability study of the SPSU.edu web site. It explains how the 900ers shaped and onduted the test, as well as the results found. It inludes the following information: Methodology explanation of how we seleted partiipants and formed qualitative surveys, test objetives, and proedures Key Findings disussion of the major findings from eah senario, both positive and negative, with supporting evidene Reommendations suggestions based on our findings on how to improve SPSU.edu for prospetive online graduate students The report also inludes appendies that inlude the information used to reate the usability test and the artifats generated from it. We would like to thank you for your assistane and support on this projet. We would also like to thank our professor, Dr. Barnum, for her advie, assistane, and support throughout the projet. We hope our findings will help SPSU enhane the spsu.edu web site so prospetive graduate students an find the information they need to hoose SPSU as their university.

2 SPSU.edu Usability Test Report April 28, 2013 Prepared for Stephanie Coleman, Diretor of Web Servies, SPSU The 900ers: Ann Beebe Niole MLaughlin Wendy AFG Stengel

3 Contents Exeutive summary... 1 Introdution... 2 Partiipant seletion... 4 User profile... 4 Partiipant sreening and questionnaire... 4 Test methodology... 6 Test environment... 6 Data olletion... 6 Senarios... 7 Post-senario questions... 8 Post-test questionnaire... 8 Findings... 9 Summary of key findings... 9 Senario and task summaries... 9 Senario 1 Home page... 9 Task 1.1 First impression... 9 Task 1.2 Labels Task 1.3 First lik Task 1.4 Look and feel Partiipant quotes Senario 2 Areditation and programs Task 2.1 Areditation Partiipant quotes Task metris Task 2.2 Program of interest Task metris Post-senario ratings Usability issues Senario 3 Program information Task 3.1 Required lasses Partiipant quotes Task metris Task 3.2 Eletive ourses Task metris Task 3.3 Online-only program Partiipant quotes The 900ers SPSU.edu Usability Test Report i

4 Task metris Task 3.4 Completion requirements Partiipant quotes Task metris Task 3.5 Program length Partiipant quotes Task metris Post-senario ratings Usability issues Senario 4 Online learning experiene Task 4.1 Prospetive online students Partiipant quotes Task metris Usability issues Post-senario rankings Senario 5 Apply Task 5.1 Program ost Partiipant quotes Task metris Task 5.2 Appliation deadline Partiipant quotes Task metris Task 5.3 Appliation requirements Partiipant quotes Task metris Task 5.4 How to apply Partiipant quotes Task metris Post-senario rankings Usability issues Post-test questionnaire Program quality Partiipant quotes Easiest parts of the web site to use Most diffiult parts of the web site to use Web site s influene on deision to apply Reommendations Further researh The 900ers SPSU.edu Usability Test Report ii

5 Conlusion Referenes Appendix A: Personas Persona 1: Potential undergraduate student Samantha Farber Persona 2: Potential undergraduate student Carrie Butler Persona 3: Potential graduate student Travis Treviño Appendix B: Heuristi evaluation Exeutive summary Introdution Methodology Persona Senario Heuristis Rankings Findings Tasks Evaluating SPSU s redentials and graduate programs Researhing degree requirements for a partiular program Learning about the online lass experiene Preparing to apply for a program Rankings Conlusion Appendix C: Test plan Exeutive summary Problem statement and test objetives User profile Demographis Tehnologial savvy Motivation Environmental and behavioral attitudes Subgroups Test methodology Seleting users Session length Roles and responsibilities The 900ers SPSU.edu Usability Test Report iii

6 Additional usability test logistis Test environment and equipment Software Hardware Inentives Senarios Data olletion Usability test shedule Deliverables Referenes Appendix D: Sreening questionnaire Appendix E: Pre-test onsent form Appendix F: Pre-test questionnaire Appendix G: Post-test questionnaire Appendix H: Moderator sript Overview/Briefing Senario Senario Senario Senario Senario Post-test wrap-up Appendix I: Session logs Partiipant 1 session log Partiipant 2 session log Partiipant 3 session log Partiipant 4 session log Partiipant 5 session log Partiipant 6 session log The 900ers SPSU.edu Usability Test Report iv

7 List of Figures Figure 1 Task 2.1 suess rates Figure 2 Task 2.2 program of interest Figure 3 Senario 2 onfidene and ease ratings Figure 4 Task 3.1 suess rates Figure 5 MBA eletives information 16 Figure 6 Task 3.2 suess rates Figure 7 Searh result showing perentages 5.aademiatalogsearh.om/isearh2/Mville / Figure 8 Task 3.3 suess rates Figure 9 Searh result for "MBA Thesis" Figure 10 MBA Transition Certifiate Program FAQ 21 Figure 11 Outdated atalogs page 22 Figure 12 Task 3.4 suess rates Figure 13 Task 3.5 suess rates Figure 14 Senario 3 onfidene and ease ratings Figure 15 Online Learning: Prospetive Students page 25 Figure 16 Desire2Learn Faulty Support 26 Figure 17 Senario 4 suess rates Figure 18 Senario 4 onfidene and ease ratings Figure 19 Tuition & Fees page 29 Figure 20 Online Learning Tuition and Fees page 30 Figure 21 Task 5.1 suess rate Figure 22 Graduate Studies page 32 Figure 23 Task 5.2 suess rate Figure 24 Task 5.3 suess rates Figure 25 Graduate Appliation Form sign in page 34 Figure 26 Task 5.4 suess rate Figure 27 Senario 5 onfidene and ease ratings Figure 28 Post-test questionnaire ratings The 900ers SPSU.edu Usability Test Report v

8 List of Tables Table 1 Partiipant demographis... 5 Table 2 Home page link ontent expetations Table 3 Areditation information loations Table 4 Senario 2 usability issues Table 5 Senario 3 usability issues Table 6 Senario 4 usability issues Table 7 Senario 5 usability issues Table 8 Reommendations by issue severity and level of effort The 900ers SPSU.edu Usability Test Report vi

9 Exeutive summary This report ontains our usability evaluation of the Southern Polytehni State University web site ( The 900ers team, a team of three graduate students enrolled in Dr. Carol Barnum s IDC 6120 lass, onduted a usability study foused on learning whether prospetive graduate students who are seeking a distane learning program are able to suessfully use the spsu.edu web site. The usability study measured how well the site helped prospetive graduate students: Evaluate SPSU s redentials and program offerings Researh degree requirements for a partiular program Learn about the online lass experiene Prepare to apply for a program We onduted the tests remotely, using online meeting tools, allowing us to use partiipants from aross the ountry. Partiipants shared their omputer sreens with team members who logged partiipant omments and browser ations as the partiipants ompleted a series of senarios. After partiipants ompleted all senarios, the moderator also asked them to omplete a post-test questionnaire. This questionnaire asked more detailed questions about the partiipants experiene using the web site. Partiipants onsistently had a positive reation to two aspets of the web site: Searh results are helpful Header and sidebar navigation work well We identified the following key issues with the web site, based on partiipants responses during the study: Partiipants have diffiulty finding areditation information Partiipants have diffiulty determining if ourses or programs are available as online-only ourses Partiipants are opposed to registering before seeing the appliation Text-heavy pages with inline links obsure important information Partiipants are onfused by inomplete and sometimes seemingly ontraditory information Some links led partiipants to information that was signifiantly different than what the partiipants expeted This report details our test methodology, findings, and reommendations for addressing the usability issues disovered during the test. It inludes supporting doumentation, inluding test logs, links to test reordings, and planning deliverables. The 900ers SPSU.edu Usability Test Report 1

10 Introdution SPSU is interested in learning whether prospetive graduate students who are seeking a distane learning program are able to suessfully use the spsu.edu web site. The university s goals for prospetive students are to get them interested in the shool, to give them a sense of SPSU s ulture, and to give the students enough information and flavor to make them want to begin and attain their degree. It is important that the web site leave users thinking. I was able to find what I was looking for and It worked like I expeted it to. From the university s perspetive, the ideal path for prospetive students is: Find information that makes them feel SPSU is the right shool for them Determine what it would be like to be an SPSU student Apply A heuristi evaluation that the 900ers onduted using our targeted persona indiated that the spsu.edu web site has the following problems: Navigation paths are unlear information is not always in logial plaes and, therefore, is sometimes diffiult to find The look and feel aross pages is inonsistent headers, sidebars, and in-page navigation treatments differ between the SPSU main web site and program-speifi web sites Link behavior is inonsistent while most links open pages in the urrent browser window, some links trigger new tabs without warning Credential information is diffiult to find both university areditation and professor qualifiations are diffiult to loate Based on the findings from the heuristi evaluation, the 900ers reated a usability test plan that foused on learning users experienes in these noted areas. The target users for the usability test were prospetive graduate students who were interested in a distane learning program. The usability study measured prospetive graduate students suess in: Finding information that will help them learn more about SPSU s redentials and program offerings Understanding what the degree requirements are for a partiular program, inluding what lasses are offered, whether a thesis is required, and how long it will take to omplete a partiular program Learning about SPSU s distane learning experiene, inluding whether the program they are interested in an be ompleted entirely through distane learning Understanding what they need to do to apply, inluding understanding program osts, whether a GRE is required, what douments are required, and how to loate an appliation The study also gathered feedbak about the overall look and feel of the spsu.edu web site. The 900ers SPSU.edu Usability Test Report 2

11 This study foused on funtional and strutural feedbak about the spsu.edu web site rather than web site design. It foused on the broader SPSU web site s key funtions, rather than the web site design unique to a partiular program. This report provides the findings from the usability study of the SPSU.edu web site. It explains how the 900ers shaped and onduted the test, as well as the results found. It inludes the following major information: Methodology explanation of how we seleted partiipants and formed qualitative surveys along with test objetives and proedures Findings disussion of the major findings from eah senario, both positive and negative, with supporting evidene Reommendations suggestions based on our findings on how to improve SPSU.edu for prospetive online graduate students The report also inludes appendies that inlude the information used to reate this usability test and the artifats generated from it. The appendies inlude: Appendix A: Personas list of the personas generated from initial interviews with potential online students; used to determine our primary user profile Appendix B: Heuristi Evaluation report generated from our initial review of the web site; used to frame the areas of fous for the test Appendix C: Test Plan plan for onduting the usability test and gathering findings Appendix D: Sreening Questionnaire list of questions we asked potential partiipants, to exlude those who did not onform to our primary persona Appendix E: Pre-test Consent Form doument ed and read to partiipants releasing their session reordings to SPSU and the 900ers team Appendix F: Pre-test Questionnaire list of questions asked at the beginning of eah test to reord fators of eah partiipant s unique testing environment Appendix G: Post-test Questionnaire questions asked at the end of the eah to gauge the partiipant s general impression of the web site and how it affeted their attitude toward the shool Appendix H: Moderator Sript doument that inludes the sript used by the moderator during eah test Appendix I: Session Logs observations and logs reorded for eah test The 900ers SPSU.edu Usability Test Report 3

12 Partiipant seletion User profile When reating user profiles for our usability study, we interviewed 11 people who are potentially interested in attending a university that offers distane learning degrees. As a result of our interviews, we reated three personas (see Appendix A): Travis an older potential graduate student Samantha an older potential undergraduate student Carrie a younger potential undergraduate student We foused our researh on potential graduate students and seleted Travis as our primary persona. To be sure that our study results would be broadly appliable aross the SPSU web site, we inluded three subgroups of partiipants. The subgroups were made of prospetive students who were interested in three different graduate degree or graduate ertifiate programs: Business Administration Information Design and Communiation Information Tehnology Partiipant sreening and questionnaire We reated a sreening questionnaire (see Appendix D) designed to reruit partiipants that resembled the demographis and attitudes of our primary persona. We reruited six partiipants who met all of the above riteria. Furthermore, eah partiipant was interested in one of three possible majors, with no more than two partiipants for eah major. Our partiipant demographis are shown in Table 1: Persona P1 P2 P3 P4 P5 P6 Name Travis Ann Tina Stefan Beky Jim Kristin Program of interest IDC IT IDC IDC MBA MBA IT Age 40s 40s 50s 30s 30s 50s 30s Employment FT FT FT FT FT FT FT Computer type Hours spent online weekly Laptop Laptop Laptop Laptop MaMini Desktop Laptop The 900ers SPSU.edu Usability Test Report 4

13 Persona P1 P2 P3 P4 P5 P6 Plans to start within 18 months Interested in online graduate program Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Motivation Improve job skills, keep learning Improve job skills, keep learning Career swith, keep learning Career swith, improve job skills Career advanement Improve job skills, keep learning Improve job skills, keep learning Table 1 Partiipant demographis The 900ers SPSU.edu Usability Test Report 5

14 Test methodology Test environment We onduted the usability test through online meeting tools, using GoToMeeting. We asked partiipants to do a small amount of tehnial set-up work prior to the test, and to meet some basi hardware requirements. The required hardware and onnetivity requirements inluded: Either a PC or Ma omputer Mirophone or the ability to dial in to the GoToMeeting all using a telephone while also using their omputer Before eah usability test started, we asked the partiipant to tell us the type of omputer they were using, what operating system they had installed, the Internet browser they were using, and what type of onnetion they had (T1, Cable Broadband, et.). We asked the partiipants to share their sreens with us, by making them the presenter in GoToMeeting so we ould observe their key strokes. In eah test, a moderator spoke to the partiipant and failitated the usability test, following a sript. The logger and observer for eah test remained behind the senes, not speaking to the partiipant. We reorded eah session. Data olletion During the study, we olleted both quantitative and qualitative data. Quantitative data inluded: Task ompletion suess rate Task ease ratings Suess onfidene ratings Qualitative data inluded: Partiipants omments and questions during the usability test Non-verbal ommuniation ues, suh as sighs, long pauses, using the mouse to hover over something for a long time, et. We logged task start and stop times in a Mirosoft Exel logging spreadsheet (Travis, 2010). We also used Mirosoft Word to note observations during eah test. We asked partiipants to omplete a post-test questionnaire (see Appendix G) that asked both qualitative and quantitative questions. The 900ers SPSU.edu Usability Test Report 6

15 Senarios The usability test inluded the following senarios and tasks. Goal: Get feedbak about the overall look and feel of the web site Senario 1 You ve reently deided that you would like to pursue a graduate degree or ertifiate program through a university that offers distane learning. You ve been searhing on the Internet, and you ome aross the web site for Southern Polytehni State University ( Tasks Goal: Learn more about SPSU s redential s and program offerings Senario 2 You re only interested in attending a university that is aredited and, of ourse, that has a ertifiate or degree program that you re interested in. Look around the home page. What do you think you an do from this web site? What do you think will happen if you lik on the tabs and links on the page? Are there any labels you don t understand? What is the first thing you d most likely lik on? What is your general impression of the web site from the homepage? Tasks From the spsu.edu home page, loate the information that allows you to answer: Is Southern Polytehni an aredited university? You had mentioned that you were interested in a <state partiipant s area of interest> program. Does Southern Polytehni offer a program like that? Goal: Understand what the degree requirements are for a partiular program Senario 3 You ve deided that Southern Polytehni looks like a good shool for you to attend, provided that the graduate program that you re interested in inludes lasses that you will enjoy, that you an take ompletely online, and that you an finish within three years. Tasks Using the spsu.edu web site, loate the information that allows you to answer the following questions: What are the required lasses for the program that you re interested in? What is an eletive lass for the program that you re interested in? Can you omplete the program that you re interested in ompletely online? If you re interested in a Master s Degree, does the program that you re interested in require you to omplete a thesis? If you re interested in a ertifiate program, what are the ompletion requirements? How long will it take you to omplete the program that you re interested in? The 900ers SPSU.edu Usability Test Report 7

16 Goal: Learn about SPSU s distane learning experiene Senario 4 You ve never taken online lasses before, and you re a little onerned about what the whole distane learning experiene might be like. Goal: Understand how to apply Senario 5 You ve got a lot of your questions answered, but you have a few final ritial questions before you deide to apply. Tasks Loate information that gives you an overview of what it s like to take online lasses and how you would interat with professors and other students. Tasks Loate the information that allows you to answer these questions: How muh will the program that you re interested in ost? When would you have to apply if you wanted to attend in Fall 2013? What appliation requirements do you need to meet in order to apply? How do you get an appliation form? Post-senario questions After Senarios 2, 3, 4, and 5, partiipants were asked to rate how easy the senario was to omplete and how onfident they were that they d found the information, on a sale from one to five. Post-test questionnaire After partiipants had ompleted all senarios, we asked them to rate the web site on a sale from one (negative) to five (positive). We foused on two types of evaluation: overall web site impressions, inluding ease of use, onfidene in results, and web site look and feel. We also asked them to rate their experiene finding information related to the university and their prospetive area of study, inluding their onfidene in SPSU s online lass experiene, whether their prospetive area of study is a respeted program, and their likelihood of applying to SPSU. The 900ers SPSU.edu Usability Test Report 8

17 Findings Our partiipants noted positive web site attributes and identified potential opportunities for hange. We present our findings below, in three parts: Summary of findings lists of positive findings and highest-severity usability issues found Senario and task summaries analysis of partiipants test sessions, task by task Questionnaire summaries analysis of pre- and post-test questionnaires Summary of key findings Partiipants onsistently had a positive reation to two aspets of the web site: Searh results are helpful Header and sidebar navigation work well We identified the following key issues with the web site, based on partiipants responses during the study: Partiipants an't find areditation information Partiipants an't determine if their programs are available as online-only ourses Partiipants are opposed to registering before seeing the appliation Many links do not take partiipants to the information they expeted Text-heavy pages with inline links obsure important information Partiipants are onfused by inomplete and sometimes seemingly ontraditory information Senario and task summaries Our findings from eah senario are summarized below. Senario 1 Home page Partiipants gave us their first impressions of the spsu.edu home page. In this senario, we asked them not to lik on anything on the page, but just to look at the page and tell us: What they think they an do from the web site Whether there are any labels they don t understand What they would likely lik on first Their general impression about the look and feel of the web site Task 1.1 First impression Partiipants thought they ould learn about: The ollege in general, inluding the admission proess, how muh it osts to attend, and what degree programs are offered The faulty and their bakgrounds The 900ers SPSU.edu Usability Test Report 9

18 Ways to keep up with shool ativities through soial media Information about upoming events and opies of news releases Donating to the shool Job openings and how to apply for jobs Partiipants expeted to find the following types of information under the following home page links. Under: About Us Researh Admission Campus Life Calendar Faulty Aademis Table 2 Home page link ontent expetations Information about: General information about the university or university programs Programs at the university with Researh Grants, what researh the university is doing Requirements, things you need to get in Current students Current events More about the faulty What degrees are offered Task 1.2 Labels All six partiipants indiated that they did not know what GaViewD2L or Banner Web meant, but most inferred that they were tools used by students, and therefore not intended for them as prospetive students. The presene of these links on the home page did not ause onfusion for any of the partiipants. Task 1.3 First lik All six partiipants said they would lik on the following links first (some gave two links): Aademis (five partiipants) Prospetive Students (five partiipants) About Us (two partiipants) Task 1.4 Look and feel Three of six partiipants found the homepage to be too luttered and busy. Two partiipants also ommented that there were a lot of links on the page. All six reognized that there were some links that, as a prospetive student, led to information they would be interested in, and other links that pertained to other audienes instead. Four partiipants ommented on the rotating photos on the home page: One said that she liked them Two said they were too big One said she would rather see more information on the page instead One thought they went by too quikly Two partiipants felt the homepage was lear, lean, and attrative. The 900ers SPSU.edu Usability Test Report 10

19 Partiipant quotes It s very fatual, blak and white, not friendly, it s very informational you get what you re looking for but graphially speaking, it s not a fun web site to go through. (P1) Senario 2 Areditation and programs We asked partiipants to determine whether SPSU is an aredited university and whether the shool offers a graduate degree or ertifiate program in the area of study they are interested in (Information Tehnology, Information Design and Communiation, or Business Administration). Task 2.1 Areditation This was the most diffiult information in the usability test for partiipants to find. All six partiipants began this task by looking under About Us. Partiipants expeted to find areditation information displayed prominently. Beause they didn t readily find the information, some partiipants initially inferred that the shool must not be aredited. In addition, partiipants were onfused about whether they would find areditation information under the university as a whole or under speifi programs. This onfusion generally ourred when partiipants used Searh to find the word aredited. They found information saying speifi programs were aredited, leaving them wondering whether just the program was aredited and not the university. After not finding the information on the About Us pages, partiipants looked in numerous plaes for it, and eventually found the areditation information in a variety of loations (see Table 3). Partiipant Loated areditation information P3 On the History page P4 Using the Index, under Our Aademi Programs P1, P6 Using Searh, in the PDF aademi atalog (one in the undergraduate atalog and one in the graduate atalog) P5 Using Searh, on the Computer Siene program page P2 Did not find the information Table 3 Areditation information loations Partiipants who found the areditation information in the ourse atalog (via Searh) were surprised to find it there; they expeted to find it somewhere more prominent on the web page rather than in a separate PDF doument. Two partiipants speifially mentioned they would like to see a prominently displayed seal or logo announing areditation so they don t have to read through text to searh for the information. Partiipant quotes It s usually something to brag about but I guess not. (P6) I would immediately go to About Us and start looking for the word aredited. Nothing. (P5) I would have liked to see a seal or something so I didn t have to read. (P3) I d say stik a logo right here on the homepage. (P1) The 900ers SPSU.edu Usability Test Report 11

20 Task metris Figure 1 Task 2.1 suess rates Task 2.2 Program of interest Our partiipants were interested in the following areas of study: Two in a Master s Degree in Information Design and Communiation Two in an MBA or Business Administration ertifiate Two in a graduate ertifiate in Information Tehnology Partiipants were easily able to find information about their seleted programs by starting with the Prospetive Students link. One on their respetive program pages, one partiipant (P5) had diffiulty determining whether a program was a Master s degree program or a ertifiate program. The same partiipant, looking for information about an MBA, had diffiulty determining the differene between a bridge ertifiate program, a Transition Certifiate program, and an MBA ertifiate program. He also wasn t sure what the differene was between a Management Tehnology program and a Tehnology Management program, whih were both listed. The 900ers SPSU.edu Usability Test Report 12

21 Figure 1 Department of Business Administration Bridge program page at Task metris Sueed with Diffiulty, 1 Sueed, 5 Figure 2 Task 2.2 program of interest The 900ers SPSU.edu Usability Test Report 13

22 Post-senario ratings Figure 3 Senario 2 onfidene and ease ratings Usability issues Issue Areditation information diffiult to find Differene between Masters and Certifiate programs not lear Differene between MBA, MBA Transition, and MBA bridge programs not lear Table 4 Senario 2 usability issues Severity High High High Senario 3 Program information All partiipants are interested in enrolling in an online-only graduate program. In Senario 3, we asked them to: Find out more about their speifi program of interest Confirm their program of interest is online-only Determine how long the program will take Task 3.1 Required lasses Five of the six partiipants began their task from the program page they loated in Task 2.2. Three of six partiipants found the required lasses quikly. Partiipants who did not find the information quikly either lost trak of the requested task while sanning through page text (P2) or were onfused by the differenes between Masters and Certifiate programs. Partiipant quotes This [IT ertifiate page] leads me to believe I m doing the same amount of work for a [ertifiate] or a Masters. (P6) The 900ers SPSU.edu Usability Test Report 14

23 Task metris Fail, 1 Sueed, 5 Figure 4 Task 3.1 suess rates Task 3.2 Eletive ourses Three of six partiipants found the requested information quikly, through diret navigation. One partiipant (P4) expressed pleasure at how quikly she found the information. One partiipant (P5) who was interested in the MBA program was frustrated by how diffiult it was to distinguish required ourses from eletive ourses. The MBA program page lists general areas students an hoose eletives from, but the partiipant never found the list of speifi lasses that he expeted. The 900ers SPSU.edu Usability Test Report 15

24 Figure 5 MBA eletives information The 900ers SPSU.edu Usability Test Report 16

25 Task metris Abandoned, 1 Fail, 1 Sueed, 4 Figure 6 Task 3.2 suess rates Task 3.3 Online-only program Four of six partiipants either failed to find information that onfirmed programs were online-only, or inorretly inferred that a program was not available exlusively online. While partiipants interested in all three programs had diffiulty loating this information, both IT Certifiate partiipants abandoned the task without finding the information. Partiipants interested in the Master s degree in IDC assumed the ourses are online beause of information they had seen on previous pages, but didn t speifially loate this information during this task. One partiipant interested in an MBA was onfused by the desription of flexible delivery, and assumed it meant that some lasses were online while others were not. The other MBA partiipant wrongly inferred during this task that the MBA program and ertifiate programs do not offer online lasses, but during a subsequent task found the information and was relieved. Partiipants who had diffiulty diserning if a program was online-only tried to figure it out by looking at individual ourse desriptions. In many ases, they saw passing mention of online ourses, but weren t sure how muh of the program is online. Partiipants expeted to see whether a ourse is offered online in the ourse desription. They also expeted this information to be prominently displayed. One partiipant (P1) searhed for online from a Graduate Catalog page. The searh results listed perentages behind them, referring to how relevant a result was to the searh terms (see Figure 7). However, the partiipant interpreted those perentages as the perentage of the program that was online. The 900ers SPSU.edu Usability Test Report 17

26 Figure 7 Searh result showing perentages One partiipant (P1) expeted this information to be in the ourse atalog, but was frustrated beause the atalog doesn t say when or how the lasses meet. Previous experiene had led her to believe the ourse atalog should have that information. Another partiipant (P6) found the PDF version of the ourse atalog assuming that she would find a list of available lasses. She did not, and was frustrated. One partiipant (P6) mistakenly navigated to the PDF undergraduate ourse atalog. She looked up Information Tehnology in the table of ontents. The table of ontents gave the page number 111 for Information Tehnology. When she went to page 111, the page was about the History program. She noted that the table of ontents was inorret. Partiipant quotes I might assume at this point that it s not online, sine it s not splashed in front of me. I would think this would be a big thing students would be looking for. (P1) [I m] not 100% onfident I an omplete everything online, but at least a portion an be. (P4) This would get a little frustrating if I had to researh this lass by lass. (P1) The 900ers SPSU.edu Usability Test Report 18

27 It says some ourses are fully online. I would assume no [the entire MBA is not fully online]. So, I would assume SPSU is not the right plae for me. (P5) I was thinking that I was going to open this [PDF] and just see the lasses that are available. I don t know if I m going in the right diretion. (P6) Task metris Sueed, 2 Fail, 4 Figure 8 Task 3.3 suess rates Task 3.4 Completion requirements One Master s partiipant (P2) suessfully answered the question about the IDC program not requiring a thesis, though she never found ontent that expliitly told her that information. Another Master s partiipant (P3) had seen requested information during an earlier task and foused on re-finding that same page. He was frustrated when he ouldn t quikly find the page again, but eventually loated it and found the requested information. Both prospetive MBA students abandoned the task and deided they needed to ontat someone at SPSU. One partiipant (P4) liked multiple plaes and was having diffiulty. She searhed on MBA Thesis and liked on a searh result that opened a PDF file from 2006, whih said no thesis was required (see Figure 9). She abandoned the task beause she felt the information was too old to be reliable. The 900ers SPSU.edu Usability Test Report 19

28 Figure 9 Searh result for "MBA Thesis" Certifiate partiipants started this task by loating their program pages. Two partiipants found the information easily, one through the program page and the other through Searh. One partiipant (P1) found the ompletion requirements for a graduate ertifiate in IT, but it took her a long time. She tried to lik on eah ourse on the program page, but ouldn t. She was frustrated beause she had explored these lasses before in the atalog. She didn t want to have to look through the atalog again, but she went there anyway. She found the program and the likable ourses. She found the prerequisites for one required ourse, but wasn t sure if she d need to take them. She remarked about the diffiulty of going through all of these ourses. One partiipant (P5) was partiularly hesitant about the information he was finding. He lost trak of the task goal, and delved deeper into the program requirements information. He liked on the MBA Transition Certifiate left hand link, and then on the FAQ link. He found a link to ourse ost information (see Figure 10). The 900ers SPSU.edu Usability Test Report 20

29 Figure 10 MBA Transition Certifiate Program FAQ This link led the partiipant to a page using an old visual template, ontaining out-of-date ontent (see Figure 11). The partiipant was inreasingly skeptial of the information on the web site from this point in the test forward. The 900ers SPSU.edu Usability Test Report 21

30 Figure 11 Outdated atalogs page Partiipant quotes To me, a PDF from 2006, that s 7 years ago, that s legay information that an t be trusted. (P4) I think I d need to take a pen and paper and write down what I d need to take. (P4) This is kind of sales-y; I ll go to the FAQ. (P5) I m surprised at this; other ertifiate programs don't have as many lasses and have some eletives. (P5) The 900ers SPSU.edu Usability Test Report 22

31 Task metris Fail, 2 Sueed, 2 Sueed with Diffiulty, 2 Figure 12 Task 3.4 suess rates Task 3.5 Program length Five partiipants quikly found the information through diret navigation their program pages. One partiipant orretly guessed at program length based on the number of required ourses. Partiipant quotes Eight ourses of eight weeks eah, sixty four weeks, a year, a year and a quarter, maybe some breaks in there. I m thinking it will take me under a year and a half to omplete. (P5) The 900ers SPSU.edu Usability Test Report 23

32 Task metris Sueed with Diffiulty, 1 Sueed, 5 Figure 13 Task 3.5 suess rates Post-senario ratings Figure 14 Senario 3 onfidene and ease ratings Usability issues Issue Outdated ontent appears high in searh results Online programs not learly identified Online lasses not learly identified Pages link to outdated ontent IDC program pages text-heavy MBA program pages text-heavy PDF ourse atalogs hard to navigate Undergraduate and graduate ourse atalogs diffiult to distinguish Searh behavior inonsistent between web site setions Overall program length not learly alled out Table 5 Senario 3 usability issues Severity High High High High Medium Medium Medium Medium Medium Low The 900ers SPSU.edu Usability Test Report 24

33 Senario 4 Online learning experiene As our partiipants want to join an online-only program, we asked them to find out what an online learning experiene would be like. Task 4.1 Prospetive online students All six partiipants quikly found the Online Learning page by liking on Prospetive Students, then Online Learning, and then Prospetive Students (see Figure 15). The two partiipants who suessfully ompleted the task were satisfied with what they found on that page. Figure 15 Online Learning: Prospetive Students page Four of the six partiipants liked links on this page to learn more speifi information about online learning. These partiipants reported that they were looking for examples with sreenshots of the online lassroom tools they d be using and more information about how they would do their work and interat with other students and their professors. The information they found did not support their information goals, and partiipants noted that the ontent seemed oriented toward faulty or urrent students. Indeed, the first in-page link led partiipants to the Desire2Learn Faulty Support page (see Figure 16). The 900ers SPSU.edu Usability Test Report 25

34 Figure 16 Desire2Learn Faulty Support From the Online Learning page, there is a link to SPSU's L.V. Johnson Library and a link alled online library materials that launhes the same page, but this page requires students to log in to see any information, so this isn t what the prospetive students expeted to see. From the Online Learning page, the Blakboard Wimba link takes prospetive students to a page that provides no useful information unless they already have an SPSU student user ID. From the Online Learning page, there is a link to Hornet Connet, but the page doesn t say what Hornet Connet is, and the link takes prospetive students to a page where they have to have an SPSU student user ID to log on. Partiipant quotes This information is a little tehnial, whih is fine, but if I m just trying to find out if I m going to like the online aspet of this whole thing, I d like some more examples of this. (P1) For me, for my detail oriented-ness, I always like sreen shots and what the lass would look like. (P4) I was thinking it would give sreen shots or something. (P4) Student organizations I m not sure what this has to do with online learning. It talks about on ampus, but I m not going to be on ampus. (P5) The 900ers SPSU.edu Usability Test Report 26

35 This gives me an overview, but it s not really satisfying. It doesn t give me any kind of flavor of what it s like. (P5) Everything looks good exept I don t know how I m going to talk to my professor. (P6) What I d really love to see is a video showing me how this works. For example, when I go to Stanford s web site, it shows a video for eah ourse and it shows what happened in the lass. (P5) Task metris Fail, 1 Sueed, 2 Sueed with Diffiulty, 3 Figure 17 Senario 4 suess rates Usability issues Issue Content not targeted to prospetive students Lak of multimodal ontent Table 6 Senario 4 usability issues Severity High Medium The 900ers SPSU.edu Usability Test Report 27

36 Post-senario rankings Figure 18 Senario 4 onfidene and ease ratings Senario 5 Apply At this point, partiipants have looked at their programs of interest and learned about SPSU s distane learning experiene. The next step would be for them to apply. We asked them to: Find out the how muh their program will ost Understand appliation deadlines and requirements Start an appliation Task 5.1 Program ost Five of the six partiipants found the Tuition & Fees page fairly easily using Searh or by starting at the Admissions or Prospetive Students links on the home page. However, five partiipants found the tuition and fees breakdown onfusing. Partiipants didn t know whether they should look at the fees for Graduate Studies or for Online Learning (see Figure 19). This was espeially true for partiipants who were still unsure if their program ould be ompleted entirely online or not. The 900ers SPSU.edu Usability Test Report 28

37 Figure 19 Tuition & Fees page Two partiipants were also unlear whether ampus-speifi fees applied to them, and if so, how often they would be harged (see Figure 20). Two partiipants ame to inorret onlusions about how muh their programs would ost. The 900ers SPSU.edu Usability Test Report 29

38 Figure 20 Online Learning Tuition and Fees page Partiipant quotes I don t know whether the lasses I need are all online or not; there s quite a prie differene. It s also not lear whether the graduate fees would be the same as a ertifiate or not? Beause I an t find the info I need, I d have to get pen and paper out, and find out the list of lasses, what pre-reqs, then find whether they re online, and then get the prie info. This ould be pretty time-onsuming. It seems like the info is on the web site, but it s just a matter of finding it. (P1) I m not sure what the fees mean. I m assuming those are one-time fees. (P3) The 900ers SPSU.edu Usability Test Report 30

39 Task metris Fail, 2 Sueed, 3 Sueed with Diffiulty, 1 Figure 21 Task 5.1 suess rate Task 5.2 Appliation deadline Five of the six partiipants began by using Searh or the Admissions or Prospetive Students links. They generally expeted to see the deadlines on the Graduate Admissions page (see Figure 22). Four partiipants liked on the appliation sign-in page expeting to see dates. One partiipant (P6) looked at the alendar for the dates first. Two partiipants made inorret onlusions about the deadline. When they landed on the Graduate Admissions page, only one partiipant (P6) hose the Priority Deadlines link first. Two liked on multiple links in the left navigation before finding the deadline link. The 900ers SPSU.edu Usability Test Report 31

40 Figure 22 Graduate Studies page Partiipant quotes I keep thinking I m going to find the date somewhere in here. (P2) I saw something that said apply. I guess I d lik on Apply, I d want graduate appliation form, looks like I an apply online, but I don t see anything about a deadline. (P3) The 900ers SPSU.edu Usability Test Report 32

41 Task metris Fail, 2 Sueed, 4 Figure 23 Task 5.2 suess rate Task 5.3 Appliation requirements Five of the six partiipants began by going to the Graduate Admissions page. Three of the partiipants found the information quikly. Two partiipants liked on Apply Now or various links in the Graduate Admissions area before they found the information. The one partiipant (P5) who abandoned the task was also the only partiipant who didn t look on the Graduate Admissions page. Instead, he found some information in an MBA Transition Certifiate FAQ page. He was onfused by the information. One partiipant (P6) liked on the Graduate Admissions Forms link and was surprised to see an appliation for readmission instead of a regular admission form. Partiipant quotes I m going in irles. (P2) If I m applying, then why am I applying for readmission? Am I in the wrong plae? (P6) Create aount here, but then it says to lik on reate aount. I would expet it to open in new page, but it doesn t so I don t have my instrutions anymore. I would assume these instrutions are wrong and just go to that page and go from there. It says to pay $50, not sure how I would do that. I assume it would ask me for a redit ard. I d have to all. (P5) The 900ers SPSU.edu Usability Test Report 33

42 Task metris Abandon, 1 Sueed, 5 Figure 24 Task 5.3 suess rates Task 5.4 How to apply All six partiipants easily found the online appliation, typially beause they had seen it during a previous task. Two partiipants were unhappy beause they thought they needed to register online before starting the appliation (see Figure 25). One partiipant (P6) expeted to download an appliation in the form of a PDF and abandoned the task, even though she had found the orret page. Figure 25 Graduate Appliation Form sign in page The 900ers SPSU.edu Usability Test Report 34

43 Partiipant quotes I don t like being asked for my address. I expet to see a form or something. It s kind of assumptive to ask me to register. (P6) Task metris Abandon, 2 Sueed, 4 Figure 26 Task 5.4 suess rate Post-senario rankings I'm onfident I found the information I intended to find Disagree, 1 Neutral, 1 Agree, 2 Strongly Agree, 2 It was easy to find the information I was looking for Neutral, 3 Agree, 2 Strongly Agree, 1 Figure 27 Senario 5 onfidene and ease ratings Usability issues Issue Requiring sign in to begin appliation is off-putting Distintion between graduate tuition and online tuition unlear Upoming deadlines not featured in body text Fee struture inadequately explained Table 7 Senario 5 usability issues Severity High High Medium Low The 900ers SPSU.edu Usability Test Report 35

44 Post-test questionnaire We used the post-test questionnaire to gauge how eah partiipant s experiene navigating spsu.edu affeted their overall impression of SPSU. We asked them to respond to the following questions on a sale of 1 (strongly negative) to 5 (strongly positive). Figure 28 Post-test questionnaire ratings The partiipants responses to the questionnaire were tepid, with the average rating for eah task being neutral. Program pages didn t give partiipants a sense that the programs were respeted or of high quality. The web site did not help partiipants assess what their online student experiene would be like. Program quality Fators that negatively influened partiipants onfidene level in the quality of the program inluded information on the web site being too shallow as ompared to other shools web sites (P5) and unertainty about whether the diffiulty they had using the web site would be similar to their experiene taking online lasses (P1, P5). Fators that positively influened partiipants inluded the shool offering programs that are wholly online (P4), and being able to find some of the information they were looking for easily. (P2, P3) Partiipant quotes They are aredited, but I didn't see anything else saying people love us or we're turning out the brightest and best. (P6) The 900ers SPSU.edu Usability Test Report 36

45 If the online ourse was offered on a site like this, I d be very unonfident but if it was offered on a blakboard type site, I d be onfident. (P1) I didn t get a lot of rih info about program itself or how the online experiene works. I wanted real rih info and I didn t get that. I got high level sales-y info and it wasn t good sales either. (P5) My feeling is if it s like using this site, it s probably not that great. (P5) Now I m more informed, and it seems like a lear, basi web site that has the big things I need. (P3) There were a few areas of the web site that I didn't immediately find, but I think I probably ould find it one I get used to the website. Most of the information was readily available...one I got the hang of an online lass, I think it would be easy. (P2) Easiest parts of the web site to use Partiipants found the following areas of the web site easiest to use: Quik links (P6) Prospetive students (P1) Admissions (P1) Searh (P4, P5) Home page (P3) About Us (P2) Aademis (P2) Most diffiult parts of the web site to use Lists of lasses in a partiular program had too muh text (P3) Finding areditation information (P2, P6) Finding the differene between Masters and Certifiation programs (P6) What ourses are needed for a speifi program (P1) Whether a thesis is required (P4) Information for prospetive students (P5) Finding appliation deadlines (P2) Web site s influene on deision to apply When asked how the web site would influene their deision about whether or not to apply to SPSU, responses were both positive and negative. Partiipants feedbak about this is best expressed in their own words: If my online lass experiene is like the web site experiene it doesn't give me a lot of onfidene. (P6) Just beause it was hard to navigate doesn t mean the shool itself is a bad plae. (P1) More likely [to apply] definitely. Mainly beause I've used other online web sites, speifially when I was teahing ommunity ollege when I was teahing. Those sites were so muh more work. This is definitely an improvement over what I used 7 years ago. (P2) The 900ers SPSU.edu Usability Test Report 37

46 Reommendations Below, we have provided reommendations to address eah usability issue identified in Senario and task summaries. We reommend prioritizing the low-hanging fruit : low level of effort hanges that address high-severity issues. We have provided our estimates of the level of effort required for eah improvement, but SPSU will be better able to determine the atual level of effort. Reommendation Issue Addressed Severity Level of Effort Add areditation information to Home and About Us landing page Rewrite Tuition & Fees pages Rewrite program ontent to highlight program delivery options Rewrite ontent Rewrite ontent Areditation information diffiult to find Distintion between graduate tuition and online tuition unlear Online programs not learly identified Differene between Masters and Certifiate programs for IT not lear Differene between MBA, MBA Transition, and MBA Bridge programs not lear Outdated ontent appears high in The 900ers SPSU.edu Usability Test Report 38 High High High High High Low Medium Medium Medium Medium Prioritize reently updated High High ontent in searh results searh results Rewrite lass and program Online lasses and programs not High High ontent to highlight lass delivery learly identified options Arhive outdated ontent Pages link to outdated ontent High High Rewrite ontent to be userentered Some ontent under Prospetive High High Students links an only be Allow users to start appliation without registering for an aount Add appliation deadlines to admission pages Chunk ontent into more sannable units Chunk ontent into more sannable units Produe PDFs with hotlinked bookmarks Group ontent by audiene Add video and sreen aptures to show proesses and interations aessed by registered students Requiring sign in to begin appliation is off-putting High High Upoming deadlines not featured Medium Low in body text IDC program pages text-heavy Medium Medium MBA program pages text-heavy Medium Medium PDF ourse atalogs hard to navigate Medium Medium Undergraduate and graduate Medium Medium ourse atalogs diffiult to distinguish Lak of multimodal ontent Medium High

47 Reommendation Issue Addressed Severity Level of Effort Use the same searh results Searh behavior inonsistent Medium High format for all searh engines between web site setions Rewrite program ontent Overall program length not learly alled out Low Medium Table 8 Reommendations by issue severity and level of effort The 900ers SPSU.edu Usability Test Report 39

48 Further researh We reommend that these senarios be tested with a larger audiene to assess the web site s usability in more depth. One the site has been updated, we reommend re-testing these same senarios with a similar audiene to see if the results show improvements. In addition, we suggest testing the web site: on mobile devies with assistive tehnologies with potential undergraduate students with non-native English speakers The 900ers SPSU.edu Usability Test Report 40

49 Conlusion The 900ers usability evaluation of the SPSU site analyzed the user experiene of potential online graduate students as they: Evaluated SPSU s redentials and graduate programs Researhed degree requirements for a partiular program Learned about the online lass experiene Prepared to apply for a program Our evaluation revealed an effetive site searh engine and easily-understood header and sidebar navigation. However, we also found areas for improvement, inluding: Hard to find information university areditation and appliation deadlines are diffiult to loate Unlear, inomplete, and outdated information ontent tuition rates, online lass offerings and experiene, program lengths, and the differenes between Master s and Certifiate programs are unlear Content targeted to other audienes pages for prospetive graduate students ontain links to information relevant only to registered students or faulty While we have made reommendations to address eah usability issue identified in our evaluation, we suggest SPSU prioritize the low hanging fruit hanges with a low level of effort that will address high severity issues. These inlude: Add areditation information to Home page and About Us landing page Rewrite program ontent to learly identify whih programs an be ompleted ompletely online Rewrite Tuition & Fees page to larify distintion between graduate tuition and online tuition Rewrite IT program ontent to larify differene between Master s and Certifiate programs Rewrite MBA program ontent to larify differene between Master s, Bridge Certifiate, and Transition Certifiate programs Add appliation deadlines to admission pages After these hanges have been made, we reommend that a seond usability test be onduted, using the same senarios that were followed for this test. The results of the seond study will provide a basis of omparison with our results, to give SPSU an indiation of whih revisions have been the most effetive and where improvements are still needed. The 900ers SPSU.edu Usability Test Report 41

50 Referenes Travis, D. (2010, April 1). Log usability tests like a pro. In UserFous. Retrieved from The 900ers SPSU.edu Usability Test Report 42

51 Appendix A: Personas Persona 1: Potential undergraduate student Samantha Farber I an go to shool and get my degree without having to sit with a bunh of kids who I don t have anything in ommon with. Samantha is a 45 year-old potential undergraduate student. She is married and is the mother of two hildren, both of whom live at home. She finished high shool when she was 18 and has been thinking about going bak to shool for years. She has been working in the aounting department at a regional book publisher loated in Marietta, Georgia, for seven years. Samantha is onsidering getting her Bahelor s degree in Business so she an move up the ladder at work. Most of her olleagues have some type of ollege degree or ertifiate and Samantha feels she may get passed over for promotions if she doesn t have a redential. Samantha has always enjoyed learning, so she is exited for the hallenge of earning a degree. Furthermore, she wants to prove to her sons that you an ahieve anything as long as you work hard. She uses the Internet daily, athing up with old friends via soial media and using various aounting programs in her job. Beyond that, however, she doesn t use many different omputer programs. She often struggles to keep up with new programs and omputer tehnology. Online shool fits into Samantha s life beause she works full-time and still has hildren she tends to daily. Also, she d like to avoid sitting in a lassroom with a bunh of kids with whom she doesn t have anything in ommon. She understands that it will be new for her, but she plans on bloking out times in the evenings and on the weekends for shoolwork. Also, she plans on ommuniating regularly with her professor and lassmates to make sure she understands the material and assignments. Demographis Computer Profiieny Motivation Behavior/Attitudes 45 Daily Internet usage Career advanement Will do shool from Marietta, GA Uses soial media, Personal fulfillment home Spouse, 2 hildren aounting software Example for hildren Feels unsure about Works in aounting Uses a PC lassroom setting due dept. Avoids new tehnology to age The 900ers SPSU.edu Usability Test Report 43

52 Persona 2: Potential undergraduate student Carrie Butler I want to give my daughter a more stable life and be a role model for her. N06/ Carrie is a 28 year-old potential undergraduate student. She is a single mother of a 7 year-old hild and lives with her mother. When she graduated high shool, Carrie went to ollege for one semester. She urrently works part-time as an Assistant Manager at a shoe store at her loal mall in Tampa, Florida. She works different shifts eah week, so her shedule has been a big hindrane to going bak to shool. She wants to finish shool to provide a more stable life for her hild and to prove that she an aomplish something. She also wants to set a good example for her daughter. Carrie is onsidering getting her degree in Business Administration beause she would like to move into a areer with better pay and onsistent work hours. She feels that a Business Administration degree would lead to a real areer rather than just a payhek-to-payhek job. She d like to start this fall. Carrie uses the Internet every day, mostly for soial media and online shopping. She has even played some popular role-playing games in the past. She has some experiene with MS Word from writing reports in high shool, but doesn t use it muh anymore. Beause of her inonsistent work shedule and young hild, Carrie needs a degree program that is flexible. That is why she has been researhing online shools. Her biggest onern is if she an keep up with the material, espeially without the benefit of fae-to-fae interation with a professor. She plans on ombatting that issue by regularly ontating her professors and asking her friends, some of whom are in shool, for help. She will do her shoolwork from her home omputer during her time off from work and in between taking are of her daughter. Demographis Computer Profiieny Motivation Behavior/Attitudes 28 Daily Internet usage Better areer Will do shool from Tampa, FL Uses soial media, Stable work home Lives with mother, online shopping, MS environment Will need aess to has 1 hild Word, Internet games Provide finanially for professor and peers Assistant Manager at Uses a PC, iphone hild shoe store Never taken lasses Example for hild Inonsistent work shedule online The 900ers SPSU.edu Usability Test Report 44

53 Persona 3: Potential graduate student Travis Treviño Learning is a lifetime pursuit. Travis is a 43 year-old potential graduate student. He lives with his spouse and his 17 year-old hild from a previous marriage. He joined the Marine Corps after graduating high shool and ompleted a 4-year enlistment. Travis then went on to ollege and graduated with a Bahelor s in Journalism after five years. He urrently works as the sports and loal news editor for a newspaper in Midlothian, Virginia. With his son getting ready for ollege, Travis has been inspired to go bak to shool. He wants to show his son that learning is a lifetime pursuit and that he an do whatever he puts his mind to. Also, the inreasingly desperate state of the newspaper industry leaves Travis unertain about his future. He loves journalism, but knows that he and his newspaper will have to build an online presene. Therefore, he has deided to pursue a graduate degree in Tehnial Communiation, whih he feels will help him gain the experiene needed with new media to push his newspaper forward. He plans to start in the fall so he and his son an start bak to shool together. Travis is profiient in multiple desktop publishing programs due to his job duties. He uses the Internet multiple times eah day for soial media, online banking, and . He also has experiene with MS Offie. He uses a PC, Ma, iphone, and ipad for his work and personal interests. He has never taken a lass online before. Travis likes the flexibility of online shools and feels onfident that he an navigate the interfae. His main onern is the lak of immediate feedbak and disussion that he feels an online program ould have. He wants to be sure that the program he hooses is professional and interative. Demographis Computer Profiieny Motivation Behavior/Attitudes 43 Daily Internet usage Personal fulfillment Will do shool from Midlothian, VA Uses soial media, MS More experiene home Spouse, 1 hild Word, desktop New professional Confident he an Sports/Loal News publishing software, opportunities adapt to shool online editor at loal online banking, Example for hild Wants interation in newspaper Uses a PC, Ma, lasses iphone, and ipad Wants a redible Never taken lasses online program The 900ers SPSU.edu Usability Test Report 45

54 Appendix B: Heuristi evaluation Exeutive summary This report ontains our heuristi evaluation of the Southern Polytehni State University web site ( Three members of the 900ers team independently evaluated the SPSU web site from the perspetive of a representative persona: Travis, a prospetive online ommuniations graduate student. Team members walked through the same senario and tasks, and evaluated the user experiene using Whitney Quesenbery s heuristi approah, the 5Es. Together, we ompared our findings and ranked issue severity. These findings will diret our upoming usability test plan. For this evaluation, we hose the following tasks: Evaluating SPSU s redentials and program offerings Researhing degree requirements for a partiular program Learning about the online lass experiene Preparing to apply to a program Evaluators found that the SPSU web site s strength lies in its ontent. Positive findings inlude: Clear, reader-entered ontent partiularly in the English, Tehnial Communiation, and Media Arts setion Effetive web site searh the searh engine was so reliable that one evaluator adapted her behavior to favor web site searh over in-page navigation The heuristi evaluation unovered several issues with the web site. We identified these top issues: Unlear navigation paths information is not always in logial plaes Inonsistent look and feel aross pages headers, sidebars, and in-page navigation treatments differ between the SPSU main web site and the ETCMA web site; the ourse information web site laks SPSU branding elements altogether Inonsistent link behavior while most links open pages in the urrent browser window, some links trigger new tabs without warning Credential information diffiult to find both university areditation and professor qualifiations are diffiult to loate This report details our findings and highlights web site strengths and weaknesses. These findings will inform upoming usability testing ativities, as well as indiate areas for web site improvement that may not be identified by test partiipants. The 900ers SPSU.edu Usability Test Report 46

55 Introdution Heuristi evaluations are a user experiene researh tehnique to assess the overall usability of a web site. A small group of trained evaluators assumes the persona of a representative user and eah team member walks through the web site, evaluating it against a set of heuristis usability priniples. Heuristi evaluations help identify web site strengths and weaknesses. There are many different sets of heuristis in the user experiene industry. Our team hose to analyze the SPSU web site using Whitney Quesenbery s 5Es, a user-entered, task-foused set of heuristis (Quesenbery, n.d.). Quesenbery s 5Es evaluates tasks and web sites by asking five sets of questions. Is the web site: Effiient How long does it take to omplete the task? This an be measured by time-on-task or number of liks. Effetive Are users able to omplete the task, with a minimum of errors? Engaging How does the user feel about the experiene? Is the user satisfied? Error tolerant How did the system handle user errors? Easy to learn Are users able to understand how the web site worked? Do they build understanding of web site funtions and features as they move from task to task? These 5Es form the framework for our heuristi evaluation. This report overs the 900ers: Heuristi evaluation methodology Task-by-task findings Severity ranking for identified issues Our goal for this heuristi evaluation is to identify any ritial problems with the web site. We will perform user testing to onfirm or refute our findings, and then make reommendations to enhane SPSU.edu s overall user experiene. Methodology Three members of the 900ers projet team independently evaluated SPSU.edu. Then, we disussed our individual findings to form a unified understanding of the web site s strengths and weaknesses. Persona Team members assumed the role of a representative user: Travis, a non-traditional prospetive graduate student interested in online eduation. (See Appendix A.) Travis is 43 years old. Married, with hild at home Comfortable using the internet on several devies Employed full time Potential areer hanger Holds Bahelor s degree The 900ers SPSU.edu Usability Test Report 47

56 Senario Our team used the following senario for our heuristi evaluation: Travis has deided that he wants to pursue a ommuniations-related master s degree online. He wants to get his degree through a respeted shool so his degree will be more valuable. A oworker told him about Southern Polytehni State University. He wants to gather information about the shool and program to understand if SPSU will help him meet his goals. After he has gathered this information, he should understand what he needs to do to apply. We defined a likely task flow for Travis through the web site: Evaluating SPSU s redentials and graduate programs Researhing degree requirements for a partiular program Learning about the online lass experiene Preparing to apply to a program Heuristis Our evaluation was guided by Quesenbery s 5Es. We used this set of heuristis to evaluate the web site against Travis s expetations as he performed relevant tasks. Effiient Travis is a web-savvy, eduated person, so expets he an find things very quikly. Effetive He expets that if he needs information, he will be able to find it. If he an t find it, he assumes it does not exist. Engaging Travis takes pride in his intelligene and ability to figure things out. He thinks of himself as a reader, but prefers to san things on the web. Error tolerant He has little patiene for tehnial errors on web sites, and will often abandon a web site if links are broken or he enounters error messages. Easy to learn He has been using the web for a long time, and feels as though he has a good understanding of how web sites should work. He s not opposed to putting in some effort to learn the quirks of a new web site, but he expets that a web site will be internally onsistent. Rankings Our team defined twelve tasks to bring Travis through his senario. For eah of those tasks, we ranked the 5Es. We hose a simple three point sale to represent task performane: Aeptable Travis was either pleased with or neutral about the task. Problemati Travis was annoyed by it. Unaeptable Travis was unhappy. We use a olor bar to represent the 5Es and their rankings after eah task. We hose stoplight olors to make it easy to see where problem spots are. The following example shows a representative task ranking olor bar: Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 48

57 In this example, the rankings are: Effiient Problemati Effetive Unaeptable Engaging Unaeptable Error tolerant Problemati Easy to learn Aeptable Findings Our evaluation found positive web site attributes and identified potential opportunities for hange. We present our findings below, in two parts: Tasks Narrative of Travis s experiene of the web site as he moves through the senario, task by task Rankings Table of findings, shown in order of issue severity The 900ers SPSU.edu Usability Test Report 49

58 Tasks Evaluating SPSU s redentials and graduate programs Travis omes to SPSU.edu for the first time. He wants to know if SPSU is a legitimate university, not some diploma mill. Travis sans the page and quikly finds the About Us link. Travis assumes that a university will highlight its areditation prominently. The 900ers SPSU.edu Usability Test Report 50

59 After sanning the About Us page and not finding the information he wanted, Travis thinks that SPSU must highlight its areditation in its student reruitment setion. Travis easily finds a link for prospetive students like himself. The 900ers SPSU.edu Usability Test Report 51

60 Travis sans through the links for prospetive students, frustrated that he an t find areditation information. Travis assumes that he must have missed the areditation information on the About Us page, so he goes bak there. The 900ers SPSU.edu Usability Test Report 52

61 Travis reads through the About Us page. He thinks the University System of Georgia referene hints at redibility, but he is not sure. Frustrated, Travis resorts to using the Searh funtion, typing in areditation. The 900ers SPSU.edu Usability Test Report 53

62 Travis finds areditation information through his searh. Travis sees that some speifi programs are aredited. Travis is amazed that the university-level areditation is buried in the undergraduate atalog. Travis knows now that SPSU is aredited, but the experiene has left him feeling autious. He wonders if SPSU is really reputable, and if he should onsider other shools. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 54

63 Travis deides to follow through with his researh, even though he is skeptial, beause he trusts his oworker s opinion. He still wants to know if SPSU offers a ommuniations master s degree. Travis remembers that the Prospetive Students page had a lot of links about aademis. He liks on the Majors, Minors, and Certifiates link. The 900ers SPSU.edu Usability Test Report 55

64 Travis sans through the list of programs, looking for one that starts with Communiations. He doesn t see a ommuniations program, so he goes bak to the Prospetive Students page and reads through the Aademis links again. Frustrated, he is sure that the Majors, Minors, and Certifiates page must have the right information. The 900ers SPSU.edu Usability Test Report 56

65 Travis hits CTRL-F and searhes the page for ommuniation. He isn t interested in a ertifiate, but the Information Design & Communiation major looks promising. Travis is moderately frustrated by the amount of effort it took to find the Information Design & Communiation (IDC) graduate major. He thinks to himself that if the web site is frustrating, being a student must be frustrating, too. Travis finds the program title intriguing, though, so he pushes on to learn more about the program. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 57

66 Researhing degree requirements for a partiular program Travis srolls through the page, sanning for links. He is happy to find the ourse options link without muh effort. Travis wants to understand what sort of lasses he ll have to take as an IDC student. He liks on the Information Design & Communiation (graduate major) link. Travis wryly notes that this page has areditation information. The 900ers SPSU.edu Usability Test Report 58

67 Travis quikly finds the Required Courses link. Travis is pleased at how easy it was to find the required ourses. He is surprised, though, that the list of Required Courses was inomplete, as the Foundation Courses are also required. Travis noties he an find out about eletives here, too. Travis wanted to know about his eletive options, and was happy he didn t even have to leave this page. He feels good about himself, and he is starting to trust the ETCMA setion of the web site, though he hasn t bothered to figure out what ETCMA stands for. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 59

68 Beause Travis is onerned about the value of his degree, he wants to ensure that his ourses will be taught by Ph.D.-level instrutors. He liks on the Required Courses link, sure that he an find out who will be teahing the Required Courses that way. Travis sees that he an find out who is teahing a lass from the Shedule of Classes. Travis liks on the Shedule of Classes straight away, happy that he s going down the right path. The 900ers SPSU.edu Usability Test Report 60

69 Travis wants to start shool in Fall Travis is sad to see that Fall 2013 isn t an available option. Travis is onerned that there aren t any IDC ourses for Summer He thinks that maybe the IDC program might not offer enough lasses for him to omplete his degree in a timely manner. Travis goes bak to the term seletor one last time, and finds the Spring 2013 IDC lasses. The 900ers SPSU.edu Usability Test Report 61

70 Travis is surprised that the instrutors are only identified by last name and initial, and don t inlude their titles. Travis realizes he doesn t remember whih ourses are required. Travis feels like he should have known to jot down the required ourses. He resigns himself to going bak, writing down the lass numbers, and oming bak to this sreen. The 900ers SPSU.edu Usability Test Report 62

71 Travis is annoyed to find he is in a new tab. Travis hits the bak button several times, but an t get bak to the IDC information page. He looks up at his browser bar and realizes that he s in a new tab. He wishes that the term seletion feature had popped up in his original sreen, and let him toggle between the required list and the shedule. After retraing his steps, he finds that IDC 6001 is being taught by Palmer L. He goes bak to the Required Courses page. The 900ers SPSU.edu Usability Test Report 63

72 The 900ers SPSU.edu Usability Test Report 64 Travis sees that he must go to the Faulty and Staff Diretory to learn more about the instrutors.

73 Travis noties some of the faulty are only professors. Travis finds Dr. Laura Palmer and assumes that this is the same person as Palmer L. Travis is assured that most of the faulty hold Ph.Ds, and that at least one of them teahes a required ourse. However, he is frustrated that the program didn t talk about its professors redentials on one of the program pages. He thinks that a strong program should brag about its professors aomplishments. He also doesn t like that he had to go bak and forth between the list of ourses and the Faulty and Staff Diretory to look up the redentials of all of his prospetive instrutors, one at a time. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 65

74 Learning about the online lass experiene Travis lives in Virginia, and needs to be sure that he an omplete his degree entirely online. Travis remembers that the ETCMA Prospetive Students page had a lot of information. The 900ers SPSU.edu Usability Test Report 66

75 Travis quikly finds out that the program is online. Travis is happy at how quikly he verified this program was online. He is relieved that this was so muh easier than finding out about the instrutors. He thought there d be more information about this, though. He wonders if this is all he needs to know. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 67

76 Travis knows that many graduate programs require a thesis. He would rather not write a thesis, so he wants to know what his options are. He liks on the Review Your Program Options link on the Prospetive Graduate Students page. Travis quikly sans down the page until he finds the IDC program link. Travis is navigating through the web site with speed and onfidene. He s getting omfortable with the web site, and feels like he an easily flow from page to page, but wonders why the thesis information isn t listed up front with the degree desription. The 900ers SPSU.edu Usability Test Report 68

77 Travis liks on the link for the MS Information Design and Communiation program. Not finding any information about thesis requirements here, he liks on the graduate program requirements link found under Next steps at the bottom of the page. He then liks on the link for Requirements: MS Information Design and Communiation. Srolling down, he finds the thesis information he was looking for on this page. Travis is thrilled to know he has non-thesis options. Travis is happy to know he has multiple options. He thinks other prospetive students would want to know about this right away, however, and wishes the information wasn t buried so deep. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 69

78 Now that he knows he an take all of his lasses online and won t have to write a thesis, Travis wonders how long it will take to omplete his degree. Travis expeted he d find the answer on the same page, and was happy to be right. He was also happy to learn more about what it would be like to take lasses online. Travis is urious about how he is eligible for in-state tuition. Travis feels like he was very luky to find the MS of IDC page it s answering questions he didn t even know he had! He feels onfident, knowing a bit more about the online learning experiene. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 70

79 Travis had thought he d have to hunt for tuition information, but found it on the same page he was already on. He s very happy to see the prie of a lass instead of the prie of a redit. He wonders, though, what else goes into the overall prie, so liks on the In-State Tuition link on the MS IDC page. Travis is overwhelmed by all the variables. Travis wonders why he s in a new tab again. Travis deides to trust that the perlass figure on the MS IDC page was aurate. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 71

80 Preparing to apply for a program Travis has deided that the IDC program might be the right program for him. He wants to know if it s too late to apply for the Fall 2013 semester. He goes bak to the MS in IDC page. Travis assumes that he ll find the deadline here. The 900ers SPSU.edu Usability Test Report 72

81 Travis is happy to find that he still has time to apply for Fall Travis is happy to know he isn t too late to apply. He wants to know what he needs to do next. Travis wants to apply. He wonders why he has to leave the apply page to get information on how to apply. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 73

82 Travis is onfident he s found the information he needs. Travis is surprised to find a long list of things he ll need to do to apply. He wishes there was an online appliation, but is satisfied that he now has all the information he needs to apply for the IDC program. Effiient Effetive Engaging Error Tolerant Easy to Learn The 900ers SPSU.edu Usability Test Report 74