June Usability Test Results of the Perform Application

Transcription

1 June Usability Test Results of the Perform Application July 2012 Written by Theresa Wilkinson

2 Table of Contents Document overview... 5 Test objectives... 5 Executive summary... 5 Business objectives... 5 High-level findings... 6 Overall task performance... 6 User difficulties and frustrations with the application... 6 Recommendations... 7 Methodology... 8 What happened during the usability test... 8 Who we tested... 8 What participants did... 9 What data we collected Usability metrics General performance metrics Specifc performance metrics Where we tested Findings and recommendations Task 1 New Hire wizard Time to complete task Success rate for task Time dealing with issues for Task Specific information on issues Recommendations for Task Task 2 Company deductions Time to complete task Success rate for task Issues that impacted Task 2 (Company Deductions) Time dealing with issues for Task Specific information on issues Recommendations for Task Task 3 Employee deductions Time to complete task Success rate for task Issues that impacted Task 3 (Employee Deductions) Time dealing with issues for Task Flow preference Specific information on issues Recommendations for Task Task 4 (Employee s salary Pay Rates) Time to complete task Success rate for task Issues that impacted Task 4 (Employee s salary Pay Rates) Specific information on issues Recommendations for Task Page 2

3 Task 5 Part 2 (Add column) Time to complete task Success rate for task Issues that impacted Task 5 Part 2 (Add column) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Task 5 Part 3 (Add column) Time to complete task Success rate for task Issues that impacted Task 5 Part 3 (Add column) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Task 5 Part 4 (Add amounts in columns) Time to complete task Success rate for task Issues that impacted Task 5 Part 4 (Add amounts in columns) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Task 5 Part 5 (Add amounts in columns) Time to complete task Success rate for task Issues that impacted Task 5 Part 5 (Add amounts in columns) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Task 5 Part 6 (Add amounts in columns) Time to complete task Success rate for task Issues that impacted Task 5 Part 6 (Add amounts in columns) Specific information on issues Recommendations for Task 5 Part Task 5 Part 7 (Net Amt of check) Time to complete task Success rate for task Issues that impacted Task 5 Part 7 (Net Amt of check) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Task 5 Part 8 (Net cash for payrun) Time to complete task Success rate for task Specific information on issues Page 3

4 Recommendations for Task 5 Part Task 5 Part 9 (Add a manual check) Time to complete task Success rate for task Specific information on issues Recommendations for Task 5 Part Task 5 Part 10 (Review your payrun and open Pre-post Payroll Journal) Time to complete task Success rate for task Issues that impacted Task 5 Part 10 (Review your payrun and open Pre-post Payroll Journal) Time dealing with issues for Task 5 Part Specific information on issues Recommendations for Task 5 Part Appendix - Glossary of terms Usability goals Problem severity Problem severity classification Page 4

5 Document overview This document details the results of the June usability test for the Perform application. The goals of usability testing include establishing a baseline of user performance, establishing and validating user performance measures, and identifying potential design concerns to be addressed in order to improve the efficiency, productivity, and end-user satisfaction. Test objectives To understand if the users who had viewed the Webex recording could complete tasks faster and more efficiently than those users who did not. To identify the most confusing, difficult or time-intensive areas of the application and gather recommendations for improvements. To identify if test users accessed the system help information. Establish baseline user performance and user-satisfaction levels of the user interface for future usability evaluations. Executive summary This report documents the findings of the June usability test of the Perform application and Perform Upgrade Summary Webex recording. Usability Testing is defined as one-on-one sessions where a "real-life" user performs tasks in order to identify user frustrations and problems within the application. Business objectives To understand if the users who had viewed the Webex recording could complete tasks faster and more efficiently than those users who did not. To identify the most confusing, difficult or time-intensive areas of the application and gather recommendations for improvements. To identify if test users accessed the system help information. NOTE: Include high-level summary of findings here, including: Overall task performance User difficulties and frustrations with the site Significant usability findings (may include positive as well as negative findings) Page 5

6 High-level findings Overall task performance Total points possible = 13, Had training, Failed User Average Task 1 6:55 5:31 7:37 F F 7:21 7:10 7:28 7:20 Task 2 2:37 F 2:46 F F F 5:33 Task 3 F F 2:29 4:25 3:18 4:16 1:39 Task 4 F :24 1:56 2:20 F :27 1:29 Task 5-2 1:25 F 2:22 F F 2:19 :30 Task 5-3 :14 1:32 F N/A F 1:25 :59 Task 5-4 F 1:02 1:10 2:34 F 2:35 F Task 5-5 1:25 :19 :40 :51 :10 :09 1:32 Task 5-6 :24 1:21 :30 1:12 1:05 :03 F Task 5-7 F 1:30 F 2:24 F F 1:11 3:01 2:33 :45 F F :18 :10 :10 1:20 3:29 3:33 1:00 1:49 1:23 1:40 :50 :59 2:18 Task 5-8 1:15 N/A 1:19 :20 1:02 1:03 :21 1:22 :57 Task 5-9 5:15 5:58 F F F F F F 5:37 Task :08 N/A :34 3:04 :09 1:01 :14 :22 1:24 Total points scored User difficulties and frustrations with the application All users complained about scrolling and multiple scroll bars on all screens. All users wanted to see the entire screen on one page. New Hire wizard most users complained about that scrolling did not keep up with tabbing through fields. Page 6

7 Most users were confused with New Hire screens that did not progress and no error messages displayed. Most users were confused about which fields were required when none were marked. Most users were confused about screen layouts. Some users are still not noticing headings. All users complained about columns on the Paydata Grid. Most users could not locate the Net Amount on the EE drawer. No users accessed online help during the test. Most users were confused on how to add and calculate a manual check. Recommendations Recommendations appear after tasks in the Findings section. Theresa Wilkinson July, 2012 Page 7

8 Methodology What happened during the usability test The usability evaluation of the Perform product was conducted by Theresa Wilkinson in the Cincinnati office on June 6, 7, and 12, and on June 8 in the Columbus office. During the usability evaluation, eleven participants were asked to spend up to two hours within the application. During this time, participants: Were asked user background questions Performed real-world tasks within the application while thinking aloud Answered questions about their overall satisfaction Took an end-of-test survey Who we tested Eleven participants, having the following profile characteristics, evaluated the Perform application. Age TOTAL (participants) 8 Years of experience with Paycor >10 TOTAL (participants) 8 Gender Women 5 Men 3 TOTAL (participants) 8 Participants were compensated a total of $150 for their time: $50 gift card and $100 credit. Page 8

9 What participants did Prior to the usability test, half of the participants were asked to view the Perform Upgrade Summary Webex recording. Total of three users viewed the recording -- one of those users watched less than half of the recording. During the usability evaluation, all participants were asked to complete five scenarios or real-life tasks in the application. The following tasks were supplied by the marketing and training teams for the test: # Scenario 1 Hire an employee for the Market Place: Name John Smith Department 201 Union Center Store SSN Paygroup Biweekly DOB 09/15/1970 Payrate $2,000 per pay DOH 03/25/2012 Filing Status Single, 1 exemption Address 556 Elm St, Covington KY Please setup a deduction for Market Place on the client level: Refer to list. 3 Please setup a deduction on the employee level: Deduction added in previous task, $50 per pay 4 Please look up the following information on the employee level: What is Greg George s annual salary? 5 Pay employees for the Market Place: 1. Open Cincinnati Payroll, BiWeekly paygroup, 4/6/12 check date 2. Add a column for H125 deduction amount 3. Add a column for Vacation earning hours 4. Dan Davis, 40 hrs, $50 H125 make up 5. Ernie Evers, 40 hrs, 5 OT 6. Tony Simpson, 32 hrs, 8 Vac 7. What is the net amount of Ernie Evers check? 8. What is the Net Cash for this payrun? 9. Add a manual check for Dan Davis, 3 Regular hrs 10. Review your payrun and open Pre-post Payroll Journal Page 9

10 What data we collected Usability metrics Usability metrics refers to user performance measured against specific performance goals necessary to satisfy usability requirements. Task completion success rates, adherence to dialog scripts, error rates, and subjective evaluations will be used. Time-to-completion of tasks will also be collected. General performance metrics A performance metric is a measure of the application s activities and performance. Performance metrics should support a range of stakeholder needs from customers to employees. General performance metrics: Easy to learn: The user can quickly go from not knowing the system to getting some work done with it. Efficient to use: Once the user has learned the system, a high level of productivity is possible. Easy to remember: The infrequent user is able to return to using the system after some period of not having used it, without having to learn everything all over. Few errors: Users do not make many errors during the use of the system, or if they do make errors they can easily recover from them. Also, no catastrophic errors should occur. Pleasant to use: Users are subjectively satisfied by using the system; they like it. Specifc performance metrics Specific performance metrics should be constructed to encourage performance improvement, effectiveness, efficiency, and appropriate levels of internal controls. They should incorporate "best practices" related to the performance being measured and cost/risk/benefit analysis, where appropriate. Specific performance metrics: Time to complete a scenario or set of tasks Time spent using help or documentation Task (completion) success rate Number of issues per task (critcal and non-critical) Number of system issues Page 10

11 Number of users making a particular error Number of users completing a task successfully (reverse metric) Number of the user expressing frustration or dissatisfaction Number of user who gave up a task This information is detailed under each task heading. Where we tested Following is a summary of the participants computing environment: URL of tested application: Computer platforms: HP laptop Browser tested: Internet Explorer 8.0 Screen resolution: 1024 X 768 Operating system: Connection speed: Voice and screen capture software Windows XP Shared T1 Camtasia Studio 7 loaded on the laptop User 11 was video-taped during part of the usability test. Page 11

12 Findings and recommendations Task 1 New Hire wizard Hire an employee for the Market Place: Name John Smith Department 201 Union Center Store SSN Paygroup Biweekly DOB 09/15/1970 Payrate $2,000 per pay DOH 03/25/2012 Filing Status Single, 1 exemption Address 556 Elm St, Covington KY Time to complete task User Average Task 1 6:55 5:31 7:37 F F 7:21 7:10 7:28 7:20 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored in test Issues that impacted Task 1 (New Hire Wizard) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes User scrolling up because screen goes to bottom of page User tabbing through fields but system is not scrolling the page user having to both physically tab and 3 Defect 3 Defect Page 12

13 scroll When users selected State and hit tab button, they went to Address 1 field and had to tab or scroll to Zip field 2 Defect Confusion on pre-filled required fields 2 User Users not filling in required fields and system not displaying error message 3 UX/Defect Confusion about effective date meaning 3 User Users 5, 7 failed task due to this issue Users are entering all fields but screen not progressing -- having to re-enter a field to progress - no error message. User having to guess which one to fill out again 3 User /Defect Time dealing with issues for Task 1 User Average time dealing with issues Task 1 Estimate: 6:00 6:00 6:00 6:00 6:00 6:00 6:00 6:00 Task time: 6:55 5:31 7:37 7:21 13:14 7:21 7:10 7:28 Issue time: 1:21 1:39 2:45 3:04 4:01 2:14 2:07 3:31 2:45 Specific information on issues 1 Screen displaying bottom of page making the user to have to scroll up. Tabbing through fields does not seem to scroll the user down the page so user must phyically scroll down the screen after tabbing so they can see what to enter in fields. All Users User 1 tried to use calendar control but entered wrong date because page did not scroll down when user tabbed. Had to enter Start Date twice. User 2: That is going to be a pain in the butt to scroll down all the time. That is one of my pet peeves right now. User 4: That is kind of weird -- the scroll there. Page 13

14 (You would rather have the page go to the top?) Yeah. I don t think you should have to scroll up. I don t like that. I think it would be better if you did not have to [scroll]. The screen comes up blank. And you are looking at it and thinking well wait a minute am I on the right page? You realize that you are when you scroll up. User 5: (Did you know that you had to scroll up?) Yes based on having to do it before and it is annoying. That IS very annoying. User 7: (User sits on next page. There is nothing else to do? (User clicks or mouses over Hire button. Do you think there is anything else to do?) I don t see anything else to do. (So you think that is all that is on the page?) Yes there s nothing. (I scroll up.) Oh, it s up there. User 8:.(TIME SITTING ON WHITE SCREEN 7:00 7:07. Was that obvious that you had to scroll up when it went to the next screen because it was white?) No. I would have thought it would come to where I want to be. User 9: (You used the scroll on the mouse why? Did you see the scroll bars?) I see three of them and I don t know which one to pick. There is a lot of them there. User 11: I will say that your other format has eveything on one screen and I did not have to scroll down. (Did you prefer that?) Yes. I like it to be all on one screen. And you do have room. I definitely don t like the scrolling. (What do you think is going on?) It is a blank screen. (She scrolls up.) Oh there we go. 2 On the Position screen, having required fields pre-filled confused users. Users: 1, 2, 7, 11 User 1: (On Position screen, Employee Type field defaults to Regular. This info is not included in the task.) I assume that is just regular? (You don t have all the information.) Okay. Employee Page 14

15 number I don t see that either. (Employee number displays default number.) Employee number I don t see that either. Will it creates its own employee number then? If it is new... okay... (I did not answer him because I did not want to help him.) User 2: Employee number? Why do we put in an employee number, that should just be automatic that should just be the next number. You should not have to put it in, not that we did but it was starred. (It is already filled in on the screen.) User 11: They automatically give that to you correct? Employee number? (Is that what it looks like?) Yeah. It looks like they assigned it. (Is it confusing that it is required and already filled-in?) Yeah. If it is automatically assigned, it should not have an asterick. At least that is what I think. 3 On the Compensation screen, User 1: (User clicks into the Effective Date field.) Page 15

16 users did not understand the label Effective Date. Will that just be the Date of Hire? (What do you think?) Yes. It would be? (I can t always answer your questions. Do you think that is what it would be?) Well I am assuming that since it is a new hire yes. User 4: I need to enter the Effective Date right? (Yes.) So I go back to my Date of Hire? (Is that what you think it is? Would it be more clear if Date of Hire?) It is not clear. But on the current system that is usually what I enter is date of hire. (But you don t think it is clear?) It is not as clear as it could be. (Info icon to explain this date?) User 7: I don t see an Effective Date. (Is that not clear?) Effective Date -- that be his date of hire, I guess. (Is that what you think it is?) That is the only thing that I can think that it would be. Or it could be today s date. (Is it clear?) No. (I tell her that it is his hire date.) It is not clear because after I thought about it, I thought it was today s date. User 8: Effective Date. I am going to say it is probably his hire date. (Are you assuming that?) I am assuming because it says Effective Date. It should say, in my opinion, Payroll Period Date or Start Date. (User did not enter the Effective Date. No error message displayed. User clicked the Next button but nothing happened.) User 9: Effective Date I would say that is the same. (All users are pausing on this label. Is that clear? Do you know what that means?) Um yeah. It means what date you want to start the $2000/pay. (That may not be the hire date correct?) Correct. I have a lot of that that I come across. I usually pick a random number as long as it falls within that pay period. (Do you think Start Date would be more clear?) Yes but I just put in a random date when I come across this. It is really not that important. (Users don t seem to want to enter this date! You do have to enter something here it is required.) I think it would be more of a hassle to have to do that having to do an extra step. User 11: Effective date does not really say so I put in today? (What do you think that means?) Page 16

17 Effective date is... oh. A date that the pay goes into effect. This does not mean effective date of hire right? (What does that mean to you?) Effective pay rate date. 5 Error messages are still not displaying when users do not enter a required field all screens OR Next button is not working when all fields are filled in. Users: 2, 4, 5, 7, 8, 11 User 2: That should pop-up in a different color when you go to Next, it should tell you what you missed. (Did you click Next?) It just sat there. See I am clicking Next now and it is not going anywhere. So I am missing something. (User scrolls up and down screen.) And I bet you it is this employee number. It wants me to re-enter it. (User re-enters employee number.) Yup that was it. It already pulled it up but it wanted me to re-enter it. That is kind of weird. User 4: (User clicked Next button but page not progressing. All fields are filled in.) It does not seem to be working right. (I re-enter Exemptions and it works.) User 8: (User did not enter the Effective Date. No error message displayed. User clicked the Next button but nothing happened.) Recommendations for Task 1 Have all screens in New Wizard flow start at top of screen so users do not have to scroll up. Page 17

18 Enable the page to scroll as the user tabs down into fields. Enable tabbing through fields in the order they display on the screen. Remove required mark (asterisk) if fields display default values. Enable error messaging on all screens in wizard. Add information icon to define Effective Date. Put all information on one screen to eliminate scrolling. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Task 2 Company deductions Please setup this deduction for Market Place on the client level: Facilitator will provide Time to complete task User Task 2 2:37 F 2:46 F F F 5: :01 Average 3:29 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 2 (Company Deductions) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Users confused about required fields - Required fields not marked on screen 3 User Users confused about what flags mean 3 User Users confused about Frequency 3 User User 5 failed task due to this issue Page 18

19 Users confused about Factor 2 User Users having to click Add Deduction button and plus button multiple times 2 Defect Users not scrolling -- not seeing see Save button 3 User User 7 failed task due to this issue Users confused about screen layout 3 User Users 2, 8 failed task due to this issue Not sure which fields are required 2 User Time dealing with issues for Task 2 User Average time dealing with issues Task 2 Estimate: 4:00 4:00 4:00 4:00 4:00 4:00 4:00 4:00 Task time: 2:37 4:14 2:46 4:38 3:57 6:23 5:33 3:01 Issue time: :50 2:18 1:31 1:42 1:21 3:05 1:53 :22 1:43 Specific information on issues 1 Users confused about values in Frequency field User 1: (And once a month. Whichever one would be once per month. User is going up and down the pull-down menu.) I don t see once per month. (User selects first pay period of the month. Okay we will go with that. User clicks Save.) User 2: (What would you expect it to say there?) Once a month. First pay period of the month I guess that would be once per month. (Is that clear?) No. User 5: (He selects the Frequency of Special Occurence when he needs something to come out once a month. Would that come out once per month?) No. (Is it obvious what would be Page 19

20 once per month in those values? He is still scrolling up and down.) No it is not obvious to me. (HE ASKED FOR HELP SO I HELPED HIM SELECT THE CORRECT FREQUENCY.) User 9: (User opens Frequency pull-down. She is confused as to what value means once per month.) First pay period of the month? (Is that clear that is once per month?) No. You ve got first, fourth or third pay of the month. Last pay period. 2 Users confused what flags mean User 2: (Where do you think you would go? User is clicking through flags. Do you think you would do it from there?) I am going to do a deduction from one of these. (Are you sure? User clicks Deductions in side navigation.) Okay there it is. What was that what I was on? That is confusing. Page 20

21 3 Scrolling User 4: I did not like that I had to scroll down to see the Save button. User 7: (USER DID NOT SAVE. SHE DID NOT SCROLL DOWN PAGE. Okay, I want to ask you a question. Did you see this [scroll bar]?) I did see that. (You used it. You didn t think you needed to go down and Save?) Oh. You know what? (Did that occur to you?) It did not occur to me. (Do you think if this [Save button] was up closer?) If it were up closer to the field, definitely. (You would have seen it?) Right. (I want to ask you another question. It is great that you used the Find field. But I want to know why. Did you think there could be anything below here [Garnishment drawer] and that you could scroll down and see?) You know what? For some reason, I don t know why, but I am not looking at those. (That is okay. That is what I want to know.) I am just not. (Does it look like the page is done and that there is nothing below?) It does. Even though that scroll bar is there, which I should know from using the computer, i just keep... (It s okay. You are not doing anything wrong. You are helping me. People are not Page 21

22 scrolling in this application and I am trying to figure out why. So you are really helping me. You did see the scroll bar and you did use it. But once we got in there you did not go down [the screen]. And it is great that you used the Find field. But I am wondering why you did not scroll down. Does it look like there is nothing below?) Right it does not look like anything is there. And I guess I am thinking that everything I want to do should be on one page so I can see it. (So you shouldn t have to scroll?) Yes.I think that in my mind some of this stuff likes kind of big and it does not need to be that big and they could fit everything on one page. 4 Confused if they added deduction or not User 4: I keep hitting save and it is not taking it. (How do you know it did not take it? Do you see it? It was not obvious to the user that the system saved the deduction. He does finally see it on the left.) Page 22

23 5 Not sure what fields are required User 5: (I ask him what else he would do to complete the task and he keeps opening the other drawers on the page. He does not see the Save button at the bottom of the page. I think he is confused by not knowing which fields are required.) The bouncing back and forth I don t like. User 7:. (She adds the info but is confused about whether to enter factor IF REQUIRED FIELDS WERE MARKED, SHE WOULD NOT BE CONFUSED.) There is a Factor but I don t know... (You don t need to worry about that.) Okay. (So what else would you do?) Calculate on. User 9: (Okay so what else would you do? User opens Advanced Settings. You don t need to do anything in there.) I don t know what factor means. (You don t need to do anything in there.) 6 Confused by screen layout User 8: (We are going to add a Dental125 deduction. User is scrolling through the deductions that were already added. User is reading items on page.) You want to set it up? (Right. User scrolling down and reading drawers.) I would click on Other Deductions. (Why if it is a dental?) Well cause... (User is scrolling through deductions that were already added. Those were already added.) Right. (So why are you looking at those? We want to add one. What are you thinking?) I am thinking that Dental is either here. (She is on the deductions that have already been added 10:32.) Or it has to have a drop-down that you can add another one. Or there is no place that says add additional deduction. (SCREEN LAYOUT IS CONFUSING HER. So there is no place that you would look to add one? She clicks Add Deduction button again (second time) She clicked the button again. So where else would you look for that?) I was looking at this recently added [flag] right here. Page 23

24 (Do you think that did anything?) No. (User clicks Add Deduction button again. Do you think there is anywhere else on the page where you would look?) Why should I when it says Add Deduction right here? It says Add Deduction so you would think the next screen would come up. User 11: (What do you think these mean over here [on left]?) Different things people might need individual employees? (Are these something we need to add or have been added? Can you tell?) No I can t tell. Recommendations for Task 2 Marked all required fields on all screens within application. Put all information on one screen to eliminate scrolling. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. As in the March test, users had problems understanding this screen. Redesign Configure Company Deduction to function and look like Employee Deduction. Do not include flags in redesign. Enable buttons to work when user clicks them once - eliminate multiple clicks In a focus group, ask users what Frequency values are most meaningful to them. Task 3 Employee deductions Please setup this deduction on the employee level: $50 per pay Time to complete task User Task 3 F F 2:29 4:25 3:18 4:16 1: :33 Average 3:33 Page 24

25 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 3 (Employee Deductions) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Users are not noticing headings 3 User Users 1, 2 failed task due to this issue Enabled Save and Cancel buttons on Employee Deductions confused user into thinking he had not saved deduction in modal window 2 User User 5 clicked Cancel button twice and Save button six times. Users slow to find Employee Deduction screen 2 User Scrolling 3 User Time dealing with issues for Task 3 User Average time dealing with issues Task 3 Estimate: 4:00 4:00 4:00 4:00 4:00 4:00 4:00 4:00 Task time: 1:25 3:26 2:29 4:25 3:18 4:16 1:39 2:33 Issue time: :26 1:05 :43 2:10 1:16 2:20 1:05 :20 1:06 Page 25

26 Flow preference User Tota l Compan y Deductio n (User prefered Compan y Deductio n because right side of screen was blank.) (He liked compan y because he thought it walked you through it.) N/ A 2 Employe e Deductio n I like this [Employee] better. (Was it easier to understand?) Yes I prefer the way this one works better. (User understood that deductions on the left were ones that had been already added to the user. All buttons were obvious.) (Which one did you think was simpler?) The one where it came up automaticall y [Employee > Deduction]. The less clicks the better. (What did you think? ) That was fairly easy. (Employee was this easier to understand?) Oh yeah no brainer. Any time you have selections of what you can choose instead of just trying to fill in a field, it is easier. This [Employe e Deductio n. Definitely this. This is better this is much better. N/ A 5 Specific information on issues 1 Users are not noticing headings User 1: (Okay so what would you do?) Do the amount. (User enters $50 in field. USERS ARE NOT NOTICING THE HEADINGS. Well is that what you want to do?) Well I guess we need to select the deduction first. What is the deduction going to be? User 2: User reads task. User scrolling up and Page 26

27 down looking for deductions.) He wants to deduct $50 per pay. (User enters info on 401K deduction. 10:21 Hold on. Is that the right one?) Oh that is 401K. (You did not notice that right away?) I did not notice that right away. (Why?) I was looking here [at deductions on left of screen] to see if there was a regular deduction. And I thought was just regular if I did not pick one of these. (You did not notice the heading?) No. I didn t. It s gray it does not really stick out. But now I notice it. 2 Enabled Save and Cancel buttons on Employee Deductions confused user into thinking he had not saved deduction in modal window User 5 - This is not saving. (User was clicking the save button on the main employee deduction page. What confused you into thinking the deduction was not saved having the buttons right here?) Yes. Now this save, saves the whole employee s information again right? (This won t do anything until you make a change. But that confused you because you thought you needed to save it again? Because the buttons were there?) Right. Page 27

28 Recommendations for Task 3 User confused over enabled buttons on Employee Deduction screen, see Usability and Accessibility Changes Needed in Perform, Problem area: Users not seeing light-colored buttons, page 31. Put all information on one screen to eliminate scrolling. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Enable buttons to work when user clicks them once - eliminate multiple clicks In a focus group, ask users: Which Frequency values are meaningful. Show them screen designs with different headings treatments (colors, fonts) and ask them which is more noticeable. Perform a card sorting test to determine a more meaningful label than Compensation. Task 4 (Employee s salary Pay Rates) Please look up the following information on the employee level: What is Greg George s annual salary? Time to complete task User Task 4 F :24 1:56 2:20 F :27 1:29 11 :45 Average 1:00 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Page 28

29 Issues that impacted Task 4 (Employee s salary Pay Rates) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Used Find field but when user selected employee, it did not appear in the Find field. 3 Defect Scrolling 3 User Went to wrong screen Not noticing headings 3 User User 7 failed task due to this issue User 1 failed task due to this issue Time dealing with issues for Task 4 User Average time dealing with issues Task 4 Estimate: 5:00 5:00 5:00 5:00 5:00 5:00 5:00 5:00 Task time: :37 :24 1:56 2:20 1:23 :27 1:29 :45 Issue time: :24 N/A N/A 1:08 1:23 N/A N/A N/A :58 Specific information on issues 1 Used Find field but when user selected employee, it did not appear in the Find field. User 5 : (User clicks into Find field and adds name Geo and employee comes up but he does not notice and continues to type name in Find field. He is confused because his name does not come in the Find field (like it does in Google when you select something) so he does not think it is working. He does not seem to notice the EE card displaying. 20:59 22:01 (How did you do that? Oh, you are on him.) Well that is probably why he didn t pop up. (But you did not notice that? I did not notice it either.) Because it does not go away. (I expected his whole name to display in the Find field is that what you expected?) Yes (Like Google does it.) Yes then it pops up. That is different. User 7: (Find field acted same as above but after Page 29

30 a few seconds (user did not notice this), George disappeared.) 2 Scrolling User 7: (Right. So how would you get there? User is reading navigation. She is not scrolling up.) I am going back to Home because this is dental this is not where I want to be at all. (User is looking in top navigation.) I am going to look in Payroll no that does not look right at all. (Do you know what employee you are on?) I am Page 30

31 on... I was not on Greg was I? I need help. (I scroll up.) I am not scrolling again! (No this is good because I need to know that. ) I am on Dan Davis. (What did you expect? Did you expect to see this all on one screen?) Yes and I expected, since I am going to something completely different, I am thinking. I want to go to a different person, I expect it up there to say... to give me a clue that I want to go back to another employee. ([I think she may be referring to the task] Well I did, you want to go to the employee level, so what would that mean?) Employee level. (So what would that mean?) That means I would go back to View Employees. (But you did not know to scroll up?) I want to see everything on one screen. I used to scroll all the time so I don t know why I am not using it. It just doesn t seem obvious. 3 User confused by icon User 7: This little thing here is very confusing to me I don t know what it is. I don t think I have seen that before. (The funnel icon? You don t know what that means?) No. I have never seen that. Recommendations for Task 4 Enable the Find field to display employee name when user clicks on it in results. Page 31

32 Put all information on one screen to eliminate scrolling. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. In a focus group, show users screen designs with different headings treatments (colors, fonts) and ask them which is more noticeable. Task 5 Part 2 (Add column) Pay employees for the Market Place - Add a column for H125 deduction amount. Time to complete task User Task 5-2 1:25 F 2:22 F F 2:19 :30 11 F Average 1:49 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 2 (Add column) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Scrolling Could not find added column 3 User Users 2, 5 failed task due to this issue User could not locate Deductions tab in Modal window 3 User User 7 failed task due to this issue User does not understand screen layout 3 User User 11 failed task due to this issue Page 32

33 Time dealing with issues for Task 5 Part 2 User Average time dealing with issues Task 5-2 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:25 3:15 2:22 2:29 2:30 2:19 :30 2:30 Issue time: N/A 2:46 1:07 1:28 1:02 1:38 N/A 2:30 1:59 Specific information on issues 1 Scrolling User 2: (Looking for column) So where is it? (Where do you think it is?) It should up here on the top right? These are columns. Where is it? (User not seeing horizontal scroll bar. Do you like all of these scroll bars?) No. There are too many scroll bars. (How do you think you would find it?) I have no idea. I am looking. Should I reset? I saw a reset down here. Reset? (User clicks Reset button. Message pops up. User clicks No. User is clicking horizontal scroll bar on browser and reading column names.) I don t know where it added it. (User opens Add Column button to check again if it was added. I show her how to tab into columns at 22:39.) Oh that is ridiculous. I don t like that at all. That is going to make it even harder. User 4: Where is it? (Where do you you added it, you would want to see it?) Oh yeah. Did I add it? User 5: Now where it come in at? (Did they cover that in the training how to add them? Yes they did cover how to add them but when it came in, it was just right where they had the cursor. So... (How do you think you would find it? Do you think it added it?) I do think that it added it. (OK so how do you think you would find it?) At this point, I do not know. I give up. Page 33

34 2 User could not locate Deductions tab in modal window User 7: (User finds Add column button quickly and opens modal window. Now that is a deduction. User is on Earnings. What is highlighted?) OT (Techically it is selected not highlighted. Is anything else highlighted on here?) OK [button] (Anything else? Like at the top?) Add or Remove (How about below that?) Code, Description (She is not seeing the tabs.) Sorry. (Which tab are we on?) We are Add a column. (Name of modal window.) (Which tab are we on?) We re on Adding a column. (Name of modal window. We are on Earnings the tab. And it is highlighted.) That is highlighted? That is not highlighted to me. Recommendations for Task 5 Part 2 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. In a focus group, show users modal screen designs with tab treatments (colors, fonts) and ask them which is more noticeable. In a focus group, show users different screen layouts of Paydata Grid, give them tasks to perform and see which layout is most successful. Page 34

35 Task 5 Part 3 (Add column) Pay employees for the Market Place - Add a column for Vacation earning hours Time to complete task User Task 5-3 :14 1:32 F N/A F 1:25 :59 11 F Average 1:23 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 3 (Add column) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Scrolling Could not find added column 3 User Users 4, 7 failed task due to this issue User had problem finding values in modal window 3 User User does not understand screen layout 3 User User 11 failed task due to this issue User having problems tabbing in fields 3 User Time dealing with issues for Task 5 Part 3 User Average time dealing with issues Task 5-3 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Page 35

36 Task time: :14 1:32 2:11 N/A 1:15 1:25 :59 :30 Issue time: N/A 1:27 1:55 N/A 1:10 N/A N/A :30 1:06 Specific information on issues 1 Tabbing in columns User 2: (She is having problems tabbing in the fields it is jumping all over and it is hard to get into the correct field.) 2 Scrollling Cannot find column User 4: Wow I don t remember seeing that in the 40 minute demo that they had me watch. I don t even remember that. Maybe it is not in there. Wow. Boy that is not very intuitive. (You would never had known to do that?) No. I sure wouldn t. I mean how would you? How would someone know that unless they had been told that? That is not intuitive. (You watched the 40 demo and it is not in there?) I watched the demo and I don t know that it is in the demo. I watched it and really paid attention. Maybe that was in there but I don t remember it. And intuitively, you want this to be as intuitive as possible. That is not intuitive. User 7: (I show her how to get to it. Would you have found that?) No, I would have called. Recommendations for Task 5 Part 3 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. In a focus group, show users modal screen designs with tab treatments (colors, fonts) and ask them which is more noticeable. In a focus group, show users different screen layouts of Paydata Grid, give them tasks to perform and see which layout is most successful. Enable tabbing in fields so that screen does not "jump" making it hard to read. Page 36

37 Task 5 Part 4 (Add amounts in columns) Pay employees for the Market Place: Dan Davis, 40 hrs, $50 H125 make up Time to complete task User Task 5-4 F 1:02 1:10 2:34 F 2:35 F 11 :18 Average 1:40 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 4 (Add amounts in columns) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Scrolling Cannot find column 3 User Users 1,7, 9 failed task due to this issue System does not delete information inside field when user clicks into field 1 User Time dealing with issues for Task 5 Part 4 User Average time dealing with issues Task 5-4 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:50 1:02 1:10 2:34 2:01 2:35 1:26.18 Issue time: 1:01 N/A N/A 1:03 1:35 :45 1:19 N/A 1:01 Page 37

38 Specific information on issues 2 Scrolling Cannot find column User 1 Why won t it let me go to that column? Am I missing something? I can see the edge of it here. Can I shrink these? (I used the arrow keys and tabbed into the fields and it worked.) I don t like that. I would want to be able to see that [all columns]. I am visual and I would forget if I didn t have this in front of me. User 4 - Personally, I would rather see all of them even if I had to scroll. (User does not see the horizontal scroll bar.) User 8: It was confusing because these (columns] were hidden. As you scroll across, you would think that they would all show up. And so there were hidden fields. It didn t tell you that they were hidden but by scrolling across... (It is not obvious?) No. If you did it weekly and you knew the columns were there maybe. But no. User 9: I know there should be column there. (User reads column headings.) I don t see the column. I give up. (I show her how to get to other columns.) Is the column there? It should be there. (I scroll down and use application horizontal scroll bar.) Crap. Okay, I was looking for something but I did not see that. You have to go to the bottom. (Do you like that?) No. (Not obvious? It is not obvious that there are more columns over there?) It wasn t but I knew there had to be but not how to get there. Because I am seeing this line here (browser scroll bar) but it is not moving. Recommendations for Task 5 Part 4 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Task 5 Part 5 (Add amounts in columns) Pay employees for the Market Place: Ernie Evers, 40 hrs, 5 OT Page 38

39 Time to complete task User Task 5-5 1:25 :19 :40 :51 :10 :09 1:32 11 :10 Average :50 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 5 (Add amounts in columns) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Tabbing in columns hard to read 3 User Scrolling Cannot find column 3 User Time dealing with issues for Task 5 Part 5 User Average time dealing with issues Task 5-5 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:25 :19 :40 :51 :10 :09 1:32 :10 Issue time: 1:23 N/A N/A N/A N/A N/A N/A N/A 1:23 Specific information on issues 1 Tabbing in fields makes it hard User 1: (User does not see column for regular hours so he adds one. It is there but it is hard to Page 39

40 to find information see because by tabbing through fields, it jumps all over the place. When he hits the end of the column, it goes to the next employee so he does not see it right away.) I want to see all those columns on the same screen. I don t like I just don t like it is coming together. It just seems to me to be hard to find. 2 Scrolling Cannot find column User 8: (User trying to get into column fields.) How did you do that? (User does not seem to notice what column that she is on. I tell her to change columns.) 3 Wants to see everything on one screen User 1: (What are you thinking?) I want to see all those columns on the same screen. I don t like I just don t like [how] it is coming together. It just seems to me to be hard to find. Recommendations for Task 5 Part 5 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Enable tabbing in fields so that screen does not "jump" making it hard to read. Task 5 Part 6 (Add amounts in columns) Pay employees for the Market Place: Tony Simpson, 32 hrs, 8 Vac Time to complete task User Task 5-6 :24 1:21 :30 1:12 1:05 :03 F 11 :10 Average :59 Page 40

41 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 6 (Add amounts in columns) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Scrollling Cannot find column 3 User User 9 failed task due to this issue Time dealing with issues for Task 5 Part 6 User Average time dealing with issues Task 5-6 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: :24 1:21 :30 1:12 1:05 :03 :50 :10 Issue time: N/A N/A N/A N/A N/A N/A :49 N/A :49 Specific information on issues 1 Scrolling Cannot find column User 9: (That is confusing to you because you don t know how to go back in [to the column fields]?) Yes. (I had to show her how to go back into column field.) User 11: (I ask her to look for the vacation amount column. She did tab over.) That s real bad. It would be nice to see them all in one page. Page 41

42 Recommendations for Task 5 Part 6 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Task 5 Part 7 (Net Amt of check) Pay employees for the Market Place: What is the net amount of Ernie Evers check? Time to complete task User Task 5-7 F 1:30 F 2:24 F F 1: :20 Average 2:18 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 7 (Net Amt of check) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Difficultly locating Net Amount Users were looking for it down the the Totals fields; Cannot see all of EE drawer 3 User Users 1, 4, 7, 8 failed task due to this issue User had difficulty getting in and out of EE drawer 2 User User 2, User 5 User did not understand why the Calc Check button did not work (he did not have employee selected) User thinks you cannot access net amt from paydata grid 2 User 3 User User 5 User 8 failed task due to this issue Page 42

43 Time dealing with issues for Task 5 Part 7 User Average time dealing with issues Task 5-7 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:25 1:30 3:05 2:24 1:30 1:02 1:11 1:20 Issue time: 1:12 1:22 3:03 1:54 1:30 1:02 N/A N/A 1:54 Specific information on issues 1 Difficultly locating Net Amount Users were looking for it down the the Totals fields; Cannot see all of EE drawer User 1 (Did not find it.) [Paydata grid] I don t like that you can t see everything in one screen shot. (You cannot see all of the EE drawer?) Yes. (Vertically and horizontally you can t see it?) Yes, I would like to see that all in one screen shot vertically and horizontally. User 2 - I can t see the whole thing. I don t like that. (You want to see as much information as possible on the screen?) Yes. (She is looking down by the Totals. Do you think it would be anywhere else on the screen? She found it. She was confused as to how to get out of the EE drawer.) User 4 - What is the net amount of their check? Oh, Enter Paydata. (User clicks the progress bar Review & Approve) Do I have to go to Review? (Do you?) We are about to find out. (Does that answer your question for how much his check is?) Net amount. (How much HIS check is?) No. (User clicks app back button to return to Paydata grid. Is it clear how to do this on the screen?) It is not clear at all. I am guessing. So I probably have to tab over. (User tabs on columns.) But I honestly don t know. (Give up?) Yes. Oh I do remember seeing that on the video. User 5: (What is the net amount of the check?) The net amount shall be... (user is scrolling Page 43

44 horizontally. He finds it. Is it obvious where it is up there?) No. (Would it be more obvious down below with the Totals?) Yes. (That is where it appeared that he was looking for that info.) User 7: (How would you figure that out? User selects employee on grid. How would you find out how much his check is going to be? User is tabbing in columns. She is reading items on screen but she is not reading items in side nav.) I don t know. (I show her. She does not see net amount at top. What would make that more obvious? User is silent. Different color? Bigger?) Well, it seems that it should be down in here [Totals]. (OK that makes sense.) User 8: I would say it is not in this page [screen]. Right now you are just entering hours and time data. It would not be on this page. (I show her how to do it. She does not see net amount. She is looking at the totals. Was that obvious where it was?) No. User 11: (What is the Net amount of Fred s check? User is looking at the total section. User assumed it would be down with Totals. User even scrolled down.) Page 44

45 Recommendations for Task 5 Part 7 Move Net Amt down near the Totals -- this is where all users looked first to find it. Enable the ability to exit the EE drawer even though user did not calculate check. When user clicks to exit drawer, display a message asking user if they want to calculate the check (and tell them how) or if they want to delete the check. Task 5 Part 8 (Net cash for payrun) Pay employees for the Market Place: What is the Net Cash for this payrun? Time to complete task User Average Task 5-8 1:15 N/A 1:19 :20 1:02 1:03 :21 1:22 :57 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task N/A Total points scored Issues that impacted Task 5 Part 8 (Net cash for payrun) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Totals button is not obvious 2 User Information on Reivew & Approve screen looks smashed 2 User Every user commented on this screen in at least one of the tasks. Page 45

46 Time dealing with issues for Task 5 Part 8 User Average time dealing with issues Task 5-8 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:15 N/A 1:19 :20 1:02 1:03 :21 :22 Issue time: N/A N/A N/A N/A N/A N/A N/A N/A N/A Specific information on issues 1 Totals button not obvious User 2: (Totals button is not obvious what is does of that it is there. We did find it but it was hard due to scrolling.) There is no dollar sign. If you would see the whole screen and not go up and down and up and down, it would be a lot better. Page 46

47 2 Information looks smashed on the screen User 4: (User reads net cash amount. What do you think of this screen? Is that easy to read?) No. It looks like they just crammed it into the screen. I mean it works. User 5: (Is all of it easy to read?) Most of it except for this column over here. They definitely need to add some space over here. User 7: (User is scrolling some. Is it easy to read.) I don t like this over here [smashed part]. That bothers me. There is all this space over here with nothing. Recommendations for Task 5 Part 8 Move the Totals button to a prominent location on the Paydata Grid screen. Change the layout of the Review and Approve screen -- take advantage of the extra space on the left-side of the screen to expand the areas on the right so information does not look smashed. This will also allow users to see the Back to Paydata button easily. Page 47

48 Task 5 Part 9 (Add a manual check) Pay employees for the Market Place: Add a manual check for Dan Davis, 3 Regular hrs Time to complete task User Task 5-9 5:15 5:58 F F F F F 11 F Average 5:37 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task Total points scored Issues that impacted Task 5 Part 9 (Add a manual check) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Not obvious to add and calculate manual check 3 User Users 4, 5, 7, 9, 11 failed due to this issue Users want to save to calculate check 2 User Users cannot get out of EE drawer because they did not calculate check 2 User OT on top of Regular 2 User Clicked +Check button 2 User Confused about clicking Back to Paydata grid button (Progress bar heading and button have different names) 3 User User 8 failed task due to this issue Page 48

49 Confused about not seeing entire screen (every user mentioned this at least once) 3 User Time dealing with issues for Task 5 Part 9 User Average time dealing with issues Task 5-9 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 5:15 5:58 2:00 3:26 5:12 2:21 4:30 4:01 Issue time: 3:30 3:13 :38 1:04 3:28 2:21 3:10 1:08 2:19 Specific information on issues 1 Not obvious to calculate manual check Users: 1, 2, 4, 5 7, 9, 11 User 2: (It was not obvious to clicks Gross to Net to calculate the check?) No. It should have a Save and automatically calculate it when you save it. User 4: Oh yeah the Gross to Net. (Is that intuitive? Did you know to do that?) I didn t but.. (Would you do that? Would you call for help? What would you do?) I might get stuck on that one. I do remember them [training video] at some point using the Gross to Net. User 5: (Do you know how to do the calculations?) No I don t. User 11: (How would you run the calculations? Is it obvious?) Why wouldn t it calculate automatically? (She clicks the disabled calc button. She asks for help. Gross to net that is not obvious.) No it is not obvious. Page 49

50 2 Users want to click a Save to calculate check Users: 2, 9 User 9: Do I need to save this? (Do you think you do?) I would think. (User is scrolling up and down screen.) How do I save it? (Do you save on the one you have now?) I need to go back to my grid. And I don t see how to get back there. (Did you watch the training video 34:01? She did not.) 3 Users cannot get out of EE drawer because they did not calculate check Users: 2, 9, 11 User 2: (She clicks Add manual check and opens the drawer.) Is this his? (Do you think that it is?) I don t know. This window just popped up. (I have the user scroll up and down. Since the blue is around his name, does it look like it is his?) No. (What would make it look like it is his?) If it was all in the same box. It does not look like it is all in the same box. It looked like this window just popped up. User 9: I need to go back to my grid. And I don t see how to get back there. 4 OT on top of Regular Users: 5, 7, 8, 9, 11 User 5: They have OT on top instead of... I wonder why they do that... User 9: I would think regular would be on top. Page 50

51 5 Clicked +Check button Users: 1, 9 User 1: Now it did it. I just did not do it right. Okay, now I need to add a manual check this part I am finding very confusing. (Why?) I am not sure yet. 6 Not obvious to add manual check User 8 7 Confused about clicking Back to Paydata grid button (Progress bar heading and button have different names) User 8: So I would go back to the Payroll data. (Is there another way to do that besides clicking on Enter Paydata [progress bar]? User is reading navigation tabs.) It is not obvious to me. User 7: ( Is there another way for you to get to Page 51

52 that from this screen?) Enter Paydata (There is another way. Can you find it? User is reading top navigation. I will give you a hint: scroll. User scrolls but still does not see button. Do you see the button? She hovers over the blue button. No.) Oh Paydata Grid. (You would not have known that is what it meant?) No, even if I had saw that I would not have known... Paydata Grid. 8 Confused about not seeing entire screen User 7: (EE drawer opens.) Oh now where did it go? (She is scrolling up and down screen. Are you on Dan?) I am not sure where I am. (How do you think you would find out where you are? User scrolls up but you still cannot see top of screen.) This is my manual check but it is still not telling me who I am making a manual check for. Page 52

53 Recommendations for Task 5 Part 9 Put all information on one screen so users don't scroll vertically or horizontally. For more information, see Usability and Accessibility Changes Needed in Perform, Scrolling, page 6. Change the layout of the Review & Approve screen -- take advantage of the extra space on the left-side of the screen to expand the areas on the right so information does not look smashed. This will also allow users to see the Paydata Grid button easily. On the Review & Approve screen, change either the progress bar name or button name so that the Paydata Grid has the same name in both places. Enable the ability to exit the EE drawer even though user did not calculate check. When user clicks to exit drawer, display a message asking user if they want to calculate the check (and tell them how) or if they want to delete the check. Move Regular above OT on the EE drawer. Task 5 Part 10 (Review your payrun and open Pre-post Payroll Journal) Pay employees for the Market Place: Review your payrun and open Pre-post Payroll Journal Time to complete task User Task :08 N/A :34 3:04 :09 1:01 :14 11 :22 Average 1:24 Success rate for task Complete success without assistance = 1, Partial success (with assistance) =.5, Gives up or wrong answer = 0, Total points possible = 13, Had training, Failed User Task N/A Total points scored Page 53

54 Issues that impacted Task 5 Part 10 (Review your payrun and open Pre-post Payroll Journal) 1 = Trivial, 2 = Moderate, 3 = Serious Issue Severity Type Notes Approve Payrun screen information looks smashed together 3 User Every user commented on this screen in at least one of the tasks. Total button not obvious on paydata grid 2 User Time dealing with issues for Task 5 Part 10 User Average time dealing with issues Task 5-10 Estimate: 1:06 1:06 1:06 1:06 1:06 1:06 1:06 1:06 Task time: 1:08 N/A :34 3:04 :09 1:01 :14 :22 Issue time: N/A N/A N/A N/A N/A N/A N/A N/A N/A Specific information on issues 1 Information looks smashed on the screen User 8: It kind of runs over. You have a ton of room over here and then this is scrunched in. In my opinion, this column two should not even be on here. You cannot read it. This Prepost Journal could be smaller. And if this [info on far right] is really important, have this across the bottom. User 9: Do you like this screen? Can you read the information easily?) No that is not easy to read (stuff on right side). Does everything need to be so big? I can read this without my glasses. Page 54

55 Recommendations for Task 5 Part 10 Change the layout of the Review & Approve screen -- take advantage of the extra space on the left-side of the screen to expand the areas on the right so information does not look smashed. Move the Totals button to a prominent location on the Paydata Grid screen. Page 55