Survey of severity ratings in usability. Lisa Renery Handalian

Transcription

1 Survey of severity ratings in usability Lisa Renery Handalian

2 Renery Handalian 1 Introduction A few careers ago, I was a writing instructor at U. Mass, Boston, where all undergraduates are required to pass a Writing Proficiency Exam ( WPE ) in order to graduate. The WPE consists of an essay test based on a reading set that students have a month to prepare. Three times a year experienced instructors from across the curriculum convene to grade the essays; each essay is assessed by two to three instructors before receiving a Pass or Retake grade. On the day of the grading, before actually starting, instructors engage in a calibration session during which everyone reads and evaluates four model essays; based on the ensuing discussions of why we graded as we did, the evaluators develop assessment guidelines on which to judge that day s essays. The intent of this preliminary process of consensus building, or convergence obtaining (Wilson, 2003), is to develop a workable grading rubric designed to ensure that at least for that particular exam on that particular day with that particular group of evaluators, all the papers are held to the same agreed upon standards. The guidelines behind said standards are then communicated to Retake students in the form of guidance on how to improve their writing. They also provide justification in the event that a student challenged his or her assessment with the Academic Evaluation Committee at the university. I was reminded of these calibration sessions (and their purpose) during a Rolf Molich UPA talk about the CUE (Comparative User Evaluations) studies. This article presents an examination of the different criteria by which usability professionals have categorized the degree to which a problem impede[s] task performance (Catani &

3 Renery Handalian 2 Biers, 1998, p. 1332). In doing so, I will draw upon the methods of Rubin (1994); Wilson (1999, 2003); Dumas & Redish (1999); Nielsen (2001); and, Molich & Jeffries (2003). I will then assess the actual usability case provided based on the different severity ratings reported in the literature, concluding with my assessment of the method that I find most effective. Attention will be paid to the evolution of severity standards over the past decade as a sign of the maturation of the field of usability testing. Review of the Literature Among the authors mentioned above, there is general consistency in how they rate the severity of usability problems. That is, most present a scale comprised of at least three ratings ranging from severe to low; some use a calculation, such as severity plus scope i.e. local vs. global occurrence (Dumas & Redish, 1999), or severity plus frequency (Rubin, 1994), among other factors. For the purposes of my review, I have organized authors treatment of problem severity accordingly: Pervasiveness o Frequency in terms of probability of problem occurrence o Scope of problems experienced o Number of users affected o Problem persistence vs learnability Performance o Disruption of productivity, in particular regarding the inability to accomplish business goals (Wilson, 2003) o Time delays associated with task completion and problem circumvention Impact o Probability of data loss or system damage o Risk to business reputation, profitability, and revenue The implications of these severity assessments on the health of the business, a.k.a. market impact (Nielsen, 2001), will also be discussed.

4 Renery Handalian 3 Pervasiveness Frequency. The use of calculations in determining severity is somewhat common in the literature, perhaps speaking to a desire or instinct among practitioners to emulate scientific research where possible. In the earliest treatment of usability problem ratings that I encountered in my reading, Rubin (1994) quantifies what Nielsen (2001) and Molich & Jeffries (2003) refer to as frequency of problem occurrence as experienced by users. Where Nielsen and Molich & Jeffries provide qualitative guidelines such as 1. Rarely, 2. Often, and 3. Sometimes (Molich & Jeffries, p. 3), Rubin furnishes specific numeric benchmarks for task completion and problem frequency. In determining severity, he first focuses on non criterion tasks that 70 percent of participants do not successfully complete in the allotted time (p. 274). He then prioritizes these noncriterion tasks to be fixed according to criticality which he calculates by combining the severity of a problem and the probability that it will occur (p. 277). Rubin ranks the estimated frequency of occurrence as 4: Will occur 90% product is used; 3: Will occur 51 89% of time; 2: Will occur 11 50% of time; [and,] 1: Will occur of time (279). Rubin points out that the sum of a frequency of 4 and a severity (also ranked from 1 4) of 1 is of course the same as a frequency of 1 of a problem with a severity of 4. Using this method he claims the very highest priority would be assigned to problems that made the product unusable for everyone (p. 279). Scope. Dumas and Redish (1999) also suggest a combination of attributes to determine the overall severity and scope of usability problem severity. Like Rubin, they use the term criticality to represent the scale of problems: Level 1 problems prevent

5 Renery Handalian 4 completion of a task Level 2 problems create significant delay and frustration Level 3 problems have a minor effect on usability Level 4 problems are more subtle and often point to an enhancement that can be added in the future (pp ). However, Dumas and Redish are alone in their mention of scope (or, how widespread is the problem? p. 322) as it pertains specifically to global vs local issues: Global problems are more important than local problems. The scope of local problems is restricted. In their simplest form, they apply to only one screen or dialog box or window or page of a manual Global problems have a scope that is larger than one screen or page Usually, the global problems are the most important to address (pp ) Since neither severity nor scope alone is sufficient to identify which problems should be fixed first, Dumas and Redish recommend taking both together in consideration to organize problems into an order of priority (p. 324). Number of users affected. While Rubin s (1994) and Dumas and Redish s (1999) works address how frequently occurring or widespread usability problems are, neither specifically adds a consideration of the number of users affected not just the frequency with which they are interrupted or lost. Wilson s rough example of a formal definition of usability severity codes mentions that a problem with a Level 2 High rating affects many users (2003, paragraph 7). He also refers to Level 1 catastrophic errors as those that prevent many people from doing their work (1999, paragraph 10). Although Wilson explores the effects on users as members of the greater business / organizational context (discussed in the Impact section of this paper), it would be helpful here to have a scale by which to determine the different degrees of many.

6 Renery Handalian 5 Problem persistence. Nielsen (2001) identifies three factors that combine to determine the severity of a usability problem: frequency (see above), impact, and persistence. These factors are also used by Molich and Jeffries (2003) in the CUE 4 study. I have combined impact and persistence here under the heading of problem persistence. Nielsen rates impact as to the ease with which users will be able to overcome a problem; his persistence attribute refers to the learnability of the problem, that is, whether they will be able to overcome [it] once they know about it or repeatedly be bothered by [it] (1999, paragraph 3). As I see it, the disruption of work and the user s ability to overcome the disruption are inseparable. In their instructions for the preparation of CUE 4 evaluation reports, Molich and Jeffries (2003) rate both impact and persistence on a 1 3 scale in increasing severity. Again, I treat the two concepts as one. Their explanations follow: Impact (how serious is the problem when it occurs) [is rated:] 1. Minor (delays users briefly), 2. Seriously (delays users significantly but eventually allows them to complete the task), [and] 3. Catastrophic (prevents users from completing their task). Persistence (will users learn how to get around the problem) [is rated:] 1. Yes, quickly, 2. Only after encountering the problem several times, [and] 3. No. (p. 3) Consistent with other authors, Molich and Jeffries also suggest combining the numeric values to determine the severity rating score. Performance Disruption of productivity. In assessing severity, Wilson (1999) describes effects on performance as often a primary usability attribute. Whether poor or too good (paragraph 2) a products lack of usability can significantly disrupt or impede user

7 Renery Handalian 6 productivity. He lists several productivity related attributes that should be taken into consideration when assessing severity. His Level 1 Catastrophic problems often indicate performance so bad that the system cannot accomplish business goals (paragraph 10); to a lesser degree, Level 2 Severe problems point to performance so poor that the system is universally regarded as pitiful (paragraph 11). Levels 3 (Moderate) and 4 (Minor but Irritating) problems typically affect the ability for users to work with out being interrupted by inconsistencies that result in increased learning or error rates although mistakes caused by them are typically recoverable (paragraphs 12 and 13). Time delays. All of the authors mention time delays as sources of user frustration and performance obstruction. However, Wilson (2003), Nielsen (2001), and Molich & Jeffries (2003) include reference to both the time lost by trying to figure out a usability problem and the waste of time associated with trying to devise workaround solutions or circumvent a function altogether. Wilson s severity rating for one mentions workarounds into consideration; specifically, he categorizes Severe or showstopper errors as having no workaround (paragraph 1). Furthermore, High errors impair the operation of one or more functions and cannot be easily circumvented or avoided (paragraph 7). In a rating system he published in 1999, Wilson also characterizes Level 3 problems as moderate causing no permanent loss of data, but waste of time. There is a workaround to the problem unlike Level 2 problems that have no workaround (paragraph 12 and 11). In general the time loss associated with usability errors is accepted by all the authors as a source of inefficiency and waste. Impact

8 Renery Handalian 7 Data loss/damage. In addition to loss of time and productivity, Wilson (2003) also judges data loss or system damage as an indicator of the severity of usability problems. Among all of the authors he (and Nielsen to a lesser extent) focuses the most on this increasingly critical issue. Wilson bestows a Severity Code of 1 Severe to an emergency condition that causes the customer s system to fail or causes customer data to be lost or destroyed (paragraph 1). A 1999 example used by Wilson is the wrong default button on the confirmation message Do you want to delete this message?. Choosing Yes for the default button would be very serious because it would cause the loss of the entire database (paragraph 3). With enterprise wide database infrastructures powering entire organizations more than ever before, such usability errors have the potential to be extremely costly. Business risk. As a continuation of the above discussion, Nielsen (2001) and Wilson (2003) raise the important point about the effect of usability errors on the reputation and revenue of businesses. Nielsen warns one needs to assess the market impact of the problem since certain usability problems can have a devastating effect on the popularity of a product, even if they are objectively quite easy to overcome (paragraph 6). He asserts that Level 4 usability catastrophe(s) [are] imperative to fix before the product can be released (paragraph 12). He states that if severity ratings indicate that several disastrous usability problems remain in an interface, it will probably be inadvisable to release it (paragraph 1). Wilson (2003) concurs, suggesting if a usability problem could be connected directly to a business goal, that would insure [sic] that it was important enough to fix before shipping (paragraph 18). Placing

9 Renery Handalian 8 usability problems in a greater business context may well be a strategy to promote closer adherence to implementation suggestions. Please consult the Appendices for tables delineating the different severity rating methods proposed by each author or set of authors. Case Study I will evaluate the following case study by the severity rating methods of Rubin (1994), Dumas & Redish (1999), Molich & Jeffries (2003), and Wilson (2003). I will not apply Catani & Biers (1998) because their scale was not detailed enough, or Nielsen (2001) because he clearly informed the rating system of Molich & Jeffries who then expanded upon his categorization. I have highlighted information I found significant. Two of the five participants in a usability test have a problem that causes loss of data and makes it impossible to complete the task. After editing a page of data, they forget to click the Save button. Instead they close the window using the Close Window control. The edits they made are not saved. The other three don t have the problem because they click Save. Rubin (1994) Although it s a bit of a stretch on what he considers frequency, Rubin might use the fact that 40% of the test participants experienced a usability problem to give this problem a Frequency rating of 2 (moderate). His definitions of Severity do not have the benefit of colleagues fine tuning like those of later authors and as such are rather vague. A rating of 3 might fit his description in that the user will probably use or attempt to use the product, but will be severely limited in his or her ability to do so (p. 278). Adding the two scores together would give this problem a criticality rating of 5.

10 Renery Handalian 9 The fact that Rubin s method does not take into account important elements such as loss of data, scope, impact or time delay severely limits the application of his ratings scale. Thus, I would not choose this method over any of the others. Dumas & Redish (1999) Dumas and Redish combine the concepts of scope and severity to rate usability problems. If the Save button issue is one that is global, then it is of greater importance than if the error occurs only in that one editing function. Along their severity scale, this would be a Level 1 problem since the product design has prevent[ed] the completion of a task (p. 324). The authors do not modify their ratings with the number of users who encounter problems, thereby furnishing little guidance as to how to handle the 40% error incidence in the case test. Assuming that this is a global issue, it is unclear what scope does to increase the severity rating, as no numeric value was attributed to either local or global issues. Dumas & Redish merely explain that scope and severity should both be taken into account in the prioritization of problems to be fixed, yet they do not tell us how. Because of the lack of specificity, I would probably not use this method alone to assess usability problems. Molich & Jeffries (2003) Molich and Jeffries use a three fold scale by which they rate problem severity. Again in terms of frequency it is necessary to stretch the definition to encompass number of users rather than how often the problem occurs since it was presented here as an isolated issue. In this case, they might rate the experience of 2 out of 5 participants as 2 Often (note: even though the concept of Sometimes seems more

11 Renery Handalian 10 appropriate here, its rating is actually higher than that of Often ). With respect to Impact, they would be justified in attributing a Catastrophic rating of 3, as some of the users were prevent[ed] from completing the task (p. 3). Again, no formula has been supplied to determine how to treat severe problems relative to the number of users experiencing them. In terms of persistence, I believe this would take a rating of 2, whereby users could learn how the Edit screen functions after encountering the problem several times (p. 3). Although this method has honed severity ratings more than those developed earlier, I would still have questions in knowing how to apply them to usability situations. Wilson (2003) I find Wilson s rough example of a formal definition of severity codes (2003) to be the most descriptive, helpful and all encompassing of the four reviewed here. Although I found it useful to break down the assessment of problems into the three categories as done by Nielsen (2001) and Molich & Jeffries (2003), I feel that the greater business context in which Wilson places usability issues makes his rating methods extremely helpful to the assessment of problem severity. His descriptions contain rich detail that position the severity very clearly. Regarding the case at hand, I believe Wilson would take the following issues into consideration: 1. Editing a page of data Since the re editing of information is irksome to the user but not mission critical (i.e. that information could be elicited again albeit with difficulty; it can be re entered, though likely causing inconvenience and perhaps time

12 Renery Handalian 11 delay to the customer), Wilson would probably see this as a Medium ( non critical, limited problem no data lost or system failure ) issue. 2. Two out of the five participants Wilson categorizes Level 2 High errors as those that affect many users. It is up for debate whether 40% of users can be called many or not. I would tend to choose the severity rating of High based on that fairly significant proportion. 3. They forget to click the Save button Level 2 of Wilson s severity scale also indicates that with High errors, [t]he software does not prevent the user from making a serious mistake which really does characterize this usability error. Because of the not insignificant number of users experiencing problems with an issue that is not severe but does cause data loss caused by the application s inability to protect the user against error, I believe that the appropriate rating for this issue based on Wilson s scale would be 2 High. Again, some guidance as to how to handle the proportion of users affected would be very helpful. Conclusions Comparing the severity rating systems published in earlier texts (Rubin, 1994; Catani & Biers, 1998; Dumas & Redish, 1999), usability testers have increased the level of specificity and detail in their definitions and attribution of problem severity. It can be hoped and expected that refinements in ratings will also promote greater agreement among problem classifications. One problem identified by Molich, et al. (2003) in the reports produced by usability professions participating in CUE 1 and 2 was the lack of severity classification of problems. Some reports did not distinguish between

13 Renery Handalian 12 disastrous problems and cosmetic details. All problems appeared to be equally serious (p. 5). Catani and Biers (1998) also note disparity and attribute the apparent lack of agreement in severity rating among professionals to methodological issues and problem perception (p. 1335). I will close with Rolf Molich s remark: if we want to become a science, we must describe our core processes in great detail (UPA meeting, October 9, 2003). Whether or not usability testing will ever become scientific research doesn t really impact his call for enhanced standardization of rating methods in order to provide the maximum impact upon the implementation of usability recommendations. He says: For usability tests to be a reliable tool for making informed decisions, we must require that two usability tests of the same piece of software, for instance a web site, produce results that are reasonably similar. In particular, we must require that there is reasonable agreement on the problems that are defined as critical, and on the overall conclusion. Otherwise a project manager, who uses a professional usability test report to allocate development resources for correcting critical usability problems, may be mislead [sic]. (Molich, R.; Kindlund, E.; Seeley, J.; et al., 2003, paragraph one) Perhaps further opportunities, like subsequent CUE studies, will present themselves as to develop the kind of convergence in ratings (Wilson, 2003) that will enable those both performing usability tests and receiving their results to feel more secure in their consistency and acceptance. Such an evolution of standards may just contribute to the increased professionalization and scientification to which Molich and his colleagues referred.

14 Renery Handalian 13 References Catani, M. B. & Biers, D. W. (1998). Usability evaluation and prototype fidelity: Users and usability professionals. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting. Santa Monica, CA: HFES Dumas, J. & Redish, G. (1999). A practical guide to usability testing, London: Intellect Books, Revised Edition. Molich, R. & Jeffries, R. (2003). Comparative expert reviews: CHI 2003 workshop description. [Available online at Accessed on October 10, 2003] Molich, R.; Kindlund, E.; Seeley, J.; et al. (2003, in press). Comparative Usability Evaluation. Nielsen, J, (2001). Severity ratings for usability problems. [Available online at Accessed October 10, 2003] Rubin, J. (1994) Handbook of Usability Testing. NY: John Wiley and Sons, Inc. Wilson, C. (1999). Readers questions: Severity scales. Usability Interface, 5(4) April [Available online at severity scale.html. Accessed October 10, 2003] Wilson, C. (2003). Defining, categorizing, and prioritizing usability problems: UPA 2003 Idea Market. July 11, [Available online at ty problems_wilson.doc. Accessed October 10, 2003]

15 Renery Handalian 14 Appendices 1. Rubin (1994, pp. 278 and 279) Severity + Frequency Ranking Ranking Description Definition The user is either not able 4 Unusable to or will not want to use a particular part of the product because of the way that the product has been designed and implemented. 4 Will occur 90% of time product is used 3 Severe 2 Moderate 1 Irritant The user will probably use or attempt to use the product here, but will be severely limited in his or her ability to do so. The user will have great difficulty in circumventing the problem. The user will be able to use the product in most cases, but will have to undertake some moderate effort in getting around the problem. The problem occurs only intermittently, can be circumvented easily, or is dependent on a standard that is outside the product s boundaries. Could also be a cosmetic problem Will occur 51 89% of time Will occur 11 50% of time Will occur of time 2. Catani & Biers (1998)

16 Renery Handalian 15 [P]roblems were rated by usability professionals on their severity on a 5 point scale with 4 being sever and 0 being no problem at all (p. 1332).

17 Renery Handalian Dumas & Redish (1999, pp ) Problem importance has two dimensions: Scope How widespread is the problem? Global Local Severity How critical is the problem? Level 1 problems prevent completion of a task. Level 2 problems create significant delay and frustration. Level 3 problems have a minor effect on usability. Level 4 problems are more subtle and often point to an enhancement that can be added in the future. 4. Nielsen (1999, paragraphs 7 12) The following 0 to 4 rating scale can be used to rate the severity of usability problems. Ratings are based on a combination of frequency, impact, and persistence and also require attention to market impact of the problem. 0 I don t agree that this is a usability problem at all 1 Cosmetic problem only: need not be fixed unless extra time is available on project 2 Minor usability problem: fixing this should be given low priority 3 Major usability problem: important to fix, so should be given high priority 4 Usability catastrophe: imperative to fix this before the product can be released.

18 Renery Handalian Molich & Jeffries (2003, p. 3) Each problem must be rated on three scales A gross way to determine whether a problem should be fixed could be to add the three numberic values and see whether the problem gets a high score Frequency Impact Persistence How serious is the problem Will users learn how to get when it occurs? around the problem? How often does the problem occur? 1. Rarely 1. Minor (delays user briefly) 2. Often 3. Sometimes 2. Serious (delays users significantly but eventually allows them to complete the task) 3. Catastrophic (prevents users from completing their task) 1. Yes, users quickly learn how to get around the problem. 2. Users only learn to get around the problem after encountering it several times 3. No, users never learn how to get around the problem. 6. Wilson (2003) Rough example of a Formal Definition of Usability Severity Codes Severity Code 1 Severe (1 = Showstopper) 2 High 3 Medium Description An emergency condition that causes the customer s system to fail or causes customer data to be lost or destroyed. A showstopper usability bug can also be one that is likely to cause frequent data integrity errors. There is no workaround to these problems. A key feature needed by many customers is not in the system. A serious condition that impairs the operation, or continued operation, of one or more product functions and cannot be easily circumvented or avoided. The software does not prevent the user from making a serious mistake. The usability problem is frequent, persistent, and affects many users. There is a serious violation of standards. A non critical, limited problem (no data lost or system failure). It does not hinder operation and can be temporarily circumvented or avoided. The problem causes users moderate confusion or

19 Renery Handalian 18 4 Low irritation. Non critical problems or general questions about the product. There are minor inconsistencies that cause a slight hesitation or small aesthetic issues like labels and fields that are not aligned properly.