Heuristic Evaluation of Groupware. How to do Heuristic Evaluation of Groupware. Benefits

Transcription

1 Kimberly Tee CPSC 681 Topic Heuristic Evaluation of Groupware Heuristic evaluation [9] is a discount evaluation method for finding usability problems in a singleuser interface design. With this methodology, a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the heuristics ). Evaluators describe each problem found, rate its severity, and can suggest possible fixes. In 2001, Kevin Baker adapted the heuristic evaluation methodology to find usability problems in real-time, shared workspace groupware systems [1]. Real-time, shared workspace groupware systems support real-time collaboration between distance-separated people through a shared workspace a medium sized flat work surface where people collaborate by manipulating visible tools and task artefacts [6]. Some examples of shared workspace groupware systems are shared editors and multi-player games. The activities in a shared workspace can be divided into taskwork and teamwork, where taskwork is the actual execution of a task (by a single user) and teamwork is the work of working together [6]. Thus, groupware usability is defined as the degree to which a system supports both taskwork (single-user usability) and teamwork [3]. How to do Heuristic Evaluation of Groupware Baker s adapted heuristic evaluation methodology uses a set of groupware heuristics (see Groupware Heuristics below) designed to evaluate shared workspace interfaces for teamwork usability problems. Like Nielsen s heuristic evaluation methodology, it consists of three stages: 1. Orientation: If required, evaluat ors are introduced to the heuristics and the methodology. They are given a short description of the system under test, and (optionally) a scenario to help them explore and evaluate the system. 2. Evaluation: Evaluators go through the interface several times and inspect specific interface elements for compliance with the list of heuristics. They describe each usability problem found, rate its severity, and can suggest possible fixes (see Problem Report section). 3. Debriefing: After all evaluations are complete, evaluators, observers, and representatives of the design team group together all duplicate problem reports. They discuss their findings, prioritize the problems they found, and suggest solutions. Heuristic evaluation of groupware may be used to evaluate medium or high-fidelity prototypes, or almost-finished products. Unlike traditional heuristic evaluation which can be done on low-fidelity prototypes, heuristic evaluation of groupware is restricted to systems that are partly functional. However, Baker suggests that the groupware heuristics can be used to guide the design of groupware so that many potential problems can be addressed prior to any formal evaluation [3]. Benefits is a discount evaluation method quick and cheap [2] requires a small number of evaluators (three to five evaluators will typically identify about 40-60% of the total known teamwork problems) [2] doesn t require end-users [9] most of the problems found will be major ones, and most of the missed problems will be minor ones [2] non-usability experts can use heuristic evaluation fairly successfully [2]

2 Problems is limited to real-time, shared workspace groupware systems [1] some problems will go unnoticed, particularly if few evaluators are used [2] since real users aren t involved, surprise problems relating to their needs are not found [9] some problems reported are not actually usability problems [9] there is quite a bit of redundancy in the problems found by evaluators (the ratio of consolidated problems to original raw problem reports is about 1:6) so results synthesis can take some effort [2] does not provide a systematic way to generate fixes to the usability problems or a way to assess the quality of any redesigns [9] Groupware Heuristics The following heuristics (based on the mechanics of collaboration framework [1, 3]) are designed to evaluate shared workspace groupware systems for teamwork usability problems. These heuristics are tailored for workspaces that are used by groups of approximately five people [3]: 1. Provide the means for intentional and appropriate verbal communication: the groupware system should provide a channel for intentional communication so that collaborators can effectively communicate with one another. 2. Provide the means for intentional and appropriate gestural communication: the groupware system should support intentional gestural communication by making all explicit gestures and other visual actions visible to all group members, as well as maintaining the association between the gesture/action and its author. The system should also display the gestures in relation to the objects or people they are referencing, when applicable. 3. Provide consequential communication of an individual s embodiment: the groupware system should capture and transmit the explicit and subtle visual and verbal cues that are given by participants during collaboration, such as locations, gazes and views, actions, presences, and identities. 4. Provide consequential communication of shared artifacts: the groupware system should transmit the local user's feedback to all remote users, and should indicate when an artifact is being manipulated to help users determine what others are doing to it and how it has changed. 5. Provide protection: the groupware system should enable users to concurrently interact with the shared virtual workspace with minimal conflicts. 6. Manage the transitions between tightly and loosely-coupled collaboration: the groupware system should allow each individual to navigate the workspace independently. The system should also enable smooth transitions between joint work and individual work. 7. Support people with the coordination of their actions: the groupware system should allow people to effectively coordinate their activities by providing awareness of others and their actions in the workspace, by providing context for the actions which take place, and by providing collaborators a means of communication. 8. Facilitate finding collaborators and establishing context: the groupware system should provide information on potential collaborators so that they can be easily found and their availability for group work can be determined. The system should allow collaborators to initiate contact with minimal effort.

3 Problem Report The following form (adapted from [3]) can be used by an evaluator to report a groupware usability problem: The heuristic violated: Provide the means for intentional and appropriate verbal communication Provide the means for intentional and appropriate gestural communication Provide consequential communication of an individual s embodiment Provide consequential communication of shared artifacts Provide protection Manage the transition between tightly and loosely-coupled collaboration Support people with the coordination of their actions Facilitate finding collaborators and establishing contact Other Description of usability problem: Severity rating: Major obstacle to effective collaboration Minor obstacle; people can work around it Solution (optional):

4 Example Notification Collage is a groupware system where members of a community can post media elements such as Sticky Notes and Photo Shows onto a real-time collaborative surface that all members can see. For this example, we will only be looking at the NC Sketch media element, which allows people to simultaneously draw on a shared canvas (figure below). We will not be conducting a full heuristic evaluation of NC Sketch since most of the process is the same as heuristic evaluation of single-user applications. Instead, we will concentrate on finding usability problems using the groupware heuristics. Example problem reports are on the next page. Exercise /Homework Find at least 3 more usability problems using the groupware heuristics before next class (so that Phuoc can use them in his Results Synthesis presentation). Instructions for downloading and connecting to the Notification Collage will be posted later.

5 Example Problem Reports for NC Sketch Problem Report #1 The heuristic violated: Provide the means for intentional and appropriate verbal communication Description of usability problem: The chat window isn t resizable so it s hard to follow conversations when lots of people are involved Severity rating: Major obstacle to effective collaboration Problem Report #2 The heuristic violated: Provide the means for intentional and appropriate gestural communication Description of usability problem: When two or more people have the same paint colour, it is impossible to see who is painting what Severity rating: Major obstacle to effective collaboration Problem Report #3 The heuristic violated: Provide consequential communication of an individual s embodiment Description of usability problem: When someone isn t drawing, can t tell if they have stopped work, are typing into the chat window, or are choosing a control Severity rating: Minor obstacle; people can work around it Problem Report #4 The heuristic violated: Provide protection Description of usability problem: No undo function; easy to lose sketch Severity rating: Major obstacle to effective collaboration Problem Report #5 The heuristic violated: Manage the transitions between tightly and loosely-coupled collaboration Description of usability problem: Can t work privately; all work is immediately visible to others Severity rating: Minor obstacle; people can work around it

6 Resources on Heuristic Evaluation of Groupware 1. Baker, K., Greenberg, S. and Gutwin, C. (2001). Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration. I M. R. Little and L. Nigay (Eds.), Engineering for Human- Computer Interaction, LNCS 2254, London, UK: Springer-Verlag. This is the best resource for a quick overview of heuristic evaluation of groupware. This paper presents a set of groupware heuristics for identifying teamwork problems in shared workspace groupware systems. For each of the heuristics, it provides an explanation of how the heuristic was derived from the mechanics of collaboration framework [6] along with some examples of how groupware applications typically realize and support its criteria. This publication was based on research from [3]. 2. Baker, K., Greenberg, S. and Gutwin, C. (2002). Empirical Development of a Heuristic Evaluation Methodology for Shared Workspace Groupware. Proceedings of CSCW 2002, ACM Press, This paper justifi es that heuristic evaluation of groupware [1] is a discount evaluation method. It reports on a study where 27 evaluators used the groupware heuristics to evaluate two groupware applications. It was found that individual evaluators uncover around 20% of the total known teamwork problems and that three to five evaluators will find 40-60% of the total known teamwork problems. These results are similar to Nielsen s [7, 8]. It concludes that the groupware heuristics can be effectively applied in heuristic evaluation of groupware to identify teamwork usability problems in a low-cost and practical way. It also speculates that the results could be even better if evaluators had better training and more practice evaluating groupware. This publication was based on research from [3]. 3. Baker, K. (2002). Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. MSc Thesis, Department of Computer Science, University of Calgary, Calgary, Alberta, Canada. This is the best resource for an in-depth look at heuristic evaluation of groupware. Baker adapts Nielsen s heuristic evaluation methodology [9] for evaluating real-time, shared workspace groupware by developing a set of groupware heuristics based on Gutwin and Greenberg s mechanics of collaboration framework [6]. He also revises another set of groupware heuristics based on the Locales Framework [4]. To demonstrate that the adapted heuristic evaluation for groupware remains a discount usability technique, he shows that individuals who are not necessarily experts in groupware can learn and apply the heuristics to identify groupware usability problems in a study replicating Nielsen s original work [7, 8]. While inspectors were able to use the mechanics of collaboration heuristics with minimal expertise and training, the Locales Framework heuristics were found to be less successful in that they were more difficult to understand and thus more difficult to effectively use. [1, 2] are publications from this research. 4. Greenberg, S., Fitzpatrick, G., Gutwin, C. and Kaplan, S. (2000). Adapting the Locales Framework for Heuristic Evaluation of Groupware. Australian Journal of Information Systems (AJIS), 7(2), This paper describes a set of groupware heuristics based on the Locales framework [5]. It reports on a heuristic evaluation done on a commercial groupware system using these heuristics. The authors were able to identify groupware usability problems, but suggest that these groupware heuristics may require a greater level of evaluator training and experienc e (compared with the conventional heuristics, which tend to be easy to learn and apply) to use effectively. They also suggest that these heuristics are best used for evaluating general groupware environments rather than specific groupware applications.

7 Resources on Groupware Theory 5. Fitzpatrick, G. (2002). The Locales Framework: making social thinking accessible for software practitioners?. In C. Floyd, Y. Dittrich, and R. Klischewski (Eds.), Social Thinking - Software Practice, Cambridge, MA: MIT Press. This paper describes the Locales Framework, a sociological theory of how people interact, collaborate and perform work in a complex social environment. It includes discussion on the strengths and weaknesses of this framework. 6. Gutwin, C. and Greenberg, S. (2000). The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. IEEE 9 th International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE 00). June 14-16, held at NIST, Gaithersburg, MD USA. Awarded Best Paper of the workshop. This paper suggests that a groupware system must support both taskwork and teamwork to be usable. It further breaks teamwork down into two areas: the social and affective elements that make up group dynamics, and the mechanics of collaboration. It identifies and describes seven activities that comprise the mechanics of collaboration and proposes evaluating groupware for the mechanics of collaboration by examining if a group can perform them effectively, efficiently, and pleasantly. It illustrates ways existing discount usability evaluation methodologies can incorporate this framework to uncover groupware usability problems. Resources on Heuristic Evaluation 7. Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of CHI 1990, ACM Press, This paper reports on four experiments run to evaluate how practical heuristic evaluation is. It was found that while individual evaluators only find 20-50% of the usability problems in the interfaces they evaluate, three to five evaluators can be expected to find at least two-thirds of the usability problems. 8. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings of CHI 1992, ACM Press, This paper extends previous work [7] evaluating how practical heuristic evaluation is. It suggests that usability specialists are better than non-specialists at performing heuristic evaluation, and usability specialists with expertise in the kind of interface being developed perform even better (so fewer of them can be used). It also discusses the types of usability problems that tend to be found and missed by evaluators. 9. Nielsen, J. (1993). Usability Engineering, & 224. Cambridge, MA: Academic Press. This text provides a useful overview of heuristic evaluation and justifies why it is a discount evaluation method. Examples are also provided.