MAPI - MUSiC Assisted Process Improvement

Transcription

1 MAPI - MUSiC Assisted Process Improvement Jurek Kirakowski, Human Factors Research Group, University College Cork, Ireland Nigel Bevan National Physical Laboratory, Teddington, Middx. TW11 0LW, UK Abstract The MAPI ÒMUSiC Assisted Process ImprovementÓ project in the ESPRIT Human Comfort and Security area uses methods for usability measurement derived from the successful Framework III MUSiC project ÒMeasuring the Usability of Systems in ContextÓ. These methods implement the principles of the draft ISO standard 9241, part 11 which defines usability as: Òthe extent to which a product can be used to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of useó. The service providers of MAPI have identified a number of different configurations in which MUSiC methods can be delivered in order to satisfy the industrial objectives of the MAPI industrial partners. The objectives of the MAPI project are to show that MUSiC methods can assist companies to improve their processes, and to highlight ways in which the current MUSiC tools need to be developed and delivered in order to be cost-effective. Introduction The MAPI project is a set of trial applications of the use of MUSiC methods for usability evaluation. The acronym MAPI stands for MUSiC Assisted Process Improvement. MUSiC is a set of methods developed in ESPRIT for Measuring the Usability of Systems in Context. The MUSiC project ran from ; MAPI began in the autumn of 1995 and will last approximately 15 months. MUSiC Methods MUSiC contributed to the development of the draft ISO standard 9241, part 11 which defines usability as: Òthe extent to which a product can be used to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of useó. The MUSiC project developed methods for measuring usability based on this definition. A major benefit of using MUSiC methods for usability measurement is that they provide a means of specifying usability goals and evaluating whether they have been achieved. The goals are expressed in terms of the purpose of business systems which is to enable users to achieve tasks effectively, efficiently and with satisfaction. MUSiC provides methods for measuring effectiveness and efficiency: _ whether (or what proportion of) typical users can correctly complete the task

2 _ the efficiency (productivity) of typical users _ how the efficiency of typical users compares with an expert user: this gives an indication of the point on the learning curve and may highlight a need to improve the user interface or to provide more training; _ the proportion of the time for which a user is productive: this may also highlight a need to improve the user interface or to provide more training. But following the ISO definition, it can be seen that it is also important to measure user satisfaction, particularly as poor satisfaction is likely to lead to deterioration in performance, as well as low job satisfaction, which in turn leads to rapid staff turnover, absenteeism, and a host of other consequences of poor staff morale. A graphic recent account of the sometimes dire consequences of neglecting end-user satisfaction is given by Ellen Bravo (1993). Satisfaction can be measured with the SUMI questionnaire developed within the MUSiC project. In addition, measures of cognitive workload recommended by MUSiC can be used to assess mental effort. These may be useful indicators of situations where users have to expend excessive mental effort to achieve acceptable performance, and are particularly important in safety-critical applications. Usability Context Analysis Guide Unless the evaluation can take place in conditions of actual use, it is necessary to decide which attributes of the actual or intended context of use are to be represented in the evaluation. When specifying or evaluating usability it is important that the context selected is representative of the important aspects of the actual or intended context of use. The Usability Context Analysis Guide provides a systematic method for describing the context of use and specifying the context to be used in an evaluation, and is a fundamental part of the MUSiC method. To get valid and reliable results, representative users should perform typical tasks in representative environments. The users and the tasks they perform are selected as a result of a context study assisted by the Usability Context Analysis Guide (Thomas and Bevan, 1995). Performance Measurement Method The Performance Measurement Method was developed at NPL as part of MUSiC. Observing how users interact with a system gives valuable information about usability, unobtainable by other means. Analysis of what is observed, with the help of a video recording, provides a highly effective means of evaluating usability. The method is fully documented in the Performance Measurement Handbook, and is supported by a software tool (DRUM) which greatly speeds up analysis of the video, and helps manage the evaluation (see Macleod and Rengger, 1993, for a description of DRUM). The Performance Measurement Method gives reliable measures of the effectiveness and efficiency of system use, by evaluating the extent to which specific task goals are achieved, and the times taken to achieve those goals. It also gives measures of time spent unproductively (for example, overcoming problems and seeking help), plus valuable data about the source of such difficulties. These measures enable comparison of prototypes of alternative designs with earlier versions of a system, for example, or with competing products. The -2

3 diagnostic information helps identify where specific problems are encountered and where improvements need to be made. Video-assisted usability analysis DRUM, the Diagnostic Recorder for Usability Measurement, is a software tool originally developed at NPL within the MUSiC Project. It supports the Performance Measurement Method, and also has wider applicability. Video clips of end-users working with a system provide convincing evidence for designers and developers of the usability of their system, and of specific problems. However, analysis is required to convey this information effectively, as an unanalysed interaction log contains too much low-level detail. Video analysis has previously been very time-consuming, with analysis times of ten hours for every hour of video being typical. It can now be performed much more quickly using DRUM Ð one or two hours to analyse one hour of video. A first pass analysis can be performed in real time during recording. DRUM supports the management and analysis of usability evaluations, including the derivation of usability metrics, and the identification of evaluator-defined critical incidents for design feedback. User satisfaction Ð SUMI Measures of satisfaction describe the perceived usability of the overall system by its users and the acceptability of the system to the people who use it. SUMI, the Software Usability Measurement Inventory, was developed by the Human Factors Research Group at University College Cork, to measure user satisfaction, and hence assess user perceived software quality. SUMI is an internationally standardised 50- item questionnaire, available in several languages. It takes a maximum of 10 minutes to complete and needs only small user sample sizes (SUMI is described in Kirakowski and Corbett, 1993). The results that SUMI provides are based on an extensive standardisation database built from data on a full range of software products such as word processors, spreadsheets, CAD packages, communications programs etc. SUMI results have been shown to be reliable, and to discriminate between different kinds of software products in a valid manner. In particular, the SUMI database allows evaluation of a product against what is considered to be the prevailing market norm, and the statistical background to SUMI enables the analyst to pinpoint quite precisely the relative standing of the product being assessed to the market as a whole. SUMI results are analysed by a computer program called SUMISCO, which produces a standard SUMI report that can be easily imported into a wider product evaluation report. SUMISCO relies on a summarised statistical database that is revised at least once a year to keep up with changes in the market. SUMISCO provides an overall assessment and a usability profile which breaks the overall assessment down into 5 sub-scales: Affect, Efficiency, Helpfulness, Control, and Learnability. These scales have been derived by an iterative process of factor analysis of large databases, and present a view of subjective usability for which there is a high level of solid empirical support. In addition, Item Consensual Analysis (ICA) can be used to list those items on which the software being rated was significantly better or worse than average. ICA is a technique developed specifically for the SUMI questionnaire, and it too has been proved in numerous industrial trials. ICA enables the analyst to pinpoint -3

4 topics about the software product that are considered to be above or below the market standard. It can be used in conjunction with an interview method to focus in to those features of the product that are giving rise to user dissatisfaction. The Human Factors Research Group, are, in the MAPI project, developing a number of different ways in which SUMI may be administered (by , as a Web site, on a standalone computer) as well as a number of different ÔflavoursÕ of SUMI, for instance and most importantly, a multi-media ÔedutainmentÕ products version, and a version specifically for the evaluation of public-access terminals. These developments have been made in response to the demands of the industrial partners of MAPI. Cognitive workload Cognitive workload relates to the mental effort required to perform tasks. The Technical University of Delft has developed a Guide to Cognitive Workload Measurement as part of the MUSiC project (Houwing, Weithoff and Arnold, 1993). European health and safety regulations implementing the EC Display Screen Directive (EC, 1990) specifically require evaluation to avoid problems of mental stress. Two specific measures for evaluation of Cognitive Workload, the SMEQ questionnaire and the NASA TLX questionnaire are recommended by this report. More detailed examination of cognitive workload can be achieved by taking physiological measures of change in heart rate but these are not always practical or necessary. Mental stress can become a problem specifically in situations where the work of the end users is demand-paced (such as in ticket sales or process control applications). Bravo (1993) reports an end user in this kind of application who was told to Ôtake 10 minutes to pull (herself) togetherõ. The end user reportedly suffered a nervous breakdown and was in therapy for 8 months. However, even in self-paced work, cognitive workload must be monitored as an increase in workload will indicate a nexus point in the application that may benefit from re-design in order to simplify or de-intensify the need for decision making on the part of the end user. MAPI project The industrial partners in MAPI have identified processes and products that are critical for them with respect to end-user quality. These partners are: _ LloydÕs Bank PLC (lead partner), UK, _ Sogei SpA, Rome, _ State Museums, Berlin, _ Cyco Ltd., the Netherlands, _ Ericsson, Stockholm, Sweden, _ AMPS, Stockholm, Sweden _ Inland Revenue, GB. These partners are in various degrees of sophistication with regard to the employment of usability technology, but all are agreed that the adoption of technology of the kind developed during the MUSiC project is of potential benefit to them and indeed to the future of their businesses. Each organisation has nominated a trial application so that seen from an overview, the MAPI project targets a number of different sectors within the software industry, requiring usability engineering at different parts of the software lifecycle. -4

5 Each industrial partner is associated with one or more organisations which function as service providers in the project. The service providers are: _ National Physical Laboratory, Teddington, UK, _ Human Factors Research Group, Cork, Ireland, _ WIT-Lab, Delft, the Netherlands, _ Data Management SpA, Milan, Italy, _ NOMOS Management AB, Stockholm, Sweden. The first three service providers have full ownership rights to the MUSiC project results being exploited in MAPI; Data Management was an industrial partner in MUSiC and is currently playing a leading role in disseminating MUSiC results in the FINSIEL group of companies in Italy, and NOMOS Management was the first organisation to develop links with the MUSiC project group in order to specifically exploit the project results in Scandinavia. NOMOS is also since May 1994 a Microsoft Solution Provider and Usability Partner. Project objectives in the current industrial context The MAPI project came about because of the need of the industrial partners for an effective integration of quality of use technology into their mainstream practices and products, and the need of the MUSiC service providers to identify modes of delivery of MUSiC technology that are industrially effective and costefficient. The project objectives are to: _ Change the evaluation culture of the companies concerned, so that the companies take up MUSiC as a long term investment _ Demonstrate the uptake of MUSiC technology successfully by companies involved in a broad range of software developments, thus showing the industrial applicability of MUSiC to other companies in the sector _ Demonstrate the uptake of MUSiC in many parts of a development lifecycle, thus showing the broad applicability of MUSiC tools and techniques. The MAPI project is therefore seen as an initial move leading to a strategic leveraging action to make an effective contribution to European practices with regard to quality of use engineering. MUSiC technology allows organisations to quantify quality of use of products, and it enables them to specify targets and identify work that needs to be done to achieve these targets. This technology is applicable at all stages of the life cycle of an IT product: not only in those time critical phases where the product is being created, but also in prior market positioning and definitional phases and in post-release phases to support activities such as marketing, product support, and product enhancement. MUSiC methods and associated technology stand as the leading edge technology in European quality of use engineering (cf Bevan and Macleod, 1994, Preece et al., 1994, Kirakowski, 1995, Redmond-Pyle and Moore, 1995, and Bšsser, in press, to name but a few of the recent books and articles in which aspects of MUSiC are reviewed). In comparison with the state of the industry in the USA, it can be seen that European technology is leading the market. Indeed, all the major international IT companies have now purchased and use parts of the MUSiC toolset, both in the USA and in Europe. However, it must be said that the way MUSiC technology is -5

6 supplied at present is craft-like. For MUSiC service providers, the way of providing MUSiC must be improved, and the markets for MUSiC technology must be opened by means of trial applications and leveraging actions. Outside of MUSiC the current international situation is characterised by two major trends: 1. informal assessment and diagnostic activities that rely on a craft-like approach handled by individuals with often considerable personal expertise in the field but few standardised tools or procedures (cf for instance the excellent account by Neilsen, in his book Usability Engineering, 1993) 2. approaches that are qualitative and are therefore difficult to place in a management perspective with regard to cost justification and process improvement (cf the summary of the state of the art by Bias and Mayhew, 1994, in their aptly-named book Cost Justifying Usability) We see many companies which are starting to consider the incorporation of user-based quality of use assessment into their processes, but it is clear that for most, these are as yet uncharted waters. In the more commercial end of the market place, we see that many popular magazines now incorporate usability reviews of software. The methods used by some are informal, to say the least, but an evaluation service provider to an influential group of magazines has recently purchased MUSiC tools and training and now is seen to regularly publish usability profiles based on this technology in PC User magazine. Delivery of usability technology The major phases of each sub-project depend on the particular circumstances of each of the industrial partners, and the delivery packages are explained in greater detail below. However, overall, each delivery package corresponds to the following stages: _ User training and problem definition _ Implementation of MUSiC in Trial Application _ Assessment of results and definition of future company strategy. These stages are now explained in greater detail. MUSiC partners already have an informal network which allows one partner to be the co-ordinating Ôindustry interfaceõ with a company, but which nevertheless enables other MUSiC partners to contribute training or consultancy effort as needed. The key element of the service delivery is to provide MUSiC products and services in a seamless and integrated manner, allowing the participating industrial partners to feel that they ÔownÕ those parts of MUSiC they have chosen to integrate. A phenomenon which happened early in the project was that many of the industrial partners began quite spontaneously to generate their own, internal ÔUsability Evaluation GuidelinesÕ. Although there appears to be much commonality between these guidelines, what is most interesting is the range of diversity between them as each organisation reflects on the most appropriate way to assimilate the technology on offer to improve their internal processes. There is quite clearly no single prescriptive Ôgolden pathõ to the attainment of this objective just as there is no single software lifecycle model to which an organisation must subscribe if they wish to incorporate end-user involvement into their process. MAPI therefore seeks to deliver MUSiC technology through a number of different Ôtemplate packagesõ. These are more in the spirit of hypotheses about -6

7 what industrial partners want MUSiC to deliver than tested routes to application. As MAPI progresses, these template packages are subjected to a process of refinement and evolution. These packages, defined in consultation the industrial partners, are as follows: _ Whole Life Cycle: This package refers to applying MUSiC methods to the whole developmental lifecycle. After the initial activities of Process Analysis, Context of Use Analysis and Stakeholder identification, the Usability Test Plan is drawn up, and a Cost Estimate is issued. Thereafter the precise MUSiC measurements depend on the companiesõ activities within their lifecycle as specified in the Usability Test Plan. At the end of the process a Summative Evaluation report is compiled. Other elements of the MUSiC process may or may not be included in the Usability Test Plan, depending on the precise nature of the kind of process involved and the interactions between the identified stakeholders. _ Partial Life Cycle: This package refers to applying MUSiC methods to part of a lifecycle (although the MUSiC service supplier should be brought in at the start of the development process, their activity will be limited to certain phases as determined by the needs of the client). Process Analysis is therefore crucial, as is Context of Use Analysis and Stakeholder identification. The Usability Test Plan and Cost Estimates are issued. They are likely to be of smaller duration and cost less than if the service were for a Whole Life Cycle. A Baseline Usability measure may need to be made, in order to set targets against which improvement is to be measured. The overall evaluation will be Cumulative over the Baseline rather than Summative. _ Version Improvement: This package is applicable when a product exists as a previous release and a user base is established. It comprises of all the activities of the Whole Lifecycle package, as well as Goal Elicitation, Baseline Usability Measurement (on the basis of the current usage patterns; this may also form part of the Goal Elicitation procedure). Usability Bugs need to be identified for the Baseline version, and the evaluation is reported Cumulatively on the basis of improvement over the Baseline. Progressive Improvement reporting is relevant because the whole concept of version improvement implies re-working and adapting work practices if necessary. _ Alpha Test: This is the most traditional view of usability: a working alpha test version is in existence, and one or two quick improvements are envisaged. The Goal Elicitation and StakeholdersÕ meetings usually suffice to generate sufficient material for a Usability Test Plan and Cost Estimate since the Process is usually fixed by the time alpha testing is envisaged. Otherwise the package is similar to Version Improvement, only that the scale and cost of testing should be less than if a full Version improvement was requested. Context of Use is important, because the user population may not have been identified by the Alpha Test stage whereas with a released product, a user population exists de facto. Usability bugs of the first alpha version are not usually considered worthy of a formal deliverable, since it is known that usability bugs will exist. They are treated within the Cumulative Evaluation report as part of the engineering process. _ Workflow Analysis: This package takes a product as implemented and used on a regular basis in one or two sites and uses MUSiC tools and methods to analyse the patterns of usage of the product and to present a report on what can be -7

8 improved in the product or what kind of uses the product best lends itself to. Context of Use evaluation is important, the evaluation is Summative, and identification of Usability Bugs is of prime consideration for the company. These packages were identified at the start of the project as an initial hypothesis on which to plan the intended work. The final shape of these delivery packages, and indeed an appreciation of whether all these different delivery packages are commercially viable, is an important end result of the MAPI project. Assessment of results and future company strategy The major question is whether the industrial partners involved are willing to make a long-term investment in incorporating MUSiC technology into their future plans. All the industrial partners have already indicated a likely continuation scenario. Thus one yardstick with which to measure the effectiveness of the MAPI project is to examine how many of the industrial partners and associate partners undertake to carry on with MUSiC. In order to overcome a vague Ôwish listõ approach to assessing this goal, each company proposes to frame quantitative future projections. However, the benefit of the project is not only to the industrial participants directly. In order to benefit other industries in the software market, an assessment must be made of the kinds of delivery packages that have been successful, reasons for their success, and likely improvements to ensure greater effectiveness and cost benefit. The MUSiC tools clearly, in the eyes of the present industry participants, present a useful initial working set. However, the need for additional tools has been noted, and there are moves within the service provider network to furnish such tools in the near future. In addition, a more formal framework for co-operation among MUSiC service providers must be considered, as well as proposals for an accreditation scheme to which other service provider centres will be able to join. In effect, the process of creating such a scheme has already been started by a wider group of service providers, with the MUSiC partners as the nucleus, acting in concert independently of the MAPI project. Acknowledgements MUSiC and MAPI are partly funded by the European Commission. NPL work on MUSiC was supported by the UK Department of Trade and Industry. Further information Further information regarding the MAPI project may be obtained from Mr K Bray, Project Manager, MAPI, Lloyds Bank PLC, 100 Pall Mall, London, SW1Y 5HP, UK. Further information about the MUSiC project may in the first instance be obtained from either of the authors of this paper. References Bevan N, Macleod M (1994) Usability measurement in context. Behaviour and Information Technology, 13, Bias R G and Mayhew D (1994) Cost-justifying usability. Boston: Academic Press. -8

9 Bšsser, T., (in press, 1995) Measures and Methods for Quality of Use, Springer Verlag, Berlin. Bravo, E, (1993) The hazards of leaving out the users. In: D. Schuler and A Namikoka, (Eds), Participatory Design: Principles and Practices, Earlbaum, Hillsdale NJ. EC (1990) Minimum Safety and Health Requirements for Work With Display Screen Equipment Directive (90/270/EEC) Official Journal of the European Communities No L 156, 21/6/90. Houwing E.M., Wiethoff M., and Arnold A.G. (1993). Introduction to cognitive workload measurement. Delft University of Technology, Laboratory for Work & Interaction Technology (WIT Lab). ISO (1995) International Standards Organisation DIS : Ergonomic requirements for office work with visual display terminals (VDTs), Part 11: Guidance on usability. Kirakowski J & Corbett M, (1993) SUMI: the Software Usability Measurement Inventory, B J Ed Tech, Kirakowski, J (1995) in Pfleger, S (Ed), Human Comfort and Security, Springer Verlag, Berlin. Macleod, M. and Rengger, R. (1993). The development of DRUM: a software tool for video-assisted usability evaluation. In: JL Alty et al. (eds.) People and Computers VIII (Proc. of HCI'93 Conf., Loughborough UK, September 1993). Cambridge: CUP, Nielsen J (1993) Usability Engineering. Academic Press. Preece, J et al. (1994) Human Computer Interaction, Prentice Hall International. Redmond-Pyle, D, and Moore, A, (1995) Graphical User Interface Design and Evaluation: a Practical Process. Prentice Hall International. Thomas C and Bevan N (eds). (1995) Usability Context Analysis: A practical guide, Version 4. National Physical Laboratory, Teddington, UK. -9