Better Games Through Usability Evaluation and Testing By Sauli Laitinen Gamasutra June 23, 2005

Size: px

Start display at page:

Download "Better Games Through Usability Evaluation and Testing By Sauli Laitinen Gamasutra June 23, 2005"

Annis Russell
5 years ago
Views:

1 1 of 10 7/1/2008 5:12 PM CMP Game Group Presents: Better Games Through Usability Evaluation and Testing By Sauli Laitinen Gamasutra June 23, 2005 URL: No one wants to play gam es that are either frustrating or difficult for the wrong rea sons. The best way to m ake sure that unintended problem s do not hinder enjoying the gam e is to take usability into account in gam e developm ent. This article presents how this can be done and what kind of results to expect. Usability is an integral part of software developm ent and has been so for the past 20 years. For one reason or another, usability has not gained sim ilar popularity in gam e develo pm ent. This, however, is about to change. Ease of use and optim al user experience are alread y im portant in gam es and will becom e even m ore so in the future. What is Usability? Usability is about m axim izing effectiveness, efficiency and satisfaction. This definition originate s from the traditional software industry, but it translate s we ll to gam e de ve lopm e nt. In gam es, usability is about delivering a better and deeper experience with less unne cessary interruptions or challenges that have not been designed by the developers. Why is Usability Important? There are m any reasons why usability is im portant in gam es. For one, playing gam es is voluntary. If the player has to struggle with problem s that m ake playing less fun tha n doing som ething else, then there is nothing to stop the player from switching off the conso le. This is a serious risk as the user experience is very sensitive to usability problem s. Even the sm allest glitch or hiccup in the user interface m ay render an otherwise good gam e into a rathe r annoying experience. For exam ple, if m anaging the inventory in a role playing gam e is not fluent enough or restarting a race in a driving gam e is tedious the player is not lik ely to enjoy playing the gam e. Another reason why usability is im portant in gam es is the com petition. Com petition in the m arket is fierce. The gam ers can choose which gam e to buy from a wide variety of titles; if the controls are not fluent in one soccer gam e, there are five m ore titles left from which to choose. Usability is one of the key factors that m ake the gam e stand out of the crowd. The delicacy of the user experience and heavy com petition actually m ake usability m ore im portant in gam es than it is in other software. There are not too m any word processo rs to choose from, and having fun at work is not usually a top priority. There are also other reasons why ease of use is im portant. One of them is that m odern gam es are large and com plex program s. In even the m ost focused gam es there are tons o f

2 2 of 10 7/1/2008 5:12 PM m enus and ways to interact within the gam e, not to m ention gam es like the Grand Theft Auto series. Usability is im portant when m aking a gam e as easy and intuitive to play as po ssible. Good exam ples of com plex gam es m ade easy are World of Warcraft and Xbox Live gam es. Both have succeeded in m aking traditionally difficult m ultiplayer gam ing easy. Usability is also im portant for the future of gam ing. For gam ing to continue to incre ase its popularity, the ease of setting up gam es and a fluent gam ing experience are of the es sence. This is because newcom ers are not fam iliar with the conventions and com m on pitfalls o f gam ing. For them, learning the peculiarities that old gam ers are already fam iliar with can be too m uch. This Article The focus of this article is not only to tell why usability is im portant in gam es, but also to introduce two usability m ethods and the kind of results they yield. The m ethods are e xpert e valuation and usability te sting. It will be presented how these were applied in the developm ent of Frozenbyte's Shadowgrounds ( e.com ) gam e. Adage C orporation ( ) was responsible for the usability related activities in the project. Before m oving on to the m ethods, the gam e will be presented briefly. Frozenbyte's Shadowgrounds Game Frozenbyte Inc. is an independent gam es studio, based in Helsinki, Finland, founded in year Frozenbyte is currently working on a PC gam e called Shadowgrounds, which is to be released in the third quarter of Shadowgrounds is an action gam e vie we d from the top-down/3rd pe rson pe rspe ctive (se e Figure 1). The m ain fe ature s of the gam e are adre naline-pum ping old-school action, re alistic lighting, de structible e nvironm e nt and upgrade able we aponry. Usability Expert Evaluation In a usability expert evaluation, usability experts review the gam e and search for potential usability problem s. Based on both their knowledge and experience they review the gam e system atically and re port the findings. In a typical expert evaluation three usability experts review the gam e. The experts first evaluate the gam e independently. After that the experts have an evaluation m eeting together. In this m eeting, the usability problem s found are discussed and rated for severity. After that a report is written where the usability proble m s found are liste d syste m atically. For each problem there is a title, severity rank, and a detailed description of the problem. A solution to each problem is also suggested. Figure 1. Intense battles, lightning effects and upgradeable weapons are key elements of Shadowgrounds. This process takes approxim ately a week. Prelim inary results, however, can be given in a few days. After handing in the final report, a m eeting between the usability experts and the gam e developers is arranged. In this m eeting, the findings are discussed in m ore detail, a nd the usability experts can answer questions that com e about from the report. Expert evaluation is a very flexible m ethod, and it can be done at alm ost any point o f gam e developm ent. In the case of Shadowgrounds the expert evaluation was done approxim ately six m onths before the planned deadline of the gam e. In the version evaluated there was one playable level and the basic gam eplay m echanics were im plem ented. Everything was, however, not ready. For exam ple, the destructible environm ent was not fully im plem ented yet,

3 3 of 10 7/1/2008 5:12 PM voice acting was m issing, and m any sm aller bugs were still present. It is also possible to conduct an expert evaluation even in earlier stages of gam e developm ent. For exam ple, m enus and displays can already be evaluated on the basis of paper prototypes, and potential usability problem s affecting gam eplay can be spotted from the design docum ent. The earlier the expert evaluation is done, the easier and m ore cost efficient it is to m ake changes. The depth and scope of an expert evaluation are also easy to change. For exam ple, if there is a desire for constant input from the usability experts to the developm ent process, then conducting several sm aller usability evaluations with less experts and faster reporting m ay be a good idea. Results of the Expert Evaluation In total 135 usability problem s were found in the expert evaluation of Shadowgrounds. Out of these problem s 2 were classified catastrophic, 30 severe and 60 interm ediate. The rem aining 43 problem s were either m inor or cosm etic. The distribution of the problem s is illustrated in Figure 2. Figure 2. Classification of the usability problems found in the expert evaluation. The num ber of usability problem s found m ay seem high, but in reality it is not exceptional by any m eans. Expert evaluation gives feedback to alm ost every area of gam e developm ent. In a typical evaluation, usability problem s are found in m enus, in-gam e displays, contro ls, gam eplay and level design. Because of this, the total num ber of problem s found can be quite high. Here are som e exam ples of usability problem s along with the gam e developers' view on the proble m.

4 4 of 10 7/1/2008 5:12 PM Problem: Rating: No feedback is given if the player cannot Severe pick an item. Description: Som etim es it happens that the player cannot pick up an ite m be cause the re is no room in the inve ntory. If this happens, the user is not given any feedback. This is problem atic as the user m ay not know why s/he cannot pick up the item. It is likely that the user will figure it out eventually, but the confusion and extra effort required are likely to cause frustration. Solution Give the user proper feedback in every situation where the user interacts with the environm ent. If the item cannot be picked up, inform the user about this with a sound and/or textual feedback. Developers' comment This was addressed only vaguely in the design docum ent, and was overlooked by the program m ers at the tim e. It is likely that this feature would have been im plem ented in due tim e, and goes to show that som etim es developers leave m any of the sm aller usability issues to the final stretch of developm ent.

5 5 of 10 7/1/2008 5:12 PM Problem It is not im m ediately obvious whether the upgrades presented next to the weapon are installed or not. Description: The upgrade s available for the we apons look active e ve n be fore purchasing them. This is because they are m arked with a sim ilar color schem e as the weapons next to them, which are active. Rating Interm ediate This is especially problem atic early in the gam e. Because the upgrades look active, the user m ay not understand that this m enu can be used to purchase new upgrade s. Solution Make the upgrades that have not been purchased yet clearly inactive. This can be done for exam ple by using a darker shade of blue to present the upgrades that have not been bought yet. Developers' comment The weapon upgrade feature had not been fully im plem ented by the tim e of the testing, but regardless, this particular problem had not been taken into consideration. Later user-testing showed that this was definitely a problem with gam ers, especially in the beginning of the gam e.

6 6 of 10 7/1/2008 5:12 PM After the expert evaluation som e of the m ost central usability problem s were fixed to prepare the gam e for the usability testing that was scheduled to start two weeks later. For e xam ple, the m ap problem discussed earlier and m inor usability problem s related to the radar were fixed. Usability Testing Usability testing is the m ost fundam ental usability m ethod. In a usability test, the gam e is tested with players that represent the target group of the gam e. Just like in an expe rt evaluation, the goal is to pinpoint the challenges in the gam e that were not intended by the gam e developers. The reason why testing is im portant is that it provides direct and objective inform ation about how real players play the gam e and what are the exact usability problem s that players face when playing the gam e. This data is irreplaceable when developing the gam e and m aking it easy to use. In a typical usability test 3 6 players from each target group of the gam e are recruited to com e and play the gam e in a usability laboratory (se e Figure 3). The playe rs com e to the laboratory one at a tim e and play the selected parts of the gam e for 1 2 hours. Figure 3. A usability laboratory consists of two rooms with a one-way mirror and a sound proof wall in-between. The user and the test instructor are in the laboratory room. The test is recorded in the observing room and the developers can follow test in the nearby meeting room. In the laboratory, there is a test instructor together with the player. The role of the expert is to give tasks to the player and observe playing the gam e. The player is instructed to think aloud while playing. In practice this m e ans that the playe r te lls what s/he is doing while playing the gam e. Som etim es the usability expert m ay interrupt the player and ask que stions about what the player was doing and why. The purpose of these questions is to get in-depth inform ation about the problem s and the reasons behind them. Gam e developers are also

7 7 of 10 7/1/2008 5:12 PM invited to com e and observe the tests. The onscreen action and an overall view of the laboratory are displayed in a m eeting room where the developers can follow the tests and discuss the findings with a usability expert. After the tests, each recording is carefully analyzed to find all the usability problem s and to understand the reasons behind them. Then the final report is created. In the final re port, all the proble m s found are prioritize d and solutions for the proble m s are provide d. Typically usability testing takes 2 3 weeks from the start to delivering the final re port. Initial findings can be provided even earlier, and it is also possible to change the version tested between the players. This m akes it possible to fix the problem s during the testing and thus to verify the changes later on in the sam e test. In the case of Shadowgrounds the usability test was conducted two weeks after the expert evaluation. The focus of the testing was broad, and all the aspects but the m ain m enu of the gam e were tested. The testing was done with a version sim ilar to the version used in the expert evaluation. Like m entioned earlier, som e usability problem s were fixed between the expert evaluation and the test. Next, exam ples of the problem s found will be presente d. Results of the Usability Test In the usability tests 97 usability problem s were found. Finding less problem s in usa bility testing than in expert evaluation is typical: players do not necessarily use every fe ature of the gam e, and som e of the m inor problem s found in the evaluation do not necessarily show in the tests. One of the problem s found was classified catastrophic, 26 severe, 28 interm ediate, and 42 m inor or cosm etic. This is illustrated in Figure 4. Figure 4. Classification of the usability problems found in the expert evaluation. Next, five exam ples of problem s classified as severe are presented. In addition to presenting the problem s and their suggested solutions there is also gam e developer's com m ents after each problem.

8 Gamasutra - Features - "Better Games Through Usability Evaluation and... 8 of 10 Problem The characte r m ove s too slowly. Description Two out of six use rs com m e nte d spontane ously that the characte r m ove s too slowly. The m ove m e nt spe e d was conside re d e spe cially proble m atic in ope n space s. Inside the buildings the spe e d was conside re d to be good. Se e also the surve y re sults. Rating Se ve re Solution C onside r im ple m e nting a running option. Developers' comment This could be one of the bigge st issue s in gam e s - the m ove m e nt spe e d of the units/characte rs. In our case, the proble m was m ostly outdoors, which would sugge st that the outdoor are as we re not inte re sting by the m se lve s, and the playe r wante d to ge t to the ne x t inte re sting are a (building). It will be inte re sting to se e if this is the case, whe n the final gam e is te ste d 7/1/2008 5:12 PM

9 9 of 10 7/1/2008 5:12 PM Problem Finding doors is difficult. Description The use rs had difficultie s in finding doors, and it happened often that the user thought that either a window or som e othe r structure was a door. Rating Severe Solution If the door is not hidden on purpose m ark it clearly. Sym bols and/or light e ffe cts should be conside re d. C hange the shape of the windows and the buildings' support structures so that they are not m istaken as doors. Developers' comment This is a tough problem, and hard to predict in the design phase. Additional lights could be adde d, and the buildings could be re m ode le d, but this takes a lot of effort. In the end, this problem seem ed big enough to warrant changes, and each door was scrutinized individually to m ake sure it worked alright.

10 10 of 10 7/1/2008 5:12 PM Benefits of Usability Expert Evaluation and Testing Usability expert evaluation and testing are system atic m ethodologies that provide inform ation to support gam e developm ent. Together they provide both the experts' view on usability and experim ental data of usability problem s in the gam e. The usefulness of the m ethods is illustrated by the following quotes from Joel Kinnunen, the developm ent director of Froze nbyte: On expert evaluation: "Expert evaluation is a fast and effective way to check the usability of a game. In our case, the results arrived in a couple of weeks, and they helped us solve some major design issues. We were also able to fix numerous smaller usability problems, and avoid a couple of potential pitfalls in designing and implementing new features. On usability testing: "Usability testing provided us with a new perspective on the game. It is difficult to know how the game is played without testing it with the real users - gamers are not predictable, especially as it comes to navigating a given level. In hindsight, I wish more of the development team could have been present at the tests, so the endless amount of choices the player can make would be more clear to everyone on the team. Level designers would do well to study the various player behaviors." On usability engineering: "As a whole the usability expert evaluation and testing was a very positive endeavour. Frozenbyte will continue to use usability testing for its future games - starting right from the design document and lasting throughout the entire production phase. The impact of a smooth gaming experience cannot be overestimated." One of the fundam ental strengths of the expert evaluation and the testing is that the y provide com prehensive and prioritized lists of problem s in the gam e. This list, together with the suggested solutions, is a handy tool in focusing the developm ent work especially when tim e and/or resources are lim ited. Usability expert evaluation and testing also have other benefits, especially if the usability work is done by usability specialists external to the developm ent team. They will save both tim e and effort of the gam e developers by organizing, conducting and reporting the us ability work. External usability professionals are also likely to have fresh views on the des ign issues. Acknowledgements I would like to thank Frozenbyte for com m enting the results and giving the perm ission to publish them. Many views on this article are based on the pioneering work done by the usability e ngine e rs at Microsoft. Se e gsuse rre se arch.com /publications/ for m ore article s on the topic. Copyright 2004 CMP Media Inc. All rights reserved.

Better Games Through Usability Evaluation and Testing

1 of 8 3/31/2010 2:00 PM Better Games Through Usability Evaluation and Testing By Sauli Laitinen No one wants to play games that are either frustrating or difficult for the wrong reasons. The best way