CHARM Choosing Human-Computer Interaction (HCI) Appropriate Research Methods

Transcription

1 CHARM Choosing Human-Computer Interaction (HCI) Appropriate Research Methods Introduction The Role of Theory in HCI Development Methods Ethnographic Methods Logging & Automated Metrics Irina Ceaparu, Pankaj Thakkar Department of Computer Science University of Maryland, College Park, MD USA October 2001 Controlled Experimentation Survey Methods Logging & Automated Metrics 1. Introduction "Usability is the measure of the quality of the user experience when interacting with something - whether a Web site, a traditional software application, or any other device the user can operate in some way or another." said Jacob Nielsen in one of its articles. The measuring can be done in many ways, by choosing one or more of the usability methods that are available to the designers/developers of a product. This study focuses on two categories of research methods, logging and automated evaluation, each with its subcategories. 2. Definitions Logging: the collection and recording of information. Server (hit) logs: collections of data about which pages are getting visited on a website and which path people are taking through the website. Client (event) logs: collections of data about user-initiated activities within a web page of a visited website. Proxy logs: collections of data about users action on the web; the proxy mediates between the client browser and the web server and logs all communication between the two. Self-reporting logs: paper-and-pencil journals in which users log their actions and observations while interacting with a product. Journaled session: a user testing situation in which usage data are automatically recorded into logs. Automatic evaluation: measuring the usability of a system automatically, by standard inspections, simulated human interaction sequences, automated capture of user interaction data and user feedback.

2 3. Description of methods Logging can be manual or automated. Example of manual logging: taking and recording a patient s temperature. Example of automated logging: automated weather station. Automated logging involves having the computer collect statistics about the detailed use of a system. Typically, an interface log will contain statistics about: the frequency with which each user has used each feature in the program the frequency with which various events of interest have occurred Statistics showing the frequency of use of commands and other system features can be used to optimize frequently used features and to identify the features that are rarely used or not used. In addition, an analysis on patterns of use can be made using the logging data. Statistics showing the frequency of various events, such as error situations and the use of online help, can be used to improve the usability of future releases of the system. Usability studies on web-based applications make use of two types of logging techniques: server logs and client-side logs. These logs offer useful data about the users-website interaction. The data may be studied to generate inferences about the website design, to test prototypes over time and to test theoretical hypotheses about the effects of different design variables on web users' behavior. Server logs provide a high level overview of the pages that a site user had visited. They contain information about which document was requested, at what time, whether it was successfully delivered, and the address that the request came from. There are four types of server log files: access logs - they provide the IP address of the computer making the request for a document, the timestamp, the user's request (e.g. whether or not the user downloaded a document or an image) agent logs - they provide data on the browser and the operating system of the accessing user error logs - they contain information on specific events such as "file not found", stopped/interrupted transmissions, configuration errors referrer logs - provide information on what Web pages, from both the site itself and other sites, contain links to documents stored on the server Client logs can identify problems or difficulties that a user is experiencing with the interface. They contain information about user-initiated action performed while viewing a web page, such as scrolling, clicking, filling out a form, path taking, etc. Client-side logging tools are predominantly used as a mean of collecting data in a controlled study environment, rather than in commercial applications. Some types of automatic evaluation do not involve users at all (as in an HTML-checking tool that tests for cross-platform compatibility or a dialog-layout tool that verifies spacing and alignment properties). A computer can also simulate human interaction sequences (mouse clicks, text entry) to test a product's robustness.

3 Other forms of automatic evaluation include using logging tools to capture user interaction data and using online feedback forms to capture user impressions and automatically provide summaries of those responses. The evaluation can be done on the transaction log, which contains records of users' actions and the system response to them, or on the full-screen log, which contains a complete session transcript for detailed analysis. The evaluation requires a data retrieval from the logs, which can be done by special programs designed to extract the data or by built-in report facilities. The next step in evaluating the data is the actual analysis, which can be done by statistical software packages, report generators or playbacks of the recorded data. Proxy logs are a logging technique that is easy to deploy for any web site and is compatible with a number of operating systems and browsers. Proxy-based logging is done on an intermediate computer, and avoids many of the deployment problems faced by client-side and server-side logging. A good example of a proxy log is the WebQuilt project [9]. Journaled sessions data can later be analyzed to determine a user's pattern of behavior, find trouble spots, examine learning times, and corroborate observations in other media. This approach may help in automating the analysis of large volumes of usage data and helps in gathering data from remote sites. Journaled sessions allow one to perform usability evaluation across long distances and without much overhead. Once the code to journalize user's actions is in place, it is relatively inexpensive to distribute the test disk to a large number of participants. Self-reporting logs are best used when there is no time or there are no resources to provide the interactive package required for journaled sessions, or when the level of detail provided by journaled sessions is not needed. For example, one might want just general perceptions and observations from a broad section of users. 4. Advantages/Disadvantages Advantages: Disadvantages: automated evaluation enables more accurate measurements than the manual recording of data logging reduces the time spent on gathering the data and increases the number of users from which the data is gathered logging software can be run continuously automated evaluation is fast and systematic recordings can be reviewed/playbacked as many times as needed direct observation may not be available in the case of automated evaluation automated methods can not provide subjective data about the user's experience and behavior log files can become too voluminous most of the methods require integration of software/hardware components

4 5. Examples Tools 1) NIST Web Metrics ( : Web Static Analyzer Tool (WebSAT) - checks web page HTML against typical usability guidelines Web Category Analysis Tool (WebCAT) - lets the usability engineer construct and conduct a web category analysis Web Variable Instrumenter Program (WebVIP) - instruments a website to capture a log of user interaction Framework for Logging Usability Data (FLUD) - a file format and parser for representation of user interaction logs VisVIP Tool - produces a 3D visualization of user navigation paths through a website TreeDec - adds navigation aids to the pages of a website Example of WebSAT session: Single page URL Whole Site URL** for results **If the HTML document for the homepage is not named "index.html", please include its name in the URL path (i.e., Select Desired Analysis Categories (Default is all) Accessibility Form use Performance Maintainability Navigation Readability Please Note: The rules that are used in the above categories do not form a comprehensive set of guidelines; however, they are a sample set of typical rules to demonstrate the feasibility (and limitations) of an automatic checker. Submit Reset After the user hits the submit button, a set of analysis results is displayed in the form of tables like this one: Navigation Rule A NO if the page does not contain at least one link. A NO if link colors are not default. Stat Line Index

5 A NO if the average # of words used in a link is less than 2 or greater than 5. # of links that do not have a line break before or after the link. # of links not found 1 90 # of links that open a new browser window , 31, 66, 68, 69, 71, 75, 79, 82, 83, 84, 88 2) NetTracker ( a log file analysis software that provides detailed web site traffic reporting and web data mining 3) NetIntellect ( bit Log Analysis Tool that generates reports (Tables & Graphs) that show Statistical, Geographic and Marketing trends in the performance and usage of any Web site. 4) WebTracker ( A web service that provides graphical logfile analysis. 5) HTTP-Analyze ( - A log analyzer for web servers 6) WET ( - Web Eventlogging tool for the client side. 7) Bobby ( Web-based tool offered by CAST that analyzes web pages for their accessibility to people with disabilities. 8) W3C HTML validation service ( - a free service that checks documents like HTML and XHTML for conformance to W3C Recommendations and other standards. Example session: URI: Last Modified: Fri Oct 26 20:21: Server: Apache/1.3.6 (Unix) PHP/ Content Length: Detected Character us-ascii Encoding: Select Character (detect automatically) Encoding: Document Type: XHTML 1.0 Transitional Root Namespace: Select Doctype: (detect automatically)

6 Options: Show Source Outline Parse Tree...no attributes Revalidate Below are the results of checking this document for XML well-formedness and validity. No errors found! * Congratulations, this document validates as XHTML 1.0 Transitional! Experiments and Studies 1) WebTrends conducted an analysis study [6] using the LogAnalyzer software package. Here are some of the results: The Visits graph displays the overall number of visits to a Web site. This graph and the following table identify how often ads were viewed.

7 This table shows the total number of hits for the site, how many were successful, how many failed, and it calculates the percentage of hits that failed. Technical Statistics and Analysis Total Hits 172,062 Successful Hits 170,698 Failed Hits 1,364 Failed Hits as Percent 0.79% Redirected Hits 291 Redirected Hits as Percent 0.16% Cached Hits 291 Cached Hits as Percent 0.16% This table identifies the most popular browsers used by visitors to your site. Top Browsers Browser Hits % of Total Hits Visits 1 Microsoft Internet Explorer 144, % 15,671 2 Netscape Navigator 16, % 1,792 3 Others 2, % Netscape Compatible % 33 Total For Browsers Above 163, % 17,707 2) S. Trewin at the University of Edinburgh conducted a study [7] about input device manipulation difficulties. This paper describes the pilot study for an experiment intended to gather detailed information about input errors made with keyboards and mice. This work is a step towards provision of dynamic, automatic support for the configuration of systems and applications to suit individual users. A detailed log of keyboard and mouse input was kept in order to analyze performance and errors. Here are some screenshots from the study with examples of data analysis from the logs: 3) M. Good's study "The Use of Logging Data in the Design of a New Text Editor" [8] examines how one technique, the use of logging data, was used throughout the design of a new text editor which is measurably easy to learn and easy to use. Logging data was used in four areas: keyboard design, the initial design of the

8 editor's command set, refinements made later in the design cycle, and the construction of a system performance benchmark. 4) Melody Ivory and Marti Hearst research has 2 studies relevant to the subject of automated metrics and analysis. One of the studies [3] discusses the taxonomy for automated usability analysis and illustrates it with an extensive survey of evaluation methods. The study surveyed 58 usability evaluation methods applied to WIMP (Windows, Icons, Pointer, and Mouse) interfaces, and 50 methods applied to Web UIs. Of these 108 methods, only 31 apply to both Web and WIMP UIs. The second study [4] is a quantitative analysis of a large collection of expert-rated web sites which reveals that page-level metrics can accurately predict if a site will be highly rated. The analysis also provides empirical evidence that important metrics, including page composition, page formatting, and overall page characteristics, differ among web site categories such as education, community, living, and finance. Metric Word Count Body Text % Emphasized Body Text % Text Positioning Count Text Cluster Count Link Count Page Size Graphic % Graphics Count Color Count Font Count Table: Web page metrics computed for this study. Description Total words on a page Percentage of words that are body vs. display text (i.e., headers) Portion of body text that is emphasized (e.g., bold, capitalized or near!'s) Changes in text position from flush left Text areas highlighted with color, bordered regions, rules or lists Total links on a page Total bytes for the page as well as elements graphics and stylesheets Percentage of page bytes that are for graphics Total graphics on a page (not including graphics specified in scripts, applets and objects) Total colors employed Total fonts employed (i.e., face + size + bold + italic) Others Finally, we provide a link to a free automated web testing web site that provides test tools, test plans, consulting and resources for testing and usability, and also a list of previous studies /web_tests/usability/white_papers.htm 6. Recommendation

9 1. Follow these steps in running a logging and evaluation process: - identify users and tasks - prepare the equipment and the materials - establish scenarios - record, compile, analyze, evaluate data 2. The inherent imperfections in collecting and analyzing data may be overcome by triangulating server logging, client logging and usability testing thus increasing the effectiveness. 7. References 1. Online Guide to Usability Resources 2. Hom, J. - The Usability Methods Toolbox 3. Ivory, M. - State of the Art in Automated Usability Evaluation of User Interfaces /survey.html 4. Ivory, M.; Hearst, M. - Empirically Validated Web Page Design Metrics 5. Burton, M.; Walther, J. - The Value of Web Log Data in Use-Based Design and Testing 6. WebTrends LogAnalyzer 7. Trewin, S. - A Study of Input Device Manipulation Difficulties /p15-trewin.pdf 8. Good, M. - "The Use of Logging Data in the Design of a New Text Editor" 9.Hong, J.; Heer, J.; Waterson, S.; Landay, J. - WebQuilt: A Proxy-based Approach to Remote Web Usability Testing Last updated October 28, 2001