Understanding Web Application Test Assertion Failures

Transcription

1 Understanding Web Application Test Assertion Failures by Sheldon Sequeira B.A.Sc., The University of British Columbia, 2012 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in The Faculty of Graduate and Postdoctoral Studies (Electrical & Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) October 2014 Sheldon Sequeira 2014

2 Abstract Developers often write test cases that assert the behaviour of a web application from an end-user s perspective. However, when such test cases fail, it is di cult to relate the assertion failure to the faulty line of code. The challenges mainly stem from the existing disconnect between front-end test cases that assert the DOM and the application s underlying JavaScript code. We propose an automated technique to help developers localize the fault related to a test failure. Through a combination of selective code instrumentation and dynamic backward slicing, our technique bridges the gap between test cases and program code. Through an interactive visualization, our approach, implemented in a tool called Camellia, allows developers to easily understand the dynamic behaviour of their application and its relation to the test cases. The results of our controlled experiment show that Camellia improves the fault localization accuracy of developers by a factor of two. Moreover, the implemented approach incurs a low performance overhead. ii

3 Preface Apart from Chapter 2, the work presented in this thesis is under review as a submission to the ACM/IEEE International Conference on Software Engineering (ICSE), I conceived the idea for the project and was responsible for its design and described implementation. The work described in Chapter 2 appears in the proceedings of ICSE 2014, [6]. The paper appeared as part of the research track and received the ACM SIGSOFT Distinguished Paper Award. I was involved in the conception and design of this previous project. In terms of implementation, I was involved in many facets of the project including the instrumentation technique and produced visualization. The section on Capturing Timeouts and XHRs was originally drafted by Alimadadi, S. The controlled experiment described in Chapter 4 was conducted under the approval of the University of British Columbia (UBC) Behavioural Research Ethics Board (BREB) with the certificate number H In addition to conducting the study, I also analyzed the collected data using the statistical analysis package R [13]. iii

4 Table of Contents Abstract ii Preface iii Table of Contents iv List of Tables vii List of Figures viii List of Algorithms ix Acknowledgements x Dedication xi 1 Introduction Background Document Object Model (DOM) JavaScript Testing Web Applications Motivation Running Example Challenges Claims and Contributions Approach: Modelling Web Application Behaviour JavaScript Transformation and Tracing Interposing on DOM Events Capturing Timeouts and XHRs Recording Function Traces Tracking DOM Changes iv

5 Table of Contents 2.2 Capturing a Behavioural Model Episode Nodes Edges Story Visualizing the Captured Model Semantic Zoom Levels Tool Implementation: Clematis Results and E ectiveness of Proposed Model Approach: Test Case Understanding Relating Test Assertions to DOM Elements Contextualizing Test Case Assertions Slicing the JavaScript Code Selective Instrumentation Computing a Backwards Slice Visualizing the Inferred Mappings Semantic Zoom Levels Stepping Through the Slice Tool Implementation Evaluation Experimental Design Experimental Object Participants Task Design Independent Variable (IV) Dependent Variables (DV) Data Analysis Procedure Results Accuracy Duration Qualitative Feedback Performance Overhead Instrumentation Overhead Execution Overhead Dynamic Analysis Overhead Discussion Task Completion Accuracy v

6 Table of Contents 5.2 Task Completion Duration Reducing Variance in User Performance Strengths Improvements Limitations Threats to Validity Related Work Fault localization Program Slicing Behavioural Debugging Visualization for fault localization Conclusion Bibliography Appendices A Experiment Tasks A.1 Task A.2 Task B Information Consent Form C Pre-Questionnaire D Post-Questionnaire vi

7 List of Tables 4.1 Injected faults for the controlled experiment vii

8 List of Figures 1.1 Example event registration technique for web applications Example of a Selenium test case Running example Instrumented JavaScript function declaration Instrumented JavaScript return statement Instrumented JavaScript function calls Three semantic zoom levels in Clematis Relating test case assertions to DOM accesses Instrumented JavaScript code for the running example Processing view of approach Visualization for a test case Box plots of (a) task completion accuracy, (b) task completion duration viii

9 List of Algorithms 1 Selective Instrumentation Backward Slicing ix

10 Acknowledgements I o er my enduring gratitude to my supervisors and my fellow students in the SALT lab at UBC. In particular, I would like to thank Dr. Ali Mesbah and Dr. Karthik Pattabiraman for their unwavering guidance, support, and patience. I am indebted to Dr. Mesbah for inviting me into such a supportive community of researchers. I would like to acknowledge Dr. Matei Ripeanu for acting as chair in my project defence and would like to thank Dr. Mesbah and Dr. Pattabiraman for taking time out of their busy schedules to act as examiners during my defence. A special thank you to the Natural Sciences and Engineering Research Council of Canada (NSERC) for supporting me financially in my studies. I would also like to thank SAP Labs Vancouver for providing me with an environment to cultivate my technical skills during my postgraduate studies. Last but not least, I would like to thank my parents for their support throughout my years of education. I attribute any of my success to their hard-work and dedication. A special thanks to my father for driving me to and from school during obscene hours of the morning and night. x

11 Dedication I dedicate this thesis to my family, for all the sacrifices they have made for me. I also dedicate this work to my friends who have kept me laughing through the years. xi

12 Chapter 1 Introduction Fault localization has been found to be one of the most di cult phases of debugging [30], and has been an active topic of research in the past [5, 9, 19, 35]. The fault localization process involves finding the exact line of code responsible for some abnormal program behaviour. Although testing of modern web applications has received increasing attention in the recent past [8, 23, 29], there has been limited work on what happens after a test reveals an error. After a test fails, the fault localization process much be undertaken by developers in order to identify the fault responsible for the failure. To test their web applications, developers often write test cases that check the application s behaviour from an end-user s perspective using popular frameworks such as Selenium. 1 Such test cases are agnostic of the Java- Script code and operate by simulating a series of user actions followed by assertions on the application s runtime Document Object Model (DOM). As such, they can detect deviations in the expected behaviour as observed on the DOM. However, when a web application test assertion fails, determining the faulty program code responsible for the failure can be a challenging endeavour. The main challenge here is the implicit link between three di erent entities, namely, the test assertion, the DOM elements on which the assertion fails (checked elements), and the faulty JavaScript code responsible for modifying those DOM elements. To understand the root cause of the assertion failure, the developer needs to manually infer a mental model of these hidden links, which can be tedious. Further, unlike in traditional (e.g., Java) applications, there is no useful stack trace produced when a web test case fails as the failure is on the DOM, and not on the application s JavaScript code. This further hinders debugging as the fault usually lies within the application s code, and not in its surfacing DOM. To the best of our knowledge, there 1 1

13 1.1. Background is currently no tool support available to help developers in this test failure understanding and fault localization process. In this thesis, we present an automated technique, called Camellia, to help developers understand the root cause of a test failure. Camellia helps programmers in three ways. First, it automatically links a test assertion failure to the checked DOM elements, and subsequently to the related statements in the JavaScript code. Second, it incorporates a novel dynamic slicing method for JavaScript that reduces the amount of noise encountered by developers when debugging. Our slicing method is based on a selective instrumentation algorithm to reduce the performance overhead and trace size. Third, once a test failure is connected to the relevant slice, our technique visualizes the output to abstract away details and relate the test failure to the corresponding episodes of code execution, DOM mutations, and the computed JavaScript slice. Camellia aids developers in understanding existing test cases by capturing the DOM dependencies of a test case, as well as an application s DOM evolution and JavaScript code execution when running the test case. 1.1 Background Modern web applications are largely event-driven. Their client-side execution is normally initiated in response to a user-action triggered event, a timing event, or the receipt of an asynchronous callback message from the server. A fault in their client-side execution may be detected by a DOMbased test case. Still, the implicit link between the test assertion, the DOM, and the application s underlying JavaScript code makes the debugging process for web applications challenging. As an analogy, the client-side component of a web application follows a similar structure as the one outlined in the model-view-controller design pattern [22]. Here, the DOM captures the structure of the application and also serves as the interface between the user and the application, acting as both the model and view. JavaScript code is executed in response to actions on the DOM and is responsible for updating the DOM, acting as the application s controller. 2

14 1.1. Background var button = document.getelementbyid("mybutton"); button.addeventlistener('click', function() { // Respond to click event, execute some JS... }); Figure 1.1: Example event registration technique for web applications Document Object Model (DOM) The DOM acts as an interface between JavaScript applications and the browser s front-end. This dynamic tree-like structure represents the UI at runtime and is updated by client-side JavaScript code to represent state changes. A website can use JavaScript to bind event listeners to HTML elements such as buttons. These registered listeners execute a specified JavaScript function when a particular event occurs on their parent HTML element. For example, the addeventlistener JavaScript method is available to most HTML elements and can be passed an event type, such as click, along with a callback function in order to register a listener. Once a listener is registered, the browser will execute the specified callback function if the user clicks on the listener s associated HTML element. Figure 1.1 illustrates the use of the JavaScript addeventlistener method to register a callback function with an existing button, which exists within the DOM. Once registered on Line 3, the callback function is executed once the button with id mybutton is clicked by the application user JavaScript JavaScript developers often include JavaScript code within their sites in order to improve the level of interactivity. JavaScript allows for the mutation of the DOM at runtime without the need for a page reload, and has become widely-adopted in websites and online applications. Additionally, JavaScript can be used to communicate with a web server. Web browsers provide a single thread for web application execution. To circumvent this limitation and build rich responsive web applications, developers take advantage of the asynchronous capabilities o ered by modern browsers, such as timeouts and XMLHttpRequest (XHR) calls. Timeouts: Events can be registered to fire after a certain amount of time or 3

15 1.1. Background public void testslideshow() throws Exception { driver.get(" } assertequals("1", driver.findelement(by.id("ss_n")).gettext()); driver.findelement(by.linktext("next")).click(); assertequals("2", driver.findelement(by.id("ss_n")).gettext()); Figure 1.2: Example of a Selenium test case used to test web applications. at certain intervals in JavaScript. These timeouts often have asynchronous callbacks that are executed when triggered. XHR Callbacks: XHR objects are used to exchange data asynchronously with the server, without requiring a page reload. Each XHR goes through three main phases: open, send, and response. These three phases can be scattered throughout the code. There is no guarantee on the timing and the order of XHR responses from the server Testing Web Applications Currently, frameworks such as Selenium and CasperJS are used to automate the execution of test cases related to an application s client-side. These frameworks work by simulating a series of user actions specified by the tester on the application under test. Testers expect that by executing a series of event-based actions on the application, its DOM state will be brought to a desirable state to verify the presence and properties of certain elements. This black-box approach to testing requires little knowledge of the application s client-side JavaScript or server-side code, and hence is easily applicable. Testers may either either record a series of actions, such as mouse clicks, during a user session and have these automatically translated into a script representation, or manually specify a series of actions in a script-like format. Then, the framework is able to replay these actions while executing certain assertions on the application s DOM. Figure 1.2 shows an example of one such test case that navigates to a specific application page (Line 3), makes an initial assertion on a DOM element with id ss n (Line 5), and then simulates a click action before executing a second assertion (Lines 7-9). The driver object within the written test case acts 4

16 1.2. Motivation 1 var currentpage = 1; 2 var sorttype = 'default'; 3 var gridsize = 8; 4 var infinitescroll = false; 5 6 var renderassets = function(url, size) { 7 var data = assetsfromserver(url); 8 9 var temp = '<div class="asset-row">'; 10 for (i = 0; i < size; i++) { 3 11 temp += ' <div class="asset-icon">'; // Reading from variable 'data' 13 temp += ' </div>'; 14 } 15 temp += '</div>'; return $('#assets-container').append(temp); 2 18 }; $(document).on('click', '#sort-assets', function(){ 21 $('#sort-assets').removeclass('selected-type') 22 $(this).addclass('selected-type'); 23 currentpage = 1; 24 sorttype = $(this).attr('type'); 25 4 gridsize = 12; 26 renderassets(url + sorttype + currentpage, gridsize) 27 infinitescroll = true; 28 }); var scroll = function() { 31 if(infinitescroll) { 32 currentpage++; 33 renderassets(url +sorttype + currentpage, gridsize/2) 34 } 35 }; 36 $(window).bind('scroll', scroll); (a) All Categories Bar Chart Bubble Chart Date Time Directions by Google (b) 1 <div class="row-sort-assets"> 2 <div class="sort-assets"></div> </div> 5 <div id="assets-container" data-pages=""> 6 <div class="asset-row"> 7 <div class="asset-icon"></div> </div> 10 </div> (c) 1 public void testsortbydefaults() { 2 driver.get(" 3 driver.findelement(by.css("i.icon-star")).click(); 4 int s1 = driver.findelements(by.css(".asset-icon")).size(); 5 assertequals(12, s1); 6 7 scrollwindowdown(); 8 int s2 = driver.findelements(by.css(".asset-icon")).size(); 9 assertequals(4, s2 - s1); 10 } assertequals(4, s2 - s1), AssertionFailure: expected <4> but was: <6> (d) (e) A 1 Figure 1.3: Running example (a) JavaScript code, (b) Portion of DOMbased UI, (c) Partial DOM, (d) DOM-based (Selenium) test case, (e) Test case assertion failure. The dotted lines show the links between the di erent entities that must be inferred. as the interface between the test case and the web application, which is accessed via a web browser. 1.2 Motivation Our work is motivated by the fact that Selenium test cases do not adequately represent the execution of an application s underlying code. Mapping actions and assertions from a front-end test case to the related JavaScript code is di cult without some supplementary information. A way of automatically inferring and presenting the relationship between these assertions and the application s underlying code would reduce the e ort required to localize a test assertion fault. 5

17 1.2. Motivation Running Example We use a small code snippet based on the open-source WSO2 estore application [4] as a running example to demonstrate the challenges involved and our solution. The store allows clients to customize and deploy their own digital storefront. A single row of four placeholder assets are shown in Figure 1.3b; three small buttons in the upper-right corner allow for these assets to be re-ordered according to some pre-existing categories. A partial DOM representation of the page is shown in Figure 1.3c. Figure 1.3d shows a Selenium test case, written by the developers of the application for verifying the application s functionality in regards to sorting and viewing the existing assets. The JavaScript code responsible for implementing the functionality is shown in Figure 1.3a. The estore application makes prominent use of the jquery 2 JavaScript library, which is used to retrieve DOM elements using the $ symbol. Invoking the $ function with an argument is equivalent to retrieving elements via the getelementbyid method, which shown on Line 1 of Figure 1.1. The test case first creates a browser instance and navigates to the asset page (Figure 1.3d, Line 2). Then, the star icon shown in the upper-right corner of Figure 1.3b is clicked (Figure 1.3d, Line 3) to sort the assets by popularity. Then an assertion is made to check whether the expected assets are present on the DOM of the page (Line 5). The second portion of the test case involves scrolling down on the webpage and asserting the existence of four additional assets on the DOM (Lines 7 9). In the underlying JavaScript code (Figure 1.3a), an event listener is registered (Lines 20 28) with each of the sorting icons in the upper-right portion of the UI. Thus, clicking on any of these icons flushes the asset grid and repopulates it according to the sort criteria (Line 26). The scroll function (Lines 30 35) is registered to the scroll event of the window object (Line 36) and is responsible for appending additional assets to the DOM, as the user scrolls down the page Challenges While the mapping between the test case and related JavaScript code may seem trivial to find for this small example, challenges arise as the JavaScript 2 6

18 1.2. Motivation code base and the DOM increase in size. As a result, it can be di cult to understand the reason for a test case failure or even which features are being tested by a given test case. C1: Identifying Dependencies of a Test Case. The test case in Figure 1.3d fails when executed and displays the message shown in Figure 1.3e. Based on the failure message alone, it is almost impossible to determine the cause of failure. Closer examination of the test case, however, reveals that the assertion fails on variables s1 and s2, which in turn depend on DOM elements with class asset-icon. This link between the assertion and the DOM is labelled in Figure 1.3 as. C2: Finding the Point of Contact. However, it remains unclear which Java- Script code, if any, was responsible for modifying this subset of DOM elements, which eventually led to the assertion failure. The link to the actual faulty line of code is not easy to find. In order to make this association, developers must tediously examine the application s underlying JavaScript code in search of any statements that mutate the DOM. In the context of our example, a developer would eventually conclude that Line 17 of Figure 1.3a is actually responsible for appending elements to the DOM. This implicit link between the checked DOM elements and JavaScript code is depicted on the Figure as. Further, lines 9 15 are responsible for generating new DOM elements expected by our failing assertion ( ). C3: Mapping Test Case Execution to Events. At this point, it is still not clear which events would cause the responsible lines of JavaScript code to execute. Further examination of the code reveals that the renderassets() function (Figure 1.3a, Line 6) can be called from within two event-handlers in response to either a click or a scroll event (Lines 26 and 33, respectively, shown as π). Now that the developer has gathered some contextual information related to the failure, the relation between the failed test case and the triggered event handlers must be derived. While in our running example it may be straightforward to link the call to scrollwindowdown() (Line 7 of Figure 1.3d) to the execution of event handler scroll (Line of Figure 1.3a) due to the similarity in naming convention, such a linear mapping is neither possible in all cases nor easily inferable. C4: Tracking Dependencies. Finally, going back to the initial point of contact between the DOM and JavaScript code (Line 17 of Figure 1.3a), to fully understand an assertion and its possible cause of failure, the data and 7

19 1.3. Claims and Contributions control dependencies for the DOM-altering statements must be determined and examined by the developer in order to identify all possible points of failure. Often, such dependencies may exist across the occurrence of multiple DOM or asynchronous events (e.g., timing, XMLHttpRequest). In the case of estore, the modification of the DOM within renderassets() depends on the arguments passed into the function. Argument url is used to retrieve asset information from the server (Line 7), and argument size is used as a control dependency when generating assets (Line 10). Dotted line 4 shows possible invocations of renderassets(), revealing dependencies on global variables (url, sorttype, currentpage, gridsize). An erroneous update to any of these global variables, in response to a DOM event, could be the cause of our assertion failure. Tracing the data dependencies for the call to renderassets() on Line 33 reveals that an update to global variable gridsize on Line 25 of Figure 1.3a is the root cause of our application s unusual behaviour, i.e., gridsize is changed to 12, while line 33 assumes it is still 8, the initial value. Thus, a code change to either Line 25 or 33 would fix the fault and resolve the assertion failure reported by our test case. 1.3 Claims and Contributions Our work makes the following key contributions: An automated technique, called Camellia, for connecting test assertion failures to faulty JavaScript code through checked DOM elements. A selective instrumentation algorithm that statically detects and instruments only related lines of code, to minimize the trace size and performance overhead. A generic JavaScript slicing technique and stand-alone open source tool called SliceJS. An interactive visualization to enhance the debugging experience. An evaluation of Camellia through a controlled experiment with 12 participants, assessing its e cacy and usefulness in understanding test failures and localizing faults. Our measurements indicate that the selective instrumentation algorithm is e ective in reducing the trace size (by 70%), the instrumentation time (by 41%), and the execution overhead (by 47%). The results of our controlled 8

20 1.3. Claims and Contributions experiment show that Camellia is capable of improving user accuracy by nearly a factor of two for tasks related to localizing and repairing JavaScript faults. This thesis is structured as follows. The next two chapters contain detailed descriptions of our approach to designing and implementing our solution. An evaluation of our tool, Camellia, is contained within Chapter 4 and a discussion of its implications follows in Chapter 5. Chapter 6 presents related work. We conclude in Chapter 7. 9

21 Chapter 2 Approach: Modelling Web Application Behaviour In the next two chapters, we describe our approach for addressing the challenges mentioned previously in Chapter 1.2. The overall process consists of two main phases, the first of which is detailed in this chapter. This initial phase is a general technique for capturing and modelling an application s behaviour, and consists of (1) capturing a fine-grained execution trace for the application under test, (2) extracting a behavioural model from the captured trace, and (3) producing a visualization based on the inferred model. During the second phase of our approach, we expand on this general model to include test case information. As a result, we are able to relate each test failure to the relevant areas of an application s execution, which assists developers to better understand test failures. This process is necessary as program comprehension is known to be an essential step during software maintenance and debugging activities [11]. Below, we describe each of the aforementioned three steps further. Our technical report [7] contains a more elaborate description of the technical details of this initial phase of the approach. 2.1 JavaScript Transformation and Tracing Our approach aims to help developers understand the context of their assertions by monitoring test-related JavaScript execution, DOM events, asynchronous events, and DOM mutations (C3 ). Tracking this information allows us to generate a trace comprising multiple trace units with each unit representing the occurrence of an asynchronous event or the execution of a JavaScript function. Below, we discuss how we monitor all this activity with the purpose of relating it to test assertions. 10

22 2.1. JavaScript Transformation and Tracing Interposing on DOM Events There are two ways event listeners can be bound to a DOM element in JavaScript. The first method is programatically using the DOM Level 1 e.click=handler or DOM Level 2 e.addeventlistener methods [31] in JavaScript code. To record the occurrence of such events, our technique replaces the default registration of these JavaScript methods such that all event listeners are wrapped within a tracing function that logs the occurring event s time, type, and target. The second and more traditional way to register an event listener is inline in the HTML code, e.g., <DIV onclick= handler(); >. The e ect of this inline assignment is semantically the same as the first method. Our technique interposes on inline-registered listeners by removing them from their associated HTML elements, annotating the HTML elements, and re-registering them using the substituted addeventlistener function. This way we can handle them similarly to the programmatically registered event handlers Capturing Timeouts and XHRs For tracing timeouts, we replace the browser s settimeout() method and the callback function of each timeout with wrapper functions, which allow us to track the instantiation and resolution of each timeout. A timeout callback usually happens later and triggers new behaviour, and thus we consider it as a separate component than a settimeout(). We link these together through a timeout id and represent them as a causal connection later. In our model, we distinguish between three di erent components for the open, send, and response phases of each XHR object. We intercept each component by replacing the XMLHttpRequest object of the browser. The new object captures the information about each component while preserving its functionality Recording Function Traces To track the flow of execution within a JavaScript-based application, we instrument three code constructs, namely function declarations, return statements, and function calls. Each of these code constructs are instrumented di erently, as explained below. 11

23 2.1. JavaScript Transformation and Tracing var scroll = function() { send(json.stringify({messagetype: "FUNCTION ENTER", fnname: "scroll", args: null,...})); if(infinitescroll) { currentpage++; renderassets(url +sorttype + currentpage, gridsize/2) } send(json.stringify({messagetype: "FUNCTION EXIT", fnname: "scroll",...})); };... Figure 2.1: Instrumented JavaScript function declaration. Function Declarations: Tracing code is automatically added to each function declaration allowing us to track the flow of control between developer-defined functions by logging the subroutine s name, arguments, and line number. In case of anonymous functions, the line number and source file of the subroutine are used as supplementary information to identify the executed code. As this communication is done each time a function is executed, argument values are recorded dynamically at the cost of a slight overhead. Figure 2.1 contains the simple scroll() JavaScript function from the running example shown in Figure 1.3a (Line 30), which has been instrumented to record both the beginning and end of its execution (Lines 31 and 36). Return Statements: Apart from reaching the end of a subroutine, control can be returned back to a calling function through a return statement. There are two reasons for instrumenting return statements: (1) to accurately track nested function calls, and (2) to provide users with the line numbers of the executed return statements. Without recording the execution of return statements, it would be di cult to accurately track nested function calls. Furthermore, by recording return values and the line number of each return statement, our approach is able to provide users with contextual information that can be useful during the debugging process. Figure 2.2 illustrates the instrumentation for the return statement of renderassets(), a function originally shown in the running example (Figure 1.3a, Lines 6-18). The wrapper function RSW receives the return value of the function and the line number of the return statement and is responsible for recording this information before execution of the application s Java- Script is resumed. Function Calls: In order to report the source of a function invocation, our approach also instruments function calls. When instrumenting function calls, it is important to preserve both the order and context of each 12

24 2.1. JavaScript Transformation and Tracing var renderassets = function(url, size) { var data = assetsfromserver(url); }; var temp = '<div class="asset-row">'; for (i = 0; i < size; i++) { temp += ' <div class="asset-icon">';... // Reading from variable 'data' temp += ' </div>'; } temp += '</div>'; return RSW($('#assets-container').append(temp), 17); Figure 2.2: Instrumented JavaScript return statement. $(document).on('click', '#sort-assets', function(){ $('#sort-assets')[[fcw("removeclass", 21)]]('selected-type'); $(this)[[fcw("addclass", 22)]]('selected-type'); currentpage = 1; sorttype = $(this)[[fcw("attr", 24)]]('type'); gridsize = 12; FCW(renderAssets, 26)(url + sorttype + (currentpage)); infinitescroll = true; }); function FCW(fnName, lineno) { // Function Call Wrapper send(json.stringify({messagetype: "FUNCTION_CALL",..., targetfunction: fnname})); return fnname; } Figure 2.3: Instrumented JavaScript function calls for running example. dynamic call. To accurately capture the function call hierarchy, we modify function calls with an inline wrapper function. This allows us to elegantly deal with two challenging scenarios. First, when multiple function calls are executed from within a single line of JavaScript code, it allows us to infer the order of these calls without the need for complex static analysis. Second, inline instrumentation enables us to capture nested function calls. Figure 2.3 depicts the instrumentation of function calls within an event-handler method from Figure 1.3a. Once instrumented using our technique, the function call to renderassets() is wrapped by function FCW (Line 26). The interposing function FCW() executes immediately before each of the original function calls and interlaces our function logging with the application s original behaviour. Class methods removeclass(), addclass(), and attr() are also instrumented in a similar way (Lines 21, 22, and 24 respectively). 13

25 2.2. Capturing a Behavioural Model Tracking DOM Changes Further, we track the DOM s evolution, as information about DOM mutations can help developers relate the observable changes of an application to the corresponding events and JavaScript code. We leverage the onload event of the document object to track the initial DOM state una ected by any JavaScript execution. Building on this, we use an observer module to log any subsequent changes to the DOM. This information is interleaved with the logged information about events and functions, enabling us to link DOM changes with the JavaScript code that is responsible for these mutations. 2.2 Capturing a Behavioural Model We use a graph-based model to capture and represent a web application s event-based interactions. The graph is multi-edge and directed. It contains an ordered set of nodes, called episodes, linked through edges that preserve the chronological order of event executions. 3 We describe the components of the graph below Episode Nodes An episode is a semantically meaningful part of the application behaviour, initiated by a synchronous or an asynchronous event. An event may lead to the execution of JavaScript code, and may change the DOM state of the application. An episode node contains information about the static and dynamic characteristics of the application Edges In our model, edges represent a progression of time and are used to connect episode nodes. Two types of edges are present in the model: Temporal: The temporal edges connect one episode node to another, indicating that an episode succeeded the previous one in time. 3 Because JavaScript is single-threaded on all browsers, the events are totally ordered in time. 14

26 2.3. Visualizing the Captured Model Causal: These edges are used to connect di erent components of an asynchronous event, e.g., timeouts and XHRs. A causal edge from episode s to d indicates episode d was caused by episode s in the past Story The term story refers to an arrangement of episode nodes encapsulating a sequence of interactions with a web application. Di erent stories can be captured according to di erent features, goals, or use-cases that need investigation. An episode terminates semantically when the execution of the JavaScript code related to that episode is finished. DOM mutation units that were interleaved with other trace units are organized and linked to their respective episode for a given story. 2.3 Visualizing the Captured Model In the final step of the first phase, our technique produces an interactive visualization of the generated model, which can be used by developers to understand the behaviour of the application. The main challenge in the visualization is to provide a way to display the model without overwhelming the developer with the details. To this end, our visualization follows a focus+context [10] technique that provides the details based on a user s demand. The idea is to start with an overview of the captured story, let the users determine which episode they are interested in, and provide an easy means to drill down to the episode of interest. With integration of focus within the context, developers can semantically zoom into each episode to gain more details regarding that episode, while preserving the contextual information about the story Semantic Zoom Levels The visualization provides 3 semantic zoom levels. The first level displays all of the episodes in an abstracted manner, showing only the type and the timestamp of each episode (Figure 2.4, top). When an episode is selected, the view transitions into the second zoom level, which presents an outline of the selected episode, providing more information 15

27 2.4. Tool Implementation: Clematis about the source event as well as a high-level JavaScript trace (Figure 2.4, middle). The trace at this level contains only the names of the (1) invoked functions, (2) triggered events, and (3) DOM mutations, caused directly or indirectly by the source event. The user can view multiple episodes to have a side-by-side comparison. The final zoom level exhibits all the information embedded in each episode, i.e., detailed information about the source event, the DOM mutations caused by the episode, and the low-level trace (Figure 2.4, bottom). Upon request, the JavaScript code of each executed function is displayed and highlighted. 2.4 Tool Implementation: Clematis The first phase of our approach is implemented in a tool called Clematis, which is freely available [2]. We use a proxy server to automatically intercept and inspect HTTP responses destined for the client s browser. When a response contains JavaScript code, it is transformed by Clematis. The trace data collected is periodically transmitted from the browser to the proxy server in JSON format. To observe low-level DOM mutations, we build on and extend the JavaScript Mutation Summary library [14]. The model is automatically visualized as a web-based interactive interface. 2.5 Results and E ectiveness of Proposed Model A controlled experiment was conducted at a large software company in Vancouver to investigate Clematis s e ectiveness in aiding developers in program comprehension tasks [6]. For the study, we recruited professional developers as participants and used an open-source web application as the experimental object. The evaluation of Clematis points to the e cacy of the approach in reducing the overall time and increasing the accuracy of developer actions, compared to state-of-the-art web development tools. Specifically, for tasks related to program comprehension, developers using Clematis took 47% less time on assigned tasks related to program comprehension, compared to developers using other web development tools. Moreover, the results showed that developers using Clematis performed more accurately across all tasks by 61% on average. 16

28 2.5. Results and E ectiveness of Proposed Model Episode #1 Event Episode #2 Event Episode #3 Timeout Episode #4 Event Episode #5 Event 1 Zoom Level 0 to Level 1 Episode #1 Event Episode #2 Event Source Episode #4 Event Episode #5 Event TO: 1 Trace TID: 1 ss_slideshow() ss_update() hideelem(x) dg(x) inlineelem(x) ss_run() TID: 2 DOM Mutations tex removed text removed text removed text added text added text added 2 Zoom Level 1 to Level 2 Episode #1 Event Episode #2 Event Episode #4 Event Episode #5 Event ss_slideshow() ss_update() hideelem(x) dg(x) TID: 1 callback: ss_slideshow duration: Figure 2.4: Three semantic zoom levels in Clematis. Top, Level 0: overview. Middle, Level 1: zoomed one level into an episode, while preserving the context of the story. Bottom, Level 2: drilled down into the selected episode. 17

29 Chapter 3 Approach: Test Case Understanding The second phase of our approach builds on the model and visualization described in the previous chapter (Chapter 2). Given this method for modelling an application s behaviour, we now focus on relating the produced model to a test case s actual execution. Linking test cases to the available model assists developers in localizing faults from test failures. The overall process has four main steps, described subsequently in the next few sections. 3.1 Relating Test Assertions to DOM Elements The DOM acts as the interface between a front-end test case and an application s JavaScript code. Therefore, the first step to understanding the cause for a test case failure is to determine the DOM dependencies for each test assertion. While this seems simple in theory, in practice, assertions and element accesses are often intertwined within a single test case, convoluting the mapping between the two. Going back to the test case of our running example in Figure 1.3d, the first assertion on Line 5 is dependent on the DOM elements returned by the access on the previous line. The last assertion on Line 9 is more complex as it compares two snapshots of the DOM and therefore has dependencies on 2 DOM accesses (Lines 4 and 8). Figure 3.3 summarizes the test case s execution and captures the temporal and causal relations between each assertion and DOM access. To accurately determine the DOM dependencies of each assertion, and to address C1 ( in Figure 1.3), we apply dynamic backward slicing to each 18

30 3.2. Contextualizing Test Case Assertions DOM Access 1 (icon-star) DOM Access 2 (asset-icons) DOM Assertion 1 Access 3 Assertion 2 (asset-icons) + Figure 3.1: Relating assertions to DOM accesses for the test case of Figure 1.3d. test case assertion. In addition, we track the runtime properties of those DOM elements accessed by the test case. This runtime information is later used in our analysis (in Section 2.1) when analyzing the DOM dependencies of each assertion. 3.2 Contextualizing Test Case Assertions In the second step, our approach aims to (1) help developers understand the context of their assertions by monitoring test-related JavaScript execution, asynchronous events, and DOM mutations (C3 ); (2) determine the initial link between JavaScript code and the checked DOM elements (C2, in Figure 1.3). In order to monitor JavaScript events, we leverage the tracing technique outlined in Chapter 2.1, which tracks the occurrence of JavaScript events, function invocations, and DOM mutations. We utilize the tracked mutations in order to focus on the segments of JavaScript execution most relevant to the assertions in a test case. As we are only interested in the subset of the DOM relevant to each test case, our approach focuses on the JavaScript code that interacts with this subset. Recall that the previous step described in Section 3.1 yields the set of DOM elements relevant to each assertion. We cross reference these sets with the timestamped DOM mutations in our execution trace to determine the Java- Script functions and events (DOM, timing, or XHR) relevant to each assertion. Once the relevant events and JavaScript functions have been identified for each assertion, we introduce wrapper functions for the native JavaScript functions used by developers to retrieve DOM elements. Specifically, we redefine methods such as getelementbyid and getelementsbyclassname to 19

31 3.3. Slicing the JavaScript Code track DOM accesses within the web application itself so that we know exactly where in our collected execution trace the mutation originated. The objects returned by these methods are used by the application later to update the DOM. Therefore, we compute the forward slice of these objects to determine the exact JavaScript lines responsible for updating the DOM. Henceforth, we refer to the references returned by these native methods as JavaScript DOM accesses. We compare the recorded JavaScript DOM accesses with the DOM dependencies of each test case assertion to find the equivalent JavaScript DOM accesses within the application s code. Moreover, the ancestors of those elements accessed by each assertion are also compared with the recorded JavaScript DOM accesses. This is important because in many cases a direct link might not exist between them. For instance, in the case of our running example, a group of assets are compiled and appended to the DOM after a scroll event. We compare the properties of those DOM elements accessed by the final assertion (assets on Lines 4 and 8 of Figure 1.3d), as well as the properties of those elements ancestors, with the recorded JavaScript DOM accesses and conclude that the assets were added to the DOM via the parent element assets container on Line 17 of Figure 1.3a ( ). 3.3 Slicing the JavaScript Code At this point, our approach yields the set of JavaScript statements responsible for updating the DOM dependencies of our test case. However, the set in isolation seldom contains the cause of a test failure. We compute a backwards slice for these DOM-mutating statements to find the entire set of statements that perform the DOM mutation, thus addressing C4. Slicing can be done statically or dynamically. Static slicing is often overly conservative when identifying dependencies for a statement or variable, especially for a dynamic language such as JavaScript. In our approach, we have opted for dynamic slicing, which enables us to produce thinner slices that are representative of each test execution, thus reducing noise during the debugging process. Moreover, by using dynamic analysis we are able to present the user with valuable runtime information that would not be available through static analysis of JavaScript code. For example, if we were to compute the slice for variable temp on Line 17 of the running example (Figure 1.3a), it would be useful to know the runtime values of the arguments 20

32 3.3. Slicing the JavaScript Code passed into renderassets, as these values a ect variable temp. To produce a trace for dynamic slicing, our approach selectively instruments the JavaScript code Selective Instrumentation An ideal test case would minimize setup by exercising only the relevant Java- Script code related to its assertions. However, developers are often unaware of the complete inner workings of the application under test. As a result, it is possible for a test case to execute JavaScript code that is unrelated to any of its contained assertions. In such a case, instrumenting an entire web application s JavaScript code base would yield a large trace with unnecessary information. This can incur high performance overheads, which may change the web application s behaviour. Therefore, instead of instrumenting the entirety of the code for dynamic slicing, our approach intercepts and statically analyzes all JavaScript code sent from the server to the client to determine which statements may influence the asserted DOM elements. Then, this subset of the application s code is instrumented. This approach has two advantages. First, it minimizes the impact our code instrumentation has on the application s performance. Second, selective instrumentation yields a more relevant and concise execution trace, which in turn lowers the processing time required to compute a backward slice. Algorithm 1 summarizes our approach to selective code instrumentation. Our algorithm first converts the code into an abstract syntax tree (AST). This tree is traversed in search of a node matching the initial slicing criteria (Line 6, LocateVariableInSourceCode()). Once found, the function containing the initial definition of the variable-in-question is also found (Line 7), henceforth referred to as the parent closure. Based on this information, the algorithm searches this parent closure for all references to the variable of interest. This is done in order to find all locations in the JavaScript code where the variable may be updated, or where a new alias may be created for the variable. As JavaScript is not type-safe, using static analysis to determine a variable s possible type can be complex. Instead of such complex analysis, our approach assumes that any variable may have a non-primitive value at runtime, and instead uses collected runtime information to determine whether alias analysis is required for each variable. Runtime value types are important since only those variables with non-primitive types can be assigned aliases. Moreover, for each variable update pertaining to the 21