Regression Test Case Prioritization Based on Historical Test Case Performance Data

Transcription

1 Regression Test Case Prioritization Based on Historical Test Case Performance Data Andreas Ljung Department of Computer Science Lund University, Faculty of Engineering May 14, 2010

2 Contact information Author: Andreas Ljung Supervisors: Emelie Engström Lund University, Faculty of Engineering Roger Hassel Sony Ericsson Communications AB Hares Mawlayi Sony Ericsson Communications AB Examinator: Per Runeson Lund University, Faculty of Engineering

3 Abstract Regression testing is performed to ensure that the overall quality is maintained in a growing and evolving software project, to be able to perform an resourceefficient regression test there is a need for a selective regression test approach. Some kind of test case selection technique or test case prioritization technique is commonly used in large organizations regression test processes. These techniques allow the regression tests to be performed with a constant amount of resources in a growing project by making sure that only the most effective and important test cases are being executed. This master s thesis started with an idea about automatizing the regression test case selection based on the test cases historical performance. The literature study focused on finding such a technique that would suite the test environment. The test environment and the regression test case selection technique currently used at one of the integration test departments at Sony Ericsson Mobile Communications was analyzed. The advantages and disadvantages of the currently used technique was identified and together with the articles found in the literature study a new regression test case selection technique was presented. The technique uses the test cases historical performance and will remedy some of the current techniques disadvantages. It prioritizes the test cases based on the test cases execution history, historical fault detection effectiveness and pre-set priority and the selection is made based on these priorities. This technique automates the currently manual process of selecting a test suite for regression test and focuses on the historically most efficient test cases. To evaluate this technique a prototype application was developed. The application was used to automatically select test cases for six test suites that were compared to their corresponding manually selected regression test suites. The automatically selected test suites were found significantly more efficient, in fault detection efficiency, than the manually selected ones at a significance level of 5%. The automatically selected test suites were also shown to detect the same amount of defects as the manually selected test suites with fewer test cases. A manual evaluation was performed by five testers and it concluded that the technique could be useful but to some extent included irrelevant test cases and that some important test cases were missing. Despite this the new selection technique was shown to be more efficient. Keywords Regression test, Test case prioritization, History-based test case prioritization.

4 Contents 1 Introduction Background for the project Research questions Outline Theoretical background and related work Introduction to software testing Black box and white box testing Regression test Regression test approaches Regression test case selection techniques Regression test case prioritization techniques Regression test case selection/prioritization techniques in a black box environment History based regression test case prioritization techniques for a black box environment The verification process at Sony Ericsson Mobile Communications Organization Development and verification process Tools and technical environment Current regression test case selection technique Research Results Regression test case prioritization based on execution history and pre-set priority Methodology Criteria for the test case prioritization technique Inspiration and base equation The developed test case prioritization equation Test case selection

5 6 Evaluation Evaluation criteria found in literature study Evaluation settings Evaluation results Discussion and analysis Conclusion and future work The currently used test case selection and its disadvantages Solution to the problems and disadvantages Changes needed in the test process Changes in the implementation of the application A Evaluation form 47 A.1 The form A.2 Results

6 Chapter 1 Introduction 1.1 Background for the project The original idea for this project came from a section manager at Sony Ericsson Mobile Communications, he had studied the control system of self registration shoppers at the local ICA store. The store allows their customers to scan their products while picking them from the shelves and then on the way out they hand in the scanner and pay the amount that is registered in it. This demands a certain level of trust between the store and their customer which is kept by performing random controls, where all the products are checked towards the registration in the scanner. Every time this control is performed and everything is found to have been scanned correctly, the customer gains some more trust which leads to fewer controls. But if an unregistered product is found the customer loses all his gained trust and has to go through controls more often. The idea was to develop a similar control system to sort out which test cases that has been most effective in finding defects in the software. 1.2 Research questions The questions this master s thesis answers are listed below. 1. How is the test case selection for regression test currently performed at the integration test department at Sony Ericsson Mobile Communications? 2. What are the problems and disadvantages with the current routines? 3. Is it possible to automatically calculate priorities for the test cases with the available data about their historical performance? What data is available and what data could be used to prioritize test cases? How could the data be combined and weighed against each other to generate a good priority value? 3

7 Could the prioritization and selection of the test cases be performed by a semi automatic tool? What changes in the routines are necessary before a semi automatic regression test case selection tool could be integrated into the current process? 1.3 Outline This master s thesis report consists of six more chapters. In chapter 2 the theoretical background and an overview of the research made in the area are presented. In chapter 3 the current verification process and organization at Sony Ericsson is described and chapter 4 the current process s problems and disadvantages are presented. Chapter 5 presents the method for solving the presented problems with a new regression test case selection technique. The application and technique are evaluated in chapter 6 and in chapter 7 the project is concluded with discussion and analysis. 4

8 Chapter 2 Theoretical background and related work 2.1 Introduction to software testing Testing is the process of exercising a software component using a selected suite of test cases, with the intent of revealing defects and evaluate quality [6]. The tester s task is to reveal defects, find weak points, inconsistent behavior and circumstances where the software does not work as expected. Testing is performed by executing test cases, test cases are written documents or small pieces of software with a specific pre condition and/or input parameters and an expected post condition and/or output variables. A test case validates one or more of a software s requirements, when the test case is executed it generates a pass if the actual post condition and output variables match the expected ones. A group of test cases selected to be executed together are called a test suite. [7] Testing is generally performed at four different levels; unit test, integration test, system test and acceptance test [8]. Unit testing covers testing of individual components in the code. Integration testing tests the interface between units and that the assembling of the different units did not cause any defects. After the integration the system should be complete and ready for system testing. System testing tests the system when it is complete and the goal is to ensure that the system performs according to its requirements. Finally the acceptance test is performed by the clients so that they can be sure that their requirements and demands have been met. 2.2 Black box and white box testing In software verification there are two basic strategies for designing test cases; black box and white box. When designing a test case for black box testing the 5

9 tester considers the software to be an opaque box with no knowledge about the inner structure. The tester only knows what function the software has. The test case is designed with a set of input parameters and an expected output. When the test case is executed the outcome is determined by matching the generated output to the specified output [9]. The white box approach focuses on the inner structure of the software. The tester has to have knowledge about the code and designs test cases that execute specific parts of it. For example a test can exercise all statements or true/false branches that occur in a module [9]. 2.3 Regression test The word regression means to relapse to a less perfect or developed state [10]. Regression in software development means that the quality of the software becomes affected by changes and added features, which could lead to new defects in the parts of the software that was working properly before. To instill confidence that the unchanged parts of the software is unaffected after the integrations has been made regression testing is performed. An important difference between regression testing and development testing is that during regression testing an established suite may be available for reuse [11]. Regression test is usually performed at the system test level, but could also be a part of the integration test level. Li and Wahl give an overview of regression testing and summarize data from several other reports in their paper [12]. They conclude that regression testing could be divided into two different types, corrective and progressive testing. Corrective regression testing is when specifications are unmodified and test cases can be reused and progressive regression testing applies when specifications are modified and new test cases must be designed. In this paper regression testing implicitly means corrective regression testing if nothing else is specified. 2.4 Regression test approaches When reusing test cases the main question is whether to run all the available test cases or just some of them. The retest-all strategy may waste time and resources but it is the safest method that ensures that all functions that was working before is still working. Another approach is the selective strategy; it uses a subset of the existing test cases to reduce the retesting costs [12], i.e. a test suite. But this reduction is a trade-off against the risk to miss an introduced defect and the cost involved with fixing this defect at a later stage in the process. When using the selective strategy the cost of selecting the test cases has to be taken into account, this cost has to be less than the cost of running the omitted test cases so that the total time for both selecting and executing the tests is reduced in comparison to the re-test all technique [11]. An approach that is similar to test case selection is test case prioritization, where every test case is given a priority and are then sorted and executed in 6

10 that order until the resources and time has been exhausted [13]. Kim and Porter acknowledge a third approach they call Random/Ad-Hoc, which is when testers select cases randomly or rely on their prior knowledge or experience to select test cases [14]. 2.5 Regression test case selection techniques The majority of regression test selection techniques that have been developed are based on information about the difference in the code in the main version and the modified version. Rothermel and Harrold have divided the code-based techniques into three different groups. Coverage techniques that locate which components that has been modified and selects tests that exercise these components, minimization techniques that are similar to coverage techniques but select minimal sets of tests and safe techniques that select every test that can expose one or more defects in the software [11]. Examples of different code-based techniques are linear equation techniques, the path analyses technique and the firewall technique [11]. The challenge in the code-based techniques is to identify the dependencies between a test case and the program entities it covers [12]. 2.6 Regression test case prioritization techniques Most regression test case selection techniques ignore software development realities such as resource and time constraints that may prevent using the techniques as intended. For example if regression testing has to be performed in one day but the selection technique selects a test suite that requires two days worth of tests [14]. By giving a priority to all the available test cases the test suite could easily be adjusted in size depending on the session s time and resource constraints and also depending on the session s importance. Different prioritization techniques use different criteria for calculating the priority for each test case. For example code coverage or expected frequency of use. The prioritization technique gives the tester the ability to achieve the desired goal, depending on criteria, as early as possible. For example if the goal is full code coverage, then the test cases with the highest code coverage will be executed first and as soon as the goal is reached the testing could be stopped [15]. Test prioritization could also be used in conjunction with test case selection to set the execution order in the selected test suite. This will in turn ensure that if the session is unexpectedly terminated the time spent testing will have been more beneficially than if they were not prioritized [15]. 7

11 2.7 Regression test case selection/prioritization techniques in a black box environment The focus in this master s thesis is on a black box testing environment; therefore the code-based selection techniques are not applicable. When testing is performed without access to the source code (i.e. black box testing) there is a need for a different approach for regression test case selection. Engström et al. perform a systematic review of empirical evaluations of regression test selection techniques [16]. They found and evaluated 28 different techniques and only two of them were not depending on source code access and no technique used information from the test cases execution history. Srikanth et al. claim that also most of the test prioritization techniques are code coveragebased [17] and Sajeev et al. have come to the same conclusion; most regression test prioritization techniques are white box strategies [18]. Some approaches for test case selection and prioritization in black box testing environment have been found in the research area. Instead of the source code they require other inputs such as the requirements specifications, customer priority, implementation complexity, fault proneness and other version information meta data. The summary of this research is that test case selection in a black box environment has not been researched as much as white box testing. 2.8 History based regression test case prioritization techniques for a black box environment During the research described in the following three articles, describing different techniques, were found to be the most suitable ones for the black box environment considered in this project; Test Case Prioritization for Black Box Testing, Dynamic Regression Test Selection Based on a File Cache and Prioritizing Test Cases for Resource Constraint Environments Using Historical Test Case Performance Data. They are prioritizations techniques and not selection techniques but by ordering them in priority order and selecting a desired number of test cases a selected suite is derived Test Case Prioritization for Black Box Testing Qu, Nie, Xu and Zhang present a method for prioritizing test cases in a black box testing environment [13]. The initial test suite is given priorities using test case history where test cases are ordered by their prior performance of detecting faults. Then a matrix R with test cases is built on available information to define the fault detection relationship of test cases. In R, test cases are grouped by the fault types they have revealed, so once a test case reveals a new regression fault all the test cases in that group can be adjusted to a higher priority. Then the test cases are executed according to their priority and the matrix R is updated 8

12 and reordered with the latest information after every execution. This is repeated until the resources are exhausted. The fault detection relationship matrix R would have been possible to use in the current environment for this project and was considered to be added later on but during the development it became clear that there were not enough time to investigate and set up the link between the error report system and the developed prototype application. Therefore this approach was not further considered to be a part of the developed test case prioritization technique Dynamic Regression Test Selection Based on a File Cache The approach to link test cases to error reports have also been considered by Wikstrand et al. [19]. They present a recommendation system that links test cases to source code files, determines which source code files are fault prone and detects which fault prone files have been updated and recommends test cases to be executed. The link between the source code file and the test case is determined through the error report that is created when a test case reveals a defect. When the defect is fixed, the error report is updated with information about which files were affected thus creating a link from the test case that revealed the test case to the file that was updated to fix the defect. A list of fault prone files is obtained based on the Fix-cache algorithm. This list is a subset of all the source code files, containing the top most of fault prone files. The subset is pre-populated based on an initial fault proneness score which is a linear combination of; lines of code, number of high severity error reports associated with the source file and modification frequency. Then the subset is updated daily based on information from the error reports that have been closed during that day. For each file that is referenced by an error report and that is also in the cache, it is considered a hit. If the file is new it is considered a miss, both hits and misses are added to the cache. The recommendation algorithm then derives a list of test cases that are linked to the source code files in the subset which are recommended for the next regression testing cycle. Though this approach could be suitable for some black box environments the department where this project is located at does not have access to the pre-population properties such as lines of code and modification frequency. The technique is also dependent on data from the error report system. Due to these reasons this approach was not considered to be the most suitable for this project Prioritizing Test Cases for Resource Constraint Environments Using Historical Test Case Performance Data In another paper Fazlalizadeh et al. presents a solution that is also based on historical performance data [21]. They also reflect over the fact that most test case prioritization techniques are based on the analysis of source code and are 9

13 memory less. Since regression testing is a continuous activity a new equation to compute the priority in each regression test session is developed. The presented equation incorporates three factors: historical effectiveness in fault detection, each test case s execution history in regression test and finally the last priority assigned to the test case. The historical effectiveness in fault detection -factor is the test cases fault detection performance in number of times it has been executed in successive tests. The test cases execution history -factor is the to keep track of how many times the test case has been executed in a period of time to ensured that after multiple sessions it will cycle through all test cases. The last priority assigned to the test case factor makes the selection smoother in successive executions and its limits severe changes between sessions. Also it will separate the test cases that have the same historical fault detection effectiveness and execution history. In empirical studies they evaluate the prioritization technique with eight different C programs, the Siemens suite, which includes seven programs, and Space. For each program, single-fault versions of the program have been created. Their empirical studies show considerable improvements in fault detection rate, measured in APFD (Average Percentage of Fault Detection). APFD is the weighted average percentage of fault detection in test suite lifetime and the metric was introduced by Rothermel et al. [22]. It measures how rapidly a prioritized test suite detects faults, higher APFD value mean faster fault detection rates. Most of the incorporated factors in this equation are available in the black box environment considered in this project. Therefore this algorithm is the most suitable and it is used as a base for the test case prioritization equation that is presented in chapter 5. Another study, by Kim and Porter, also emphasise the need to view regression testing as a ordered sequence of test sessions where the history of the test case executions should be considered [14]. They conclude that if the test selection technique only focus on the parts of the program that have been changed since the last testing session it is possible that a change, once tested, is never retested, leaving only one chance to find a defect. 10

14 Chapter 3 The verification process at Sony Ericsson Mobile Communications 3.1 Organization At Sony Ericsson Mobile Communications the software verification is carried out at different levels by different departments. The software is divided into different parts for each function and for each part there is a group of developers and a group of testers. The developers in each function group are responsible for unit testing the code before delivery. The integration test is carried out at a test department by a test group for each function. The testing at this level is performed on temporary builds of the software or on the main branch, usually in a hardware prototype. The system test is carried out by another department and is performed on the main branch. Finally a release candidate of the software is sent for acceptance test to the carrier service providers. This master s thesis focuses on the regression test at the integration level, since it is carried out at the function test team responsible for testing the Bluetooth and USB functionality. But since the regression test case selection technique that is a product of this thesis only requires data that is available in all parts of the organization it is applicable for any part of the organization that performs regression testing, at any level. The department for integration test has several different roles involved in making the verification processes work efficiently and to ensure the overall quality of the software. The different roles are: Tester Part of a function group. Responsible for designing test cases based on the requirements, writing the test case step-by-step description, automate test cases if possible, selecting test suites for regression test sessions, exe- 11

15 cuting test cases in the test sessions and reporting defects and test suite status after execution. Technical project leader Leader of a function group. Responsible for planning test sessions, follow up test session executions, reporting test results and general resource and time management. Section Manager Responsible for organizing the resources, time and budgets for several function groups. 3.2 Development and verification process The software development process at Sony Ericsson is an incremental process, where each component is developed and integrated in small iterations. After each iteration every function group tests the new feature that have been implemented, unit tested and integrated in a temporary build by the developers. If the feature can be verified and no new defects have been introduced, it is integrated on the main branch. However, to ensure that the overall quality of the software is maintained, after several iterations, a range of regression test sessions are performed by each function test group. For every new feature that is developed several new test cases are created based on the feature requirements. All the test cases are written for black box testing and are not connected to a specific part of the code. A test case consists of a precondition, a step by step description, and an expected result. The test cases are added to the test database which contains all the test cases relevant to a specific product. The amount of features increases in each project and therefore the total amount of test cases that is available for regression testing is too vast for a re-test all regression test session to be executed on a regular basis. Since regression testing is an expensive but necessary maintenance activity it has to be performed in a balanced manner. The solution is to perform a selective regression test session that is further described in chapter Tools and technical environment All test case descriptions and the execution data are stored in HP Quality Center. The error reports are stored in IBM Rational Clear Quest and linked via an id number, called DMS number, to the revealing test case in Quality Center and also to other test cases that are affected by the defect HP Quality Center Description HP Quality Center is a web based test database software that supports essential aspects of test management [23]. 12

16 The system is based on a SQL database which is accessed by the users through the web browser based client application. Figure 3.1: QC architecture [24] HP Quality Center at Sony Ericsson Mobile Communications Sony Ericsson uses QC as its main test management tool. All test case data and executions results are held in a globally accessible database. All test planning, test design and test execution is performed in the QC environment. New test cases are designed and created in QC by the tester based on the functions requirements. The test cases are stored in folders in the Test Plan database, organized under platform and function. For every new test session a new suite of test cases are selected from the Test Plan database and added to a new test suite in the Test Lab. The test suites are organized in folders, one root folder for each main project which in turn contains a folder for each test session. These test session folders contain folders for each function group and within these folders the test suites are stored. When a test session is performed all the test cases in the test suite should be executed. Each executed test case should contain the following information; Software version, hardware version, test status (Blocked, Failed, N/A, No Run, Not Completed, Passed), execution date and time, tester id and if the status is Failed a DMS number from the error report in Clear Quest [25]. The data needed to solve the problems in this master s thesis is found both in the Test Plan part of the system and in the Test Lab part. External applications development possibilities for HP Quality Center The Quality Center (QC) Open Test Architecture (OTA) Application Programming Interface (API) is a Component Object Model (COM) library. COM is a 13

17 software architecture that allows the components made by different software vendors to be combined into a variety of applications. It defines a standard for component interoperability and is not dependent on any particular programming language or platform. QC OTA API enables integration of external applications with QC, interaction with QC without having to use the GUI front-end and interaction directly with the QC database, bypassing the database administrator. It could be used with for example Microsoft Visual Basic, C++ or C#. Some examples of what could be accomplished with OTA programming [26]: Read information from a QC project for use by an external tool or report generator. Integrate other site tools and data with QC. For example have you project planning tool create new QC projects. Integrate with other applications such as MS Excel and Word. Creating test suites. Since the system is based on a SQL database the data could also be accessed directly through SQL queries. This could be useful when gathering historical data because of the possibility to create specific SQL queries to get the data presented in a desired way in result sets. But this should be avoided when submitting data, for example when creating new test suites, to ensure that all the default fields and tables are updated correctly. 14

18 Chapter 4 Current regression test case selection technique This chapter focuses on answering the two first research questions, listed in 1.2. First the method for gathering the information is described and then the results are presented. 4.1 Research To gather information about the current regression test case selection technique used at Sony Ericsson discussions were held with employed testers and technical project leaders. Some information was also based on my own experience after working in the environment for several months. The information about the current process, test case selection technique and the problems and disadvantages with the current situation are presented in section 4.2. In discussions with a test engineer currently working at the integration test department about the test process and the test selection technique the following discussion topics were discussed. How do you select the test cases to be run during a regression test? How do you evaluate how much time each test case will add to the whole test suite? What differentiate the test cases from your point of view? Do you feel confident that the correct suite has been selected every time? What is most important when running selected test suite; time/resources or quality? What do you think of the equation from [21]? 15

19 Another open discussion was held with a technical project leader at the department and the following was discussed: Which properties of the executed test cases should affect the prioritization equation in the test case selection technique? Define the development and test process. When is regression testing performed? What do you think of the equation from [21]? 4.2 Results How the test case selection currently is performed The tester has through experience gained knowledge about which areas that have detected defects in the recent regression test sessions and this is affecting his selection, the test cases covering these areas are prioritized. The current selection technique is performed by first selecting the basic test cases from each area, those that verifies that the basic functionality is working. Then those that executes the basic use case for the customer. In general function test cases are prioritized higher than system test cases (system test cases are test cases that exercise interaction with other functions). The test cases are also given a priority when they are created which will help in giving an indication about how important they are. In this manner the regression test suite is filled until the tester is satisfied that all the areas are sufficiently covered, the overall quality is maintained and the regression test suite is within the time and resource limit. The time and resource that each test case adds to the whole test suite is only known though experience. The total amount of test cases available for the function group where the thesis was performed is about 2500 and the normal suite of selected test cases for a regression test includes about test cases Disadvantages with the current technique The discussions and the research within the organization concludes that the currently used technique could lead to these problems and disadvantages: It requires senior testers with experience of the test area. The selected test suite could be too extensive or too narrow, a tester with lack of experience in the area could have trouble estimating the required time and resources. The selected test suite could be inefficient and misleading, since it is just based on judgment. The tester has to know which test cases that, recently, have been more prone to detect faults. 16

20 Even an experienced tester could select an ineffective test suite. If the test cases are selected in a routinely manner by just selecting the same test cases for every regression test session. Since the selection is based on judgment it is not statistically proven to be the most effective test suite. An analysis of these problems and disadvantages together with the research in the area of regression test case selection and prioritization led to the hypotheses that a semi automatic test case selection tool based on a history based test case prioritization technique offers a solution for the disadvantages in this verification process. 17

21 Chapter 5 Regression test case prioritization based on execution history and pre-set priority The following section describes the different methods that were used to find the answer to the third research question stated in chapter 1. First the methodology is presented and then the findings and development of the prioritization equation is presented. 5.1 Methodology Article search for applicable prioritization strategies To initiate the project and to get at a broader view of the research performed in the area an extensive information search was performed. Three main electronic sources where used, mainly ELIN [1] but also Inspec [2] and Compendex [3]. ELIN covers about journals, 500 databases and e-books [4] and Inspec and Compendex were used to refine the search [5]. Several title searches were performed in ELIN [1] generating a large amount of articles. A first selection was made based on their presumed relevance based on their title then out of this selection the final selection was made based on their presumed relevance based on their abstract chapter. The articles were presumed relevant if the title or abstract indicated that they were about test case selection or prioritization, specifically in a black box environment. Articles specifically about code based regression test case selection or prioritization were discarded. 18

22 Initially the keyword regression test selection were used which resulted in a final selection of eight articles that was selected for further investigation. Then the keyword regression test was executed and seven more articles were chosen. The results from the keywords black box test, regression test prioritization and testing was also searched through and three more relevant articles were selected. On top of this seven other articles were selected based on recommendation from Per Runeson (Prof) and Emelie Engström (PhD stud.) who both have extensive experience in the area. With background of the current black box testing environment and the available data at the integration test department in mind the first selection of the 25 articles were studied and three specific articles were chosen as most relevant to be applicable for this project, due to their specific focus on history based test case prioritization. Nine others were studied to gather general information about test case selection and prioritization. Five were discarded because they were specifically and only about code based test case selection and three more were discarded because they were found not to be applicable in the current black box testing environment, with the specific available input data. Four other were about processes for case studies, principles for software product line testing, analysis of fault distributions and managing software engineering and not found applicable in this project. The articles are presented and summarized in chapter 2 to give an overview of the research performed in the field and to describe how they were considered in this master s thesis project Development of the test case prioritization technique With the theoretical background information, presented in chapter 2, and the current situation with its disadvantages identified, presented in chapter 4, the work continued by more thoroughly investigating the selected articles in search for specific test case prioritization techniques that included the test cases historical performance, could fit this environment and would help solve the presented problems. As described in chapter 2 several articles were used as inspiration and one in particular used as a base for the solution to the problems. The available data in the test process at the integration test department was analyzed to get an overview of what was possible to use when prioritizing the test cases. Discussions with a test engineer and a technical project leader was held about what they would want to affect the selection of the test cases. The base equation combined with the analysis of the available data and the outcome of discussions with employees about a possible solution resulted in the proposed solution; a new regression test case prioritization technique. 19

23 5.2 Criteria for the test case prioritization technique In the discussions with the test engineer and the technical project leader, described in chapter 4, the following properties were found desired to be integrated into the test case prioritization equation. The first group contains test case properties that should affect the calculation of the prioritization value, the second group contains test case properties that should determine which test cases that should be included or excluded from the priortization calculations and the third group is properties that should affect the size of the selection. The test cases properties that should affect the prioritization value (in no specific order): 1. (a) The defect detecting frequency of the test case during a period of time. This property determines the test cases effectiveness, if a test case often reveal defects it could be an indication that it exercise parts of the software where new defects often appears. A test case that detects a defect is linked via an id to the error report in the error report database. But the reason why a test case often detects defects has to be considered. If it is an old defect that has not been fixed since the last session the test case should not be selected. Also a defect found with one test case can block other test cases and these test cases are also linked to the error report via an id and should not be selected until the defect has been fixed. (b) The number of regression test sessions that have been executed since the test case was executed last. It is important to ensure that all test cases eventually are executed over a period of time. So the number of sessions that has been executed without the test case should increases the test case s priority until it is executed in a regression test session. (c) The importance of the test case, for business purposes and for the overall functionality. This is to ensure that some basic test cases, that are important, are given higher priority. This property is set manually when the test case is created and should affect the priority in every selection. (d) The test case s creation date. To ensure that new functions are more thoroughly tested the creation date should be used to determine which test cases that is new. (e) If the test case should be executed automatically of manually. 20

24 This property should be used to estimate the execution time for each test case. An automated test case is assumed to consume less time and resources. The following properties were found desired to affect the selection of which test cases that should be prioritized: 2. (a) The test case type (for example Duration test, Performance test, Certification). This property should be used to determine which test cases that should be excluded from the prioritization equation, test cases that are marked for duration, performance and certification should not be executed in a regression test session. (b) What type of session the test case was intended for (Some test cases have a field that specifies this; Focus test, regression test, sanity test etc.). This factor is should also be used to exclude test cases that are not intended to be executed during a regression test session. (c) If the test case is written for a specific feature that is not available in all products (e.g. 2nd display, 3.5 mm plug). This property should be used to exclude test cases that are not applicable for the product about to be tested. 3. The importance of the targeted regression test session (the one that the test suite is selected for) were also found desired to be used. This property should affect how many test cases are selected from the prioritized list of test cases. 5.3 Inspiration and base equation The black box test environment at the integration test department requires a technique that does not depend on any information about the source code. The most research that has been made in the area of test case selection and prioritization are depending on knowledge about the source code and there are only a few that considered the black box testing approach. Among the articles that have been presented in chapter 2 an article presented by Fazlalizadeh et al. [21] specifically presents an equation for test case prioritization that seems to suit the environment at the integration test department and the data available there. This technique also focuses on the test cases historical performance, which was the main idea for this project when it started. The prioritization equation presented by Fazlalizadeh et al. [21] includes two of the desired test case properties that are listed in 5.2; how frequent the test case has revealed defects during a period of time and how many regression test sessions that has been executed since the test case were included, i.e. how long 21

25 ago was the test case executed measured in number of regression test sessions. It was therefore chosen as a base for the prioritization equation. The equation that is presented in the paper consists of these parts: Historical effectiveness: fc k ec k fc k is the number of times that the test case has failed and ec k is the number of test case executions during k number of test sessions. Execution history: h k Each time a test case is not executed, its execution history will be increased by one. Once the test case is executed, execution history becomes 0 and the operation is repeated. Previous priority: P R k 1 The last priority of the test case in the last priority calculation during the past executions. The initial priority, P R 0, is defined as the percentage of code coverage of the test case. Three control constants: α, β, γ To control the effects of the factors the constants α, β and γ can be changed. γ should be smaller than the other two constants, because h k increases one unit per time according to whether or not test cases executes. But the variables fc k ec k and P R k 1 are always between 0 and 1. The equation is constructed as follows 1 : P R k = α fc k ec k + β P R k 1 + γ h k 0 α, β, γ < 1, k The developed test case prioritization equation Due to lack of available data only a few of the test case properties that was desired for the prioritization calculation, listed in 5.2, could be integrated into a new equation. The proposals of which test case properties to include and how they could be integrated are based on the discussions with the testers about the properties, estimations made together with the testers of the importance of the properties and reflection about the present variables and their proportions in the base equation. 1 In the paper the P R k 1 on the right hand side is actually presented as P R k but this is assumed to be a typing error. 22

26 5.4.1 Included properties The test case properties historical effectiveness (5.2: 1a) and execution history (5.2: 1b) are already included in the base equation and are used in the same way. Since there is no data about the test cases code coverage P R 0 has to be defined as something else. The priority that is manually set when the test case is created is used to differentiate the test cases (5.2: 1c). The P R 0 value is selected so that it will add a proportional amount to the total priority sum and the term (β P R 0 ) will be between 0 and 1. Since P R 0 in the base equation is based on how much code the test case covers the values could not be larger than a few percentage, the test cases are given the following P R 0 value: 0.04 Priority 1 P R 0 = 0.02 Priority Priority 3 Since the base equation is a recursive equation that is based on the test cases execution history the other desired test case properties could not be included as constants in this recursive equation, this would lead to that these properties would soon dominate the total value. They are instead added separately to the priority value for each time the test cases are presented for selection. The test case s original priority (I k ) should affect the calculated priority (5.2: 1c), since the most important test cases always should be given higher priority. I k is set to three different values added separately to the recursively calculated priority value for every time a regression test case selection should be made. The value is selected with the same difference between the priorities as in P R 0 but the value is adjusted so that it adds a proportional value to P R k which is between 0 and 1: I k = 0.4 Priority Priority Priority 3 The creation date of the test case reveals new test cases and ensure that they are given higher priority (5.2: 1d). The limit for how long a test case is defined as new is set to 3 months. To increase the priority to these test cases a value is added separately to the recursively calculated priority value in the same way as the original priority. The value is chosen so that it will be proportional to the average value of P R k, between 0 and 1, and increase the priority as much as an Priority 1 test case: { 0.4 current date creation date < 3 months N k = 0 current date creation date > 3 months The importance of the regression test session is determined by the project leaders and the limit to the selected regression test case selection is set by the user in the application (5.2: 3). 23

27 If the test cases are marked as performance, duration or certification tests they are excluded from the test case prioritization equation (5.2: 2a). The test cases that are written for a specific feature that is not applicable for all products (e.g. 2nd display, 3.5 mm plug) should be excluded (5.2: 2c), but the information about this is often only specified in the test case s description text. In some cases this property could be found in the test case name and these are excluded from the prioritization equation (e.g. 2nd display) Excluded properties The following properties were found desired to be integrated in the the priority calculation but were due to different reasons not possible to include. To avoid selecting a test case that reveal a defect that has not been fixed yet the link to the error report should be checked. If the error report that is linked to the test case still is unresolved then the test case should not be selected (5.2: 1a). A factor that sets the whole priority to zero could be used. This link however could not be set up during the development of the prototype application due to lack of knowledge and time, so this property is excluded from the equation. The way the test cases are executed, automatically or manually, could affect the time and resources they add to the total amount of time and resources that the test case selection demands (5.2: 1e). However since there is no data about how much time and resources either of the types will consume this property is not included into the equation. The information about what specific session (5.2: 2b) (regression test, focus test, reception control) the test case is written for has not been added in the majority of the test cases at this time, therefore this is not included in the equation The smoothing constants The α and β constants were kept at the same value as in the base equation, 0.7 each. But the γ constant were set to a much smaller value, 0.001, since the execution history, h, in some cases is very large and will have an excessive effect against the other two factors The test case prioritization equation The equation consists of a base prioritization equation that calculates a priority value for each test case based on certain input data. The test case is given a new calculated priority for each performed regression test session, this is because the equation is recursive and needs the test case s last calculated priority. When a test case should be selected two extra types of values are added, one for the original priority and one for if the test case is a new test case. Theses extra values are added outside the recursive equation since they are not dependent on the execution history but always constant. 24