A choice relation framework for supporting category-partition test case generation

Transcription

1 Title A choice relation framework for spporting category-partition test case generation Athor(s) Chen, TY; Poon, PL; Tse, TH Citation Ieee Transactions On Software Engineering, 2003, v. 29 n. 7, p Issed Date 2003 URL Rights This work is licensed nder a Creative Commons Attribtion- NonCommercial-NoDerivatives 4.0 International License.; 2003 IEEE. Personal se of this material is permitted. However, permission to reprint/repblish this material for advertising or promotional prposes or for creating new collective works for resale or redistribtion to servers or lists, or to rese any copyrighted component of this work in other works mst be obtained from the IEEE.

2 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY A Choice Relation Framework for Spporting Category-Partition Test Case Generation T.Y. Chen, Pak-Lok Poon, Member, IEEE, and T.H. Tse, Senior Member, IEEE Abstract We describe in this paper a choice relation framework for spporting category-partition test case generation. We captre the constraints among varios vales (or ranges of vales) of the parameters and environment conditions identified from the specification, known formally as choices. We express these constraints in terms of relations among choices and combinations of choices, known formally as test frames. We propose a theoretical backbone and techniqes for consistency checks and atomatic dedctions of relations. Based on the theory, algorithms have been developed for generating test frames from the relations. These test frames can then be sed as the basis for generating test cases. Or algorithms take into consideration the resorce constraints specified by software testers, ths maintaining the effectiveness of the test frames (and hence test cases) generated. Index Terms Category-partition testing, choice relation framework, choice relation table, specification-based testing, test case constrction, test frame. æ 1 INTRODUCTION ACCORDING to varios athors [10], [14], [18], software testing is a labor-intensive and expensive process, which may accont for 50 percent of the total project cost. In order to improve on its effectiveness, testing shold be well planned and organized. In particlar, the constrction of test cases is an important aspect becase it affects the scope and, hence, the qality of the process [2], [8], [11]. This inspired varios researchers to develop test case constrction methods. Among them, Ostrand and Balcer [17] have developed the category-partition method (CPM). A category is defined as a major property or characteristic of a parameter or an environment condition. An example is the Accont Balance in a typical acconting application. Sch categories can easily be identified from the fnctional specification of a system. Each category is partitioned into a set of choices, which represent all the different kinds of vales that are possible for the category. Examples are Accont Balance 0 and Accont Balance < 0. Then, all valid combinations of choices are generated as test frames. Invalid combinations of choices are sppressed via varios constraints. Finally, test cases are constrcted from the generated test frames. Following p on the work of Ostrand and Balcer, several stdies on CPM were condcted. For instance, Amla and Ammann [1] and Ammann and Offtt. T.Y. Chen is with the School of Information Technology, Swinbrne University of Technology, Hawthorn 3122, Astralia. tychen@it.swin.ed.a.. P.-L. Poon is with the Department of Accontancy, The Hong Kong Polytechnic University, Hng Hom, Hong Kong. acplpoon@inet.poly.ed.hk.. T.H. Tse is with the Department of Compter Science and Information Systems, The University of Hong Kong, Pokflam Road, Hong Kong. tse@csis.hk.hk. Manscript received 10 Ag. 2001; revised 9 Jly 2002; accepted 18 Feb Recommended for acceptance by G. Rothermel. For information on obtaining reprints of this article, please send to: tse@compter.org, and reference IEEECS Log Nmber [2] stdied the viability of applying the method to Z specifications. Offtt and Irvine [1] investigated the faltdetection effectiveness of CPM when applied to objectoriented programs. Or stdy of CPM reveals the following problems: 1. All the constraints among choices mst be defined manally. This can be ineffective and prone to hman errors in real-life sitations where there are many sch constraints. 2. There is no precise mechanism for checking for consistency among constraints. This may affect the correctness and completeness of the test frames generated. 3. The generator for processing the test specification is meant to be rn repeatedly, with additional constraints being imposed in each rond, thereby redcing the nmber of test frames generated, ntil the software tester can afford to rn the test cases generated from test frames [17]. Sch an approach can be avoided if resorce constraints are considered dring, rather than after, the test frame generation process. To address these problems, we propose a choice relation framework to spport CPM. Or framework incldes the following featres:. a more rigoros approach for representing different types of constraints among individal choices,. consistency checks of specified constraints among choices,. atomatic dedctions of new constraints among choices whenever possible, and. a more effective test frame constrction process. The rest of this paper is strctred as follows: Section 2 otlines the major steps of CPM. Section 3 is the core of the paper, proposing a choice relation framework for CPM. Section 4 describes the work related to category-partition testing. Finally, Section 5 concldes the paper /03/$17.00 ß 2003 IEEE Pblished by the IEEE Compter Society

3 578 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY CATEGORY-PARTITION METHOD CPM is a specification-based testing techniqe developed by Ostrand and Balcer [17]. It helps software testers create test cases by refining the fnctional specification of a program into test specifications. It identifies the elements that inflence the fnctions of the program and generates test cases by methodically varying these elements over all vales of interest. The method consists of the following steps: 1. Decompose the fnctional specification into fnctional nits that can be tested independently. 2. Identify the parameters (the explicit inpts to a fnctional nit) and environment conditions (the state of the system at the time of exection) that affect the exection behavior of the fnction. 3. Find categories (major properties or characteristics) of information that characterize each parameter and environment condition. 4. Partition each category into choices, which inclde all the different kinds of vales that are possible for that category. 5. Determine the constraints among the choices of different categories. For example, one choice may reqire that another is absent or has a particlar vale.. Write the test specification (which is a list of categories, choices, and constraints in a predefined format) sing the test specification langage TSL. 7. Use a generator to prodce test frames from the test specification. Each generated test frame is a set of choices sch that each category contribtes no more than one choice. 8. For each generated test frame, create a test case by selecting a single element from each choice in that test frame. 3 CHOICE RELATION FRAMEWORK FOR CATEGORY-PARTITION TESTING Motivated by problems 1 to 3 of CPM as sggested in Section 1, we propose a choice relation framework to spport the method. Basically, or framework helps constrct test cases from fnctional specifications via the notion of a choice relation table. The intition of this table is to captre the constraints imposed on the choices by the specification. These constraints are expressed as relations between pairs of choices. They are essential information for the atomatic generation of test frames. Or approach consists of the following major steps: 1. Decompose the fnctional specification into fnctional nits that can be tested separately. 2. For every fnctional nit, identify its parameters and environment conditions and, hence, define the categories and their associated choices. 3. Constrct a choice relation table T for each fnctional nit. 4. For each T, constrct the corresponding choice priority table P, which captres the relative priorities for the se of the choices in generating test frames. 5. From each T and the corresponding P, constrct the set of test frames.. Create a test case from each generated test frame. Steps 1, 2, and above are identical to Steps 1, 2-4, and 8, respectively, of CPM described in Section 2. We shall therefore concentrate on Steps 3, 4, and 5 in or discssions in Sections 3.1, 3.2, and 3.3, respectively. 3.1 Constrction of the Choice Relation Table As mentioned above, the choice relation table T is intended to captre the constraints imposed by the specification on the choices. To constrct T, we need to determine the relation between each pair of choices. This is explained in the following sections Determination of Relations among Choices The steps prior to the constrction of T correspond to Steps 1-4 of CPM mentioned in Section 2 and, hence, detailed explanations are not repeated here. Instead, we shall simply illstrate the concepts of categories and choices throgh an example. Example 1 (Loan Example). Sppose a software tester is given the following specification: Develop a program loan for se by ABC Bank to process applications by its cstomers for personal loans, based on their employment and credit card details. In order to evalate an application, the program will accept the following details from the applicant. The evalation criteria are not specified here.. Employment Stats: Either Employed or Unemployed.. Type of Employment (if the applicant is working): Either Self-Employed or Employed by Others.. Type of Job (if the applicant is working): Either Permanent or Temporary.. Monthly Salary S (if the applicant is working): Either $0 <S $2; 000, $2; 000 <S $3; 000, or S >$3; Type of Applicant: Either Cardholder or Non-Cardholder.. Type of Credit Card (if the applicant is a cardholder): Either Gold or Classic.. Credit Limit (applicable only to a classic card): Either $2,000 or $3,000. It shold be noted that there is no credit limit for a gold card. Sppose the software tester decides that loan can be tested as a whole and ths frther break down into smaller fnctional nits is not reqired. Additionally, sppose the categories and their associated choices for loan are simply defined based on the above inpt details. For example, Employment Stats is defined as a category. Its two associated choices are Employment Stats = Employed and Employment Stats = Unemployed. When there is no ambigity, we can

4 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 579 simply refer to these two choices as Employed and Unemployed. 1,2 Before we proceed frther, let s define the concepts of test frames, valid choices, and relations among choices. Definition 1 (Test Frames and their Completeness). A test frame B is a set of choices. A test frame B is said to be complete if, whenever a single element is selected from each choice in B, a standalone inpt will be formed. Otherwise, it is said to be incomplete. Definition 2 (Set of Complete Test Frames Related to a Choice). Let TF denote the set of all complete test frames. Given any choice x, we define the set of complete test frames related to x as TFðxÞ ¼fB 2 TF : x 2 Bg. A choice x is valid if and only if TFðxÞ is nonempty. Different types of relations are possible for a given pair of valid choices, as illstrated in the following example: Example 2 (Relation between Two Choices). Consider the specification of the program loan in Example 1. TF( Classic ) contains all the complete test frames containing the choice Classic in the category Type of Credit Card. The relation between Classic and any other choice x can be one of three types:. x 2 B for any B 2 TFð}Classic}Þ. An example of x is the choice Cardholder nder the category Type of Applicant.. x 2 B for some, bt not all, B 2 TFð}Classic}Þ. Consider the following two complete test frames B 1 and B 2 2 TFð}Classic}Þ: B 1 ¼fUnemployed; Cardholder; Classic; $2; 000g; B 2 ¼fUnemployed; Cardholder; Classic; $3; 000g: Sppose x ¼ }$2; 000:} It is obvios from B 1 and B 2 that x appears in some, bt not all, B 2 TFð}Classic}Þ.. x 2 B for any B 2 TFð}Classic}Þ. An example of x is the choice Non-Cardholder nder the category Type of Applicant. Another example of x is the choice Gold nder the category Type of Credit Card. Becase of the importance of the above distinction, we define the three corresponding types formally as follows: Definition 3 (Relation between Two Choices). Given any valid choice x, its relation with another valid choice y (denoted by x 7! y) is defined in terms of one of three relational operators as follows: 1. x is flly embedded in y (denoted by x y) if and only if TFðxÞ TFðyÞ. 2. x is partially embedded in y (denoted by x P yþ if and only if TFðxÞ TFðyÞ and TFðxÞ\TFðyÞ ¼;. 1. One may arge that we need a frther category Stats of Cstomer Master File to reflect the environment condition, with three associated choices File Does Not Exist, File is Empty, and File is Not Empty. For simplicity of illstration, however, we shall only concentrate on the categories and choices defined for inpt parameters in this example. 2. It shold be noted that a choice can be defined for a range of vales. For example, $0 <S $2; 000, $2; 000 <S $3; 000, and S >$3; 000 are three possible choices for the category Monthly Salary ðsþ. 3. x is not embedded in y (denoted by x yþ if and only if TFðxÞ\TFðyÞ ¼;. The choice relations fll embedding and nonembedding in the above definition have straightforward meanings in ordinary logic and, hence, the motivation behind them is fairly obvios. On the other hand, the motivation behind the choice relation partial embedding merits some discssion. Example 3 (Motivation behind the Partial Embedding Relation). Consider the following simple specification in a typical credit card system: If Total Transaction Amont > $1,000, then add 200 bons points. If Average Transaction Amont > $100, then add 50 bons points. 1. According to Definition 3, the two choices Total Transaction Amont > $1,000 and Average Transaction Amont > $100 are partially embedded in each other. There is no logical relationship between them. However, this specification is important to the ser and sefl to the implementer. 2. The notion of partial embedding will be sefl for testing against problematic implementations sch as the following: If Total Transaction Amont > $1,000, then add 200 bons points else if Average Transaction Amont > $100, then add 50 bons points. Since the three types of choice relations in Definition 3 are exhastive and mtally exclsive, x 7! y can be niqely determined. It shold be noted that, immediately from the definition, the relational operator for x 7! x is and the relational operator for x 7! y is ifxand y are two different choices in the same category. Example 4 (Choice Relation Table). Consider the loan example again. The choice relation table T loan is constrcted and depicted in Fig. 1. Let w be the total nmber of choices. Let tði; jþ denote the element at the ith row and jth colmn of T loan, i; j ¼ 1; 2;...;w. For tð12; 14Þ (Self-Employed 7! Employed), the relational operator is becase Self-Employed always reqires Employment Stats to be Employed. For tð15; 10Þ (Unemployed 7! Permanent), the relational operator is becase Unemployed and Permanent are mtally exclsive, as the latter reqires Employment Stats to be Employed. For tð5; 15Þ (Cardholder 7! Unemployed), the relational operator is obviosly P. The relational operator for the remaining elements in Fig. 1 can be determined in a similar way. Readers may note that the choice relations defined in Definition 3 focs on the constraint between a pair of choices. In sitations where the relationships among three or more inpt variables have to be considered at the same time, we can define a single category that involves these

5 580 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 Fig. 1. Choice relation table for the loan example. inpt variables. Consider, for instance, the relationship ða þ B þ C ¼ 10Þ, where A, B, and C are inpt variables. In this case, we can define a single category A þ B þ C with two associated choices ¼ 10 and ¼ 10. It wold obviosly be qite inefficient and error-prone if all the relational operators in the choice relation table had to be defined manally, especially when the nmber of choices is large. To solve the problem, we have developed techniqes for consistency checks and atomatic dedctions of choice relations. These techniqes are rendered possible by a set of properties of the relational operators described in the next section Properties of Relations among Choices Some sefl properties for the three relational operators are stated in the following propositions and corollaries. Readers may refer to Appendix E for the proofs of these propositions and corollaries. Proposition 1 (Symmetry of the Nonembedding of Choices). For any pair of valid choices x and y, x y if and only if y x. Immediately from Proposition 1, we have the following corollary: Corollary 1 (Reverse of Fll and Partial Embedding of Choices). Let x and y be valid choices. 1) If x y, then y x or y P x. 2)Ifx P y, then y x or y P x. It shold be noted that the reslts in Corollary 1 cannot be narrowed down any frther. This is frther discssed in Appendix E of the paper. Proposition 2 (Fll Embedding of Choices). Let x, y, and z be valid choices. 1) If x y and y z, then x z. 2)Ifx y and x z, then y z or y P z. Proposition 3 (Fll Embedding and Nonembedding of Choices). Let x, y, and z be valid choices. 1) If x y and y z, then x z. 2)Ifx y and x z, then y P z or y z. Immediately from Propositions 1 and 3, we have the following corollary: Corollary 2 (Fll Embedding and Nonembedding of Choices). Let x, y, and z be valid choices. 1) If x z and y z, then x y. 2)Ify z and x y, then z P x or z x. Proposition 4 (Fll and Partial Embedding of Choices). Let x, y, and z be valid choices. 1) If x y and x P z, then y P z. 2) If x z and y P z, then y P x or y x. 3)Ify z and x P y, then z x or z P x. Proposition 5 (Partial Embedding and Nonembedding of Choices). Let x, y, and z be valid choices. 1) If x P y and y z, then x P z or x z. 2)Ifx P y and x z, then y P z or y z. Immediately from Propositions 1 and 5, we have the following corollary: Corollary 3 (Partial Embedding and Nonembedding of Choices). Let x, y, and z be valid choices. 1) If x P z and y z, then x P y or x y. 2)Ify P z and x y, then z P x or z x. Each of the above propositions and corollaries provides a certain scope of consistency checks for the relations among choices. For example, we know that x P y and y x cannot coexist or else it wold contradict Proposition 1. However, not all incorrectly defined relations can be identified as inconsistencies. For example, sppose x y and y x are correct bt somehow mistakenly defined as x y and y P x. This mistake is not inherently inconsistent.

6 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION System Dedction Rles for Choice Relations Consider again Propositions 1, 2.1, 3.1, and 4.1, and Corollary 2.1. The then parts of these propositions and corollary contain definite relations, which provide a basis for atomatic dedctions. Ths, when appropriate relations have been defined, other relations can be dedced. For example, once we know that x y and y z, we can conclde from Proposition 2.1 that x z. The effectiveness of atomatic dedctions, however, varies with the chronological order of defining the relations. The following example illstrates this point. Example 5 (Chronological Order of Defining Choice Relations). Consider the choices $2,000, Classic, and Permanent in Fig. 1 for the loan example. Sppose the relations ($2; 000 7! Classic), ($2; 000 7! Permanent), and (Classic 7! Permanent) have yet to be defined. If ($2; 000 Classic) and ($2; 000 P Permanent) are manally defined first, then (Classic P Permanent) can be dedced sing Proposition 4.1. On the other hand, if ($2; 000 Classic) and (Classic P Permanent) are manally defined first, then ($2; 000 P Permanent) mst still be manally defined since it cannot be dedced from any of the propositions or the corollary. In view of Example 5, we shall propose a heristic approach in Section to determine the chronological order of defining relations in order to improve on the effectiveness of atomatic dedctions. First, however, we discss some important system dedction rles that form the basis of or heristic approach. System Dedction Rle (1). Given x y, we shold next define y 7! z, z 7! y, x 7! z, and z 7! x if they have not yet been defined or dedced. If y z, then we can dedce x z by applying Proposition 2.1. If y z, then we can dedce x z by applying Proposition 3.1. If z y, then we can dedce x z by applying Corollary 2.1. If x P z, then we can dedce y P z by applying Proposition 4.1. If z x, then we can dedce z y by applying Proposition 2.1. System Dedction Rle (2). Given x P y, we shold next define x 7! z if it is not yet defined or dedced. If x z, then we can dedce z P y by applying Proposition 4.1. System Dedction Rle (3). Given x y, we shold next define z 7! x and z 7! y if they have not yet been defined or dedced. If z x, then we can dedce z y by applying Proposition 3.1. If z y, then we can dedce z x by applying Corollary 2.1. The above system dedction rles provide the basis of or heristic approach for the atomatic identification of the next relation to be defined Table Constrction As explained in Section 3.1.2, an incorrectly defined choice relation may not lead immediately to any inconsistency highlighted by Propositions 1-5 and Corollaries 1-3. Other choice relations may have been defined manally before the incorrect choice relation is identified. By this time, additional choice relations may have already been dedced atomatically by the system, some of which are erroneosly based on the original incorrect relation. It is therefore desirable to provide a fnction to the tester so that an incorrectly defined relation and the erroneosly dedced relations can be corrected dring the constrction of the choice relation table. This correction fnction will be discssed later in this section. Before that, we mst first explore the possible types of elements in the choice relation table T. Each element tði; jþ in T can be classified into one of the following three types:. A defined element if it is manally defined.. A dedced element if it is atomatically dedced.. A yet-to-be-defined element if it has not been defined or dedced. An element tðm; nþ in T is said to be an ancestor of another element tði; jþ if the former has been sed to dedce the latter, either directly or indirectly. When the ser identifies a defined element for correction, the system will also check (and amend if necessary) all the elements that have been dedced from it. When the system amends a dedced element, it will also check (and amend if necessary) all the relevant ancestors that have been defined. Ancestor information is therefore vital to the correction of incorrectly defined or dedced elements in T. We se an element relation table with parent linked lists to captre this information. Besides, this table will also serve as a gide for atomatically identifying the next relation that shold be manally defined, as explained in Section and frther elaborated below. Given w choices, the dimension of T is w w. A corresponding element relation table E has the same dimension. Each element of E, denoted by eði; jþ, consists of the following for fields:. Type: It contains an integer vale of 1, 0, or 1, indicating that the corresponding tði; jþ is a defined, yet-to-be-defined, or dedced element, respectively.. Parent-Pointer: An element tðm; nþ in T is called a parent of another element tði; jþ if the former is an immediate ancestor of the latter. The set of all parents of tði; jþ is represented by a parent linked list PL i;j. If tði; jþ is a dedced element and the corresponding pair of choices belong to different categories, then the parent-pointer contains the header address of the parent linked list. Otherwise, the parent-pointer is set to a nll vale.. Dedction-Pointer: If i ¼ j or the type is 0, then the dedction-pointer contains a nll vale. Otherwise, it contains the header address of a dedction linked list DL i;j. Each node of DL i;j contains a yet-to-bedefined element in T which, according to the three system dedction rles in Section 3.1.3, shold be a candidate for the next manal definition. Obviosly, DL i;j may be empty.. Conter: This field is sed to determine the relative chance of dedcing some other relations if we know the relational operator for the corresponding tði; jþ.it

7 582 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 TABLE 1 Contents of eði; jþ of Element Relation Table contains an integer vale 1. Intitively, a higher vale in this field indicates a greater chance of dedcing other relations when we know the relational operator. It shold be noted that the vale of this field is 1 if the corresponding tði; jþ has been defined or dedced, and this field may be pdated once T is pdated. Table 1 smmarizes the possible vales for the above for fields. Appendix A shows an algorithm bild table for the constrction of T. It incorporates the featres for atomatic dedctions and consistency checks. The main philosophy is to perform atomatic dedction for every yet-to-be-defined tði; jþ as far as possible and to perform consistency checks whenever a tði; jþ is manally defined. We note that: 1. The procedre correct operatorðþ will not only correct the erroneos elements selected by the ser, bt also all the dedced elements reslting from these erroneos elements. As a reslt, the ser need only contine constrcting the choice relation table from that point, rather than repeat the entire table constrction process. 2. For every incorrectly defined choice relation, the nmber of exections reqired to correct this relation as well as other related relations is in the order of w 4 in the worst case, where w is the total nmber of choices. Ths, if m choice relations have been defined incorrectly, then the nmber of exections is in the order of mw 4. The algorithm also atomatically identifies the next tði; jþ to be manally defined, as gided by the system dedction rles explained in Section It wold be sefl to otline the main idea behind this featre. Each element tði; jþ in T is associated with a conter at eði; jþ of E, which provides an estimate of the nmber of other relations that can be dedced. The higher the vale of the conter at eði; jþ, the greater will be the chance of the corresponding tði; jþ to be selected for the next manal definition so that the chance of dedcing other relations will be increased. This is illstrated by the following example: Example (System Dedction Rles). Assme that we are constrcting the choice relation table for the loan example, as shown in Fig. 1. Sppose that the choice relation ($2; 000 Gold) has jst been defined. According to System Dedction Rle 3 in Section 3.1.3, we shold next define (z 7! $2; 000) and (z 7! Gold), where z is a choice sch as Permanent. Sppose frther that (z 7! $2; 000) is yet-to-be-defined and (z 7! Gold) has been defined or dedced. Then, the conter vale for (z 7! $2; 000) will be increased by one, bt not that for (z 7! Gold). This will effectively increase the chance of (z 7! $2; 000) being selected for the next manal definition. We have bilt a prototype system implementing the algorithm bild table, in which previosly presented techniqes for consistency checks and atomatic dedctions have been incorporated. Fig. 2 shows the inpt screen for defining the relation between a pair of distinct choices. It also provides sers with the option of defining grop constraints by means of a single manal definition (see Step 5d of the procedre bild tableðþ). This will frther redce the nmber of manal definitions reqired. When sers select the option of grop constraint definitions in Fig. 2, for instance, the relational operator will not only be assigned to (Gold 7! $2; 000), bt also to (Gold 7! $3; 000). Fig. 3 depicts a system screen that alerts sers abot detected inconsistencies among relations and allows them to choose the erroneos relations to be removed. Fll- EmbedIn, PartEmbedIn, and NotEmbedIn in the figre represent the relational operators, P, and, respectively Empirical Stdies We have condcted empirical stdies to evalate the effectiveness of or table constrction techniqe and to compare or approach with the original CPM. Or stdies involve for real-life commercial specifications:. The specification S register is for the inventory registration modle of an inventory management system Fig. 2. Inpt screen for the relation between a pair of distinct choices.

8 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 583 the qantity for every type of meal to be prepared and loaded onto the aircrafts served by the company. Empirical Stdy 1: Effectiveness of Table Constrction Fig. 3. Inpt screen to trigger the correction of erroneos relations. sed by a grop of pblic hospitals. The main fnctions of the modle are to record inventory details, to captre parent-child relationships among the inventory items, and to generate bar-code labels.. S prchase is for a prchase-order generation modle sed by the same grop of pblic hospitals. The modle allows prchasing officers to add procrement information to replenishment reqests sent from varios hospital nits and generates prchase orders atomatically.. S inqiry is for an online telephone inqiry system sed in a large telecom company. The system handles more than 0,000 inqiries a day and spports varios modes of inqiries sch as incomplete name searches, alternative name searches, and complex phonetic searches.. S meal is for the meal schedling modle of a meal ordering system sed by an international airline catering company. The main fnction is to determine 1. Effectiveness of Atomatic Dedctions and Grop Constraint Definitions. First, we identified the categories and choices for the for specifications described above in the sal manner. For the categories and choices for each specification, we randomly generated five different initial seqences of choices. It shold be noted that the actal seqence of manally defined choice relations depends on the initial seqence of choices and the dedction heristics described in Sections and Table 2 shows the effectiveness of atomatic dedctions and grop constraint definitions in the algorithm bild table. On average, abot 41.9 percent of the choice relations were atomatically dedced. This atomatic dedction featre was not available in the original CPM, in which all the constraints among choices had to be defined manally. In addition, becase of grop constraint definitions, an extra 28.0 percent of choice relations need not be defined individally. As a reslt, the amont of hman effort was significantly redced only abot 30.1 percent of the total nmber of choice relations had to be specified manally. 2. Effectiveness of Consistency Checks. At the conclsion of the 20 trial rns, we experienced seven erroneos choice relations, as shown in the rightmost colmn of Table 2. All these problematic cases were defined at an intermediate stage of the table constrction process. Each error was detected by the consistency checking mechanism almost TABLE 2 Empirical Stdy 1: Effectiveness of Table Constrction

9 584 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 TABLE 3 Empirical Stdy 2: Comparison with the Original CPM immediately after the manal definition, rather than after a series of other manal definitions. A plasible explanation is that real-life specifications involve a fairly large nmber of categories and choices and, hence, a large nmber of choice relations. The chance of erroneosly defining a choice relation at an intermediate stage, and yet slipping throgh the consistency checking mechanism, is very slim. The immediate detection of inconsistencies cased by erroneosly defined choice relations also means that the nmber of related choice relations to be corrected by the procedre correct operatorðþ of the algorithm bild table can be kept to a minimm. This is becase, whenever an incorrect choice relation is defined, it will be detected and, hence, retracted immediately before it is erroneosly sed for dedcing other choice relations. As mentioned earlier, the nmber of exections reqired to correct m incorrectly defined choice relations is Oðmw 4 Þ in the worst case. Or actal experience shows, however, that the amont of work de to corrections is mch less than that portrayed by this formla becase:. Only six of the 20 trial rns involved erroneos manal definitions. Each erroneos case involved only one or two incorrectly defined choice relations.. In the six trial rns that involved erroneos manal definitions, the incorrect choice relations were detected almost immediately before they were sed for dedcing other relations. Hence, the nmber of steps involved in removing an error was mch less than Oðw 4 Þ. By virte of the consistency checking mechanism, all the choice relations are verified before the constrction of test frames, as described in Section 3.3 below. This proves to be very handy. For the original CPM, on the other hand, sers have to check manally for incorrectly defined constraints against test specifications. If some incorrect constraints are detected only after the exection of the associated generator, certain effort will be wasted. Empirical Stdy 2: Comparison with the Original CPM. It wold also be sefl to frther evalate the effectiveness of consistency checks in the choice relation framework by comparing or approach with the original CPM. The stdy involves two sbjects, to be referred to as Person-A and Person-B. Both of them have postgradate qalifications in compter science or information technology and have abot 10 years of working experience in indstry. Neither of the sbjects has been involved with the development of the original CPM or or choice relation framework. Before the stdy started, both sbjects were given the relevant sections of this paper as well as the literatre relating to the original CPM [4], [17] for self-stdy. We then gave them a sample specification and asked them to constrct a TSL specification in the original CPM and a choice relation table sing the prototype system. The exercise was followed by a thorogh discssion of the reslts. The idea was to familiarize them with CPM and or framework. After the initial training, Person-A first constrcted the choice relation tables followed by the TSL specifications for the specifications S register and S prchase. He also constrcted the TSL specifications followed by the choice relation tables for the specifications S inqiry and S meal. On the other hand, Person-B first constrcted the TSL specifications followed by the choice relation tables for S register and S prchase. She also constrcted the choice relation tables followed by the TSL specifications for S inqiry and S meal. Table 3 smmarizes the reslts of the stdy. It shows that the sbjects have inclded erroneos definitions in both the TSL specifications and the choice relation tables. We have two observations: 1. The nmber of error cases for TSL specifications (19) was not sbstantially different from that of the choice relation tables (15). The nmber of errors varied little independent of whether the TSL specifications or the choice relation tables were constrcted first. 2. All 15 error cases in the choice relation tables were corrected dring table constrction with the help of the consistency checking and correction mechanisms. Abot 8.7 percent of these errors were detected immediately. The rest were detected after the next manal definition. As a reslt, all the

10 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 585 definitions were correct at the conclsion of the table constrction processes. This featre was not available in the TSL specifications. None of the 19 errors were detected by the sbjects themselves. We note that, when erroneos constraint definitions are left ndetected, the reslting nmber of erroneos test frames is sally many times that of the incorrect definitions. 3.2 Constrction of the Choice Priority Table In most real-life sitations, resorce constraints are imposed on the software tester. Hence, not all complete test frames may be sed for generating test cases for a program. A possible approach is to define the relative priorities for the choices based on the software tester s expertise and experience in the application domain. In this way, the choices with higher priorities can first be sed to generate test frames, ths respecting both the resorce constraints and the relative importance of the choices. Users are reqested to define the following parameters after the choice relation table T has been constrcted: 1. Preferred Maximm Nmber of Test Frames M. Software testers are reqired to define a preferred maximm nmber of test frames M that they are willing to handle. The word preferred implies that M is not absolte, as the limit may be overwritten by the minimally achievable priority level m in Relative Priority of Every Choice. Given w choices, a choice priority table P with a dimension of w 2 is constrcted, captring the relative priority of every choice. Let pði; 1Þ and pði; 2Þ denote the first and second elements, respectively, of the ith row of P. Basically, pði; 1Þ contains a valid choice x i and pði; 2Þ contains a positive integer r i. The smaller the vale of r i, the higher will be the priority of the corresponding x i for inclsion as part of a test frame. Or framework assmes that the smallest vale of r i is Minimally Achievable Priority Level m. The definition of a minimally achievable priority level m allows the software tester to ensre that those x i s with r i m will always be selected for inclsion as part of a test frame, independent of whether the nmber of generated test frames exceeds M or not. Or framework assmes that m 0. In the sitation where M shold not be waived by m, the software tester shold set m to zero. In this way, M becomes the absolte maximm nmber of generated test frames. In the above, the vale of M is largely dependent on the testing resorces. The more the available resorces, the higher shold M be defined. As pointed ot in [12], [13], [15], 1) it wold be far more effective to have an idea of the kinds of falts that are most probable or most damaging and then to constrct test cases that are likely to reveal these significant falts and 2) this falt-gessing process depends largely on the software tester s expertise and experience. Smaller vales of r i, representing higher priorities, shold be assigned to those crcial choices x i that are likely to reveal the significant falts. Frthermore, m shold not be assigned a vale smaller than any of these r i. Example 7. Consider the program loan in Example 1. Sppose the software tester defines M to be 10 and assigns the relative priority for all the choices, as illstrated partially in Table 4. Now, sppose m is set to 3. In this sitation, the choices Employed, Unemployed, Self-Employed, and Employed by Others will always be sed for the constrction of test frames, regardless of whether the nmber of generated test frames exceeds 10 or not. 3.3 Constrction of Test Frames Test Frames and Their Relations According to Definition 1, a test frame consists of a grop of choices. Frthermore, a test frame B is complete if and only if, whenever a single element is selected from every choice in B, the reslt will constitte a standalone inpt. Consider, for instance, the following test frame for loan in Example 1: B 0 ¼fType of Applicant ¼ Cardholder; Type of Credit Card ¼ Classic; Credit Limit ¼ $2; 000g: is incomplete becase additional details sch as Employment Stats = Employed mst be spplied before we have sfficient information to generate a test case for exection. Althogh B 0 is incomplete, it may be a sbset of a complete test frame, sch as the following: B 0 TABLE 4 Choice Priority Table for Loan Example B ¼fType of Applicant ¼ Cardholder; Type of Credit Card ¼ Classic; Credit Limit ¼ $2; 000; Employment Stats ¼ Employed; Type of Employment ¼ Employed by Others; Type of Job ¼ Permanent; Monthly Salary ðsþ ¼ $0 <S $2; 000g: Definition 4 (Set of Complete Test Frames Related to a Given Test Frame). Let TF denote the set of all complete test frames. Given any test frame B 0, we define the set of complete test frames related to B 0 as the set TFðB 0 Þ¼ fb 2 TF : B 0 Bg. A test frame B 0 is valid if and only if TFðB 0 Þ is nonempty. It follows immediately from Definition 4 that a complete test frame mst be valid. Like the treatment of choices, the relations between valid test frames can also be classified into three types:

11 58 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 Fig. 4. Some relations between x and B ¼fy; zg when y P x and z P x. Definition 5 (Relation between Two Test Frames). Given any pair of valid test frames B 1 and B 2, 1. B 1 is flly embedded in B 2 (denoted by B 1 B 2 Þ if and only if TFðB 1 ÞTFðB 2 Þ. 2. B 1 is partially embedded in B 2 (denoted by B 1 P B 2 Þ if and only if TFðB 1 ÞTFðB 2 Þ and TFðB 1 Þ\TFðB 2 Þ¼;. 3. B 1 is not embedded in B 2 (denoted by B 1 B 2 Þ if and only if TFðB 1 Þ\TFðB 2 Þ¼; Properties of Relations among Test Frames In light of Definition 5, we can extend Propositions 1-5 and Corollaries 1-3. For example, Proposition 1 and Corollary 1 can be extended into the following dal versions: Dal Proposition 1 (Symmetry of the Nonembedding of Test Frames). For any valid test frames B 1 and B 2, B 1 B 2 if and only if B 2 B 1. Dal Corollary 1 (Reverse of Fll and Partial Embedding of Test Frames). Let B 1 and B 2 be valid test frames. 1) If B 1 B 2, then B 2 B 1 or B 2 P B 1.2)IfB 1 P B 2, then B 2 B 1 or B 2 P B 1. Propositions 2-5 and Corollaries 2-3 can be extended in a similar fashion. Readers may refer to Appendix D for a fll list of dal propositions and corollaries. Proposition (Generalization Property). Given any valid choice x, TFðfxgÞ ¼ TFðxÞ. Becase of Proposition, a choice x and the test frame fxg will be sed interchangeably in this paper Dedction of Relations among Test Frames Two very important special cases of B 1 7! B 2 are B 7! x and x 7! B. Propositions on these special cases are practically sefl for atomatic dedctions of relations. They will be discssed in this section. Given any valid choice x and any valid test frame B, we can only have three scenarios: 1) There exists some y 2 B sch that y x, 2) there exists some y 2 B sch that y x, and 3) y P x for every y 2 B. Proposition 7 below shows that these three scenarios are not only exhastive bt also mtally exclsive. Before we present Proposition 7, we need the following lemma. Lemma 1 (Fll Embedding Lemma). Given any valid test frames B 1 and B 2,ifB 1 B 2, then B 2 B 1. Proposition 7 (Mtal Exclsion of Flly Embedded and Nonembedded Choices). Let x be a valid choice and B be a valid test frame. We cannot have any y and z 2 B sch that y x and z x. The following proposition shows how we can niqely determine B 7! x and x 7! B when there exists some y 2 B sch that y x. Proposition 8 (Test Frame Containing Nonembedded Choice). Let x be a valid choice and B be a valid test frame. If there exists some valid choice y 2 B sch that y x, then B x and x B. The next proposition shows how we can niqely determine B 7! x and x 7! B when there exists some y 2 B sch that y x. Proposition 9 (Test Frame Containing Flly Embedded Choice). Let x be a valid choice and B be a valid test frame. If there exists some valid choice y 2 B sch that y x, then the following will hold: 1) B x. 2) If there exists some z 2 B sch that x P z, then x P B; otherwise, x B. The following example shows that we cannot niqely determine B 7! x and x 7! B when y P x for every y 2 B. Example 8 (Test Frame Containing Partially Embedded Choices Only). Let x be a valid choice and B ¼fy; zg be a valid test frame. Sppose y P x and z P x. In this case, B 7! x and x 7! B may take one of the forms depicted in Fig. 4. It can be seen from the figre that the relational operator for B 7! x and x 7! B may be, P, or. In spite of the above limitation, the next proposition shows that x 7! B can still be determined in some circmstances.

12 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 587 Proposition 10 (Test Frame Containing Flly Embedding Choices Only). Let x be a valid choice and B be a valid test frame. We have x B if and only if x y for every y 2 B. In smmary, except when y P x for every y 2 B, both the relational operators for B 7! x and x 7! B can be niqely determined sing Propositions 8 and 9. Even in the partial embedding case, x 7! B can still be determined sing Proposition 10 nder certain circmstances. Ths, a manal definition of relational operators is not reqired in all these cases. This proves to be very handy in test frame constrction, which will be discssed in Section Test Frame Constrction Or approach for the constrction of test frames is incremental, consisting of the following two steps: 1. For every nprocessed choice x, combine it with each test frame B to form one or more test frames B new s. Contine with this process ntil the resorce constraints in terms of M and m have been exceeded. a. The selection of x depends on its relative priority so that choices with higher priorities are selected first for combining with existing Bs to form B new s. b. The relational operators for B 7! x and x 7! B determine how to generate B new s. Hence, these operators are indispensable in test frame constrction. They are determined atomatically sing Propositions 8 and 9 except when y P x for every y 2 B. In the latter case, x 7! B can be determined sing Proposition 10 nder certain circmstances. Frthermore, when B consists only of a single choice y, B 7! x and x 7! B are effectively the same as y 7! x and x 7! y becase of Proposition. Hence, they can be fond directly from the choice relation table. 2. For every incomplete test frame B new generated in Step 1, extend it into a complete test frame. Before we present or constrction algorithm for test frames, we have to discss the constrction rles first. These rles help s to generate new test frames in an incremental manner. Given a valid choice x and a valid test frame B, it follows from Dal Proposition 1 and Dal Corollary 1 that the only possible relations between x and B are 1. B x and x B, 2. B x and x B, 3. B x and x P B, 4. B P x and x B, and 5. B P x and x P B. If we know the relations between x and B, we can se the following rles to constrct new test frames: Constrction Rle 1. If B x and x B, then any complete test frame containing x will not contain B and any complete test frame containing B will not contain x. To generate all the complete test frames for this sitation, we need to retain the original test frame B and constrct a new test frame fxg. Constrction Rle 2. If B x and x B, then any complete test frame containing B will contain x and vice versa. To reflect the sitation, we replace B by a new test frame B [fxg. Constrction Rle 3. If B x and x P B, then any complete test frame containing B will contain x bt not vice versa. To reflect the sitation, we replace B by new test frames fxg and B [fxg. Constrction Rle 4. If B P x and x B, then any complete test frame containing x will contain B bt not vice versa. To reflect the sitation, we need to retain the original test frame B and constrct a new test frame B [fxg. Constrction Rle 5. If B P x and x P B, then complete test frames containing B may or may not overlap with complete test frames containing x. To generate all the complete test frames for this sitation, we retain the original test frame B and constrct new test frames fxg and B [fxg. Let x be a valid choice and B be a valid test frame. If there exists some valid choice y 2 B sch that y x, then by Proposition 8 and Constrction Rle 1, we need to retain the original test frame B and constrct a new test frame fxg. If there exists some valid choice y 2 B sch that y x, then, by Proposition 9 and Constrction Rles 2 and 3, we shold replace B by a new test frame B [fxg and add a new test frame fxg if appropriate. If x y and y P x for every y 2 B, then, by Proposition 10, Constrction Rle 2 or 4 shold apply. Since every test frame generated by Constrction Rle 2 will also be generated by Constrction Rle 4 bt not vice versa, we recommend applying Constrction Rle 4 in order to play it safe and avoid possible omissions. Ths, we shall retain the original test frame B and constrct a new test frame B [fxg. For the case where y P x for every y 2 B and no other information is known, we recommend applying Constrction Rle 5 so as to play it safe and avoid possible omissions since every test frame generated by any of Constrction Rles 1 to 4 will also be generated by Constrction Rle 5 bt not vice versa. Ths, the original test frame B will be retained and new test frames fxg and B [fxg will be constrcted. By Proposition 7, the above scenarios are mtally exclsive and, hence, the recommended procedre is well-defined. Appendix B shows the algorithm bild test frame for the incremental constrction of test frames. As we can see from this algorithm, the nmber of exections is in the order of c w M in the worst case, where c is the total nmber of categories, w is the total nmber of choices, and M is the absolte maximm nmber of generated test frames. On the completion of bild test frame, all the test frames generated are stored in K. Frthermore, becase of Step 2.2.b.v in bild test frameðþ, all the generated test frames are distinct Test Frame Extension In the algorithm bild test frame, a preferred maximm nmber of test frames M and a minimally achievable priority level m are sed by the software tester to specify the resorce constraints. As a reslt, some of the choices

13 588 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 may remain nprocessed or partially processed when the algorithm is terminated becase of resorce limitations. In this way, some test frames in K may not be complete. Obviosly, these incomplete test frames shold be extended frther. Given any incomplete test frame B, we wold like to extend it to inclde valid choices x that remain nprocessed or partially processed, as long as B and x do not mtally exclde each other. The extension rle can easily be formlated sing the relation B 7! x, as follows: Extension Rle. When B is flly or partially embedded in x, extend B into B [fxg. When B is not embedded in x, do not change B. We observe that the nmber of test frames will remain nchanged when the extension rle is applied. Hence, we can preserve the constraint on the nmber of generated test frames as imposed by M and m. To determine B 7! x for the extension rle, we note the following: 1. If B consists only of a single choice, we can determine B 7! x from the choice relation table. 2. Otherwise, if there exists some valid choice y 2 B sch that y x, then, by Proposition 8, we have B x. 3. Otherwise, if there exists some valid choice y 2 B sch that y x, then, by Proposition 9, we have B x. 4. Otherwise, if x y for every valid choice y 2 B, then, by Proposition 10, we have x B. Hence, by Dal Corollary 1.1, we mst have B x or B P x. For those cases not covered above, sers will be reqested to define B 7! x manally. Based on the above logic, we have developed an algorithm called extend test frame (as shown in Appendix C), which extends every incomplete test frame generated by bild test frame. For this algorithm, the nmber of exections involved is on the order of c w N in the worst case, where c is the total nmber of categories, w is the total nmber of choices, and N is the total nmber of generated test frames. Once we have a set of complete test frames, the generation of test cases is straightforward. Given any complete test frame B, we randomly select a single element from each choice contained in B. The set of elements ths selected will constitte a test case corresponding to B. Consider, for example, the following complete test frame generated by the algorithm extend test frame: B 1 ¼fType of Applicant ¼ Cardholder; Type of Credit Card ¼ Classic; Credit Limit ¼ $2; 000; Employment Stats ¼ Employed; Type of Employment ¼ Employed by Others; Type of Job ¼ Permanent; Monthly Salary ðsþ¼ $0 <S $2; 000g: The following may be selected as a test case corresponding to B 1 : Type of Applicant ¼ Cardholder; Type of CreditCard ¼ Classic; Credit Limit ¼ $2; 000; Employment Stats ¼ Employed; Type of Employment ¼ Employed by Others; Type of Job ¼ Permanent; Monthly Salary ðsþ ¼ $1; 358: 3.3. Merits of Test Frame Constrction Or choice relation framework spports CPM mainly in 1) consistency checks and atomatic dedctions of choice relations and 2) the atomatic bt constrained generation of test frames. The effectiveness of consistency checks, atomatic dedctions, and grop constraint definitions has been discssed in Section The approaches in constraining the total nmber of generated test frames can be compared as follows:. In the original CPM, special annotations [error] and [single] can be attached to a choice x so that a complete test frame containing only x will be generated [4], [17]. The [error] annotation is designed to test a particlar vale that will case an exception or other error state. It is assmed that any call of the fnction with this particlar vale in the annotated parameter or environment condition will reslt in the same error. On the other hand, the [single] annotation is intended to describe special, nsal, or redndant conditions that do not need to be combined with other possible choices. The main prpose of sing these two annotations is to redce the nmber of complete test frames generated. Given a special choice x, this objective can also be achieved throgh or framework by defining all the choice relations x 7! y as x y for any choice x ¼ y. In this case, x will not be combined with any other choices to form complete test frames. Instead, a single complete test frame containing only x will be generated.. In the original CPM, the nmber of generated test frames can only be redced by means of incorporating additional constraints among choices. As a reslt, the tester does not have direct control of the exact nmber of test frames generated. After all the constraints have been taken into consideration, frther redction will not be possible even if the nmber of generated test frames is still too large for the available testing resorces. On the contrary, or framework provides a means of frther redcing the nmber of test frames after all the choice relations (that is, constraints) have been considered. This is achieved sing M, m, and the relative priorities of individal choices, as explained in Section RELATED WORK It wold be worth reviewing other work related to the original CPM and or choice relation framework: 1. Amla and Ammann [1] sggest that CPM is applicable to natral-langage fnctional specifications, which may be incomplete and nstrctred. Software testers will need nde effort to define testing reqirements, ths hampering the effectiveness of

14 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 589 the method. On the other hand, they arge that testing reqirements are, to a large extent, already captred in formal specifications. They analyze the feasibility of applying CPM to Z specifications and verify that testing reqirements can be derived from formal specifications more easily. 2. Following p on the above stdy, Ammann and Offtt [2] define a minimal coverage criterion, called the base-choice-coverage criterion, for category-partition testing. They develop a procedre for converting Z specifications into test specifications that satisfy this criterion and introdce a method to prodce test scripts from the test specifications. 3. Using the notion of test templates, Stocks and Carrington [19] develop a nified, flexible, and formal framework for specification-based testing. Their framework provides not only a formal model of tests and test sites, bt also a method for applying the model in testing. In this way, test sites can be constrcted in a concise and formal manner. They also investigate several application areas of the framework, inclding test oracles, refinement, and regression testing. 4. Zeil and Wild [20] observe that test descriptions generated by testing criteria are, effectively, sets of constraints that define test cases. Soltions to a set of constraints correspond to actal test data. 3 There may, however, be more that one soltion and a common practice is to choose one of them arbitrarily. Zeil and Wild arge that some soltions may have a higher probability of revealing failres. Hence, they sggest a refinement process to redce the soltion set with the aim of identifying test data with a higher failre-revealing capability. Refinement is achieved by imposing frther constraints incrementally. 5. Offtt and Irvine [1] investigate the effectiveness of falt detection in object-oriented programs sing test cases generated by CPM. Common types of falts in C++ programs are identified. Sch falts are inserted into two programs. Test cases are then generated sing CPM with a view to ncovering these seeded falts. Their reslts show that these test cases help identify almost all the falts, except those involving memory management. They propose that C++ programs can be tested effectively by combining CPM with a tool for detecting memory management falts. They frther conclde that traditional testing techniqes, sch as CPM, are also effective for testing object-oriented programs and, hence, software developers do not need new testing methods in the object-oriented paradigm.. Grochtmann and Grimm [11] propose a classificationtree method to help constrct test cases from fnctional specifications. In this method, a classification tree, which is in the form of a hierarchical strctre, organizes classifications and classes at alternate levels. 4 The basic approach of the classification-tree method is very similiar to that of CPM, 3. Note that the term test case sed by Zeil and Wild corresponds to test frame in or choice relation framework, while their term test data corresponds to test case in or terminology. 4. Classifications and classes in the classification-tree method correspond to categories and choices in CPM, respectively. namely, to bild a model of the constraints in the inpt domain with a view to generating all the valid test frames while sppressing invalid ones as far as possible. Chen et al. [7] frther stdy how to improve on the tree strctre to facilitate the constrction of test frames. 7. Previos work has also been done on test case prioritization. For example, Avritzer and Weyker [3] develop load testing strategies to generate test sites to check the resorce allocation behavior of software systems according to operational profiles. Elbam et al. [9] stdy version-specific test case prioritization techniqes in regression testing, with a view to improving the rate of falt detection. The major difference between approaches 1-3 and ors is that the former are based on formal specifications. In or project, rather than formalizing the specification langage, we attempt to improve on CPM by proposing a rigoros and systematic framework. On the other hand, both approach 3 and ors provide a formal framework to systematically define test sites for specification-based testing. In 3 and 4, the notion of refinement has been sed to derive test cases. Or framework can also spport the generation of test cases via refinement, sch as by splitting choices or imposing additional constraints on choices. It will be interesting to investigate, as ftre research, how the concept of refinement can be sed to enhance the choice relation framework so as to frther facilitate the constrction of test cases from test frames. With regard to 5, we note that discssions on the testing of object-oriented software are beyond the scope of the crrent paper. Readers may refer to [5], [] for or perspective on this topic. CPM and differ in how invalid test frames are modeled and sppressed. CPM achieves this by captring constraints in textal form, whereas represents constraints in the form of tree strctres. It has been fond that the latter approach may not be applicable to every scenario. Finally, the work highlighted in 7 stdies the generation of test sites to cover states in proportion to their se [3], or the prioritization of existing tests in regression testing [9]. On the other hand, or work addresses prioritization from the perspective of the specification-based CPM. 5 CONCLUSION In this paper, we have developed a choice relation framework for spporting category-partition test case generation. The major merits of the framework are: 1. We captre the constraints among choices in a rigoros and systematic manner via the introdction of varios relations. 2. We improve on the effectiveness and efficiency of complete test frame constrction by means of consistency checks and atomatic dedctions of relations. 3. We provide a means of removing only the incorrectly defined relations and any related ones, thereby saving the effort of repeating the entire constrction process for the choice relation table. 4. We provide a direct way to control the maximm nmber of generated test frames.

15 590 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY We enable the software tester to specify the relative priorities for choices that are sed for the sbseqent formation of complete test frames. We have applied or approach to real-life sitations and reported on the effectiveness of consistency checks and atomatic dedctions of choice relations. APPENDIX A ALGORITHM FOR CONSTRUCTING THE CHOICE RELATION TABLE Procedre bild tableðt Þ 1. Initialization of Bffer of Manally Defined Relations BR and List of Previosly Defined and Dedced Relations LR a. Initialize BR as an empty linked list. It will be sed for storing temporarily the choice relations defined manally by sers. b. Initialize LR as an empty linked list. Each element of LR is a linked list that captres the contents of the choice relation table T immediately before the system carries ot a correction of erroneos choice relations. 2. Initialization of Element Relation Table E 3. Initialization of Diagonal Elements 4. Initialization of Elements for Different Choices of the Same Category. For every element tði; jþ in T sch that i ¼ j and the corresponding pair of choices belong to the same category, initialize tði; jþ and pdate the relevant entries and dedction linked lists in E sing the procedre pdate dedction detailsði; jþ. 5. Updating of Choice Relation Table T. Repeat the following ntil all the elements tði; jþ in T have been defined or dedced: a. Choose an element tði; jþ with the largest conter vale of eði; jþ. If there is more than one sch tði; jþ, then arbitrarily choose one of them.. If the relational operator for tði; jþ appears in the bffer BR, then move it to T.. Otherwise, prompt the ser to define the relational operator for tði; jþ and store it in T. b. Set the type and the conter of eði; jþ to 1. c. Update the relevant conters and dedction linked lists in E sing the procedre pdate dedction detailsði; jþ: d. Sppose tði; jþ in Step 5.a corresponds to the choice relation x 7! y, where y ð¼ xþ is nder the category Y. If, for every choice y 0 ð¼ yþ nder Y, the choice relation x 7! y 0 is not fond in T and BR, then confirm with the ser whether the relational operator for tði; jþ shold also be applied to all sch x 7! y 0. If so, store the relational operators for all x 7! y 0 into the bffer BR. e. Perform consistency checks for all the defined and dedced elements in T sing the propositions and corollaries of Section If no inconsistency is detected, then proceed to Step 5f. Otherwise, check whether the combination of defined and dedced elements in T exists in the linked list LR. If so, alert the sers that they have encontered this problem before and prompt them to define another relational operator for tði; jþ. Otherwise, perform the following:. Append a copy of the contents of T to the linked list LR.. For any set of inconsistent elements S ¼ ftðm 1 ;n 1 Þ;tðm 2 ;n 2 Þ;...;tðm k ;n k Þg in T, alert the sers abot the following: i. every element in S that is manally defined, ii. every element in S that is atomatically initialized in Step 4 of this procedre, and iii. the manally defined ancestor(s) of every element in S that is atomatically dedced in Step 5f of this procedre. Then, prompt the ser to select the erroneos ones from i and iii above.. Correct the selected elements sing the procedre correct operatorðþ. Repeat Step e ntil no inconsistency is detected. f. Whenever possible, perform atomatic dedctions for yet-to-be-defined elements, sing Propositions 1, 2.1, 3.1, and 4.1, and Corollary 2.1. For every element tðp; qþ whose relational operator has been dedced:. Set the type of eðp; qþ to 1 and the corresponding conter to 1.. Update the relevant conters and dedction linked lists in E sing the procedre pdate dedction detailsðp; qþ.. Initialize PL p;q as an empty linked list, and store its header address in the parentpointer of eðp; qþ.. Append each parent element of tðp; qþ to PL p;q. Procedre pdate dedction detailsði; jþ 1. Initialize DL i;j as an empty linked list and store its header address in the dedction-pointer of eði; jþ. 2. Identify all the elements tðm; nþ of T sch that the system dedction rles in Section can be applied. For every sch tðm; nþ: a. If the conter in the corresponding eðm; nþ is smaller than the largest integer vale spported by the system, then increase it by one and b. Append a node ðm; nþ todl i;j. 3. For every defined or dedced element tðp; qþ in T, delete the node ði; jþ, if any, from the corresponding DL p;q.

16 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 591 Procedre correct operatorðm; nþ 1. Delete the following from T : a. the manally defined element tðm; nþ and b. all the dedced elements, if any, that have tðm; nþ as an ancestor. Note that the ancestor information of an element can be obtained from its parent linked list. To delete an element tðp; qþ from T : i. Set the type of the corresponding eðp; qþ to 0 and its parent-pointer to nll. ii. Set the conter of the corresponding eðp; qþ to the largest integer vale spported by the system. This ensres that tðp; qþ has the highest priority for the next manal definition. iii. For every element tðx; yþ of T corresponding to an element ðx; yþ of the dedction linked list DL p;q, decrease the conter of eðx; yþ by one. iv. Set the dedction-pointer of eðp; qþ to nll. 2. Prompt the ser to define the relational operator for tðm; nþ and store it in the bffer BR. APPENDIX B ALGORITHM FOR GENERATING TEST FRAMES Procedre bild test frameðt ; P; KÞ. In this procedre, a linked list K is sed to store all the generated test frames. In terms of data strctres, each element of the linked list K points to a linked list whose elements are the choices of a test frame. 1. Initialization 1.1. Inpt the preferred maximm nmber of complete test frames M and the minimally achievable priority level m Sppose r denotes the relative priority of a choice in the choice priority table P. Set the crrent priority level (denoted by L) to the minimm r defined Initialize K as an empty linked list. 2. Constrction of Possible Test Frames. Let NðKÞ denote the total nmber of test frames stored in K. While (there exists any nprocessed choice) and ðl m or NðKÞ < MÞ, do the following: 2.1. Select an nprocessed choice x with priority level L If NðKÞ ¼ 0, then store the test frame fxg into K. Otherwise, perform the following: a. Initialize K new as an empty linked list. This linked list has the same strctre as K. b. Let NðK new Þ denote the total nmber of test frames stored in K new. While NðKÞ > 0 and ðl m or ðnðkþ þ NðK new Þ < MÞÞ, do the following: i. Select any test frame B from K. ii. Determine B 7! x and/or x 7! B sing Propositions 8, 9, and 10. Then, generate new test frames for K new according to the recommendations in Section iii. Store all the newly generated test frames in K new. If, according to the constrction rle, B shold be retained, then store it into K new also. iv. Delete B from K. v. Remove all bt one dplicated test frame from K new. c. For every B new in K new, move it to K ntil NðK new Þ¼0 or ðl >m and NðKÞ MÞ Set L to the smallest relative priority of all the nprocessed choices, if any. APPENDIX C ALGORITHM FOR EXTENDING INCOMPLETE TEST FRAMES Procedre extend test frameðt ; P; KÞ. For every nprocessed or partially processed choice x in the choice priority table P, repeat the following for all the test frames B in K: 1. If B consists only of a single choice, determine B 7! x from the choice relation table. Extend B into B [fxg if B x or B P x. 2. Otherwise, if there exists some valid choice y 2 B sch that y x, then take no action. 3. Otherwise, if there exists some valid choice y 2 B sch that y x, then extend B into B [fxg. 4. Otherwise, if x y for every valid choice y 2 B, then extend B into B [fxg. 5. Otherwise, prompt the ser for B 7! x. Extend B into B [fxg if B x or B P x. APPENDIX D DUAL PROPOSITIONS AND COROLLARIES Dal Proposition 1 (Symmetry of the Nonembedding of Test Frames). For any valid test frames B 1 and B 2, B 1 B 2 if and only if B 2 B 1. Dal Corollary 1 (Reverse of Fll and Partial Embedding of Test Frames). Let B 1 and B 2 be valid test frames. 1) If B 1 B 2, then B 2 B 1 or B 2 P B 1.2)IfB 1 P B 2, then B 2 B 1 or B 2 P B 1. Dal Proposition 2 (Fll Embedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 B 2 and B 2 B 3, then B 1 B 3.2)IfB 1 B 2 and B 1 B 3, then B 2 B 3 or B 2 P B 3. Dal Proposition 3 (Fll Embedding and Nonembedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 B 2 and B 2 B 3, then B 1 B 3. 2) If B 1 B 2 and B 1 B 3, then B 2 P B 3 or B 2 B 3. Dal Corollary 2 (Fll Embedding and Nonembedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 B 3 and B 2 B 3, then B 1 B 2. 2) If B 2 B 3 and B 1 B 2, then B 3 P B 1 or B 3 B 1. Dal Proposition 4 (Fll and Partial Embedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 B 2 and B 1 P B 3, then B 2 P B 3.2)IfB 1 B 3 and B 2 P B 3,

17 592 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 29, NO. 7, JULY 2003 then B 2 P B 1 or B 2 B 1.3)IfB 2 B 3 and B 1 P B 2, then B 3 B 1 or B 3 P B 1. Dal Proposition 5 (Partial Embedding and Nonembedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 P B 2 and B 2 B 3, then B 1 P B 3 or B 1 B 3. 2) If B 1 P B 2 and B 1 B 3, then B 2 P B 3 or B 2 B 3. Dal Corollary 3 (Partial Embedding and Nonembedding of Test Frames). Let B 1, B 2, and B 3 be valid test frames. 1) If B 1 P B 3 and B 2 B 3, then B 1 P B 2 or B 1 B 2.2)IfB 2 P B 3 and B 1 B 2, then B 3 P B 1 or B 3 B 1. APPENDIX E PROOFS AND DISCUSSIONS OF LEMMA, PROPOSITIONS, AND COROLLARY Proof of Proposition 1 (Symmetry of the Nonembedding of Choices). This proposition follows directly from the definition of. t Discssions on Corollary (Reverse of Fll and Partial Embedding of Choices). Given x y, the set of possible complete test frames may be divided into the following three disjoint partitions:. Pðx ^ yþ: Its complete test frames mst contain x, y, and possibly other choices.. Pðy ^:xþ: Its complete test frames mst contain y and possibly some other choices, bt not x.. P ð:x ^:yþ: Its complete test frames contain neither x nor y. Partition P ðx ^ yþ cannot be empty becase x is a valid choice and x y. On the other hand, P ðy ^:xþ or Pð:x ^ :yþ may be empty. If P ðy ^:xþ is empty, we have y x. Otherwise, we have y P x. Now, sppose x P y. In this case, the set of possible complete test frames may be divided into the following for disjoint partitions:. Pðx ^ yþ: Its complete test frames mst contain x, y, and possibly other choices.. Pðx ^:yþ: Its complete test frames mst contain x and possibly some other choices, bt not y.. Pðy ^:xþ: Its complete test frames mst contain y and possibly some other choices, bt not x.. P ð:x ^:yþ: Its complete test frames contain neither x nor y. Partitions Pðx ^ yþ and P ðx ^:yþ cannot be empty becase x is a valid choice and x P y. On the other hand, P ðy ^:xþ or P ð:x ^:yþ may be empty. If P ðy ^:xþ is empty, we have y x. Otherwise, we have y P x. Proof of Proposition 2 (Fll Embedding of Choices). 1. The proof follows directly from the definition of. 2. Sppose x y and x z. By Definition 3, TFðxÞ TFðyÞ and TFðxÞTFðzÞ. Hence, any complete test frame B 2 TFðxÞ is also in TFðyÞ and TFðzÞ. Since x is valid, TFðxÞ is nonempty. Ths, TFðyÞ\TFðzÞ ¼;, which means that y z cannot be tre. In other words, y z or y P z. t Proof of Proposition 3 (Fll Embedding and Nonembedding of Choices). 1. The proof follows directly from the definition of and. 2. Sppose x y and x z. Let s assme y z. It wold follow from Proposition 2.1 that x z, which wold contradict x z. Hence, we mst have y P z or y z. t Proof of Proposition 4 (Fll and Partial Embedding of Choices). 1. Sppose x y and x P z. If we assmed y z, then it wold follow from Proposition 2.1 that x z, which wold contradict x P z. On the other hand, if we assmed y z, then it wold follow from Proposition 3.1 that x z, which wold also contradict x P z. Hence, we mst have y P z. 2. Sppose x z and y P z. Let s assme y x. It wold follow from Proposition 2.1 that y z, which wold contradict y P z. Hence, we mst have y P x or y x. 3. Sppose x P y and y z. Let s assme z x. By Proposition 3.1, we wold have y x. It wold follow from Proposition 1 that x y, which wold contradict x P y. Hence, we mst have z x or z P x. t Proof of Proposition 5 (Partial Embedding and Nonembedding of Choices). 1. Sppose x P y and y z. Let s assme x z. It wold follow from Corollary 2.1 that x y, which wold contradict x P y. Hence, we mst have x P z or x z. 2. Sppose x P y and x z. Let s assme y z. It wold follow from Corollary 2.1 that y x, which wold contradict x P y according to Proposition 1. Hence, we mst have y P z or y z. t The proofs of Dal Propositions 1 to 5 are similar to those of Propositions 1 to 5. Proof of Proposition (Generalization Property). Given any valid choice x, for any B 2 TF, B 2 TFðfxgÞ, fxg B, x 2 B, B 2 TFðxÞ. t Proof of Lemma 1 (Fll Embedding Lemma). Sppose B 1 B 2. For any B 2 TFðB 2 Þ, by Definition 4, we mst have B 2 TFðB 1 Þ. By Definition 5, therefore, we have B 2 B 1. t Proof of Proposition 7 (Mtal Exclsion of Flly Embedded and NonEmbedded Choices). Assme there exists a valid choice y 2 B sch that y x. By Lemma 1, we have B y. Hence, by Dal Proposition 2.1, we have B x. Assme there exists a valid choice z 2 B sch that z x. By Lemma 1, we have B z. Hence, by Dal Proposition 3.1, we have B x. Ths, we have a contradiction. t Proof of Proposition 8 (Test Frame Containing Nonembedded Choice). Since y 2 B, by Lemma 1, we have B y. Hence, by Dal Proposition 3.1, if y x, then B x. By Dal Proposition 1, we can also conclde that x B.t

18 CHEN ET AL.: A CHOICE RELATION FRAMEWORK FOR SUPPORTING CATEGORY-PARTITION TEST CASE GENERATION 593 Proof of Proposition 9 (Test Frame Containing Flly Embedded Choice). 1. Since y 2 B, by Lemma 1, we have B y. Hence, by Dal Proposition 2.1, if y x, then B x. 2. Sppose there exists some z 2 B sch that x P z. By Lemma 1, we have B z. Hence, by Dal Proposition 4.2, we have x P B or x B. Now, according to part 1 of this proposition, we have B x. Hence, by Dal Proposition 1, we cannot have x B. Ths, we mst have x P B. Sppose there is no z 2 B sch that x P z. Since B x, by Dal Corollary 1.1, we can only have x B or x P B. Assme that x P B. From Definition 5.2, we wold have TFðxÞ TFðBÞ. By Definitions 2 and 4, there wold exist some B 0 2 TF sch that x 2 B 0 and B B 0. In other words, there wold exist some B 0 2 TF and z 2 B sch that x 2 B 0 and z 2 B 0. This wold contradict the nonexistence of z 2 B sch that x P z. Hence, we mst have x B. t Proof of Proposition 10 (Test Frame Containing Flly Embedding Choices Only). The reslt follows directly from Definitions 4 and 5. t ACKNOWLEDGMENTS This research is spported in part by grants of the Research Grants Concil of Hong Kong (Project Nos. HKU 7029/01E and CityU 1048/01E) and a discovery grant of the Astralian Research Concil (Project No. DP ). REFERENCES [1] N. Amla and P. Ammann, Using Z Specifications in Category- Partition Testing, Systems Integrity, Software Safety, and Process Secrity: Bilding the Right System Right: Proc. Seventh Ann. IEEE Conf. Compter Assrance (COMPASS 92), pp. 3-10, [2] P. Ammann and J. Offtt, Using Formal Methods to Derive Test Frames in Category-Partition Testing, Safety, Reliability, Falt Tolerance, Concrrency, and Real Time Secrity: Proc. Ninth Ann. Conf. Compter Assrance (COMPASS 94), pp. 9-79, [3] A. Avritzer and E.J. Weyker, The Atomatic Generation of Load Test Sites and the Assessment of the Reslting Software, IEEE Trans. Software Eng., vol. 21, no. 9, pp , Sept [4] M.J. Balcer, W.M. Hasling, and T.J. Ostrand, Atomatic Generation of Test Scripts from Formal Test Specifications, Proc. Third ACM Ann. Symp. Software Testing, Analysis, and Verification (TAV 89), pp , [5] H.Y. Chen, T.H. Tse, F.T. Chan, and T.Y. Chen, In Black and White: An Integrated Approach to Class-Level Testing of Object- Oriented Programs, ACM Trans. Software Eng. and Methodology, vol. 7, no. 3, pp , [] H.Y. Chen, T.H. Tse, and T.Y. Chen, TACCLE: A Methodology for Object-Oriented Software Testing at the Class and Clster Levels, ACM Trans. Software Eng. and Methodology, vol. 10, no. 1, pp , [7] T.Y. Chen, P.L. Poon, and T.H. Tse, Classification-Tree Restrctring Methodologies: A New Perspective, IEE Proc.: Software, vol. 149, no. 2, pp. 5-74, [8] T. Chsho, Test Data Selection and Qality Estimation Based on the Concept of Essential Branches for Path Testing, IEEE Trans. Software Eng., vol. 13, no. 5, pp , May [9] S. Elbam, A. Malishevsky, and G. Rothermel, Test Case Prioritization: A Family of Empirical Stdies, IEEE Trans. Software Eng., vol. 28, no. 2, pp , Feb [10] R. Fergson and B. Korel, The Chaining Approach for Software Test Data Generation, ACM Trans. Software Eng. and Methodology, vol. 5, no. 1, pp. 3-8, 199. [11] M. Grochtmann and K. Grimm, Classification Trees for Partition Testing, Software Testing, Verification and Reliability, vol. 3, no. 2, pp. 3-82, [12] P. Jorgensen, Software Testing: A Craftsman s Approach. Boca Raton, Fla.: CRC Press, [13] C. Kaner, J. Falk, and H.Q. Ngyen, Testing Compter Software. New York: Wiley, [14] B. Korel, Atomated Test Data Generation for Programs with Procedres, Proc. 199 Int l Symp. Software Testing and Analysis (ISSTA 9), pp , 199. [15] G.J. Myers, Software Reliability: Principles and Practices. New York: Wiley, 197. [1] A.J. Offtt and A. Irvine, Testing Object-Oriented Software Using the Category-Partition Method, Proc. 17th Int l Conf. Technology of Object-Oriented Langages and Systems (TOOLS 17), pp , [17] T.J. Ostrand and M.J. Balcer, The Category-Partition Method for Specifying and Generating Fnctional Tests, Comm. ACM, vol. 31, no., pp. 7-8, [18] J. Sanders and E. Crran, Software Qality: A Framework for Sccess in Software Development and Spport. Wokingham, U.K.: Addison- Wesley, [19] P. Stocks and D. Carrington, A Framework for Specification- Based Testing, IEEE Trans. Software Eng., vol. 22, no. 11, pp , Nov [20] S.J. Zeil and C. Wild, A Knowledge Base for Software Test Refinement, Proc. Eighth Knowledge-Based Software Eng. Conf., pp , T.Y. Chen received the BSc and MPhil degrees from The University of Hong Kong, MSc degree and DIC from the Imperial College of Science and Technology, and the PhD degree from the University of Melborne. He is crrently the Professor of Software Engineering in the School of Information Technology, Swinbrne University of Technology, Astralia. His research interests inclde software testing, debgging, software maintenance, and software design. Pak-Lok Poon received the Master of Bsiness (information technology) degree from the Royal Melborne Institte of Technology and the PhD degree in software engineering from the University of Melborne. He is crrently an assistant professor in the Department of Accontancy at The Hong Kong Polytechnic University, where he teaches corses on information systems. He is on the editorial committee of the Information Systems Control Jornal. His research interests inclde software testing, reqirements inspection, compter adit and control, electronic commerce, and compters in edcation. He is a member of the IEEE and the IEEE Compter Society. T.H. Tse received the PhD degree from the University of London and was a visiting fellow at the University of Oxford. He is crrently an associate professor of compter science at The University of Hong Kong. He was the program chair of COMPSAC 2001, a program cochair of APAQS 2000, the steering committee chair of APAQS 2001, and a standing committee member of COMPSAC He is the steering committee chair of QSIC 2003 and a standing committee member of COMPSAC Dr. Tse is a fellow of the British Compter Society, a fellow of the Institte for the Management of Information Systems, a fellow of the Institte of Mathematics and Its Applications, a fellow of the Hong Kong Instittion of Engineers, a senior member of the IEEE, and a member of the IEEE Compter Society. He was decorated with an MBE by the Qeen.. For more information on this or any compting topic, please visit or Digital Library at