A Practical Strategy for Testing Pair-wise Coverage of Network Interfaces

Transcription

1 A Practical Strategy for Testing Pair-wise Coverage of Network Interfaces Alan W. Williams Robert L. Probert Telecommunications Software Engineering Group Department of Computer Science University of Ottawa 150 Louis Pasteur Priv. Ottawa, Ontario, K1N 6N5 Canada Abstract Distributed systems consist of a number of network elements that interact with each other. As the number of network elements and interchangeable components for each network element increases, the trade off that the system tester faces is the thoroughness of test configuration coverage, versus limited resources of time and expense that are available. An approach to resolving this trade off is to determine a set of test configurations that test each pair-wise combination of network components. This goal gives a well-defined level of test coverage, with a reduced number of system configurations. To select such a set of test configurations, we show how to apply the method of orthogonal Latin squares, from the design of balanced statistical experiments. Since the theoretical treatment assumes constraints that may not be satisfied in practice, we then show how to adapt this approach to realistic application constraints. Motivation A common source of system faults are unexpected interactions between system components. The risk is magnified when, for each element in a system, there are a number of interchangeable network components. A manufacturer of these system components would want to test as many of the potential system configurations as possible, to reduce the risk of interaction problems. However, the number of potential system configurations grows very quickly. A system tester faces the constraints of time and money, and it is quite likely that all possible configurations cannot be tested within any reasonable allotment. How, then, can the risk of interaction faults be managed in a realistic test plan? One approach is to at least test for all two-way interactions among various system components. This leads to a reduced set of test configurations. The assumption is that the risk of an interaction among three or more components is balanced against the ability to complete system testing within a reasonable budget. This paper investigates this approach, using the method of orthogonal Latin squares to determine the set of test configurations that cover all two way interactions. Background The method of orthogonal Latin squares is a technique from the literature of experimental design in statistics [1]. A Latin square is a balanced two-way classification scheme used to construct balanced experiments that achieve pair-wise parameter value coverage in cases where it is not practical to test all possible combinations. Depending on the number of factors (parameters) and levels (values) in the experiment, several orthogonal Latin squares may be required. The construction and properties of orthogonal Latin squares derive from both algebraic and combinatorial theories. An introduction to the algebraic construction of orthogonal Latin squares is contained in Gilbert [2], while the combinatorial theory and properties are covered in detail in Hall [3]. The use of orthogonal Latin squares for testing software is introduced by Mandl [4] in testing an Ada compiler. An example application of the method is to choose the types of the left and right sides of logical operators to achieve pair-wise type coverage. Mandl

2 also presents some of the restrictions imposed by the theory. Brownlie, et al. [5] provide a case report on the use of orthogonal Latin squares to choose system test configurations. This paper focuses on the specific system to be tested, how to apply the method of orthogonal Latin squares to the problem, and the benefits of using the method. Burroughs, et al. [6], apply the idea of systematically achieving pair-wise parameter coverage to protocol testing, in the testing of the call control requirements for primary rate ISDN. This paper presents the coverage goal, and the results, without specific details of the methodology. Cohen, et al.[7], have applied this method to protocol, telecommunications feature, and interoperability testing with their Automatic Efficient Test Generator. Again, the details of how the methodology is implemented are not provided. White [8] has also applied the same approach to testing graphical user interfaces. The configurations generated by using orthogonal Latin squares are compared with direct enumeration of combinations, and randomly generated configurations. The aim of this paper is to provide a self-contained guide to applying the theory of orthogonal Latin squares for constructing system test configurations to achieve pair-wise coverage at realistic cost. The paper extends previous work by showing how to apply the method in a wider range of situations that arise in practice. We also describe a prototype tool that implements the methods. First, we present the problem of testing combinations of configurations, along with a description of pair-wise parameter coverage. Next, we present a construction algorithm for generating orthogonal Latin squares. The theory includes some constraints, so we describe steps to take to apply the method in practice. Finally, we present some conclusions and further work. Testing of interfaces over networks In a telecommunications network, there are frequently a large number of allowable configurations of the network. In particular, a network may consist of various components, and each component may have several different types that can be freely substituted for one another. For example, in a tele network, the calling might be an ordinary, a business set, or a coin. A manufacturer of telecommunications equipment may also produce tele switches intended for various markets. Figure 1 illustrates a possible scenario. Figure 1: A network with substitutable components calling Phone type: business coin tele switch Market: Canada US Mexico called Phone type: business coin A system tester for more realistic-sized networks faces a large number of system configurations. The trade off that the system tester must deal with is the thoroughness of test coverage, versus the limited resources of time and expense that are available. It may be the case that the customer can configure the network for their purposes. That is, it may not be possible to predict in advance which configurations the customer will select, and concentrate on testing those configurations. This reduces to a general problem: having a number of parameters, or factors, each of which have a number of possible values, or levels. For the moment, let us assume that all the parameters are independent. That is, selecting a specific value for any single parameter does not restrict the choice of selecting values for any other parameters. Suppose there are k independent configuration parameters for the system. For each parameter i, there are v i possible values of the parameters. Then there are v 1 v 2 v k possible system configurations. After setting up each system test configuration, there are normally a number of test cases to run for each configuration. An exhaustive test of all system configurations would therefore require t = c v 1 v 2 v k test cases, where c is the number of test cases required for each configuration. For even small values of k and v i, the value of t can become quite large. For example, if there are four parameters, and each parameter can have four values, then there are 4 4 = 256 possible system configurations. If there are 100 tests required for each configuration, the complete system test would require execution of 25,600 tests.

3 Pair-wise coverage of parameter values A possible trade off between test coverage and the number of tests is to determine a set of test configurations that tests each pair-wise combination of parameter values. That is, for any two parameters in the system, each possible combination of the values of the two parameters occurs in at least one test configuration. This goal provides a well-defined level of test coverage, with a reduced number of system configurations. Pair-wise parameter coverage can detect any consistent problem with any value of any single parameter, or detect any consistent problem with pair-wise compatibility of parameters. However, it cannot detect problems involving a combination of three or more parameter values. To select a set of test configurations that achieve coverage of all pair-wise combinations of parameter values, we can apply the method of orthogonal Latin squares from the design of statistical experiments. Application of orthogonal Latin squares to testing A Latin square is a balanced two-way classification scheme used by statisticians in the construction of experiments, and is usually represented by a square matrix: The defining property of a Latin square is that each entry appears exactly once in each row and column. A single n n Latin square can handle three parameters, each of which have n possible values. The square shown above can be used for a problem with three parameters, each of which can take three possible values. Each entry in the Latin square represents the values for a single test configuration. The first parameter takes the value of the row index of the entry. The second parameter takes the value of the column index of the entry. The third parameter takes the value of the matrix entry. For example, the highlighted matrix entry represents the test configuration (3,2,1). To handle more than three parameters, we need a set of orthogonal Latin squares. A set of s squares is said to be orthogonal if the combined matrix formed by ordered s-tuples of the individual matrix entries has no repeated elements. Suppose we have two matrices [a ij ] and [b ij ]. The combined matrix is [c ij ] where the elements are c ij = (a ij,b ij ). If c ij c rq whenever r i and q j, then the squares [a ij ] and [b ij ] are orthogonal. If there are k system parameters, then k 2 orthogonal Latin squares are required. With a set of orthogonal Latin squares, two parameters take their values from the row and column indices as before. The remaining parameters take their values from the ordered (k 2)-tuple formed from the superimposed squares. In a system with k parameters and v possible values for each parameter (that is, v k system configurations), then the method of orthogonal Latin squares will reduce the number of system configurations to v 2. If c is the number of test cases required for each configuration, then the total number of test cases to be executed is reduced from cv k to cv 2. For example, four parameters require two orthogonal Latin squares: Superimposing these matrices, we obtain: (1,1) (2,2) (3,3) (2,3) (3,1) (1,2) (3,2) (1,3) (2,1) The highlighted matrix entry represents the configuration (3,2,1,3): row 3, column 2, entry (1,3). Table 1 shows the complete set of test configurations: Table 1: Test configurations for four parameters, each with three values Configuration number Parameter Parameter Parameter Parameter Restrictions on the existence of orthogonal Latin squares It is always possible to construct a single n n Latin square for any size n. However, as listed in

4 Mandl [4], and proved in Hall [3], there are restrictions on the existence of a set of orthogonal Latin squares: 1. For any n, there are at most n 1 mutually orthogonal n n Latin squares. 2. If p is the smallest prime in the prime factorization of n, and occurs j times in that factorization, then there exist at least p j 1 orthogonal Latin squares. This guarantees the existence of the maximum of n 1 orthogonal Latin squares when n = p m, for some prime p and integer m 1. The cases with the maximum number of orthogonal Latin squares are of interest, because they can handle the greatest number of parameters, and because there is a useful construction algorithm for them. Unfortunately, this result also points out a particularly problematic case when n = 4t + 2, for some integer t 1. Since the prime factorization of n has only a single 2, the prime factorization result guarantees only the existence of a single Latin square. This worst case is true for n = 6; one cannot find two 6 6 orthogonal Latin squares. When t > 1, it has been shown (Hall [3]) that at least two orthogonal Latin squares exist. Algorithm for the construction of orthogonal Latin squares Construction algorithms for a set of orthogonal Latin squares are presented in Gilbert [2] and Hall [3]. The algorithm presented here is from Gilbert [2], and uses results from the theory of finite fields in algebra. A finite field with p m elements is called a Galois field of order p m, and is denoted by GF(p m ). For any prime p, and any integer m 1, a Galois field GF(p m ) exists. In the case where m = 1, the integers modulo p, Z p, is a Galois field of order p. For m > 1, the Galois field GF(p m ) can be constructed from Z p by extending the field in a manner similar to the method by which the field of real members is extended to complex numbers. In the field of real numbers, the polynomial q(x) = x is irreducible. However, by defining a new field element i as a root of x = 0, the real numbers are extended to complex numbers. Let Z p [x] denote the set of polynomials of x with coefficients in Z p. To extend Z p to GF(p m ), find a polynomial q(x) of degree m, irreducible in Z p [x], and define α such that q(α) = 0. Then, the polynomials Z p [α], with degree less than m can be used as new field elements to create GF(p m ). Algorithm 1: Let GF(n) = { x 0, x 1, x 2,, x n 1 } be a finite field of order n = p m, where x 0 = 0, and x 1 = 1. Define the elements of squares L = ( a l ) for 1 l n 1, by ij l i j l l a = x x + x, for 0 i n 1. Then, L l is a Latin square, and {L 1, L 2,, L n 1 } are mutually orthogonal. Example: Construct three squares of order four To demonstrate the use of Algorithm 1, consider the case of constructing three orthogonal 4 4 Latin squares. First, we need to generate the Galois field GF(4) = {x 0, x 1, x 2, x 3 }, with x 0 = 0, and x 1 = 1. GF(4) = GF(2 2 ), so we start with integers mod 2, Z 2 = {0, 1}. Next, we need to find a polynomial q(x) in Z 2 [x], such that q(x) is of degree 2, and irreducible in Z 2 [x]. A choice for q(x) is x 2 + x + 1, and define new element α such that α 2 + α + 1 = 0 in GF(4). Then, GF(4) = Z p [α] = {0, 1, α, α + 1}. Since x 1 = 1, aij 1 = xi + x j, we can form L 1 from the addition table for GF(4): 0 1 α α α + 1 α α α α + 1 α 1 0 ij L 2 can be obtained by a 2 = α x + x. Note that ij i j since α 2 + α + 1 = 0 in GF(4), α 2 = α α α + 1 α α α + 1 α α + 1 α ( ) + 3 ij i j L 3 can be obtained by a = α + 1 x x. 0 1 α α + 1 α + 1 α α + 1 α α α Derivation of test configurations With the set of squares we have just generated, we can now proceed to determining the set of test configurations.

5 While the field elements in the squares are { 0, 1, α, α + 1 }, we can renumber these elements as { 1, 2, 3, 4 } to correspond to the parameter values. If we superimpose the three orthogonal squares with the renumbered elements, we obtain: (1,1,1) (2,2,2) (3,3,3) (4,4,4) (2,3,4) (1,4,3) (4,1,2) (3,2,1) (3,4,2) (4,3,1) (1,2,4) (2,1,3) (4,2,3) (3,1,4) (2,4,1) (1,3,2) Note that all sixteen ordered triples are distinct, so the squares are indeed orthogonal. Each ordered triple represents a test configuration for the situation where there are five parameters, each of which have four values. To obtain a test configuration, the first parameter takes the value of the row index of the ordered triple, the second parameter takes the value of the column index of the ordered triple, and parameters three through five take the values contained in the ordered triple. For example, the highlighted triple represents the test configuration (3,1,3,4,2). Table 2 shows the complete set of test configurations. Table 2: Test configuration for five parameters, each with four values Configuration Parameter number Applying the method in practice The method of orthogonal Latin squares rests on several assumptions that may not be satisfied in practice. In this section, we outline strategies for adapting the method to fit realistic situations. To this point, we have assumed that: 1. All parameters have the same number of values. 2. Each parameter is independent. 3. Enough orthogonal Latin squares exist. In practice, none of these assumptions may hold. Therefore, we need strategies to deal with each of these situations. Moreover, assumption 3 may not hold for two reasons: A. The number of values represents a case where there are few (or no) orthogonal Latin squares. For example, if there are six values for each parameter, there are no orthogonal Latin squares, and more than three parameters cannot be handled in a 6 6 square. B. There are more than v + 1 parameters, where v is the number of values. The strategy for handling cases A and B are different, and are treated separately in the discussion to follow. We now proceed to discuss each of these situations. Case 1: Differing numbers of values The basic orthogonal Latin squares method assumes that each parameter has the same number of values, v, and produces v 2 configurations. Suppose instead that there are k parameters, and for each parameter i, there are v i possible values of the parameters. For convenience, order the parameters so that v i v i+1 for all i k. Clearly, at least v 1 v 2 configurations are necessary to achieve pair-wise coverage. White [8] notes that columns (or rows) of a Latin square can be repeated to handle additional values for the parameter that obtains its value from the column (or row) index. Therefore, one can set the problem up so that the parameter with v 1 values is determined in this manner, and base the square size on v 2, assuming the appropriate number of squares exist (see case 3A). To handle the parameter with v 1 values, add rows or columns as required. The result is a v 1 v 2 matrix that produces the minimal number of test configurations. For parameters that have fewer than v 2 values, don t care values can be used as required.

6 There are additional considerations for best selecting the square size, given the values of v 1, v 2, and v 3, that we are presently investigating when case 3A affects unbalanced cases. Case 2: Dependent parameters Another common situation in practice is when there are dependencies among two or more parameters. Specific combinations of parameter values may not be allowed, or one parameter value may require another parameter to have one of a reduced set of values. The existence of a parameter itself may even be dependent on other parameters having certain values. The following approach to handling dependent parameters is suggested by Taguchi [1]. Suppose for example that there are two types of interfaces between a tele switch and a. Interface A supports, business, or coin s; while interface B supports business and ISDN s. Suppose also that there are five varieties of switch, independent of the other network elements. Figure 2 illustrates this situation. calling Phone type: coin business ISDN Figure 2: Dependent parameters line interface Interface: A B switch Market: 1-Canada 2-US 3-Mexico 4-France 5-UK line interface Interface: A B called Phone type: coin business ISDN The objective is to turn the dependent parameters into independent parameters, to be able to apply the method of orthogonal Latin squares. The strategy is to form a hybrid parameter that includes each allowed combination of dependent parameters. For the interface- dependency, construct the following renaming: A 1 coin A 2 business A 3 business B 4 ISDN B 5 If values 1 through 5 of the hybrid parameter are covered, then all possible allowed combinations of the interface and dependency are covered. Therefore, the hybrid parameter is independent, and replaces the dependent parameters. The overall problem therefore reduces to three independent parameters, each of which have five possible values. A 5 5 Latin square can be applied to this problem: Assuming the switch markets are numbered according to Figure 2, the highlighted configuration (4,3,1) in the square above represents: calling : business left interface: switch market: right interface: called : B Mexico A Case 3A: Insufficient number of orthogonal Latin squares Because of the restrictions on the existence of orthogonal Latin squares, there may be an insufficient number of squares to exactly fit the number of parameters and values in the system. As noted previously, the case where there are six values for each parameter is particularly problematic, since only three parameters can be handled. When there are not enough squares, a strategy is to increase the square size to the next value for which enough squares exist. This approach should be taken only when there are fewer parameters than the number of values: for example, four parameters, each having six values. At most, the square size should be increased to the next largest integer n such that n = p m, for some prime p and integer m 1. The extra values are treated as don t care values, as in the previous case. Case 3B: Many parameters and few values A common situation in practice is a large number of parameters, each of which have only a few possible values. However, the number of orthogonal Latin squares that exist is restricted by the number of values. It is possible to increase the square size to the point where enough orthogonal Latin squares exist. Then, the number of don t care values swamp the

7 Table 3: First set of configurations with Latin squares repeated within blocks Configuration number Parameter Block Block Block Block number of actual values, and the number of configurations is unnecessarily large. For example, suppose there are sixteen parameters, each with three values. Handling this many parameters with direct application of the method requires a set of fourteen orthogonal Latin squares. This results in using squares that are at least 15 15, and using the construction algorithm presented here generates a set of fourteen squares (16 = 2 4 ). The result is 256 test configurations, and for each configuration, there are typically three actual values, and thirteen don t care values. Of course, 256 configurations is better than 3 16 = 43,046,721 possible configurations, but we can do better. For sixteen parameters with three values, we can use two 3 3 squares, and divide the sixteen parameters into four blocks of four parameters. In each block, use the two squares to generate all pair-wise combinations of the four parameters. The result, as shown in Table 3, is four repeated blocks. Within each block, each pair-wise combination of parameter values is covered. If you select any parameter, the pair-wise combinations with twelve out of fifteen of the other parameters are covered. The only parameters where the pair-wise combinations are not Table 4: Additional configurations created by assigning Latin square values to each block (B = Block number, P = Parameter number) Configuration number B P covered are the ones with the identical row of values. Therefore, our next concern is these corresponding parameters in the other blocks To cover the remaining pair-wise combinations, we now treat each block as a single parameter, and apply the method again. Use the two 3 3 squares to cover pair-wise combinations between blocks. Assigning a value to a block means that each parameter within the block takes that value, as shown in Table 4. For any parameter, there is one parameter in every other block where the pair-wise combinations remain uncovered. However, by assigning the same value to all parameters in a block, and then covering all pair-wise combinations of block values, we achieve complete pair-wise coverage of parameter values. We can handle up to sixteen parameters with this approach because there are four blocks of parameters, and this can be handled by two orthogonal Latin squares. If there were five or more blocks, subsequent blocks would have to repeat the block values. However, we can repeat the block method by creating super blocks of sixteen parameters, and adding a set of nine configurations that cover the pair-wise

8 combinations of the super blocks. This process can be repeated to cover an arbitrary number of parameters. This leads to the following result: Lemma 1: For the case of k parameters, each of which have v values, let n v be the next largest integer such that n = p m for some prime number p and integer m 1. Then, the number of test configurations produced by this method will be log n k n ( ) Proof sketch: Let n v be the next largest integer such that n = p m for some prime number p and integer m 1. Using algorithm 1, we can construct n 1, n n orthogonal Latin squares. This can handle n + 1 parameters with n values in n 2 configurations. We can k then form n + 1 k blocks. If n n + 1 > + 1, then we still have too many parameters to be handled by the n + 1 orthogonal Latin squares. However, we can k repeat the process by creating ( n + 1) 2 super blocks, and continue in this manner. k Eventually, n b ( n + ) + 1 for some integer 1 b 0. The value b represents the number of times that we repeat the blocking process, and n 2 configurations are added each time, so that the number of configurations created is (b + 1)n 2 But: k ( n + 1) b+ 1 b+ 1 k ( n + 1) log ( k) b + 1 n+ 1 = 1 logn+ 1( k) = b 1 Therefore, we have log n k n configurations. Tool implementation + 1( ) We have implemented the methods described in this paper in a prototype tool that generates test configurations achieving pair-wise parameter coverage. At present, all the methods presented in this paper have been implemented, except for dependent parameters. Because of the nature of the methods, we have created a class library that performs some elements of symbolic algebra. Table 5 shows the classes comprising the tool implementation. Table 5: Classes in tool implementation Class Description IntMod Implements integers modulo p, for some prime p. Subclass of Number in Smalltalk, but is a base class in C++. PolyMod Implements polynomial arithmetic, with coefficients being instances of IntMod QuoPoly Implements PolyMod equivalence classes based on a quotient polynomial. Derived class of PolyMod. Field Generates a Galois field for a given prime order. Field elements are instances of QuoPoly. LatinSquare Creates a set of LatinSquares of specified size, using an instance of Field in the generation process. ConfigSet Represents a set of test configurations. Uses an instance of LatinSquare, and implements the algorithm for many parameters with few values. TGenUI Implements the user interface. The key classes in Table 5 are QuoPoly (derived from PolyMod) and Field. Instances of Field are implementations of the Galois fields GF(p m ) (p is prime) used by Algorithm 1. In particular, Algorithm 1 requires implementation of the and + operators for the field elements. Constructing the field involves finding the quotient polynomial q(x), where q(x) is of degree m, and irreducible in Z p [x]. This polynomial is used to construct instances of class QuoPoly, which represent the elements of the field. Instances of class QuoPoly represent polynomial equivalence classes based on q(x). Class PolyMod implements polynomial arithmetic where the polynomial coefficients are in Z p (integers modulo p). Class QuoPoly extends PolyMod so that the and + operators preserve the property that each polynomial is represented by its equivalence class as the remainder of division by the quotient polynomial q(x). The tool allows the user to enter the names of each parameter, and then enter name of each parameter value. It then generates the set of test configurations using these names.

9 Conclusions and further work In this paper, we have presented a guide to the theory and practical application of the method of orthogonal Latin squares to generate system test configurations that achieve pair-wise parameter coverage. We have presented a method for generation of a set of orthogonal Latin squares where the size of the square is a prime power. The theory is directly applicable when all configuration parameters are independent, have the same number of possible values, and enough orthogonal Latin squares exist. We have shown how to apply the theory in practice when there are dependent parameters, additional orthogonal Latin squares are required, or there are differing numbers of parameters. In particular, we have provided a method to handle the case where there are many parameters each having a small number of values a common case in practice. If there are k parameters in the system, each having v values, there are v k possible system configurations. The method presented here generates 2 log n k n configurations, where n is the next + 1( ) largest integer v such that n = p m for some prime p and integer m 1. The set of configurations meets the pair-wise parameter value coverage criterion. Further investigation is needed for handling differing numbers of parameter values, and in improving the method for handling many parameters with few values. In both of these situations, the number of configurations can be further reduced in some cases. When there are many parameters and few values, if logn+ ( k) log ( n+ k ) 1 1, it may be possible that pair-wise parameter coverage can be achieved with fewer configurations than produced by the method described here. For example, thirteen parameters each having three values can be covered with only fifteen configurations [7], instead of the eighteen generated by the method described here. An improved approach to generating the last block of configurations should be investigated. The combined case of many parameters with unbalanced numbers of values also needs further work, and either heuristics, or combining the separate approaches given here are promising directions for investigation. Telecommunications Software Engineering Research Group at the University of Ottawa for their useful comments on preliminary versions of this material. The authors gratefully acknowledge support for this work from the Telecommunications Research Institute of Ontario (TRIO), the Natural Sciences and Engineering Research Council of Canada (NSERC), Nortel, and the University of Ottawa. References [1] G. Taguchi, System of Experimental Design, Vol. 1, UNIPUB/Kraus International Publications (1987). [2] W.J. Gilbert, Modern Algebra with Applications, Wiley Interscience, New York, (1976). [3] M. Hall Jr., Combinatorial Theory, Wiley Interscience, New York (1986). [4] R. Mandl, Orthogonal Latin squares: An application of experiment design to compiler testing, Communications of the ACM, Vol. 28 No. 10 (October 1985) pp [5] R. Brownlie, J. Prowse, and M.S. Phadke, Robust Testing of AT&T PMX/StarMail using OATS, AT&T Technical Journal, Vol. 71 No. 3 (May/June 1992) pp [6] K. Burroughs, A. Jain, and R.L. Erickson, Improved Quality of Protocol Testing Through Techniques of Experimental Design, proc. Supercomm/ICC (1994) pp [7] D.M. Cohen, S.R. Dalal, A. Kajla, and G.C. Patton, The AETG System for Feature and Protocol Testing, Int. Conference on Testing Computer Software, Washington DC, June [8] L.J. White, Regression Testing of GUI Event Interaction, submitted to ICSM 96. Acknowledgments This work was inspired by a discussion with Rick Greenfield and his staff at Nortel. The authors would also like to thank the members of the