ET-based Test Data Generation for Multiple-path Testing

2016 3 rd International Conference on Engineering Technology and Application (ICETA 2016) ISBN: 978-1-60595-383-0 ET-based Test Data Generation for Multiple-path Testing Qingjie Wei* College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China Yijin Li & Ying Qian College of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing, China ABSTRACT: Automatic test data generation for path testing is one of the difficult problems of software testing And the existing methods have low efficiency and stability. We have conducted studies on the research field about ET-based test data generation for multiple-path testing, and we have improved the efficiency and the stability of the method by improved the fitness function of currently existing method. By the comparison experiments, it showed that the method in this paper have high efficiency and stability. Meanwhile, we gave the conclusion how the basic parameters impacted on the efficiency, and suggested the setting range of parameters. Keywords: software testing; evolutionary testing; path coverage; test data; genetic algorithms 1 INTRODUCTION Software testing is involved in the whole life cycle of software development, not only as a verification and validation process for the software quality improvement, but also as an essential method to assure the completion and to enhance the quality of a software project [1,2]. Meanwhile, it costs a lot to design the high quality test cases and check the results manually [3]. Finding the input data for these test cases manually is very time-consuming, especially when the logical structure of the program under test is very complicated [4]. And automated software testing can significantly reduce the cost of developing software. Therefore, Ould believes that the research of automatic test data generation will be the most important field for automatic testing [5]. Path coverage is the strongest coverage criterion in software testing, and automatic test data generation for path testing has become one of the difficult problems of automatic test data generation. By the intensive study of automatic test data generation, this research field attracted more and more attention. Through the survey, we found the achievements in this field have been massive increased in the *Corresponding author: eagin322@163.com past few decades, especially in the last decade. The survey result is shown in Figure 1 and Figure 2. Figure 1. The growth trend of the research of automatic test data generation. Figure 2. The growth trend of the research of automatic test data generation for path testing. 377

As one of the research fields of automatic test data generation, evolutionary testing (ET) was first proposed by Xanthakis et al. in 1992 [6]. It is based on the basic framework of genetic algorithm (GA), which can find the test data in the input domain by heuristic searching. The method has high convergence and automation level, and can significantly reduce the test cost and improve the quality of testing [7-10]. Although the study of automatic test data generation for multiple-path testing received a lot of attention, most of them only consider the single path coverage (one search only aimed at single target path). At the same time, a few methods for multiple target paths have low efficiency and stability. Therefore, this paper adjusts and improves existing method to optimize the efficiency and stability. In addition, we first propose the impact factors on convergence rate of parameters and the recommended setting range of parameters. 2 BASIC CONCEPTS 2.1 ET The basic idea of ET is: at first, in order to solve the problem of automatic test data generation, it is transformed into a function optimization, and the fitness function is defined; on the second, to encode the test data into the chromosomes and perform genetic manipulation; at last, to put decoded data to the program under test and calculate the performances of current test data. After certain generations of GA iteration, we can get the test data sets which are satisfied with the particular test adequacy criteria. The flow graph of ET is shown in Figure 3. evaluation of the individuals, and the outer loop is responsible for the population evolution. 2.2 Representation of the path The representation of the path is the foundation of automatic test data generation for multiple-path testing, because the design of the fitness function depends on it. Gong et al. [11] have proposed that the path can be expressed as a kind of binary string by Huffman coding. When Huffman coding was used to represent the path, programs under test should be instrumented via making bit 1 to represent the true branch and making bit 0 to represent the false branch. By this way, there is a one-to-one correspondence between the path of the program and the binary string. Then, we use a regular array A to store the execution path, and set its initial value to \0. The specific rules are shown in Formula (1). A is the number of elements in the array, viz., the length of the path string. Among them, the different indexes of the array mean different branches. The assignment of the same index elements depend on the executions of the branches. If executed the true branch, the element will be assigned to 1, otherwise it will be assigned to 0. '1', executed true branch Ai, i 0,1,, A 1 (1) '0', executed false branch Triangle Classifier [11-17] has been widely used in the field of software testing because of its clear and complicated logical structure. Meanwhile, only a tiny part of the test data in the whole input domain can cover the certain paths, so the Triangle Classifier also became the most common benchmark function of automatic test data generation for path testing. The sample of the Triangle Classifier is shown in Figure 4. As shown in the figure, by instrumentation, the 4 paths of the Triangle Classifier can be expressed as: 000, 001, 01, 1. And after the execution of the program, the assignment result of the array A shows the path that the current test data can cover. 2.3 Fitness function design Figure 3. Flow graph of ET. It can be seen that the outer loop is the basic framework, which implements the population evolution in the input domain. The inner loop is the calculation for the individuals, which monitors the execution result and evaluates the fitness of the individuals. In other words, the inner loop is responsible for the As the core to transform automatic generate test data to the function optimization, the fitness function determines the quality of this method to some extents. In the methods based on Huffman coding, their design idea is to compare the execution path string with the target path string bit by bit. Through calculating the matching degree between them, let the path with higher matching degree have the higher fitness value. In the definition of fitness function of Gong [11], X is the test data, P(X) is the execution path string corresponds to X, Pj is the j-th target path string, P(X) and Pj correspond to the length of P(X) and the length of Pj. And djk(p(x)) is the bit matching, it reflects 378

Figure 4. The sample about Triangle Classifier. whether the k-th bit of P(X) and Pj are equal. If the k-th bit of P(X) and Pj are equal, then we let djk(p(x)) equal 1; otherwise, we let djk(p(x)) equal 0. The expression of djk (()) is shown in Formula (2). 1, the k-th bit of PX ( ) and Pj are equal d jk ( P( X)) (2) 0, the k-th bit of PX ( ) and Pj are not equal The path strings of P(X) and Pj are compared bit-by-bit min( P(X), Pj ) times. If the number of 1 in (()) equals Pj, it means that P(X) entirely matches with Pj. In other word, the test data X covers the target path Pj. Otherwise it means that P(X) partially matches with Pj. If there are more numbers of 1 in (()), the matching of P(X) and Pj will be higher. The different bits of the path string correspond to the different matching degrees, and the later bit has the higher matching degree. So the different bits of the path strings should be given different weights. So, we introduce a variable mjk, which indicates the number of same bits when comparing () with Pj to the k-th bit. And, is the matching degree of P(X) and Pj, its expression is shown in Formula (3). Thus, if there are more numbers of the same bits of P(X) and Pj, the matching degree and the value of will be higher. In addition, if there are more consecutive identical bits starting from the first bit, the similarity degree between the execution path and the target path will be higher. Therefore we have to adjust the Formula (3). We introduce a variable cj(p(x)), which represents the number of the consecutive identical bits between P(X) and Pj from the first bit. The adjusted fitness function fj (X) is shown in Formula (4). j j j f ( X) ( c ( P( X)) 1) f ( X) (4) If there are more numbers of the same bits of P(X) and Pj, and more consecutive identical bits starting from the first bit, the matching degree and the value of fj (X) will be higher. For an example, when the target path P1 = 100, the execution path P(X1) = 101 and P(X2) = 001, the calculations of their matching degrees are as follows: min( P( X), P j ) j k1 jk jk f ( X ) m d ( P ( X )) (3) 379

3 1 1 k 1 1k 1k 1 1 1 1 1 1 1 f ( X ) m d ( P( X )) 1121203 f ( X ) ( c ( P( X )) 1) f ( X ) (2+1) 3=9 f ( X ) m d ( P( X )) 00+00+11=1 f ( X ) ( c ( P( X )) 1) f ( X )=( 1+1) 1=2 ' 3 1 2 k 1 2k 2k 2 2 1 2 2 1 2 From the result, the matching degree of P(X1) and P1 is higher than the matching degree of P(X2) and P1. In conclusion, the fitness function of automatic test data generation for multiple-path testing is shown in Formula (5). m 1 i j i m j 1 fit( X ) f ( X ) (5) In this function, the test data X is an n-dimensional vector in the input domain D(X D), the target paths are P1, P2,, Pm, and m is the number of remaining the target paths. 3 NEW FITNESS FUNCTION The definition of the fitness function of Gong [11] is unreasonable when the execution path and the target path have different lengths. Especially when the lengths of the target path and the execution path are very different, this situation is more serious. For example, when the target path P1 = 1001 and the execution path P(X1) = 1000, the fitness value f1(x) = 24. And when the target path P1 = 1000000000 and the execution path P(X1) = 1000111111, the fitness value f1(x) = 50. In general, the former fitness value should be much higher than the latter fitness value. But the former is lower than the latter, because the design of the fitness function is unreasonable. Therefore, we introduce a variable Fj, which indicates the fitness value when the execution path entirely matches with the target path. By this way, the adjusted fitness function is shown in Formula 6. m 1 f j( Xi) fit( Xi) m F (6) j1 j According to this, when the target path P1 = 1001 and the execution path P(X1) = 1000, the fitness value f1(x) = 0.48. And when the target path P1 = 1000000000, and the execution path P(X1) = 1000111111, the fitness value f1(x) = 0.0826. By this way, we effectively solve the problem that the matching error when the lengths of the target path and the execution path are different. 4 EXPERIMENTS 4.1 Parameters setup In order to get the optimized scheme of the parameters, we have conducted the experiments on the basic parameters of ET-based test data generation for multiple-path testing. Then we gave the conclusion that how the basic parameters impacted on the method efficiency, and suggested the setting range of parameters. As a frequently-used benchmark program, Insertion Sort [11,16,17] has abundant target paths(sum to 120) and complicated logical structure. Due to the random factors have little impact on its results, and the experimental results are typical, We select the Insertion Sort program as the benchmark program. As a replacement strategy of the individuals, generation gap indicates the scale of the generation genes replacement in the evolution [18]. Different generation gaps have different effects on the convergence rate of the method. We ran the program 200 times and took the average when the crossover probability was 0.7, the mutation probability was 0.7, the population size was 5, the number of parameters was 5, the range of the values was [0, 2 5-1] 5, and the maximum number of the iteration was 500. The experimental results are shown in Figure 5. As shown in the figure, generation gap impacted on the convergence rate of ET-based test data generation for multiple-path testing, has some rules as follows: at first, when generation gap setting is low(0-0.4), ET cannot cover the whole target paths; on the second, the convergence rate rapidly accelerates with the gradually enlarging generation gap(0.4-0.7); in the third, as the generation gap reaches a certain zone(0.7-1), the convergence rate is speedy and stable. Figure 5. Influence of generation gap on the convergence rate. 380

Figure 6. The impact of crossover probability and mutation probability on the convergence rate. Therefore, in ET-based test data generation for multiple-path testing, the recommended setting range of generation gap should be [0.7,1]. In addition, as the important parameters of ET, crossover probability and mutation probability have significant impact on the convergence rate of the method. As one of the primary operators to produce new individuals has some impact on the convergence rate. Besides, mutation keeps the population's diversity during evolution, avoiding the evolution halted and the immature convergence. We ran the program 200 times and took the average when generation gap was 0.9, the population size was 5, the number of the parameters was 5, the range of the values was [0, 2 5-1] 5, and the maximum number of the iteration was 1000. The experimental results are shown in Figure 6. The figure shows that the impact on the convergence rate from the crossover probability is relatively weak. Instead, the mutation probability has a relatively significant impact on the convergence rate. The impact of mutation probability on the convergence rate is shown in Table 1. Table 1. The impact of mutation probability on the convergence rate. Mutation probability (P m ) Influence cannot cover the whole target 0 paths the rate of convergence rapidly 0<P m <0.1 accelerates the rate of convergence is 0.1 P m 0.9 speedy and stable the rate of convergence rapidly 0.9< P m <1 decelerate cannot cover the whole target 1 paths From this, we can see that the suggested setting range of mutation probability of ET-based test data generation for multiple-path testing should be [0.1,0.9]. 4.2 Experiments design To validate that our method has faster convergence rate and the better stability, we did the contrast experiments on 4 different mainstream benchmark programs. These benchmark programs are Triangle Classifier [11-17], Minimaxi [11,17,20], Bubble Sort [11,17,21], Insertion Sort. The description of these benchmark programs are shown in Table 2, and the descriptions of their parameter settings are shown in Table 3. In order to make the results more accurate and typical, we got each result following these steps: Step 1: do 200 times of experiments, and take the average as R n ; Step 2: do extra 50 times of experiments, and take the average as R n50; Step 3: calculate the difference between the two averages: Rn50 R ; Step 4: if 5 10, get the average as the result, otherwise return to step 2. 4.3 Experimental results and analysis We have conducted these experiments as the above. The results of these experiments are shown in Figure 7, and the data of the experimental results are shown in Table 4. According to the experimental results, the method in this paper has little advantage in the convergence rate and the stability, when the benchmark programs are Triangle Classifier, Minimaxi, and Bubble Sort. On the contrary, when the benchmark program is Insertion Sort, our method expresses the tremendous advantage in the convergence rate and the stability. As the typical of the complicated logical structure programs, Insertion Sort has abundant target paths with different lengths. And our target is to improve the efficiency when the programs under test have complicated logical structure. Therefore, the experiments of Insertion Sort can fully demonstrate the advantages of our method. In conclusion, the method in this paper has a slight advantage for the programs under test which have simple logical structure and same lengths of the target paths. And, when the programs under test have complicated logical structure and abundant target paths with different lengths, our method expresses the significant advantage that our method has the higher convergence rate and the higher stability. 381

Table 2. Description of benchmark programs. Benchmark programs Loops and branches Structure Number of parameters Target path Triangle Classifier 3 branches 3 level nested branch 3 4 paths with different lengths Minimaxi 1 loop and 2 branches 2 if statement nested in a loop 3 9 paths with same lengths Bubble Sort 2 loops and 1 branch 2 level nested loop and 1 if statement nested in the inner loop 5 24 paths with same lengths Insertion Sort 2 loops and 1 branch 2 level nested loop, 1 if statement nested in the inner loop, and 1 break statement 5 120 paths with different lengths Table 3. Description of parameters setting. Benchmark programs Range of Population Maximum Crossover Mutation Selection Crossover Mutation values size iteration probability probability operator operator operator Triangle Classifier [0,2 5-1] 3 5 500 0.7 0.7 sus xovsp mut Minimaxi [0,2 5-1] 3 5 500 0.7 0.7 sus xovsp mut Bubble Sort [0,2 5-1] 5 5 500 0.7 0.7 sus xovsp mut Insertion Sort [0,2 5-1] 5 300 500 0.7 0.7 sus xovsp mut Table 4. Data of the experimental results. Benchmark programs Average of evolution times Variance of evolution times Gong s method our method Gong s method our method Triangle Classifier 23.58241 23.5229 470.0710 468.5026 Minimaxi 46.5320 45.9363 1768.7451 1707.0842 Bubble Sort 43.5161 43.5559 193.3143 195.4922 Insertion Sort 60.967 54.9810 218.9439 160.2194 Figure 7. Results of the experiments. 5 CONCLUSION In this paper, we have conducted studies on the research field about ET-based test data generation for multiple-path testing, and we have conducted experiments on our method and existing method.then we gave the conclusion how the basic parameters impacted on the algorithm efficiency, and suggested the setting range of parameters. In addition, we have improved the efficiency and the stability of the method by improved the fitness function of currently existing method. REFERENCES [1] Friedman, M.A. & Voas, J.M. 1995. Software Assessment: Reliability, Safety, Testability. Chichester: John Wiley & Sons, Inc. [2] McMinn, P. et al. 2012. Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation. IEEE Transactions on Software Engineering. 38(2): 453-477. [3] Osterweil, L. 1996. Strategic directions in software quality. ACM Computing Surveys. 28(4): 738-750. [4] Michael, C.C. & Mcgraw, G.E. 1997. Genetic algorithms for dynamic test data generation. Proceedings of the 12th IEEE International Conference on Automated software engineering. pp: 307-308. [5] Ould, M.A. 2004. Testing-a challenge to method and tool developers. Software Engineering Journal. 25(4): 50-52. 382

[6] Xanthakis, S. et al. 1992. Application of genetic algorithms to software testing. Proceedings of the 5th International Conference on Software Engineering and Applications. pp: 625-636. [7] Wegener, J. et al. 2002. Suitability of evolutionary algorithms for evolutionary testing. Proceedings of the 26th Annual International on Computer Software and Applications Conference. pp: 287-289. [8] Xu, B.W. et al. 2007. Application of Genetic Algorithms in Software Testing. Pennsylvania: IGI Global. [9] Clarke, J. et al. 2003. Reformulating software engineering as a search problem. IEEE Proceedings Software. 150(3): 161-176. [10] Xie, X.Y. et al. 2008. Survey of evolutionary testing. Journal of Frontiers of Computer Science and Technology. 2(5): 449-466. [11] Gong, D.W. et al. 2010. Novel Evolutionary generation approach to test data for multiple paths coverage. Acta Electronica Sinica. 38(6):1299-1304. [12] Jones, B.F. et al. 1996. Automatic structural testing using genetic algorithms. Software Engineering Journal. 11(5): 299-306. [13] Pargas, R.P. et al. 1999. Test-data generation using genetic algorithms. Software Testing Verification and Reliability. 9(4): 263-282. [14] Lin, J.C. & Yeh, P.L. 2000. Using genetic algorithms for test case generation in path testing. Proceedings of the 9th Asian Test Symposium. pp: 241-246. [15] Alba, E. & Chicano, F. 2008. Observations in using parallel and sequential evolutionary algorithms for automatic software testing. Computers & Operations Research. 35(10): 3161-3183. [16] Sagarna, R. & Yao, X. 2008. Handling constraints for search based software test data generation. Proceedings of the 8th IEEE International Conference on Software Testing Verification and Validation Workshop. pp: 232-240. [17] Ahmed, M.A. & Hermadi, I. 2008. GA-based multiple paths test data generator. Computers & Operations Research. 35(10): 3107-3124. [18] Beasley, D. et al. 1994. An overview of genetic algorithms: Part 1. Fundamentals. University Computing. 55(6): 1171-1177. [19] Zhou, S.X. & Yang, B. 1996. The influence factors of genetic algorithm and the improved measures Automation of Electric Power Systems. 20(7):24-26. [20] Pei, M. et al. 1994. Automated software test data generation using a genetic algorithm. Michigan State University. 6(1): 1-15. [21] Pargas, R.P. et al. 1999. Test-data generation using genetic algorithms. Software Testing Verification and Reliability. 9(4): 263-282. 383