MT2Way Interaction Algorithm for Pairwise Test Data Generation

MT2Way Interaction Algorithm for Pairwise Test Data Generation K. F. Rabbi 1, S. Khatun 2, A.H.Beg 1, 1 Faculty of Computer Systems & Software Engineering 2 Faculty of Electronics and Electrical Engineering University Malaysia Pahng Gambang-26300, Kuantan, Malaysia sabira@ump.edu.my, fazley.rabbi@ymail.com, ahbeg_diu@yahoo.com Abstract. To reduce the number of test cases by utilized the minimum possible amount of time throughout the testing process of software and hardware is highly desirable. For ensuring the stability of the technique the combination of a complete set of available inputs is recommended to be executed. But generally an exhaustive numbers of test cases are hard to execute. Therefore, test data generation is an NP-hard (non-deterministic polynomial-time hard) problem. This is likely to present significant issues in difficulties interpreting the best possible method for generating the test data. The reduction of test cases usually depends on the interaction level. 2-way interaction or pair wise test data can reduce high number of test cases. This efficiently addresses most of the software errors. In this paper has been addressed an effective 2-way interaction algorithm to generate the test data which is more acceptable in terms of the number of test cases. Keywords: Combinatorial interaction testing, Software testing, pairwise testing, Test case generation 1 Introduction Software testing and debugging is one of the important part of software development life cycle in software engineering but this process is still very labor-intensive and expensive [1]. The most practical and realistic project management is characterized by an effective and well balanced allocation of project resources. But this is generally perceived that half of the project resources are absorbed in in initiatives for presenting highly efficient software applications and hardware devices. All the researcher and practitioners focus on finding automatic cost-effective software testing and debugging techniques which can maintain the high error detection rate and ability to ensure high quality of software product release [1, 2]. Nowadays, software engineering research on software testing focus on test coverage criterion design, test generation problem, test oracle problem, regression testing problem and fault localization problem [1]. Therefore, test generation problem is an important issue to produce error free software [1]. Just to recognize the volume of the problem a system which is constituted by 7

considering only six parameters with ten values is predicted to produce 106 numbers of test cases. Obviously this much number of test cases is un-realistic as it is expected to miserably fail to keep itself well within the boundaries of time and resource constraints. Therefore it is very important to reduce the number of test cases keeping the effectiveness of detecting errors [3]. To fix this issue pairwise interaction technique is one of the most useful as it keeps good stability between the quality and effectiveness of combinations. To comprehend the issue more temporarily, the follow the option dialog in Microsoft excel view tab option, there are 20 possible configurations needed to be tested along with the Gridline colors (which takes 56 possible values). Each other configuration takes two values (checked or unchecked). Therefore, to test this view tab exhaustively there is 220 x 56 i.e. 58,720,256 (2 is the values and 20 is the number of configurations) number of test cases need to be executed. Assuming each test case may consume 5 minutes to execute; results around 559 years to complete the exhaustive test of this view tab [4]. This is also identical for hardware products as well. A product which has 20 on/off switches will take 220 = 1,048,576 test cases. It will take 10 years if each single test case needs around 5 minutes to execute [4]. Currently, research work in combinatorial testing aims to generate least possible test cases [5]. The solution of this problem is NP-hard (non-deterministic polynomial-time hard) [6]. So far many approaches have been proposed and also many tools have been developed to find out the least possible test suit in polynomial time [5, 12]. 2 Related Work There are some strong empirical facts [7], [8] that lack of testing of both functional and nonfunctional is one of major source of software and systems bug/errors. National Institute of Standard and Technology (NIST) estimated that the cost of software failure to the US economy at $6 X 1010, which was the 0.6 percent of GDP [9, 10]. Another review found that more than one-third of this price can be reduce by improving software testing structure. To improve software testing structure automatic testing is a critical issue [10, 11]. In the course of automation, software can become more practical and scalable. However, the automated generation of test case still represents challenges [10]. The underlying problem is known to be un-decidable and NP-hard thus researchers have focus on the techniques that search to identify near optimal test sets in a reasonable time [12]. Basically, each test suite constructed by N k array where N represents the number of test cases. Each of the single test cases is a combination of the k parameter values [13]. In test data generation, it is normally focused on all t-way combinations of inputs and the values occur in the test suite. In the t-way t determines the interaction strength of test suite. Thus, the single setting of the system that needs to be tested is defined by the single test case. In a nut shell, test data generation strategies can generate test cases in non-deterministic or deterministic way. Fig. 1 shows the most available test data generation strategies. 8

Fig.1: Test case generation categories The Automatic Efficient Test Generator or AETG [14, 15] and its deviation maetg [16] generate test data using computational approach. This approach uses the Greedy technique to build test cases based on covering as much as possible uncovered pairs. AETG uses a random search algorithm. Genetic Algorithm, Ant Colony algorithm is the variants of AETG. Genetic algorithm [17] creates an initial population of individuals (test cases) and then the fitness of those individuals is calculated. Then it starts discarding the unfit individuals by the individual selection methods. The genetic operators such as crossover and mutation are applied on the selected individuals and this continues until a set of best individuals found. Ant Colony algorithm [17, 18] candidate solution is associated with the start and end point. When an ant chooses one edge among the different edges, it would choose the edge with a large amount of pheromone which gives the better result with the higher probability. The In- Parameter-Order [19] or IPO Strategy starts with an empty test set and adds one test at a time. It creates the test cases by combination of the first two parameters, and then add third and calculate how many pair is been covered then add fourth and calculate and then fifth and calculate until all the values of each parameter is checked. This approach is deterministic approach. G2Way [17] is based on backtracking algorithm uses computational deterministic strategy. G2Way uses customized markup language to describe base data. The G2Way backtracking algorithm tries to combine generated pairs so that it covers highest pairs. Finally when it covers all the pairs, the test case treats as a final test suite. 9

3 Proposed MT2Way Algorithm The MT2Way algorithm has been divided into three steps. The three steps are as follows: 3.1 Pair generation 3.2 Tuple generation 3.3 Test case construction 3.1 Pair generation Pair wise algorithm first takes the parameters and corresponding values, makes the 2 way combinations. The combinations stored into the memory. As an example Parameter A has 3 values a1, a2, and a3 Parameter B has 2 values b1, b2 Parameter C has 4 values c1, c2, c3, c4 The pair generation algorithm first generates header pairs. The generated header and value is shown in Table 1. Table 1: Generated header and value pairs Pairs Value AB (a1, b1), (a1, b2), (a2, b1), (a2, b2), (a3, b1), (a3, b2) AC (a1, c1), (a1, c2), (a1, c3), (a1, c4), (a2, c1), (a2, c2), (a2, c3), (a2, c4), (a3, c1), (a3, c2), (a3, c3), (a3, c4) BC (b1, c1), (b1, c2), (b1, c3), (b1, c4), (b2, c1), (b2, c2), (b2, c3), (b2, c4) After creating the header pairs, it calculates the possible values into all the pairs. In this example, AB contains 3 x 2 = 6 values (pairs). The expression can be written as follows: NP = [V1 * V2,] where NP = Number of Pairs, V1 = Value of Parameter 1, V2 = Value of Parameter 2 Therefore it can be find that, in the pair generation algorithm stores all the value pairs into the memory which are 6, 12, and 8 respectively for AB, AC, and BC. 3.2 Tuple generation Tuple generation algorithm first generates the N-tuple based on the pairs, where N is the number of the parameters. To make a complete N-tuple from a pair, the values are replaced by 0 which is not exists in that pair. As an example, the generated 3-tuples can from Table 1, which can make the algorithm more understandable. Table 2 illustrate the N-tuple, 10

Table 2: Generated N-Tuple from Table 1 Pairs AB AC BC N-Tuple from Pairs (a1, b1, 0) (a1, b2, 0) (a2, b1, 0) (a2, b2, 0) (a3, b1, 0) (a3, b2, 0) (a1, 0, c1) (a1, 0, c2) (a1, 0, c3) (a1, 0, c4) (a2, 0, c1) (a2, 0, c2) (a2, 0, c3) (a2, 0, c4) (a3, 0, c1) (a3, 0, c2) (a3, 0, c3) (a3, 0, c4) (0, b1, c1) (0, b1, c2) (0, b1, c3) (0, b1, c4) (0, b2, c1) (0, b2, c2) (0, b2, c3) (0, b2, c4) From Table 2, all the pair values converted to a set of test case i, e, N-tuples and the missing parameter is replaced by 0. 3.3 Test case construction In this part of the algorithm, the missing parameters try to adjust with the all possible values. It selects the value from the missing parameter position which gives the highest possible pair coverage. From N-Tuple the 0 indicates the missing parameter and it s positing. 4 Result and Discussion In order to study the effectiveness of the proposed approach, there have been eight system configurations was tested with the proposed strategy. The following systems are: 11

S1: 3 3-valued parameters, S2: 4 3-valued parameters, S3: 13 3-valued parameters, S4: 10 10-valued parameters, S5: 10 15-valued parameters, S6: 20 10-valued parameters, S7: 10 5-valued parameters, S8: 1 5-valued parameters, 8 3-valued parameters and 2 2-valued parameters. The generated test case by each algorithm is shown in Fig.2. The comparative result shows that, MT2Way generates test cases each time which is competitive to other well-known strategies. Fig.2: Comparison Based on the Test Size 12

5 Conclusion In this paper has been presented Tuple based algorithm MT2Way for test case generation for pairwise testing. The algorithm is able to reduce test suites effectively compared to other algorithms. The proposed algorithms could be further extended to support higher t-way interaction testing. Acknowledgements Appreciation conveyed to Ministry of Higher Education Malaysia and University Malaysia Pahang for project financing under the Fundamental Research Grant Scheme RDU100361 References 1. Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: A general strategy for t-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems. pp.549--556 (2007) 2. Cui,I., Longshu, L., Sheng, Y.: A New strategy for pairwise test case generation. In: Third international Symposium on Intelligent Information Technology Application (2009) 3. Mohammed, I. Y., Kamal, Z. Z., Nor A. M.I.: Algebraic Strategy to Generate Pairwise Test Set for Prime Number Parameters and Variables. In: IEEE international conference on computer and information technology (2008) 4. Xiang, C., Qing, G., Jingxian, Q., Daoxu, C.: Applying Particle Swarm optimization to Pairwise Testing. In: 34th Annual IEEE Computer Software And Application Conference (2010) 5. Mohammad, F. J. K., Sangeetha, M., Ahmad, N., Sidek, R.: A Tree Based Strategy for Test Data Generation and Cost Calculation for Uniform and Non-Uniform Parametric Values. In: 10th IEEE international conference on computer and information technology (2010) 6. Harman, M., Jones, B.F.: Search-based Software Engineering & Information and Software Technology. Vol: 43, pp.833-839 (2001) 7. Leffingwell, D., Widrig, D.: Managing Software Requirements: A Use Case Approach. Addison Wesley (2003) 8. Glass, R.L.: Facts and Fallacies of Software Engineering. Addison Wesley (2002) 9. National Institute of Standards and Technology: The Economic Impacts of Inadequate Infrastructure for Software Testing. Planning Report 02-3 May (2002) 10. Mark, H., McMinn, P.: A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search. IEEE Transactions On Software Engineering, vol. 36, no. 2 (2010) 11. McMinn P.: Search-Based Software Test Data Generation: A Survey. Journal of Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105--156 ( 2004) 12. Gong D., Yao, X.: Automatic detection of infeasible paths in software testing. IET Software, iet-sen.0092 (2009) 13

13. Chen, X., Qing, G., Jingxian, Q., Daoxu, Chen, D.: Applying Particle Swarm Optimization to Pairwise Testing. In: IEEE 34th international conference on Computer Software and Applications Conference (COMPSAC), vol. 17, pp. 1(2010) 14. Cohen, D. M., Dalal, S. R., Fredman, M. L., Patton, G. C.: The AETG System: An Approach to Testing Based on Combinatorial Design. IEEE Transactions on Software Engineering, vol. 23, pp. 437 -- 444 (1997) 15. Cohen, D. M., Dalal, S. R., Kajla, A., Patton, G.C.: The Automatic Efficient Test Generator (AETG) System. In: Proceedings of the 5th International Symposium on Software Reliability engineering, Monterey, CA, USA, pp. 303--309(1994) 16. Younis, M.I., Zamli, K.Z., Mat I. N.A.: IRPS - An Efficient Test Data Generation Strategy for Pairwise Testing, in Proceedings of the 12th International conference on Knowledge- Based Intelligent Information and Engineering Systems, Lecture Notes In Artificial Intelligence; Springer-Verlag, Vol. 5177, pp. 493--500 ( 2008) 17. Shiba, T., Tsuchiya, T., Kikuno, T.: Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing. In: 28th Annual International Computer Software and Applications Conf. (COMPSAC 04), pp. 72-77, Hong Kong (2004) 18. Chen, X., Qing, G., Zhang, X., Chen, D.: Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization. In: IEEE 9th International Conference on Quality Software, pp. 347 -- 352 (2009) 19. Lei, Y., Tai, K. C.: In-Parameter-Order: A Test Generation Strategy for Pairwise Testing. In: 3rd IEEE International High-Assurance Systems Engineering Symp, Washington, DC, USA, pp. 254--261(1998) 14