Issues in Testing Distributed Component-Based. Systems. Sudipto Ghosh. Aditya P. Mathur y. Software Engineering Research Center

Transcription

1 Issues in Testing Distributed Component-Based Systems Sudipto Ghosh Aditya P. Mathur y Software Engineering Research Center 1398 Department of Computer Sciences Purdue University West Lafayette, IN fghosh,apmg@cs.purdue.edu March 22, 1999 Abstract Issues in testing distributed component-based systems are discussed. Dierences in testing such systems and other systems are identied. Several limitations and shortcomings of the existing test methodologies are also identied and a new methodology proposed. Keywords: CORBA, DCOM, component-based distributed systems, fault-tolerance, Java RMI, test adequacy, test methodology. 1 Introduction Testing software systems is a complex problem in itself. With the increasing trend in using distributed software, the task of testing becomes even more complicated. The scalability of testing methodologies and development of testing tools need to keep up with new technologies such as CORBA, DCOM and Java RMI. The process of testing is further complicated by the use of COTS components in the systems. Testers need to test the behavior of such components in systems even if the components have been tested before. Sometimes the components that are reused may not have been designed for systems with the same specication. However, the entire system needs to be tested in order to observe if the system fails because of the use of such components. Thus, software testers need a methodology for: 1. Testing components 2. Testing systems In this work we examine the issues involved in the testing of components and systems. We explain how the distributed nature of the systems makes testing such systems dierent from others. In Section 2 and 3, we describe the issues related to component and system testing respectively. We propose a methodology for testing distributed component-based systems in Section 4. Sudipto Ghosh's work was supported in part by NSF award CCR , a grant from Bellcore, and the Purdue Research Foundation. y Aditya P. Mathur's research was supported in part by NSF award CCR and a grant from the Software Engineering Research Center. All correspondence may be addressed to apm@cs.purdue.edu. 1

2 2 Issues in Testing Components A component is a program that can either request service, provide service or do both. Several techniques are available for testing computer programs [2]. Often a testing methodology provides a measure that quanties \how much" was tested. Code coverage of some sort is a measure of \how much" code is tested. The term coverage domain denotes a set of program related entities that are being counted for measuring coverage. These entities include requirements, functions, statements, decisions, and denition-use pairs. For example, the set of all functions in a C program forms a coverage domain. A test set T is adequate with respect to a coverage domain D, if for each element e D, there exists a t T such that e is covered when program P is executed on t. Empirical evidence provided by many researchers supports the hypothesis that measuring test adequacy and improving a test set against a sequence of well-dened, increasingly strong, coverage domains leads to improved reliability of the system under test [13]. Ideally, one would wish to determine a test set such that the removal of errors detected while testing against this test set would ensure an error-free program. Exhaustive testing, in an ideal environment, would ensure that all errors are revealed. However, exhaustive testing is usually inapplicable in software development environments. Therefore it is recommended that realistic goals be set that account for the resources, the criticality of the components and the system under test and testing be performed such that one or more coverage criteria are satised. We identify the following issues in testing components: 1. Scalability of the test adequacy criteria 2. Test data generation 3. Selection of subsets of components to test 4. Sequence of testing components 2.1 Scalability of the Test Adequacy Criteria If the source code of a component is available, traditional test adequacy criteria can be used. These include control-ow coverage, data-ow coverage, and mutation coverage [2]. For example, the test cases generated by using control ow attempt to cover all the blocks or decisions in the program. A test set is considered adequate with respect to block coverage if all the blocks have been covered. Similarly, a test set is considered adequate with respect to decision coverage if all the decisions have been exercised by the tests. Data-ow coverage looks at the denition-use pairs in the program. The c-use coverage looks at all the denition and subsequent computational use pairs. The p-use coverage looks at all the denitions and subsequent predicate-use pairs of a variable [9]. Data-ow based test adequacy criteria have been extended for use on classes in object-oriented programs [6]. Testing is dened at three levels: 1) Intra-method; 2) Inter-method and 3) Intra-class. Various issues arise during the testing of object-oriented programs. Merely exercising methods is not enough. The order in which these methods are called is also important. Inheritance allows one to create subclasses that share many attributes of the parent class. One could design a testing methodology that tests each class separately. However, that would require re-testing many of the attributes in the subclass that were already tested in the parent class. In [7], an incremental testing technique of object-oriented class structures is proposed. However, these criteria suer from a lack of scalability. While they are useful in the assessment of tests of relatively small programs, such as functions in a C program, they are reported to be not cost-eective when used in the testing of large systems. The diculty arises because of the explosion in the number of possible paths that could be taken to reach the low-level element with an increase in the number of such elements. A decision or a denition-use pair is considered to be 2

3 a low-level element whereas a specic requirement or a function is considered to be a high-level element. To illustrate the path explosion, consider a software system with two functions f and g. Let n be the total number of decisions in f and g combined. Suppose that there is a global variable x used in f. The code of f is as follows: void f() { : if(x > 0) { : } : } The code for g is similar, with x being dened. Then the composition f g, as in a linear execution of f and g, would lead to O(2 n ) computation paths some of which are infeasible. However, if there are no global variables, the number of paths in a composition would increase linearly with the number of functions. The size of coverage domains usually increases exponentially with an increase in the size of components. A few years back, formal test adequacy measures were dicult to use because of the lack of suitable testing tools. Though this may not be a problem any more, the amount of testing required in order to achieve test adequacy is far higher that what is normally permitted by the resources available to most software companies. 2.2 Test Data Generation Test data generation in order to make the test sets adequate with respect to a coverage criterion is a dicult task. Experimental and anecdotal evidence reveals that it is dicult to obtain high coverage for large systems. This is because in a large software system it is often dicult to design test cases that can cover a low-level element. Because the number of possible paths increases exponentially, it is dicult to generate test input that executes all the paths and exercise all the elements of a low-level coverage domain. 2.3 Selection of Subsets of Components to Test A component may need the services of other components. When one tests a component, the question is how to provide these services to the component. One could use stubs that give the results expected by the component from others. However, this will not create the kind of conditions that exist in an actual system environment. There could be a possibility of deadlocks and race conditions only when a component is used with others. Even though correct behavior is exhibited when the component executed alone, it may not be correct when other components are used. The same problem arises when a component is multithreaded. The number of clients used to test the component may create a dierence. There may be no errors revealed when only one client is used because only one thread in the component executes. However, when many clients are used, several component threads may be started and race conditions may be revealed in the code. 2.4 Sequence of Testing Components From the previous sub-section we saw that components may have to be tested along with other components. This brings up the problem of what order should be followed in testing components. One would begin with components that do not require the services of any other component. Subsequent components would be selected such that they use services of components that have already 3

4 been tested. However, this assumes that the application architecture is layered, with the bottommost layer containing components not requiring the services of other components. Each layer l i uses the services of layer l i?1. However, if the interactions within components do not t into the layered architecture, then there could be cycles among components. It is not immediately clear which component one should start with rst. All components may have to be tested together. 3 Issues in Testing Systems We identify the following issues in testing systems: 1. Scalability of test adequacy criteria 2. Redundant testing during integration of components 3. Availability of source code 4. Heterogeneity of language, platforms and architectures 5. Monitoring and control mechanism in distributed software testing 6. Reproducibility of events 7. Deadlocks and race conditions 8. Testing for system scalability and performance 9. Testing for fault tolerance 3.1 Scalability of Test Adequacy Criteria for Testing Systems The problem of scalability of the traditional test adequacy criteria also arises in the context of testing systems. The criteria need to be scalable with the number of components. 3.2 Redundant Testing during Integration of Components Often, components are tested separately and then together in some incremental manner. When the components are tested separately, some test adequacy criteria are used for assessing the quality of test sets. However, when the entire system is tested, the same criteria are used along with coverage domains from other components. This leads to a lot of re-testing of the components. There is a need for studies that helps testers quantify how much of the testing is redundant. If the components are tested well separately, how much extra testing is needed during integration. This issue will be addressed again when we describe the issues that deal with the distributed nature of the systems. 3.3 Availability of Source Code Software components may be built in-house or used o-the-shelf. The developer of a component has access to its source code. The user of an o-the-shelf component usually does not have access to its source code. Depending on the availability of code, dierent testing techniques need to be used for system testing. 4

5 Q Rpq P Rrp Rqr R R1 R2 C1 C2 Figure 1: Deadlock in a component-based distributed System. 3.4 Heterogeneity of Language, Platforms and Architectures The components of a system may by written in dierent programming languages and for use on dierent hardware and software platforms. With middleware conforming to standards like CORBA and DCOM, components can interact with each other independent of the language and the platforms. When a system composed of such components is tested, the testing methodology and tool used must be independent of the language and the platforms. 3.5 Monitoring and Control Mechanism in Distributed Software Testing Distributed software systems bring with them multiple computers on a network. The test monitoring and control mechanism is more complex than that used to control a centralized software system. Mechanisms for centralized or distributed monitoring and control need to be devised. Distributed data collection and storage mechanisms need to be designed taking care of possible data buer overows. Usually the amount of data collected for monitoring and control is large because of the number of components and their long life-times. Care must be taken to ensure that the monitor and controller do not become bottlenecks during the monitoring and control process. 3.6 Reproducibility of Events Concurrent processing and asynchronous communication occur frequently in distributed systems. These together with a lack of full control over the environment leads to the inability to reproduce execution behavior. This is one reason why only testing of components is not enough to predict the reliability of the system. 3.7 Deadlocks and Race Conditions Distributed or concurrent systems occasionally have problems related to race conditions and deadlocks. It is desirable to detect such problems while testing. Consider a case, shown in Figure 1, where there are three components P, Q and R. All of them are single threaded. When a client C 1 makes a request R 1 to P, a request R pq is sent from P to Q. This leads to a request R qr from Q to R. On the other hand, client C 2 makes a request R 2 to R which happens before R qr arrives at R. To serve this request R sends a request to P. However, P waits for Q to reply. Q waits for R and R waits for P. Thus all the components are deadlocked. 5

6 Request m Client Server Callback c Figure 2: Deadlock when using a call-back mechanism. CORBA provides the facility of making one-way calls which are delivered with best-eort semantics. In case the client makes several such calls, the order in which such calls arrive could result in erroneous behavior in case there are race conditions. CORBA also provides call-back mechanisms by which a server can call a client back. This could potentially lead to deadlocks. In Figure 2, the client make a request r to the server and waits. The server makes a call-back c to the client and waits for a response. Thus both are involved in a deadlock. 3.8 Testing for System Scalability and Performance To improve performance, components may be implemented using threads. The same single-threaded system that has been tested perfectly may not work when re-implemented using multiple threads. Concurrency issues, race conditions and deadlocks could occur. Various threading models are used in server implementations and object adaptors in CORBA, each having dierent consequences. Testing for the thread-safe property is a major concern. Stress testing has to be done to ensure that the system performs well under high load factors. The same server components may perform well under small loads, but may cause poor performance or even crash under a heavy load of requests. 3.9 Testing for Fault-Tolerance Software systems are often required to be fault-tolerant. They need to tolerate failures that occur due to faults in its components or in the environment in which the components execute. Applications for nuclear plants, space missions, medical equipment and defense establishments require fault-tolerant software because of the high cost of life and resources attached to them. Any failure in the system would lead to catastrophic results. Stringent rules apply to telephone communication and air-trac control for being available most of the time. These systems have to be fault-tolerant with the availability of backup components or alternative routing mechanisms. In the context of component-based distributed systems like CORBA, there are several faults that could occur. When a client makes a one-way call, there is no guarantee that the request arrives at the server. A client that makes a request to a server may be blocked forever (or may time-out) if the server goes into an innite loop or crashes. It is reasonable to assume that the requirements for fault-tolerance are known at the time of testing. To test for fault tolerance, it is necessary to test the entire system to observe the eects of failures occurring in individual components or in the underlying communication mechanism. Testing for fault-tolerance is a challenge because in most systems, the fault-recovery code rarely gets executed in the test-bed as faults rarely get triggered. As the ability of a system to perform well in the eld in the presence of faults depends on the correctness of the fault-handling code, it is imperative that this code be tested. Fault injection testing [5, 8] is used to test for fault-tolerance by injecting faults into a system under test and observing its behavior. 6

7 The site where a fault is injected is called a fault-site. Code is inserted into the site so that a fault is triggered if control reaches the fault-site. However, testers have faced two kinds of problems with this methodology: 1. Reachability: Injection of faults into a program is not enough to cause a failure. The fault-site must be reached during some execution. 2. Latency: Although a fault has been triggered, the tester might see the manifestation of the fault after control has passed through several modules or components. It may take days before a fault manifests itself. A set of faults is required for the purpose of injection. Having a generic set of faults [1, 3, 4, 10, 11, 12] for a class of system helps in automating the process of fault injection. The faults can be provided in a fault-injection tool and can be selected by the tester for injection. Fault injection has mostly been performed only on selected systems because of a lack of tools that can perform fault-injection testing. MESSALINE, FERRARI, FIAT, SFI, FINE, Tamer, ORCHESTRA and Pisces Safety Net are examples of fault injection tools. 4 Proposed Testing Methodology To address the above issues, we propose a methodology called TDS. Our goal is to reduce testing to what is necessary. We propose test adequacy criteria that are interface-based and are at a higher level than the traditional test adequacy criteria. The leads to a reduction in the size of coverage domains. The basis for our methodology is the need for having a precise denition of an interface of a component. This is possible by using existing technologies like COM, CORBA and Java RMI. In COM and CORBA, an interface description language (IDL) is used to describe the interface of components. In Java RMI, components have Java interfaces. We focus on testing a component through its interface. 4.1 Test Adequacy Criteria An interface in CORBA-IDL contains the declarations of method and exception signatures. We dene the following criteria based on methods and exception coverage: 1. Method coverage: 2. Exception coverage: # methods executed in the component # methods dened in the component's interface # exceptions thrown by the component # exceptions dened in the component's interface However, coverage of methods and exceptions is not enough. The code implemented in the servers will access and modify the parameters passed to the methods and the exceptions. The code will have several paths that depend on the values and types of these parameters. Merely covering a method or an exception by executing it once does not imply that all the paths will be covered. Hence, there is a need to execute the methods by using dierent values of the parameters. Mutation techniques allow us to do that. We dene a set of mutant operators as follows: 1. Replacing an inout parameter with an out parameter and vice versa. 2. Swapping parameters of the same type. 3. Increment/decrement operators for simple types. 4. Set parameter to boundary values. 7

8 5. Nullify object references. 6. For complex types, use the mutant operators on the simple types that are contained. Mutants are created when these programs are applied to the original version of the component. Tests are run in order to distinguish non-equivalent mutants from the original program. A test set is adequate with respect to mutation coverage if the tests kill all the non-equivalent mutants. 4.2 Proposed Methodology We propose that testers create test sets that are adequate to our proposed test adequacy criteria. If errors are revealed, the testers modify the code and remove the errors. 4.3 Assessment of the Methodology Work is in progress for conducting experiments using CORBA systems. Systems range from classroom-type applications to a large-scale help-desk system consisting of several components, being developed in a company. Experiments are needed to compare the proposed test adequacy criteria with the traditional test adequacy criteria. The basis for comparison will be the eectiveness of the criteria in revealing known seeded errors in the components. The eort in generating test cases will also be measured in the number of test cases required for meeting the adequacy criteria. Experiments will also be carried out for evaluating our strategy for testing systems containing such components. 5 Summary We have described the issues in testing component-based distributed systems related to the heterogeneity of platforms, dierences in languages, concurrency, scalability and underlying communication protocols. We have proposed an interface-based testing methodology for testing components and systems. This methodology is currently a part of TDS 1.0, a tool to test CORBA-compliant systems. References [1] J. Barton, E. Czeck, Z. Segall, and D. Siewiorek. \Fault Injection Experiments using FIAT". IEEE Transactions on Computers, 39(4):576{582, April [2] Boris Beizer. \Software Testing Techniques". Van Nostrand Reinhold Company, Inc., New York, [3] R. Chillarege, I. S. Bhandari, J. K. Charr, M. J. Halliday, D. S. Moebus, B. K. Ray, and M. Y. Wong. \Orthogonal Defect Classication a Concept for in-process Measurements". IEEE Transactions on Software Engineering, SE(18):356{363, [4] R. Chillarege and N. Bowen. \Understanding Large System Failures a Fault Injection Experiment". In Proceedings 19th International Symposium Fault-Tolerant Computing, pages 356{363, [5] Clark and Y. K. Pradhan. \Fault Injection". IEEE Computer, 19(11):1087{1094, June

9 [6] M. J. Harrold and G. Rothermel. \Performing Data Flow Testing on Classes". In ACM SIGSOFT Software Engineering Notes Vol. 19, No. 5 in: SIGSOFT '94. Proceedings of the second ACM SIGSOFT symposium on Foundations of Software Engineering, pages 154{163, December [7] M.J. Harrold, J.D. McGregor, and K.J. Fitzpatrich. Incremental Testing of Object-oriented Class Structures. In Proceedings of the 14th International Conference on Software Engineering, pages 68{80, May [8] M. C. Hsueh, T. K. Tsai, and R. K. Iyer. \Fault Injection Techniques and Tools". IEEE Computer, pages 75{82, April [9] P. Jalote. An Integrated Approach to Software Engineering, Second Edition. Springer Verlag, New York, USA, [10] T. Nakajo and H. Kume. \A Case History Analysis of Software Error Cause-eect Relationships". IEEE Transactions on Software Engineering, SE(17):830{838, December [11] T. Nakajo, K. Sasabuchi, and T. Akiyama. \Structure Approach to Software Defect Analysis". Hewlett Packard Journal, 43(23):50{56, [12] D. Siewiorek, J. Hudakund B. Suh, and Z. Segall. \Development of a Benchmark to Measure System Robustness". In Proceedings 23rd International Symposium Fault-Tolerant Computing, pages 88{97, [13] W. E. Wong. \On Mutation and Data Flow, a Thesis". Technical Report SERC-TR-149- P, Software Engineering Research Center, Purdue University, West Lafayette, IN, December