Issues in Testing Electronic Commerce Systems Kassem Saleh 1 and Robert Probert University of Ottawa, School of Information Technology and Engineering (SITE) P.O.Box 450, Stn A, Ottawa, Ontario, Canada K1N 6N5 Abstract. Electronic Commerce (EC) systems can be considered as specialized instances of distributed processing systems. As for any distributed system, an EC system is architecturally composed of multiple, decentralized and autonomous processing entities or agents exchanging messages across the network. EC systems are also reactive, real-time, concurrent and open systems. They are financially-critical (as opposed to safety-critical) systems since they perform one or more distributed business functions or transactions whose successful execution contributes positively to the overall corporate business. Testing an EC system implementation requires ensuring the conformance of the system implementation to its business function specification. The use of sound and formal methods for testing such systems is essential for guaranteeing a high level of reliability and trustworthiness. In this paper, we discuss how existing testing methodologies can be applied to testing EC systems. Keywords. Electronic commerce, distributed processing, software systems, testing. 1. Introduction Electronic Commerce (EC) systems [1] can be considered as specialized instances of distributed processing systems. As for any distributed system, an EC system is architecturally composed of multiple, decentralized and autonomous processing entities or agents exchanging messages across the network. EC systems are also reactive, real-time, concurrent and open systems. They are financially-critical (as opposed to safety-critical) systems since they perform one or more distributed business functions or transactions contributes positively to the overall corporate business. Testing an EC system implementation requires ensuring the conformance of the system implementation to its business function specification. The use of 1 On sabbatical leave from Kuwait University, Department of Electrical and Computer Engineering
sound and formal methods for testing such systems is essential to guarantee a high level of reliability. Obviously, traditional black box and white box software testing techniques [2] are very useful for ensuring that each software component conforms to its specification. However, testing the quality of assembling and integrating these components in a distributed environment requires more sophisticated testing tools, environments and procedures. In such environments, we have to deal with concurrency, synchronization and communication issues. Also, in the context of EC systems, testing the system s security, integrity and vulnerability, and the data/databases integrity is of great importance. In addition to the basic test architectures, a distributed test architecture and setup is needed to provide a realistic testing environment with a high level of testability (i.e., observability and controllability). Finally, because of the need for unambiguous (deterministic) testing, a formal standard test specification language is required [14]. Basically, an electronic commerce software system provides many functionalities distributed over several business areas. The degree of distribution and complexity varies among these areas. Such areas include product cataloguing, advertising and marketing, order execution, billing, and post-order support and management. A typical EC system requires the collaboration of several entities (or agents) to complete an order transaction (Figure 1). The orderly and timely exchange of messages among those agents ensure the correct execution of an EC transaction. These agents are, in addition to the consumer and business, the financial institution, the payment server, the shipper and the supplier. Figure 2 shows a typical message sequence diagram representing a successful termination of a single transaction. In this diagram, a *-labeled interaction indicates that the message must be encrypted. Figure 3 shows a partial finite state machine description of the interactions between these agents (corresponding to Figure 2). At a more detailed level, each agent in turn can be described using the extended communicating finite state machines model. The
improper handling of concurrency resulting from the simultaneous execution of order transactions by the business and other agents may lead to abnormal and non-deterministic behaviors. These issues must be dealt with at the agents design level and tested for conformance later on. In this paper, we propose a generalized framework for testing EC systems. The framework includes the adaptation of the ISO s Conformance Testing Methodology and Framework (CTMF) [3] for protocols and the necessary extensions needed for EC systems [4]. The rest of this paper is organized as follows. Section 2 introduces testing issues in the context of EC systems. Section 3 briefly presents the research and standards in protocol systems testing and proposes an Electronic Commerce Testing Framework (ECTF) as a specialization of the CTMF for protocols. Finally, in Section 4, we conclude the paper and discuss some future work to be addressed. 2. Testing Issues This section presents some of the issues and aspects that have to be considered in the process of testing electronic commerce software systems. First, we start by addressing the different software testing goals pertinent to EC systems. Then, we deal with the mechanisms to achieve our testing goals. 2.1. What to test? Like any software system, EC software must be tested against its functional specification. In addition, because of its distributed and real-time (on-line) nature, other pertinent software qualities have to be checked thoroughly and formally. Basic (Functional) Transaction Testing The functional software specification must be decomposable into functions or business transactions with multiple scenarios. Ideally, these scenarios are described formally or semiformally using use cases or Message Sequence Charts-like (MSC) representations as shown
in Figure 2. The aim here is to test each transaction separately without interference or concurrency from other transactions. Also, this test can be performed locally, i.e., consumer and client are co-located on the same machine. Robustness and Fault Tolerance For each of the transaction, we should be able to develop additional scenarios that may or may not be explicitly included in the functional specification. Specifically, we should aim at scenarios dealing with abnormal behavior of the operating environment that must be dealt with properly in the software. Avoiding testing these aspects often leads to intolerant software, and hence, severe financial consequences may result. It is important to check the error recoverability aspect of the system since it affects its reliability. Ideally, secondary use cases must exist in the specification, and can be used to design the corresponding test cases. Load and Stress The scenario shown in Figure 2, involves a single transaction. However, in real-life, some agents may have to deal with a substantial number of simultaneous or concurrent requests. Obviously, the business, financial institution and payment server agents must be able to deal with a heavy load of simultaneous transactions. Test cases must be designed to check how the software behaves under such operating environment (i.e., stress testing). Ideally, in the system specification, an upper limit on the number of simultaneous transactions must be explicitly mentioned. Quality of Service (QoS) Test cases must be designed to deal with QoS issues and parameters specified in the system specification [5]. For example, multimedia objects transfer delays are of importance in the interactions between consumer and business agents when browsing and downloading catalogue products.
Performance Testing Test cases for the measurement of some predetermined performance indicators for EC systems must be developed. These indicators are obtained from the real-time system requirements and may include various transfer delays and end-to-end connectivity delays. Real-Timeliness Hard real time requirements may be included in the specification (i.e., ideally in the use cases themselves). These requirements must be met in the system s implementation. Test cases dealing with this aspect must be designed. Moreover, these requirements have to be checked along with stress and robustness testing. For example, the timing requirements may be satisfied for a single transaction but not when considering a heavy load or when dealing with an abnormal condition (i.e., the recovery time is substantial). Security / Integrity Test cases must be designed to ensure that financially critical message exchanges are highly secure. Any attempt to modify such messages must be detected. The setup for these types of test cases must be carefully designed. For example, as shown in Figure 2, the messages carrying consumer credit card information and bank authorizations must be secured. Data / Database Integrity Another aspect of testing, often discarded in traditional software testing (although it is a big source of software errors), is to ensure the integrity of data flows and data in permanent data stores (i.e., databases). Test cases must be designed to check the validity of the data store before and after the execution of a transaction. These test cases must be performed under different operating environment (i.e., under stress and abnormal conditions). For example, a transaction that was not approved by the financial institution must leave certain databases intact, except for log files. All designed test cases must specify which data stores are affected and how.
Interoperability / Platform Independency / openness All the above aspects of testing must be repeated for all supported operating environments. These environments may include operating systems, database systems among other system components. This requirements is vital for the success of EC systems, since the proper support of the EC system implies more consumer usage. 2.2. How to Test In the following, we briefly discuss the mechanisms that are needed to perform the testing activities and achieve the objectives described above. Architecture and Setup Since we are dealing with a distributed system, most of the test cases require execution in a distributed test environment. However, basic functionality test cases may be executed in a local test environment. A distributed test setup may require various test coordination procedures, the specification of points of observability and controllability to monitor the progress and traceability of the transaction execution and the correctness and diagnosability of the test outcomes. Test Case Generation, Selection and Specification Ideally, the various test cases can be (semi) automatically generated from the different use cases described in the specification. In addition, test cases can be generated from the various communicating finite state machine descriptions of the EC agents. Often, the generation process is exhaustive, therefore, we should aim at selecting enough test cases to cover all the functionalities at least once, in addition to the test cases ensuring robustness, recoverability and other aspects as discussed earlier. Moreover, the use of formal and readable test specification language would be desirable for writing unambiguous test cases and for test automation and results analysis.
Realization: Application and Automation The automation of test case application is very desirable since we are be dealing with a large number of test cases that may be executed very often. In addition to reducing the testing time, automation will make the testing process less error prone. Consequently, a more reliable software will be delivered. Test Results Analysis After executing the test cases, the process of analyzing the test outputs can be automated. Also, in case a test case fails (i.e., the expected test output specified in the test case itself does not match the test execution results), we can use diagnostics and traceability information to interprete the results to be able to fix the error in a shorter time. 3. Testing in Protocols and Distributed Systems As stated earlier, electronic commerce systems are specialized types of distributed systems consisting of many collaborating entities. Moreover, communications protocols constitute the backbone of such systems. Therefore, we believe that most research results on testing protocols and distributed systems are applicable to electronic commerce systems. This is in addition to the basic research on traditional software testing techniques. 3.1. Protocol Testing Research In the following, we present a brief overview of the basic research results obtained in protocol testing. Readers interested in a recent tutorial article on the state-of-the-art in protocol and distributed systems testing can refer to [6]. Also, books [7,8] and international conferences and workshops [9-13] deal with protocol testing issues. Test case generation, selection and coverage Many formal methods have been introduced to provide test cases for testing protocol implementations starting from finite state machine descriptions of the communicating protocol entities. Tools have been developed for automating these test case generation methods. Moreover, test cases are automatically generated from uses cases and System
Description Language (SDL) specifications. After the generation of test cases and because of the limitations in testing time, techniques for selecting the error-revealing high-yield test cases are needed. Techniques for developing test coverage metrics need to be further developed. Testing CFSMs Recently, formal methods have been introduced to generate test cases to test the complete interactions between communicating finite state machine specifications of collaborating agents (i.e., the distributed system as a whole compared to testing individual entities one at a time). Testing in Context Also, in a system of many interacting entities, testing only one entity in the context of its environment has been addressed. Testing in context can be useful to effectively generate test cases and execute them when doing regression testing. 3.2. Protocol Testing Standards In addition to the basic research, significant achievements have been made in developing international standards for testing protocols and distributed systems. These standards include a test specification language (TTCN the Tree and Tabular Combined Notation), and test architectures and setup guidelines. Conformance Testing An international standard has been developed to provide a framework and methodology for testing protocols systems and distributed systems in general. IS 9646 Conformance Testing Methodology and Framework CTFM - is divided into three parts. Part 2 describes the different abstract test architectures and models of conformance testing based on degrees of controllability and observability.
Test Specification Language Part 4 of IS 9646 provides a description of a notation for describing test cases the Tree and Tabular Combined Notation (TTCN) [3, 14]. This language can be used for the concise documentation of test cases allowing easy readability and maintainability, and facilitates test automation. 3.3. Highlights of an Electronic Commerce Testing Framework Given the wealth of research, tools and standards developed for protocol and distributed systems testing, we should be able to develop a complete framework and guidelines for testing electronic commerce systems. The main components of this framework are: 1) guidelines on how to apply the most suitable high-yield protocol and software testing activities in the context of EC software, 2) the use of test specification language to describe business transaction oriented test cases, 3) guidelines on how to describe and test the realtime and quality of service requirements for EC systems, and finally, 4) a test architecture involving multi-party distributed testing with high testability requirements in mind, 4. Conclusions and Future Work In this paper, we have addressed some issues related to testing electronic commerce systems. We argued that testing such systems can benefit from the advances and results achieved in protocol testing. As a future work, we intend to develop, as a contribution towards an Electronic Commerce Testing framework (ECTF), a UML-like methodology for capturing and validating service requirements specifications which is also compatible with Message Sequence Charts (MSC), an international standard notation used to describe individual interaction scenarios between distributed customers and systems. This provides a means of integrating the best methods in both Protocol Engineering and Object-Oriented Software Engineering. We aim at developing automated and semi-automated techniques and tools to generate high yield functional tests and service conformance tests in a suitably extended notation based on the international test specification language, concurrent TTCN. This will
help to make the high-yield software quality engineering process cost effective and minimize overall time to market. Acknowledgement The authors would like to acknowledge support of this work by an NSERC operating research grant. References [1] N. Adam, O. Dogramaci, A. Gangopadhyay and Y. Yesha, Electronic Commerce: Technical, Business and Legal Issues, Prentice Hall, 1998. [2] Beizer, B, Software testing techniques (2nd Edition) Van Nostrand Reinhold (1990) [3] ISO International Standard 9646 (1995) Information Technology Open Systems Interconnection Conformance Testing Methodology and Framework. [4] T. Walter, I. Schieferdecker and. Gabrowski, Test architectures for distributed systems State of the art and beyond, Invited paper at the International Workshop on Protocol Testing, August 1998. [5] G. Bochmann, First IBM Symposium on Technological Challenges in Electronic Commerce, September 1998. [6] R. Dssouli, A. En-Nouaary, and K. Saleh "Test development for communication protocols: Towards automation", To appear in Computer Networks and ISDN Systems. [7] B. Sarikaya, Principles of Protocol Engineering and Conformance Testing, Ellis Horwood, 1993. [8] K. Tarnay, Protocol Specification and Testing, Plenum Press, 1991. [9] Computer Communications, Special Issue on Protocol Engineering, Vol. 19, No. 14, December 1996. [10] IFIP Transactions on Protocol Specification, testing and verification (PSTV) (yearly since 1980). [11] IFIP Transactions on Formal Description Techniques for Distributed Systems and Protocols (FORTE) (yearly since 1989). [12] IFIP Transactions on Protocol Test Systems (IWPTS) (yearly since 1988). [13] International Conference on Network Protocol (ICNP) (yearly since 1993). [14] R.L. Probert and O. Monkewich, TTCN: The international notation for specifyingtests for communications systems, Computer Networks and ISDN Systems, Vol. 23, No. 5, pp. 417-438, 1992.
Consumer Financial Institution Business Transport Supplier Shipper Payment Server Figure 1. Electronic commerce collaborating entities.
Shopping Cart Catalogues Supplier Shipper Consumer Business Payment Server Financial Institution order form download browsing place order * order completed ship ordered items order confirm order date update inventory updated forward order for bank approval order approved * approval request *transaction approved Figure 2. Message sequence diagram of a typical order execution transaction. 2* Consumer 13 1 3 Business 8 12 11 10 4 7 9 Payment Server 6* 5* Financial Institution Shipper Supplier Figure 3. Partial description of agents interactions.