Complex event processing in reactive distributed systems Ján JANÍK Slovak University of Technology Faculty of Informatics and Information Technologies Ilkovičova 3, 842 16 Bratislava, Slovakia xjanikj@is.stuba.sk 1 Introduction Abstract. Specific to the reactive computing is processing of potentially large volumes of incoming messages or events from various sources. It can be assumed that the number of resources and their diversity will only grow. For that reason we have analyzed common approaches in distributed systems development. We then introduce complex event processing and try to evaluate its assets concerning design, implementation, modifiability and flexibility. For this purpose, we implemented a functional model of an electronic toll collection system, based on system specification for Slovak conditions. Significant flexibility is achieved compared to those of the existing solutions. Thanks to growing computational power, ubiquitous network and the significance of IT in everyday s businesses, distributed systems are one of the most prevalent application types being developed today. Furthermore, as information systems take over more and more responsibility, more and more realtime (or reactive) systems are being developed. A distributed system can be defined as a system consisting of multiple independent (selfcontained) components that communicate with each other and coordinate their activity to achieve certain goal. Self-contained components means that each component has its own control mechanism and system resources. Typically, a distributed system consists of several computers communicating across network. A reactive system is a system that needs to response to changes in conditions (events) in a specific time interval, typically very short. Examples of reactive systems include all types of security and monitoring systems, fraud detection, industrial process controllers, flight operation systems or even a video player software. Logically, as distributed systems consist of more components, its complexity gets bigger as well. Another problem is the everincreasing amount of digital data. Although computational power increases rapidly too, Kenneth Cuckier assumes in an article in The Economist, that by this year Master study programme in field: Information Systems Supervisor: Dr. Ján Lang, Institute of Informatics and Software Engineering, Faculty of Informatics and Information Technologies STU in Bratislava IIT.SRC 2011, Bratislava, May 4, 2011, pp. 1 8.
2 UNSPECIFIED FIELD (2011), the amount of information created will exceed available storage more than two times [2]. Information systems, which are being more and more interconnected, now have to process a lot of diverse data. In such conditions, developing a real-time system, which needs to response promptly, without proper efficient technique(s) will in most cases lead to failure. In this paper, we looked at complex event processing (CEP) as a new approach addressing these problems and tried to evaluate its assets in reactive distributed systems design. In section 2 we briefly introduce common approaches in distributed systems design. Section 3 explains the basics of complex event processing. Section 4 explains how we used complex event processing in designing and implementing an electronic toll collection system. Results are mentioned in section 5. 2 Common approaches in distributed systems design Distributed systems first emerged with the development of computer networks. As the development progressed, common techniques for designing and implementing such systems were established. Two interesting concepts gained importance - service oriented architecture and event-driven architecture. 2.1 Service oriented architecture Service oriented architecture is an architectural style, in which functionality is provided by services, which can be discovered and used by clients. Clients and services are thus two main components defined by this architectural style. Service is a software entity which provides some basic or complex functionality to its clients. A client can be any software program, including another service. Other elements specified by SOA are mechanism of message transfering, communication protocol, service registry and others. It is almost a standard to use internet and HTTP protocol for communication, while services are located on different computers, thus creating a distributed system. It is however important to remember that SOA is more about concept and philosophy than about implementation specifics. The main goal of service oriented architecture is to increase reusability of individual components and flexibility and modifiability of large business systems, which were having problems with ever changing requirements, technologies et. al. [3]. It is thus very suitable for system integration. 2.2 Event-driven architecture Most computer systems are based on continuous execution of instructions, which means that the system itself determines what to do next. Event-driven architecture, on the contrary, is an architectural pattern in which system reacts to incoming data, represented by events. David Luckham defines an event as anything that happened or is currently happening [5]. When modeling an event-driven architecture, one has to identify all types of events that can occur in the problem domain. In an event-driven system, real world events are represented by software events. Software events need to have a common structure so they can be automatically processed by machines. Each software event is an instantiation of an event type, which defines what data an event carries. It is an analogy to object oriented programming, with event types standing for classes and (software) events standing for objects. Basic components of an event-driven system are event publishers (or event sources), event consumers and communication channels. Most important characteristics of event-driven architecture are that events are published asynchronously (the publisher does not wait for the consumer to process the previous event) and right after they occur. Main advantage of EDA is loose coupling of components (components don t know about each other). When compared to SOA, event-driven systems are asynchronous, opened to unexpected input, program flow is determined by data and relations between components are M:N (as opposed to SOA, where they are 1:1) [1]. We can see that characteristics of SOA and EDA are complementary and both architectural styles can be well used together.
Ján Janík: Complex event processing in reactive distributed systems 3 3 Concepts of complex event processing Complex event processing is one of three general styles of event processing in event-driven architecture (the others being simple event processing and event stream processing). A complex event is created by analyzing and identifying the cause-and-effect relationships among other events, which is done by event correlation and aggregation. In a simplified way, a complex event consists of other events, which can be also complex, thus creating an event hierarchy. Furthermore, time and quantity constraints can be applied to event correlation, which significantly increases the capabilities of CEP. A good example can be an electronic banking system detecting a credit card fraud when two withdrawals from distant places are detected in a short time (short enough to be impossible to get from one place to the other) [6]. In this case, the system registers two events of type Withdrawal, each originating from a different place, that happen over a short period of time. When all of the conditions are met, the system detects a complex event Credit Card Fraud and can automatically carry out any necessary actions, like freezing the account and alerting the bank s employees. Event correlation is conducted by a special software component called event processing engine. Its efficiency is crucial to the whole system, as it must be able to process large amount of data in a very short time. Usability is also critical. CEP engine must provide an easy way to define complex events. Most existing event processing engines provide a specialized language, which is very similar to SQL (an overview of existing CEP engines can be found in [4]). Statements defining complex events contain a SELECT-FROM-WHERE clauses, with the only difference from SQL being that data are not persistent and stored, but processed as they arrive in a data stream. Because complex event processing is part of event-driven architecture, CEP systems have the same characteristics as EDA. The main contribution of CEP is simplification of implementation and efficiency of the system. Complex event processing is used in a range of applications, most notably in business process management, fraud detection systems, monitoring systems (industrial process monitoring, network monitoring, traffic monitoring), algorithmic trading, system integration and others [5]. We focused on the possibilities of utilization of CEP in real-time distributed systems. 4 Application of CEP in electronic toll collection system design We formulated following assumptions about the CEP technology: High modifiability and flexibility of CEP systems. Ability to change the system run-time. Suitable for system integration. Simple implementation (compared to common design and implementation techniques). High efficiency (processing up to tens of thousands of events per second). The goal of our work is to evaluate these assumptions when applied in reactive distributed system design. For this purpose, we decided to implement an electronic toll collection (ETC) system using complex event processing. Such system meets all the requirements: it works in an asynchronous environment, with large amount of data that need to be processed in real time, it should be easily modifiable and integrated with other systems. Another reason why we decided to implement this system is that it is an actual topic not only in Slovakia, but around the world. It is however important to mention that we implemented only a functional model, not the whole system, which exceeds the extent and focus of our work. 4.1 Designing a complex event processing system When designing a CEP system, additional steps must be conducted. First, event sources must be defined. In our functional model, we defined two event sources: on-board units (OBU) and tollgates.
4 UNSPECIFIED FIELD The former is a small device installed in clients vehicles, which constantly monitors their location and sends data to server as soon as it detects a transit through a toll road segment. The latter is a static enforcement equipment build along the toll road segments, which can communicate with OBUs of passing vehicles and check if they have OBU set up correctly and running. Second step consists of identifying all the event types in the problem domain. We defined ten event types - three basic, which originate directly from identified sources, and seven complex events. Every event type must have defined what data it carries (more precisely, what data events of this type carry), its source, input condition and what action to execute. Basic event types in proposed tolling system, their sources and data, are showed in table 4.1. Event type (event ID) Event source Data Toll Road Segment Transit (TRST) OBU vehicle ID, segment IT, timestamp Tollgate Vehicle Identified (TVI) Tollgate gate ID, vehicle ID, license number, timestamp Tollgate Vehicle Unidentified (TVU) Tollgate gate ID, license number, timestamp Table 1. Basic event types Toll Road Segment Transit occurs whenever a vehicle passes a toll road segment. In reaction to this event, payment from proper client s account must be carried out. Tollgate Vehicle Identified originates from tollgates and occurs whenever a vehicle passing a tollgate is properly identified, that means the tollgate successfully communicated with the on-board unit of the vehicle. On the contrary, Tollgate Vehicle Unidentified events arise when a vehicle passes a tollgate and the communication with the on-board unit was unsuccessful. In this case, the system should trigger an alarm and notify enforcment department. We will only present one complex event type as an example because mentioning all would exceed the extent of this paper. A good example is the event type Toll Road Transit Unregistered. Events of this type occur when a vehicle passes through a toll road segment, but the back office system receives no data about this fact from the on-board unit of the vehicle. To detect this event by CEP engine, two preconditions must be met: 1. The passing vehicle must be detected. This can be done only by tollgate, therefore a Tollgate Vehicle Identified event must occur first. Moreover, this event says that the vehicle s OBU is up and running and has responded correctly. 2. No data about passing the segment from the OBU are recieved. That means, no Toll Road Segment Transit event (with proper vehicle ID and segment ID) is received in a given time interval. This event can indicate a technical problem, for example a malfunctioning GPRS module of the OBU, but also a potential fraud. In reaction, the system should issue a warning. With all the event sources and event types identified, the system design then continues with usual steps like architecture definition etc. Small portions of the rest of the design and implementation are explained in the next section. 5 Results In this chapter we present the results we reached mainly in the stages of design and implementation.
Ján Janík: Complex event processing in reactive distributed systems 5 Figure 1. System architecture without CEP 5.1 Assets of CEP in system design To demonstrate the advantages of complex event processing in system design, we proposed a second system architecture, which does not use CEP, and compared it to the actual architecture of our system. This architecture is displayed in figure 1. There are eight components. Adapter OBU and Adapter Tollgate are software and hardware modules, which are responsible for receiving data from on-board units and tollgates and transforming them into proper format, so they can be processed by the rest of the system. Because there is limited space, we won t explain the purpose of the rest of the components. As virtually all of the components represent individual subsystems, effective and transparent communication is cruicial for the system. We decided to use enterprise service bus to address this requirement and services to implement most of the functionality. This will have a positive impact on the system from several aspects (reusability, flexibility, transparent communication, simplified logic), however, components are still tightly coupled and form a spaghetti achitecture. Consider the following example. When vehicle detects a transit through a toll road, the data flow from OBU Adapter to Monitoring and Billing modules. Both Modules need data from Client Account Administration module. If Monitoring detects something suspicious, it might communicate with external systems and Enforcement. Enforcement might communicate with external systems independently again, and so on. When considering a system that uses CEP, the architecture remains the same, with the only new component being CEP Server, which contains a CEP engine and communication interfaces. Although it is almost the same as the previous architecture, using CEP significantly affects the system behaviour. The data now flow from adapters to CEP Server, where they are processed. The Engine then decides what actions to carry out. For example, if a vehicle passes through a toll road segment, the data are first sent to the engine, which analyzes it and invokes proper services from other modules. The role of the engine is to look at the system as a whole and react to what is happening. The logic is now centralized and simplified, with the only component that needs to know about the others being CEP server. It is obvious that modifying a system with decoupled components and centralized logic is much easier than modifying one with interconnected components and the logic spread across the whole system.
6 UNSPECIFIED FIELD 5.2 Assets of CEP in implementation To demonstrate the assets of CEP in implementation, we will show an example of a complex event definition of the complex event specified in section 4.1. SELECT tvi.vehicle_id AS vehicle_id, tvi.gate_id AS gate_id, tvi.timestamp AS timestamp FROM TVI.win:length(1) AS tvi, TRST.win:time(5 sec) AS trst WHERE tvi.vehicle_id!= trst.vehicle_id OR tvi.segment_id!= trst.segment_id This rules compares each Tollgate Vehicle Identified event with a 5 second window of Toll Road Segment Transit events. The complex event is detected if no TRST event that matches TVI s vehicle ID and segment ID occured during the 5 second window. If we should program this functionality in a common programming language, the service processing data from tollgates would have to launch a timer each time it recieves data about a vehicle passing through a tollgate and call another service to check if the OBU did not send data about passing that segment. This is of course very impractial and decreases flexibility. Our assumptions are that CEP might reduce the time needed for implementation a certain functionality up to 5 times. 6 Conclusion In this paper we discussed the application of complex event processing in reactive distributed system design. One of the reasons why we decided to address this issue is that it is a new and interesting concept, yet it has not expanded to its full potential. We tried to evaluate the assets of CEP from several asspects by implementing a functional model of an electronic toll collection system. Despite the fact that our work is not finished, some results were reached already in the stages of design and implementation. Our next goal is to perform a benchmark test of system s performance and conduct a more in-depth analysis of system s modifiability and flexibility, as well as provide a more formal metrics for evaluating the heftiness of implementation. Acknowledgement: This work was partially supported by the Slovak Research and Development Agency under the contract No. APVV-0233-10. References [1] COVINGTON, R. Event-Driven Architecture and SOA: Complimentary Architectures for the Enterprise. [online] December 2005. Available at http://soa.omg.org/uploaded Docs/SIG Meetings/Dec. 05/05-12-04.pdf [2] CUKIER, K. Data, data everywhere. [online] February 25th, 2010. Available at http://www.economist.com/node/15557443 [3] FRONCKOWIAK, J. SOA Best Practices and Design Patterns: Keys To Successful Service- Oriented Architecture Implementation. [online] March 3rd, 2008. Available at http://www.emacromall.com/techpapers/soa Best Practices and Design Patterns - Key to Successful Service Oriented Architecture Implementation.pdf [4] CEP Vendors. [online] February 2nd, 2010. Available at http://rulecore.com/cepblog/?page id=47 [5] LUCKHAM, D. The Power of Events. 5. act. publication. Addison-Wesley, August 2007. 376 s. ISBN 0201727897. [6] NEAL, L. Complex-Event Processing Poised for Growth. Computer [online] April, 2009, Volume 42, Issue 4, pp 17-21.