Esper. Luca Montanari. MIDLAB. Middleware Laboratory

Transcription

1 Esper Luca Montanari

2 Esper Open Source CEP and ESP engine Available for Java as Esper, for.net as NEsper Developed by Codehaus (write esper complex event processing on Google) Esper enables rapid development of applications that process large volumes of incoming messages or events. Esper filters and analyzes events in various ways, and responds to conditions of interest in real-time

3 Esper How does it work Works as real time engine that triggers actions when event conditions occur among event streams An Event Processing Language allows registering rules (sql-like) in the engine, using Java objects to represent events

4 Reversed DB concept DB# DB# DB# Queries Alert! query satisfied Data

5 Esper Architecture

6 Outline Events representation Esper input Esper logic - Processing (queries and event processing language) Esper Output Performance

7 Events representation How the events are represented inside the engine? Each event is a Plain Old Java Object (POJO) The instance varables of the object are the properties of the event. The object must have the getters. The class specifies the event type (stream).

8 Events representation Example: Through the POJOs, Esper can take the properties of the events by name.

9 Events representation Other representations: Java Map: key-values pairs, can contain objects, further maps, arrays... XML: Document Object Model (DOM) XML: Streaming API for XML (StAX)! Adapters (csv, spring jms template, db output, http...) Multiple inner streams can have different representations - this is transparent to the processing engine

10 Esper Input How the events are given to the engine? Esper sockets - optimized java sockets (very high throughput) Send event inside the engine - Through the reference of the engine, whithin the java application. - useful if you integrate esper in your application

11 Esper logic Almost all CEP systems have a language (or something equivalent) in order to specify the correlation rules No DataBase to interrogate, the data flow through the queries (if SQL extention) (that become long running queries )

12 Esper EPL language Esper comes with their SQL-Like language, called Event Processing Language (EPL) Very similar to SQL: select, where, having, group by and so on, but with some important improvements (later). In EPL the queries are called statements Statements are given to engine in form of string parameters

13 EPL vs SQL Each event belonging to an event stream is like a tuple (a row of the table). The event stream is like the table. The properties of an event are the attributes of the tuple. People Name Surname Age Mario Rossi 24 Gianni Bianchi 56 Roberto Verdi 23 Stream name: People Gianni Bianchi 56 Mario Rossi 24 Roberto Verdi 23 NO ORDERING, it is a set! ORDERING, it is a stream! Time

14 EPL vs SQL People Name Surname Age Mario Rossi 24 Gianni Bianchi 56 Stream name: People Gianni Bianchi 56 Mario Rossi 24 Roberto Verdi 23 Roberto Verdi 23 Time Select Name, Surname from People Where (Age<24) If you don t have to capture the concept of time the EPL and SQL queries are equal! The output will be provided in a different way.

15 EPL improvements Generally the Stream Processing languages are more powerful then SQL cause they can capture the concept of time. EPL can also modify magic numbers at runtime, can execute user defined functions and can capture relations among events. Windows Pattern Variables User Defined Functions

16 A EPL improvements Windows D B A A C A Time

17 EPL improvements Windows batch-space: wait for n events, calculate among n events fixed in size - variable in time - one calculation each n events, among n events. A B A A C D A select... from MyEvent.win:length_batch(3) batch-time: wait for n seconds, calculate among the events arrived in the n seconds fixed in time - variable in size - one calculation per n seconds A B A A C D A select... from MyEvent.win:time_batch(1 sec)

18 EPL improvements Windows sliding-space: wait for the firsts n events and after calculate among the last n events as soon a new event arrives moving, fixed size window - one calculation per event among n events A B A A C D A select... from MyEvent.win:length(3) sliding-time: when a new event arrives, calculate among the events arrived in the last n seconds moving window fixed time, one calculation per event. A B A A C D A select... from MyEvent.win:time(2 sec)

19 EPL improvements Pattern In a given window, pattern captures relations among (at least 2) events: e.g. A => B - an A event after a B event e.g. every A =>B - every A event after a B event A B A A C D A I want to be alerted when an A event is followed by a B event in the last 10 events select * from pattern [(every A -> B).win:length(10)] The window is needed to avoid excessive memory consumption

20 EPL improvements Several cases available, e.g. : Pattern I want to be alerted when an A or a B event arrives in the last 10 events select * from pattern [every (A or B).win:length(10)] I want to be alerted each time an event is between an A event and a B event in the last 10 events select * from select * from pattern [(A until B).win:length(10)] I want to be alerted each time an event with a given property satisfied is followed by an event with another property satisfied in the last 10 seconds pattern [(a.prop1 = true -> a.prop2 = true). win:time(10 sec)]

21 EPL improvements Pattern Pattern can also be used to correlate different streams stream1 A B A A C D A stream2 A A C B D A I want to be alerted each time an event of the stream1 having id 49 is followed by an event of the stream2 having id 49 Select * from pattern [every ( a=stream1(ids = 49) -> b=stream2(idr = 49) )].win:time(30 sec)

22 EPL improvements Variables The EPL queries can be very long running queries. The variables allow to change numerical values at runtime. select symbol, sum(price) from TickEvent group by symbol having sum(price) > var_threshold

23 EPL improvements User Defined Function Within an EPL statements can be executed a user defined (static) Java method and can be used the returned value. select 3 * com.mycompany.myclass.myfunction(price, volume) as myvalue from StockTick.win:time(30 sec)

24 Esper output What happens when a query is satisfied? three possible solutions: Esper triggers a listener or a subscriber bound to the statement; Esper inserts into another stream the result of the statement; both of them.

25 Esper Listeners When you register a new statement (query) to an esper engine you can specify a related listener or subscriber A Simple java object that is instantiated the first time that a query (registered to him) is satisfied and that contains an update method invoked each time the registered queries are satisfied. The attributes of the event are passed via parameters (subscriber) or via an eventbean object (listener)

26 Esper insert into When a statement is satisfied, it returns events or properties that can be used by other statements creating a logical network of streams. The insert into clause acts like a dispatcher separating meaningful events among others. Insert into replies select * from eventstream where type = reply This statement creates a new events stream containing only replies. This Stream can be used by the other statements in the from clause or in the pattern clause

27 Esper insert into eventstream Rq Rp Rq Rp Ack Rq Rp Insert into replies select * from eventstream where type = reply replies Rp Rp Rp Select * from replies where cond

28 Esper insert into Join of streams is supported (also by sql) - multiple inner stream are also supported

29 Esper insert into Insert into replies example select * from eventstream where type = reply Insert into requests select * from eventstream where type = request select (replies.timstamp - request.timestamp) as XXX from replies, requests where requests.requestid = replies.replyid and requests.dest = replies.source

30 Esper performance

31 Esper performance 2 Ghz dual core laptop

32 Conclusion Pro & Cons Very easy to use Open Source, GPL license (download and try it!) Very well supported (API and documentation) Impressive performance! Centralized System (but can be distributed)

33 Conclusion What we are doing with it? Online Failure Prediction in Air and Naval traffic control systems Collaborative Security in financial critical infrastructure

34 thanks