High-Performance Event Processing Bridging the Gap between Low Latency and High Throughput Bernhard Seeger University of Marburg

Transcription

1 High-Performance Event Processing Bridging the Gap between Low Latency and High Throughput Bernhard Seeger University of Marburg common work with Nikolaus Glombiewski, Michael Körber, Marc Seidemann

2 1. Motivation reactive monitoring of timecritical business processes predictions about the near future and recommendations for action 2 Bernhard Seeger

3 Situations of Interest Impact Root Cause Event Benefit Opportunity Reaction Costs Options E-2 E-1 E E+1 E+2 E+3 E+4 E+7 E+8 E+9 E+10 Time 3 Bernhard Seeger

4 Many application domains Algorithmic trading Logistics Traffic management Internet of Things System Monitoring & Security 4

5 5 Monitoring IT infrastructures

6 6 Event-based Security within a VM

7 Agenda Review of CEP Architecture Event Store Pattern Matching Conclusions 7 Bernhard Seeger

8 2. A Critical Review of CEP The history of CEP Charles Forgy Inventor of the RETE-algorithms (1981) David Luckham Rapide Project The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems (published 2002) Jennifer Widom Stream Project (2002) 8 Bernhard Seeger

9 Basic Ideas Event Sources Continuous Production of Events Continuous Processing of EPAs Events are flowing through a network of EPAs Event Sinks 9

10 Functionality of EPAs Basic set of of operators Filter Select applications that throw an error message Sliding Window Aggregation Compute number of all running applications within the last minute Window-based Correlator Correlate applications with servers they are running on within the last 10 seconds. Window-based Pattern Matching Detect faulty and anormal application state transitions User-defined operators 10

11 Many CEP-systems available SQL-based systems MS StreamInsight, Esper Tech, Siddhi, Systems with special-purpose language Tibco, Apama, Plain distributed stream systems Twitter Storm, Spark Streaming, Flink, 11 but no agreed semantics

12 Problems and Issues Performance High Throughput vs. Low Latency Scalability Event Store Persistent Management of Events Information extraction from historical events Functionality Support for application time Powerful pattern matching 12

13 The Performance Issue of CEP Esper low latency? Spark Streaming low throughput high throughput high latency Spark 13

14 The Persistence Issue of CEP CEP systems are designed for in-memory processing only. Volatile Data and Persistent Queries Applications require a persistent management of events. Extremely high input rates (millions of events/s) Time-based queries on massive databases 14 Are standard DBMS or NoSql systems the right tools?

15 The Functionality Issue Pattern Matching is the Core Operator for Event Processing Detect a sharp increase in temperature together with sufficiently large amount of smoke within a short period of time. Despite its importance Pattern Matching requires domain knowledge. User-defined implementation vs. General-purpose operator offered by the system 15

16 Summary Problems and Issues in current systems optimized either for low latency or high throughput persistence is still a big issue and often delegated to Apache Kafka very weak or no support for pattern matching 16

17 3. Our Architecture Basic Ideas Combination of an event store and a CEP-engine Similar to the Lambda-architecture Both components run under a unified interface (JEPC) It allows to exchange specific technologies (your most preferred CEP engine, you most preferred store) JEPC acts as a federation platform A continuous query can run in parallel on multiple target platforms. 17

18 Our Architecture C++, Groovy, Realtime Reports (e.g. Grafana) WebSockets Java SQL-like query language JEPC ChronicleDB Bridge Bridge Bridge Native CEP-system JDBC Esper 18 throughput layer H2 PostgreSQL Flink low-latency layer

19 Important Concepts EPAs (aka continuous queries) Queries come with latency constraints Visualization dashboard: 1 min. Security: 1 s Alarm as fast as possible Assignment of queries based on the latency constraint Low-latency layer High-throughput layer 19

20 Our Architecture CEP-only C++, Groovy, WebSockets Realtime Reports (e.g. Grafana) Java SQL-like query language JEPC 20 throughput layer low-latency layer

21 Our Architecture DBS only C++, Groovy, WebSockets Realtime Reports (e.g. Grafana) Java SQL-like query language JEPC 21 throughput layer low-latency layer

22 Our Architecture ideal C++, Groovy, WebSockets Realtime Reports (e.g. Grafana) Java SQL-like query language JEPC 22 throughput layer low-latency layer

23 Necessary Requirements for the Throughput Layer Time to update the database < latency constraint of query Time to process the query on the database < latency constraint of query 23

24 4. ChronicleDB 24 Our Database system for the management of historical events to achieve high throughput. Properties Optimized for fast writes Utilization sequential write performance of magnetic disks Compression Queries Efficient support of temporal predicates Analyze events within a range of four hours Temporal aggregates Number of ssh logins last Tuesday Fast garbage collection of outdated events Bernhard Seeger

25 Architecture of ChronicleDB SQL-like query language Command Line Interface REST-API Java TCP-based protocol ChronicleDB Compression Secondary Indexes (LSM, COLA, ) PAX-Layout Aggregate Temporal B- tree

26 The Gist of ChronicleDB External Memory Main memory optionally secondary indexes Append-only B-tree Event Queues CEP external event streams t 1 t 2 time 26

27 Append-Only B-tree The entire tree is sequentially written in one stream. kept on your favorite technology: UNIX-fs,HDFS,Ceph Record in a leaf consists of Timestamp List of attribute values Index entry in an internal node consists of Temporal routing information Aggregates of the non-indexed attributes min, max, top-k, sum, 27

28 Compression Column Layout (PAX) within a page. A multidimensional time series is split into multiple one-dimensional time series within each page. Compression of one-dimensional time series using a standard algorithm LZ4. LZ4 is very fast in decompression. 28

29 Experimental Results Limited to a central system Maximum disk speed 187 MB/s Measures Events per second Event streams 29

30 Comparison (write performance) Our results for ElasticSearch: events/s to load DEBS 30

31 What is possible? gross data rates 31

32 32 Comparison (Read Performance)

33 33 Performance of Temporal Aggregation vs. Temporal Scans

34 34 Recovery Time of ChronicleDB

35 Summary ChronicleDB provides large performance improvements over other systems. Inserion rate, recovery time search performance ChronicleDB either runs on (parallel) file system HDFS Scalability of ChronicleDB using one of the popular distributed frameworks 35

36 5. Pattern Matching Pattern: Sequence of conditions A B + C Matching: Search for pattern in event stream Stream e 1 e 2 e 3 e 4 e 5 e 6 e 7 e 8 e 9 36

37 Pattern Matching Pattern: Sequence of conditions A B + C Matching: Search for pattern in event stream Stream e 1 e 2 e 3 e 4 e 5 e 6 e 7 e 8 e 9 Match! 37

38 Example Query FROM Sensors s DEFINE AS s.temperature > 60 DO prev = s.temperature AS s.temperature > prev DO prev = s.temperature AS s.smoke = true PATTERN WITHIN 60 seconds RETURN ALERT 38

39 Event Processing Implementation E.g. via NFA (Nondeterministic Finite Automaton) S 0 S 1 S 2 S 3 39

40 Event Store Implementation Index incoming data Attributes involved in conditions E.g. A : s.temperature Determine most selective sub-pattern Leverage index to restrict search space 40

41 6. Conclusions 41 Lambda-like architecture for event processing Low latency & High throughput Due to the smart indexing capabilities in ChronicleDB Performance of ChronicleDB Superior to competitive systems Full Support of Pattern Matching Event Processing Engine & ChronicleDB Available under open source Bernhard Seeger

42 Thanks JEPC is common work with Bastian Hoßbach and Marc Seidemann Dieter Gawlick for our great discussions Our team: Nikolaus Glombiewski, Michael Körber, Andreas Morgen, Franz Ritter BMBF for funding ACCEPT 42 Bernhard Seeger