Partition and Compose: Parallel Complex Event Processing. Martin Hirzel, IBM Research Tuesday, 17 July 2012 DEBS

Transcription

1 Partition and Compose: Parallel Complex Event Processing Martin Hirzel, IBM Research Tuesday, 17 July 2012 DEBS 1

2 ? CEP = Stream Processing? Event (Stream) Processing Aggregate Enrich Join Filter Parse Complex Event Processing Use pattern over simple events to detect and report composite events CEP as an in a streaming language? 2

3 Background: SPL IBM Streams Processing Language SPL is the language for InfoSphere Streams (IBM Product) This paper is based on System S = research branch of InfoSphere Streams 3

4 Scenario: Financial analysis Series of rising peaks and troughs Deep drop below start of match M-shape (double-top) stock pattern Source: 4

5 M-Shape pattern in SPL Composite events Simple events Regular expression Key Aggregation Operator only, no extensions to SPL syntax 5

6 Regular expressions Pattern language familiar from string matching 6

7 Aggregations Operator-specific intrinsic functions 7

8 Matching semantics Standard regular expression semantics Non-greedy (right-minimal) Partition-isolated (Partition-)Contiguous Non-overlapping (submit longest: left-maximal) 8

9 Implementation overview MatchRegex invocation Upstream instance param, output Simple events MatchRegex generator MatchRegex instance Automaton Composite events At compile-time At runtime Downstream instance All C++ s in SPL are code generators 9

10 Automaton. rise+ drop+ rise+ drop* deep Update and filter partial match Create new partial match rise 2 rise rise 0. 1 drop 3 drop rise 4 deep drop 5 6 deep drop Report completed match and flush NFA (non-deterministic finite automaton) 10

11 Partitioning :SimpleEvent :PartitionMap ts symbol price size seqnum key 0..* :PartialMatch state aggr rise 2 rise rise 0. 1 drop 3 drop rise drop 4 5 drop deep deep 6 11

12 Generated C++ code Incremental aggregation rise 2 rise rise 0. 1 drop 3 drop rise drop 4 5 drop deep deep 6 12

13 Paralleli- zation Schneider et al. [PACT 12] Up-stream Simple events :SimpleEvent symbol Composite MatchRegex events PartitionMap key Down-stream Parallelize Simple events MatchRegex (replica 0) PartitionMap Composite events Up-stream key for hash-split :SimpleEvent symbol key for partition map MatchRegex (replica 1) PartitionMap MatchRegex (replica 2) PartitionMap Down-stream 13

14 Safety and determinism SPL compiler checks Syntax and names in expressions Expression and function types MatchRegex checks Syntax and names in regular expression pattern Starting predicate aggregation-free Auto-parallelizer checks Partitioning Absence of stateful expressions Sequence numbers and pulses Enables simple output validation with diff 14

15 Data sets and benchmarks 15

16 Absolute throughput in events per second Large speedup when low sequential throughput 16

17 Speedups 1 Machine x 8 Cores 4 Machines x 8 Cores = 32 Motivates elasticity and auto-width controller 17

18 Related work 2000 today Engine / language Complex events Parallelism NiagaraCQ / XML-QL Algebraic No SQL-TS Back-tracking No Amit Back-tracking No NFA b / SASE Automaton No MATCH_RECOGNIZE ANSI proposal No EventScript Automaton No Cayuga / CEL Automaton Yes, by hand EventJava Index data structures Yes, per task [Woods,Teubner VLDB] Automaton Yes, on FPGA This paper Automaton Yes, partitioned 18

19 Conclusions CEP as an SPL Use CEP for pattern matching Use other s for filtering, enrichment, parsing, joining, etc. Up to 830K events/second Incremental aggregation C++ code generation Parallelism (up to 14x speedup) 19

20 Backup 20

21 Shuffle in twitter02 and twitter03 ParseTweet (replica 0) MatchRegex (replica 0) Source ParseTweet (replica 1) MatchRegex (replica 1) Down-stream Raw tweets as XML documents ParseTweet (replica 2) Tweets as simple events MatchRegex (replica 2) Composite events 21