Rule-Based Multi-Query Optimization

Transcription

1 Rule-Based Muli-Query Opimizaion Mingsheng Hong Dep. of Compuer cience Cornell Universiy Johannes Gehrke Dep. of Compuer cience Cornell Universiy Mirek Riedewald Dep. of Compuer cience Cornell Universiy Alan Demers Dep. of Compuer cience Cornell Universiy Chrisoph Koch Dep. of Compuer cience Cornell Universiy ABTRACT Daa sream managemen sysems usually have o process many long-running queries ha are acive a he same ime. Muliple queries can be evaluaed more efficienly ogeher han independenly, because i is ofen possible o share sae and compuaion. Moivaed by his observaion, various Muli-Query Opimizaion (MQO) echniques have been proposed. However, hese approaches suffer from wo limiaions. Firs, hey focus on very specialized workloads. econd, inegraing MQO echniques for CQL-syle sream engines and hose for even paern deecion engines is even harder, as he processing models of hese wo ypes of sream engines are radically differen. In his paper, we propose a rule-based MQO framework. This framework incorporaes a se of new absracions, exending heir counerpars, physical operaors, ransformaion rules, and sreams, in a radiional RDBM or sream processing sysem. Wihin his framework, we can inegrae new and exising MQO echniques hrough he use of ransformaion rules. This allows us o build an expressive and scalable sream sysem. Jus as relaional opimizers are crucial for he success of RDBMes, a powerful muli-query opimizer is needed for daa sream processing. This work lays he foundaion for such a muli-query opimizer, creaing opporuniies for fuure research. We experimenally demonsrae he efficacy of our approach.. INTRODUCTION Query opimizers have been insrumenal for he success of relaional daabase echnology. The cos difference beween a good and a bad query plan can be several orders of magniude. For This maerial is based upon work suppored by he AFOR under Award No. FA , by he U.. Deparmen of Homeland ecuriy under Gran Award Number , and by he Naional cience Foundaion under Gran No Any opinions, findings, and conclusions or recommendaions expressed in his publicaions are hose of he auhors and do no necessarily reflec he views of he sponsors. Permission o copy wihou fee all or par of his maerial is graned provided ha he copies are no made or disribued for direc commercial advanage, he ACM copyrigh noice and he ile of he publicaion and is dae appear, and noice is given ha copying is by permission of he ACM. To copy oherwise, or o republish, o pos on servers or o redisribue o liss, requires a fee and/or special permissions from he publisher, ACM. EDBT 29, March 24 26, 29, ain Peersburg, Russia. Copyrigh 29 ACM /9/3 $5. daa sream sysems he sakes are even higher. Insead of onesho queries in a relaional DBM, a sream sysem is processing many coninuous queries simulaneously. These queries are acive for long periods of ime and hey process massive sreams in real ime. A poor query implemenaion choice can negaively affec sysem performance for he lifeime of he query. The key o achieving good sream processing performance is o opimize muliple queries ogeher, raher han individually. In a sream query workload, i is ofen he case ha muliple concurrenly acive queries can share sae and compuaion. Query evaluaion echniques ha exploi his propery are referred o as Muli- Query Opimizaion (MQO) echniques. The imporance of MQO for sream processing is widely acceped and various sream MQO echniques have been proposed [, 6, 2, 22, 4, 5, 7]. Unforunaely, exising MQO echniques apply only o very specific queries or workload properies. For example, predicae indexing [, 6] is ailored for a se of selecion operaors ha all read he same inpu sream. In addiion, work on MQO echniques so far has happened in parallel for CQL-syle sream engines [2, 5], referred o as Relaional Engines (RE), and even paern deecion engines [8, 2], referred o as Even Engines (EE). The former use an operaor model similar o relaional daabases, while he laer implemen queries wih auomaa. This has led o an unsaisfacory sae of MQO, characerized by a confusing variey of individual echniques ha apply o specific workloads or implemenaion models only. This prevens effecive MQO for complex queries and leads o a siuaion where similar approaches migh be re-invened by he differen communiies for REs and EEs. To address hese problems, we propose a Rule-based MQO framework, called RUMOR. I is inspired by he classical Query Graph Model (QGM) of RDBMes [7], where query opimizaion echniques for single queries can be naurally modeled as ransformaion rules on query plans. RUMOR provides a modular and exensible framework, enabling new opimizaion echniques o be developed and incorporaed incremenally ino he sysem. To suppor rule-based MQO, we have o exend he key absracions ha are used in a radiional RDBM or sream sysem: physical operaors, ransformaion rules, and sreams. We inroduce a small number of carefully designed absracions ha ogeher creae a powerful MQO framework. In fac, RUMOR incorporaes all previously proposed MQO echniques for sream processing. In paricular, i successfully incorporaes MQO echniques from boh relaional sream engines and auomaa-based even processing engines. Hence an addiional benefi of RUMOR is ha i enables he unificaion of hese diverse camps of sream processing sysems. Experimenal resuls for our prooype implemenaion indicae ha we can efficienly process a large number of CQL-syle relaional

2 sream queries, even processing queries, as well as hybrid queries involving query feaures from boh ypes of query workloads. RUMOR lays he foundaion for muli-query opimizers (MQOpimizers) for daa sream processing. I opens up opporuniies for exciing fuure research on finding new rewrie rules and on exending he approach o cos-based MQOpimizers, incorporaing ideas from he classical dynamic programming approach o cos-based single query opimizaion in RDBMes [8]. Conribuions and roadmap. Our conribuions can be summarized as follows. We propose RUMOR, a rule-based MQO framework, which naurally exends he rule-based query opimizaion and query-plan-based processing model used by curren RDBM- es and sream sysems. We show how new and exising MQO echniques for relaional sream engines and for even engines can be inegraed ino RUMOR. This is done by defining a small number of carefully designed absracions. We demonsrae he efficacy of our approach by presening experimenal resuls using a prooype implemenaion of RUMOR. RUMOR inegraes MQO echniques for REs and EEs. For ease of exposiion, in ecion 2 and 3, we inerleave he descripion of RUMOR and inegraion of MQO echniques for REs ino RU- MOR. We hen describe he inegraion of MQO echniques for EEs in ecion 4. The experimenal resuls are presened in ecion 5. Finally, we survey relaed work in ecion 6, and conclude in ecion RUMOR: PART I RUMOR incorporaes hree absracions, respecively exending physical operaors, ransformaion rules, and sreams. For ease of exposiion, in his secion we inroduce only he firs wo absracions (ecions 2.2 and 2.3), and show how hey can be used o express a se of ineresing MQO echniques (ecion 2.4). We describe he las absracion in ecion 3. Due o space consrains, we choose o presen RUMOR in an inuiive way, accompanied by examples. 2. Background We briefly review he relaed conceps in a relaional query processing engine. A logical query is specified by a user hrough a query language such as CQL, which has well-defined semanics. A query opimizer reads a logical query as inpu, and produces a physical query, also known as a query plan, as he resul of opimizaion. The opimizaion process involves he applicaion of ransformaion rules, also known as rewrie rules, on he query plans. A ransformaion rule maps one query plan o anoher semanically equivalen plan (e.g. pushing selecion below join). The query plan produced by he opimizer is execued by he query engine o produce resuls conforming o he logical query semanics. We say he query plan implemens is corresponding logical query. For efficiency we wan he query engine o process muliple queries ogeher. We herefore exend he noion of a query plan o be one ha implemens all he currenly acive logical queries. A query plan is composed of physical operaors, he basic scheduling and execuion unis in he engine. A physical operaor consumes one or muliple inpu sreams, and i produces one oupu sream. A physical operaor is called he consumer operaor of is inpu sreams, and he producer operaor of is oupu sream. This paper focuses on rewrie rules for query plans. 2.2 Physical Muli-Operaor MQO echniques idenify opporuniies for sharing beween operaors, and hey modify pars of he query plan o exploi hese opporuniies. For example, consider a query plan wih muliple selecion operaors reading he same inpu sream. The predicae indexing MQO echnique shares work among hem by indexing he selecion predicaes of he operaors. For each incoming sream uple his index is probed. I reurns all saisfied predicaes a a much lower cos han he naive sraegy of evaluaing each selecion predicae individually one-by-one [, 6]. To model a se of operaors wih shared compuaion, we propose an absracion called physical muli-operaor (or m-op). We say ha an m-op implemens a se of operaors. An m-op is defined as follows. For every sream, is an inpu (resp. oupu) sream of he m-op, if and only if i is an inpu (resp. oupu) sream of a leas one of he operaors he m-op implemens. The semanics of he m-op are defined as follows. Le be an inpu uple arriving in sream. Then he m-op concepually execues all is operaors ha have inpu sream, and i wries he oupu produced for by hese operaors o he corresponding oupu sreams. The sae of he m-op concepually is a vecor; each enry in he vecor is equivalen o he sae of one of he implemened operaors if his operaor was execued in isolaion. Noice ha he definiion of m-op semanics is based on he oneby-one execuion of he implemened operaors wihou sharing sae. This defines he correc semanics, bu of course our goal is o find more efficien m-op implemenaions ha sill guaranee he same inpu-oupu behavior as defined by he above semanics. Inuiively, he m-op consumes he se of inpu sreams of he physical operaors i implemens, and i produces a corresponding se of oupu sreams. The noion of consumer and producer operaors for physical operaors exends naurally o m-ops. The m-op absracion generalizes he radiional physical operaor absracion. I herefore akes he place of a physical operaor in RUMOR: A query plan is composed of m-ops, and an m-op is he new scheduling and execuion uni in he query engine. We illusrae he use of m-ops in he following example. EXAMPLE. Figure (a) shows wo queries Q and Q 2, where σ and σ 2 are selecion operaors, and α denoes a sliding window aggregaion operaor, occurring in boh queries. Noe ha we use he query name o denoe is oupu sream name. Le σ {,2} denoe he m-op implemening σ and σ 2 wih predicae indexing. I produces wo oupu sreams, respecively corresponding o he oupu sreams of σ and σ 2 in Figure (a). Figure (b) shows he query plan using σ {,2}. uppose uple in sream saisfies boh σ and σ 2. In Figure (a), an oupu uple is produced by boh σ and σ 2. In Figure (b), an oupu uple is produced by σ {,2} on each of is wo oupu sreams. 2.3 Transformaion Rules on m-ops We now exend he radiional ransformaion rules, which operae on query plans composed of physical operaors, o muliquery ransformaion rules, or m-rules for shor. M-rules operae on query plans composed of m-ops. imilar o a radiional ransformaion rule, an m-rule consiss of a pair of condiion and acion funcions [7]. The condiion funcion is a Boolean side-effec-free funcion on he query plan o idenify opporuniies for sharing. Once a sharing opporuniy is idenified among a se of operaors in a query plan, he acion funcion modifies he query plan by replacing ha se of operaors wih a single m-op. We say he m-rule maps a se of m-ops o a single m-op, or i merges hese operaors.

3 Q Q 2 α σ (a) Queries α σ 2 Inpu Q Q 2 α α σ {,2} (b) Plan wih elecion m-op Q Q 2 α {,} σ {,2} (c) Plan Channel,[,2] wih Figure : Query Plans in RUMOR (Red Recangles Represen ream Tuples; he Blue Recangle is a Channel Tuple) More precisely, he condiion of an m-rule is a funcion from he powerse of he se of all possible m-ops o {rue, false}. For a given se of m-ops, he rule can only be applied if he condiion funcion evaluaes o rue. The acion of an m-rule is anoher funcion. This funcion maps a se of m-ops (for which he condiion funcion evaluaed o rue) o a single m-op, referred o as he arge m-op, which implemens he inpu m-ops more efficienly wih an MQO echnique. In he query plan, we simply replace all edges ha previously conneced oher operaors wih he o-be merged operaors by edges o he corresponding inpu and oupu sreams of he arge m-op. 2.4 Expressing MQO Techniques wih m-ops and m-rules Mos of he exising specialized MQO echniques share work among operaors reading he same sream(s). These can be implemened in RUMOR hrough m-ops and m-rules. For example, we can model predicae indexing for equaliy predicaes on a single aribue as an m-rule as follows. The condiion of he m-rule evaluaes o rue only for a se of selecion operaors ha all read he same sream and whose selecion predicae is an equaliy predicae on he same aribue A. The acion rule hen replaces hem wih a arge m-op ha uses a hash index on aribue A for a more efficien evaluaion of he selecion predicaes of hese operaors. I is no hard o see ha all hese previously proposed MQO approaches for muliple selecion [, 6], aggregaion [22], and join [2] operaors can be expressed similarly hrough corresponding m-ops and m-rules. The firs hree rows in Table summarize hese rules. Noice ha in daa sream processing sysems, join and aggregaion operaors usually conain window specificaions o preven unbounded memory consumpion. Also, aggregaion operaors may conain opional group-by specificaions. For each operaor ype τ, we name he corresponding m-rule s τ, indicaing ha i is an m-rule for insances of operaor τ ha all process he same inpu sream(s). The remaining rows in Table will be discussed laer in his paper. The curren se of m-rules is no inended o be complee he exensible naure of rule-based query opimizaion allows for adding new rules. 3. RUMOR: PART II In ecion 2, we have shown how o use he wo absracions m-op and m-rule o express a se of exising MQO echniques, including predicae indexing [, 6], muliple aggregae processing wih differen group-by specificaions [22], and shared join evaluaion [2]. All of hese echniques aemp o share work among similar operaors reading idenical sream(s). A complemenary MQO approach is o suppor sharing also in he case where similar sreams are processed by idenical operaors. Consider he example in Figure (a). The same aggregaion operaor α occurs in boh queries, and i aggregaes some subse of he uples from. However, if σ and σ 2 have differen selecion predicaes, hen m-op σ {,2} will have wo differen oupu sreams, as shown in Figure (b). This in urn implies ha he wo insances of α read differen inpu sreams and herefore canno be combined using m-rule s τ. On he oher hand, he selecion predicaes migh be similar so ha many uples ha pass σ migh also pass σ 2. In ha case i would be beneficial o avoid duplicaion of sream uples as well as duplicaion of work for he aggregaion. This example migh appear like a rare corner case, bu sreams wih common uples occur frequenly in pracice as a resul of muliple operaors processing he same inpu sream. Recen work on MQO echniques like precision sharing join [4] and shared fragmen aggregaion [5] have shown ha exploiing hese sharing opporuniies can resul in significan performance improvemen. In his secion, we propose an absracion o model such MQO echniques in RUMOR. We refer o i as a channel. Channels generalize and replace sreams in RUMOR (ecion 3.). We hen describe how o decide which sreams should be replaced wih channels in he query plans in RUMOR (ecion 3.2). Finally, we add a new se of m-rules and m-ops leveraging channels, which express boh exising and new MQO echniques (ecion 3.3). 3. Exending reams o Channels Logically, a channel is equivalen o he union of a se of sreams, The sreams ha are combined o form a channel are required o have compaible schemas. This can always be achieved by padding he schemas of individual sreams wih he aribues from oher sreams afer appropriae aribue renaming. Unlike a union of sreams, a channel keeps rack of which original sream a uple belongs o. We say he channel encodes hese sreams. More formally, a channel encodes a se of daa sreams wih union-compaible schemas as follows. The channel is defined as he union of is sreams, bu each sream uple has an addiional aribue called membership componen. The membership componen specifies he se of sreams o which his uple belongs. For efficiency, he membership componen is implemened by a bi vecor. Through he use of a channel we can share work in wo ways. Firs, when idenical uples from differen sreams are encoded as a single channel uple, heir space is shared. econd, when muliple sreams are encoded ino he same channel, he compuaion of heir consumer operaors may be shared. Clearly, channels generalize sreams. In RUMOR, hey ake he place of sreams as he inpu and oupu of m-ops. For each m-op, he inpu (resp. oupu) channels ogeher pariion he se of inpu (resp. oupu) sreams of his m-op. When an m-op o processes an inpu channel uple, a decoding and an encoding sep are involved as follows. o firs deermines o which se of inpu sreams belongs, so ha i concepually only evaluaes hose physical operaors implemened by o ha ake his uple as inpu. This is he decoding sep. imilarly, when o is abou o produce a se of oupu uples, i needs o encode i ino a se of channel uples wih he appropriae sream membership componen, and hen wrie hem o he appropriae oupu channels. This is he encoding sep. Noe ha he decoding and encoding seps can ofen be implemened very efficienly, or migh acually no be necessary a all. For example, consider an m-op π {,,n} implemening n projecions wih he same projecion specificaion, bu wih differen inpu

4 m-rule name e of inpu operaors o which he m-rule is applicable Targe m-op s σ A se of selecion operaors which read he same sream Predicae indexing [, 6] s α A se of aggregaion operaors which read he same sream, wih he same aggregae funcion bu poenially differen group-by specificaions hared aggregae evaluaion [22] s A se of join operaors which read he same wo sreams, wih he same join predicae bu poenially differen window lenghs hared join evaluaion [2] c α A se of aggregaion operaors reading sharable sreams, wih he same definiion hared fragmen aggregaion [5] c A se of join operaors which read sharable sreams, wih he same definiion Precision sharing join [4] s ; (or s µ) A se of; (or µ) operaors reading he same wo sreams, wih he same definiion Common ubexpression Eliminaion (ecion 4.3) c ; (or c µ) A se of; (or µ) operaors which a) have he same definiion b) read sharable inpu sreams for he firs inpu sream parameer, where hese inpu sreams are produced by he same m-op c) read he same inpu sream for he second inpu sream parameer Channel Based MQO (ecion 4.4) Table : Represenaive m-rules o Express Exising and New MQO Techniques sreams hrough n. uppose hese n inpu sreams are encoded by channel C, and he n oupu sreams are encoded by channel D. In his case, for each inpu channel uple from C, π {,,n} needs o perform projecion only once and o produce only one oupu channel uple in D, keeping he membership componen of inac in he oupu D uple. To coninue Example, we can use a channel o encode he wo oupu sreams of σ {,2} in Figure (b), resuling in he query plan shown in Figure (c). Here he dashed arrow represens he channel, and α {,} represens he aggregaion m-op, implemened by he shared fragmen aggregaion echnique described in [5]. uppose an inpu uple from sream saisfies boh predicaes in σ {,2}. σ {,2} hen produces a single oupu channel uple, represened by he blue recangle in Figure (c). Tha channel uple has he same conen as he inpu uple, bu is associaed wih a membership componen denoed as [,2], indicaing ha i belongs o boh oupu sreams of σ {,2}. Noe ha ideas similar o channels were used for specific MQO algorihms for joins and aggregaes in relaional engines [4, 5]. Our conribuion is o propose he addiion of he channel concep o an MQO framework as a general absracion for sharing work. As we will show in ecion 4, he combinaion of m-ops, m-rules, and channels also leads o powerful new MQO echniques for even processing queries. 3.2 Mapping reams o Channels Channels are a powerful mechanism ha allows us o aggressively share work among operaors ha read even differen sreams. Given a se of sreams, how do we decide which ones o map o he same channel? The following radeoffs have o be aken ino accoun. Firs, if wo sreams i and j are encoded ino he same channel, hen sream uples wih he same conen can share sorage by being represened as he same channel uple. econd, if he consumer operaors of i and j have he same definiion, he evaluaion on channel uples will be more efficien han evaluaing uples from sream i and j separaely. Third, mapping muliple sreams o he same channel creaes overhead. Time-wise, wih muliple sreams being mapped o he same channel, he consumer m-op of his channel now has o process he membership componen of each inpu uple. pace-wise, each channel uple has o carry he membership componen. Based on hese radeoffs, i is clear ha sreams should only be mapped o he same channel if here is a large enough fracion of channel uples ha belong o muliple sreams and if he sreams are consumed by idenical operaors. We now propose a simple lighweigh heurisic for deciding which sreams o map o he same channel. This heurisic was used in our experimenal evaluaion and works very well in pracice. More sophisicaed cos models can be developed, bu are lef for fuure work. Our proposed algorihm for deciding which sreams o merge ino a single channel is based on he concep of sharable sreams. Two sreams and 2 are sharable, denoed 2, if he following holds: Base case. If = 2, hen 2. Base case 2. If and 2 are produced by wo sream sources ha are labeled o be sharable, hen 2. Oupu of unary ops. For any unary operaors o, o 2, if := o (T ), 2 := o 2(T 2), o = o 2, and T T 2, hen 2. Oupu of binary ops. For any binary operaors o, o 2, if := o (T, U ), 2 := o 2(T 2, U 2), o = o 2, T T 2, and U U 2, hen 2. pecial case for selecion. For a selecion operaor ha reads T and produces, T. ymmery, 2 : 2 2 Transiiviy, 2, 3 : ( 2 2 3) 3 Inuiively, sreams are sharable if hey are he resul of he same query plans, modulo any selecion operaors anywhere in he plan, applied o he same inpu sreams. Clearly, is an equivalence relaion and i generalizes he sream ideniy relaion =. This makes very efficien o compue and sore. Even if sreams are sharable, we map hem o he same channel only if hey originae from he same m-op. If hey are produced by differen m-ops, he runime sysem would have o synchronize hese operaors o ensure idenical uples are available a he same ime for he channel encoding sep. This is concepually no hard, bu an analysis of he radeoffs is beyond he scope of his paper. Furhermore, if sharable sreams are consumed by m-ops ha canno share any work, here is no benefi in encoding hem wih he same channel. Typically wo m-ops reading differen sreams can effecively share work only if hey have exacly he same definiion. For example, wo selecion operaors wih he same predicae, wo projecion operaors wih he same projecion specificaion, or wo aggregaion operaors wih he same aggregae funcion and group-by specificaion can share work when reading wo differen inpu sreams ha are sharable. We only consider his ype of work sharing wih channels in his paper.

5 Tradiional absracion RUMOR absracion physical operaor m-op (ecion 2.2) ransformaion rule m-rule (ecion 2.3) sream channel (ecion 3) 2 Operaors of ype τ 2 o X o2 Operaors of ype τ Table 2: Correspondence beween new and exising absracions for building a sream sysem Inpu sreams ha are similar Inpu sreams ha are similar To conclude, given a se of sreams hrough n, we map hem o he same channel, only if (a) he i s belong o he same equivalence class defined by, (b) he i s are produced by he same m-op, and (c) he consumers of he i s have he same definiion. These crieria, referred o as channel-based MQO sharing crieria, are currenly used in RUMOR. When hese crieria are me, we map he sreams o a single channel and hen combine he (idenical) consumers of he i s ino he same m-op, achieving effecive work sharing among hem. The above sharing crieria may appear resricive, bu are me surprisingly ofen in pracice. E.g., hey apply when queries conain paramerized componens ha differ in some selecion predicaes bu oherwise follow he same query emplae. For example, precision sharing join [4] and shared fragmen aggregaion [5] are boh implicily based on he above crieria, addiionally limied o join operaors and aggregaion operaors, respecively. 3.3 Expressing MQO Techniques wih Channels To benefi from channels, we add he following new m-rules. For each operaor ype τ (e.g. selecion, join, aggregaion), we add an m-rule which idenifies operaors of ype τ whose inpu sreams saisfy he channel-based MQO sharing crieria defined a he end of ecion 3.2. I hen maps hese operaors o a single m-op. We refer o his m-rule as c τ, indicaing ha his is an m-rule for operaors τ processing uples from he same channel. For example, he fourh and fifh m-rule in Table respecively express shared fragmen aggregaion [5] and precision sharing join [4]. Noe he ineresing dualiy beween he wo m-rules s τ (ecion 2.4) and c τ of an operaor ype τ. s τ is applicable o a se of sharable operaors (i.e., operaors of ype τ) reading he same sream(s), whereas s τ is applicable o a se of operaors of he same definiion, reading sharable sream(s). Assuming τ is unary, we presen an illusraion for he difference beween s τ and c τ. In Figure 2, he enclosing recangle denoes he se of unary operaors of ype τ, reading sharable sreams. Each row labeled i corresponds o a subse of operaors of ype τ, reading he same sream i. Each applicaion of s τ will pick a row of operaors here, and map hem o an m-op. Repeaed applicaions of his m-rule herefore form a pariion of his se of operaors. The seing of Figure 3 is similar o ha of Figure 2. Each column corresponds o a se of operaors of ype τ wih he same definiion, reading a se of sharable inpu sreams, 2,. One applicaion of c τ selecs a column of operaors, and maps hem o an m-op. Repeaed applicaions of c τ herefore also form a pariion of his se of operaors. As a resul, for any operaor in he shaded region X (i.e., any operaor wih definiion o, reading sream ), boh s τ and c τ are applicable o i. Therefore, as in many oher rule-based applicaions, differen orderings of m-rule applicaions may resul in differen opimized query plans. To summarize, Table 2 shows he newly proposed absracions in RUMOR, and heir correspondences wih exising absracions. Figure 2: R Applied o a e of Operaors of Type τ Figure 3: R 2 Applied o a e of Operaors of Type τ 4. INTEGRATING MQO TECHNIQUE FOR EVENT ENGINE In ecions 2 and 3, we have presened RUMOR, as well as how he MQO echniques for REs can be inegraed ino RUMOR. In his secion, we describe how he MQO echniques for EEs can be inegraed as well. This is a more challenging ask, as EEs are ofen based on auomaa, insead of query plans composed of relaional operaors. On he oher hand, if we are able o inegrae he MQO echniques for boh REs and EEs ino RUMOR, we will be able o build an expressive and scalable sream sysem unifying REs and EEs, of which here are obvious and significan benefis (ecion 4.). The challenge is o inegrae EE MQO echniques wihou cluering RUMOR wih numerous new absracions or complex special rules for even queries. Insead, he inegraion should resul in a clean simple opimizaion framework where only new operaors and new rules need o be added o exend is funcionaliy. Our soluion consiss of wo pars. Firs, we ranslae auomaa ino query plans (ecion 4.2), second, we express he MQO echniques designed for auomaa in RUMOR (ecion 4.3). To illusrae he benefi of inegraing RE and EE MQO echniques ino RUMOR, we show new channel-based MQO echniques for EEs ha combine he channel concep wih radiional EE MQO approaches (ecion 4.4). 4. A Moivaing cenario for Unifying REs and EEs The separaion of sream processing sysems ino REs and EEs has led o parallel developmens of MQO echniques ha are ailored o hese sysems. Due o he lack of a common MQO framework, similar ideas have o be re-invened and opimizaion opporuniies are missed because some echniques exis in only one world. Also, queries ha require funcionaliy from boh REs and EEs are no effecively opimized by eiher sysem. Consider he following scenario. In performance monioring of compuer sysems [2, 9], each sream corresponds o readings of a paricular performance couner, such as he amoun of curren CPU consumpion of a paricular hread or process. Users can regiser coninuous queries in a sream sysem; e.g. o compue he average CPU load in a ime-based sliding window, or o raise alers on specified condiions and opionally o perform cerain acions, such as erminaing resource hogging processes. The following simple example illusraes performance monioring workloads. Inpu sreams. We assume he following inpu sream schema: CPU(pid, load; s), indicaing he CPU load of each process in he sysem. pid denoes process ID; load denoes CPU load; s denoes he required imesamp aribue for each sream. In pracice here are more performance couners han jus CPU,

6 Queries. Query workloads for sysem monioring ofen have he following wo characerisics. Firs, here may exis a large number of concurren queries in he sysem, since differen queries may be regisered o monior he behavior of differen processes. Furhermore, for a paricular process, differen monioring condiions may be posed in differen queries. To obain high hroughpu, i is crucial o apply MQO echniques o hese queries. econd, some performance monioring queries demand funcionaliy from boh CQL-syle queries suppored by a relaional engine and paern maching queries suppored by an even engine. We refer o such queries as hybrid queries. Consider he following hybrid query, which deecs processes ha are ramping up in CPU consumpion. This query combines he funcionaliy of sliding window aggregaes (which have received a lo of aenion in work on REs) for smoohing he incoming performance couner readings, and he funcionaliy of even paern deecion (suppored by an even engine) for finding a monoonically increasing sequence in CPU load consumpion. QUERY. For a paricular process p, smooh he CPU load value by replacing he curren CPU load for p wih an average load of p over he las 5 seconds. Call he smoohed sream MOOTHED. Nex, find in MOOTHED an even paern composed of a sequence of monoonically increasing CPU loads on p, where his sequence paern saisfies a cusomizable saring condiion θ s, e.g., θ s = CP U.load < 2, and a fixed sopping condiion, say CP U.load > 9. To efficienly process such a query, one has o combine opimizaion echniques from REs (for he sliding window aggregae) and EEs (for he monoonic sequence). Having a common framework like RUMOR grealy simplifies his process, especially when i comes o he even more challenging problem of processing a large number of such hybrid queries: QUERY 2. We have a se of queries {Q,, Q n}, where each Q i differs from Query only in he saring condiion θ s. Noe ha for his comparably simple example workload, i is possible o manually consruc query plans ha achieve good compuaion sharing. However, he focus of his paper is o auomae MQO wih RUMOR so ha more complex workloads can be opimized as well. In he remainder of ecion 4, we will revisi his query workload and describe how auomaed MQO is achieved in RUMOR. This is however predicaed on he undersanding of how he MQO echniques for EEs are inegraed ino RUMOR, which we discuss nex. 4.2 Translaing Auomaa o Query Plans Even Engines are ofen based on auomaa [4,, 7, 2]. In order o inegrae he MQO echniques for EEs ino RUMOR, our firs sep is o model he auomaa used in EEs as query plans in RU- MOR. ince he Cayuga sysem [7] is he EE represenaive wih he by far sronges emphasis on MQO, we chose i as he example o show how o express auomaa as query plans in RUMOR. I is possible o inegrae oher even engines, such as AE [2], ino he RUMOR framework in a similar manner. A a high level, he ranslaion of auomaa o query plans is based on he idea ha auomaon saes can be mapped o operaors while auomaon edges correspond o sreams where uples flow from one operaor o he nex. However, he challenges are such as for memory and disk. The sreams schemas also involve more aribues. We simplify he scenario here for ease of presenaion. Θ f q Θ r,f r Θ fo,f fo Acive insances a sae q: Insance Insance2 Insance3 Insance4 Figure 4: ingle ae q of Cayuga Auomaon in he deails: Auomaon edges have predicaes and here is nondeerminism. Also, our main goal is o inegrae EE funcionaliy ino RUMOR wihou significanly increasing RUMOR s complexiy. Ideally, one should only have o add new specific m-ops and m-rules o suppor EE syle MQO inside RUMOR. The following discussion shows how his can be achieved for Cayuga auomaa. Our general approach can be exended o oher auomaon-based sysems, including he recenly proposed NFA b [] auomaa. The basic building block of Cayuga auomaa are saes like he one shown in Figure 4. A sae has an associaed inpu sream and i mainains a se of acive auomaon insances. These insances correspond o parially processed queries ha have advanced o his sae. Each insance has he same fixed schema; is values record relevan daa from previously mached evens. A sae has hree ougoing edge ypes: a single filer edge (op loop edge), one or more forward edges (horizonal ougoing edge), and a single rebind edge (boom loop edge). aes can only be conneced hrough forward edges, resuling in auomaa ha are direced acyclic graphs. Concepually, whenever a new even arrives on sream, he Cayuga engine checks for each insance a sae q if any of he edge predicaes is saisfied. These predicaes can reference aribues of boh he incoming even a well as he insance. Insances for which no edge predicae is saisfied are deleed. All ohers non-deerminisically raverse all edges whose predicaes are saisfied. Non-deerminism is implemened by duplicaing he insances and leing each copy raverse he corresponding edge. When an insance raverses he filer edge, i remains a sae q unchanged. When i raverses he rebind edge, he auomaon execues formula F r on he concaenaion of he insance and he incoming even. I hen sores he modified insance a sae q. For forward edges, he insance is also modified based on formula F fo and he incoming even, bu i is sen o he corresponding nex sae. F r and F fo are schema map funcions. A schema map funcion can rename and projec aribues, as well as inroducing new aribues via simple arihmeic compuaion or user-defined funcions. I is similar o a QL projecion operaor (which implemens he QL ELECT clause). Deails can be found in [7, 8]. A complee Cayuga auomaon wih saes q, q 2, and q 3 is shown in Figure 5(a). aes q and q 3 are sar and final sae, respecively. The sar sae has only forward edges, while he final sae has no ougoing edge. For sae q 2 he rebind edge is omied, which is equivalen o having a rebind edge wih θ r = false. Ideally we would like o map all edge predicaes θ o selecion operaors and all schema map funcions F o he corresponding combinaion of projecion, renaming, and arihmeic manipulaions. Unforunaely, he semanics of filer and rebind edges makes i necessary o inroduce wo special m-ops ino RUMOR. Given an auomaon sae wih a filer edge bu no rebind edge (θ r = false), ha sae will be ranslaed ino an m-op denoed as;. Is semanics is he same as is counerpar in he Cayuga algebra [7]. Inuiively, ; is a sequence operaor concaenaing wo inpu evens. imilarly, 2 3 A B A D

7 Oupu ream π F2 Q Q Q i Q n Q Q i Q n σ θ2 Θ f ; θf σ e σ e σ ei σ en σ {e,, en} Θ, F Θ 2, F 2 2 π F σ θ 2 μ μ μ i μ n μ{,,n} q q 2 q 3 (a) (b) σ s σ {s,, sn} σ {s,, sn} Figure 5: (a) An Example Cayuga Auomaon, and (b) he Equivalen Query Plan α α α an auomaon sae wih a filer and a rebind edge is ranslaed ino an m-op denoed as µ, whose semanics is also he same as is counerpar in he Cayuga algebra. µ is an ieraive version of ;, capable of concaenaing an unbounded number of inpu evens ino an even sequence paern. The formal definiions of ; and µ can be found in [7]. Forward edges, however, can be mapped o a selecion operaor followed by a schema mapping operaor as expeced. We have developed a formal mapping from Cayuga auomaa o RUMOR query plans. For simpliciy, we illusrae his process hrough an example. For he Cayuga auomaon in Figure 5(a), we sar wih he inpu sream, read by q. Predicae θ on he forward edge of q is ranslaed o σ θ in he query plan. imilarly, he schema map funcion F on he same edge is ranslaed o π F in he query plan, reading he oupu sream of σ θ. 2 Nex, we ranslae sae q 2 ino a binary operaor ; θf, reading he oupu sream of π F as well as 2. Finally, he forward edge from q 2 o q 3 is ranslaed in a similar way as he forward edge from q o q 2. We use σ θ2 and π F2 respecively o implemen he predicae θ 2 and he schema map funcion F 2 on ha forward edge. The oupu sream of π F2 is equivalen o he oupu sream of he auomaon. This finishes he ranslaion. The resuling query plan is shown in Figure 5(b). The ranslaion of a Cayuga auomaon involving saes wih rebind edges is similar. For example, if sae q 2 in Figure 5(a) also had a rebind edge wih predicae θ r, hen he operaor; θf in Figure 5(b) would be replaced wih µ θf,θ r. EXAMPLE 2. The RUMOR query plan for Query in ecion 4. is shown in Figure 6(a). For clariy, we omi projecion operaors and he parameers of some operaors in he query plan. The inpu sream is denoed as. α denoes he sliding window aggregae operaor for smoohing he CPU load readings of each process. σ s and σ e are respecively he saring and sopping condiions. µ builds up he even sequence paern consising of monoonically increasing values in he CPU loads of a paricular process. Finally, as in Example, we use he query name Q o denoe is oupu sream name. 4.3 Expressing Auomaa Based MQO Techniques in RUMOR In ecion 4.2, we achieved he unificaion of an RE and an EE on a single auomaon level, by ranslaing an auomaon ino a query plan. This was done by adding only wo operaors o RUMOR. 2 Here π denoes he more expressive QL projecion operaor (E- LECT clause), as opposed o he projecion operaor in relaional algebra. (a) Query Plan for Query (b) Query Plan for n Insances of Query 2 Wihou Channel (c) Query Plan for n Insances of Query 2 Wih Channel Figure 6: RUMOR Query Plans for he Moivaing Queries in ecion 4. (Omiing Projecion Operaors for Clariy) To efficienly process a large number of even paern queries, an EE ofen employs a se of MQO echniques specially designed for auomaa. To make RUMOR pracical for even processing applicaions, we have o express hese echniques as m-rules. We again use Cayuga as a represenaive EE which adops MQO, and express all is MQO echniques by m-rules and m-ops in RUMOR. New MQO echniques for EEs could be inegraed similarly. Given ha we have inroduced wo new operaors ; and µ ino RUMOR in ecion 4.2, nex we add a new m-rule for each of hese wo operaors. The m-rule s ; for ; (resp. s µ for µ) maps a se of ; operaors (resp. µ operaors) o an m-op, if hey read he same pairs of sreams, and have he same definiion. These wo m-rules are shown in he second-o-las row in Table. There are wo major caegories of MQO echniques in Cayuga, sae merging and indexing. We show how hese echniques can be expressed by m-rules. ae merging. The firs ype of sae merging in Cayuga is prefix sae merging. Inuiively, given an exising auomaon F, and a new inpu auomaon A, A can be merged ino F by idenifying he longes prefixes of F and A ha are idenical, and share he wo prefixes in he merged auomaon. As a concree example, he exising auomaon and he inpu auomaon o merge are shown respecively in Figure 7(a) and 7(b). In his example, suppose inducively ha sae P and P have been merged, and sae Q and Q read he same sream (in his case i is 2), hen we can merge saes Q and Q. The resuling auomaon is shown in Figure 7(c). This prefix sae merging echnique is expressed by he m-rules s ; and s µ ogeher. We illusrae his wih he above example. The query plans corresponding o he exising auomaon and he inpu auomaon are shown respecively in Figure 8(a) and 8(b). Noe ha he operaors; θf in Figure 8(a) and in Figure 8(b) respecively implemen saes Q and Q in he corresponding auomaa. uppose inducively ha he common sub-expressions below operaor; θf in Figure 8(a) and in Figure 8(b) have been merged. The m-rule s ; corresponding o;is now applicable o; θf in boh figures,

8 Θ f 2 Θ, F Θ 2, F 2 P Q (a) Curren NFA Fores before Merge Θ f Θ, F Θ 2', F 2' 2 P Q (b) The Query Auomaon o Merge Θ, F P Θ f 2 Q Θ 2, F 2 Θ 2', F 2' (c) Curren NFA Fores afer Merge Figure 7: Cayuga Auomaa ae Merging Process ; P π F σ θ π F2 σ θ2 ; θf 2 (a) Curren Query Plan before Merge π F ; P σ θ π F2' σ θ2' ; θf 2 (b) The Inpu Query Plan o Merge ; P π F2 π F σ θ σ θ2 ; θf π F2' σ θ2' 2 (c) Curren Query Plan afer Merge Figure 8: RUMOR Query Plans for Cayuga Auomaa since by assumpion hey read he same pair of sreams, and have he same definiion. Hence, hey are mapped by he m-rule s ; o he same m-op, which we sill denoe as; θf, since i has he same definiion as he wo inpu operaors ha are merged. The resuling query plan is shown in Figure 8(c). The prefix sae merging performed on muliple µ operaors can be done in a similar way. Noe ha we have ranslaed he prefix sae merging MQO echnique on auomaa ino he well-known MQO echnique on query plans: Common ubexpression Eliminaion (CE). This is a good example for how a common opimizaion framework can help avoid duplicae work (in his case he developmen of new specialized auomaon sae merging echniques). In addiion, in RUMOR, we have more opporuniies for inlining, illusraed as follows. An inpu Cayuga query ha is no lefassociaive, such as ;( 2; 3), has o be implemened by wo Cayuga auomaa A and B, where A implemens 2; 3, producing an inermediae sream, and B implemens ;. This is referred o as resubscripion in [7], and in his case A canno be inlined ino B. However, his query can be implemened by a single query plan, which effecively inlines he query plan corresponding o A o ha corresponding o B, providing addiional MQO opporuniies. Auomaon indexing. There are hree ypes of indices in Cayuga. Below we describe how o express he Forward-Rebind (FR) Index echnique in RUMOR. The oher wo indices, Acive Node Index and Acive Insance Index [7, 8], are handled similarly. We omi heir descripion due o space consrains. FR Index is a per-sae index on some of he predicaes of forward/rebind edges of is associaed sae. For example, in Figure 7(c), he predicaes θ 2 and θ 2 associaed wih he forward edges going ou of sae Q can be managed by an FR Index. For an incoming even e from sream 2 which does no saisfy he filer edge predicae θ f associaed wih sae Q, e will be used o probe his FR index o obain he se of saisfied predicaes associaed wih he forward edges (i.e., a subse of {θ 2, θ 2}). An FR Index on sae q can be expressed by he m-rule s σ in RUMOR as follows. Le he operaor corresponding o q in he ranslaed RUMOR query plan be o (o is of ype; or µ). For hose selecion operaors ha are consumers of he oupu sream of o, we apply he m-rule s σ o map hem o he same m-op. Tha m-op effecively implemens he FR Index for he ranslaed query plan. For example, recall ha for he auomaon shown in Figure 7(c), is corresponding query plan in RUMOR is shown in Figure 8(c). For he FR Index on he forward edges of sae Q in Figure 7(c), which we described above, i can be expressed by applying s σ o σ θ2 and σ θ 2 above he; θf operaor in Figure 8(c). To conclude, we have shown ha wih RUMOR, all he MQO echniques employed by Cayuga can be expressed eleganly as m- rules on RUMOR query plans. Hence, asympoically, he evaluaion efficiency of a se of even paern queries in RUMOR is a leas as good as ha in he Cayuga engine, as is confirmed by our experimens. EXAMPLE 3. The RUMOR query plan for n query insances of Query 2 in ecion 4., denoed as Q hrough Q n, is shown in Figure 6(b). The aggregaion operaor α is shared by all n queries. I produces a single sream called MOOTHED in Query, and muliplexes i o all is consumer operaors. The n saring condiions are implemened by he m-op σ {s,,s n}, which produces n oupu sreams corresponding o ha of σ s hrough σ sn. µ i builds he even sequence paern for query Q i. I reads he wo sreams produced by σ si and α respecively. Is oupu sream is consumed by σ ei, he sopping condiion for Q i. Noe ha in Example 3, even hough he µ i operaors have he same definiion, hey canno share work, since heir lef inpu sreams are differen. The same observaion holds for he σ ei operaors. This is a limiaion for RUMOR wihou channels, and is also he case for Cayuga auomaa. We will show in he nex subsecion how o use channels o overcome his limiaion. 4.4 Query Plans wih Channels In he previous subsecion, we have discussed how o express all Cayuga MQO echniques as m-rules and m-ops in RUMOR. In his subsecion, we demonsrae one of he major benefis of inegraing MQO echniques from RE and EE ino a single framework. More precisely, we show ha, somewha surprisingly, here are even paern queries ha can be evaluaed more efficienly in he form of

9 RUMOR query plans han in he Cayuga engine. This is due o new MQO opporuniies wih channels, illusraed hrough he following example. EXAMPLE 4. Le us revisi Query 2 from ecion 4., and consider how o process n insances of his query more efficienly han he query plan shown in Figure 6(b). The sliding window aggregaion par of hese queries for smoohing he inpu sream is already shared. For he paern maching par, a good evaluaion sraegy is o firs evaluae he saring condiions in he n queries. If any subse of hem is saisfied, we remember his informaion and coninue o mach he monoonic sequence paerns of hese queries, implemened by he µ operaors. When he sopping condiion is saisfied, we hen use he informaion we remembered for which θ si s are saisfied o produce resul uples for he righ se of queries. The RUMOR query plan implemening his evaluaion sraegy is shown in Figure 6(c). As in Figure (c), we use dashed arrows o represen channels. However, noe ha his evaluaion sraegy is ouside of he Cayuga auomaa model, and herefore canno be used by he Cayuga engine. In order o produce he desired query plan wih channels shown in Figure 6(c), we add one m-rule for; and µ each. The m-rule for ;, denoed as c ;, maps a se of; operaors o a single m-op, if hese operaors saisfy (a) hey have he same definiion, (b) hey read sharable inpu sreams for he firs inpu sream parameer, where hese inpu sreams are produced by he same m-op, and (c) hey read he same inpu sream for he second inpu sream parameer. In his case, we encode he firs inpu sreams of hese operaors wih a channel. The new m-rule for µ works in a similar way. These wo m-rules are shown in he las row in Table. The sream shareabiliy compuaion and he channel-based MQO sharing crieria defined in ecion 3 are exended accordingly for; and µ. We now show how o use he m-rules o opimize n insances of Query 2, denoed as Q hrough Q n. aring from he query plan in Figure 6(b), we firs apply he m-rule s σ o he se of saring condiions in Q,..., Q n, and encode heir oupu sreams wih a channel C. Nex, we apply he m-rule c µ o he se of µ operaors in he n queries, and again encode heir oupu sreams wih a channel D. Finally, we apply he m-rule c σ o he se of sopping condiions in Q,..., Q n, resuling in a selecion m-op ha reads channel D as inpu, and produces n oupu sreams for he n queries. 5. PERFORMANCE EVALUATION We have implemened in Java a prooype sream engine based on RUMOR, which is capable of processing RE queries, EE queries, as well as hybrid queries. In his secion we repor he performance of our engine in evaluaing he opimized query plans. The experimens are conduced on a machine wih Inel Penium D 2.8 GHz processor and 2 GB main memory, running un Java Hospo erver VM.6.2 on Windows Visa. To leverage he JVM jus-in-ime code opimizaion, for each experimen, we firs process he inpu sream for a few ieraions, before we sar o measure hroughpu. To reduce experimenal variance, we perform each experimen for en imes, and repor he average hroughpus we measured. 5. eup We firs use a synheic benchmark o measure he performance of our sysem for processing even paern queries and hybrid queries. We do no measure he performance for RE queries, because RUMOR is query-plan based like REs. RUMOR herefore can use he same query plans as hese sysems, which have been well sudied [2, 5]. Variable Defaul Value Number of queries Number of aribues in sream schemas Consan domain size Window lengh domain size Zipfian parameer.5 Table 3: Parameers (defaul values) The sream schema we choose consiss of ineger aribues, denoed as a[],, a[9], and (ineger) imesamp aribue. We generae wo sreams conforming o his schema, denoed as and T, as follows. The generaed sream uples have consecuive imesamps, saring from. For each uple, we se is ineger aribues o ineger values from o 999 chosen uniformly a random. We inerleave he generaion of uples for and T. Tha is, uples wih imesamps, 2, belong o, and uples wih imesamps, 3, belong o T. For each experimen, we generae a oal of a leas uples, and feed he uples from and T in heir imesamp ordering. We use he following common parameers o generae query loads. For each randomly generaed query, we choose a window lengh for i from o, where is he defaul domain size for generaing window lenghs. Each window lengh is chosen wih a Zipfian disribuion, favoring larger windows (i.e., a window of lengh is mos likely o be chosen). The defaul Zipfian parameer value is.5. The Zipfian disribuion is o model commonaliy among queries ha is ofen observed in real, large-scale workloads. The parameers are summarized in Table Even Paern Queries In ecion 4, we have chosen Cayuga as a represenaive even engine, and shown how o express is auomaa queries and MQO echniques in RUMOR. In his subsecion, we compare he performance of our sysem based on RUMOR wih Cayuga. Due o he significan differences in he archiecure and implemenaion plaform of boh sysems, a comparison of heir absolue performance is no meaningful. Insead, we follow he experimenal approach used in AE [2], and repor normalized hroughpu obained as follows: as he query processing load changes from ligh o heavy in each experimen, we use he hroughpu for he lighes workload o normalize oher measuremens. This approach allows us o observe and compare he performance rends of boh sysems when we vary he values of experimenal variables, indicaing sysem scalabiliy. Workload. In he firs query workload, we generae a se of queries of emplae σ θ () ; θ2 θ 3 T, where ; is he Cayuga sequence operaor. θ is of form a[] = c, where c is chosen a random beween and 999 wih he same Zipfian disribuion as for window lenghs. imilarly, θ 3 is of he same form and generaed in he same way, bu i is evaluaed on each T uple, whereas θ is evaluaed on each uple. θ 2 is a duraion predicae in Cayuga erminology i expresses he window lengh of his query. Noe ha his query workload benefis from he AN index and FR index in Cayuga, which we described in ecion 4.3. In paricular, he θ s of he se of queries we generae can be indexed by an FR index, while he θ 3 s can be indexed by an AN index. We firs vary he number of queries. Figure 9(a) shows ha by expressing AN indexes and FR indexes wih m-rules in RUMOR, our sysem scales very well. Noe ha even if he predicaes θ and θ 3 on each query are quie selecive, his is no a rivial query workload wih K queries in he sysem, he inpu sream of