Utilizing IDs to Accelerate Incremental View Maintenance

Transcription

1 Utilizing IDs to Acceleate Incemental View Maintenance Yannis Katsis UC San Diego Kian Win Ong UC San Diego Yannis Papakonstantino UC San Diego Kevin Keliang Zhao UC San Diego ABSTRACT Pio Incemental View Maintenance (IVM algoithms specify the view tples that need to be modified by compting diff sets, which we call tple-based diffs since a diff set contains one diff tple fo each to-be-modified view tple. idivm assmes the base tables have keys and pefoms IVM by compting ID-based diff sets that compactly identify the to-be-modified tples thogh thei IDs. This wok makes the following contibtions: (a An ID-based IVM system fo a lage sbset of SQL that incldes the algebaic opeatos selection, join, goping and aggegation, genealized pojection involving fnctions, antisemijoin (and theefoe negation/diffeence and nion. The system is based on a modla appoach, allowing one to extend the sppoted langage simply by adding one algebaic opeato at-a-time, along with eqations descibing how ID-based changes ae popagated thogh the opeato. (b An efficient algoithm that ceates an IVM plan fo a given view in fo passes that ae polynomial in the size of the view expession. (c A fomal analysis compaing the ID-based IVM algoithm to pio IVM appoaches and analytically showing when one otpefoms the othe. (d An expeimental compaison of the ID-based IVM algoithm to pio IVM algoithms showing the speioity of the fome in common se cases. Categoies and Sbject Desciptos H.2 [Database Management]: Systems Keywods mateialized views; incemental view maintenance 1. INTRODUCTION Mateialized views ae widely sed to speed p qey evalation by stoing the eslts of commonly asked qeies. Being mateialized, these views have to be boght p to date when the ndelying data change. This is typically done thogh Incemental View Maintenance (IVM. Abstacting ot the details of diffeent IVM Sppoted by NSF III and NSF III , PI d by Pof. Papakonstantino, who has commecial inteest in this eseach. Also sppoted by NSF III Pemission to make digital o had copies of all o pat of this wok fo pesonal o classoom se is ganted withot fee povided that copies ae not made o distibted fo pofit o commecial advantage and that copies bea this notice and the fll citation on the fist page. Copyights fo components of this wok owned by othes than ACM mst be honoed. Abstacting with cedit is pemitted. To copy othewise, o epblish, to post on seves o to edistibte to lists, eqies pio specific pemission and/o a fee. Reqest pemissions fom pemissions@acm.og. SIGMOD 15, May 31 Jne 4, 2015, Melbone, Victoia, Astalia. Copyight is held by the owne/atho(s. Pblication ights licensed to ACM. ACM /15/05...$ devices(did, categoy pats(pid, pice devices_pats(did, pid (a Database schema CREATE VIEW V AS SELECT did, pid, pice FROM pats NATURAL JOIN devices_pats NATURAL JOIN devices WHERE categoy = "phone" (b View definition Fige 1: Database schema and view fo nning example appoaches, a typical IVM algoithm takes as inpt thee diff tables D + R, D R and D R pe base elation R, containing the tples that wee inseted, deleted and pdated in R and comptes the coesponding diff tables D + V, D V and D V fo the view V, containing the changes that have to be pefomed on V to bing it p to date. In pio IVM wok, each diff table D + V, D V and D V contains one diff tple fo each view tple that has to be inseted, deleted and pdated, espectively. This is why we efe to sch diffs as tple-based diffs (in shot t-diffs. In this wok we show that if the base tables contain keys, one can epesent the view modifications in a mch moe compact way thogh a novel type of diffs, called ID-based diffs (in shot i-diffs, which identify the to-be-modified view tples thogh thei IDs. In contast to t-diff tples, a single i- diff tple can epesent modifications to mltiple view tples. This diffeence is ccial, as i-diffs ae moe efficient to compte than t-diffs, eqiing in geneal fewe base table accesses as we will explain next. This leads to moe efficient ID-based IVM algoithms, nde common assmptions. The following example demonstates the diffeence between t-diffs and i-diffs. To diffeentiate between the two types of diffs, in the est of the pape we will be sing the standad symbol to efe to the newly intodced i-diffs and the symbol D to efe to taditional t-diffs. EXAMPLE 1.1. Conside the database of an electonic device manfacte, stoing a list of devices and thei pats. Fige 1a shows the espective database schema, consisting of the elations devices, pats and devices_pats. The key attibtes of each elation ae shown ndelined. Also conside the view V of Fige 1b etning the list of pats fo devices of type phone. Fige 2 shows an example of tple-based and ID-based incemental maintenance of V. The initial database and view instance ae shown on the left and t-diffs and i-diffs fo a sample change in elation pats on the ight in Figes 2a and 2b, espectively. Conside the action of pdating the pice of pat P1 fom $10 to $11. This modification is epesented thogh identical t-diff and i- diff tples, as seen in the t-diff table Dpats and i-diff table pats, espectively. Howeve, this is no longe te when we look at the diffs that epesent the eslting pdates that have to happen to the view. Note that the change of the P1 pice in pats has to be popagated as pdates of all P1 tples in the view (which in this case ae the fist two tples. While the t-diff table DV descibes these changes by two diff tples, each descibing an entie view t-

2 Ini/al#Database#Instance#DB# devices' devices_pats' pats' pats D'''''''''#! Δ'''''''''''''! pats did# categoy# did# pid# D1' phone' D1' P1' D2' phone' D2' P1' D3' tablet' D1' P2' Ini/al#View#Instance#V(DB# pid# pice# P1' 10' P2' 20' pid# pice_# old# pice_# new# P1' 10' 11' SELECT did, pid, pice_old, pice_new Q D FROM D pats INNER JOIN devices_pats INNER JOIN devices pid# pice_# old# pice_# new# P1' 10' 11' #' Q Δ SELECT * FROM Δ pats V' did# pid# pice# D1' P1' 10' D2' P1' 10' D1' P2' 20' D''''#####! V did# pid# pice_# old# pice_# new# D1' P1' 10' 11' D2' P1' 10' 11' Δ'''''''''! V pid# pice_# old# pice_# new# P1' 10' 11' Initial Instances (a Tple-based diffs (b ID-based diffs Fige 2: Example of tple-based and ID-based IVM ple that has to be pdated, the i-diff table V descibes the same modifications thogh a single diff tple (which intitively instcts pdating all P1 tples in the view. Obviosly ID-based diffs ae moe compact than thei tplebased contepats. Moe impotantly thogh i-diffs ae in geneal moe efficient to compte than t-diffs. Intitively, the pefomance gains come fom the fact that in contast to t-diffs, i-diffs do not need to eceate the entie view tples to be modified and ths can avoid some base table accesses. We se this obsevation to design an ID-based IVM algoithm, which is shown both analytically and expeimentally to pefom in most cases fewe base-table and view accesses than pio tple-based appoaches. EXAMPLE 1.2. Qeies Q D and Q show how the diffs fo the view can be compted fom the base tables and the diff fo base elation pats. While compting the t-diff eqies joining D pats with the base tables devices_pats and devices (see Q D to find all devices containing pat P1, podcing the i-diffs simply amonts to finding the modified pats (and not the devices in which they ae contained and can theefoe be accomplished by accessing only pats and avoiding all joins with the base elations (see Q. Note that the fewe base table accesses of i-diff comptations ae not, jst by themselves, an absolte poof of speio pefomance of the i-diffs, as maintaining the view shold also cont the cost of applying the i-diffs to the view. While both the ID-based and tplebased appoaches will have to join the eslting view diffs with the view, the ID-based appoach has the dawback of potentially tying to join moe diff tples by ceating dmmy i-diff tples, i.e., i-diff tples descibing changes fo tples that do not even exist in the view. Fo example, assme that pats inclded a tple (P3, 20 and pats inclded a change of P3 s pice. Then V wold inclde a dmmy P3 tple, i.e., the system wold pay the pice of attempting to pdate the P3 s in V, albeit thee wold be no P3 in V. We call this effect oveestimation. Nevetheless, o theoetical and expeimental analysis show that nde common cicmstances the i-diff appoach is indeed speio. Fthemoe, we show that the advantage of ID-based IVM gows as the qeies become moe complex. Contibtions. This pape makes the following contibtions: (a An ID-based IVM system, called idivm, applicable when the base elations have pimay keys. The idivm is based on a modla, algebaic appoach, allowing one to extend the sppoted view definition langage simply by adding one elational algeba opeato at-a-time and poviding i-diff popagation eqations descibing how ID-based changes ae popagated thogh it. (b A set of i-diff popagation eqations fo a lage sbset of SQL (denoted by Q SP JADU that incldes the algebaic opeatos select, poject, join, goping and aggegation with associative fnctions, genealized pojection involving fnctions, antisemijoin 1 and nion. Althogh the famewok can be easily extended to moe expessive view definition langages as descibed above, o analytical and expeimental eslts focs on Q SP JADU, as it is expessive enogh to cove a lage nmbe of pactical se cases. (c An efficient 4-pass algoithm that ceates an IVM plan fo a given algebaically-expessed view and a given set of modification types in fo passes that ae polynomial in the size of the view expession: The fist pass comptes the IDs of intemediate eslts. The second pass instantiates the opeato IVM eqations to the specifics of the view s opeatos. The thid pass composes individal eqations into the qeies of the IVM plan. Finally, the foth pass applies minimization and othe optimizations paticla to the IVM poblem. Unlike geneal ppose minimization the consideed minimization is polynomial. (d An algoithm that given a view expession decides what types of i-diffs shold be mined fom the modification log o capted fom tigges. The poblem is non-tivial since, as we will see, the nmbe of types of i-diffs that ae applicable, given a base schema and a view schema, is exponential in the size of the schemas. The pesented algoithm ses the view definition to decide the mch smalle nmbe of sfficient and efficient i-diffs. (e A fomal analysis poving that fo Q SP JADU views in many se cases the ID-based IVM with the i-diff popagation eqations descibed in the pape is moe efficient than tple-based IVMs. The analysis is based on a fine-gained cost model conting data accesses and incldes a discssion nde the specific conditions nde which tple-based IVMs can pefom bette. (f An expeimental evalation of the poposed IVM system fo Q SP JADU views indicating that in most cases it significantly otpefoms taditional tple-based appoaches. The expeimental eslts show speedps of 2 to moe than 50 ove tple-based IVMs. Note that ID-based IVM optimization is othogonal and can be combined with many of the othe IVM isses stdied in the liteate [12, 8], sch as mateialized view selection [3, 25, 19], self maintenance [5, 11] and compilation into code [2]. We biefly descibe these pio IVM woks and thei synegies in Section 8. Otline. The pape is stcted as follows: Section 2 defines ID-based diffs (i-diffs. Section 3 pesents the achitecte of idivm. It consists of two main pats: The fist tansfoms base table mod- 1 Theefoe capting qeies with negation. The diffeence opeato is a special case of antisemijoin.

3 ifications to i-diffs and the second, given a set of i-diffs, ceates a DML scipt fo maintaining the view. Fo ease of exposition, we pesent them in the evese ode, i.e. Section 4 descibes the algoithm fo the DML scipt geneation and Section 5 descibes the tansfomation of changes to i-diffs. Sections 6 and 7 compae analytically and expeimentally the efficiency of the geneated scipt to those podced by tple-based IVM appoaches. Finally, Sections 8 and 9 discss elated wok and conclde the pape, espectively. 2. ID-BASED DIFFS Fo the following discssion we conside a elational database DB whose base tables contain keys and a elational view V (Ī, Ā ove DB, containing a set of key attibtes (which we will efe to as IDs Ī and a set of non-key attibtes Ā. EXAMPLE 2.1. The view V of o nning example contains IDs Ī = {did, pid} and non-id attibtes Ā = {pice}. In the following we will be sing the initial instance of V of Fige 2. View definition langage. Althogh, as we will see, the famewok can be easily extended to moe expessive view definition langages, nless othewise stated, we conside views fom the langage Q SP JADU, which contains all SQL qeies that can be fomlated sing the algebaic opeatos Selection, Pojection (involving fnctions, Join (with abitay join conditions, Aggegation with associative fnctions sm, avg and cont, Antisemijoin (and ths Diffeence, and Union 2. Geneal Stcte of an i-diff. Let V (Ī, Ā be a view with IDs Ī and non-id attibtes Ā. An ID-based diff (in shot i-diff of type t {+,, } fo elation V is in its most geneal fom a elation t V (Ī, Ā pe, A post satisfying the following popeties: It contains a sbset Ī of the view s IDs Ī. These ae sed to identify the tples to be modified. It may contain two sets Ā pe and A post of attibtes, sch that Ā, A ae sets of non-id attibtes of V. An attibte A pe and A post intitively stoes the pe-state vale (i.e., initial vale befoe the change and espectively post-state vale (i.e., new vale afte the change of attibte A of V. Depending on thei type, i-diffs may not contain both pe-state and post-state attibtes. In paticla, inset i-diffs (i.e, of type t = +, do not contain pe-state attibtes, since they epesent insetions of tples that did not exist befoe. Similaly, delete i- diffs (i.e., of type t = do not contain post-state attibtes. Only pdate diffs (i.e., of type t = may contain both old and new attibte vales. We next descibe the semantics fo each i-diff type: Update i-diff. An pdate i-diff instance V fo view V (Ī, Ā is a elation instance with schema V (Ī, Ā pe, A post, whee Ī is a sbset of the IDs Ī of V and Ā, Ā ae potentially diffeent sbsets of the non-id attibtes Ā of V (with Ā being potentially the empty set. Intitively, each tple (ī, ā pe, ā post in V specifies that all tples in V with vales ī fo thei Ī attibtes shold have the vales of thei Ā attibtes pdated to ā post. Fomally, applying V on an instance I V of V is eqivalent to applying the following DML statement on I V : FROM V WHERE V.Ī = V.Ī 3 In the est of the pape, the instance I V will be implied fom the context and theefoe fo simplification we will simply efe to a diff as being applied on the view V. Note, that althogh not affecting its semantics, an pdate i-diff may also contain pe-state vales of some non-id attibtes of V. As we will see late, this additional infomation is leveaged by the IVM algoithm to edce the nmbe of accesses to the database. EXAMPLE 2.2. Applying the following pdate i-diff V pid pice pe pice post P leads to the pdate of the pice of both tples in V with pid = P1" fom 10 to 11. Remak. In the following we conside only i-diffs whee Ī foms a pimay key of the i-diff. The eason is that if Ī is not a key, then pdate i-diffs ae not well-defined and inset i-diff applications may lead to pimay key violations. Inset i-diff. An inset i-diff instance fo a view V (Ī, Ā is a elation instance with schema (Ī, Āpost, o in othe wods a elation containing the post-state vales fo all attibtes of the view and no pe-state vales. Intitively, an inset i-diff instance contains a set of tples that shold be inseted into V. Fomally, applying + has the same effect as applying the following DML statement on V : APPLY : INSERT INTO V SELECT Ī, Ā post AS Ā FROM + V WHERE ROW(Ī, Ā post NOT IN (SELECT Ī, Ā FROM V EXAMPLE 2.3. Applying the following inset i-diff did pid pice post D3 P2 20 D4 P3 30 insets tples <D3, P2, 20> and <D4, P3, 30> in V. Remak. The WHERE clase in the above DML statement enses that an attempt is made to inset a tple into V only if it is does not aleady exist in V in the exact same fom. This allows mltiple inset i-diffs to ty to inset the same tple. Delete i-diff. A delete i-diff instance fo a elation V (Ī, Ā is a elation instance with schema (Ī, Ā pe, whee Ī is a sbset of the IDs Ī of V and Ā is a potentially empty sbset of the non-ids Ā of V. Intitively, specifies the tples that shold be deleted fom V based on the vales of the Ī attibtes. Fomally, applying has the same effect as applying the following DML statement on V : APPLY : DELETE FROM V WHERE ROW(Ī IN (SELECT Ī FROM APPLY V : UPDATE V SET Ā = Ā post Note that, similaly to pdate i-diffs, a delete i-diff may also specify the pe-state vales of the deleted tples, which ae sed to ceate moe efficient IVM soltions. 2 To maintain the IDs fo the bag nion, we employ a special nion all opeato, otptting a special attibte b, denoting which child banch (b = 0/1 fo left and ight, esp. a tple came fom. 3 Note that fo conciseness the UPDATE statement is witten sing PostgeSQL s special UPDATE FROM syntax. Howeve, it cold be eqivalently witten sing standad SQL syntax.

4 EXAMPLE 2.4. Applying the following delete i-diff pid pice pe P1 10 leads to the deletion of both tples with pid = P1 fom V. Effective i-diff instances. Given a set of i-diff instances fo a elation V, applying them on V leads in geneal to diffeent eslts depending on the ode of application. Howeve, in this wok we only look at sets of i-diffs whee any ode of applying them on V yields the same eslt. To this end, we define the notion of effective i-diff instances. Given the pe-state V pe and post-state V post of a elation V, an i-diff instance t V is said to be effective w..t. V pe and V post if fo each vale of a tple of V it eflects its final vale. Fomally, it is effective iff it satisfies the following popeties: If t V is an inset i-diff: Evey tple inseted by the i-diff exists in the post-state (i.e., V post. If t V is a delete i-diff ove schema (Ī, Ā pe: Evey tple deleted by the i-diff does not exist in the post-state elational instance (i.e., π Ī V πī V post =. post: Ev- If t V is an pdate i-diff ove schema V (Ī, Ā pe, Ā ey tple pdated by V that exists in the post-state instance, has all pdated attibtes Ā post set to the coesponding vales in that instance (i.e., π Ī,Ā post V V post Ī π Ī,Ā V post. It can be shown that a set of effective i-diffs lead to the same eslt egadless of the ode in which they ae applied. In the following the i-diff instances we conside ae assmed to be effective. We will discss in Sections 4 and 5 how idivm makes se that it always opeates on effective i-diff instances. i-diff schemas. It shold be obvios that a single modification cold be epesented thogh i-diffs of diffeent schemas. In paticla, one can inclde pe-state o post-state vales fo diffeent sets of attibtes. Moe impotantly, diffeent base table i-diffs may lead to IVM soltions of diffeent efficiencies. Fo instance, an pdate of a t tple of elation R(Ī, A1, A2 on attibte A1 can be epesented by eithe an i-diff that contains only the post-state of A 1, o both the post-state of A 1 and A 2 (even thogh the vale of A 2 did not change. Howeve, the fist i-diff will in geneal lead to a moe efficient soltion, since fo the second i-diff the IVM algoithm will have to accont also fo the change of A 2, althogh this is not needed. This geneates a novel challenge of selecting which base tables i-diff schemas to ceate, as explained next. IDs and fnctional dependencies. As discssed ealie, the set Ī of ID attibtes of a view V foms a key of that view. Moeove, any i-diff V fo the view V identifies the tples of V to be pdated, deleted o inseted thogh a sbset of that key. Howeve, this cannot be an abitay sbset of the key. The key Ī of a view is split into components (Ī1, Ī2,... I n, sch that fo each component of the key thee is a fnctional dependency Īi Āi fom this component Īi to a sbset Āi of non-key attibtes of the view. The i-diff can identify the tples of the view to be modified thogh a sbset of these key components. EXAMPLE 2.5. Fo instance, in o nning example, V has ID/key Ī = {did, pid}, which can be decomposed into two components Ī1 = did and Ī2 = pid, since thee is the fnctional dependency pid pice (in a moe geneal view V containing also attibtes of the devices elation thee wold also be a fnctional dependency fom did to those attibtes. Ths the tples of V can be identified by an i-diff eithe thogh did o thogh pid (as is the case in o example. Modification Logge Base table i-diff schema geneato Δ-scipt geneato View definition Modification log Base table i-diff schema epositoy Base table instances Δ-scipt epositoy View instance Base table i-diff instance geneato Δ-scipt execto Cache instances Data modification time View definition time View maintenance time Fige 3: idivm achitecte 3. SYSTEM ARCHITECTURE Fige 3 depicts the achitecte of idivm; an ID-based IVM system based on i-diffs and bilt on top of a elational DBMS. The modles of the system ae shown as onded boxes, while the system s data stctes ae depicted as white ectangles. idivm can be setp to maintain the view eithe eagely (i.e., wheneve the base data change, known as eage IVM [6, 7, 10] o lazily at some late point in time (known as defeed IVM [9, 15, 17]. In eithe case, the modification logging modle of the idivm emains the same. Fthemoe, the only pat of the achitecte that is sbstantially diffeent in the two appoaches is the i-diff popagation les and cache maintenance les (see Fige 4. This pape descibes the defeed IVM les. idivm contains modles exected at thee diffeent times (shown thogh colo-coding in Fige 3: (a when the views ae defined (oange, (b wheneve the data in the ndelying database change (geen, and (c wheneve the views ae maintained (ble. We pesent next biefly each of these stages: View definition time. The most inteesting and novel comptations happen when a view is added to the system. At that point idivm pecomptes in the fom of DML scipts how to tanslate i-diffs on the base tables to view pdates. This comptation happens thogh the synegy of two components: Fist, it employs a base table i-diff schema geneato to decide which i-diff schemas to geneate fo the base tables. As we have discssed in Section 2, this is a non-tivial poblem, as the same pdate cold be modelled thogh i-diffs of diffeent schemas. Once the base table i-diff schemas have been decided, idivm invokes the -scipt geneato ceating a DML scipt that accesses the geneated i-diffs, the base tables and the potential caches (which as we will see can be sed to speedp the IVM to maintain the view. The eslting -scipt is stoed in a epositoy to be sed at view maintenance time. Data modification & view maintenance time. Given this offline comptation, the system s online component is simple: When the base data ae modified, a modification logge logs these changes fo late se. When the time comes to maintain the view, the base table i-diff instance geneato conslts the modification log and convets it to instances of the base table i-diff schemas pecompted at view definition time. A -scipt execto then etieves the -scipt coesponding to the view fom the -scipt epositoy and exectes it to popagate the changes epesented by the base table i-diff instances to the view instance. We next descibe the -scipt geneation, leaving the discssion of how to convet base table modifications to i-diffs fo Section 5.

5 Δ& & % 1& Δ& & pats& ID Infeence Opeato Rle Instantiation Rle Composition Minimization Δ-scipt geneato Δ& & 2& =& Δ& % & 1& =& Opeato ID infeence les Opeato i-diff popagation les & cache maintenance les Opeato extensibility laye Fige 4: -scipt geneato achitecte Δ& & γ& =&f(δ&&&&&&,&inpt,&otpt& & 2& % % Δ& & pats& SCAN% pats& %pid& did,&pid& did& γ% did;&sm(pice& &cost& %did& did,&pid& SCAN% devices_pats& did& σ% &categoy&=&"phone"& pid& did,&pid& did& (a Algebaic plan (annotated by the -scipt geneato CREATE VIEW V AS SELECT did, sm(pice AS cost FROM pats NATURAL JOIN devices_pats NATURAL JOIN devices WHERE categoy = "phone" GROUP BY did (b View definition SCAN% devices& Fige 5: View definition and plan fo extended nning example 4. -SCRIPT GENERATION ALGORITHM Given a view definition and a set of base table i-diff schemas, the -scipt geneation algoithm ceates a DML scipt that incldes (a qeies ove the base table i-diffs, the base tables and the axiliay caches (which as we will see ae sed by idivm to speed p the IVM that compte the coesponding view i-diffs and (b UP- DATE / INSERT / DELETE statements of the fom descibed in Section 2 that apply these i-diffs on the view. The -scipt geneato is based on the algebaic IVM appoach [21, 10, 22]: Each elational opeato type (e.g., selection, pojection, join, etc. is annotated with a set of les, descibing how to tansfom an (effective i-diff ove its inpt schema to an (effective i-diff ove its otpt schema. Given this infomation, the -scipt fo a view V can be composed fom the individal les fo each opeato that appeas in an algebaic plan of V. Intitively, the algoithm is compting how to maintain the entie view by fist compting how to maintain all intemediate sbviews in the algebaic plan. This appoach enables a modla implementation in which the sppoted view definition langage can be easily extended by adding les fo additional elational algeba opeatos. In this wok we pesent the les fo all opeatos inclded in Q SP JADU (i.e., selection, join, genealized pojection involving fnctions, goping with aggegation, antisemijoin, and nion. Similaly to pio algebaic IVM appoaches, we assme that the algebaic plan of the view on which the algoithm opeates is given as inpt. Opeato SCAN(R σ φ (R π D(R R S R φ S R φ S bag nion R S gop by γ M...(R Ḡ,f( Table 1: Opeato ID infeence les Δ& & γ& =& Δ& % & 2& % π& cost& Cache& EXAMPLE 4.1. To showcase the algoithm, we extend the view Δ& & of o nning γ& =& Δ& % & example to also pefom an aggegation, etning the total cost 2& % π& of the did&& Cache& pats fo each device. Figes 5b and 5a show the view definition V and a coesponding algebaic plan, espectively. The shaded components ae annotations inseted by the algoithm, which we explain below. Δ& Otpt ID attibtes key(r ID(R ID(R ID(R ID(S ID(R ID(S ID(R ID(R ID(S {b} Ḡ idivm pefoms the 4 efficient passes of Fige 4. Pass 1: Infeing ID infomation fo intemediate views. Since i-diffs detemine the view tples that have to be modified thogh thei π IDs, the view and all intemediate sbviews did,totalpice costpe,totalpice+c Δ cost post (V_ shold contain as pat of thei otpt schema a set of ID attibtes that fom a key ofδ the coesponding view. idivm detemines the ID attibtes that shold be contained in the otpt schema of each sbview thogh the se of ID infeence les. An ID infeence le is spplied fo each opeato type sppoted by the system and descibes how the IDs of the view ooted at an opeato p can be compted fom the IDs of the sbviews ooted at p s childen. Table 1 shows the ID infeence les fo opeatos in Q SP JADU. 4 idivm ses these les to pefom a postode tavesal of the plan checking at each opeato whethe the IDs infeed by these les exist in the opeato s otpt schema. If this is not the case, idivm atomatically extends the plan to inclde the eqied ID attibtes. EXAMPLE 4.2. Fige 5a shows the set of IDs fo each opeato in a shaded oval on the top ight side of the opeato. Note that extending the view with additional ID attibtes simply inceases the width of the view instance (i.e., the nmbe of colmns bt does not affect its cadinality (i.e., the nmbe of tples. In paticla, if V oig is an oiginal view with attibtes Ā and V ID is the view infeed by the ID-infeence algoithm, then fo all instances of the base tables the oiginal view can be compted fom the oiginal view simply by pojecting ot the additional attibtes intodced by the ID-infeence algoithm (i.e., V oig = π Ā V ID. Given that the nmbe of additional ID attibtes is sally small compaed to the nmbe of attibtes aleady in the view, we do not expect the extension of the view schema with ID attibtes to significantly affect the qey evalation pefomance. Impotantly, the above obsevations hold not only fo opeatos in Q SP JADU bt fo any SQL opeato (fo the eade wondeing how this can be the case fo the dplicate elimination opeato δ, given that in geneal δ(π A,B(R is diffeent fom δ(π ID,A,B(R, conside an implementation whee the dplicate elimination opeato δ above is eplaced by the gop by opeato γ A,B. Pass 2: Instantiating les fo each intemediate opeato. To constct the -scipt fo the view, the algoithm employs opeato les that descibe how each opeato can popagate i-diffs ove its inpt to i-diffs ove its otpt. Opeato les. An opeato descibes how to tansfom an i-diff t inpt ove one of its inpt schemas to an i-diff t otpt ove its 4 Union efes to the nion all opeato descibed in Section 2.

6 otpt schema thogh a set of qeies known as i-diff popagation les. These qeies can access (a the opeato s inpt i-diff t inpt and (b the data coesponding to the sbview ooted at the opeato o at one of its child opeatos. The latte is the way in which idivm allows opeato les to access data fom the base tables. Since an opeato does not have knowledge of the exact place in the qey plan whee it appeas to ask fo a qey eslt ove the base tables, it can access the base table data only indiectly by asking fo the sbview ooted at one of its childen thogh the se of the Inpt i=l, (standing fo left and ight inpt, esp. fo binay opeatos o fo the sbview ooted at itself sing the Otpt keywod, espectively. The inpt sbviews can be eqested eithe in thei pe-state fom (i.e., sing the instances of the base tables befoe the diffs wee applied to them o in the post-state (i.e., sing the final instances of the base tables afte the diffs wee applied to them. An i-diff popagation le can specify which of the two vesions of the inpt it needs thogh the sbscipts pe and post. The otpt is always povided in pe-state. EXAMPLE 4.3. Fo instance, a geneal goping and aggegate opeato V = γ X c(inpt Ḡ,f( contains among othes the i- diff popagation le: V = γ X c(inpt Ḡ,f( post Ḡ + Inpt, which semijoins the post-state of the sbview ooted at the opeato s child with an inpt inset i-diff to find all tples that belong to gops affected by the insetions and se them to ecompte the vale of the aggegate fnction fo these gops. The Inpt post keywod is the way in which the opeato asks fo the (post-state of some base data (in this case the base data defined by the sbview ooted at the opeato s child, which fo the aggegate opeato of Fige 5a is the sbview pats pid devices_pats did σ categoy= phone devices. Thee ae two diffeent classes of opeatos in idivm: The fist consists of opeatos which can podce an otpt effective i-diff by looking at one inpt i-diff at a time. These opeatos ae called nonblocking opeatos, in contast to blocking opeatos which need to inspect the entie set of inpt i-diffs befoe ceating an effective otpt i-diff. The opeato type affects how the opeato s i-diff popagation les ae expessed. Fo non-blocking opeatos, each le is expessed ove a single inpt i-diff, while fo blocking opeatos, a le is expessed ove all inpt i-diffs. EXAMPLE 4.4. The geneal aggegate opeato γ of Example 4.3 is a non-blocking opeato, since it can decide how to popagate an inpt inset i-diff withot looking at othe inpt i-diffs (e.g., delete o pdate i-diffs. The eason is that fo each inset i-diff tple it ecomptes the entie affected gop fom the base data ths eflecting indiectly also the changes inced by othe inpt i-diffs. On the othe hand, imagine a moe efficient aggegate opeato designed specifically fo the SUM aggegate fnction. This opeato avoids ecompting entie gops by combining all inpt i-diffs to fige ot the amont by which the aggegate vale of each gop has changed. While it avoids some base table accesses, it eqies knowledge of all inpt i-diffs and is ths a blocking opeato. Tables 4-13 show the i-diff popagation les fo the opeatos consideed in this wok, inclding join, nion, genealized pojection involving fnctions, antisemijoin and aggegation. Rles fo aggegation ae povided in fo diffeent vesions (see Tables 7, 9, 11, and 12; one fo geneal aggegation fnctions and othes fo specialized fnctions, sch as SUM, COUNT and AVG. Some opeato les may also benefit fom special caches to speed p IVM. Fo instance, an aggegate AVG opeato in the pesence of a COUNT and SUM cache can incementally maintain its otpt withot accessing the base tables. To accomodate sch cases, idivm allows opeatos to declae special opeato caches Δ" " Δ" " γ" Δ" +" γ" π" Δ" +" π" Δ" " γ" Δ" " π" Δ" +" π" π% [non*blocking]% Δ" " Δ" " R" Δ" +" γ% sm" % SCAN% R" SCAN% S" Fige 6: Rle DAG stcte [blocking]% [non*blocking]% and associated cache maintenance les, descibing how to compte the i-diffs that maintain the caches. The i-diff popagation les can then be expessed also ove the opeato cache schemas and the opeato cache i-diffs. Table 12 shows the cache maintenance les and i-diff popagation les fo the AVG opeato. Rle instantiation. In its second pass, the -scipt geneato algoithm employs the pedefined opeato les to compte how each base table i-diff is popagated fom opeato to opeato in the view plan. This is done as follows: Fo each base table i-diff schema t R, the algoithm stats fom the scan opeato of the coesponding base table R and in a bottom-p fashion instantiates the les fo all opeatos in the path fom the scan opeato to the oot of the plan. 5 The le instantiation simply consists in selecting fom all les fo the paticla opeato the ones that apply in the paticla case (based on the inpt i-diff schema and othe conditions and eplacing the abstact schema sed in the les with the concete schema of the paticla opeato instance (e.g., fo an opeato V = π x(r the geneal pojection i-diff popagation le = π ā,f( X c + R becomes + V = πx + R. EXAMPLE 4.5. Conside an pdate i-diff schema pats(pid, pice pe, pice post modeling pdates on the pice attibte of table pats of o nning example. Fige 5a shows on the left the coesponding instantiated les geneated by the algoithm. The exact le fo the aggegate opeato is omitted de to lack of space. Howeve, it is impotant to note that it is a le that mentions the inpt i-diff and the inpt and the otpt of the opeato, espectively. Note that fo a single inpt i-diff an opeato may ceate mltiple otpt i-diffs. Fo instance, an pdate i-diff going thogh a selection opeato may lead to inset, pdate and delete i-diffs, depending on whethe a tple satisfied the condition befoe and afte the change. Wheneve the les of an opeato ceate mltiple otpt i- diffs, the above comptation contines conceptally in paallel fo each geneated i-diff schema. This leads to a diected le DAG, whose nodes ae instantiated les and whose edges point fom a le to all les that wee ceated sing its otpt schema. Fige 6 shows sch a stcte. Note how blocking les convet the stcte that wold othewise be a tee into a DAG. Pass 3: Composing opeato-level instantiated les into a - scipt. Each le in the DAG is a qey expessed ove the otpt schema of its paent les (note that the DAG in Fige 6 is shown inveted with its oot shown at the bottom. Ths each i-diff fo the view (which coesponds to a leaf can be compted by composing the instantiated les of its ancestos. The exact ode in which these compositions ae pefomed does not matte, since all consideed i-diffs ae effective. This is gaanteed by the fact that (a the base table diff instance geneato ceates effective diffs (as we will discss in Section 5 and (b i-diff popagation les tansfom effective inpt i-diffs to effective otpt i-diffs. 5 If the base table R appeas with mltiple aliases, this pocess is epeated fo evey scan opeato of R.

7 1 Cache = pats; 2 APPLY Cache; 3 V = π did,cost cost pe,cost+cost cost post (V γ did,sm(pice cost ( π did,pid,(picepost pice in pice ( Cache π pice picein Cache; 4 APPLY V ; Fige 7: -scipt fo nning example To make the geneated plan moe efficient, idivm employs also additional caching, othe than the caching sed intenally by opeatos. In paticla, fo aggegate opeatos, whose les typically ask fo the base data coesponding to thei inpt/otpt (thogh the Inpt i and Otpt keywods, esp., idivm attempts 6 to ceate an intemediate cache in which it mateializes this eslt. This cache is teated as any othe view and maintained ding the IVM pocess. In paticla, idivm fist composes all les that ceate the i-diffs fo the cache and then sing them as inpt, composes the est of the les p to the next cache ntil it eaches the view. EXAMPLE 4.6. Fo instance, as we have seen in Fige 5a, the instantiated le fo the aggegation mentions both the inpt and the otpt of the opeato. Ths, idivm ties to geneate two intemediate caches; one befoe the aggegate and anothe afte the aggegate. Since howeve the otpt of the aggegate coincides with the view (which is aleady mateialized, idivm ceates only the fist cache and tilizes the aleady existing view as the second. The eslt of this composition is a -scipt, containing qeies that compte i-diffs fo an intemediate cache/view and APPLY opeatos that se the DML statements coesponding to each i-diffs type (shown in Section 2 to apply these i-diffs to the cache/view. EXAMPLE 4.7. In o nning example idivm employs an intemediate cache below the aggegate opeato. Ths, it composes the les p to that point, pdates the cache and then ses it as inpt to compose the les p to the view, which is sbseqently pdated. This leads to the -scipt of Fige 7. Pass 4: Optimizing the geneated -scipt. As a last step, idivm optimizes the -scipt by pefoming semantic optimization, which minimizes each individal qey inclded in the plan. In contast to geneal minimization, this minimization takes into accont the special semantics of i-diff tables. As descibed in Section 2, given a base table R(Ī, Ā in its post-state and i-diffs + R (Ī, Ā post, R (Ī, Ā pe, and R(Ī, Ā pe, Ā post ove this table, the following constaints hold: (a C 1 : + R R, (b C2 : πī R π Ī R =, and (c C 3 : π Ī,Ā post R Ī R π Ī,Ā R. idivm minimizes w..t. constaints C 1 - C 3 by employing on top of the standad elational ewite les also the ewite les pesented in Fige 8. Semantic minimization is ccial in eliminating inefficiencies intodced by composing individal opeato les, impoving in some cases pefomance by moe than 50%. Designing opeato i-diff popagation les. The efficiency of the -scipt obviosly depends on the povided i-diff popagation le definitions. Reasoning abot the efficiency of individal les is had, as les affect each othe (e.g., a le avoiding base table accesses may not bing in some infomation that cold be sed by les late in the plan, ths leading to highe access cost late. 6 Intemediate caches ae not geneated when they ae expected to contain mlti-valed dependencies (fo instance de to a many-tomany join, since in that case eading the eslt fom the cache wold inc moe tple accesses than simply ecompting it on the fly fom the base tables. idivm exploits foeign key constaints to infe the absence of mlti-valed dependencies. Fo semijoin + R R.Ī σ φ( X R σ φ( Xpost + R R R. Ī σ φ( Ȳ + R πī,āpost Āσ φ( Ȳ + R R R. Ī σ φ( X R σ φ( Xpost R, if X Ā R R. Ī σ φ( Ȳ R π Āσ Ī,Ā post φ( Ȳ R, if Ā = Ā R R.Ī σ φ( X R R R. Ī σ φ( X R Fo antisemijoin + R R. Īσ φ( X R σ φ( Xpost + R R R. Īσ φ( X R σ φ( Xpost R, if X Ā R R. Īσ φ( X R R R R. Īσ φ( X R R Fo join + R R.Ī R + R R R. Ī R R R R.Ī R * p to enaming Fige 8: Rewite les fo semantic optimization Howeve, in this wok we show that we do not have to get into this easoning pocess. Simply ceating les that individally avoid accessing the base tables when possible leads to efficient -scipts, as shown by o analytical and expeimental eslts. To avoid data accesses, the les ae even allowed to oveestimate, i.e. skip some filteing that wold eqie base table accesses and popagate to thei otpt i-diffs dmmy tples that do not affect the view. EXAMPLE 4.8. Fo instance, the selection opeato allows a delete inpt i-diff to pass thogh the opeato nmodified. Howeve, this means that the otpt i-diff will also instct the deletion of tples that do not satisfy the selection conditions and ths do not exist in the view. Althogh this is an oveestimated i-diff, it does not affect the coectness of the geneated -scipt, since the latte will simply ty to delete some tples fom the view that do not exist. On the othe hand, this le locally minimizes the base table accesses, as it avoids accessing the base tables to filte ot the tples that do not satisfy the selection condition. 5. FROM MODIFICATIONS TO I-DIFFS We saw above how given a set of base table i-diffs, idivm maintains the view. In this section we explain how these base table i- diffs ae geneated fom base table modifications. This is a nontivial poblem, since as explained in Section 2, a single modification can be epesented thogh i-diffs of diffeent schemas, each leading potentially to -scipts of diffeent efficiencies. idivm solves the i-diff geneation poblem thogh the synegy of thee components shown in Fige 3: (a a modification logge ecoding the base table modifications at data modification time, (b a base table i-diff schema geneato deciding at view definition time which base table i-diff schemas to geneate, and (c a base table i- diff instance geneato, tanslating at view maintenance time the modifications ecoded in the log to instances of the pe-compted i-diff schemas. Logging changes to the base tables can be easily pefomed thogh known techniqes, sch as DBMS log inspections, timestamp qeies o tigges (cently sed by idivm. Theefoe we focs next on the othe two components. Geneating i-diff schemas. Given a view definition V, idivm geneates sitable base-table i-diff schemas fo all base tables mentioned in V. Insetions and deletions ae staightfowad cases: Conside a base table R(Ī, Ā with key attibtes Ī and non-key attibtes Ā. Fo each sch table, the i-diff schema geneato ceates a single inset i-diff schema + R (Ī, Āpost (containing all attibtes of R and a single delete i-diff schema R (Ī, Āpe (containing all non-id attibtes of R in pe-state fom. This is based on the obsevation that pe-state vales can lead only to a moe efficient -scipt as they may edce oveestimation and the espective view

8 index lookps. Fo instance, as shown in Table 6 with ble, a selection opeato can exploit pe-state attibtes to filte ot the tples of an incoming delete i-diff that do not satisfy the condition. The same does not hold thogh fo post-state attibtes inclded in pdate i-diffs. Inclding moe post-state attibtes in an pdate i-diff schema leads to a geneally less efficient -scipt, as it has to accont also fo changes in these attibtes. Ceating one pdate i-diff schema fo each sbset of attibtes of each base table is obviosly not an option, de to the exponentiality involved. In idivm we solve this poblem by obseving that the base table attibtes can be divided into sets of attibtes whose pdates lead to the same -scipt and can ths be goped togethe in a single i-diff schema. Fo each opeato op in the algebaic view plan, let C op, be the set of (non-key base table attibtes involved in any condition (e.g., selection, join etc 7. We efe to C op as the set of conditional attibtes fo op. The set of (non-key base table attibte not inclded in any C op fo any opeato op in the view s plan is efeed to as the set of non-conditional attibtes NC. Non-conditional attibtes may still affect the view (since they cold be inclded in the view s otpt, bt intitively they do not affect the geneated -scipt (p to pojections. Updates on each set of conditional attibtes C op on the othe hand may lead to a diffeent -scipt, since the pdated vales may affect whethe the i-diffs make it past op s condition. Theefoe fo each base table R(Ī, Ā, the i-diff schema geneation algoithm ceates (a fo each set C op an pdate i-diff R(Ī, Āpe, Ā post, s.t. Ā = Ā Cop and (b an additional pdate i-diff R(Ī, Āpe, Ā post, containing the non-conditional attibtes of R (i.e., Ā = Ā NC. Poplating i-diff instances. Evey time idivm is invoked to maintain the view, the i-diff instance geneato simply poplates the i-diff tables ceated at view definition time. This is done by extacting the changes since the last view maintenance fom the modification log and adding them as diff-tples to all i-diff tables that contain at least one of the modified attibtes (in the case of pdates and to the single inset and delete i-diff tables (in the case of insets and deletes, espectively. Note, that when extacting the modifications fom the log, the algoithm combines mltiple modifications to the same tple to a single modification, so as to geneate effective diffs. As discssed in Sections 2 and 4, this is ccial fo the algoithm s coectness. 6. PERFORMANCE ANALYSIS We next analytically compte the speedp atio of ID-based ove tple-based cost tple-based IVM (i.e., the atio. A speedp atio geate ID-based cost than 1 signifies that the ID-based appoach has a lowe cost than the tple-based appoach and ths is moe efficient than the latte, while a speedp atio lowe than 1 signifies the opposite. To compte the speedp we fist compte the individal cost of each IVM appoach. The cost of the ID-based/tple-based appoach is meased in the combined nmbe of tple accesses and index lookps inced by the /D-scipt, geneated by the coesponding appoach. Fo the pposes of this analysis, we assme that both appoaches have access to view indices on the view IDs and additionally the tple-based IVM has access to appopiate base table indices (which ae not eqied by the ID-based appoach. We also ty to be as geneal as possible egading the qey plan that the DBMS might choose to execte a paticla /Dscipt. Howeve, since DBMSs employ complex optimizations that cannot be compehensively acconted fo in an analytical model, the compted cost and the associated speedps epoted in this Section shold only be sed as ogh estimates of the actal cost of pefoming IVM that illstate the diffeence in pefomance be- 7 Base table key attibtes do not need to be consideed fo pdates as they ae immtable. tween the two appoaches. Fo an expeimental compaison of the two IVM appoaches, please efe to Section 7. We next pesent the speedp fo two epesentative cases: (a SPJ views, which by defalt do not involve intemediate caches and (b Aggegate views involving goping and associative fnctions, which (by defalt ae sppoted by caches. Fo a detailed analysis explaining how this speedp was compted, please efe to Appendix A. 6.1 SPJ Views Conside the SPJ view V spj: SELECT S FROM R, R 1,..., R n WHERE c whose FROM clase involves a single alias of a table R, (b a t-diff D R on R and (c a coesponding i-diff R. 8 Paametes affecting speedp. The speedp of the ID-based appoach ove the tple-based appoach can be expessed in tems of two paametes: the i-diff compession facto p and the tplebased comptation cost pe base table diff tple a. The i-diff compession facto p = D Vspj / Vspj is the atio of the size of the tple-based diff to the size of the ID-based diff fo the view. p may be less than 1 (when i-diffs smmaize the modifications to the view in a moe compact way than t-diffs, as shown in Fige 2 bt may also be geate than 1 (when i-diffs ae oveestimating and tying to modify tples that do not exist in V spj. The second paamete is the nmbe of accesses a that the tple-based appoach has to pefom on aveage to compte the t-diff tples fo the view that eslt fom a given t-diff tple fo the base table. This cost will typically vay, depending on the plan chosen by the DBMS to evalate the tple-based D-scipt. Speedp atio. The speedp atio of the ID-based appoach ove the tple-based appoach is given by the following fomla: (a if R/D R is an pdate i-diff/t-diff on attibtes of R that ae not involved in selection o join conditions in V spj, then A: Speedp atio = a+2p 1+p (b (i.e., if R/D R is any othe pdate i-diff/t-diff o it is an inset o delete i-diff/t-diff ( B: Speedp atio min a+2p, 1 1+p Discssion. Let s fist explain why pdate diffs on attibtes of R that ae non-conditional may lead to a diffeent speedp atio than othe types of diffs. Since the pdates do not affect how a tple behaves w..t. selections o joins, they ae gaanteed to lead to pdates (i.e., neithe insets, no deletes on the view. When this is the case (case (a above, the ID-based IVM algoithm can simply popagate the base table i-diffs to the view withot accessing the. This speedp is in most pactical cases geate than 1 (meaning that the ID-based is moe efficient than the tple-based appoach. Fo the tple-based appoach to pefom bette, it shold be the case that a < 1 p, which can be satisfied only in the cone case when the following conditions simltaneosly hold: (a the tple-based appoach incs a < 1 tple accesses fo each tple in DR on aveage (which can only happen if many tples of DR shae the same join attibte vales and ths the joined tples can be etieved once and esed fo all of them and (b the ID-based appoach is seveely oveestimating (i.e., p 1. We have expeimentally veified that by following this patten it is possible to ceate contived scenaios in which the tple-based IVM otpefoms the ID-based appoach. base tables, leading to a speedp atio of a+2p 1+p 8 Recall that we se the symbols and D to epesent i-diffs and t-diffs, espectively.

9 Howeve, in all othe cases, the ID-based appoach pefoms bette (with a diffeence that aises popotionally to p. When the base table diffs ae inset o delete diffs o they ae pdates on attibtes involved in conditions (case (b above, then they will lead in geneal to pdates, insets and/o deletes on the view. If they lead to pdates and deletes only, then the eslting speedp and ths the ID-based appoach is expected to pefom bette in most cases. Howeve, if they lead to insets, then the two appoaches will pefom identically and hence exhibit a speedp of 1. Finally, if the base table diffs lead to a combination of pdates, deletes and insets, then the speedp will be a linea combination of the above two speedps and ths will be geate than the smalle of the two (hence the se of ineqality and the min fnction in the above fomla. Ths fo SPJ qeies the ID-based IVM will always (p to the cone case descibed above pefom at least as good as the tplebased IVM and in most cases bette then the latte. The two appoaches will only pefom identically if the IVM wokload is heavy on modifications that lead to insetions to the view. is the same as in the fist case (i.e., a+2p 1+p 6.2 Aggegate Views Conside the aggegate view V agg: SELECT Ḡ, f( X AS g FROM R, R 1,..., R n WHERE c GROUP BY Ḡ whose FROM clase involves a single alias of R, and f is an associative aggegation fnction sch as sm (b a t-diff D R on R and (c a coesponding i-diff R. To ease exposition, we isolate the aggegation opeato of the qey, expessing it thogh the plan V agg = γ Ḡ,f( X gv spj, whee V spj is the plan fo the SPJ qey pesented in Section 6.1. We stdy the inteesting case, whee the ID-based IVM has identified that an intemediate cache stoing the inpt of the aggegate opeato, which is the eslt of the SPJ qey, is beneficial (since withot cache both appoaches wold pefom identically. The tple-based appoach does not se a cache, since it cannot benefit fom it. Both appoaches opeate in two two stages: They fist compte the diff to maintain the SPJ sbview V spj and then se it to maintain the final aggegate view V agg. The second step is the same in both cases. Ths the diffeence in pefomance comes fom compting the diff fo the V spj, which in the case of the ID-based appoach is also sed to maintain the cache. Let a and p be defined as above fo the sbview V spj (i.e., let p = D Vspj / Vspj and let a be the aveage cost inced by the tple-based diff to compte fo each base table diff tple the coesponding diff tples fo the view V spj. Speedp atio. The speedp atio of the ID-based appoach ove the tple-based appoach is given by the following fomla: (a if R/D R is an pdate i-diff/t-diff on non-conditional attibtes of R, then (b Speedp atio = a+x 1+p+x ( Speedp atio min a+x, 1+p+x a+x a+k+x whee x is a cost elated to goping that is shaed by both appoaches and ths can be safely ignoed fo the pposes of o discssion and k is a paamete concening inset diffs that we will explain late. Discssion. Similaly to SPJ views, we diffeentiate between cases whee the base table diffs lead to pdate o delete diffs on the view V spj and cases whee they lead to inset diffs on V spj. a+x 1+p+x In the fist case the speedp atio is s 1 =. This speedp is always going to be at least 1, meaning that the tple-based appoach Relation Use FiendList Micoblog (i.e. tweets Retweets Mentions Rel_Event_Micoblog Tples 1M 100M 20M 4M (#tweets 10% 2 etweets pe tweet 8M (#tweets 20% 2 mentions pe tweet 16M (#tweets 40% 2 events pe tweet (a BSMA elation sizes Qey Desciption Q7 Mentioned ses within a time ange Q10 Uses who ae etweeted within a time ange Q11 Pai of etweeting ses, goped by etweeting times Q15 Uses talking abot events within a time ange Q18 Paiwise cont of mentions Q*1 Aggegate of fiends of fiends within the same city Q*2 Aggegate of etweetes fo evey se Q*3 Aggegate of ses who tweet abot topics (b BSMA qeies and additional Qeies Fige 9: Configation of social analytics expeiments can neve pefom bette than the ID-based appoach. This happens becase the cost a inced by the tple-based appoach fo each diff tple in D R is at least 1 + p, since fo each sch tple it will have to inc at least one index access (to find the tple of the othe elations it joins with and p tple accesses (to ead the tples it joins with to ceate the coesponding p tples in the view. Note that these ae lowe bonds that apply when the view V spj contains only one join. If it contains moe joins, the speedp atio and ths the pefomance benefit of ID-based IVM will be even highe. On the othe hand when base table diffs lead to inset diffs on the view V spj, the ID-based appoach will be pefoming the same plan with the tple-based appoach bt will also be inseting tples into the cache. Ths if k is the nmbe of tples ceated in V spj on aveage as a eslt of a single diff in D R, then the speedp will be s 2 = a+x. This speedp is less than 1 (meaning that the a+k+x tple-based appoach will be pefoming bette, bt now the loss is bonded, as it is always 1 pe tple inseted into V spj. Similaly to SPJ views, when the base table diffs ae pdates on non-conditional attibtes (case (a above, the geneated diffs on the view V spj ae gaanteed to be pdate o delete diffs and ths the speedp atio will be eqal to s 1. In any othe case (case (b, the geneated view diffs will in geneal be combinations of pdate, delete, and inset diffs and ths the speedp will be a linea combination of s 1 and s 2 (and ths bonded by the smalle of them. To smmaize, fo all base table diffs that do not lead to insets in the cache, the ID-based appoach is gaanteed to pefom not wose (and the moe complex the qey the bette than the tplebased appoach. It only pefoms wose in wokloads heavy on modifications that lead to insetions to the view, becase it has to maintain a cache, so that the pdate and delete diffs can benefit fom it. Howeve, even this loss is bonded and we expect it to not be significant in pactice. Moeove, this loss will be balanced ot by the speedp on diffs that lead to deletes and pdates in the view, which benefit fom the cache. 7. EXPERIMENTAL EVALUATION To compae the pefomance of ID-based and tple-based IVM, we an two sets of expeiments. In o fist set of expeiments we stdied the pefomance of both appoaches on a divese wokload of views, by applying them on views commonly sed in social netwoks. In o second set of expeiments we stdied the effect of vaying diffeent paametes on both the view and the data (sch as selectivity, fanot, nmbe of joins and base-table diff size.

10 Speedp&& 60# 50# 40# 30# 20# 10# 0# 29x# 54x# 26x# ID#IVM#(sec Tple#IVM#(sec x# 14x# 26x# 7x# 9x# Q7# Q10# Q11# Q15# Q18# Q*1# Q*2# Q*3# Fige 10: Speedp & IVM time fo extended set of BSMA qeies In all cases we sed idivm to geneate the -scipt and D- scipt (the latte was podced sing o implementation of idivm with tple-based diff popagation les instead of the standad IDbased popagation les. We then meased the times to n each scipt in PostgeSQL. All expeiments wee n sing Ubnt LTS 12.04, OpenJDK JRE 7 and PostgeSQL 9.1, on top of an Amazon Web Sevices (AWS m1.lage dedicated instance configed with 1,200 inpt/otpt opeations pe sec (IOPS fo 16KB blocks. The configed IOPS povide pedictable thoghpt fo andom block accesses. Each expeiment was n with cold PostgeSQL page bffes and Linx disk bffes, which is the common case when lage nmbe of views need to be maintained. 7.1 IVM in social analytics To stdy the elative pefomance of ID-based and tple-based IVM on a divese set of eal qeies, we applied both IVM appoaches to a wokload of analytics views ove social media. Maintaining analytics ove social media is a pimay se case fo IVM techniqes becase: (a lage base tables ae podced by social media sch as Twitte and Facebook, (b apid, feqent pdates occ on the base tables, and (c analytic views that monito metics and tends need to be pdated continosly. To geneate the wokload, we tilized the Benchmak fo Social Media Analytics (BSMA [26]. Fige 9a shows the size of the elations geneated, while Fige 9b povides a smmay of the wokload, which compises: 100 pdate diffs on the Use table fo attibtes tweetsnm (i.e nmbe of tweets and favonm (i.e. nmbe of favoites Views coesponding to qeies Q7, Q10, Q11, Q15 and Q18 fom BSMA, which exhibit join chains and aggegates, hence eslting in high cost fo view e-comptation. These qeies ae also minimally extended to: (a extend the SELECT clase with attibtes tweetsnm and favonm (b emove the ORDER BY and LIMIT and the ID paamete in the WHERE clase in ode to ceate lage views whee the benefit of the IVM becomes appaent. An additional 3 aggegate views ove the BSMA schema, labeled as Q*1, Q*2 and Q*3. Wheeas qeies Q7, Q10, Q11, Q15 and Q18 inclde aggegation, this aggegation is not affected by the pdated attibtes. Since, as we have discssed in Section 6 the ID-based and tple-based appoaches behave diffeently in the pesence of aggegates affected by the pdates, we designed views Q*1, Q*2 and Q*3 to inclde sch aggegates. Fige 10 shows the speedp atio of the ID-based ove the tplebased appoach fo each of the views. The speedp vaies widely between the 8 diffeent views and its vale does not seem to be detemined by whethe a view contains an aggegate affected by an pdate o not. Fom the epoted views, it is inteesting to look at the exteme cases with eithe vey high o low speedp. Qeies Relation Tples Size pats 5M 170MB devices 5M 170MB devices_pats 50M 3GB (a Relation sizes Paamete Defalts d: Diff size 200 s: Selectivity 20% f: Fanot 10 j: Joins 2 (b Paametes pats = D pats (pid, picepe, picepost (c Base-table Diff Fige 11: Configation of vaying paamete expeiments Q10 and Q*1 ceate a hge benefit fo the ID-based IVM de to the fact that the tple-based IVM has to inc a lage nmbe of data accesses, which can be avoided by the ID-based IVM. Inteestingly, this need fo data accesses is ceated in diffeent ways by each of the two qeies. In Q10 it is ceated by a long join chain (Q10 joins 4 elations, while in Q*1 it is ceated by a combination of a long join chain with a high selectivity that appeas at the end of the join chain (so that the tple-based has to pefom a lot of data accesses befoe it can decide that a tple will be dopped. Finally, Q15 displays a elatively low speedp becase the eslting view and the nmbe of tples that need to be pdated in the view is vey lage. This makes the view pdate time component (which is shaed by both the ID-based and the tple-based appoaches dominate the IVM cost, ths leading to a elatively small speedp. Howeve, it shold be noted that even in this case the ID-based appoach otpefoms the tple-based appoach by a facto of Effect of data & qey paametes To stdy in a moe contolled fashion how the stcte of the view and the data affects the ID-based and tple-based IVM, we next an a second set of expeiments, in which we picked a single view and meased the pefomance of both ID-based and tplebased IVM on maintaining the view, while vaying diffeent paametes of both the view definition and the ndelying data. Fo ease of exposition we employed the view sed in o nning example and shown in Fige 5a. Expeimental setp. Fige 11 shows the popeties of the dataset and the view sed in the expeiments. Fige 11a shows the sizes of the elations, Fige 11c pesents the employed basetable diff (which captes pdates on the pices of pats and Fige 11b lists the paametes that we vaied in the expeiments and thei defalt vales. We vaied fo paametes: (a The size d of the base-table diff (i.e., the nmbe of pice pdates that happened, (b the nmbe of joins pefomed by the view (as we will explain late we extended the view that by defalt pefoms two joins with additional joins, (c The selectivity s of the selection condition categoy= phone (i.e., the pecentage of devices tples that satisfy the condition, and (d the fanot f fom pats to devices_pats, i.e. the nmbe of pats fo each device. (Note that the fanot fom association table devices_pats to entity table devices is always 1. Fo each expeiment vaying a paamete, we sed fo all othe paametes thei defalt vales shown in Fige 11b. Fige 12 shows the view maintenance times fo ID-based IVM vess tple-based IVM and the eslting speedp of ID-based ove tple-based IVM. The cost of each appoach is boken down to its components. Colmn A epesents ID-based IVM, with the top stack (diagonally stiped coesponding to cache pdate time (ecall that the inpt of an aggegate is mateialized as an intemediate cache, and the bottom stack (solid coloed coesponding to view pdate time. No stack is shown fo the cache/view diff comptation time as both ae negligible. Colmn B epesents tple-based IVM, with the top stack (hoizontally stiped coesponding to view diff comptation time, and the bottom stack (solid coloed coespond-