Journal of Network and Computer Applications

Transcription

1 Jounal of Netwok and Compute Applications 34 (211) Contents lists available at ScienceDiect Jounal of Netwok and Compute Applications jounal homepage: Optimization of sub-quey pocessing in distibuted data integation systems Gang Chen, Yongwei Wu n, Jia Liu, Guangwen Yang, Weimin Zheng Depatment of Compute Science and Technology, Tsinghua National Laboatoy fo Infomation Science and Technology, Tsinghua Univesity, Beijing 184, China aticle info Aticle histoy: Received 28 Octobe 29 Received in evised fom 21 Apil 21 Accepted 7 June 21 Available online 15 June 21 Keywods: Cloud computing Gid computing Data integation Quey Data flow abstact Data integation system (DIS) is becoming paamount when Cloud/Gid applications need to integate and analyze data fom geogaphically distibuted data souces. DIS gathes data fom multiple emote souces, integates and analyzes the data to obtain a quey esult. As Clouds/Gids ae distibuted ove wide-aea netwoks, communication cost usually dominates oveall quey esponse time. Theefoe we can expect that quey pefomance can be impoved by minimizing communication cost. In ou method, DIS uses a data flow style quey execution model. Each quey plan is mapped to a goup of mengines, each of which is a pogam coesponding to a paticula opeato. Thus, multiple sub-queies fom concuent queies ae able to shae mengines. We econstuct these sub-queies to exploit ovelapping data among them. As a esult, all the sub-queies can obtain thei esults, and oveall communication ovehead can be educed. Expeimental esults show that, when DIS uns a goup of paameteized queies, ou econstucting algoithm can educe the aveage quey completion time by 32 48%; when DIS uns a goup of non-paameteized queies, the aveage quey completion time of queies can be educed by 25 35%. & 21 Elsevie Ltd. All ights eseved. 1. Intoduction As cloud and gid computing is becoming moe and moe popula, inceasing numbe of applications needs to access and pocess data fom multiple distibuted souces. Fo example, a bioinfomatics application needs to quey autonomous databases acoss the wold to access diffeent types of poteins and potein potein inteaction infomation located at diffeent stoage clouds. Data integation in Clouds/Gids is a pomising solution fo combining and analyzing data fom diffeent stoes. Seveal pojects (e.g., OGSA-DQP Lynden et al., 29; CoDIMS-G Fontes et al., 24; and GidDB-Lite Naayanan et al., 23) have been developed to study data integation in distibuted envionments. Fo example, OGSA-DQP (Lynden et al., 29) is a seviceoiented, distibuted quey pocesso, which povides effective declaative suppot fo sevice ochestation. It is based on an infastuctue consisting of distibuted sevices fo efficient evaluation of distibuted queies ove OGSA-DAI wapped data souces and analysis esouces available as sevices. Queies to data integation systems ae geneally fomulated in vitual schemas. Given a use quey, a data integation system n Coesponding autho. Tel.: addesses: c-g5@mails.tsinghua.edu.cn (G. Chen), wuyw@tsinghua.edu.cn (Y. Wu), liu-jia4@mails.tsinghua.edu.cn (J. Liu), ygw@tsinghua.edu.cn (G. Yang), zwm-dcs@tsinghua.edu.cn (W. Zheng). typically pocesses the quey by tanslating it into a quey plan and evaluating the quey plan accodingly. A quey plan consists of a set of sub-queies fomulated ove the data souces and opeatos specifying how to combine esults of the sub-queies to answe the use quey. As Clouds/Gids ae geneally built ove wide-aea netwoks, high communication cost is the main eason of leading to slow quey esponse time. Theefoe, quey pefomance can be impoved by minimizing communication cost. In this pape, ou objective is to educe communication ovehead and theefoe impove quey pefomance, though optimizing sub-quey pocessing. We optimize sub-quey pocessing by exploiting data shaing oppotunities among sub-queies. IGNITE is a method poposed in Lee et al. (27) to detect data shaing oppotunities acoss concuent distibuted queies. By combining multiple simila data equests issued to the same data souce, and futhe to a common data equest, IGNITE can educe communication ovehead, theeby incease system thoughput. Howeve, IGNITE does not utilize paallel data tansmission so that it does not always impove quey pefomance. Ou appoach poposed hee enhances IGNITE by addessing its dawbacks so that quey pefomance in distibuted systems can be futhe impoved. Ou data integation system employs an opeato-centic data flow execution model, also poposed in Haizopoulos et al. (25). Each opeato coesponds to a mengine, which has local theads fo data pocessing and data dispatching. Queies ae pocessed by outing data though mengines. All the mengines wok in paallel, thus they can fully utilize inta-quey paallelism. Based /$ - see font matte & 21 Elsevie Ltd. All ights eseved. doi:1.116/j.jnca

2 136 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) on such an opeato-centic data flow execution model, all simila quey plans ae allocated to the same goup of m Engines. Theefoe sub-queies fom diffeent queies ae gouped in a common place fo pocessing to enable data shaing acoss the sub-queies. In the mengine fo pocessing sub-queies, a quey econstuction mechanism with a Mege-Patition (MP) econstuction algoithm is developed. The quey econstuction mechanism can constuct a set of new queies to eliminate data edundancy among the sub-queies being pocessed by the mengine. All the sub-quey answes can be obtained by evaluating the new queies and theefoe the equied communication ovehead can be educed. The est of the pape is oganized as follows. Section 2 pesents elated wok. Section 3 descibes the execution model of ou DIS. Section 4 poposes the Mege-Patition (MP) quey econstuction algoithm used in ou DIS. Section 5 discusses the expeiments that we conducted to evaluate ou solution. Section 6 concludes the pape. 2. Related wok IGNITE system poposed in Lee et al. (27) was developed based on the PostgeSQL database, and is a wok mostly elated to the wok pesented in this pape. IGNITE decouples the souce wappes fom the execution engine (adopted fom the PostgeSQL database), and enables the execution engine to send subqueies to same souce, which theefoe makes data shaing acoss sub-queies possible. Meanwhile, IGNITE employs the iteato model poposed in Gaefe (1993) so that sub-queies may have delay oppotunities a sub-quey can wait fo othe simila equests. Because of this, IGNITE develops a Stat-Fetch wappe with Request Window mechanism. The wappe combines a goup of simila sub-queies to a common sub-quey and only sends the common sub-quey to the data souce, so that edundant answes among sub-queies can be eliminated. Thee ae two majo diffeences between ou method and IGNITE. Fist, ou method econstucts oiginal sub-queies to altenative sub-queies, which may not eliminate all edundant answes, but neve intoduce unnecessay data. IGNITE combines a goup of sub-queies into a single common sub-quey to eliminate edundant answes; howeve by doing so it may intoduce unnecessay data and in some cases may incease the size of quey answes. IGNITE inceases communication taffic in two ways: (1) it equies not only output attibutes, but also pedicate attibutes to identify sub-quey answes; (2) all tuples including common tuples must contain all equied attibutes fo all sub-queies. The second majo diffeence between ou method and IGNITE is that if the souce wappe manages multiple wok theads, ou method can take advantage of paallel sub-quey pocessing, wheeas IGNITE cannot. A significant amount of wok on data integation (i.e. Ives, 22; Halevy et al., 26; Deshpande et al., 27; Haas et al., 1997) has been conducted. Seveal pojects (e.g., OGSA-DQP (Lynden et al., 29); CoDIMS-G (Fontes et al., 24); and GidDB- Lite (Naayanan et al., 23)) paticulaly focus on data integation in Clouds/Gids. With a sevice-oiented achitectue, OGSA-DQP suppots pipeline and patition paallelism fo efficient evaluation of distibuted queies. Diffeent fom ou method, OGSA-DQP uses iteato model and elevant eseach on OGSA-DQP often focuses on impoving the pefomance of a single quey. Similaly, CoDIMS-G and GidDB-Lite also focus on impoving the pefomance of a single quey. Many effots have been made on exploiting data shaing in data integation aea as well as database aea (e.g., Dalvi et al., 21; Haizopoulos et al., 25; Lee et al., 27; Goldstein and Lason 21; Sacco and Schkolnick, 1986), including: (1) Multiplequey optimization (MQO) techniques (e.g., Dalvi et al., 21), which exploit data shaing by identifying common sub-expessions in quey execution plans duing optimization; (2) buffe pool management (e.g., Sacco and Schkolnick, 1986), which typically euses disk pages in a buffe pool; (3) caching and view mateialization (e.g., Kossmann, 2; Goldstein and Lason, 21), which typically euse pe-stoed data in cache o mateialized view. Thee ae also techniques poposed in the distibuted data pocessing aea, aiming to impove quey efficiency (e.g. paallel quey pocessing techniques poposed in Gounais (25) and adaptive quey pocessing techniques poposed in Deshpande et al. (27) and Gounais (25)). The technique poposed in Kossmann (2) is one of them, which achieves the objective by deceasing communication cost. Fo example, semi-joins ae poposed in Kossmann (2) to educe data tansition while pocessing joins between tables stoed at diffeent sites, and ow blocking is used to educe the numbe of communication occuences by deliveing tuples in batches. 3. Quey engine In this section, we discuss the execution engine of ou DIS. The engine employs a data flow style execution model (Section 3.1), based on it, sub-queies can be gatheed to a common place fo evaluation though souce wappes (Section 3.2). We also discuss in Section 3.3, in detail, why is equied to have a delay fo each equest in ode to bette utilize data shaing Data flow execution model As peviously discussed, ou DIS employs a data flow style execution model, also efeed to as opeato-centic (oneopeato, many-queies) model in Qpipe (Haizopoulos et al., 25). In this model, each opeato uses an independent mengine. mengines seve equests fom submitted queies. Each equest specifies input and output data buffes, and opeato aguments. By linking a mengine s output to anothe s input, poduce consume elationships can be established among mengines. Queies can then be evaluated by pushing data though mengines. Fig. 1 descibes the untime model of ou data flow execution. In this model, thee ae fou kinds of elements: Quey Plans, equests, dispatche and mengines. Quey Plans: a Quey Plan consists of a set of sub-queies fomulated ove the data souces and opeatos specifying how to combine esults of the sub-queies to answe the use quey. equests: ae geneated accoding to the Quey Plans. They can be consideed a goup of opeations need to be pefomed by mengines. dispatche: is a component which is esponsible fo sending the equests to pope mengines. mengines: Each ound box in Fig. 1 epesents a mengine and the text in the box indicates its coesponding opeato. In Fig. 1, mengines labeled with wappe o WSP is used to pocess sub-queies o invoke web sevices, espectively. mengines labeled with Sot, Selection and Hashjoin ae used to pocess elational opeatos Sot, Selection and Hashjoin, espectively. The pocess of evaluating a quey plan is as follows. Afte the aival of the quey plan, the dispatche ceates as many equests as the nodes in the quey plan and dispatches these equests to thei coesponding mengines. Then, the mengines wok in paallel

3 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) Quey Plans Q 1 Q 2 Q 3 dispatche equests µengine Sot μengine Wappe μengine Output HashJoin µengine Selection µengine WSP tuples Fig. 1. Runtime achitectue of the execution engine. to pocess the equests, and the data tuples flow among the buffes of diffeent mengines in push mode fo evaluation. The easons why we choose the opeato-centic data flow model instead of the iteato model ae as follows. Fist, in the data flow model, all mengines can wok in paallel; theefoe inta-quey paallelism can be achieved, and quey pocessing can be acceleated. This featue is paticulaly impotant to DIS fo the eason that queies of DIS (e.g., OGSA-DQP (Lynden et al., 29) and CoDIMS-G (Fontes et al., 24)) always contain timeconsuming opeatos such as extenal web sevice calls and local function calls. Second, the data flow model can goup equests with the same natue togethe, and can pocess each goup of simila equests using a dedicated mengine. This featue enables the execution engine to send all sub-queies to same souce and theefoe enables data shaing acoss sub-queies Sub-quey evaluation though souce wappes Wappes ae used to evaluate sub-queies in data integation systems, whee wappes hide the heteogeneity of accessing data souces of diffeent types. Fo example, OGSA-DAI acts as a gidenabled wappe, poviding sevice access intefaces fo vaious data souces. Ou DIS has one specific mengine, efeed to as mengine-w, to invoke wappe pocedues of evaluating sub-queies. As shown in Fig. 2, the mengine-w consists of the following elements. Sub-quey equests: a Quey Plan consists of a set of sub-quey equests. Quey Reconstucto: the component which is esponsible fo econstucting the equests by using Quey Reconstuction Algoithm poposed in this pape (Section 4). Reconstucted sub-queies: the sub-queies geneated by Quey Reconstucto. Coodinato: the component is esponsible fo eoganizing the esults of econstucted sub-queies fo the oiginal subqueies. Wappe Handle: the component is esponsible fo pefoming the queies on Data Souces. A data shaing mechanism is poposed in this pape (Section 4) and applied in mengine-w to optimize sub-quey pocessing. Fig. 2 shows the oveview achitectue of mengine-w with a quey econstuction mechanism. To pocess sub-quey equests, mengine-w fist econstucts a set of sub-queies (say W) contained in the sub-quey equests into a substitute set of sub-queies (say W ) using the Quey Reconstucto. This step identifies and eliminates edundant data among the sub-queies Fig. 2. Achitectue of mengine-w. in W. Then mengine-w evaluates the econstucted sub-queies W to get equied answes fo the oiginal queies W using the Wappe Handle Sub-quey delay to bette facilitate data shaing This subsection discusses when is appopiate fo mengine-w to tigge a econstuction. The taget of the econstuction algoithm is the sub-quey equests waiting in mengine-w. Assume that mengine-w schedules an available wok thead to emove and pocess a equest immediately afte the equest aives. The only case in which concuent waiting equests occu is that the equests aive at the exact same time o mengine-w has no idle wok thead. Theefoe, the oppotunity fo educing quey cost though exploiting data shaing in this case is vey low and thus the expected cost saving is quite limited. In ode to incease the cost saving, mengine-w makes use of the allowed delays of sub-quey equests. As descibed in IGNITE (Lee et al., 27), when complex queies ae executed, some subqueies sent to the data souces may have a delay oppotunity so that it is toleable to wait fo othe simila sub-queies. Let s take the following quey fo example: select *fom t1, t2 whee t1.id¼t2.id. Suppose we use a hash join to calculate the tuples

4 138 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) satisfying t1.id¼t2.id. The left child node of the hash join is a subquey to fetch tuples fom t1, and the ight child node is a subquey to fetch tuples fom t2. The pocessing of the hash join consists of two phases: (1) building the hash table using tuples fom t1, and (2) pobing the hash table using tuples fom t2. We can see that thee is an inteval between the aival time of the sub-quey to t2 and the time that the hash join takes its esult. Though IGNITE discusses the delays in the context of the iteato model, we can see that the same mechanism can be similaly applied in ou context: the opeato-centic data flow model. By setting a delay to each sub-quey equest, the oppotunity of the pesence of concuent equests can be inceased and the cost saving of the quey econstucting mechanism can be inceased accodingly. Afte adding a delay to each sub-quey equest, mengine-w emoves the equests fom its queue, classifies them into goups accoding to thei taget data souces, and then tigges a econstuction in the following two situations: (1) the waiting time of a equest has eached its toleable delay and (2) the esult of a equest is about to be consumed and mengine-w is notified by the consume mengine of the equest to tigge a econstuction to its coesponding goup. 4. Quey econstuction algoithm In this section, we intoduce the Mege-Patition (MP) econstuction algoithm applied in ou DIS. Fist, we model the poblem of quey econstuction in Section 4.1. Then, in Section 4.2, we pesent the algoithm to see how it econstucts a set of queies and computes the answes of the queies Poblem desciption We assume evey sub-quey is a Select Poject Join (SPJ) quey with duplicate-peseving semantics. Evey quey is in the fom of p L ðs P ðr 1 R 2 R a ÞÞ, whee L is the list of output attibutes, P is the selection pedicate, and R 1,R 2,,R a ae queied elations. Fo a given quey q i : Let L o (q i )be the set of output attibutes. Let P(q i ) be the selection pedicate of q i. Let L c (q i ) be the set of attibutes that appea in P(q i ). Let Sðq i Þ¼fR 1,,R a gbe the set of souce elations. Let R(q i ) be the answe of q i. Fo two queies q i and q j, let R(q i \q j )be thei common answe. The poblem of quey econstuction can be descibed as follows. Given a set of SPJ queies Q ¼fq 1,,q n g, we compute anothe set of queies Q * ¼fq * 1,,q* mg such that: (1) we can poduce the answes to the oiginal queies Q ¼fq 1,,q n g fom the answes to Q*; (2) Size(RðQ * Þ)Size(R(Q)), which means the netwok ovehead incued by deliveing RðQ * Þis smalle than that of R(Q). The quey econstuction mechanism consists of two activities: (1) econstucting the set of oiginal queies befoe dispatching them to the data souces, and (2) computing the answes to the oiginal queies based on the answes to the econstucted queies MP quey econstuction algoithm In this section, we descibe ou Mege-Patition quey econstuction algoithm, which econstucts queies by two steps: quey meging and quey patitioning, fo both paameteized queies (Section 4.2.1) and non-paameteized queies (Section 4.2.2) Paameteized quey Paameteized queies ae queies that have one o moe embedded paametes in a SQL statement. The main advantages of a paameteized quey ae: (1) it makes a SQL statement less pone to eos and (2) it saves quey pepaation time since you can pepae the quey one time, and execute the quey as many times as you wish. Thee ae two steps to econstuct a goup of paameteized queies: meging all the queies using the method poposed in IGNITE (Lee et al., 27) and using a ange-patition method to patition the meged quey into quey fagments. Let us take the following example to illustate ou method: Q: select* fom a_table whee key4? Q1: select* fom a_table whee key41 Q2: select* fom a_table whee key42 Q3: select* fom a_table whee key45 Suppose Q1 Q3 ae thee sub-queies, which ae geneated by embedding values 1, 2 and 5 in the paameteized quey Q. To econstuct the queies Q1 Q3, the fist step is to mege these thee queies, which esults the following meged quey: Q4: select* fom a_table whee key 41. Then, the MP quey econstuction algoithm patitions the meged quey Q4. Assume that the domain of attibute key is 2, and the numbe of the fagments is 4. An example of quey patitioning is Q5 Q8: Q5: select* fom a_table whee key41 and (key5); Q6: select* fom a_table whee key41 and (key45 and key1); Q7: select* fom a_table whee key41 and (key41 and key15); Q8: select* fom a_table whee key41 and (key415 and key2); To compute the answe of Q1, we should apply its pedicate (key41) to the answe of the meged quey Q4. It is obvious that: RðQ4Þ¼RðQ5Þ[RðQ6Þ[RðQ7Þ[RðQ8Þ The answes to Q2 and Q3 can be computed in a simila way Non-paameteized quey Mege queies. Fo any two queies q i and q j in Q ¼fq 1,,q n g, they can be meged if all of the following thee conditions can all be satisfied: (1) L c (q i )DL o (q i ); (2) L c (q j )DL o (q j ); (3) L o (q i )¼L o (q j ). Let q m be the meged quey of q i and q j. The answes to q i and q j can be computed fom the answe to q m because of the following easons: (1) the answe to q m get all the tuples equied by q i and q j ; (2) the answe to q m get all the columns equied by q i and q j ; (3) the answes to q i and q j can be computed since the attibutes equied by thei pedicates ae povided in the answe to q m. Besides, fom the thid condition L o (q i )¼L o (q j ) (see above), we can see that no unnecessay data is intoduced by evaluating the meged quey q m to get answes fo q i and q j.

5 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) We use the set Q 1 ¼fq 1 1,,q1 s g to epesent the esult afte the step of meging queies. In this step, we ecod all the pais of queies that can be meged. Let MP¼{/q 11,q 12 S,y,/q p1,q p2 S}be these quey pais. The communication ovehead can be saved by this step issize(r(q 11 \q 12 ))+?+size(r(q p1 \q p2 )). Fo each quey pai, say /q i, q j S, we also ecod the following infomation as lineage expessions: (1) /q m, func i, q i S (2) /q m, func j, q j S Whee func i is the function used to compute the answe to q i fom the answe to q m, andfunc j is the function used to compute the answe to q j Patition queies. In this step, we futhe patition the queies in Q 1 ¼fq 1 1,,q1 s g to eliminate ovelapping data. We use an example to explain the basic idea of the quey patition fist. Fig. 3 shows two queies: q 1 and q 2. We use a twodimensional aea to epesent a quey answe. As shown in Fig. 3(a), if you pocess the two queies diectly, the common data epesented by the yellow aea is tansmitted twice. Fig. 3(b) shows the way afte applying quey patition. Queies q 1 and q 2 ae patitioned into thee fagments: (1) q * 2 : to get the ovelapping tuples (i.e., a quey with condition(p(q 1 )\P(q 2 ))), (2) q * 1 : to get the emaining tuples of q 1 (i.e., a quey with condition(p(q 1 ) P(q 2 ))), and (3) q * 3 : to get the emaining tuples of q 2 (i.e., a quey with condition(p(q 2 ) P(q 1 ))). Afte the answes to q * 1, q* 2 and q* 3 ae etieved, the answe to q 1 can be computed diectly fom the answes to q * 1 and q* 2. The answe to q 2can be extacted diectly fom the answes to q * 2 and q* 3. Given two queies, if the amount of thei common data is small, the method of quey patition may lead to pefomance degadation. The eason is that patition inceases the numbe of queies and each quey bings an initial delay. Thus, we need to estimate if a quey patition can bing in pefomance impovement befoe take any actions. Given two queies q i and q j, without a quey patition, the cost of sub-quey pocessing is befoe ¼ sizeðrðq iþþ þ sizeðrðq jþþ þ2id whee denotes the data ate ove the link between the quey engine and the taget data souce, IDdenotes the initial delay of pocessing a sub-quey. With a quey patition, the cost is afte ¼ sizeðrðq* ij1 ÞÞþsizeðRðq* ij2 ÞÞþsizeðRðq* ij3 ÞÞ þ3id afte which can also be expessed as afte ¼ sizeðrðq iþþþsizeðrðq j ÞÞ sizeðrðq * ij2 ÞÞ þ3id To detemine if the quey patition can impove the quey pefomance, we need to compae befoe withcij. Geneally, if afte befoe Cij Za is satisfied (the amount of impovement exceeds afte a specified theshold a), a quey patition can be consideed. We q 1 q 2 Fig. 3. Patition the queies q 1 and q 2 to the queies q * 1, q* 2 and q* 3, to eliminate ovelapping data (each two-dimensional aea epesents a quey answe). * q 1 * q 2 * q 3 use imp deived: to denote Cij befoe Cij, and the following equation can be afte imp ¼ sizeðrðq* ij2 ÞÞ ID To check if Za is satisfied, an estimate of the amount of imp the common data Rðq * ij2þ is equied. We can use statistic techniques in database aea (e.g., Ahad et al., 1989; Getoo et al., 21) to pefom the estimation. Due to space limitations, we omit the discussion of the estimaiton fom this pape. Afte the basic idea of patition and the basic conditions to conside a patition ae clea, we now descibe the pocess of pefoming patitions ove the queies in Q 1 ¼fq 1 1,,q1 s g. Step 1 estimates the amount of common data between evey two queies, and build the following matix M pi to denote the pioities of quey patitions: M pi ¼ q 1 1 q 1 2 ^ q 1 s q 1 1 q 1 2 q 1 s 1 p 11 p 12 p 1s p 21 p 22 p 2s B ^ ^ & ^ A p s1 p s2 p ss whee, p ij denotes the pioity of the quey patition between q 1 i and q1 j.ifi¼j, then p ij¼; othewise, p ij can be computed as p ij ¼ a imp ¼ sizeðrðq* ij2 ÞÞ ID a Step 2 selects the biggest and positive pioity fom the matix M pi. Biggest one means the one with the highest pioity, and positive one means that the equiements of pefoming quey patition ae satisfied. if p ij is the biggest one, emove the i th and j th ows, and the i th and j th columns fom the matix M pi, pefom a patition to the queies q 1 i and q1 j, and ecod the following infomation as lineage expessions: (1) /q * ij1, null, q1 i S (2) /q * ij2,func i, q 1 i S (3) /q * ij2,func j, q 1 j S (4) /q * ij3, null, q1 j S whee, /q * ij1, null, q1 i S means the answe to q* ij1 is a subset of the answe to q 1 i. null means no computation on R(q* ij1 ) is needed befoe etuning it to q 1 i. func i is the function used to compute the answe to q 1 i fom the answe to q * ij2, which is actually a pojection opeation. Othe symbols can be explained similaly. Step 3 epeats step 2 until thee is no positive numbe in the matix M pi. We use the set Q * ¼fq * 1,,q* mgto epesent the esult afte the step of patitioning queies. Then, we can begin to evaluate queies in Q *. When the data etuned fom the data souce aives, we can fist use the lineage expessions ecoded in the step of patitioning queies to compute the answes of the queies Q 1 ¼fq 1 1,,q1 s g, and then use the lineage expessions ecoded in the step of meging queies to compute the answes to the oiginal sub-queies Q ¼fq 1,,q n g. 5. Evaluation In this section, we pesent ou evaluation method and esults. The oveall expeimental setup is discussed in Section 5.1,

6 14 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) followed by the detailed discussion of each expeiment and its esults in Section Expeimental setup The expeimental setup consists of two pats: the seve side to deploy databases and the client side to un ou data integation system. The expeimental envionment is pesented in Fig. 4, including the location and configuation of each machine, and the data distibution ove the machines. On the seve side, we use a TPC-H database ( (scale facto 1) as the dataset of ou expeiments. The TPC-H database has eight elations: REGION, NATION, CUSTOMER, SUPPLIER, PART, PARTSUPP, ORDERS, and LINEITEM. To build a distibuted envionment, we ceated eight PostgeSQL (vesion: 8.3) databases on fou diffeent machines, as shown in Fig. 4. Each of the eight databases povides one of the above elations. The dataset in the elations was geneated by DBT ( On the client side, ou DIS was deployed on a sepaate machine, as shown in Fig. 4. The value of sub-quey delay is set accoding to the cadinality of tables and the stuctue of the whole quey tee. This pinciple is followed by Lee et al. (27) Expeiment analysis and esults Thee expeiments have been designed and conducted to evaluate (1) the effectiveness of the opeato-centic data flow quey model by compaing it with the iteato model, (2) the effectiveness of the Mege-Patition algoithm by compaing it with IGNITE having the mege algoithm used in mengine-w and an oiginal appoach whee mengine-w pocesses sub-queies without using any quey econstuction mechanism, when pocessing a goup of paameteized queies, and (3) the effectiveness of the Mege-Patition algoithm when pocessing a goup of non-paameteized queies Expeiment 1 Expeiment 1 evaluates the effectiveness of the opeatocentic data flow quey model while executing DIS queies. We evaluated a goup of queies that follow the paameteized quey descibed in Fig. 5(a). The paameteized quey contains two web sevices WS 1 and WS 2. The quey plan descibed in Fig. 5(a) is used to answe the queies. In this expeiment, the selectivities of WS 1 and WS 2 ae 1, and the aveage esponse time of WS 1 and WS 2 ae.6 and.11 s, espectively. Duing diffeent uns, the paamete values of the quey wee set as , , , y, and , espectively. We an the queies in both the iteato model and ou data flow model. The expeiment esult is shown in Fig. 5b), whee the x-axis and y-axis denote the paamete value the quey and the completion time of the quey, espectively. Fom the figue, we can see that the data flow model achieves 4% less quey completion time than the iteato model. This is a easonable esult because the data flow model invokes web sevices WS 1 and WS 2 in paallel to pocess data, which is the ationale why we chose the data flow model athe than the iteato model employed in IGNITE Expeiment 2 This expeiment evaluates the effectiveness of the Mege- Patition algoithm by pocessing a goup of paameteized queies. In this expeiment, we compae the aveage quey completion time and communication taffic of the following thee appoaches. (1) MP: the MP algoithm is used in mengine- W; (2) IGNITE: the mege algoithm is used in mengine-w; quey completion time Q: select o_odekey, o_odedate, output1, output2 fom odes WS 1( o_odedate, output1) WS 2( o_odedate, output2) whee o_odedate <? WS ( o_odedate, output2) 2 WS 1( o_odedate, output1) Sub-quey: select o_odekey, o_odedate fom odes whee o_o dedate <? iteato dataflow paamete value of the quey Fig. 5. (a) Paameteized quey and quey plan. (b) Quey completion time. Fig. 4. Expeimental envionment.

7 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) (3) Oiginal: mengine-w pocesses sub-queies without using any quey econstuction mechanism. In this expeiment, 5 queies (q 1 q 5 ) ae geneated complying with the quey descibed in Fig. 6(a). 49 andom numbes ( 1 49 ) in the ange of 6 ae geneated as quey intevals. Queies q 1 q 5 ae submitted sequentially. The time inteval between submitting q i (1i49) and submitting q i +1 is i seconds. Fig. 6(b) and (c) pesents the esults of the expeiment. Fig. 6(b) ecods the total communication taffic of the queies of each of the thee appoaches. Compaed with the Oiginal appoach, IGNITE and communication taffic (m) Hashjoin Sub-quey1 Sub-quey2 Sub-quey1: select o_odekey, o_odedate fom odes whee o_odedate <? Sub-quey2: select l_odekey, l_commitdate, l_eceiptdate, l_discount, l_comment fom lineitem whee l_discount<? Oiginal IGNITE MP ou MP appoach can educe the communication taffic by aound 2% and 15%, espectively. Fig. 6(c) shows two plots of the quey completion time of the thee appoaches at each quey inteval. The y-axis of each plot is the completion time of q 1 q 5,thex-axisisthe inteval between the aival time of a quey and the aival time of the fist quey q 1, and each point in each plot epesents a quey. The aveage quey completion time of the Oiginal, IGNITE and MP is , and in deceasing ode, espectively. Fom Fig. 6(c), the following conclusions can be dawn. Fist, compaed with Oiginal, ou MP appoach can educe the aveage completion time by 48%. This is obviously because MP can exploit data shaing among sub-queies. This is due to one fact. Though econstucting small and lage queies togethe may take shote time than econstucting them sepaately, this may lead to econstucting small queies takes moe time. Thid, the aveage completion time of MP is smalle than that of IGNITE by 32%. It is because MP can get a highe degee of paallelism duing the phase of patition than IGNITE Expeiment 3 This expeiment evaluates the effectiveness of the Mege- Patition algoithm by pocessing a goup of non-paameteized queies. Similaly to Expeiment 2, in this expeiment, 5 queies (q 1 q 5 ) and 49 andom numbes ( 1 49 ) in the ange of 6 ae geneated as quey intevals. The geneated queies ae submitted sequentially. Fig. 7 shows the expeimental esults. Fig. 7(a) ecods the total communication taffic of the queies of each appoach. Compaed with Oiginal, both IGNITE and MP can educe the communication taffic by about 15%. Fig. 7(b) shows a plot of the aveage quey completion time of evey five queies of the thee appoaches. Each point in the plot epesents an aveage completion time of each five queies. Fo example, the point 1 quey completion time (s) quey completion time (s) Oiginal MP quey inteval (s) IGNITE MP quey inteval (s) Fig. 6. Expeimental esults of paameteized queies: (a) paameteized quey, (b) total communication taffic of queies and (c) quey completion time of each quey. communication taffic (m) quey completion time (s) Oiginal IGNITE MP Oiginal IGNITE MP quey inteval (s) Fig. 7. Expeimental esults of non-paameteized queies: (a) total communication taffic of queies, (b) aveage completion time of each quey goup.

8 142 G. Chen et al. / Jounal of Netwok and Compute Applications 34 (211) coesponding to the numbe 1 of the x-axis epesents the aveage completion time of queies q 1 q 5. Fom Fig. 7(b), we can obseve the following facts. (1) Compaed with Oiginal, MP can educe the aveage completion time of queies by moe than 35%. (2) Fo most of the queies, MP takes less time to complete the queies than IGNITE; MP can educe the aveage completion time of all the queies by 25%, to compae with IGNITE. 6. Conclusion Distibuted data souces can be heteogeneous, and managing, analyzing, and pocessing data fom diffeent souces in an integated way is becoming moe and moe impotant. Distibuted data integation applications ae always pocessed on distibuted infastuctues, and communication cost becomes the main facto of detemining quey esponse time. Theefoe we can expect that quey pefomance can be impoved by minimizing communication cost. The objective of this pape is to popose an appoach to impove the quey pocessing pefomance of data integation systems by optimizing sub-quey pocessing. Ou data integation system adopts a data flow style execution model, which allows the system to exploit data and wok shaing oppotunities acoss queies duing the pocess of quey evaluation, at the same time, impoves inta-quey paallelism. An expeiment was conducted to demonstate the effectiveness of the data flow model of ou choice by compaing it with the iteato model employed in IGNITE. The expeiment esult confims ou choice: the data flow model achieves 4% less quey completion time than the iteato model. We also developed a Mege-Patition quey econstuction algoithm in a mengine (Haizopoulos et al., 25) fo pocessing sub-queies. The poposed econstuction algoithm is able to exploit data shaing oppotunities among the concuent subqueies, which can educe the aveage communication ovehead equied by the sub-queies and can theefoe impove the oveall quey pefomance. Two expeiments have been designed and conducted to evaluate the effectiveness of ou Mege-Patition algoithm by compaing it with two appoaches: IGNITE having the mege algoithm used in mengine-w and an oiginal appoach whee mengine-w pocesses sub-queies without using any quey econstuction mechanism, when pocessing a goup of paameteized and non-paameteized queies, espectively. The esults show that, by applying quey econstuction mechanism with ou Mege-Patition algoithm, communication ovehead of executing sub-queies can be educed, and pefomance of DIS queies can be impoved coespondingly. Moe detailedly, compaed with the oiginal appoach, ou MP appoach can educe the aveage completion time of queies by moe than 48% and 35% fo paameteized and non-paameteized queies, espectively. Compaed with IGNITE, MP can educe the aveage completion time of all the queies by 32% and 25% fo paameteized and non-paameteized queies, espectively. Acknowledgement This Wok is suppoted by Natual Science Foundation of China (683121, , , 98121, 69635), National High-Tech R&D (863) Pogam of China (29AA1A13, 26AA1A11, 26AA1A18, 26AA1A111, 26AA1A117), MOE-Intel Foundation and Tsinghua National Laboatoy fo Infomation Science and Technology (TNLIST) Coss-discipline Foundation. Refeences Ahad Rafiul, Bapa Rao KV, Mcleod Dennis. On estimating the cadinality of the pojection of a database elation. ACM Tansactions on Databases 1989;14(1):28 4. Dalvi Nilesh N, Sanghai Sumit K, Roy Pasan, Sudashan S. Pipelining in Multi- Quey Optimization. In PODS 21. Deshpande Amol, Ives Zachay, Raman Vijayshanka. Adaptive quey pocessing. Foundations and Tends in Databases 27;1(1):1 14. Fontes V, Schulze B, Duta M, et al. CoDIMS-G: a data and pogam integation sevice fo the gid. In: Poceedings of the 2nd wokshop on Middlewae fo gid computing. Ontaio, Canada, 24. p Gaefe Goetz. Quey evaluation techniques fo lage databases. ACM Comput Suv 1993;25(2): Getoo Lise, Taska Benjamin, kolle Daphne. Selectivity estimation using pobabilistic models. SIGMOD 21. Goldstein J, Lason P. Optimizing queies using mateialized views: a pactical, scalable solution. SIGMOD 21: Gounais Anastasios. Resouce awae quey pocessing on the gid. PhD thesis, School of Compute Science of the Univesity of Mancheste, 25. Haizopoulos S, Shkapenyuk V, Ailamaki A. QPipe: a simultaneously pipelined elational quey engine. In: Poceedings of SIGMOD, 25. Alon Halevy, Rajaaman A, Odille J. Data integation: the teenage yeas. In: Umeshwa Dayal, Kyu-Young Whang, editos. Poceedings of the VLDB 6. Seoul, Koea, 26. p Haas LM, Kossmann D, Wimmes EL, Yang J. Optimizing queies acoss divese data souces. In: Poceedings of VLDB, p Ives Z. Efficient quey pocessing fo data integation. PhD thesis, Univesity of Washington, 22. Kossmann Donald. The state of the at in distibuted quey pocessing. ACM Comput Suv 2;32(4): Lee Rubao, Zhou Minghong, Liao Huaming. Request window: an appoach to impove thoughput of dbms-based data integation system by utilizing data shaing acoss concuent distibuted queies. In: Chistoph Koch, Johannes Gehke, editos. Poceedings of the VLDB 7. Vienna, Austia, 27. p Lynden Steven, Mukhejee Aijit, Hume Alastai C, et al. The design and implementation of OGSA-DQP: A sevice-based distibuted quey pocesso. Futue Geneation Compute Systems 29;25(3): Naayanan S, Kuc TM, Saltz J. Database suppot fo data-diven scientific applications in the gid. Paallel Pocessing Lettes 23;13(2): OGSA-DAI, / Sacco GM, Schkolnick. M. Buffe management in elational database systems. ACM TODS 1986;11(4): / /