2PC AGENT METHOD: ACHIEVING SERIALIZABILITY IN PRESENCE OF FAILURES IN A HETEROGENEOUS MULTIDATABASE

2PC AGENT METHOD: ACHIEVING SERIALIZABILITY IN PRESENCE OF FAILURES IN A HETEROGENEOUS MULTIDATABASE Antoni Wolski nd Jri Veijlinen Technicl Reserch Centre of Finlnd Lortory for Informtion Processing Lehtisrentie 2A, 00340 Helsinki Finlnd Astrct A method for integrted concurrency control nd recovery, pplicle to heterogeneous multidtse systems is proposed 1. The role of the prticipnt in the two-phse commit protocol is lid on n entity clled 2PC Agent ssocited with the locl dtse system. The min importnce of the method is in preserving glol serilizility in the presence of unilterl orts nd site filures. The method requires the prticipting dtse systems to use the strict two-phse locking or comprle rigorous concurrency control policy. Introduction There hs recently een much interest in integrting pre-existing dtses mnged y heterogeneous dtse mngement systems (DBMS). This is understood to e cused y the need to eliminte "islnds of informtion" [11] nd, generlly, the necessity to improve the interoperility of dtse systems [23]. There re vrious rchitectures supporting these ojectives. The multidtse rchitecture [25] is chrcterized y preserving vrious spects of locl system utonomy [31, 13] to gret extent. The dtse systems retin their design utonomy, i.e. neither their functionl chrcteristics nd the existing interfces cn e modified, nor the dtse structures chnged. In ddition, they hve execution utonomy which is reflected y their freedom to execute dtse opertion in ny loclly suitle order or to ort n opertion. The federted dtse rchitecture [19] represents n interoperility pproch sed on glol frmework for schem informtion exchnge. In order to fully utilize integrted dtses, fcilities to support trnsctions spnning distinct dtses re needed. The utonomy properties of the systems mke them incple of prticipting in conventionl glol concurrency control s well s commit nd recovery procedures required to ttin the full qulity of trnsction processing. We propose method for integrted concurrency control nd recovery, pplicle to heterogeneous multidtse system. Such system includes redy-mde dtse mngement systems (clled Prticipting DBMSs here) 1 This pper is n enhnced version of the originl PARBASE-90 puliction. The min chnges re the introduction of the two-step sutrnsction certifiction nd the inclusion of the certifiction lgorithm descriptions. The sketch of the proof hs een updted ccordingly. 28 Jnury, 1991. To pper in: N. Rishe, S. Nvthe nd D. Tl (eds.), Dtses: Theory, Design nd Applictions, IEEE Computer Society Press, 1991.

2 locted, essentilly, t distinct computing nodes (Prticipting Sites). The Prticipting DBMSs re heterogeneous with respect to locl DBMS softwre. Trditionlly, the concept of correctness of trnsctions in heterogeneous dtses [16] hs een sed on recoverle nd serilizle trnsction executions [3]. Other trnsction models were lso proposed [2, 14, 22, 28], with the emphsis on compenstion sed recovery which ws extensively studied,e.g. in [32]. We shll focus our ttention on serilizility-preserving concurrency control nd recovery methods in the sequel. Most of the ltest reserch work in the re hs concentrted on the concurrency control issues limited to filure-free opertion. The exmple methods re the site grph method [4], the ltruistic locking [29], the cycle detection method of [30], the optimistic lgorithm of [11], the integrtion method using oservility nd controllility [21], the superdtses [27], the topdown pproch of [10], the vlue dte method [24] nd the ticket methods [15]. Prolems with llowing locl trnsctions hve lso led to new correctness criterion, the qusi serilizility, which is weker notion thn conflict serilizility [8]. Locl trnsctions re trnsctions executing within the scope of Prticipting DBMS, nd they re not ville for nlysis y ny entity outside the Prticipting DBMS. The common ssumption mde in the ove studies is tht the locl concurrency control mechnism of prticipting locl dtse system is unknown. This leds to pproches which re chrcterized y serious limittions imposed on the concurrency of glol trnsctions. Also, the ove methods typiclly require centrlized glol trnsction mnger, nd some of them require modifictions to prticipting dtse mngement systems. For comprison of some of the methods, see [9, 7]. A notle progress in deling with filures is represented y the commit grph method of [6]. Under certin restrictions the method gurntees the conflict serilizility of glol nd locl trnsctions, nd the dedlock detection. Typicl filures re tken cre of. The locl trnsction mngers need not e modified. The method is sed on centrlized trnsction mngement fcility utilizing pessimistic nd highly restrictive concurrency control policy. In this work, we set up set of similr gols reflecting prcticl point of view. We ssume the prticipting DBMSs retin their design utonomy nd the execution utonomy with respect to glol requests, which is reflected, mong others, y their freedom to unilterlly ort n opertion t ny time. Thus, the recovery from such filures must e included in the method. We lso llow for sumission of locl trnsctions through the existing locl interfce lthough we shll hve to restrict them considerly. The significnce of our pproch is in tht we consider centrlized concurrency control solutions imprcticl, nd we intend to tke dvntge of decentrlized dynmic concurrency control policies (e.g. locking) which led to higher concurrency rtes thn centrlized methods [1]. We ssume the ojective of the trnsction mngement system is to preserve the trditionl chrcteristics of trnsction (so-clled ACID): tomicity, consistency, isoltion nd durility [18]. The view serilizility [3] is ssumed s sufficient consistency criterion. Typiclly, in distriuted dtse systems, the ACID properties re ttined y mens of the sic two-phse commit protocol (2PC) [17] or its vritions [26], nd the relted recovery protocols. In the sic 2PC scheme, Coordintor responsile for the glol trnsction commitment communictes with Prticipnts executing the sutrnsctions. It is typicl of the scheme tht every Prticipnt hs to move sutrnsction to recoverle prepred stte efore the

3 glol trnsction is finlly committed. In this stte, the Prticipnt does not hve the execution utonomy it is oliged y the protocol to execute the finl decision commnd (Commit or Aort) issued y the Coordintor. Systems supporting the prepred stte re clled here two-phsed DBMSs, wheres systems without n pproprite 2PC interfce will e clled single-phsed DBMSs. It is evident tht if ritrry single-phsed DBMSs re used s Prticipting DBMSs nd, dditionlly, sumission of locl trnsctions is llowed, then the ojective to gurntee ACID properties cnnot e met in generl. The fct tht most of the existing systems re single-phsed, nd thus neither support the prepred stte nor hve n externl interfce for prticipting in the 2PC protocol, is mjor ostcle in the wy of heterogeneous DBMS integrtion. Therefore, our pproch is sed on the importnt restricting ssumption tht the Prticipting DBMSs re cple of producing histories of the type produced y strict two-phse locking [3] schedulers. The ssumption is derived from the oservtion tht most of the commercilly ville systems use the strict two-phse locking (S2PL) [12] policy. Since this pproch is the previling one, the design utonomy of the Prticipting DBMS is not too much reduced y this requirement in prctice, while opening up prcticl possiilities to chieve the ACID properties. As concerns the pplicility of strict 2PL to distriuted dtses, it hs een shown tht, in the sence of filures, glol trnsction scheduler using the distriuted two-phse locking protocol produces serilizle histories for glol trnsctions [3]. It hs lso een proved tht this is true for mix of glol nd locl trnsctions [5]. We shll use the ove result s the sis of norml opertion, i.e. in sence of filures. In the following, we shll concentrte on chieving tomic trnsction commitment nd lso recovery from the two most common types of filures, i.e. the Prticipting Site system filure nd the unilterl sutrnsction ort y Prticipting DBMS. The resulting 2PC Agent Method for integrting single-phsed prticipting DBMSs into the 2PC scheme is presented elow. The ssumed rchitecture nd requirements re descried in the next section. A short outline of the method follows. Then, simplified proof of correctness is presented. At the end, the remining prolems re discussed. A note on the prototype implementtion of the method is included too. Assumptions nd requirements Overll rchitecture The rchitecture model of multidtse trnsction mngement system we re proposing is shown in Fig. 1. It is uilt of softwre modules (oxes) nd interfces (horizontl segments). An interfce is descried in terms of commnds which fcilitte dt exchnge nd control flow etween the interconnected modules. The interction through n interfce consists of issuing commnd nd the susequent response returned in the opposite direction. All the interfces of the model re synchronous, i.e. commnd elonging to given trnsction cn not e strted efore the processing of the previous commnd of the s m e tr n s c tio n hs een completed t the interfce. By n opertion we men the execution of commnd. An opertion is sid to e performed fter the result or confirmtion is received through the interfce in question. The modules re multithreded in the sense tht opertions elonging to different trnsctions my e processed in prllel.

4 A (multidtse) glol trnsction is understood s finite series of dtse opertions (in prcticl sense, e.g. executions of SQL commnds) ech of which origintes within the ppliction execution nd provides return vlues to the ppliction t the Glol Interfce (GI). A glol sutrnsction is series of ll the opertions of glol trnsction, performed t the 2PC interfce of prticulr site. Thus, similrly to trnsction models used elsewhere [16, 5, 11, 9], for every glol trnsction there is t most one glol sutrnsction per site. A locl sutrnsction is series of opertions pertining to glol sutrnsction, performed t the locl DBMS interfce. The distinction etween the glol nd locl sutrnsctions is essentil in chieving the ility to del with trnsction filures fflicted y the Prticipting DBMS, s will e shown in the sequel. The commnds of trnsction re not known priori to the system when the trnsction is strted. In prticulr, the invoction nd prmeters of some commnds my depend on the results of other opertions. Coordinting Site Glol trnsctions Glol trnsctions Coordintor GI GI Prticipting Site Coordintor (...) Glol sutrnsctions Coordintor 2PC 2PC Agent LI EI LTM Locl trnsctions 2PC Agent LTM (...) Locl sutrnsctions Elementry commnds 2PC Agent LTM LI EI Dtse Dtse Dtse Fig. 1. Proposed rchitecture of multidtse trnsction mngement system. The ssumptions out GI re s follows: (GI.1) (GI.2) (GI.3) The interfce supports high level dtse mnipultion commnd (e.g. SQL DML: Select, Updte, Insert nd Delete ) tht re trnsformle (y ech underlying DBMS) to sequence of Red nd Write commnds performed on elementry dtse ojects; Ech dtse mnipultion opertion is explicitly relted to given Prticipting DBMS; The interfce supports trnsction mngement commnds Begintrnsction, Commit nd Aort.

5 The Coordintor decomposes glol trnsctions into glol sutrnsctions, sumits the corresponding commnds to the Prticipting Sites nd returns the results to the ppliction. Upon receiving the glol Commit, it strts the distriuted commitment procedure ccording to the sic 2PC protocol [17]. The Prticipnt role is plyed y the 2PC Agent (2PCA) modules. There is single instnce of the 2PCA ssocited with Prticipting DBMS. The connecting lines in Fig. 1 denote potentil ssocitions etween the modules. The functionlity of the 2PC Agent interfce is defined y the 2PC protocol. In prticulr, the Prepre commnd (PREPARE messge) is request to move the sutrnsction to the prepred stte. The response my either e ffirmtive (READY messge) or negtive (REFUSE messge). The decision commnds Commit or Aort (COMMIT or ABORT messge) hve to e ccepted nd cknowledged unconditionlly. The Locl Trnsction Mnger (LTM) modules, represent the trnsctionl spects of the Prticipting DBMSs. There is single LTM (nd the corresponding 2PCA) per site. In fct, the concept of site my e generlized into tht of computing environment ssocited with given LTM. There is n importnt premise tht the LTMs re pre-existing systems nd their design utonomy is mximlly preserved. A glol trnsction results in one or more locl sutrnsctions seen t LI. In filure-free opertion there is only one locl sutrnsction per glol trnsction. Locl trnsctions (which re not seen y the 2PCA) lso enter the LTM t LI. The LTM cn not differentite etween the locl trnsctions nd the locl sutrnsctions. In ddition to the ove components, for the ske of model completeness, we ssume hypotheticl Elementry Interfce (EI) where the elementry commnds Red nd Write del with elementry dtse ojects. The ltter ones re the ojects recognized y the specific concurrency control method pplied y the LTM (e.g. hving grnulrity used y the locking scheme). The trnsctionl commnds Begintrnsction, Commit nd Aort re s such not supported t this interfce, ut the LTM decomposes them into suitle elementry opertions. The min ssumptions out the Locl Interfce re the following: (1PC) (DLU) (RTT) There exists single-phse trnsctionl interfce to the LTM supporting the sme dtse commnds s those t the GI level nd the locl commnds Begintrnsction, Commit nd Aort. Denied Locl Updtes. Locl trnsctions do not updte ojects red or written y locl sutrnsction, while the corresponding glol sutrnsction is in the prepred stte. Rel Time Trnsprency. Any two identicl sequences of dt mnipultion commnds executed t ritrry points of time produce the sme results (in terms of dtse stte chnges nd commnd responses) provided the dtse ojects red y them hve identicl vlues in either cse. The LTMs hve the following chrcteristics: (SRS) Serilizle nd rigorous histories. LTM enforces histories tht re conflict serilizle, nd rigorous mening they re strict [3] nd dditionlly such tht no dt oject my e written until the trnsctions tht previously red it commit or ort [15]. Typiclly, it is sufficient to hve the strict locking policy (S2PL) wherey ll ojects ccessed y trnsction re unville to other trnsctions until

6 (UAN) (TW) (LL) (RWMC) Commit/Aort. For etter clrity, the exmples included in the pper, nd some of the rgumenttion, re sed on the ssumption tht LTMs pply the S2PL policy. Unilterl Aort Notifiction. If LTM unilterlly orts locl sutrnsction, this informtion is pssed over to the 2PCA, through the LI interfce, on the next synchronous commnd return. Trustworthiness. After fixed numer of resumissions, ny glol sutrnsction tht should e committed cn e committed. LTM Log. LTM mintins its own internl trnsction log. By working off the log, LTM is cple of: rolling ck ny locl (su)trnsction or locl trnsction nd undoing nd redoing locl (su)trnsctions nd locl trnsctions during the (LTM-driven) site recovery, if necessry. Red-Write model comptiility. LTM trnsforms the high level dtse mnipultion commnds into the series of the elementry Red nd Write.commnds [3]. The elementry commnds re oservle t hypotheticl Elementry Interfce (EI). Filure modes We re considering two types of filures t Prticipting Sites: Site filure system filure (system crsh) t Prticipting Site. No progrm stte is retined. Non-voltile repositories (logs) re ville for recovery. Sutrnsction filure locl sutrnsction is orted unilterlly y the LTM. The stte of the 2PC Agent nd the LTM re preserved. This filure my occur either (or oth) efore or fter chieving the prepred stte y the glol sutrnsction. We re not deling with filures types tht do not imply specil tretment in n environment discussed, s compred to homogeneous dtse system. Such filure types re, e.g.: lost or corrupted messges, messges rriving out of order, or ny type of Coordintor filures. Also, no specil ttention need to e pid to sutrnsctions tht hve not reched the prepred stte ecuse they my e "leglly" orted nywy. It is the responsiility of the 2PC Agent to recover ll the trnsctions tht hd een in the prepred stte s the filure occurred. In cse of site filure, this lso includes restoring of the locl execution order tht corresponds to the glol order. Ojectives of the 2PC Agent Method The 2PCA method, under the ssumptions stted ove, ssures the view serilizility [3] of glol histories. A simplified version of the method produces qusi serilizle [8] histories 1. 1 We use slightly weker notion of qusi serilizility thn tht in [Du&Elmgrmid89]; To us, history H is qusi serilizle if its projection on glol trnsctions H[g] is serilizle nd ech locl history H[ i ] is serilizle.

7 Description of the 2PC Agent method Generl The prepred stte recovery scheme is sed on the ide of the sutrnsction resumission which is repeted execution of ll the commnds elonging to glol sutrnsction when corresponding locl sutrnsctions hd een orted y the LTM. A trnsction resumission results in new locl sutrnsction. In the process of recovery, the 2PCA my generte mny locl sutrnsctions for given glol sutrnsction. Then, ll ut the lst one (in the history) re in the orted stte. The lst one my e incomplete, orted or committed. Agent Log A 2PC Agent mintins its own log (Agent Log) solely for the purpose of recovery of sutrnsctions tht hd reched the prepred stte. Every time glol sutrnsction opertion hs een completed, the 2PCA writes the corresponding dtse lnguge commnd to the log. Thus log entry implicitly indictes successful lock reservtions s well. The 2PCA lso writes the PREPARED record efore the READY messge is sent. Ech time PREPARED record is written, the Agent Log is forced to disk (or other stle storge) to fcilitte site recovery. We cll this force writing of the record. When 2PCA receives the COMMIT messge, it writes the COMMIT record in the log. This is to e performed s prt of the locl sutrnsction so tht it cn e committed together with the other opertions of the sutrnsction. This gurntees tht either oth the glol decision nd the locl trnsction result re recorded in the persistent storge, or none of them is recorded. This wy of opertion is esily chievle y implementing the Agent Log in the dtse under the control of the LTM. Filure time concurrency control In filure-free opertion, the serilizility of glol trnsctions is gurnteed y the strict two-phse locking (or other equivlent) mechnism of the ssocited LTMs. The mechnism fils if there re ny unilterl orts of prepred sutrnsctions. In cse of locl filure it my re-schedule the trnsctions in such wy tht the sutrnsction execution order will chnge. For exmple, ssume we hve two glol sutrnsctions executing t given node i: T i 1[x] otined lock on oject x, nd is in prepred stte; T i 2[x] witing for lock on oject x. The resulting execution order is: T i 1[x] < T i 2[x] ("<" mens "executed efore") which complies with some glol seriliztion order. Now, ssume T i 1 is orted y LTM. This hs the consequence tht T i 1 is rolled ck nd the locks held y this trnsction re relesed. Before 2PC Agent mnges to recover the stte of T i 1 (i.e. resumit T i 1), LTM schedules T i 2 for execution. The resulting execution order would e T i 2 < T i 1 which violtes the glol seriliztion order ( cycle ppers in the seriliztion grph).

8 The min prolem here is to preserve, t ech site, the sme locl order mong locl sutrnsctions tht would e ttined y wy of locking in the filure-free sitution. There re t lest two pproches to this prolem one pessimistic nd the other optimistic. In the pessimistic pproch, the 2PC Agent hs to mintin locks on dtse ojects solely for the ske of recovery from sutrnsction filures. As no 2PC Agent hs control over physicl dtse ojects, logicl locking scheme hs to e pplied. Possile techniques re predicte locks [12], precision locks [20] or some corse grnulrity methods like tle locks. Appliction of the pessimistic pproch would men duplicting of the locking mechnism existing in the LTM nd high performnce penlty. Since, under the SRS property, the glol histories in filure-free sitution re serilizle without ny intervention of the 2PC Agent, we re proposing n lterntive optimistic pproch here. At first, the commnds of n unprepred sutrnsction re executed without ny ordering ctivity of the 2PC Agent, relying on the locl seriliztion mechnism. Then, t rrivl of the PREPARE messge, the 2PCA performs the prepre certifiction. The ojective of the prepre certifiction is to ensure tht the originl locl opertion order hs een preserved in spite of sutrnsction filures. The prepre certifiction lone gurntees qusi serilizility of glol histories in the presence of filures. It is the sis of the single-step certifier of the 2PC Agent. Once sutrnsction hs een prepre certified, it enters the prepred stte. If it is rejected, the REFUSE response is issued or s n optimiztion the certifiction is retried lter. At the rrivl of the COMMIT messge, the 2PC Agent my lso perform the second certifiction step the commit certifiction. The gol of this step is to ensure tht no cycles hve een introduced into the glol seriliztion grph y (red-only) locl trnsctions executed during the sutrnsction filures. The gol is chieved y delying commitment of certin sutrnsctions. If sutrnsction is commit certified, it my e immeditely committed. Otherwise it is scheduled for repeted certifiction t some lter point of time. A two-step certifier performing oth the prepre nd commit certifiction gurntees view serilizility of glol histories under the stted ssumptions. Both certifiction methods re discussed elow. Prepre certifiction The prepre certifiction is performed in such wy tht the following invrint is not violted: 2PCA Correctness Invrint: ) no glol sutrnsction is moved to the prepred stte if the corresponding locl sutrnsction is unilterlly orted nd ) no two glol sutrnsctions hve conflicting elementry dtse opertions while they re in the prepred stte. As usul, conflict tkes plce etween the unprepred locl sutrnsction T i u nd n nother locl sutrnsction T i t node i if: (i) (ii) T i u reds or writes locl dtse oject written y T i t EI, T i u writes n oject red y T i t EI. The sutle issue in the definition of the conflict is tht the elementry dtse ojects nd opertions, nd thus the definition of conflicts, might e different from site to site. Still, we ssume tht, t ech site, the locl system somehow orders its elementry dtse opertions in such wy tht the SRS property holds for the locl history. Thus, "loclly conflict equiv-

9 lent" seril history cn lwys e found. Since we only need seriliztion order of locl sutrnsctions t site, the possile differences in the implementtion of the concept of "conflict" etween two different sites do not ctully mtter, s long s ech implementtion of the conflict is correct (i.e. ech history is view serilizle). If the 2PCA Correctness Invrint cn relly e kept y the 2PC Agent, then the sutrnsctions cn e committed without jeoprdizing the serilizility mong glol trnsctions no mtter when nd if the COMMIT messge rrives. It is siclly possile to mintin the Correctness Invrint if only such glol sutrnsctions re moved to the prepred stte tht do not hve conflicting opertions with the lredy prepred ones. How is it possile to find those glol sutrnsctions? We sy tht locl sutrnsction or locl trnsction (oth denoted in the sequel s "locl (su)trnsction") is ctive if it is neither loclly committed nor orted. Then the following fct holds: 2PCA Implementtion Bsis: If two locl (su)trnsctions re ctive t the sme time nd they hve completed ll their opertions, then they cnnot hve conflicting elementry dtse opertions, provided tht the Prticipnt LTM produces SRS histories. To see tht the Implementtion Bsis is correct, let us ssume there is some conflict. This is possile only if oth conflicting trnsctions re ctive, or t lest one is committed. The ltter is in contrdiction with the ssumption tht oth re ctive. Let them oth e ctive. A result from the conflict etween ctive trnsctions under rigorousness is tht t lest one trnsction must e witing for some dt oject (or lock on it) to e relesed y the other one. This mens necessrily tht one trnsction would not hve completed ll its opertions, which is in contrdiction with the ssumption tht ll opertions re completed. Thus, there cnnot e conflicting opertions. The certifier using the Implementtion Bsis needs primrily to scertin tht there is moment in the pst t which oth the prepred sutrnsction to e tested nd the locl sutrnsction to e certified hve een ctive nd hve performed ll their opertions. If such moment cn e found with ech of the prepred sutrnsctions seprtely, then the glol trnsction hs pssed the test nd cn e moved to the prepred stte, otherwise the stte trnsition is not llowed (t tht moment). Resumissions of opertions might cuse the Correctness Invrint to e violted since the locl dtse stte could hve een chnged during unilterl ort y locl sutrnsctions or locl trnsctions. This is, however, impossile: since the prepred originl locl sutrnsctions do not hve conflicting elementry opertions, nd since the locl dtse recovers correctly (cf. LL property), the resumitted locl sutrnsction cnnot ccess (under RTT) new dtse ojects s compred to the originlly ccessed. Only intervening locl trnsctions could cuse the dtse stte to evolve in such wy tht conflicts might rise etween resumitted glol sutrnsctions (tht is, the red-write sets of the resumitted locl sutrnsctions might e different from the originl ones). However, sed on DLU it cn e shown tht the resumitted commnds result in locl sutrnsction tht is structurlly identicl to the originl one. Thus, the Correctness Invrint cn, t lest in principle, e kept in the environment. Techniclly, the certifier lgorithm is sed on live time intervls rther thn on distinct points of time. A glol sutrnsction is live if the corresponding locl sutrnsction is ctive. The certifier keeps the (lst) live intervl of ech prepred glol sutrnsction in centrlized

10 dt structure nd performs intervl intersection test etween ech prepred sutrnsction nd the one to e certified. If the 2PCA discovers non-intersecting live intervl pir, there is possily conflict etween the prepred sutrnsction nd the sutrnsction T i u to e certified. Thus, the certifiction fils nd the trnsction T i u is orted y 2PCA (the REFUSE messge follows), or the certifiction is retried lter. If there is non-empty intersection with ech prepred trnsction, the certifiction is sid to e successful nd the sutrnsction T i u lso enters the prepred stte. How does the 2PC Agent know tht sutrnsction is live? The locl sutrnsctions re regulrly checked for unilterl ort (sed on the UAN property, this is possile) nd the upper end of the live intervls re updted t the sme pce, s long s the locl sutrnsction is ctive. The upper end of the intervl is not updted while the locl sutrnsction is unilterlly orted nd is eing recovered. Commit certifiction When COMMIT messge rrives, the 2PC Agent should commit the locl sutrnsction. There is still one compliction due to the locl trnsctions. Even under DLU, they might cuse n intersite cycle to emerge into the glol seriliztion grph, if resumissions nd conflicts occur in certin wy. Exmple 3 of [6] illustrtes such cse. In this exmple, two glol trnsctions T 1 nd T 2 operte on two sites nd ut they do not conflict directly with ech other. There re lso two locl trnsction T nd T which hve conflicting opertions with oth glol trnsctions t sites nd, respectively. Thus, T 1 nd T 2 conflict indirectly [8] with ech other. In filure-free sitution possile glol seriliztion order is: T 1 < T < T 2 < T. However, If T 1 gets unilterlly orted while eing in the prepred stte (nd susequently recovered y the 2PCA), the locl seriliztion order t site could ecome T 2 < T <T 1. The result would e glol history eing neither conflict nor view equivlent to ny seril history. The commit certifiction prevents this from hppening. Generlly, the pproch is to dely the commitment of glol sutrnsction (in our cse, T 2) in cse the commitment would potentilly led to n illegl locl seriliztion order. One wy is to derive totl order of glol trnsction commitments, nd to enforce it upon the glol sutrnsctions t ech site. An ritrry glol order could, however, contrdict with the order generted dynmiclly y the locl SRS schedulers. Therefore, for ny set of directly or indirectly conflicting glol trnsctions such n order should e nlogous [15] to the glol seriliztion order estlished during norml (filure-free) opertion of the system. Once the 2PCA knows the correct order, it my utonomously decide whether prepred sutrnsction should e committed or not. If ll the sutrnsctions tht hve lower order position were lso committed, the sutrnsction in question is commit certified nd it will e immeditely loclly committed. Otherwise it will wit repeted certifiction performed t lter point of time. The prolem is how to derive the order unmiguously t ech site, nd in time for the commit certifiction to e performed. Vrious synchroniztion techniques cn e pplied. Advntge cn e tken of the fct tht, for directly or indirectly conflicting glol trnsctions, the corresponding glol sutrnsctions enter t ech site the prepred stte in the order which is nlogous to the glol seriliztion order. An optiml ordering technique eing good trdeoff of complexity nd restrictiveness is for further study.

11 Assume the order T 1 < T 2 is imposed on the glol trnsctions in the exmple ove. If the exmple ws run in lock sed system, the effect of the commit certifiction would e in creting dedlock illustrted y the following wit-for-grph [3]: T 2 > T 1 > T > T 2. The first rc represents witing for the correct commit order, imposed y the 2PCA. The other rcs represent typicl witing for locked ojects within the LTM. The dedlock hs to e resolved y orting T or T 2. Becuse neither LTM nor 2PCA cn detect the dedlock utonomously, timeout sed dedlock resolution hve to e used y LTM or 2PCA. If the Coordintors send COMMIT messges synchronously (i.e. they wit for the cknowledgement efore sending the next COMMIT messge) glol dedlock involving Coordintors my lso occur. This my e voided y the following optimiztion. The COMMIT messge is cknowledged immeditely, regrdless of the commit certifiction outcome. If the commit certifiction hd filed, PROMISED TO COMMIT record is force written into the Agent Log efore cknowledging the COMMIT messge. The record will fcilitte the sutrnsction stte recovery in cse of site filure. Also, the response time of the glol Commit commnd is improved with this implementtion. The disdvntge of the pproch is tht the dtse ojects ffected y the trnsction might not e ctully relesed efore the locl commit is eventully performed. Filure-free opertion The following is the summry of the 2PCA lgorithm in filure-free sitution: 1. Accept the Begintrnsction commnd nd record them it the Agent Log. 2. Accept the dtse lnguge commnds, pss them to the LTM nd record them in the Agent Log upon successful completion; mintin the timestmp of the lst commnd completion (per sutrnsction). 3. Accept the Prepre commnd; do the prepre certifiction; if the certifiction is successful check if the trnsction is live; if it is force write the PREPARED record in the Agent Log, updte the list of prepred trnsctions nd set the limits of the live intervl s <lst commnd timestmp,current timestmp>; if prepre certifiction fils or the sutrnsction gets unilterlly orted in the men time, respond with REFUSE, clen the Agent Log nd ort the sutrnsction, if still ctive. 4. Periodiclly check whether the prepred sutrnsction is live; if it is live updte the live intervl ccordingly, if not strt sutrnsction recovery (see next section). 5. Accept the Commit commnd; do the commit certifiction; if the certifiction is successful remove the sutrnsction from the prepre list, write the COMMIT record into the Agent Log, commit the sutrnsction in the LTM, nd cknowledge the COMMIT commnd to the Coordintor; if not successful force write the PROMISED TO COMMIT record into the Agent Log, cknowledge the commnd to the Coordintor nd repetedly retry the certifiction lter on until it is successful. 6. Repetedly clen the Agent Log y removing ll the committed trnsctions. Recovery from sutrnsction filures If there hs hppened locl sutrnsction unilterl ort then (i) in the unprepred stte: "forget" the sutrnsction, leding effectively to the glol trnsction ort;

12 (ii) in the prepred stte: 1. Resumit (in the originl order) the commnds of the glol sutrnsction from the Agent Log. 2. Once ll the opertions of the orted trnsction re gin completed, reinitilize the live intervl to just the point of time of the completion of the lst recovered opertion. 3. Resume norml opertion. Recovery from site filures A prticipting site filure my e considered mssive sutrnsction filure ecuse LTM rolls ck ll uncommitted trnsctions during the LTM-driven site recovery, including the ones tht hd een in the prepred stte efore the filure occurred. Additionlly, the voltile stte of 2PCA is lost. The Agent Log contins, however, the specifictions of ll the prepred glol sutrnsctions. The sutrnsctions tht hd een promised to commit re lso recorded in the log. We re concentrting on the sutrnsction stte recovery in the following. Consequently, we ssume the connections with the Coordintors cn e re-estlished nd the Coordintors retrnsmit messges (e.g. PREPARE nd COMMIT) infinitively if they re not responded to. The recovery sequence is s follows: 1. Restrt the LTM while disllowing locl trnsctions. 2. Clen the Agent Log y removing dt pertining to unprepred nd committed trnsctions. 3. Resumit ll the prepred sutrnsctions nd the promised to commit trnsctions. If the trnsction hs PROMISED TO COMMIT record in the Log, then commit it loclly s ove. Otherwise, initilize the live intervls. 4. Enle connections to the Coordintors nd llow for locl trnsctions. 5. Resume norml opertion. The glol sutrnsctions my e resumitted in ny order. Still, the glol history would e view serilizle. This is, in short, ecuse: (i) The locl trnsctions re prohiited to ccess the locl dtse while the 2PCA recovers. This Denied Locl Access (DLA) restriction holds during the 2PCA recovery. Clerly, DLA implies DLU. (ii) Since DLU is gurnteed to hold efore nd during the recovery, the Conflict Invrint is kept. Thus, no conflict is possile mong the glol sutrnsctions during the 2PCA recovery. Consequently, no new direct rcs etween glol trnsctions cn emerge in the glol seriliztion grph due to the recovery of 2PCA. (iii) Thnks to the DLA, no locl trnsction cn ccess the sme dt ny resumitted locl sutrnsction ccesses during the 2PCA recovery. Thus, locl trnsctions cnnot cuse new rcs to emerge into the glol seriliztion grph during 2PCA recovery. (iv) Since no new rcs cn emerge into the seriliztion grph during 2PCA recovery, no new cycles cn occur either. Thus, if the 2PCA method produces view serilizle histories during filure-free opertion nd in the presence of sutrnsction filures, then it produces them ll the time.

13 Correctness of the method We shll use the model nd the results of [3]. However, due to unilterl orts nd susequent resumissions of opertions, there might e, in our cse, n ritrry numer of locl ort opertions in glol trnsction nd they my occur fter the Coordintor hs decided to glolly commit the glol trnsction. Wht is trnsction tht is glolly committed ut tht hs loclly orted sutrnsctions? To us, it is glolly committed ut incomplete. It ecomes complete due to resumissions of opertions from the 2PC Agent log nd susequent locl commit. In [3] the committed projection of history, C(H), contins exctly ll committed trnsctions in H. We include only the glolly committed complete trnsctions into the committed projection. In ddition, our C(H) includes ll orted locl sutrnsctions tht elong to glolly committed trnsctions. We cnnot simply ignore them in C(H) ecuse locl ort cuses dt ojects to e relesed for other glol nd locl trnsctions in ny phse of execution nd this cn led to glolly nonserilizle or unrecoverle histories. A trnsction execution is modelled y mens of finite sequence of execution trees, nd the results re proven for ny execution tree. Ech individul tree is snpshot of certin phse of the execution of one trnsction nd it is contined in the tree modelling the next phse. An exmple of n execution tree of complete committed nd incomplete committed trnsction is shown in Fig. 2 nd 2, respectively. GI level T j : O 1 O1 O2 O2 C j 2PC level T j :O1 O2 Pj Cj T j : O1 O2 Pj Cj LI level T j0 :O1 O2 Cj T j0 : O1 O2 Cj EI level R j0 x W j0 x R j0 y () R j0 z W j0 z R j0 v GI level T k :O 1 O1 O2 O2 Ck 2PC level T k :O 1 O 2 Pk Ck T k :O 1 O2 Pk O1 O2 LI level T k0 : O1 O2 Ck T k0: O 1O 2A k0 T k1:o1 O 2 EI level R k0 x W k0 x R k0 y R k0 z W k0 z R k0 v R k1 z W k1 z R k1 v () Fig. 2. Illustrtion of () committed nd complete glol trnsction Tj nd () committed nd incomplete glol trnsction Tk with n orted locl sutrnsction.

14 If n opertion is listed in node of n execution tree, this indictes tht the opertion hs een completely executed t given interfce nd t the interfces elow it. The only exceptions re the glol C (Commit) nd A (Aort) opertions, one of which occurs in the root node, whenever the Coordintor hs got the glol ort or commit request from the ppliction nd recorded the decision in the Coordintor log. The lef level of the tree, i.e. the opertions of the locl (su)trnsctions, represents collection of trditionl trnsction histories [3], ech consisting of sequence of R (Red) nd W (Write) opertions. Fig. 2, for instnce, models glol trnsction T k, whose glol Commit opertion hs not een successfully cted upon yet, nd hence it is seen t the GI level ut not t the 2PC level. In the men time the originl locl sutrnsction T k0 hd ecome orted nd the 2PCA resumitted it in the form of T k1. The ltter locl sutrnsction is loclly neither committed nor orted, which mkes T k incomplete. The exmple ove shows tht we must model the locl commits nd orts seprtely from glol commits nd orts. Distinction etween glol nd locl commits nd orts is to us lso necessry due to the need to model the purely locl trnsctions seprtely from the glol ones. Further, we must mrk explicitly the prepred glol sutrnsctions with P opertions to e le to reson out the properties of the overll system. A P opertion occurs in 2PCA node of n execution tree, if nd only if the sttus of the corresponding glol sutrnsction hs een recorded permnently in the 2PCA log. Becuse of these chnges we must redefine the sic concepts like conflict or view equivlence nd the corresponding serilizility concepts in the spirit of [3]. Nottions: Tj denotes the jth trnsction, Cj the glol commit of Tj t the root, Aj the glol ort t the root. C i j denotes oth the locl commit nd the commit t the 2PCA t site i. P i j denotes the prepre opertion of the glol sutrnsction T i j t ith 2PCA. Finlly, A i jk is the locl ort of the kth locl sutrnsction of Tj t site i. R jk [X i ] (W jk [X i ]) denotes red (write) opertion, tht ccesses entity X i nd tht elongs to locl (su)trnsction T i jk. Definition 1: Histories. A trnsction history H(T j ) is the sequence of R, nd W opertions t the leves of n execution tree T j, enhnced with C j, C i j, A j, A i jk nd P i j opertions if they occur higher in node of the execution tree. The order of opertions in the trnsction history is consistent with their rel time execution order. A (glol) history H is n interleving of finite collection of trnsction histories H(T 1 ), H(T 2 ),...,H(T k ). Definition 2: Seriliztion grph SG(H). Let H e history. SG(H) is directed grph whose nodes re the glol nd locl trnsctions T h, T j,... occurring in C(H) nd there is n rc from T h to T j, j h, iff there is pir of conflicting opertions (R hk [X i ] < H W jt [X i ] or W hk [X i ] < H W jt [X i ] ) in H. Together with the existing theory nd new definitions we cn prove the needed results. We ssume tht locl systems produce conflict serilizle nd rigorous histories. At ech locl site, tht is, in ny restriction of glol history H onto ith site, H[ i ], we cn directly use the results of [3]. Our min concern is then to show tht the glol system only produces conflict serilizle histories. The result cn e shown only if there re no unilterl orts fter 2PCA hs voted READY for glol sutrnsction. If there re unilterl orts of the glol sutrnsctions in the prepred stte, the conflict serilizility cnnot e proven in generl ny more. The conflict serilizility cn still e proven for restriction of ny history H onto glol trnsctions, H[ g ], provided DLU is oeyed. Becuse the locl histories H[ i ] re rigorous

15 nd serilizle, the result mens tht t ny cse 2PCA method yields qusi serilizle histories. The cyclicity of seriliztion grphs SG(H) in generl lso mkes it questionle whether resumitting opertions mkes ny sense or under wht conditions it mkes sense. Wht re the resonle locl trnsctions? Using the DLU ssumption to restrict locl trnsctions nd ssuming tht locl systems only produce conflict serilizle nd rigorous histories it is possile to show tht ll histories emerging re view serilizle in spite of cyclicity of SG(H). This is our min result. It mkes the strtegy to use resumissions resonle ut lso illustrtes the most severe limittion of the method. Here we only sketch the results, detiled definitions nd proofs cn e found in [33]. Theorem 1: Serilizility theorem. A history H is serilizle iff SG(H) is cyclic. Proof: We cn use directly the theorem 2.1 of [3], since we cn define the concepts in similr wy. The only difference is tht the definition of conflicts ove includes site identity. We continue y showing tht using locl orting is pproprite in ny phse of glol or locl trnsction. The theorem is the sis of the implementtions tht use locl orts to resolve ny prolems like dedlocks, progrm errors, etc. It lso shows tht resumitting of opertions is sfe t lest s long s they re not committed. Theorem 2. Let H e serilizle history nd H e history tht is formed from H y dding some R, W, or A opertions into it so tht H is prefix of H. Then H is lso conflict serilizle. Proof sketch: Follows from the fct the seriliztion grph SG(H) does not chnge if neither new committed trnsctions re dded into H in H, nor new opertions re dded to trnsctions committed in H. Thus SG(H) = SG(H ) nd Theorem 1 yields the result. Unfortuntely, one cnnot include ritrry locl commit opertions into H so tht SG(H ) still remins cyclic. The Coordintors nd LTMs should, of course, commit only such trnsctions tht the cyclicity of SG(H ) is preserved. We cll trnsction simple if it is locl trnsction or if it is glol trnsction tht hs t most one locl sutrnsction t site. If no locl sutrnsction T i j0 ws unilterlly orted fter the glol sutrnsction T i j ws moved into the prepred stte t ny site i where T j hs glol sutrnsction, no resumissions re needed nd the whole glol trnsction T j remins simple (see Fig.3). The following theorem shows tht the overll system produces conflict serilizle histories s long s no unilterl ort hppens fter the ith 2PC Agent moved the locl sutrnsction to prepred stte (P i j occurs in H). Theorem 3. Let H e history. If ll glolly committed trnsctions in H re simple nd ech locl system produces only loclly serilizle nd rigorous histories then H is glolly conflict serilizle. Proof sketch: Ech locl trnsction is y definition simple nd ech glolly committed trnsction hs this property y ssumption. Thus, for ny committed completed glol trnsction there is t most one locl sutrnsction t ech site. Bsed on the rigorousness nd serilizility of the locl histories we cn show tht the glol seriliztion grph consists of cyclic sugrphs of the locl seriliztion grphs SG i (H) defined over loclly committed (su)trnsctions only. Assuming tht there is cycle in the union of the grphs, i.e. in SG(H), leds to contrdiction with respect to the order of glol commits in H (cf. [5], or p. 78 of [3]).

16 Note tht sufficient condition for the rigorousness nd serilizility of locl history H[ i ] is tht the ith locl system uses strict 2PL (see [3], p. 78). It is not, however, necessry condition. Any other method, or comintion of methods tht gurntees locl serilizility nd rigorousness, suffices. The previous two results re the strongest generl ones tht we cn chieve in the sense tht no restrictions re enforced upon trnsctions. If we drop the firly unrelistic ssumption tht unilterl orts do not occur fter glol sutrnsction T i j hs een moved to the prepred stte (i.e. fter P i j opertion in H) then the conflict serilizility of n ritrry history H cnnot e ny more proven. For this reson we restrict locl trnsctions y DLU. Even under DLU there might e cycles in the SG(H), tht is, the system does not produce conflict serilizle histories. But mye they still re view serilizle? This is indeed true under DLU. To show tht the 2PC Agent method relly mkes some sense in the presence of ritrry unilterl orts we show tht, under DLU, the glol histories re v i e w s e r i l i z l e. To proceed to this result let us first prove tht ignoring the locl trnsctions (or prohiiting them or sumitting them s glol ones) yields serilizle glol restrictions, H[ g ], of histories H onto glol trnsctions. Theorem 4. Let H e history nd the restrictions of it into locl histories, H[ i ], e loclly serilizle nd rigorous. Under 2PCA policy the restriction of H onto glol trnsctions, H[ g ], is conflict serilizle, i.e. SG(H[ g ]) is cyclic. Proof sketch: We drop the locl trnsctions from the history H nd look t the reduced glol history H[ g ]. On the sis of the properties of the 2PCA nd locl LTMs nd DLU we show tht no cycles cn emerge into the seriliztion grph SG(H[ g ]), even if resumissions occur. The reson is tht 2PC Agent mintins the Correctness Invrint nd thus prohiits such glol sutrnsctions to e moved to the prepred stte tht hve conflicting elementry opertions with the lredy prepred sutrnsctions t tht site. Consequently, when prepred sutrnsction is committed, it cnnot hve conflicting elementry dtse opertions with ny other glol sutrnsction tht ws simultneously in the prepred stte t tht site. Hence, there cnnot e cycles due to such trnsctions in SG(H[ g ]). Assuming other cycles in SG(H[ g ]) leds to the sme contrdiction s in the proof of Theorem 3. Theorem 5. If H is conflict serilizle then it is view serilizle. Proof sketch: We cn rgument exctly like in Th 2.4. of [3], in spite tht the definitions hd to e modified. Theorem 6. If locl trnsctions oey DLU nd locl systems produce loclly rigorous nd conflict serilizle histories then 2PC Agent method produces view serilizle glol histories. Proof sketch: Let H e n ritrry prefix of H nd SG(H ) e the seriliztion grph of H. If it is cyclic the clim follows from theorem 5 for it nd ll its prefixes. Let us ssume tht there is cycle in SG(H ). Our strtegy is to show tht in spite of the cycle in SG(H ), we re le to find seril history H s tht is view equivlent to C(H ). To find this we reduce C(H ) to C 1j (H ) in such wy tht ll orted locl (su)trnsctions re removed. Consequently, ll intrsite cycles mong glol sutrnsctions nd locl sutrnsctions t site, cused y resumissions, dispper from SG(C 1j (H )). The prepre certifiction gurntees tht no two sutrnsctions cn rech prepred stte t the sme time t ny site if they hve conflicting opertions in the originl locl sutrnsction. From DLU it follows tht they will neither hve conflicting opertions in their possile resumissions. Together these fcts gurntee tht the

17 Correctness Invrint holds, which in turn mens tht no cycles cn occur in the restriction of H onto the glol trnsctions, H[ g ]. The commit certifiction orders the locl commits of the locl trnsctions into the sme order t ech ech site. This prohiits intersite cycles which involve t lest two glol trnsctions nd t lest one locl trnsction t site in SG(C 1j (H )). Thus, the two-step certifier gurntees tht there re no cycles t ll in SG(C 1j (H )) nd topologicl sort of it cn e performed. By theorem 5, the reduced history C 1j (H ) is view equivlent to resulting sorted seril one, C ij (H s). After tht, we dd the removed sutrnsctions ck to oth histories to get H s nd C(H ). The conflict order of locl trnsctions nd locl committed sutrnsctions cnnot e different in the four histories due to locl serilizility nd rigorousness. Bsed on this we show tht the glol reds-form reltion remins identicl in C(H ) nd H s, since it is identicl with tht of the reduced histories. The rgumenttion is sed on DLU. The finl writes lso remin the sme in oth histories s ech dded write ecomes lter orted nd thus cnnot chnge the finl vlues. The previous result shows tht forwrd recovery (resumissions) used y the 2PCA is resonle under DLU. It llows glolly written dt to e red y locl trnsctions ut not updted while glol sutrnsction is in the prepred stte. Let us look t Fig. 2. It is esy to show tht if we did not restrict in ny wy the locl trnsctions, then the view serilizility cnnot e proven for histories with cyclic SG(H). The reson is tht the resumitted opertions, e.g. the locl sutrnsction T k1 in Fig 3, might red other vlues thn the first sumission, T k0, did since n intervening locl trnsction, sy T l, would hve een le to updte locl oject etween T k0 nd T k1. Therefore, T k1 gets nother view thn T k0, nd consequently T k1 might lso updte the locl dtse in wy tht differs from tht of T k0. Remining prolems We did not ddress here the prolem of when does trnsction run to end. As ws shown in [16] glol nd locl trnsctions might dedlock so tht neither glol nor locl dedlock detection is possile. Theorem 2 ove shows tht it is sfe to ort ny trnsction on the sis of time-out or some other criteri y the 2PC Agent or y the locl system t ny time even if the glol sutrnsction ws in the prepred stte nd lredy glolly committed. It is especilly importnt to understnd tht lso site filures led to such locl ort, s sideeffect. Thus, sufficient condition for hidden dedlocks [16] to e resolved is tht the sites fil from time to time or tht 2PC Agent orts locl sutrnsctions tht wit too long. To show tht glol nd locl trnsctions relly run into completion we hve to show further tht no ritrry mny livelocks occur. Actully, TW ssumption gurntees this. The occsionl site filures lso suffice to gurntee tht Coordintor filures do not lock locl or glol trnsctions forever. The overll issue of termintion is studied more closely in [33]. We did not ddress here the prolems of optimizing the certifier. There re siclly three possiilities. The certifier could store ll live intervls of sutrnsction insted of only the lst one nd use ny of them. Another ide is tht the prepre certifiction cn e retried severl times if it fils. Under which condition this mkes sense is for further study. Third possiility is to check explicitly conflicts etween opertions of the sutrnsctions if the prepre certifiction fils etween the prepred sutrnsction nd tht to e certified. This optimiztion functions correctly only if the locl conflicts cn relly e deduced y looking only t the commnds known to the 2PCA. However, usge of triggers t the locl dtse mke this ssump-