Comparing Hierarchical Data in External Memory

Transcription

1 Compring Hierrhil Dt in Externl Memory Surshn S. Chwthe Deprtment of Computer Siene University of Mryln College Prk, MD 090 Astrt We present n externl-memory lgorithm for omputing minimum-ost eit sript etween two roote, orere, lele trees. The I/O, RAM, n CPU osts of our lgorithm re, respetively, 5, 6, n 5, where n re the input tree sizes, is the lok size,, n. This lgorithm n mke effetive use of surplus RAM pity to qurtilly reue I/O ost. We exten to trees the ommonly use mpping from sequene omprison prolems to shortest-pth prolems in eit grphs. Introution We stuy the prolem of ompring snpshots of t to etet similrities n ifferenes etween them. Suh ifferening of t hs pplitions in version ontrol, inrementl view mintenne, t wrehousing, stning queries (susriptions), n hnge mngement [Ti85, LGM96, CAW98]. The RCS version ontrol system [Ti85] uses the iff progrm [MM85] to ompute n store only the ifferenes etween the new n ol versions of t tht is heke in. As nother version ontrol pplition, onsier the proess use to merge two ivergent versions of progrm or oument (e.g., the eiff/emerge funtion in Ems). The first step onsists of ompring the files ontining the two versions to etermine where n how they iffer. These ifferenes re then presente using grphil interfe tht llows user to etermine whih vrint to keep in the merge file. Permission to opy without fee ll or prt of this mteril is grnte provie tht the opies re not me or istriute for iret ommeril vntge, the VLDB opyright notie n the title of the pulition n its te pper, n notie is given tht opying is y permission of the Very Lrge Dt Bse Enowment. To opy otherwise, or to repulish, requires fee n/or speil permission from the Enowment. Proeeings of the 5th VLDB Conferene, Einurgh, Sotln, 999. Differening lgorithms lso ply key role in hnge mngement systems suh s [CAW98]. Sine mny tses, espeilly those on the We, o not offer hnge notifition filities, hnges must e etete y ompring ol n new results of query. One hnges hve een etete in this mnner, uses them to implement stning queries se on the urrent stte s well s the history of the tses eing monitore. We n lso use ifferening lgorithms to reue the mount of t trnsmitte over network in mirroring pplitions. Populr We n FTP servers often hve ozens of mirror sites roun the worl. Chnges me to the mster server nee to e propgte to the mirror sites. Ielly, the persons or progrms mking hnges woul keep reor of extly wht t ws upte. However, in prtie, ue to the utonomous n loosely orgnize nture of suh sites, there is no relile reor of hnges. Further, even if suh reor is ville, it my e se on version tht is ifferent from the version urrently t ertin mirror site. Due to suh iffiulties, effiient mirroring requires ifferening lgorithms tht ompute n propgte only the ifferene etween the version t the mster server n tht t mirror site. Similr ies enle ifferening lgorithms to improve effiieny in t wrehousing environment [LGM96]. Differening lgorithms re lso use to fin, mrk-up, n rowse hnges etween two or more versions of oument [CRGMW96, CGM9]. Suppose we reeive n upte version of n online mnul. Agin, in the iel se the new version woul highlight the wy it iffers from the ol one. However, for resons similr to those stte ove, in prtie we often nee to etet the ifferenes ourselves y ompring the two versions. For exmple, [CAW98] esries experienes in eteting n rowsing ifferenes etween ifferent versions of resturnt review tse on the We, while [Yn9] esries the implementtion of n pplition tht highlights ifferenes etween progrm versions. There is sustntil oy of prior work on ifferening lgorithms. The min istinguishing fetures of the work in this pper re the following. (See Setion 6 for etile isussion.) 90

2 We stuy lgorithms for omputing ifferenes etween snpshots of hierrhilly struture t, moele using roote, orere, lele trees. Our moel llows us to urtely pture the hierrhil struture inherent in t suh s soure oe, ojet lss hierrhies, struture ouments, HTML, XML, n SGML. For exmple, n online mnul typilly hs well-efine hierrhil struture onsisting of hpters, setions, susetions, prgrphs, n sentenes. Algorithms tht tke this struture into ount proue results tht re more meningful thn those tht tret their inputs s flt strings. While the prolem of ifferening strings n sequenes hs een thoroughly stuie n mits severl effiient solutions, the prolem of ifferening trees remins hllenging. Severl formultions of this prolem re NP-hr [ZWS95]. In this pper, we stuy simple vrition tht mits effiient solutions. We o not ssume tht the snpshots eing ifferene re smll enough to fit entirely in min memory (RAM); inste, they resie in externl memory (isk). For exmple, online mnuls for omplex mhinery, irrfts, n sumrines re tens or hunres of gigytes in size, mking it imprtile to use min-memory ifferening lgorithms to ompre their versions. When t resies in externl memory, the numer of input-output opertions (I/Os), n not the numer of CPU yles, is the primry eterminnt of running time. Therefore, externl-memory lgorithms use tehniques tht try to minimize the numer of I/Os. A seonry ut importnt onsiertion is the mount of uffer spe require in RAM. See [Vit98] for n overview of externl memory lgorithms. In this pper, we nlyze lgorithms se on their I/O, RAM, n CPU osts. As n illustrtion of the importne of using n lgorithm tht is ogniznt of the hierrhil struture of t, onsier the following exmple from [Yn9]. Figure epits epits frgments of two progrm versions tht re eing ompre. A sequene omprison progrm suh s the one in[mm85] ompres the inputs line-y-line n my result in mthing progrm text s suggeste y the soli lines in the figure. Given the neste struture of the progrm frgment, it is lerly more meningful to mth the inputs s suggeste y the she lines in the figure. However, the efinition of optimlity use y most sequene omprison lgorithms (se on longest ommon susequene) onsiers the solution epite using soli lines more esirle [Mye86]. By moeling the hierrhil struture of progrms, tree ifferening lgorithms re le to proue more meningful results. We now present rief, informl efinition of the ifferening prolem we stuy in this pper. (See Setion for etils.) A roote, orere, lele tree is tree in whih eh noe hs lel n in whih the orer mongst while(p) { x = y + z; = + ; } while(p) { x = y + z; } while(p) { = + ; } Figure : Importne of hierrhil struture silings is signifint. (The lel of noe intuitively represents the t ontent t tht noe; it is not unique key or ojet ientifier.) Trees n e trnsforme using three eit opertions: () We n insert new lef noe t speifie lotion in the tree. () We n elete n existing lef noe. () We n upte the lel of noe. Note tht the restrition tht () n () operte only on lef noes mens tht to elete n interior noe, we must first elete ll its esennts; similrly, we must insert noe efore inserting ny of its esennts. An eit sript is sequene of eit opertions tht re pplie in the orer liste. We ssoite ost with eh eit opertion n efine the ost of n eit sript to e the sum of the osts of its omponent opertions. Prolem Sttement (informl): Given two roote, orere, lele trees n, fin minimum-ost eit sript tht trnsforms to. Given trees n, we n trnsform one to the other using ny of n infinite numer of eit sripts. (For exmple, given ny eit sript tht trnsforms to, we n ppen to it opertions tht insert n immeitely elete noe, thus generting n infinite numer of eit sripts.) This ft motivtes the minimum-ost requirement in the prolem efinition. Another motivtion for the minimumost requirement is the following: Given two trees tht iffer only in one noe lel, the intuitively esirle eit sript is one tht ontins single upte opertion. We nee to wee out eit sripts tht unneessrily insert, elete, n upte noes. An eit sript of minimum ost nnot ontin suh reunnt or wsteful eit opertions (sine we n otin nother eit sript with lower ost y getting ri of the reunnies n ineffiienies). The two min ontriutions of this pper my e summrize s follows: We present n effiient externl-memory lgorithm for omputing the ifferene (minimum-ost eit sript) etween two snpshots of hierrhil t (trees). The I/O, RAM, n CPU osts of our lgorithm re, respetively, 5, 6, n, where n re the input tree sizes, is the lok size,, n. To our knowlege, this lgorithm is the first externl-memory ifferening lgorithm (for sequenes or trees). The I/O omplexity of our lgorithm is optiml over wie lss of omputtion moels ue to the lower oun for the sequene omprison prolem (whih 9

3 is simple speil se of our tree omprison prolem) [AHU6, WC6]. We reue our tree omprison prolem to shortest pth prolem in well-stuie grph lle the eit grph. This reution opens the oor for generlizing to trees severl effiient sequene omprison lgorithms tht re se on eit grphs (e.g., [MM85, Mye86, WMG90]). Outline of the pper: In Setion, we esrie our moel of trees, eit opertions, n eit sripts, followe y the forml prolem sttement. Setion riefly esries eit grphs s they re use for sequene omprison n then presents our moifitions tht llow them to e use for tree omprison. In Setion, we present simple minmemory lgorithm for our tree omprison prolem. This lgorithm illustrtes the use of our eit grphs n serves s sis for our externl-memory lgorithm. Setion 5 first explores nive extension of our min-memory lgorithm for externl memory n then presents our min lgorithm, xmiff. Relte work is isusse in Setion 6, followe y the onlusion in Setion. Moel n Prolem Sttement Hierrhil t is nturlly moele y trees. In this pper, we fous on roote, orere, lele trees. Eh noe in suh tree hs lel ssoite with it. Informlly, we n think of the lel of noe s its t vlue. The hilren of noe re totlly orere; thus, if noe hs hilren, we n uniquely ientify the th hil, for. There is istinguishe noe lle the root of the tree. (This feture istinguishes these trees from yli grphs, whih re lso lle free trees.) Formlly, roote, orere, lele tree onsists of finite, nonempty set of noes n leling funtion suh tht: () The set ontins istinguishe noe, lle the root of the tree; () The set is prtitione into isjoint sets, where eh is tree (lle the th sutree of or ); n () the lel of noe is [Sel]. The root of is lle the th hil of the noe, n is lle the prent of. Noes in tht o not hve ny hilren re lle lef noes; the rest of the noes re lle interior noes. In the rest of this pper, we use the term trees to men roote, orere, lele trees. Figure epits severl suh trees. The letter next to noe suggests its lel. The numer next to noe is noe ientifier. Note tht when we re given two trees to ompre, there is, in generl, no orresponene etween the noe ientifiers. (In ft, omputing suh orresponene is equivlent to omputing minimum-ost eit sript for our formultion of the prolem.) For exmple, when we re ompring two versions of mnul, the noe ientifiers in Figure my represent offsets within the SGML soure files for the mnuls. Sine the soure files for the mnuls re seprte, there is no orresponene etween these offsets. up(,) el(8) 5 6 el(9) 5 6 el() ins(8,,,) ins(,,,) ins(9,,,) 0 up(,) 0 Figure : Eit opertions on trees 9

4 We moel hnges to trees using the following tree eit opertions: Insertion Let e noe in tree, n let e the sutrees of. Let e noe not in, let e n ritrry lel, n let. The insertion opertion ins inserts the noe s the th hil of. In the trnsforme tree, is lef noe with lel. Deletion Let e lef noe in. The eletion opertion el results in removing the noe from. Tht is, if is the th hil of noe with hilren, then in the trnsforme tree, hs hilren. Upte If is noe in n is lel then the lel upte opertion up results in tree tht is ientil to exept tht in,. We ssume, without loss of generlity, tht the root of tree nnot e elete or inserte. An eit sriptis sequeneof eit opertions. Theresult of pplying n eit sript to tree is the tree otine y pplying the omponent eit opertions to, in the orer they pper in the sript. Consier Figure. The ownwr rrows illustrte the pplition of the following eit sript: el 8, el 9, up, el. Similrly, the upwr rrows illustrte the pplition of nother eit sript. As isusse in Setion, we formlize the esirility of eit sripts tht perform s few n s smll hnges s possile y efining ost moel for eit sripts. Let n e ritrry funtions return positive numer representing the osts of, respetively, inserting n eleting noe. Similrly, the ost of upting lel to is given y. The ost of n eit sript is the sum of the osts of its omponent opertions. We n now formlly efine the prolem of ifferening trees s follows: Prolem Sttement: Given two roote, lele, orere trees n, fin minimum-ost eit sript tht trnsforms to tree tht is isomorphi to. Eit Grphs In this setion, we introue n uxiliry struture, lle n eit grph, tht we lter use in our ifferening lgorithms. Eit grphs hve een use y severl effiient lgorithms for ompring sequenes (equivlently, strings) [Mye86, MM85, WMG90]. In effet, the prolem of fining minimum-ost eit sript etween two sequenes is reue to the prolem of fining shortest pth from one en of the eit grph to the other. Sine n eit grph hs very simple n regulr struture, this shortest pth prolem n typilly e solve very effiiently. Below, we first explin how eit grphs re use for sequene omprison n then introue our moifitions tht permit them to e use for ompring trees. The eit grph of two sequenes n is the gri suggeste y Figure. (Eh point where two lines touh or ross is noe in the eit grph.) A point intuitively orrespons to the pir, for n. In our eit grphs, the origin 0 0 is the noe in the top left orner; the -xis extens to the right of 0 0 n the -xis extens own from 0 0. There is irete ege from eh noe to the noe, if ny, to its right. Similrly, there is irete ege from eh noe to the noe, if ny, elow it. For lrity, these irete eges re shown without rrowhes in the figure. All horizontl eges re irete to the right n ll vertil eges re irete own. In ition, there is igonl ege from to for ll 0. For lrity, these eges re omitte in the figure. The eit grph epite in Figure orrespons to the sequenes (strings) n Figure : Eit grph for sequene omprison Trversing horizontl ege in the eit grph orrespons to eleting. Similrly, trversing vertil ege orrespons to inserting. Trversing igonl ege orrespons to mthing to ; if n iffer, suh mthing orrespons to n upte opertion. Eges in the eit grph hve weights equl to the osts of the eit opertions they represent. Thus, horizontl ege hs weight, vertil ege hs weight, n igonl ege hs weight. It is esy to show tht ny min-ost eit sript tht trnsforms to n e mppe to pth from 0 0 to in the eit grph. Conversely, every pth from 0 0 to orrespons to n eit sript tht trnsforms to. For etils, see [Mye86]. For the exmple suggeste in Figure, the highlighte pth orrespons to the following eit sript: el ins el 5 el 6 9

5 up el el 8 el 0 ins 0 el el ins It is esy to verify tht pplying the ove sript to proues. In orer to use eit grphs to ompre trees, we nee to moify their efinition to inorporte the onstrints impose y the struture of the trees eing ompre. For exmple, we must moel the onstrint tht if n interior noe is elete then ll its esennts must lso e elete. Before we proee, we nee to efine the l-pir representtion of tree. We efine the l-pir of tree noe to e the pir, where is the noe s lel n is its epth in the tree. We use n to refer to, respetively, the lel n epth of n l-pir. The l-pir representtion of tree is the list, in preorer, of the l-pirs of its noes. In the rest of this pper, we ssume tht trees re in the l-pir representtion. (A tree n e onverte to this representtion using single preorer trversl.) Figure : Input tree Figure 5: Input tree 9 0 Consier the trees n epite in Figures n 5. The letter next to eh noe suggests its lel n the numer next to noe is its preorer rnk, whih lso serves s its ientifier. The l-pir representtions of n re s follows: 0 0 Given tree (in l-pir representtion), we use the nottion to refer to the th noe of tree. Thus, n enote, respetively, the lel n the epth of the th noe of. We efine the eit grph of two trees n to onsist of gri of noes s suggeste y Figure 6. There is noe t eh lotion for 0 n 0. These noes re onnete y irete eges s follows: 0 For 0 n 0, there is igonl ege if n only if. (For lrity, these eges re omitte in Figure 6.) For 0 n 0, there is horizontl ege unless n. For 0 n 0, there is vertil ege unless n. l l Figure 6: Eit grph for tree omprison As ws the se for sequene eit grphs, horizontl eges represent eletions, vertil eges represent insertions, n igonl eges represent mthing of noes (with n upte opertion require if the lels of the mthe noes iffer). Similrly, eh ege hs n ege weight equl to the ost of the orresponing eit opertion. However, in ontrst to sequene eit grphs, severl horizontl n vertil eges re missing from tree eit grphs. Intuitively, the missing vertil eges ensure tht one pth in the eit grph trverses n ege signifying the eletion of noe, tht pth n only ontinue y trversing eges tht signify the eletion of ll the noes in s sutree. Similrly, the missing horizontl eges ensure tht ny pth tht trverses n ege signifying the insertion of noe n only ontinue y trversing the eges signifying insertion of ll noes in s sutree. We n show tht ny min-ost eit sript tht trnsforms to n e mppe to pth from 0 0 to in the tree eit grph; onversely, every pth from 0 0 to orrespons to n eit sript tht trnsforms to. 9

6 For exmple, the pth inite using ol lines in Figure 6 orrespons to the following eit sript, where re ritrry ientifiers for the newly inserte noes: up up 5 el 6 el el 8 el 9 el ins 0 ins ins ins Differening in Min Memory In Setion, we reue the prolem of omputing minost eit sript etween two trees to the prolem of fining shortest pth in the eit grph of those trees. We now use tht reution to present min-memory lgorithm for ifferening trees. Given eit grph, let e mtrix suh tht is the length of shortest pth from 0 0 to in the eit grph. We ll the istne mtrix for. For nottionl onveniene, let us efine the weight of n ege tht is missing from the eit grph to e infinity. Consier ny pth tht onnets the origin 0 0 to in the eit grph. Given the grph s struture, the previous noe on this pth is either the noe to the left of, the noe ove or the noe igonlly to the left n ove. Therefore, the istne of from the origin nnot e greter thn tht istne for the noe to its left plus the weight of the ege onneting the left neighor to. Similr reltions hol for the top n igonl neighors of, yieling the following reurrene for, where 0 n 0 : min where if otherwise if otherwise if otherwise This reurrene les to the following ynmi-progrmming lgorithm for omputing the istne mtrix. We ll this lgorithm mmiff, for min-memory ifferening. Algorithm mmiff Input: Arrys n, whih represent two trees in l-pir representtion. Thus n enote, respetively, the lel n epth of the th noe (in preorer) of (n nlogously for ). The numer of elements in n is n, respetively. Output: The istne mtrix where equls the length of the shortest pth from 0 0 to in the eit grph of n. Metho: Figure presents the pseuooe for Algorithm mmiff. All the pseuooe in this pper ssumes shortiruit evlution of Boolen expressions. We initilize t the origin 0 0, followe y omputtion of istnes long the top n left ege of the mtrix. The neste for loop is iret implementtion of the reurrene for. If we ssume tht the funtions,, n exeute in onstnt times,, n, respetively, the running time of mmiff is proportionl to, or. 0 0 : 0; for : to o 0 : 0 ; for : to o 0 : 0 ; for : to o for : to o egin : ; : ; : ; if ( ) then : ; if ( or ) then : ; if ( or ) then : ; : min ; Figure : Algorithm mmiff One we hve ompute the istne mtrix, it is esy to reover from it shortest pth from 0 0 to. We strt t n tre the reurrene reltion for kwrs. As we trverse horizontl, vertil, n igonl eges, we emit the pproprite eletion, insertion, n upte opertion, respetively. Thus, we hve the following lgorithm for reovering n eit sript from the istne mtrix. We ll this lgorithm mmiff-r (for mmiff-reovery). Algorithm mmiff-r Input: Arrys n representing the input trees (s in Algorithm mmiff) n the istne mtrix ompute y mmiff. Output: A min-ost eit sript tht trnsforms to. (For simpliity of presenttion, the insertion n eletion opertions only ientify the noes eing inserte n elete. Using the informtion from the originl trees, it is esy to generte n eit sript in the syntx of Setion. Further, insertions of interior noes re printe fter the insertions of their esennts; this orering is esily fixe.) Metho: Figure 8 presents the pseuooe for Algorithm mmiff-r. At eh itertion of the while loop n/or is eremente y one. Thus there re etween mx n itertions of the while loop, with eh itertion performing onstnt mount of work. Thus, the running time of mmiff-r is. 95

7 : ; : ; while ( 0 n 0) o if ( n ( or )) then egin print( el ); : ; else if ( n ( or )) then egin print( ins ); : ; else egin if( ) then print( up ); : ; : ; while ( 0) o egin print( el ); : ; while ( 0) o egin print( ins ); : ; Figure 8: Algorithm mmiff-r 5 Differening in Externl Memory In Setion, we presente lgorithms mmiff n mmiff-r to ompute minimum-ost eit istne etween two trees in min memory. These lgorithms require RAM spe for not only the input trees n, ut lso for the istne mtrix. If n hve sizes n, respetively, then the istne mtrix is of size, whih n e prohiitively lrge for even moestly size inputs tht fit in RAM. In situtions where n themselves re too lrge to fit in RAM, the prolem is muh worse. Let us first onsier nive extension of Algorithm mmiff for externl memory. We use two uffers, n, to re, s neee, the trees n, respetively, from isk into RAM. The uffers re extly lrge enough to hol one isk lok (of size ). As the istne mtrix is ompute, we write the istnes to thir uffer, lso of size. When is ompletely fille, we write it out to isk n overwrite it in RAM. Using similr uffering sheme, we n pt the lgorithm mmiff-r, use for reovering the eit sript, for externl memory. Let us ll these uffere version of mmiff n mmiff-r Algorithm miff n Algorithm miff-r, respetively. Sine Algorithm miff omputes the istne mtrix in olumn-mjor orer, eh of the loks of is re one for eh of the noes in, inurring n I/O ost of. On the other hn, eh of the loks of is re extly one, for n I/O ost of. Algorithm miff lso stores the entire istne mtrix, of size to isk, inurring n I/O ost of. Thus the totl I/O ost of lgorithm miff is, or pproximtely. In ition to spe for temporry vriles Algorithm miff nees spe for only the three uffers,, n, of size eh. Thus the RAM ost of Algorithm miff is. Other thn opertions require to re n write isk loks, Algorithm miff performs the sme opertions s Algorithm mmiff. Thus its CPU ost is. Thus, we my summrize Algorithm miff s performne s follows: I/O RAM CPU By using tehniques similr to those use for omputing neste-loop joins in reltionl tses, it is esy to reue the I/O ost of reing n from to. Intuitively, inste of omputing the istne mtrix in olumn-mjor orer, we ompute it in loke mnner: When we re in the th lok of n the th lok of, we ompute the entire sumtrix :. However, it is not ovious how we n improve on the I/O ost of storing the istne sumtrix itself, whih is of size loks. As esrie elow, our pproh is to voi storing ll of the istne mtrix, storing inste only orse gri from whih the rest of the istne mtrix n e quikly ompute. 5. Computing the Distne Mtrix Consier the istne mtrix suggeste y the rrngement of ots in Figure 9, orrsponing to input trees n of sizes n, respetively. As suggeste in the figure, we ivie into set of overlpping tiles. For lrity, the figure lterntes the use of soli n she lines for mrking the tiles. The overlp etween neighoring tiles is one item wie. Figure 9: Tiling the istne mtrix More preisely, let. For simpliity in presenttion, we shll heneforth ssume tht n re oth integrl multiples of, with n. The istne mtrix is then ivie into 96

8 tiles li out in olumns n rows. We numer these rows n olumns of tiles strting with 0, n use the nottion to enote the tile in the th olumn (of tiles) n th row(of tiles). Thus, the tile onsists of the following sumtrix of the istne mtrix : :. Our externl memory lgorithm, lle xmiff (for externl memory ifferening), is se on the following three key ies: First, inste of storing the entire istne mtrix, we store only the top n left eges of eh of the tiles esrie ove. We ll this gri onsisting of the tile eges the istne gri. Eh tile ege is of size, n there re tiles in ll. Thus the entire istne gri n e store using only loks of storge. Referring k to Figure 9, only those entries in the the mtrix tht re rosse y otte line re store on isk. (Note tht for lrity, Figure 9 ssumes tht the imension of eh tile is only items, orresponing to. For suh low vlues of, our tehnique oes not pper to yiel muh svings. However, for typil vlues of tht lie in the hunres, the svings re sustntil.) Seon, given the top n left eges of tile, we n ompute the istne sumtrix for the rest of the tile y using the reurrene for in Setion. Further, using few temporry vriles, this reurrene llows us to upte in-ple n rry tht initilly ontins the istnes for the top (left) ege of tile to one ontining the istnes for the ottom (respetively, right) ege. Thir, given the istne gri store on isk, shortest pth in the eit grph from 0 0 to (n thus the orresponing min-ost eit sript tht trnsforms tree to tree ) n e ompute in liner time y working kwrs from, s one in Algorithm mmiffr. A strightforwr lgorithm se on this ie requires RAM s working store. However, we lso esrie n enhne version of the lgorithm tht requires only RAM. We ssume tht the first lok of hs ummy first noe, 0 0. Eh suessive lok hs s its first noe opy of the lst noe from the previous lok. The loks of re lso ssume to e in this formt. We lso ssume tht the input sizes n re oth integrl multiples of, where is the lok size. These ssumptions re not neessry for our lgorithm; we mke them only to simplify the following presenttion of the lgorithm. Rell, from Setion, tht we represent tree using the preorer listing of the l-pirs of its noes. These l-pir sequenes re pke ensely into isk loks when the trees re represente on isk. Noes within lok re rrnge t fixe offsets eginning with 0. Disk loks re re n written using the funtion R- Blk n the proeure WrBlk, respetively. RBlk tkes s rguments file nme (whih we use to group relte loks) n lok numer. Blok numers for eh file re seprte, n re numere sequentilly eginning with 0. For exmple, the th lok of file lle is enote y n is re using RBlk. The nottion for WrBlk is nlogous to tht use for RBlk: WrBlk(A,i) writes the uffer to the th lok of file. In ition to one imensionl inexing of isk loks, we lso use two-imensionl inexing of the form where the rnges of n re fixe n known in vne. Suh oule inexing of loks is only nottionl onveniene. Using stnr row-mjor enoing of two-imensionl mtries, is equivlent to where is the numer of ifferent vlues. Thus eh RBlk n WrBlk opertion requires only single I/O. Algorithm xmiff Input: Arrys n representing the input trees, ontining n elements, respetively (s in Algorithm mmiff). Output: The istne gri (efine ove), giving the length of the shortest pth from 0 0 to in the eit grph of n for ll eit grph noes tht lie on the eges of tiles of imension. Metho: Figure 0 lists the pseuooe for Algorithm xmiff. The first prt of the lgorithm onsists of two for loops whih orrespon to the first two for loops of Algorithm mmiff in Figure. The first for loop in Figure 0 omputes n writes to isk the top row of the istne mtrix. Similrly, the seon for loop omputes n writes to isk the leftmost olumn of the istne mtrix. The seon prt of Algorithm xmiff (the thir for loop in Figure 0) orrespons to the neste for loops of Algorithm mmiff in Figure. Eh itertion of the inner loop omputes the istnes in one tile (sumtrix) of the istne mtrix. Rell tht we ientify tile y its tile olumn n tile row numers, with numering strting t 0. The innermost loop omputes istnes in tile. This omputtion uses, in ition to the pproprite loks of the input trees n, the istnes for noes on the top n left eges of the tile. Due to the overlp of tiles, these eges re the ottom n right eges of some other (previously ompute) tiles. The istnes for noes on the ottom n right eges of the tile re then written to isk. By refully orering the istne lultions n y using temporry vriles ( n ), we re le to upte in ple the rry, whih initilly ontins the istnes for noes on the top ege, to the istnes for noes in the ottom ege. Similrly, the rry initilly ontining the istnes of noes in the left ege is upte in ple to the istnes of noes in the right ege. The tests in the if sttements re nlogous to those in Algorithm mmiff. Anlysis Algorithm xmiff uses only four uffers in RAM:,,, n, eh of size. Thus, the RAM storge requirements re only (in ition to the smll, onstnt mount of storge neee for progrm oe n slr vriles). Let,, n. The first two for loops of the lgorithm mke totl of lls to the eh of the proeures RBlk n WrBlk, giving s the I/O ost. In the neste for loops, the first 9

9 : ; for : 0 to o egin : RBlk ; if( 0) then 0 : ; else 0 : 0; for : to o : ; WrBlk 0 ; for : 0 to o egin : RBlk ; if( 0) then 0 : ; else 0 : 0; for : to o : ; WrBlk 0 ; for : 0 to o egin : RBlk ; : RBlk 0 ; for : 0 to o egin : RBlk ; : RBlk ; 0 : ; for : to o egin : 0 ; 0 : ; for : to o egin : ; : ; : ; if ( ) then : ; if ( or ) then : ; if ( or ) then : ; : ; : min ; : ; WrBlk ; WrBlk ; Figure 0: Algorithm xmiff two RBlk sttements (for n ) re exeute times. The next two RBlk sttements (for n ) re exeute times. Finlly, the WrBlk sttements re exeute times. Thus, the totl numer of I/Os in the neste loops of xmiff is. Comining this numer with tht for the first two for loops, we onlue tht xmiff mkes I/Os. Finlly, it is esy to oserve tht the CPU time is. Thus, we n summrize Algorithm xmiff s performne s follows, where n re the sizes of the input trees, is the lok size,, n : I/O RAM CPU 5. Reovering the Eit Sript Rell, from Figure 8, the Algorithm mmiff use to reover minimum-ost eit sript in RAM. Using the istne mtrix s guie, Algorithm mmiff-r trverses the shortest pth from 0 0 to kwrs, emitting pproprite eit opertions long the wy. Unlike mmiff, lgorithm xmiff oes not store the entire istne mtrix, mking iret pplition of the pth-reovery lgorithm mmiff-r impossile. However, for eh tile of the eit grph, xmiff stores the istnes for noes on its top n left eges in the isk loks n, respetively. By reing in these loks n using lgorithm mmiff, the istne mtrix for tile n esily e ompute t CPU ost (where is the lok-size). Bse on these ies, we hve the following: Algorithm xmiff-r Input: Arrys n representing the input trees (s in Algorithm mmiff) n the istne gri ompute y xmiff, store on isk s n. Output: A min-ost eit sript tht trnsforms to (s in Algorithm mmiff-r). Metho: Figure presents the pseuooe for xmiff-r. As in lgorithm mmiff-r, we egin t the noe of the eit grph n move kwrs long the shortest pth from 0 0 to. At eh itertion of the outer while loop, the urrent position in the eit grph is move k to either,, or using the if-then-else sttement tht is very similr to tht in lgorithm mmiff-r. Using the oolen vriles n, we etet the sitution when the urrent position first moves into new tile in the horizontl n vertil iretion, respetively. When is nonzero, hs just move to new tile to the left of the ol tile. In this se, we re in the orresponing lok of the input file. Similrly, when is nonzero, we re in the pproprite lok of the input file. In oth ses, we re in the top n left eges of the istne mtrix for the new tile,, into the rrys n, respetively. We use n to initilize the top n left eges of the omplete istne mtrix for the new tile. Reomputtion of the 98

10 istnes in the rest of the mtrix is one in mnner ompletely nlogous to lgorithm mmiff. Anlysis Eh itertion of the outer while loop erements t lest one of n ; thus, there re t most itertions of tht loop. Sine rnges from own to, it follows tht is set to for itertions. Similrly, is set to for itertions. Bloks from n re re in whenever one of n is nonzero. In the worst se, this sitution n our times. Thus the I/O ost ue to reing n is t most. It is esy to oserve tht eh of the loks of the input is re extly one, for n I/O ost of. Similrly, the I/O ost inurre in reing the input is. Thus, the totl I/O ost of lgorithm xmiff-r is. The signifint RAM storge requirements of xmiff-r re ue to the rrys,,, n, n the istne mtrix. The rrys,,, n re ll of size, the lok size. Unfortuntely, the istne mtrix is of size. Thus, the totl RAM ost is. It is esy to oserve tht the CPU ost of lgorithm xmiff-r is. Thus, Algorithm xmiff-r s performne my e summrize s follows: I/O RAM CPU Reuing the RAM ost of xmiff-r We n reue the RAM spe neee to store the istne mtrix s follows. We ivie the istne mtrix into sutiles of size s suggeste y Figure. Consier the proess of tring the shortest pth kwrs through. We ivie this tsk into stges orresponing to trversing the sutiles. We egin with the sutile in the lower right orner. At the en of eh stge, we move to sutile tht is ove n/or to the left of the urrent sutile. At the eginning of the stge orresponing to sutile, we ompute (s efore) the istne mtrix, ut store istnes for only those points tht lie in this sutile. Using tehnique similr to tht use y lgorithm mmiff, this omputtion n e performe using only uffer of size s working storge. Sine the size of sutile is, the totl storge require for the sutile omputtion is. Comine with the spe require to store,,, n, we hve totl RAM ost of 6. The sutile-se enhnement esrie ove results in the istne mtrix for eh tile eing reompute severl times, thus inresing the CPU ost. After eh reomputtion, the urrent sutile moves one position to the left n/or up. Thus, the numer of sutile omputtion stges is etween n (sine there re sutiles long eh imension of the istne mtrix). It follows tht the totl CPU ost is therefore inrese y ftor of t most, to 5. : ; : ; : ; : ; : ; while ( 0 n 0) o egin : ; : ; : mo ; : mo ; if ( 0) then : ; if ( 0) then : ; if ( 0 or 0) then egin : ; : ; : 0; : 0; 0 0 : ; 0 0 : ; for : to o for : to o egin : ; : ; : ; if ( ) then : ; if ( or ) then : ; if ( or ) then : ; : min ; if ( n ( or )) then egin : ; if ( mo 0) then : ; print( el ); else if ( ) n ( or )) then egin : ; if ( mo 0) then : ; print( ins ); else egin : ; : ; if ( mo 0) then : ; if ( mo 0) then : ; if( ) then print( up ); while ( 0) o egin print( el ); if( mo 0) then : ; : ; while ( 0) o egin print( ins ); if( mo 0) then : ; : ; Figure : Algorithm xmiff-r 99

11 S S S / S / Figure : Reuing mmiff-r s RAM ost Thus, the performne of the moifie Algorithm xmiff-r my e summrize s follows: I/O RAM CPU 6 5 Thus, the osts of exeuting Algorithm xmiff followe y Algorithm xmiff-r re s follows: I/O RAM CPU Although we hve esrie to e the lok size, our lgorithm oes not epen on eing equl to the lok trnsfer unit. Another wy to interpret the ove result is the following: Suppose we hve units of RAM tht we n use for our lgorithm s uffers. We set 6 ; tht is, 6. Sustituting for in our performne results tells us tht y inresing the mount of RAM uffer spe, we n hieve qurti reution in the signifint prt of the I/O ost. 6 Relte Work Differening lgorithms hve reeive onsierle ttention in the reserh literture, with the prolem of ompring sequenes reeiving the most ttention [SK8]. Erly sequene omprison lgorithms inlue the lssi Wgner-Fisher lgorithm [WF], whih ws shown to e optiml for wie lss of omputtion moels in [AHU6, WC6]. Using the so-lle four Russins tehnique, more effiient log lgorithm for finite lphets is presente in [MP80]. More reent work on sequene omprison hs fouse on improving the expete se running time using outputsensitive lgorithms. For exmple, Myers presents n lgorithm, where n re the sizes of the input n eit sript, respetively [Mye86]. (This lgorithm forms the sis of the wiely use iff utility [MM85].) Further improvements re reporte in [WMG90]. These lgorithms re se on using the speil struture of n eit grph. Our tehnique for mpping the tree ifferening prolem to eit grphs, s esrie in Setion, llows us to pply the ove sequene omprison results to trees. For exmple, it is reltively strightforwr to exten these lgorithms in [Mye86] for ifferening trees in min memory without ny signifint inrese in running time. However, extening these results to externl memory ppers more omplite n is prt of our ontinuing work. Severl formultions of the tree omprison prolem hve lso een stuie. Erly work inlues Selkow s reursive lgorithm for simple formultion similr to the one in this pper [Sel]. More reent work inlues [ZS89], whih stuies orere trees, n [ZWS95], whih stuies unorere trees. Most formultions of the tree ifferening prolem for unorere trees re NP-hr. There re signifint vntges to esriing tree ifferenes using not only the three si eit opertions (insert, elete, n upte), ut lso sutree opertions suh s move, opy, n unopy. In [CRGMW96], we stuie vrition tht inlues sutree move opertion n esrie n effiient lgorithm tht uses simplifying ssumptions se on omin hrteristis. In [CGM9], we stuie vrition tht inlues sutree opy n unopy opertions in ition to moves. These lgorithms re use in the implementtion of the system for mnging hnge in utonomous tses [CAW98]. All the ove lgorithms re min-memory lgorithms; tht is, they ssume tht ll input t n working storge resies in RAM. To our knowlege, ours is the first externlmemory ifferening lgorithm. The esign of externlmemory lgorithms se on the prolem formultions n ies in the work esrie ove is topi for ontinuing work. The Unix progrm iff implements sequene omprison lgorithm for files tht re too lrge to fit in RAM using simple wrpper roun the stnr iff lgorithm [Mye86]. The iff progrm works y first iviing the input files into segments tht fit in RAM, n then running iff on eh pir of orresponing (y position in the respetive file) segments. This strtegy oes not gurntee minimum-ost eit sript. In ft, single inserte line n misle iff into mismthing lrge numer of other lines. The prolem of omputing ifferenes is losely relte to the prolem of pttern mthing (e.g., [WZJS9, WCM 9, WSC 9]). While there re importnt ifferenes etween the two prolems, it my e possile to shre some of the tehniques use y their solutions. Conlusion We esrie severl pplitions tht re se on ompring two snpshots of t in orer to etet the ifferenes etween them. We expline the nee for externl-memory ifferening lgorithms for oth flt (sequene) n hierrhil (tree) t. We moele hierrhil t using roote, orere, lele trees, n formlize the hierrhil t omprison prolem using the ie of minimumost eit sript etween two trees. We esrie metho to mp this tree omprison prolem to shortest-pth prolem in speil grph lle the tree eit grph. We first 00

12 presente min-memory tree ifferening lgorithm se on the eit grph reution. Next we isusse its extension to externl memory n esrie prolems with nive extension. We then presente our min lgorithm (xmiff) for effiiently ifferening trees in externl memory. As ontinuing work, we re exploring the use of our eit grph reution to trnsfer results from sequene omprison to trees. Preliminry results inite tht while this strtegy is reltively esy to use for min-memory lgorithms, extening it to externl-memory lgorithms is more hllenging. We re lso working on inorporting the lgorithm xmiff into the hnge mngement system n on relesing puli version of the implementtion. We lso pln to explore the pplition of our tehniques to relte prolems suh s pttern mthing n t mining in semistruture t (whih is often moele using trees n grphs). Referenes [AHU6] A. Aho, D. Hirsherg, n J. Ullmn. Bouns on the omplexity of the longest ommon susequene prolem. Journl of the Assoition for Computing Mhinery, ():, Jnury 96. [CAW98] S. Chwthe, S. Aiteoul, n J. Wiom. Representing n querying hnges in semistruture t. In Proeeings of the Interntionl Conferene on Dt Engineering, pges, Orlno, Flori, Ferury 998. [CGM9] S. Chwthe n H. Gri-Molin. Meningful hnge etetion in struture t. In Proeeings of the ACM SIGMOD Interntionl Conferene on Mngement of Dt, pges 6, Tuson, Arizon, My 99. [CRGMW96] S. Chwthe, A. Rjrmn, H. Gri- Molin, n J. Wiom. Chnge etetion in hierrhilly struture informtion. In Proeeings of the ACM SIGMOD Interntionl Conferene on Mngement of Dt, pges 9 50, Montrél, Quée, June 996. [LGM96] W. Lio n H. Gri-Molin. Effiient snpshot ifferentil lgorithms for t wrehousing. In Proeeings of the Interntionl Conferene on Very Lrge Dt Bses, Bomy, Ini, Septemer 996. [MM85] W. Miller n E. Myers. A file omprison progrm. Softwre Prtie n Experiene, 5():05 00, 985. [MP80] W. Msek n M. Pterson. A fster lgorithm omputing string eit istnes. Journl of Computer n System Sienes, 0:8, 980. [Mye86] E. Myers. An ifferene lgorithm n its vritions. Algorithmi, ():5 66, 986. [Sel] S. Selkow. The tree-to-tree eiting prolem. Informtion Proessing Letters, 6(6):8 86, Deemer 9. [SK8] D. Snkoff n J. Kruskl. Time Wrps, String Eits, n Mromoleules: The Theory n Prtie of Sequene Comprison. Aison-Wesley, 98. [Ti85] W. Tihy. RCS A system for version ontrol. Softwre Prtie n Experiene, 5():6 65, July 985. [Vit98] J. Vitter. Externl memory lgorithms. In Proeeings of the ACM Symposium on Priniples of Dtse Systems, Settle, Wshington, June 998. [WC6] C. Wong n A. Chnr. Bouns for the string eiting prolem. Journl of the Assoition for Computing Mhinery, (): 6, Jnury 96. [WCM 9] J. Wng, G. Chirn, T. Mrr, B. Shpiro, D. Shsh, n K. Zhng. Comintoril pttern isovery for sientifi t: some preliminry results. In Proeeings of the ACM SIGMOD Conferene, pges 5 5, My 99. [WF] R. Wgner n M. Fisher. The string-to-string orretion prolem. Journl of the Assoition of Computing Mhinery, ():68, Jnury 9. [WMG90] S. Wu, U. Mner, n G.Myers. An sequene omprison lgorithm. Informtion Proessing Letters, 5:, Septemer 990. [WSC 9] J. Wng, D. Shsh, G. Chng, L. Relihn, K. Zhng, n G. Ptel. Struturl mthing n isovery in oument tses. In Proeeings of the ACM SIGMOD Interntionl Conferene on Mngement of Dt, pges , 99. [WZJS9] J. Wng, K. Zhng, K. Jeong, n D. Shsh. A system for pproximte tree mthing. IEEE Trnstions on Knowlege n Dt Engineering, 6():559 5, August 99. [Yn9] W. Yng. Ientifying syntti ifferenes etween two progrms. Softwre Prtie n Experiene, ():9 55, July 99. [ZS89] K. Zhng n D. Shsh. Simple fst lgorithms for the eiting istne etween trees n relte prolems. SIAM Journl of Computing, 8(6):5 6, 989. [ZWS95] K. Zhng, J. Wng, n D. Shsh. On the eiting istne etween unirete yli grphs. Interntionl Journl of Fountions of Computer Siene,