Scalable Distributed Data Structures: A Survey Λ

Transcription

1 Sclble Distributed Dt Structures: A Survey Λ ADRIANO DI PASQUALE University of L Aquil, Itly ENRICO NARDELLI University of L Aquil nd Istituto di Anlisi dei Sistemi ed Informtic, Itly Abstrct This pper reviews literture on sclble dt structures for serching in distributed computing environment. Strting with system where one server mnges file of given size tht is ccessed by specific number of clients t specific rte, sclble distributed dt structures (SDDS) cn efficiently mnge file tht is n times bigger nd ccessed by n times more clients t the sme per-client rte, by dding servers nd distributing the file cross these servers. We nlyze nd compre SDDS proposls bsed on hshing techniques nd order preserving techniques. Work regrding the mngement of multi-dimensionl dt is lso reported. Moreover, issues such s high vilbility nd lod control re considered. Keywords sclble distributed dt structure, messge pssing environment, high vilbility 1 Introduction A network of computers is n ttrctive environment for mny pplictions, nd in prticulr for the ones with high performnce requirements. Among the motivtions usully cited for the use of distributed system there re: ese of expnsion, incresed relibility, the bility to incorporte heterogeneous resources, nd resource shring mong utonomous sites. When dt structure is mnged by more thn one of the processors on network one usully speks of distributed dt structure. In this re lrge number of solutions hve been proposed. A min prtition mong the proposls cn be defined on the bsis of whether the number of processors mnging the dt structure is fixed or not. For fixed number of processors, the proposed dt Λ Reserch prtilly supported by the Europen Union TMR project Chorochronos nd by the Itlin MURST COFIN project REACTION: Resource Alloction in Computer Networks. 87

2 88 Distributed Dt nd Structures 3 structures hve focus not only on efficient dt ccess, but on combintion with other relevnt issues such s, e.g., lod blncing nd concurrency control. In this pper we focus insted on the cse of vrible number of processors, since this is n essentil ingredient to obtin sclble distributed dt structure (SDDS). The min objective of n SDDS is in fct to ccommodte dynmic file growth with sclble performnce, where the key to sclbility is the dynmic distribution of file cross multiple servers of distributed system. Furthermore, n SDDS hs focused view on efficiency considertions, disregrding other spects completely. Consider file tht contins set of records nd is mnged by single server t single node of computer network, ccessed by number of clients t fixed per-client rte. Idelly, when the file grows by fctor of n nd the number of clients lso increses by the sme fctor, we should be ble to scle up the system throughput (i.e. to serve n times more clients), without ny noticeble degrdtion of the system performnce, by redistributing the file cross k servers. This redistribution of dt should tke plce continuously s the file is growing. Servers re creted dynmiclly on demnd nd frgments of the file re redistributed mong them. The widely used technique is to split server s dt into two hlves nd migrting one hlf onto new server. We ssume it is lwys possible to find new fresh server to involve in the mngement of the dt structure whenever it is necessry. As perfect sclbility (for non-trivil worklods) is chievble only theoreticlly, we usully spek of sclble pproch lredy if response time of the system is nerly constnt for resonbly lrge vlues of n nd increses only very slowly for very lrge vlues of n. Let us formlize the bsic environment we re considering. The distributed system is composed by collection of processing sites, interconnected by communiction network. The dt is tken from domin D. Ech dt item d 2 D consists of two min fields, d =(Key d ;Record d ), where Key d is key tken from mono-dimensionl or multi-dimensionl domin, nd Record d is record field contining the relevnt dt. A distributed dt structure is composed of dt orgniztion scheme, specifying collection of locl dt structures storing copies of dt items t vrious sites in the system, coupled with set of distributed ccess protocols tht enble processors to issue modifiction nd query instructions to the system nd get pproprite responses. Dt re orgnized in buckets. We ssume tht ech site mnges exctly one bucket. Communiction mong sites hppens by sending nd receiving messges. A messge cn be of the following types: ffl point-to-point messge. Such messges hve one sender site nd unique receiver site. ffl multicst messge. Such messges hve one sender site nd mny receiver sites. In generl, the set of receiver sites of multicst messge corresponds to subset of ll the sites of the structure. In our cse, the set of receiver

3 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 89 sites of multicst messge corresponds to the set of server sites of the structure. We concentrte minly on dictionry structures, which support exct serches, inserts nd deletes, nd typicl extensions supporting rnge queries, prtil queries, nerest neighbor queries nd so on. In prticulr, mny of the proposed SDDSs only consider insertions nd serch opertions. Ppers explicitly discussing nd nlyzing deletions re [6, 20, 1]. In the rest of the pper whenever we consider request for single key, we men n insert or n exct serch. 1.1 Performnce mesures The min mesure of performnce for n opertion is its communiction complexity. In the context of SDDS this is defined s the number of messges exchnged between clients nd servers to perform given opertion. For this complexity mesure, the following ssumptions hold: ffl The rel network topology is not relevnt. The grph ssocited with the communiction network is complete grph. This llows to mesure the communiction complexity in terms of the number of exchnged messges. ffl Ech messge costs one unit nd the size of messge is not relevnt. ffl The network is free of errors. Both nodes nd links never crsh. This hypothesis is relxed in the high vilbility schemes. Another relevnt complexity mesure is the globl lod fctor, formlly defined s α = x bn, where x is the number of dt items stored in the overll structure, b is the cpcity of bucket (equl for ll the buckets) nd n is the number of servers (we recll tht ech server mnges exctly one bucket). In ddition, the globl nd locl overhed (introduced in [30]) give mesure on how lrge is the wste of computtionl resources deriving from the fct tht servers re distributed over communiction network. They re defined s follows: ffl locl overhed (locl ovh) mesuring the verge frction of useless messges tht ech server hs to process; this is expressed by the verge, over ll servers, of the rtio between the number of useless messges nd the number of messges received by server. A messge is useless for server if it is received by it but it is not pertinent to it. ffl globl overhed (globl ovh) mesuring the verge frction of overhed messges trveling over the network; this is expressed by the rtio between

4 90 Distributed Dt nd Structures 3 the overll number of overhed messges nd the overll number of requests. A messge is considered to be overhed if it is not query messge issued by client. The mthemticl definition of these two prmeters is now provided. Let us denote with n the overll number of servers, nd with rec msg(i) nd pert msg(i), the number of messges received by server i nd the number of pertinent messges received by server i, respectively. Then, we hve: locl ovh = n i=1 rec msg(i) pert msg(i) rec msg(i) n globl ovh = n i=1 rec msg(i) n i=1 pert msg(i) rec msg(i) n i=1 The pper is orgnized s it follows: section 2 discuss SDDS proposl bsed on hshing, while in section 3 those bsed on order preserving dt orgniztion technique, re presented. In section 4 multi-dimensionl dt mngement in SDDS frmework is nlyzed. Section 5 shows how to tke into ccount in SDDS of high vilbility requirements. Finlly, lst section contins prtil survey of non sclble distributed dt structures. 2 Hsh bsed schemes 2.1 LH* LH* [17, 20] introduced the concept of Sclble Distributed Dt Structure (SDDS). LH* is generliztion of Liner Hshing to distributed sites. A file F is stored on server sites nd it is ccessed by client sites. Ech server site mnges one bucket, which stores some of the file records. Ech client site hs its own view of the overll file, which my be out-of-dte. Client s view is updted through requests, which my require t most two dditionl messges. The sttus of overll file is described by two prmeters, nmely the hshing level (i) nd the split pointer (n). The hshing level defines the couple of hshing functions to be used to ssign keys to buckets nd the split pointer identifies the next bucket to be split whenever collision (tht is n insertion in full bucket) hppens. Ech server knows the hshing level only of the bucket it mnges. Ech client uses its own view of the overll file sttus (hshing level nd split pointer) to send request messges to servers. If ddressing is wrong the receiving server is ble to forwrd key to nother server, which either is the correct one or is ble to identify the correct one. Moreover, the receiving server communictes bck to the client its hshing

5 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 91 level, so tht client my bring its own view closer to the file overll sttus. A designted site ct s split coordintor, which forces the server site designted by the split pointer to split its bucket nd serilizes ll the splits. The communiction network is ssumed without dely (if there is dely then n rbitrrily fst sequence of insertions my cuse n rbitrrily long sequence of forwrding messges). In LH* the splitting policy obeys to globl rule: whenever split occurs in the structure, the splitting server is lwys the n-th one. After tht n is incresed by one. A server is split even if it is not the overflowing server. This my led in some cses to poor globl nd locl lod fctor. In fct server could hve to mnge gret number of keys, witing to be the n-th server, nd in the mentime, servers with few keys re split. Moreover server does not know when it hs to split. It hs to be notified. In [17] specil split coordintor, prticipting in the bucket split opertions, is proposed. Such coordintor knows the exct configurtion of the structure (i.e. the current prmeters n nd i). It notifies the n-th server to split, nd then updtes the configurtion. The definition of such specil entity is not completely complint with sclbility gols, becuse it cn likely become bottleneck. However, in [20] some vrints of the bse technique is proposed, in prticulr version of LH* without split coordintor. The technique is bsed on token, which is stored by the next server tht hs to split (i.e., the n-th server). After the split, the token is sent to the next server (it is ssumed tht server lwys knows the ddress of the next server). Other vrints discussed in [20] regrd the control of the lod fctor, tht bsiclly consists in llowing the split of the server n only whenever n estimtion of the globl lod fctor reches given threshold. The ltter technique gives good results if it is possible to ssume tht the used hsh functions ctully hsh uniformly. In LH*, where the hshing functions re defined s h i (c) =c mod 2 i, for ech i, this mens tht the probbility of hshing key to given ddress is 1=2 i. 2.2 DDH Distributed Dynmic Hshing (DDH) is introduced in [5]. It is distributed version of Dynmic Hshing technique. A trie bsed on the rightmost i digits of the key is used s hsh function t level i. Clients hve locl imges of the overll logicl trie. In generl, logrithmic number (trie-height) of forwrding messges is required for new client to identify the correct server. Ech bucket splits s soon s it overflows. Ech client needs to store trie. In DDH ech server mnges mny smll buckets. In this wy the cost of split is reduced, nd fine-grined lod shring cross servers is chieved. Determining if its own bucket hs to be split is n utonomous decision tht cn be mde by the ffected server. Moreover the splitting server is lso the overflowing server, nd then there is more ccurte distribution of keys mong the servers. The resulting lod fctor is better thn LH* s one. Moreover, specil split

6 92 Distributed Dt nd Structures 3 coordintor is not needed. Like other SDDS proposl, in DDH site (tht it is mbiguously clled split coordintor too) knowing the vilble servers in the network, is necessry, but it hs not to mnge specil decisions or to correct globl informtions bout the structure. The bsic drwbck of DDH in respect to LH* is the communiction complexity, i.e. the number of messges needed to stisfy client request. LH* s communiction protocol ensures constnt number of ddress errors, in the worst-cse, nmely 2, hence worst-cse of 4 messges for ny request. The number of ddress errors in DDH cn be the height of the virtul trie. If the trie is unblnced we cn hve liner number of ddress errors for request, nd the sme holds for the communiction complexity. In [5] some experiments re reported, but they do not give complete ide of performnces of DDH. Moreover, they regrd only constnt number of servers, while the bsic property of n SDDS is to lwys hve the possibility to get new server to scle-up performnces. 2.3 Lod blncing Distributed liner hshing with explicit control of cost to performnce rtio is presented in [35]. Multiple buckets my be ssigned to ech server nd n ddress tble mps logicl buckets to server number. Buckets cn be redistributed through splitting or migrtion to keep the overll lod t n cceptble level. Ech server hs fesible cpcity (expressed in number of keys), bove which it becomes overloded, nd pnic cpcity, bove which it cnnot ccept ny more keys. Servers nd clients ct like in the LH* schem. Ech of them hs its own view of the hshing level of the overll file nd its own copy of the ddress tble. A client uses its own hshing level to ccess its own ddress tble nd to find the server where to send the key. This server my be the wrong one, nd in this cse it uses its own ddress tble to forwrd the messge to nother server, which either is the right one or is ble to forwrd the key to the right one. Moreover, the first server communictes bck to the client its own ddress tble nd hshing level, so tht the client my come closer to the current view of the file. While the overll lod is below specified threshold nd no server hs reched its pnic cpcity, overloding is mnged, if possible, through migrtion of buckets from more loded servers to less loded ones. If no such migrtion is possible thn new server is cquired, but only when the overll lod threshold (defined ccording to n heuristic estimtes) is reched. Whenever server reches its pnic cpcity, the possibility of llevite its lod through migrtion of buckets to less loded servers is evluted. If it is not possible, new server is cquired. A designted site, nmed file dvisor, is in chrge of mnging nd coordinting migrtions nd splittings. The file dvisor uses probbilistic function to estimte ech server lod (since ech server reports to file dvisor only every k insertions in n overloded sttus) nd the overll lod.

7 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 93 3 Order preserving SDDS 3.1 DRT DRT (Distributed Rndom Tree) [15] proposes distributed serch tree for serching both single items nd rnges of vlues in totlly ordered set of keys (llowing insertion of keys). It is bsiclly serch structure, bsed on key comprisons, mnged s generic tree. The overll tree is distributed mong the different server sites. Ech lef node is llocted to different server, together with prtil copy of the overll serch structure (clled locl tree). When lef node overflows, its bucket is split in two nd new server is brought in. The overflown lef node is trnsformed in n internl node with two sons (lef nodes). One son contins keys remining with the current node, nd the new server tkes cre of the remining keys. The lef node corresponding to new server becomes the root of new locl tree (which is the prt of the overll tree llocted to the new server). This node therefore ppers twice in the overll tree (once s lef in the old, overflown, node nd once s root in locl tree). Internl nodes re therefore distributed to the different servers ccording to the wy the tree hs grown. Client sites query the structure, ech using its own view of the overll structure. A client view is portion of the overll tree, nd my be out-of-dte since lef node my hs subsequently been split due to n overflow. A client uses its view to identify to which server the serch request hs to be sent. If this server evolved nd hs no more the key, then it forwrds the request to the server it identifies using its locl trees. This forwrding chin ends t the server hving the key. This lst server sends bckwrd chin of ICMs (Index Correction Messges), contining the informtion bout locl trees of servers trversed during the forwrding phse, follows the sme pth followed by the forwrding chin. Informtion bout locl trees in n ICM re used by ech server receiving it to updte its locl tree nd to build up, in combintion with the serch pth for the requested key in its locl tree, the view djustment to send bck, figure 1- (from [15]). The client finlly receives, together with the messge relted to the key, the overll view djustment, see figure 1-b. Since there is no explicit mechnism to keep the overll tree blnced, in the worst-cse the height of the overll tree is liner in the number of servers. However, for rndom insertions from domin with uniform distribution, the verge height of the overll tree is logrithmic. This is not surprising since for uniform distribution the expected height of serch tree under rndom insertions is logrithmic in the number of insertions [14]. The totl number of messges in the worst-cse is O(n), where n is the number of servers.

8 94 Distributed Dt nd Structures 3 ) T(s) T(s ) T(s ) 1 2 s s s 4 3 T (s ) T (s )=T(s ) b) T(c) T (s) T(c) c b c b d c b d Figure 1: A forwrding chin nd ICM messges (). Updte of locl tree T (c) of client c with correction tree T 0 (s) (b). 3.2 RP* A fmily of distributed dt structures (Rnge Prtitioning - RP*) supporting rnge queries on totlly ordered keys is introduced in [18]. The bsic structure of the fmily, RP Λ n, is bsiclly B+ -tree distributed mong servers nd without ny index. Ech bucket is llocted to different server nd serch is done through multicsting. Ech bucket covers contiguous portion of the whole key rnge disjoint from portions covered by other buckets. Ech server processes the query nd the one (or the ones for rnge serch query) whose rnge contins key s vlue nswers with point-to-point messge. Insertions re mnged similrly. When bucket overflows its server brings in new one nd ssigns hlf of its keys to it. To reduce communiction network lod ech client my mintin locl index (this structure is clled RP Λ c ) which is updted in consequence of serches. The locl index is collection of couples hbucket rnge; bucket ddressi. A client sends point-to-point request to the server identified s the potentil owner of the key in its locl index. If the client hs no informtion bout the server owning certin rnge it issues multicst request. If the receiving server is not the correct one, it issues multicst request, including the rnge of keys it mnges. Client then uses nswers from point-to-point nd multicst requests, which my include one or two couples hbucket rnge; bucket ddressi, to updte its locl index. A third vrint, clled RP Λ s, is structure which mintins indexes lso t server sites nd completely voids the need of multicsting. In this cse we hve full-blown B + -tree distributed mong server sites, one node for ech server. Lef nodes re servers mnging buckets. Internl nodes re mnged by dedicted servers nd hve structure nd behvior similr to tht of n internl B + -

9 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 95 - * 0 for 3 in 2 of 1 for nd for - 0 to the tht of 1 of it is of in 2 in i in for 3 c - * in b - in c 0 for 3 b in c 2 of 1 these 4 for nd for - 0 these the tht these of 1 of it is of in 2 in i in for 3 to this these 4 Figure 2: An RP Λ s (down). smple file with kernel with two level (up) nd tree level tree node, with seprting keys nd pointers to lower nodes, plus pointer to the prent node. The set of dedicted servers is clled kernel. If the receiving server is not the correct one, this my only hppen becuse the client hs n out-of-dte locl index where the server is mnging rnge of keys lrger thn wht the server is relly mnging. Therefore the request cn either be forwrded up in the overll B + -tree, so tht n ncestor of the receiving server cn identify the correct server to mnge it, or be forwrded down to the server which is relly tking cre of the key rnge contining the requested key (internl nodes in the B + -tree do not directly mnge keys). In both cses the server finlly nswering the request signls bck to client the up-to-dte view of the serch structure, by sending bck the pth visited in the tree during the serch for the correct serving bucket. A node which splits leves prent pointers of its sons out-of-dte. They re djusted only in two cses: either when the sons themselves split nd need to communicte this fct to their true prent, or when the sons receive request from their true prent. In figure 2 (tken from [18]) n exmple of RP Λ s file is shown. 3.3 The stright guiding property With the im to investigte intrinsic efficiency of SDDSs, Krll nd Widmyer nlyzed from theoreticl point of view performnce bounds of distributed serch trees. [16]. The min result is the impossibility to extend to the distributed cse

10 96 Distributed Dt nd Structures 3 both of two importnt properties used in mnging dt structures in the singleprocessor cse: 1. The serch process is monotone. 2. Rottions re used to keep the tree blnced. In the single processor cse serch tree is binry tree such tht every node represents n intervl of the dt domin. The overll dt orgniztion stisfies the invrint tht the serch process visits child node only if it lies inside the fther node s intervl. Krll nd Widmyer cll this behvior the stright guiding property. For stisfying the stright guiding property in the distributed cse is needed to ensure tht key received by server belongs to the set of keys the server represents, i.e. to the intervl of the node ssocited to the server. This intervl is given by the union of intervls ssocited to descendnts of the node. The stright guiding property ensures tht the serch process goes lwys down in the tree nd never goes up. In this cse bound on the height of the tree directly correspond to bound on the cost of the serch process. In [16] is proved tht if rottions in the distributed tree re used, it is impossible to keep the stright guiding property. To understnd why consider figure 3. Assume tht in the serch tree T the server s 1 mnges node v 1 nd the server s 2 6= s 1 mnges node v 2. Assume now tht rottion is needed t v 1 to reblnce the tree. T new is the tree fter the rottion, where we ssume the ssignment of nodes to servers hs not chnged. Note tht the set of keys visiting v 1 in the serch tree T (i.e. before the rottion) is superset of the set of keys visiting v 1 in the serch tree T new (i.e. fter the rottion). Thus, fter the rottion, server s 1 my receive the request for key whose serch pth ends in T 0, since v 1 is ssigned to s 1 in T. For exmple the request could be issued by client with n obsolete view, believing tht server s 1 still mnges n intervl contining T 0. But, fter the rottion, server s 1 should not mnge ny serch pth for keys in T 0. To rech T 0 from s 1 we hve to go up in the tree violting the stright guiding property. The sme problem exists if we exchnge the ssignment of nodes to server between v 1 nd v 2. In fct in this cse it is the server s 2 tht my receive the request for key whose serch pth ends in T 0. Hence whether we mintin the ssignment of servers s 1 nd s 2 to nodes v 1 nd v 2 in T new or we exchnge such n ssignment, we fil in ny cse to gurntee the stright guiding property. Moreover, Krll nd Widmyer show tht if rottions re not used nd the stright guiding property is mintined, lower bound of Ω( p n) holds for the height of blnced serch trees, hence for the worst-cse of serching.

11 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 97 v 0 v 0 S 2 v 2 S 1 v 1 rottion v 2 S 2 v 1 S 1 T 0 T 1 T 2 T 0 T1 T2 Figure 3: The rottion does not llow to keep the stright guiding property. 3.4 RBST A nturl pproch to overcome the complexity lower bound of Krll nd Widmyer is to violte the stright guiding property: this mens the serch process cn go up nd down in the distributed tree. Then bound on the height of the tree does not correspond nymore to bound on the cost of the serch process nd it is possible to use rottions to keep the tree blnced. But to obtin good complexity it is then necessry to bound the number of times serch opertion cn go up nd down in the tree. The first work in this direction is the RBST (Relxed Blnced Serch Trees) [1]. This is serch tree where nodes hve the following structure: ech node but the root hs pointer (fther pointer) to its fther (internl) node. Ech internl node hs pointer (left pointer) to node in its left subtree nd one (right pointer) to node in its right subtree. Note tht these pointed nodes my be, in generl, different from the direct sons of the internl node. Insertions nd deletions re mnged with technique similr to tht used for AVL-trees. In [1] it is shown in detil how to mintin these invrint properties of pointers coming out from nodes. Suppose request for key k rrives to node v (corresponding to server). If v is lef nd k belongs to the intervl of keys I(v) ssocited to v, then the serch termintes. Otherwise if v is n internl node nd k 2 I(v), then, ccording with routing informtion of v, the left or the right pointer of v is used. Since the stright guiding property is not stisfied, it is possible tht k =2 I(v). In this cse the fther pointer is used to forwrd the request. The serch process is therefore chnged, but the number of times serch opertion cn go up nd down in the tree is bounded by the logrithmic of the number of servers [1]. The cost for n exct serch, n insertion nd deletion is of O(log 2 n) messges in the worst-cse.

12 98 Distributed Dt nd Structures BDST Another pproch to obtin good worst-cse performnces in SDDS is discussed in [6], where BDST (Blnced Distributed Serch Trees), n SDDS tking n pproch similr to those of RBST, is presented. The stright guiding property is violted, the tree is kept blnced through rottions nd the serch process is modified. However, in the cse of BDST, the number of times serch opertion cn go up nd down in the tree is bounded by 2. Then BDST improves the result of RBST, giving cost for n exct serch, n insertion nd deletion of Θ(logn) messges in the worst-cse. In the following we discuss in some detil the dt structure. Let T be binry serch tree with n leves (nd then with n 1 internl nodes). f 1 ;::: ; f n re the leves nd t 1 ;::: ;t n 1 re the internl nodes. h(t ) is the height of T, tht is the number of internl nodes on longest pth from the root to lef. To ech lef bucket cpble of storing b dt items is ssocited. Let s 1 ;::: ;s n be the n servers mnging the serch tree. We define lef ssocition the pir ( f ;s), mening tht server s mnges lef f nd its ssocited bucket, node ssocition the pir (t;s), mening tht server s mnges internl node t. In n equivlent wy we define the two functions: ffl t(s j )=t i, where (t i ;s j ) is node ssocition, ffl f (s j )= f i, where ( f i ;s j ) is lef ssocition. To ech node x, either lef or internl one, the intervl I(x) of dt domin mnged by x is ssocited. Every server s but one, with lef node ssocition (t;s) nd lef ssocition ( f ; s), records t lest the following informtion: ffl The internl node t = t(s) nd the ssocited intervl of key s domin I(t), ffl The server p(s) mnging the prent node pn(t) of t, ift is not the root node, ffl The server l(s) (resp., r(s)) mnging the left child ls(t) (resp., right child rs(t)) oft, nd the ssocited intervl I l (t) (resp., I r (t)), ffl The lef f = f (s) nd the ssocited intervl of key s domin I( f ), ffl The server pf(s) mnging the fther node pn( f ) of f,if f is not the unique node of globl tree (initil sitution). This informtion constitutes the locl tree lt(s) of server s (see figure 4). Since in globl tree of n nodes there re n 1 internl nodes, there is one server s 0 mnging only lef ssocition, hence lt(s 0 ) is mde up by only the two lst pieces of informtion in the bove list.

13 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 99 pn(t)=t(p(s)) t=t(s) pn(f)=t(pf(s)) ls(t)=t(l(s)) rs(t)=t(r(s)) f=f(s) Figure 4: Locl tree of the server s. t(r) t(s) t(s") t(s) Client f(s ) Client f(s ) Client f(s ) Figure 5: The BDST serch process for request from new client (left), from client with ddressing error sending its request to: logiclly pertinent server (center) nd non logiclly pertinent server (right). We sy server s is pertinent for key k, ifs mnges the bucket to which k belongs. In our cse if k 2 I( f (s)). Moreover we sy server s is logiclly pertinent for key k,if k is in the key intervl of the internl node ssocited to s, tht is if k 2 I(t(s)). Note tht the server mnging the root is logiclly pertinent for ech key. Note lso tht, due to the effect of rottions, it is not necessrily I( f (s)) I(t(s)). Suppose tht request for key k rrives to server s. Ifs is the pertinent server for k then s directly mnges the request. If it is logiclly pertinent, then s forwrds the request downwrds in the tree, using l(s) or r(s). If s is not logiclly pertinent, then it forwrds the request upwrds in the tree, using p(s). In generl, request of client trvels the globl tree structure until rrives to the lef corresponding to the pertinent server. In figure 5, the vrious cses of the serch process re described. It is cler tht the number of messges for request, ccounting for the request nd the nswer messges, is 2h(T )+2 in the worst-cse. Rottions do not influ-

14 100 Distributed Dt nd Structures 3 ence the serch process. In this cse ny blncing strtegy bsed on rottions cn be used, nd the worst-cse cost for exct serch, insertion or deletion is shown to be Θ(logn) messges. 3.6 DRT* The min drwbck of BDST is represented by the fct tht the use of rottions requires locking servers involved in the rottion process. In fct locks reduce the concurrency in the system. On the contrry DRT, i.e. the implementtion of distributed tree without using blncing strtegies, suffers of the well known worst-cse cost of O(n) messges for request. Although the worst-cse result is very bd, the mortized nlysis shows very nice behvior. DRT* [8, 9], extends the DRT technique by mens of different use of correction technique bsed on ICMs. These modifictions improve the response time of request with respect to DRT, but do not influence the mortized communiction cost, in the sense tht the communiction complexity results of DRT* hold lso for the DRT. More detils on DRT* cn be found in nother pper [8] of these proceedings. 3.7 Distributed B + -tree In DRT nd DRT* the gol of reducing communiction complexity is reched by mens of continuous improvement of the system knowledge of locl trees of servers. This is obtined in lzy mnner, in the sense tht the locl tree of server s is corrected only whenever s mkes n ddress error. It is cler tht the best possible sitution is the one where ech locl tree is lwys correct. An lterntive strtegy is to try to ensure servers hve correct locl trees by sending corrections exctly when locl trees become obsolete, tht is whenever split occurs in the structure. A structure designed to this im is Distributed B + - tree [4]. For the ske of simplicity we describe Distributed B + -tree in slightly different wy with respect to the originl pper. If server s ws creted through the split of server s 0 then server s stores pointer to the server s 0. With respect to the node v ssocited to s, the pointer to s 0 is pointer to the fther node v 0 of v. Whenever split occurs, the splitting server s sends correction messge contining the informtion bout the split to s 0. s 0 corrects its locl tree nd sends the messge upwrds in the tree so to rrive to the root. This technique ensures tht server s ssocited to node v hs completely up-to-dte view of the subtree rooted t v. This mens tht it knows the exct prtition of the intervl I(v) of v nd the exct ssocitions between elements of the prtitions nd servers. This llows s to forwrd request for key belonging to I(v) directly to the right server, without sending chin of messges long the tree. This distributed tree does not use rottions, hence ech request rriving to s belongs to I(v) becuse of

15 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 101 the stright guiding property. The min result is worst-cse constnt number of messges for request, which is very good result for n order preserving dt structure. The drwbck is the cost of split. In fct, like in DRT, if keys re not uniformly distributed over the dt domin, the height of the distributed tree my be liner, nd then the cost of correcting locl trees fter split cn be liner s well. In [7] comprison between Distributed B + -tree nd BDST is performed, showing tht BDST behves better in the mortized cse, for the cse of sequence of intermixed exct serches nd inserts. In fct Distributed B + -tree hs liner mortized cost, while for BDST we clerly hve logrithmic cost. The mortized cost of sequence of intermixed exct serches nd inserts of DRT* is better thn the BDST one, hence it is better thn the Distributed B + -tree one s well. 3.8 DSL Another SDDS deling with order preserving dt mngement is DSL (Distributed Skip Lists). It is fundmentlly bsed on the extension of the skip list technique to the distributed environment. The min results, bsed on probbilistic nlysis - which is the stndrd pproch to mesure skip list performnces, re n ccess cost of O(logn) messges nd n O(1) number of reconstruction opertions fter merges or splits of nodes with high probbility. A complete presenttion of DSL cn be found in pper [3] of these proceedings. 4 Multidimensionl serch structures Mny of the newest ppliction res, like CAD, GIS, Multimedi nd others, del with very lrge mounts of complex (i.e. multi-ttribute) dt nd require high performnce. A distributed environment offers good nswer to these requirements. Some SDDSs present efficient solutions for serching multi-dimensionl dt. The min proposls for multi-dimensionl SDDSs re bsed on the extension to the distributed context of the k-d tree [2]. 4.1 Distributed k-d tree In [28, 29, 30] n SDDS version of the k-d tree [2] is proposed. Different vrints re discussed, nd their use depends on the vilbility of the multicst protocol in the communiction network. The bse technique used to design distributed k-d tree is like the one used to derive DRT from stndrd binry trees. Ech server mnges different lef of the tree, nd ech lef corresponds to bucket of dt. A k-d tree is binry tree where ech internl node v is ssocited to

16 102 Distributed Dt nd Structures 3 (bounded or not) k-d intervl (or k-rnge) I(v), dimension index D(v) nd vlue V (v). The intervl ssocited to the left (resp. right) son of v is mde up by every point in I(v) whose coordinte in dimension D(v) hs vlue less thn (resp. not less thn) V (v). D(v) is clled the split dimension for node v. V (v) is the split point for node v. Leves of the k-d tree re ssocited only to k-d intervl. To ech lef w of k-d tree one bucket exctly corresponds, denoted with the sme nme. Bucket w contins ll points within I(w). The k-d intervl I(v) of n internl node v is the initil k-rnge of the bucket which ws ssocited to node v when v ws inserted s lef into the k-d tree. When bucket v is split two leves, sy v 0 nd y, re creted nd inserted in the k-d tree s sons of node v. Bucket v, with new, reduced, k-rnge is ssocited to lef v 0, nd lef y tkes cre of the new bucket y, so tht I(v) =I(v 0 ) S I(y) nd I(v 0 ) T I(y) =/0. Therefore, for ech lef w but one it exists unique internl node z whose bucket s splitting creted the k-rnge of bucket ssocited to w. Such node z is clled the source node of lef w (nd of bucket w) nd is denoted s α(w). The lef without source node, for which we let for completeness α(:) = /0 is the lef mnging the initil bucket of the k-d tree. Clients my dd k-d points, which go in the pertinent bucket. In this cse bucket b (nd in the sme wy server b) is pertinent with respect to point p if b is ssocited to the lef node mnging the portion of the k-d spce contining p. Whenever split is needed, it is done with (k-1)-dimensionl plne nd vrious strtegies cn be used to select which dimension is chosen. A lrgely used strtegy is the round-robin one, where t ech level different dimension is selected nd fter k levels the sme sequence is used gin nd gin. Moreover clients cn issue exct mtch, prtil nd rnge queries. An exct mtch query looks for point whose k coordintes re specified. A prtil mtch query looks for (set of) point(s) for whom only h < k coordintes re specified. A rnge query looks for ll points such tht their k coordintes re ll internl to the (usully closed) k-dimensionl intervl specified by the query. Serch lgorithm for exct, prtil nd rnge serch is optiml. Optimlity is in the sense tht (1) only servers tht could hve k-dimensionl points relted to query reply to it nd tht (2) the client issuing the query cn deterministiclly know when the serch is complete. The ltter property is very importnt for the multi-dimensionl cse. In fct while for simple exct mtch query client knows tht the query is terminted whenever it receives n nswer by the single pertinent server, in prtil mtch or rnge query it is not true, in generl, tht there is exctly one pertinent server. Hence the client hs the problem of checking tht ll pertinent servers hve nswered. One wy to perform the termintion test is to clculte the volume V of the k-rnge of the query, then to clculte the volumes of the received k-d intervls nd to consider the corresponding sum S. Whenever V = S the termintion test returns true. This pproch is not good from prcticl point of view, becuse

17 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 103 infinite precision multipliction is needed for the pproch to be correct. In fct, if buckets covering very lrge nd very smll rnges exist t the sme time in the dt structure then, due to possible roundings, the termintion test my fil. An improved deterministic termintion test, nmed logicl volumes test, which does not suffer from this problem is presented in [30]. If multicst is vilble in the network, the protocol for serch process is very simple nd it is similr to the one of RP Λ n. A client issues query by simple multicsting it in the network, nd witing for the nswers of servers. If the use of multicst hs to be reduced, like in RP Λ c, client cn use locl index to ddress only servers pertinent for its requests. In this cse requests re sent using the point-to-point protocol. In cse of ddress error, the server receiving the request multicsts it in the network. A client my receive ICM messges both from server s (mnging k-d intervl I) it sent the request to, nd from the rel pertinent server s 0 (mnging k-d intervl I 0 ). In the overll index, I 0 my be ssocited to node which is vrious levels down with respect to the current height in the client s locl index of node ssocited to intervl I. Intervl I 0 certinly derives from splits of intervl I (mnged by s). But since servers do not mintin the story of splits, then the client receives intervls in ICMs without the relted sequence of splits producing them. While in RP Λ c simple binry tree is used to mnge such unrelted intervls in the locl index of client, in the multi-dimensionl cse the use of stndrd k-d tree for the locl index of client my produce incorrect results. Therefore in [28] new structure, nmed lzy k-d tree, is used to mnge the locl index of clients. A lzy k-d tree is k-d tree where: ffl there re two types of nodes: simple nodes nd compound nodes; ffl simple node my be lef or n internl node nd it corresponds to node of k-d tree. ffl compound node u hs no sons nd no piece of informtion relted to the globl index is directly ssocited with it. u is set C(u) of lzy k-d trees, whose roots re simple nodes, such tht for ech couple of distinct roots v nd w in C(u) it is I(v) T I(w) =/0. Complete detils of the structure cn be found in [30]; Ech client index strts with simple lef node. The index is then built incrementlly using nswers rriving from servers. A version of the structure with locl index t client nd server sites is defined in cse only the point-to-point protocol is used to exchnge messges. In this cse the locl indexes re locl trees like in the DRT. The difference with respect to DRT is tht now locl tree is stndrd k-d tree nd not binry tree, but the behvior is the sme. In fct now servers mnge n index. The index of server s contins t lest the portion of the globl tree built by the split of s. Moreover s cn

18 104 Distributed Dt nd Structures 3 send messges only to server belonging to its index. The ICM rriving to client s nswer to request contins the set of locl indexes of the involved servers. The ggregtion of these indexes corresponds t lest to contiguous portion of the globl k-d tree connecting the node ssocited with the server ddressed from the client to the node ssocited with the pertinent server. The most importnt thing is tht the correction technique used for the DRT* cn be used lso for this version of the distributed k-d tree. In [9] it is shown tht the sme nlysis conducted for the mono-dimensionl DRT* is vlid lso for this version of the distributed k-d tree nd then the sme results hold. In prticulr request my require liner number of messges in the worst-cse nd if we perform sequence of m requests (tht cn be exct serch nd insertions) producing n n-servers distributed k-d tree, strting with one empty server, we hve n mortized communiction cost of O mlog (1+m=n) n messges. 4.2 k-rp Λ s Another SDDS version of k-d tree is k-rp Λ s [19]. The pper presents the version used in cse of point-to-point protocol. k-rp Λ s is k-dimensionl version of the pproch used to design fmily RP*. There re servers involved in the mngement of dt (servers bucket), nd servers involved in the mngement of ddress errors (servers index). A server index mintins n internl index like in RP Λ s. The difference relies just in the fct tht while in RP Λ s stndrd binry tree is used for the internl index, k-d tree is used in k-rp Λ s. Techniques like logicl volumes re used for the termintion test fter rnge serch, nd in generl the mngement of requests is very similr to the distributed k-d tree one. Results for RP Λ s holds for k-rp Λ s s well. In prticulr, the worst-cse logrithmic bound for the communiction cost for n exct serch nd for n insertion. The min drwbck is the use of dditionl servers to mnge ddress errors, considering tht server is n expensive resource. With respect to ccess performnce, the comprison between distributed k-d tree nd k-rp Λ s is similr to the one between DRT* nd RP Λ s. k-rp Λ s nd RP Λ s re better in the worst-cse, where they gurntee n ccess cost logrithmic in the number of servers, while DRT* nd the distributed k-d trees, under prticulr dt distributions of the dt domin, my hve cost liner in the number of servers. We obtin the contrry in the mortized cse. In fct, it is esy to prove tht for both k-rp Λ s nd RP Λ s the cost remin logrithmic lso in the mortized cse. More precisely, we hve tht for m intermixed exct serches nd inserts, the mortized cost is O (mlog F n) messges, where n is the number of servers nd F is relted with the fnout of the servers in the kernel. For DRT* nd the distributed k-d trees in [8, 9] it is proved tht se-

19 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 105 quence of m intermixed exct serches nd inserts, gives n mortized cost of O mlog (1+m=n) n messges, which is generlly better thn k-rp Λ s nd RP Λ s one. Note tht while this mortized cost decreses with m, this does not hold for the mortized cost of k-rp Λ s nd RPΛ s. 5 High vilbility In this section we discuss some spects regrding fult tolernce for SDDSs. The min considertion is tht vilbility of distributed file deteriortes with the number N of sites, nd rther strongly in prctice. Assume, in fct, tht the probbility p d tht bucket is up is constnt, nd rther high in prctice, for exmple 99%. The probbility p c tht key c is vilble is p c = p d. The probbility p F tht the whole file F is vilble is p F =(p d ) N, under the usul ssumption tht bucket filures re mutully independent. If F scles modertely, to let us sy 100 buckets, it leds to p F = 37%, i.e., most of the time F is not entirely vilble. For N = 1000, one gets p F = 0:00004, i.e., zero in prctice. For mny pplictions, this my not be problem. For other pplictions however, especilly those needing relible very lrge dtbse, these numbers my men tht n SDDS scheme simply does not scle-up to files lrge enough for their needs. A k-vilbility scheme preserves the vilbility of ll records despite up to k bucket filures. The first ttempts in enhnce file vilbility re bsed on the populr mirroring technique. In [21] vrint of LH* clled LH* M is presented. This scheme mirrors every bucket nd thus preserve full ccessibility despite. The sclble nd distributed generliztions of B + -trees introduced in [4, 36] lso use the repliction. In both cses, the cost is the doubling of the storge requirements. This my be prohibitive for lrge files. High vilbility vrints of LH* with smller storge overhed hve therefore been developed. The 1-vilbility scheme LH* S stripes every dt record into m stripes, then plces ech stripe into different bucket nd stores the bitwise prity of the stripes in prity records in dditionl prity buckets [23]. The storge overhed for the high-vilbility is only bout 1=m for m stripes per record. If bucket is unvilble becuse of missing stripe then, like in the RAID schemes, LH* S recovers the missing stripe from ll the other stripes of the bucket, including the prity stripe. Striping typiclly produces meningless record frgments. This prohibits or t best hevily impirs the prllel scns, especilly with the function shipping. Those typiclly require entire records t ech site. Efficient scns re decisive for mny pplictions, especilly web servers nd prllel dtbses. In nother 1-vilbility vrint of LH* termed LH* g [22] the ppliction record, clled dt record, remins entire in LH* g. To obtin high vilbility, records re considered s forming m-member record groups ech provided with the bitwise prity record. The resulting storge overhed is bout 1=m, s for the striping. The speed of serches is tht of generic (0-vilble) LH*. It is unffected by the

20 106 Distributed Dt nd Structures 3 dditionl structure for the high-vilbility. As the file grows, 1-vilbility or even k-vilbility for ny sttic k is however not sufficient to prevent decrese in relibility. To this im one needs to dynmiclly increse k. The result is sclble vilbility schemes. The first sclble vilbility scheme ws LH* SA [25]. LH* SA retins the concept of record grouping, mking the technique more elborted. Ech dt record c is member of k or k +1 1-vilble groups tht only intersect in c nd re ech 1-vilble. The vlue of k progressively increses with the file. For ny k, LH* SA file is k-vilble. The storge overhed my vry substntilly depending on the file size. It cn be close to the miniml possible for k-vilbility which is known to be k=m. But it cn lso become over 50%. In [24, 26] n lterntive sclble vilbility scheme termed LH* RS is presented. Through record grouping, it retins the LH* generic efficiency of serches. Ech record belongs to one group only, but with k or k + 1 prity records. This provide the (sclble) k-vilbility of the file. The prity clculus uses the Reed Solomon Codes (RS-codes). This mthemticlly complex tool proves simple nd efficient in prctice. Current dvntges of LH* RS re storge overhed lwys close to the miniml possible, nd more efficient recovery lgorithm, ccessing buckets only within one group. Moreover, the bsic ides in LH* RS my be lso ported to other SDDS schemes, including the order preserving ones. 6 Relted work Mny ppers relted to SDDS nlyzed, s discussed in the introduction, distributed dt structures under the ssumption tht the number of processors (nodes, sites) is fixed. Typiclly, the min problem ddressed here is how to plce dt mong the fixed number of sites in order to blnce the worklod. We discuss some of the ppers belonging to this re, without the gol of being exhustive. Consider lso tht some of the structures presented in this section could be trnsformed into sclble ones. One importnt spect under which structures here discussed nd SDDSs differ is the mngement of lod blncing, tht is one of the min gol for both of the ctegories. In order to chieve lod blncing non sclble structure distributes dt mong ll the server from the beginning, while sclble structure strts with minimum number of servers (typiclly one) nd blnces the lod mong them to provide dequte performnces. When this is no more possible using the existing servers, new servers re clled in the structure nd now the lod blncing technique considers lso the new servers. In [10] (non sclble) distributed version of extendible hshing is presented. The directories of the tble re replicted mong severl sites (the directory mngers). Dt buckets re distributed mong sites (the bucket mngers). Every updte to the directories must be distributed to every copy. Other issues discussed

21 Di Psqule & Nrdelli: Sclble Distributed Dt Structures 107 in the pper regrd chnges in the structure to llow replictions, used to increse vilbility of the dt structure in presence of filure or to improve performnce by llowing more concurrency. In [27], the problem of implementing Bounded Disordered files in multiprocessors multi-disk environments consisting of fixed number of processor-disk pirs is considered. The used schem is vlid both for tightly coupled (shred memory) nd loosely coupled (locl network) processors. The use of Bounded Disordered files is motivted by the fct tht it chieves good performnce for single-key opertions (lmost s good s tht of hsh bsed methods), nd unlike hshing schemes, rnge serches re performed with low cost. The strightforwrd solution is to eqully prtition the file records mong processors, ech of which mintins its prt of locl Bounded Disorder file (stored in the processor s min memory nd disk). This method is highly prllel nd chieves good performnce due to the use of Bounded Disorder files. An lterntive method, clled Conceptul Bounded Disorder file, is presented, which obtins performnce similr to the bove strightforwrd solution, nd in ddition, obtins significnt cut down in min memory spce consumption. In [31] nd [11] complexity issues relted with the distribution of dictionry over fixed number of processors on network re considered. Communiction cost tkes into ccount the topology of the network, in the sense tht messge hs cost given by the number of links trversed from the source to destintion. In this cse one bsic lower bound for the communiction cost of n opertion in the worst-cse is Ω(D) trversed links, where D is the dimeter of the grph ssocited to the network. Let m be the number of informtion items stored in structure t given point in time, nd n be the fixed number of sites. The min objective in [31] nd [11] is to hve memory-blnced structures, tht is structures where the mount of storge required t the vrious sites in the system is roughly Ξ Π the sme, more precisely it hs to be O(Lod), where Lod mn =. Specil ttention is devoted to networks hving tree topology. In order to pply results for this kind of networks to networks with generic topology, the problem of embedding virtul network with tree topology in given generic network is studied. Vrious structures re proposed nd nlyzed in detil with respect to topologies of the network nd some conditions on m. Some structures present communiction complexity close to optimlity (i.e. O(D)) either in the worst-cse or in the mortized cse. In some of these cses to obtin this communiction cost results either centrl directory mnger or directory structure replicted on ech site is ssumed. Both these pproches do not llow extensions of the structures to sclble environment. Pper [12] introduces db-tree, distributed memory version of the B-link tree. A B-link tree is B + -tree where every node hs pointer to its right sibling. When node overflows due to n insertion, hlf-split opertion is performed. First the node cretes its sibling nd pss it hlf of its keys, nd second it inserts in its