Partitioning vs. Replication for Token-Based Commodity Distribution

Transcription

1 Parttonng vs. Replcaton for Token-Based Commodty Dstrbuton 8 XUdHWLQWHPHO Insttute for Advanced Computer Studes, Department of Computer Scence, Unversty of Maryland, College Park ugur@cs.umd.edu Banu Özden Bell Laboratores, Lucent Technologes ozden@research.bell-labs.com Mchael J. Frankln Computer Scence Dvson, EECS Unversty of Calforna, Berkeley frankln@cs.berkeley.edu Av Slberschatz Bell Laboratores, Lucent Technologes av@research.bell-labs.com Abstract The prolferaton of e-commerce has enabled a new set of applcatons that allow globally dstrbuted purchasng of commodtes such as books, CDs, travel tckets, etc., over the Internet. These commodtes can be represented on lne by tokens, whch can be dstrbuted among servers to enhance the performance and avalablty of such applcatons. There are two man approaches for dstrbutng such tokens replcaton and parttonng. Token replcaton requres expensve dstrbuted synchronzaton protocols to provde data consstency, and s subject to both hgh latency and blockng n case of network parttons. On the other hand, token parttonng allows many transactons to execute locally wthout any global synchronzaton, whch results n low latency and mmunty aganst network parttons. In ths paper, we examne the Data-Value Parttonng (DVP) approach to token-based commodty dstrbuton. We propose novel DVP strateges that vary n the way they redstrbute tokens among the servers of the system. Usng a detaled smulaton model and real Internet message traces, we nvestgate the performance of our DVP strateges by comparng them aganst a prevously proposed scheme, Generalzed Ste Escrow (), whch s based on replcaton and escrow transactons. Our experments demonstrate that, for the types of applcatons and envronment we address, replcaton-based approaches are nether necessary nor desrable, as they nherently requre quorum synchronzaton to mantan consstency. We show that DVP, prmarly due to ts ablty to provde hgh server autonomy, performs favorably n all cases studed. 1 Introducton The prolferaton of e-commerce and wdespread access to the WWW have enabled a new set of applcatons that allow globally dstrbuted purchasng of commodtes and merchandse such as books, CDs, travel tckets, etc., over the Internet. Companes, such as Amazon.com, Expeda, etc., that support these types of applcatons are drastcally ncreasng n number and scale. As these applcatons become more popular, centralzed mplementatons wll fall short of meetng ther latency, scalablty, and avalablty demands, thereby requrng dstrbuted solutons. Conventonal dstrbuted mplementatons, however, are also not vable for these applcatons: they nherently requre tght global synchronzaton, and, thus, break down n envronments where the nature of the communcatons medum s falure-prone and unpredctable. More effectve dstrbuted solutons can be realzed by explotng two mportant characterstcs of these applcatons: Frst, they nvolve a set of commodty types wth a lmted nventory (e.g., the latest CD of Emnem, economy class tckets for a partcular flght, etc.). Second, the operatons of nterest on these tems Part of ths work was done whle the author was workng at Bell Laboratores, Lucent Technologes Inc. 7KHIXOODGGUHVVRIWKHFRQWDFWDXWKRULV8 XUdHWLQWHPHO'HSDUWPHQWRI&RPSXWHU6FLHQFH8QLYHUVLW\RIDU\ODQG&ROOHJH Park, MD 2742, USA. Tel: (31) Fax: (31) E-mal: ugur@cs.umd.edu 1

2 typcally nvolve ncremental updates (e.g., buyng two economy tckets for a flght). It s, therefore, possble to acheve dstrbuton by usng tokens to represent the nstances of commodtes for sale. Prevous work (e.g., [5, 7, 11]) exploted the noton of tokens to enable hgh-volume transacton processng for dstrbuted resource allocaton applcatons. There are two fundamentally dfferent approaches for dstrbutng tokens replcaton and parttonng. Token replcaton requres expensve dstrbuted synchronzaton protocols to provde data consstency, and s subject to both hgh latency and blockng n case of network parttons or long delays n communcatons between groups of stes (whch are ndstngushable from network parttons). Token parttonng allows many transactons to execute locally wthout any global synchronzaton, whch results n low latency and mmunty aganst network parttons. The effectveness of token parttonng, however, reles on token redstrbuton technques that allow dynamc mgraton of tokens to the servers where they are needed. In ths paper, we examne the Data-Value Parttonng (DVP) [11] approach to token-based commodty dstrbuton. We propose novel DVP strateges that vary n the way they redstrbute tokens across the servers of the system. Usng a detaled smulaton model and real Internet message traces, we nvestgate the performance of our DVP redstrbuton strateges by comparng them aganst a prevously proposed scheme, Generalzed Ste Escrow () [7], whch s based on replcaton and escrow transactons [1]. generalzes prevous escrow algorthms for replcated databases, provdng hgher server autonomy and throughput. The man contrbutons of the paper are twofold: Frst, we extend the prevous work on DVP by proposng new token redstrbuton strateges and evaluatng ther performance under a range of workload scenaros. Second, although the basc approaches, DVP and, were both developed a number of years ago, ths paper s the frst to drectly compare ther performance. Thus, ths paper provdes valuable nsght nto the fundamental tradeoffs between parttonng and replcaton for the ncreasngly mportant problem of token-based commodty dstrbuton. The remander of the paper s organzed as follows. In Secton 2, we brefly descrbe the system model and the base and DVP algorthms. In Secton 3, we propose several token redstrbuton strateges for DVP. We descrbe the expermental envronment and methodology n Secton 5 and present our expermental results n Secton 6. Fnally, we dscuss related work n Secton 7 and offer concludng remarks n Secton 8. clent clent clent www gu server clent 2 Overvew of token parttonng In ths secton, we frst defne our system model. We then brefly gve an overvew of the basc DVP algorthm. For brevty, we leave asde many sgnfcant propertes such as recovery and concurrency control, focusng mostly on the performance-related ssues. Detals of DVP can be found [11]. 2.1 System model clent www gu server www gu server clent clent Fgure 1: System model www gu server server-server: token request/response perod/aperodc state nfo. clent-server: commodty request/response Our reference system model, llustrated n Fgure 1, conssts of a set of servers and clents that communcate va message exchange over a wde-area network. We assume, for smplcty of exposton, that the system stores a sngle commodty wth a lmted number of ndstngushable nstances and we represent each such nstance wth a sngle token. Through a web-based nterface, clents submt transactons that allocate tokens or return (.e., deallocate) tokens that have prevously been allocated. Therefore, the types of transactons that we model nvolve ncremental updates to a data tem, aval, whch denotes the number of tokens globally avalable n the system. We also assume, wthout loss of generalty, that there s a lower-bound constrant on aval: the system must ensure that aval does not become negatve at any tme (e.g., tckets for a partcular show should not be oversold). Therefore, all token return transactons can potentally commt (as they cannot volate the lower-bound constrant), whereas only some of the token allocaton transactons can commt. 2

3 Borrower server s b : 1. Lock tok b. 2. If tok b < req(t) borrow_tokens() /* borrow_tokens() gathers tokens from other stes and updates tok b as t receves tokens. */ 3. Wat untl tok b req(t) or tmeout. 4. If tmeout 5. Abort; unlock tok b; and ext. 6. Set tok b = tok b req(t). 7. Commt; and unlock tok b. Lender server s l : 1. Lock tok l. 2. Calculate the number of tokens, resp(s l, s b), resp(s l, s b) tok l, to be lent to s b; Send resp(s l, s b) tokens to s b; Set tok l = tok l resp(s l, s b); 3. Unlock tok l. Fgure 2: The Generc DVP algorthm 2.2 The DVP approach to token parttonng DVP [11] s a non-tradtonal approach for representng and dstrbutng data. It essentally apples to data tems that can be parttoned nto smaller peces such that the peces can also be regarded as nstances of the orgnal data tem. The same operators that apply to the orgnal tem should also apply to the peces (e.g., ncrement, decrement, set to zero, etc.). The basc dea underlyng DVP s to splt up the values of database tems and store each of the consttuent values (.e., tokens) at dfferent servers. Transactons are then executed locally at each server usng the tokens locally avalable at that server. Only n the event that the number of tokens locally avalable s nsuffcent to execute the transacton, the server makes requests to other servers only to borrow some ther tokens. If responses from other servers fal to arrve for any reason wthn a specfed tmeout perod, the transacton s aborted. Tokens are locked before beng accessed, however, only at the server where they resde,.e., no lock requests are made to other servers. The basc DVP algorthm s presented n Fgure 2. We refer to the number of tokens avalable at s as tok. The number of tokens requred to execute transacton t s req(t), whch s a negatve value f t s a return transacton. DVP s a fully decentralzed scheme that does not requre global synchronzaton. Due to ts non-blockng behavor, t s mmune to network parttons, and s thus partcularly well-suted for envronments wth unpredctable and falure-prone communcatons (e.g., many servers on the Internet) due to the hgh server autonomy t enables. 3 Token redstrbuton strateges Token parttonng enables servers to execute transactons locally as long as they have suffcent tokens. When a server cannot execute a transacton locally due to nsuffcent number of tokens, t must be able to locate tokens avalable at other servers and acqure them n a tmely fashon n order to contnue transacton executon. The performance of DVP reles upon how effectvely ths token redstrbuton among servers s accomplshed. The man questons that a token redstrbuton strategy needs to answer are: 1. when to request tokens, 2. whch servers to request tokens from, and 3. how many tokens to request. It s possble to construct a cost functon wth a set of constrants and solve t to fnd the optmal token redstrbuton. Unfortunately, not only t s dffcult to construct a realstc cost functon due to the dynamc, dstrbuted nature of the system, and the fact that tokens are pershable resources (.e., they cease to exst after beng used), but also t s long known that even smpler formulatons of the problem are NP-hard [4, 5]. Snce optmal solutons are mpractcal, we focus on heurstcs-based solutons n the rest of the paper. In partcular we avod global strateges that requre tght synchronzaton, and concentrate only on decentralzed strateges that make progress usng only par-wse synchronzaton. Note that prevous work on DVP [11] focused on the basc features of parttonng and gnored token redstrbuton ssues. In the rest of the secton, we descrbe several token redstrbuton strateges. We refer to the number of tokens requested by s b from s l as req(s b, s l ), the number of tokens returned by s l to s b as resp(s l, s b ). 3.1 Random redstrbuton We begn by descrbng our baselne strategy, BASIC. In ths strategy, the borrower contacts a lender server, whch the borrower pcks randomly, and requests the exact number of tokens needed: req( sb, sl ) = req( t) tokb If s b does not receve the entre amount t needs, t then randomly chooses another lender server and requests the remanng amount. The lender computes the return amount as: req( sb, sl ), f req( sb, sl ) tokl ; resp( sl, sb ) = tokl, otherwse. The messages exchanged among servers are mnmal, and only nclude token requests and responses. 3.2 Token count-based redstrbuton The BASIC strategy makes blnd redstrbuton decsons, snce t does not mantan or utlze any nformaton about the states of other servers. The 3

4 strategy attempts to make more ntellgent decsons by ncorporatng knowledge of the token counts at other servers nto ths decson process. The state mantaned by a server s s bascally a token table (tt) that stores an estmate of the number of tokens avalable at all servers;.e., tt[j]=(tok j, tstamp j ), where tok j s the token count at s j at logcal tme tstamp j, j=1 n, and n s the number of servers. makes more ntellgent redstrbuton decsons at the expense of mantanng and dssemnatng token count nformaton. Servers dssemnate token count nformaton usng pggybackng and broadcastng. Each server, when sendng a message to another server, also ncorporates ts token table nto the message. In addton to ths pggybackng, servers broadcast ther states to other servers at the followng crtcal ponts: 1. when the number of tokens avalable at a server becomes less than a certan threshold value so that servers wth fewer tokens can be dfferentated from the ones wth many more tokens; 2. when a server runs out of tokens so that other servers do not make token requests to ths server anymore, and; 3. when tokens are returned at a server that prevously had run out of tokens so that other servers may resume requestng tokens from ths server. A server s updates ts token table when s commts a transacton t: tok = tok req( t) and tstamp = tstamp + 1 and, when s receves a token table from server s k, k : k tok = tok and tstamp = tstamp k, f tstamp k > tstamp j j j j j j j = 1... n, j. A borrower server, s b, consults ts state table when choosng a lender server. It (rank) orders the servers accordng to ther token counts. The lender servers are then chosen based on ther ranks: at each step, a mnmal set of lenders, L, that together have a suffcent number of tokens s chosen as lenders. Formally, L S { s b } s chosen such that: b tok req( t) tok, and l L l L s. t. L S { s }, tok b req( t) tok, and L < L b l b l L where S s the set of all servers (see Secton for the other lender selecton schemes). The borrower then contacts the lenders n parallel, wthout watng for reples. The borrower makes a pessmstc assumpton about the avalablty of tokens at lender servers and requests req(t)-tok tokens from each lender. If the estmated token count of a server s zero, then that server s not contacted at all. The redstrbuton at a lender proceeds smlar to that n the BASIC case. b 3.3 Token demand-based redstrbuton In the prevous strateges, token redstrbuton occurs as the result of a token request, whch s ntated only when the borrower has nsuffcent tokens to execute a transacton successfully. The strategy, on the other hand, contnually redstrbutes tokens across servers based on the token request rates at each server. Such a demand-based redstrbuton s lkely to be benefcal especally when there s a skew n server workloads. Each server mantans a smple token request rate value, rr j, for each server s j, j=1 n. Ths value ndcates the number of tokens requested from a partcular server durng a certan perod of tme. Each server updates ts own request rate value and dssemnate t along wth ts token table usng pggybackng and broadcastng as descrbed before. SHARE/PRETECH employs two key technques that utlze request rate values of the servers: 1. Token sharng refers to the redstrbuton of tokens based on the token request rates as observed by the nvolved servers. The borrower server s b sends ts token request rate, rr b, along wth ts token request. The lender server s l computes the number of tokens that t should share wth s b as: rr b share( sl, sb ) = c tokl rrb + rrl amng to acheve a balanced token redstrbuton accordng to relatve request rates (where c s the sharng constant). Server s l then returns: share( sl, sb ), f share( sl, sb ) req( sb, sl ); req( sl, sb ), f share( sl, sb) < req( sb, sl ) and resp( sl, sb ) = tokl req( sb, sl ); tokl, otherwse. 2. Token prefetchng refers to the perodc, request ratebased redstrbuton of tokens n the background. Prefetchng can potentally elmnate the need to search for tokens as part of transacton executon, thereby reducng response tme and ncreasng avalablty. Perodcally, each server contacts every other server n some prefxed order (n the background). When s contacts s j, the tokens avalable at s and s j are redstrbuted among the two servers as follows: rr tok = ( tok + tok j ) rr + rrj where tok and tok = ( tok + tok ) tok j j tok j are the respectve token counts at s and s j after redstrbuton. Although the token redstrbuton mechansm we descrbed operates n a par-wse fashon and uses only the nformaton that s avalable at the two servers redstrbutng ther tokens, t s qute robust n that t n- 4

5 crementally mgrates the exstng global token dstrbuton n the system to match the global relatve request rate dstrbuton. Furthermore, we expermentally proved that, usng ths par-wse mechansm, any exstng token dstrbuton converges to any desred global dstrbuton exponentally fast. Fgure 3 demonstrates ths exponental convergence by plottng the number of par-wse token exchanges (between randomly selected servers) versus the sum of errors between the global target token dstrbuton and the exstng token dstrbuton (averaged over 1 randomly selected target and ntal token dstrbutons for a system wth 1 servers). A smlar par-wse exchange mechansm has been proposed [2] n the context of Deno, an object replcaton system based on an epdemc weghted-votng protocol, n order to mgrate exstng weght dstrbutons to target weght dstrbutons. 3.4 Prmary-based, hybrd redstrbuton Our prelmnary experments revealed that, as the number of tokens n the system decreases, t becomes ncreasngly harder to locate and gather the necessary number of tokens n a dstrbuted fashon. Borrower servers typcally make several unsuccessful attempts untl they are able to gather the tokens they need. Even though there mght be suffcent tokens globally avalable n the system, dentfyng the servers that have the tokens can be very costly, especally f the system conssts of many servers. In such cases, transacton executon costs can become so hgh that the performance benefts of usng a dstrbuted system may be overshadowed. The strategy attempts to add the benefts of usng a centralzed model for stuatons where only a small number of tokens reman n the system. The system operates n regular, decentralzed mode (usng the strategy) as long as the number of tokens avalable s above a fxed threshold value, but swtches to a centralzed mode of operaton when the total number of tokens drops below ths threshold. In ths strategy, each commodty s assgned a prmary server that s responsble for satsfyng all the requests nvolvng the tokens of a partcular commodty when the system operates n the centralzed mode. Each server contnuously observes the global number of avalable tokens for the commodtes for whch t serves as the prmary. If the number of tokens drops below a partcular callback threshold value, the correspondng prmary ntates the swtch to the centralzed mode by broadcastng callback messages. Each server, havng receved a callback message for a commodty, sends all the tokens of that commodty to the prmary. After the callback, the prmary executes all requests nvolvng that commodty locally. When a non-prmary receves sum of errors a token request after a callback, t smply forwards the request to the prmary, whch executes the request and returns the result back to the clent. 1 If the number of tokens later ncreases above the callback threshold, the system swtches back to ts decentralzed operaton: the prmary redstrbutes the tokens t has to other servers (unformly or dependng on ts knowledge of the request rates at servers). 4 Algorthms for token replcaton exponentally ft curve 1 servers number of par-wse token redstrbuton sessons Fgure 3: Incrementally convergng to global target token dstrbutons va par-wse token exchanges We now brefly descrbe the Generalzed Ste Escrow () scheme [7], an effcent replcaton scheme based on escrow transactons [1], as the representatve replcaton-based approach to dstrbuted token mantenance. In, the number of tokens avalable n the system, referred to as aval, s replcated at all servers. The escrow quantty at s, whch represents the number of tokens that s can dspense wthout contactng other servers, s computed as: aval es = n where aval s s s vew of aval, and n s the number of servers. Each server perodcally broadcasts the token allocatons t performed to lmt the extent to whch vews of aval s out-of-date. The escrow quantty as each server, therefore, dynamcally decreases as tokens are allocated. Each server estmates the escrow quanttes at other servers, whch are then used to replensh ts own escrow quantty and allocate tokens wthout contactng other servers, f possble. In order to mplement ths scheme, reles on: (1) gossp messages, whch are perodc background 1 An alternatve model, whch assumes the exstence of a set of forwardng agents, such as cluster DNSs that translate logcal names nto the IP-addresses of one of the servers [3], enables clent requests to be submtted drectly to the correspondng prmary. Ths latter model elmnates the (useless) ntal request to a non-prmary server. 5

6 messages that nclude the token allocatons known by the sender server, n order to lmt the global token allocatons unknown to a server, and; (2) quorum lockng to lmt the number of token allocatons that can be performed concurrently at the quorum servers. A server s can perform token allocatons as long as es, whch s updates dynamcally t allocates tokens and receves gossp messages, s large enough. In case es s not suffcent, s needs to contact other servers and form a synchronous quorum of servers such that the combned escrow quanttes of the quorum servers are enough to execute the transacton. Formng a quorum of servers nvolves remotely lockng the aval values at the quorum servers by sendng lock/unlock messages. Such remote lockng nvolvng multple servers, as our results wll demonstrate, s relatvely expensve to perform n wde-area and s the man performance bottleneck of replcaton-based approaches such as. Fgure 4 depcts the basc algorthm. Intally, only s s n the quorum of t (denoted by Q t ). Step 2 computes the escrow quantty es. The tght synchronzaton among the servers n Q t enables s to use the escrow quanttes of the other servers n Q t. Snce s s not up-to-date regardng the token allocatons performed by the other servers n the system, t, thus, makes a conservatve estmate regardng ts escrow sze. Server s makes ths estmate by placng an upper bound on the escrow values assumed by non-quorum servers; thereby guaranteeng that smultaneous token allocatons do not cause a lower-bound constrant volaton. In order to make ths estmate, s computes U, whch denotes the set of token allocaton transactons t u such that t u s known to s, and s s not sure that t u s known to all the non-quorum servers. The sum req( t) + req( tu ) tu U provdes an upper bound on the number of token allocatons that mght be unknown to non-quorum servers. The nequalty n Step 4 checks whether all but (at most) the last es req(t) allocatons known to s are known to all the non-quorum servers. If ths nequalty evaluates to true, t means that es s nsuffcent; addtonal servers need to be added to Q t. Ths s accomplshed by makng (remote) lock requests to nvolved servers regardng ther vews of aval, aval j for s j ). Durng ths lockng phase, the vews of aval at the quorum servers are synchronzed (.e., the aval values at the quorum servers become equal). If es s stll not suffcent and all servers are already n the quorum (Step 4), the transacton s aborted. If es s suffcent, the executon of t can proceed. After all the necessary updates and loggng are performed, t commts and all local and remote locks are released. It s mportant to emphasze that the performance of heavly depends on how up-to-date each 1. Lock aval and set Q t ={s }. 2. Set es = aval / n Q t. 3. Compute U. 4. If es < req(t) + req( t ) If Q t < n server s regardng the global token allocatons performed and the cost of synchronzaton among servers 5 Expermental envronment 5.1 Smulaton model tu U In order to evaluate the performance of and the DVP strateges, we mplemented a detaled smulaton model usng CSIM [1]. The model conssts of components that model a dstrbuted set of servers, a populaton of clents makng commodty requests, and communcaton latences among servers Server module The server module s responsble for executng clent transactons usng ether DVP or. It conssts of: a resource manager, whch schedules the CPU and dsk (usng a non-preemptve FIFO polcy); a buffer manager, whch handles transfer of data between the dsk and buffer; a communcaton manager, whch handles the passng of messages to and from the network module usng a queue to mplement ordered message retreval and processng. Every message sent or receved by the server s charged a fxed CPU cost, specfed n terms of the number of nstructons executed. No per-message-byte cost s modeled snce we assume the use of short, fxed-szed control messages; and a transacton manager, whch coordnates the executon of transactons. We now descrbe the transacton manager n more detal. The transacton manager conssts of several components. One component s the lock manager, whch handles all lockng assocated wth a gven concurrency protocol. Deadlocks are handled usng a tmeout mechansm. The transacton manager also contans several algorthm-specfc components. The escrow manager mantans all necessary nformaton and makes all the quorum-based decsons when modelng the algorthm. It also contans a gossp manager component that coordnates the sendng of gossp mes- u Lock aval j of a server s j Q t and add s j to Q t Goto Step 2. Else abort; and for all s j Q t, unlock aval j. 5. Perform requste updates 6. Commt; and for s j Q t, unlock aval j Fgure 4: The algorthm mplemented by s to execute transacton t 6

7 sages to other servers. The other algorthm-specfc component s the DVP manager, whch s responsble for mplementng the redstrbuton strateges that we descrbed. The DVP manager also contans a gossp manager component that s responsble for dssemnatng state nformaton va message exchange. Every message sent or receved by a server s charged a fxed CPU cost. No per-message-byte cost s modeled snce we assume the use of short, fxed-szed control messages Clent transacton generator module The clent transacton generator models a populaton of clents submttng transactons to the system. The model generates transactons usng exponental nterarrval tmes and submts them to the network module for delvery to a server. We model the transactons based on common characterstcs of wde-area commodty dstrbuton applcatons we dscussed n Secton 1. A clent request nvolves a sngle commodty 2 wth a specfc number of tokens (e.g., buyng two tckets for a partcular show). Infrequently, the tokens obtaned from the system are returned (e.g., the return of a book, cancellaton of a tcket). The transactons we generate, therefore, specfy a partcular commodty and a value ndcatng the number of tokens requested of that commodty. The commodty to be requested s selected unformly among all the commodtes n the database, and the number of tokens to request s chosen unformly from a gven nterval. Each successfully executed transacton potentally has a return transacton that returns the tokens obtaned by the orgnal transacton. A return transacton s submtted to the system wth a gven probablty. Return transactons, f submtted, are ssued by the clent that prevously submtted the correspondng allocaton transacton Network module The network module models Internet communcaton latences among the servers. Rather than usng a synthetc model to generate communcaton latences and nject certan falure modes (e.g., message losses, network parttons, etc.), we sampled messagng latences over the Internet. For a perod of three days, we contnuously collected traces of par-wse ICMP (.e., png) message exchange among four servers, whch are located at College Park (Maryland), Murray Hll (New Jersey), Lexngton (Kentucky) and Santa Barbara (Calforna). The traces were then formatted nto lsts whose entres are the latences of each message exchange or an 2 We do not nvestgate transactons that nvolve multple commodtes n ths work. Although t s straghtforward to generalze the algorthms for that case, such a generalzaton s beyond the scope of ths paper. ndcaton that the message was lost (approxmately 2% of the messages were lost). The server-server traces are used to drve the network component of the smulatons as follows. At the begnnng of a smulaton run, one trace s randomly (unformly) chosen for each par of servers, and one entry s selected randomly (unformly) on each trace as a startng pont. Whenever a server sends a message to another server, the current latency value for the correspondng trace s read and the current entry s ncremented. The message s delayed by ths latency value and then nserted nto the message queue of the destnaton server. If the entry ndcates a message loss, then the message s re-sent after a tmeout perod of 2 ms (twce the average round-trp tme between any par of servers). In order to model the communcaton latences between servers and clents, we used a smlar approach. We used clent machnes whose IP addresses were obtaned from the web-server traces of a large telecommuncatons company. We chose 4 machnes that were scattered over the Internet. We sent control messages from our four servers to these clents for a perod of three days, and logged the latency values observed. The communcaton between the clents and the servers conssts of a short clent message that ncludes the transacton requested and a short response message from the server ndcatng the result of the transacton submtted. Thus, the sampled latences can reasonably be used n modelng the clent-server communcaton n our envronment. The use of a clent-server trace s smlar to that descrbed above for the server-server case. The use of wde-area ICMP traces as descrbed s a reasonable technque for our purposes, snce (a) our Parameter Settng Parameter Settng Number of 1 Database sze 2 servers (commodtes) Local lock 2 (ms) Number of tokens 2 tmeout Remote lock 5 (ms) Number of clents 4 tmeout CPU speed 2 (ms) Transacton nter-arrval tme exp(1.) (msec) Dsk latency.97 (s) Update transacton 1. probablty Dsk transfer rate 4 (MB/s) Transacton sze 1 (commodtes) Buffer ht.95 Tokens requested unform(1,5) rato per transacton (tokens) Item read 1 Return transacton.1 cost (nstr.) probablty Item wrte 2 Workload skew. or 1. cost (nstr.) (θ ) Message cost 1 (nstr.) Prefetch perod 1 (s) Control message sze 64 (bytes) Callback threshold 2 (tokens) Commodty tem sze 512 (bytes) Sharng constant 1. Table 1: Prmary expermental parameters and default settngs 7

8 BASIC Number of Commtted Transactons Mean Response Tme (msec) model conssts of servers communcatng va message exchange over a wde-area network, and (b) the protocols we study only requre the transfer of control messages but not any data messages. Table 1 shows the man smulaton parameters and ther default settngs. 5.2 Methodology Tme (msec) BASIC Fgure 5: Number of commtted transactons unform, 1 servers, 1 trans/s All the results reported n the followng secton are the averaged results of ten ndependent smulaton runs. For each set of runs, the same set of ten workloads (.e., the same clents submttng the same requests at the same tme) are used when comparng dfferent approaches to ensure farness. The only factor that results n dfferent loads for the dfferent approaches s the exstence of return transactons. Snce the schedulng of a return transacton s dependent on whether or not the correspondng allocaton transacton commts, dfferent approaches wll typcally see dfferent return transactons. In partcular, the hgher the commt rate for an approach, the more return transactons that approach wll observe. Note that there are two reasons why a transacton may not commt n our envronment. Frst, a transacton may tmeout watng for a local or remote lock and abort. Second, ether there may not be suffcent tokens n the system to execute the transacton successfully or there may be suffcent tokens but the system may not be able to locate them n a tmely manner. 6 Performance experments and results Ths secton present the results of our performance experments comparng our DVP strateges and the (extended) approach. We frst dscuss two modfcatons to the basc approach. Both modfcatons sgnfcantly mproved the performance of n our experments. The frst modfcaton addresses the parallel lockng of the quorum servers. As suggested n [7], rather than requrng each server to construct a quorum sequentally, t s possble to estmate an ntal quorum sze and lock the quorum servers n parallel. If the quorum Tme (msec) Fgure 6: Mean response tme unform, 1 servers, 1 trans/s needs to be enlarged, a new set of non-quorum servers are added to the quorum and locked n parallel. The detals about quorum sze estmaton can be found n [7]. The second modfcaton nvolves the use of background messages. Recall that the performance of depends heavly on the extent to whch servers are upto-date regardng the token allocatons performed n the system. In the basc approach, servers perodcally broadcast ther states to the other servers. In order to ensure that servers have the most up-to-date nformaton possble, we modfed the algorthm so that these background messages are sent whenever an update s commtted. When a server receves such an update notfcaton, t also notfes the other servers that t s aware of that update. Whle ths approach s not lkely to be practcal for a real mplementaton, t mproves the performance of n our experments because (1) the messages are sent n the background, not n the crtcal path of updates and (2) the messages are all short control messages. Thus, the messages have lttle or no negatve mpact on update performance, whle provdng the beneft of up-to-date nformaton to all of the servers 3. By mplementng ths aperodc strategy, we gve an unfar advantage over DVP, whch uses far fewer messages for dssemnatng state nformaton, for the purposes of these experments. 6.1 Basc performance The graphs we present n ths secton demonstrate how the performance changes over tme. As to be expected, the performance of all the approaches becomes worse as the number of tokens globally avalable n the system decreases. Most of the measurements shown are cumulatve measures. For example, the response tme 3 We also expermented wth ncreasng the frequency of the perodc messages of the basc approach. The results showed, however, that for ths envronment, the aperodc approach provdes better performance wth fewer messages than the perodc one. 8

9 Number of Commtted Transactons Mean Response Tme (msec) Tme (msec) Fgure 7: Number of commtted trans. (nstantaneous) unform, 1 servers, 1 trans/s value that s plotted at tme 1, ms s the mean of the response tmes for all transactons that have commtted n the frst 1 (smulated) seconds of the experment. Whenever nsghtful, we also present nstantaneous results that depct the performance of the approaches durng a certan executon wndow. In all the experments, we fx the number of servers at 1 and the mean nter-arrval tme for clent transactons at 1 ms. At ths nter-arrval rate, all of the approaches demonstrate stable behavor Unform workload We now show the results of the algorthms under a unform workload where the transacton requests are unformly dstrbuted across servers (.e., θ =). Fgure 5 presents num-commts, the number of commtted transactons, by and the DVP strateges. The fgure reveals that the DVP strateges delver a sgnfcantly hgher num-commts than does. For all the DVP strateges, num-commts ntally ncreases rapdly. Wth each commtted allocaton transacton, however, aval, the number of tokens globally avalable, decreases. num-commts, then, settles down as very few transactons can commt due to a lack of tokens n the system. The dfferences among dfferent DVP approaches here are not sgnfcant. Fgure 6 shows the complementary resp-tme, mean response tme, results. All the DVP strateges, except BASIC, outperform throughout the entre executon range. These DVP varants manage to keep ther resp-tme qute low, whereas the performance of deterorates quckly after several hundred transactons are executed. The resp-tme of BASIC drastcally ncreases as aval decreases snce t selects the lender servers randomly and cannot tell whether a server has any tokens or not. The other DVP strateges do not suffer from the same problem, achevng much better resptme values. The approach, as explaned n Secton 4, operates by formng a quorum of servers. It therefore requres tght synchronzaton among the quorum servers Tme (msec) Fgure 8: Mean response tme (nstantaneous) unform, 1 servers, 1 trans/s Such synchronzaton turns out to be qute costly n a wde-area envronment. The DVP approaches, however, do not requre synchronzaton: they can contnue executng transactons locally as long as they have suffcent tokens at ther dsposal. Even n the case when a server runs out of suffcent tokens and has to make requests to other servers, t never has to communcate synchronously wth multple servers. In order to gan more nsght, we present the nstantaneous num-commts results for,, and n Fgure 7. Each pont represents the number of commtted transactons durng a tme wndow (of length 3, ms) that s centered at the tme value on the x-axs. Ths fgure, n contrast to Fgure 5 that demonstrates the cumulatve behavor of the system over tme, presents the number of transactons commtted durng certan executon perods. Intally, num-commts for the DVP approaches are sgnfcantly larger than that of. As aval decreases rapdly, num-commts also decreases. After about 15 seconds, s able to commt more transactons than the DVP approaches. Ths s because, durng that executon range, aval for the DVP approaches s much smaller than aval for, makng t (relatvely) more dffcult for the DVP approaches to commt transac- Number of Messages BASIC Tme (msec) Fgure 9: Number of messages sent unform, 1 servers, 1 trans/s 9

10 Number of Commtted Transactons Number of Local Transactons Tme (msec) Fgure 1: Number of commtted transactons skewed, 1 servers, 1 trans/s tons. Eventually, num-commts for the DVP approaches drop close to zero, as aval goes to zero. At the end of the executon range shown, aval s stll hgh for, so can stll contnue to commt transactons. Correspondng nstantaneous resp-tme results are shown n Fgure 8. The fgure demonstrates that both DVP strateges delver better response tmes than durng all executon perods. Note the dfference between the shapes of the resp-tme curves of and the DVP approaches. The resp-tme of the DVP approaches ntally ncreases as the number of locally avalable tokens decreases, reachng a peak. The resptme values then begn to decrease as num-commts decrease and aval becomes so small that most of the transactons that commt are local ones (whch have low response tmes). It can be seen that performs slghtly better than. The resp-tme for, however, ncreases gradually as the quorum szes ncrease wth decreasng aval. A closer look at the raw results clearly demonstrates the fundamental drawbacks of : (1) cannot execute as many transactons locally as DVP approaches do, and (2) the sze of the quorum that a server has to form also ncreases as aval decreases., therefore, not only has to contact other servers most of the tme, but also has to lock more and more servers as aval decreases, aggravatng ts neffcency. The mean quorum sze for, whch s the average number of quorum servers per transacton, s 1.1 n the ntal 1 seconds, 2.1 n the perod of [9,1] seconds, and 3.7 n the perod of [19,2] seconds. Another problem of s ts hgh abort rate, whch occurs manly due to tmeout n lock wats. Lock wats tend to become a serous problem n a wde-area system as the communcaton latences are unpredctable. DVP approaches, on the other hand, can execute transactons locally most of the tme, achevng hgher commt rates and lower response tmes by avodng latences and delays typcally encountered durng nterserver communcaton Tme (msec) Fgure 11: Number of locally executed transactons skewed, 1 servers, 1 trans/s Fgure 9 presents num-msgs, the number of messages sent, for all of the approaches studed here. Note that ths metrc ncludes all messages exchanged durng the operaton of the system. sends a large number of messages compared to the DVP strateges. Even wth ths large volume of gosspng among servers, cannot perform comparably to DVP. BASIC ntally sends fewer messages than the other DVP strateges, as t does not dssemnate state nformaton. As aval decreases, however, BASIC begns to suffer from ts lack of nformaton about the states of other servers, havng to choose the servers to ask for tokens randomly. The other DVP strateges make smarter decsons about whch servers to contact (or not to contact) and succeed n lmtng ther num-msgs. Beyond a certan pont, uses slghtly fewer messages than the other DVP approaches because once the system swtches to centralzed mode, all the tokens are handled by a sngle server, avodng the need to contact multple servers to gather tokens Skewed workload In ths secton, we study the case where servers receve transactons accordng to a hghly-skewed Zpf dstrbuton (.e., θ =1), as opposed to the unform dstrbuton case studed before. We present the num-commts acheved by and the DVP strateges under the skewed workload n Fgure 1 (we drop BASIC from presentaton n the rest of the paper, as t s consstently outperformed by the other DVP approaches). Comparson of these results wth those for the balanced case mmedately shows that (1) all approaches are negatvely mpacted by the hgh skew n the workload, (2) DVP approaches stll sgnfcantly outperform, and (3) the strategy acheves the hghest num-commts. For all the approaches, a few popular servers perform most of the token allocatons due to the skew n the workload. In, although the escrow szes vary dynamcally, the popular servers exhaust the tokens n 1

11 Number of Commtted Transactons Mean Response Tme (msec) Number of Servers Fgure 12: Number of commtted transactons unform, vared servers, 1 trans/s ther local escrows rapdly, havng to form quorums most of the tme. In DVP, the popular servers consume ther local tokens quckly, and then have to contact the less popular servers n order to obtan tokens. Comparng the ndvdual DVP strateges, observe that acheves the hghest numcommts. Ths superor performance s due to ts ablty to redstrbute tokens dynamcally across servers va sharng and prefetchng based on request rates;.e., by contnuously transferrng tokens from the less popular servers to more popular ones. and provde vrtually equvalent performance. Fgure 11, whch shows the number of transactons that are executed locally wthout contactng other servers, further justfes ths behavor. In terms of response tme also outperforms the other approaches for most of the experment (not shown). The results of the experments presented so far reveal that: (1) the proposed DVP strateges sgnfcantly outperform under varous workloads; (2),, and outperform BASIC manly because they explot state nformaton; (3) when there s a balanced workload, and demonstrate the best performance, whle demonstrates somewhat better response tmes; and (4) under skewed workloads, compares favorably to other schemes as t adaptvely redstrbutes tokens based on the loads on servers. 6.2 Scalablty Varyng number of servers We now nvestgate how the approaches perform as we scale the number of servers n the system. All the results presented here are the mean values of the relevant metrcs at 2 seconds. The choce of 2 seconds s not arbtrary. In fact, at around 2 seconds, the results for the commt rate and response tme related metrcs tend to stablze for all approaches. As before, we fx Number of Servers Fgure 13: Mean response tme unform, vared servers, 1 trans/s the transacton nterarrval tme at 1 ms. We present only the num-commts and resp-tme results under the unform workload. Fgure 12 shows the num-commts when varyng the number of servers. All approaches presented are negatvely affected as the system sze s ncreased. The reasons for ths result are twofold. Frst, the workload conssts of update transactons only. Second, at the request rate for whch we present the results, the performance of the system s not CPU or I/O bound, but s largely synchronzaton bound. The well-known benefts of usng a dstrbuted system to dstrbute the workload, thus, are not apparent here. The num-commts value for drops drastcally wth ncreasng number of servers. We observe that num-commts for always les below those of the DVP strateges. The DVP strateges scale better; wth beng the one wth the best scalablty as t s qute robust for regons wth small aval regardless of the number of servers n the system. and perform smlarly to each other. Fgure 13, whch presents the resp-tme results, shows that has the best response tme for the case where there are two servers. Its performance, however, degrades and falls behnd those of the DVP approaches very rapdly wth ncreasng number of servers. In addton, ts num-commts s below those of the DVP strateges for all system szes shown (ncludng the two-server case). Ths s not surprsng because as the number of servers ncreases, synchronzaton among them becomes ncreasngly more dffcult and expensve. The DVP results are consstent wth the correspondng num-commts results Varyng transacton rate Next, we nvestgate the performance of the approaches as the transacton nterarrval tme s vared. Smlar to the prevous case, each pont plotted s the mean value of the correspondng metrc at 2 seconds. As before, we fx the number of servers at ten. 11

12 Number of Commtted Transactons Transacton Interarrval Tme (msec) Fgure 14: Number of commtted transactons unform, 1 servers, vared transacton rate We show the num-commts results for the and the DVP strateges n Fgure 14. It can be seen that for low transacton nterarrval tmes (.e., hgh transacton arrval rates), num-commts acheved by and the DVP approaches deterorates drastcally. The system cannot sustan nterarrval tmes lower than about twothree ms, below whch the performance worsens substantally. For these and lower values, the state of the system changes very rapdly and the state nformaton mantaned at each server becomes outdated quckly. Another factor contrbutng to low num-commts s the ncreased lock contenton at servers whch results n a hgh number of transacton aborts. All systems recover as the nterarrval tmes are ncreased, and, n fact, ther performance does not mprove sgnfcantly above an nterarrval tme of fve ms. We observe smlar results for all the DVP strateges. Examnng Fgure 15, we observe (relatvely) hgh resp-tme values for the DVP strateges for nterarrval tmes below three ms (for whch the system s not stable). Although seems to do a better job n lmtng ts resp-tme values for nterarrval tmes below three ms, ts num-commts s much lower than those of the DVP strateges for all the nterarrval values shown. 6.3 Other experments In the remander of the secton, we brefly dscuss addtonal experments that we conducted n order to explore potental mprovements to the parttonng strateges dscussed earler Load balancng Comparson of the results for the balanced and skewed workload cases reveal that the DVP approaches suffer, although not as much as, from load mbalance. We, therefore, nvestgated the potental benefts of employng a smple load balancng mechansm to dstrbute the load evenly across servers. In ths mechansm each clent, rather than submttng the transacton to ts closest server, randomly pcks a server to submt Mean Response Tme (msec) the transacton 4. Servers use the strategy or any other strategy for that matter wthout any modfcatons. The advantage of such load balancng s that the workload seen by each server wll (on average) be smlar. In such a case, as we observed from the unform workload results, the DVP approaches wll demonstrate sgnfcantly better performance. The downsde s that such a randomzed selecton elmnates the potental gans of submttng the transacton to the closest server. Ths restrcton potentally results n ncreased clent-sde response tmes. Expermental results demonstrate that ths s ndeed the case: Our smple load balancng technque acheves better server-sde response tmes than the no-load balancng strategy. In fact, t acheves results qute smlar to those of for the unform workload case. Clent-sde response tmes, however, turn out to be smlar, or slghtly better for the no-load balancng case, demonstratng the aforementoned tradeoff Non-determnstc selecton of lenders The DVP strateges presented are determnstc n that they use the estmated token counts at servers to choose the lender servers. Ths determnstc selecton may lead to a stuaton where, for a gven perod of tme, most of the token requests are targeted to the same server, makng that server a hot-spot. Our experments showed that, due to the dfferences among the vews at dfferent servers, such stuatons do not occur very frequently: the probablty s no more than 1% and 3% hgher than the random selecton case on average for the unform and skewed cases, respectvely. In order to elmnate such stuatons completely, we studed two randomzed schemes. The frst scheme chooses the lenders randomly wth equal probablty (among those wth a non-zero token count). The other 4 Note that ths s a very practcal and easy-to-mplement scheme Transacton Interarrval Tme (msec) Fgure 15: Mean response tme unform, 1 servers, vared transacton rate 12

13 scheme s based on the dea of lottery schedulng [13], where lenders are chosen wth a probablty proportonal to ther token counts. Experments reveal that the lottery-schedulng scheme performs somewhat better than the completely random scheme, achevng commt rates smlar to those of the determnstc scheme, whle sufferng less than a 2% deteroraton n response tme under both the unform and skewed workloads (usng ). 7 Related work Most prevous related work focused on explotng applcaton semantcs to mprove performance n certan classes of applcatons. O Nel proposed Escrow transactons [1] to enable concurrent access to hgh traffc aggregate felds on whch only a restrcted class of operatons (such as ncremental updates) are allowed. Ths technque utlzes the commutatvty property of such operatons;.e., two operatons can run n any order and stll produce the same fnal result provded that they do not volate upper/lower bound constrants on the nvolved data tems. Escrow transactons execute a specal escrow operaton that attempts to put n escrow (.e., reserve) some of the resources that t plans to acqure. All escrow operatons that succeed are logged. Transactons consult ths log before executng an escrow operaton and see the total amount of resources that are escrowed by all uncommtted transactons. If the total quantty of unescrowed resources s suffcent, the transacton proceeds; otherwse t aborts. Haerder extended the escrow transactonal model for centralzed database envronments to DB-sharng systems [6] where multple DBMSs share the database at the dsk level. He proposed the use of a herarchcal escrow scheme that conssts of a global escrow and local dstrbuted escrows. He dscussed a technque that enables the herarchcal scheme to behave lke a purely global scheme for only a crtcal margn of aggregate values. Kumar and Stonebraker generalzed the noton of escrow transactons for replcated data [9]. Each server s assgned an escrow quantty that can be used to execute transactons locally. The escrow quanttes at servers are readjusted by the use of a perodc global snapshot algorthm. Ths algorthm has to be executed suffcently frequently for servers to have an up-to-date vew of the global state. On the other hand, frequent executon of such a costly algorthm may tself degrade performance. Both DVP and employ mechansms that elmnate the need for a global snapshot algorthm. The work most closely related to our nvestgaton of varous redstrbuton strateges s [8], where Kumar dscussed several borrowng polces for escrow transactons n a replcated envronment. Kumar devsed four smple borrowng polces that (1) select lender servers ether randomly or accordng to a pre-specfed order, and (2) that borrow ether the exact amount they need or borrow an amount such that the fnal escrow quanttes at the nvolved servers become equal. Note that one of the polces, n whch the lender server s chosen randomly and the exact amount needed s requested, s smlar to our BASIC strategy. Unlke our strateges, however, Kumar s borrowng polces do not make use of the knowledge of the global state of the system to mprove ts effectveness and adapt dynamcally to workload. Golubchk and Thomasan dscussed demand-drven token allocaton schemes n the context of a fractonal data allocaton method (FDA) [5]. One such scheme they descrbe enables token parttonng between the nvolved servers based on demand (as n our SHARE strategy). In [12], Thomasan further dscussed FDA and proposed an abstract model for optmal ntal allocaton of tokens, whch we do not address n ths paper. It s worth notng that no prevous work has explored the fundamental tenson between replcaton and parttonng for token-based resource dstrbuton, whch s our man focus n ths paper. 8 Conclusons Token-based commodty dstrbuton can meet the demands of a class of newly emergng Internet-based e- commerce applcatons. In ths paper, we expermentally evaluated and compared two fundamentally dfferent approaches to token dstrbuton parttonng and replcaton usng real Internet message traces. We also proposed several par-wse token redstrbuton strateges for the parttonng-based approach and evaluated them under dfferent workloads. Our experments reveal a number of sgnfcant results for token-based commodty dstrbuton. Frst, replcaton-based approaches are nether necessary nor desrable for the knds of applcatons and envronment we address n ths study. Parttonng-based approaches perform and scale better prmarly due to ther ablty to provde hgher server autonomy. Second, the use of nformaton about the system state turns out to be crucal for makng token redstrbuton decsons. Thrd, when there s a balanced load on the servers, the use of complcated redstrbuton schemes does not mert ther complexty; smple strateges can be as effectve, provded that they utlze mnmal state nformaton. Fnally, n the case of skewed workloads, however, the use of extra nformaton about request rates may yeld notable performance mprovements. In terms of future work, we are plannng to nvestgate herarchcal server organzatons, whch can mprove the performance sgnfcantly when the number of servers s large. 13