Distributed Data Structures xfs: Serverless Network File System

Size: px

Start display at page:

Download "Distributed Data Structures xfs: Serverless Network File System"

Cecil Atkins
5 years ago
Views:

1 Advanced Tpics in Cmputer Systems, CS262B Prf Eric A. Brewer Distributed Data Structures xfs: Serverless Netwrk File System February 3, 2004 I. DDS Gal: new persistent strage layer that is a better fit fr Internet services separate service lgic frm durability, availability, cnsistency issues interface is that f a data structure, nt a SQL query (intentinally navigatinal) autmate replicatin and partitining acrss the cluster autmate recvery and high availability CAP: CA (nt P) -- depends n a machine rm with a redundant netwrk Nit: abstract s perf numbers have t much precisin! (shuld have been limited t 2-3 significant digits) Alternatives fr persistent strage: DBMS: chses C ver A, interface is inherently slw: parse and plan each query. Enterprise Java Beans (EJB): map Java bjects nt tables. Typical use has an array f bjects map t a table with ne rw fr each bject; rw stres the serialized bytes fr that bject => expensive cpy t get the bject in and ut f the database. Als, strage is never lcal and there is little cntrl ver partitining and replicatin (which ndes). Shuld be 10x r mre slwer, but n reliable data. File system: als has an expensive interface and very hard t prvide fine-grain atmicity. (primary atmic peratin is file rename, which implies cpying the whle file). Fine chice fr large bjects, but nt fr small nes... Internet requirements: extreme scale: billin request/day high availability unknwn but ptential fast grwth -- must be able t add capacity quickly verlad will ccur -- need graceful degradatin Cluster prperties: Incremental scalability -- add ndes ver time Ptential fr high availability -- but needs t be written! Natural parallelism fr bth I/O and CPUs High BW, lw-latency, *partitin free* netwrk (CAP) Machine rm prperties: security, administratin, reliable pwer, AC, netwrking 1

2 (these generally dn t apply t P2P systems in practice) DDS design decisins: hash table interface: 64-bit keys, byte array values, put/get/delete elements, create/ destry hash tables peratins are atmic, sequences f peratins (trans-actins) are nt tw-phase cmmit acrss all replicas fr cnsistency f writes read any replica (=> higher read thrughput than write thrughput) event-driven design fr high cncurrency. this was a mistake and led t Capricci. It is being rebuilt frm scratch in C ver Capricci. (see Java prblems belw) assume n netwrk partitins (CAP) Partitins: key idea: break hash tables int many small fixed size partitins replicate partitins (nt tables); replica grups are a set f partitins with the same data recvery: recver each partitin independently small partitins mean that yu can lck the whle partitin during recvery (ther partitins perate independently) als implies that yu can lazily recver each partitin, r yu can be practive. Fault tlerance DDS library: handles tw-phase cmmit acrss replica grup. cmmit ccurs when at least ne replica receives a cmmit message failure befre this and all replicas will time ut and discver there was n cmmit by cntacting each ther failure after this and the replicas will learn abut the cmmit frm the member that recrded it, and will then all cmmit Brick: failure befre cmmit will cause 2PC t fail and everyne abrts failure after agreement t cmmit, but befre cmmit: ther replica s cmmit, this nde will get the new value during recvery Other service cde: if all durable state is in the DDS, this is recvered autmatically n restart sft state can be rebuilt (e.g. caches) if lcal durable state (nt in DDS), then service is n its wn (e.g. might stre sme data in a DBMS als) Data Partitining Map: maps HT key t partitin id uses a trie s that it is easy t split r merge partitins as needed (increment data repartitining!) replicated n all DDS clients, eventual cnsistency (stale data causes a repair and a 2

3 retry); staleness is detected by cmparing a hash f the maps t the current values. Replica Grup Map: maps partitin id t the set f replicas replicated n all DDS clients, eventual cnsistency as abve writes are 2PC t all replicas reads g t ne replica. which ne? want t spread ut reads in the same way t maximize effective cache space! Recvery: Nde failure takes dwn all bricks n that nde; brick failure takes dwn all partitins n that brick a recvering brick must catch up all its partitins key idea 1: allw a brick t say n -- this simplifies the cde greatly! DDS library will retry in a bit. NO means that brick can be incnsistent fr a while (but nt t lng) all cmmitted peratins must use up-t-date maps (DP and RG), else retry bricks catch up ne partitin at a time: just cpy that partitin frm anther replica. Updates t a partitin stp during recvery, but this is OK but partitins are small! Must decide hw practive t be in recvery; OK t be cmpletely lazy (wait fr partitin t be accessed, but hurts latency) see recvery graph (figure 8) Perfrmance: great scalability, availability, graceful degradatin fr reads (less s fr writes) Prblems: Java GC can screw up perfrmance! One a nde gets behind it needs mre memry (fr grwing queues), s it tends t get wrse and wrse! Need admisstin cntrl, cntrl ver GC, r prbably bth. SEDA wuld prbably fix this... Events make this much wrse! Event driven systems (in Java) create a lt f garbage, as events are just passed up thrugh layer and then GC d. Threads wuld have mst f this state n the stack and thus create very little gargage... (this wuld als be a bad place fr functinal languages, which need t cpy a great deal) serius prblems with Java: GC issues, extra cpying, asynchrnus I/O... We later fixed async I/O, but the ther tw issues remain. In general, Java is t high level fr this kind f wrk... II. xfs Key backgrund: RAID small write prblem LFS fr basic servers Zebra fr netwrk striping, but replace centralized manager with a distributed manager, and use a parallel cleaner and stripe grups (rather than stripe acrss all disks) 3

4 Multiprcessr cache cnsistency: aggressive lcality explitatin Three prblems t slve: scalable, distributed metadata cache cherent recnfigurable after failure ( a level f indirectin!) Scalable way break strage servers int stripe grups Scalable lg cleaning Philsphy: anything, anywhere. All parts f the file system can be anywhere: lcatin independent Any nde can take ver fr any ther lcality determines lcatin... Fur basic mdules: client strage server manager: fr each file (index number), tracks cache inf and metadata (inde) cleaner Map 1: Manager Map: glbally replicated map index number t manager (which stres the imap fr that index number), i.e. which machine manages this file? 4

5 can use t balance metadata lad Map 2: Imap: split acrss managers maps index number t inde (i.e. where n disk is the inde?) Map 3: File Directries: maps file names t index numbers stred in regular files (just like previus file systems) Hw t pick new index numbers when create a file r directry? a) pick randmly fr lad balancing, b) pick lcally fr better lcality. xfs des the latter. Map 4: Stripe Grup Map partitins strage servers int grups glbally replicated maps stripe-grup IDs t a list f strage servers Implies a disk address = (grup ID, segment number, ffset) Grups can becme bslete, which allws GC r grups rather than having t update all f the disk addresses (everywhere) Grups have better fault tlerance than striping withut grups (why and at what cst?) Caching 1. lcal blck cache at client 2. smene else s cache (knwn by the manager) 3. inde cache at manager Deltas: critical sectin fr atmic writes Since we write data and metadata t different servers, we can t ensure that they happen atmically (as a grup) Deltas just allw the client t mark the bundaries f an atmic p, s that n replay it will ignre partial deltas Cperative caching: manager tracks list f readers write causes invalidatin f all readers, acquires write permissin read frm different client cnverts writer t reader and adds new reader t list Recvery: starts at strage servers Rebuild fragment ID t lcatin map frm checkpint + rll frward 5

6 Stripe ID (and members) in every fragment ifile stres imap n disk, but it has t be in a knwn place since it s needed fr recvery (list it in a checkpint) Rllback t last checkpint, lad imap frm ifile Then rll frward thrugh client lgs t finish imap changes (using deltas) Find cnsensus n which manager handles which clients lgs... 6

CS510 Concurrent Systems Class 2. A Lock-Free Multiprocessor OS Kernel

CS510 Concurrent Systems Class 2. A Lock-Free Multiprocessor OS Kernel CS510 Cncurrent Systems Class 2 A Lck-Free Multiprcessr OS Kernel The Synthesis kernel A research prject at Clumbia University Synthesis V.0 ( 68020 Uniprcessr (Mtrla N virtual memry 1991 - Synthesis V.1