Shared (Disk) File Systems

Transcription

1 Shared (Disk) File Systems Matthew T. OÕKeefe Associate Professor Departmet of Electrical ad Computer Egieerig Uiversity of Miesota Mieapolis, MN

2 Outlie of Talk The Problem of Shared Disk File Systems Distributed File Systems Shared File Systems: Defiitio ad Key Properties Example Shared File System Desigs Global File System Ð Architecture Ð Curret Implemetatio Coclusios

3 File Systems ad Shared Disks: The Problem Shared disk etwork itercoects like Fibre Chael help solve importat techical issues Ð Itercoect legth, umber of ports, speed Ð These are of course extremely importat But there are sigificat opportuities to leverage these ew capabilities to develop SCSI-based etwork file systems These file systems could be more scalable, cheaper, more reliable tha curret etwork-attached storage that uses NFS

4 File Systems ad Shared Disks: The Problem The storage idustry will evolve SCSI towards very smart disks that do file (istead of block operatios) Ð That efficietly support cachig ad coherecy o the drive Ð That build o the iterface improvemets to allow totally ew storage architectures Ñ SMP-like multiple-cliet architectures Ð The advaced developmet ad research work i this area is happeig ow! NSIC/NASD workig group (Gibso) Compaies like Tricord out frot techically

5 The New Opportuity: Exploitig Shared Disks Shared Disks Classic NFS GFS Cliet GFS Cliet NFS Cliet CPU Memory Disks CPUs Memory Disks CPU Memory Disks GFS Cliet GFS Cliet CPU Memory Disk Storage Area Network CPU Memory Disk NFS Cliet CPU Memory Disk CPUs NFS Server Memory Disks NFS Cliet CPU Memory Disk Network Storage Pool Sub-pool 0 Sub-pool 1 Sub-pool 2 Sub-pool 3 Sub-pool 4 Solid State Sigle Disk RAID 5 Software Striped Disks RAID 3 Disk Farm

6 Explosive Data Growth Both documets ad applicatios are becomig more mediarich, drivig up file sizes Cotiued growth i capacity of memories ad disks promotes further file growth Example eviromet: digital productio houses Ð Seaker et is preferred data trasport media Ð Academy film format: about 3000x4000 pixels per frame Ð With 14-bits per RGB compoet: 42-bits per pixel Ð At 24 frames/secodñ2.0 GB for 1 secod of film Ð 30 secods Megabytes sec thru Fibre Chael

7 Potetial New Applicatios Better Web server desigs based o shared rather tha replicated disks Ð Web server desigs:

8 Potetial New Applicatios More efficiet parallel databases for data miig, other parallelquery-based applicatios Ð Sigificat advatages to shared disk i cluster eviromets Cluster-based applicatios which require high capacity ad high badwidth Ð Film ad video Ð Feature film idustry, advertisig, local TV statios Ultimately, the right way to do distributed storage across the eterprise Ð Especially with smart disks

9 Classic Cliet-Server Distributed File Systems SuÕs NFS (Network File System), Novell Netware, MicrosoftÕs La Maager Traditioal DFSÕes use cetral server approach: Ð may cliets share data through 1 cetral server Basic assumptio: disks are ot smart ad caot be attached to a etwork Geerally complex ad iefficiet from stadpoit of really large datasets Ð sychroous writes Ð cliet cachig

10 DFS Evolutio NFS got the ball rollig i the mid-1980õs Popular but is well kow to be iefficiet Ð sychroous writes ad write-through Ð statelessess meas more retries Ð protocol stack overheads NFS is popular for several reasos Ð Its sematics are fuzzy so it ca ÒworkÓ i may differet Oses Ð Brute-force hardware approaches ca be used i some cases to icrease performace

11 NFS Executio Path Cliet Server User Applicatio VFS VFS NFS Cliet NFS Server Local FS XDR RPC Cliet XDR RPC Server Buffer Cache Memory TCP IP TCP IP Memory Storage Device Drivers Network Device Drivers Network Device Drivers Host Adapter Network Adapter Network Adapter

12 Treds i Distributed File Systems AFS (DFS/DCE) ad Coda use Distributed Servers distributed ame spaces Ð redudacy Ð very aggressive cachig, loose sharig sematics Ð better scalig, usable i Wide Area Network Ð complex, difficult to cofigure, closed system Still o cocept of attachig devices directly to etworks Ð Except for iches like maiframes ad supercomputig

13 Mergig Cliets ad Servers New approaches to DFS desig allow machies to act as both cliets ad servers Merged cliet-server desigs Ð Coda ad BerkeleyÕs xfs More fuctioality migratig to cliets A atural effect give that computig systems today are drive by the desktop Cliets are gettig closer to servers i their capabilities

14 Future of Distributed File Systems NFS ad other DFSÕes rule file sharig today i LAN eviromets Ð these protocols have ot drive the Web eve though i some ways they are better protocols Ð Web techologies may displace curret LAN DFSÕes Ð DFSÕes have geerally ot exploited LAN locality to improve performace: istead focus is portability

15 Future of Distributed File Systems Traditioal distributed file systems solve a problem that o loger exists Ñ machies could ot talk to each otherõs storage devices Ð But they will cotiue to exist Ð Network-attached storage will allow NFS servers to be costructed from clusters of machies that share disks Fie-graied read/write sharig geerally ot supported Ð for example with NFS you ca ever be sure if the data you are readig is the latest copy if some other cliet has writte to that file Ð AFS has sessio sematics

16 I/O Iterfaces Chael iterfaces (e.g., SCSI) Ð Coect computers to storage devices ad other peripherals Ð High-performace, low coectivity, short coectio distaces Network iterfaces (e.g., Etheret) Ð Coects computers to other computers Ð Lower performace, high coectivity, log coectio distaces Merged iterface Ñ Fibre Chael Ð Both a chael ad etwork iterface Ð Supports storage attached to a etwork

17 Eablig Techologies Fibre Chael Ð High badwidth, low latecy etwork ad chael iterface Ð Highly scaleable, very flexible topologies Ð Becomig high-volume, hece lower-cost Ð Support from a wide-variety of adapter, computer, etworkig, ad storage vedors Network-attached Storage (NAS) Ð Have your disks ad share them too Ð Allows direct data trasfer betwee disks ad cliets Together, Fibre Chael ad NAS eable SCSI-based storage area etworks (SANs).

18 Fibre Chael Matthew T. OÕKeefe Departmet of Electrical ad Computer Egieerig Uiversity of Miesota Mieapolis, MN

19 Fibre Chael: Itroductio Fibre Chael (FC) merges features from both etworks ad chael iterfaces to create a storage iterface that is Ð Fast Fiber optic serial iterface Ð Scalable I the umber of odes, devices supported per ode, ad etwork badwidth available per ode Ð Efficiet Lots of low-level processig performed i silico Ð Ope, high-volume, idustry stadard May strog vedors ad OEMs participatig i Fibre Chael developmet ad productio Idustry groups: FCLC ad FCA

20 FC Layers FC fuctioality implemeted across multiple layers Ð Physical media ad trasmissio rates Ð Ecodig scheme Ð Framig protocol ad flow cotrol Ð Commo services Ð upper-level protocol iterfaces

21 Physical ad Sigalig Layer FC-0 Ð Covers physical characteristics of the iterface ad media icludig cables, coectors, drivers, trasmitters ad receivers Ð Sigle- ad multi-mode fiber optics 1 GHz shippig ow, up to 4 GHz defied i stadard 1000s of meters Ð Copper coax for shorter distaces (LANs) FC-1 Ð dc-balaced 8B/10B code scheme (thak you IBM) used for clock recovery, byte sychroizatio, ad ecode/decode Ð Comma character isures proper byte ad word aligmet Ð Good error detectio capabilities ad simple logic implemetatio for the ecoder ad decoder

22 FC Topologies Dedicated Poit-to-Poit Ð Direct coectio betwee sigle host ad disk F_Port Fabric F_Port Arbitrated Loop L_Port L_Port L_Port Ð N_port Shared Badwidth Arbitrated Loop Ð Hubs used to coect odes i shared loop topology N_Port N_Port N_Port Poit-to-Poit N_Port Ð NL_ports Switched Fabric for Scalable Badwidth Ð Multiple switches itercoect ito a Fabric Ð N_ports to F_ports, NL_ports to FL_ports

23 Framig ad Sigalig Layer (FC-2) Defies trasport mechaism idepedet of upper layer protocols Self-cofigurig ad supports poit-to-poit, arbitrated loop ad switched eviromets N_port is ode (server, workstatio or peripheral) Ð If port coected to a loop, it becomes a NL_port Ð Data commuicatios occur betwee itercoected ports Ð Each ode has a ASIC with a embedded FC Lik Cotrol Facility Ð Each port ca act as a origiator, respoder, or both ad has a uique idetifier: N_port or NL_port Idetifier

24 FC-2: Framig Layer Defies the framig structure ad provides the followig fuctios: Ð robust 32-bit cyclic redudacy check Ð Various classes of service to provide ad maage Circuit switchig Frame switchig Datagram services Fractioal badwidth virtual circuits Ð A flow-cotrol scheme that guaratees delivery Buffer-to-buffer Node-to-ode Ð Built i protocol to aid i maagig the lik, cotrol the FC cofig, perform error recovery, ad recover lik ad port status iformatio

25 FC Desig Philosophy Desiged to be hardware-itesive: use low-cost ICs to miimize software overheads Cotets of frame determie destiatio At gigabit speeds decisios must be made i hardware FC-2 ordered sets, frames, sequeces, exchages

26 NASD Idustry Stadardizatio efforts Disks should be smart devices (they are already pretty clever, as are HBAs) File level operatios performed at the disks Ð Rather tha blocks have objects Ð The objects are directories, files, ad objects related to recovery Joit idustry effort led by Gibso et al. Ð Seagate, StorageTek, IBM, ad others ItÕs clear SCSI will be the trasport protocol Ð This is where SCSI-4 is headed

27 Shared File Systems A architecture for distributed file systems based upo shared storage Fully exploits the special characteristics of Fibre Chael-based LANs Key feature is that cliets trasfer data directly from the device across the SAN (Storage Area Network)

28 Shared File Systems Key characteristics: Ð Hece more tha oe cliet may access the data from the same storage device Ð The device is shared betwee cliets via some kid of itercoectio etwork Ð Shared file systems must recogize the existece of other cliets accessig the same storage devices ad file system data ad metadata Directly through the metadata Through a file maager Precludes most local file systems: these cosider storage devices as owed ad accessed by a sigle host computer

29 Shared File Systems (Advatages) Advatages iclude (1) Availability is icreased sice whe a sigle cliet fails aother cliet ca still access the associated data ad cotiue the failed cliets work (2) Load-balacig a mixed workload amog multiple cliets sharig disks is simplified by cliets ability to quickly access ay portio of the dataset o ay of the disks (3) Poolig of storage devices ito a shared disk memory equally accessible to all cliets i the system is possible (4) Scalability i capacity, coectivity, ad badwidth ca be achieved without limitatios iheret i file systems desiged with cetral servers

30 Example Shared File Systems DEC Vaxcluster (mid 1980s) IBM Sysplex (maiframes) Oracle Parallel Database Server (~1990) LLNLÕs HPSS CrayÕs Shared File System (1994) Veritas Cluster File System (1998) IBMÕs Parallel Jouraled File System (1995) GibsoÕs NASD project at CMU (1995) U. MiesotaÕs Global File System (1996)

31 Classifyig Shared File Systems (SFS) Symmetric or Asymmetric? A shared file system is symmetric if Ð ay cliet ca perform ay file system operatio without ecessarily iteractig with aother cliet I asymmetric shared file systems a cliet must first make a request through a file maager executig o aother cliet Ð File maager typically maages File system metadata Checks file access permissios Provides cliet with ifo ecessary access data directly o disk via the SAN

32 Asymmetric Shared File Systems Asymmetric shared file systems replace the server with the file maager Ð cliets must access metadata through the file maager which effectively cotrols cliet access to files Ð sometime called 3rd-party trasfer: the file maager is a 3rd-party which sets up xfer betwee cliet ad device Ð oce trasfer approved, direct trasfer is possible

33 Asymmetric SFS Asymmetric shared file system advatages: Ð file maager desig will geerally re-use file server code Ð cetralized desig is simpler Asymmetric shared file system disadvatages: Ð same problem as cliet-server: server is a bottleeck to scalability (but does ease badwidth ad capacity issues) Ð must write both cliet ad file maager code Ð cetralized loggig ad lockig itroduce bottleecks Ð file maager machie is a sigle poit of failure

34 SFS Lockig Where is lockig performed? Ð i the cliets or file maager: cetralized file maager (CMU NASD, CFS, HPSS) distributed lock maager (Vaxcluster, Oracle PDS) Ð DLMÕs are otoriously difficult to desig i the cotext of cliet failures Ð i the devices or etwork: devices (GFS, Cray SFS) Ð simpler protocol to desig Ð requires pool of fie-grai locks i the device or etwork Ð these locks must be fast

35

36 IBM Sysplex Matthew T. OÕKeefe Departmet of Electrical ad Computer Egieerig Uiversity of Miesota Mieapolis, MN

37 IBM Sysplex IBM has bee clusterig maiframes for a log time JES eases task of distributig work amog a cluster of maiframes I 1994, additioal hardware ad software provided to allow more fie-grai data sharig ad parallelism Cluster of up to 32 MVS/ESA systems ruig System/390 maiframes (figure) Ð Multiple machies share data through ESCON directors Ð Provides simultaeous access to disk

38 IBM Sysplex Ð ESCON director works like Digital Star coupler ad Fibre Chael switch or hub i GFS Allows multiple simultaeous trasfers Ð ESCON is 12-Mbyte/sec coectio to S/390s Ð Sysplex timer provides commo real-time referece for all systems Cosistet data ad time stampig of trasactios File modificatios Other parallel operatios

39 Sysplex Couplig Facility Sysplex couplig facility is a hardware assist for performace ehacemet: it provides Ð Cluster-wide locks Ð Stable electroic storage for cachig Sysplex-wide ifo such as Global work queues Commo read/write data Sysplex Timer ad Couplig Facility are the ÒextraÓ hardware itroduced to improve Sysplex performace i 1994

40 Exploitig Cluster Parallelism i Sysplex Parallel applicatios executig across the cluster will exploit Couplig Facility ad Timer Pfister: Òlist of applicatios is Acroym CityÓ Ð IMS DB (Iformatio Maagemet System Database Maager) is a hierarchical database product Ð DB2 (Database 2) Ñ relatioal database product Ð VSAM (Virtual Storage Access Method) Ð RACF (Remote Access Cotrol Facility) Ñ security Ð JES2 for job etry

41 Sys Admi Tools i Sysplex System admiistratio utilities allow creatio of a sigle poit of cotrol Ð Sigle cosole from which target area ca be maaged ad admiistered for etire cluster Ð ESCON Maager moitors ad maages I/O cofiguratio ad status Ð others

42 Sysplex Summary Symmetric or asymmetric? Not clear to me from my sources Lockig? Performed o device Ñ Sysplex couplig facility or ESCON Director Ð SCSIÕs RESERVE/RELEASE is a descedat of this IBM facility for lockig devices whe multiple machies may access them Proprietary or ope storage itercoect? ESCON is basically proprietary Existig file systems modified to work with cluster

43 DEC Vaxclusters Matthew T. OÕKeefe Departmet of Electrical ad Computer Egieerig Uiversity of Miesota Mieapolis, MN

44 Vaxcluster System Developed by DEC to provide a highly available system that provides users with a sigle system image across a cluster of Vax workstatios Origial implemetatios used the CI (Cluster Itercoect) Ñ custom 70 Megabits (Mbps) etwork that itercoects both computers ad disk cotrollers Ð Star Coupler A symmetric shared file system for Vaxcluster odes Ð Lockig hadled through the DLM: Distributed Lock Maager

45 Distributed Lock Maager (DLM) DLM: a cliet-based lock maager that provides a geeralized lock service for all resources i the Vax cluster Ð Devices Ð Prit services Ð Files Ð Other abstractios defied by the OS

46 DLM Lock maager allows cliets to request ad release a lock Ð Each request specifies a lockig mode: varyig levels of exclusive cotrol o associated resource Exclusive: o other host may read or write to the resource Cocurret read access: other cliets may read or write to the shared resource

47 DLM Desig Issues DLM hadles cluster-wide sychroizatio Several goals iflueced the desig: Ð Programs usig DLM should ru i both sigle ode ad cluster cofiguratios Ð Lock services must be efficiet to support system-level software that makes frequet requests Lock maager must miimize the umber of messages eeded to maage the locks I sigle-ode eviromet the lock maager should recogize the simpler eviromet ad bypass cluster-specific overheads Ð Lock maager must recover from failures of odes holdig locks so that survivig odes ca cotiue to access shared resources i a cosistet maer

48 Distributed Lock Maager (DLM) Lock requests may be queued Ð Oce resource becomes available, the requestig cliet is so iformed via a callback Ð Implies sigificat state associated with each lock regardig the status of outstadig requests for the lock DLM is distributed across the cliets i Vaxcluster Ð Provides load balacig across the parallel resource: the locks This aids scalability Ð Locks are cached o requestig cliet if possible Ð Must work i the presece of cliet failures

49 VMS File System ad Vaxclusters Origial VMS file system was modified to use the DLM to support shared, simultaeous access to Ð Files Ð Associated metadata Directories, files, volumes Extesive cachig used i origial local VMS file system Ð This approach preserved i Vaxcluster implemetatio Ð Versio umbers were associated with lock operatios Ð Stale cache data detected as a disparity betwee versio umbers caused by earlier file update operatios performed by other cliets

50 Versio Numbers ad Cachig Example: cliets A ad B both accessig file foo Ð Assume both A ad B have the file cached ad both are allowed to read ad write the file Ð A makes first access (read) to foo: lock versio umber (v) is 1 Versio umber Ò1Ó is associated with cached data o A Ð B makes 2d access (read) to foo : lock v still 1 v Ò1Ó associated with cached data o B Ð A makes 3rd access (write) to foo: lock v icremeted to 2 v Ò2Ó ow associated with cached data o A Ð B makes 4th access (read) to foo: sice lock v is Ò2Ó but cached v is Ò1Ó, B kows it must ot use its cached copy Reads data directly from disk istead

51 Cachig Data with Deferred Writeback Software Iterrupt Optio: whe requestig a exclusive lock, process ca specify it be otified by software iterrupt Ð if aother lock request o the resource is forced to block Ð The a process owig the lock ca cache ad repeatedly modify the data Ð Data is writte back to disk ad lock released whe the lock is otified that it is blockig aother process

52 Lock Maager Features Lock maager provides a lock coversio service to chage the lockig mode o a existig lock Coversios are faster tha ew lock requests because table etry represetig the lock already exists Applicatios will frequetly hold a ull lock o a resource ad the covert it to a more restricted access mode later

53 Lock Maager Implemetatio The lock maager implemetatio iteded to Ð distribute overhead of lock maagemet throughout the cluster Ð while still miimizig the iter-ode traffic eeded to perform lock services The lock table is therefore divided ito two parts (both of which are distributed): Ð Resource lock descriptios Ð Resource lock directory system Each resource has a master ode resposible for gratig locks o the resource

54 Master Node for a Lock Ð Master maitais a list of grated locks queue of waitig requests for that resource Ð Note that the master for all operatios for a sigle tree is the ode o which the lock request for the root was made Ð The master maitais the lock data for its resource tree But ay ode holdig a lock o a resource mastered by aother ode keeps its ow copy of the resource ad lock descriptios Ð The resource directory system maps a resource ame ito the ame of the master ode for that resource The directory database is distributed amog odes willig to share the overhead associated with processig directory requests

55 Lockig a Resource Ð Give a resource ame, a ode ca trivially compute the resposible directory as a fuctio of the ame strig ad the umber of directory odes Ð To lock a resource, the lock maager seds a lock request message to the directory for that resource: Ð The directory respods i oe of 3 ways: 1) if directory is o master ode for that resource, lock request performed ad cofirmatio retured 2) if directory is ot o master ode but it fids the resource defied, it returs the idetity of the master ode 3) if directory fids resource udefied, it returs a message tellig requestig ode to master the resource itself

56 Lockig a Resource Ð I best cases (1 & 3) above, two messages required to request a lock: case 2 takes four messages Ð Ulock executed with oe message Ð If lock request is for sub-resource i a resource tree the requestig process will either Be located o the master ode (local request) or Will kow who the master for its paret is Ñ allows bypass of directory lookup Ð I ay case umber of messages idepedet of umber of machies i the cluster

57 Vaxcluster File System Summary Symmetric or asymmetric? Symmetric Lockig? Performed o cliets usig DLM Proprietary or ope storage itercoect? CI ad Star Coupler are proprietary Existig file systems modified to work with cluster

58 Cray Shared File System (SFS) Cray had a customer who required that multiple (at least 4) C90s be able to access the same data directly ad be costatly available Ð this evolved later ito a more geeral product (though to my kowledge few are istalled) This required a shared disk solutio Cray modified their ow Uicos (CrayÕs versio of UNIX) file system

59 SFS Goals Commo storage device to share file system data ad metadata High badwidth All machies i the ÒcomplexÓ are peers Ñ o server Re-use existig UNICOS file system code Works with all the stadard UNICOS utilities High availability

60 ÒSpecialÓ Versio of SFS 4 C90s shared a commo SSD (Solid State Disk) which held the file system ad metadata Separate arbitratio device to cotrol access to shared data ad metadata: Ð semaphore device kow as the SMP: serializes operatios usig atomic test-ad-set Ð SMP ca tur aroud request from C90 i 6 microsecods Ð special-purpose hardware: redudat Ñ 2048 locks Ð cofusig poit: what are usually called locks the paper calls ÒsemaphoresÓ

61 Phase 2 SFS Geeralized SFS to work with more stadard, less expesive peripherals Geerally available software i UNICOS 9.0 I phase 2 the shared media is a HiPPI disk array Arbitratio service provided by a SPARC workstatio attached to the HiPPI switch (HSMP) Ð dedicated to SMP operatio Ñ 1 millisecod turaroud Ð locks implemeted as IPI-3 disk commads

62 SFS-2 Operatio UNICOS employs typical UNIX file system desig priciples Diodes o disk reside i special Òdiode regiosó Ð diode is 256 bytes log Ð with 4096-byte disk block, 16 iodes per iode regio block Ð o RAID-5 device with 64K block size there are 256 iodes per iode block

63 SFS-2 Operatio Noe of the stadard UNIX bufferig was used i SFS Diode lockig must be doe exterally Ð differet from stadard UNIX where i-memory lock sufficiet to serialize access amog may processes I SFS, lockig accomplished both with the exteral semaphore ÒdeviceÓ (HSMP) ad by scribblig lockig iformatio ito the diode

64 SFS-2 Read May cotiguous SFS diodes map to sigle semaphore (lock) Semaphore used like a test-ad-set bit whe accessig a Òiode sectoró: a cotiguous set of diodes Ð semaphore acts like test-ad-set bit: if busy try agai later Ð if semaphore is acquired, eter ame of C90 that grabbed it i the semaphore Oce semaphore acquired, the the particular diode to be acquired must also be locked by writig ito diode fields o disk

65 SFS-2 Read If diode i sector acquired, we set lock ad write that diode sector back out to disk ad release semaphore Read of disk data the performed Followig read, repeat previous sequece to ulock the diode Ð acquire semphore Ð acquire diode lock Ð ulock ad write back diode ito Òiode sectoró

66 SFS-2 Read Lots of traffic: at least 4 HiPPI packets set per read(!) assumig o lock coflicts Diode sector must be read ad re-writte twice No bufferig as i o-shared case Sigificat performace degradatio, attack by Ð file lockig Ð large user buffers Ð reduced directory lockig Ð faster metadata devices

67 SFS-3 Operatio SFS-3 implemets series of improvemets A optimizatio proposed was to lock files i Òread-dataÓ or Òwrite-dataÓ mode (i.e. like traditioal UNIX file locks) Ð kow as Òdata locksó i SFS jargo Ð use fields i the diode itself (like the Òdiode sectoró lock) Ð lock state ad idetity of the C90 owig the lock

68 SFS-3 Operatio Multiple read locks are allowed Ð i-memory iode serializes reads from sigle C90 with multiple processes accessig same file Oly oe process o oe machie may ow a write-lock, ad the oly if there are o read locks i place

69 SFS-3 Operatio File lockig uses fctl commad (F_SETLK ad F_GETLK) Ð but i SFS these are ot advisory locks Ð always work across the whole file Lock state kept o disk ad o-processor UNICOS ca be set to always do Direct I/O Ñ as i origial SFS, whe request ot aliged UNICOS ca break up the request ito aliged ad u-aliged requests

70 SFS-3 Operatio Pre-allocatio for writes proposed to reduce metadata access ad related semaphore ops Recovery i SFS depeds o cetral shared table i HSMP device Ð oe semaphore for recovery: oly oe machie ca do recovery at ay oe time Ð HSMP maitais heartbeat status of machies i cluster Ð durig recovery semaphores held by failed cliets must be released ad disk metadata ÒfixedÓ

71 More SFS Performace Improvemets No locks o directories while beig searched Realized that lockig the diode umber i the semaphore device would be useful Ð Òiode sectoró still locked for writes Ð diode ad device umbers combied ito 32-bit umber locked i semaphore Ñ symbolic semaphores Ð keep track of machie holdig the semaphore Ñ like a versio umber but more limited i usefuless

72 SFS Differeces with GFS GFS decouples sharig from lockig also decouples sharig from clusterig GFS has o restrictios o read- ad write- sharig Diodes ot cocetrated i sigle sector o disk Ð why is this good/bad?

73 Cray SFS Summary Symmetric or asymmetric? symmetric Lockig? Performed i device or etwork switch Proprietary or ope storage itercoect? HiPPI, though ot proprietary, is low volume Existig file systems modified to work with for shared file system problem

74 NASD: Network Attached Secure Disks Matthew T. OÕKeefe Departmet of Electrical ad Computer Egieerig Uiversity of Miesota Mieapolis, MN

75 NASD Overview Gibso has proposed Network Attached Shared Disks (NASD) as a stadard for shared storage devices NASD goes beyod previous shared disk storage systems i two key areas Ð Security Ð Objects NASD-based file systems as curretly proposed use file maager for directory ad certai cryptographic operatios Ð Mechaisms provided to keep these overheads low

76 NASD Overview NASD could potetially allow symmetric shared file system desigs as well Secure commuicatios betwee disks ad cliets achieved usig capabilities that have bee cryptographically sealed Ð Support for cryptographic operatios placed o the devices

77 NASD Overview NASD systems dramatically raise the sematic level of disk drive operatios Ð From fixed-size blocks to variable-sized objects Ð Objects ca be files or directories; support for partitios (cotaiers for separate groups of files) is provided Higher sematic level for objects o devices meas that fewer disk commads eed be set over the etwork per file operatio Ð Reduces etwork overheads ad improves scalability

78 Properties of NASD (Gibso) Direct Trasfer: data accessed by a filesystem cliet trasferred betwee NASD drive ad the cliet without store-ad-forward through file server Asychroous Filesystem Oversight: file maager retais some file server fuctioality as it relates to filesystem policies Ð Goal is to reduce overheads of additioal checks required i shared storage eviromet (i.e., ca this cliet access data o this drive?) Ð Offloads this work to the drive to keep load small o file maager as more cliets ad devices added to system

79 Properties of NASD (Gibso) Cryptographic support for request itegrity: NASD drives must be capable of computig keyed digests o commad ad status fields with little or o performace pealty Ð Required sice o sychroous filesystem oversight ad o assumptio etwork betwee cliets ad disk is secure Ð Caot deped o trusted cliets Ð With hardware support for cryptography i a drive, highperformace data itegrity, data privacy ad commad privacy are feasible

80 Properties of NASD (Gibso) Storage Self-Maagemet: NASD drives kow a lot more about the ature of the data that they hold due to the higher level of abstractio (compared to SCSIÕs liear array of radomlyaccessible blocks) Ð Hece blocks associated with a sigle file i a object ca be kept cotiguous Ð Access patters to file objects as a whole discered more easily due to cocept of a object Ð Exteds SCSI already extesive error recovery ad self-checkig ad reportig of drive ad media state to file objects Ð Reduce overall cost of storage maagemet

81 NASD Objects ad Operatios Curret SCSI disks provide virtual liear-block iterface Ð Virtual because moder disks trasparetly re-map storage sectors to hide defective media ad variatios i track desities Ð Locality based optimizatios supported sice blocks small distace apart i the liear address space are physically close Ð More advaced SCSI devices also implemet: RAID Data compressio Dyamic block remappig

82 NASD Objects ad Operatios NASD abados idea that file maagers uderstad ad directly cotrol layout NASD drives store variable-legth logical byte-streams called objects

83 NASD Filesystems Filesystems request objects from the drive to hold the fileõs data Ð Read ad write operatios apply to a byte regio withi a object Ð Layout of the object o the drive determied by the NASD drive itself Ð Sequetial addressed i file NASD objects allocated o media for fast sequetial access Objects ca be clustered: several obejects placed together o proximity list (multiple liear block lists)

84 NASD Objects ad Operatios Proximity o object list ecourages proximity o physical media NASD supports secure commuicatios usig capabilities that have bee cryptographically sealed by the right authority (filesytem maager or partitio maager) Ð shared secret betwee NASD drive ad the authority Ð Four-level key hierarchy icludes master key (system level, chaged very ifrequetly, ot stored olie) ad drive key Ñ log-term o-lie key used to maipulate NASD partitios; chaged whe compromise feared

85 NASD Summary Symmetric or asymmetric? Asymmetric as curretly proposed Lockig? Performed o cliets or file maager Proprietary or ope storage itercoect? SCSI, so its ope Existig file systems like AFS ad NFS modified to work with NASD Ð Good scalig so far from 4 to 8 cliets

86 The Global File System (GFS) A shared file system developed at the Uiversity of Miesota A collaboratio with Seagate, Prisa, Brocade ad other FC vedors Ñ a ope system Hardware solutio to the problem of distributed file system desig Orieted towards applicatios that have lots of big files ad that require high badwidth New desig will provide more geeral architecture

87 GFS Executio Path Cliet User Applicatio VFS Memory GFS Cliet Storage Device Drivers Host Adapter Network Storage Pool

88 Global File System (GFS) Network Storage Pool Ñ a shared address space of disk blocks Ð like a shared memory i a SMP (symmetric multiprocessor) Ð implemeted i the Òpool driveró layer beeath the file system Ð pool layers implemets: locks stripig pool_assemble to assemble etwork devices ito shared pool of devices Fibre-Chael-specific stuff like ame service etc. Ð pool partitioed ito subpools

89 A Distributed GFS Eviromet GFS Cliet GFS Cliet CPU Memory Disks CPUs Memory Disks GFS Cliet GFS Cliet CPU Memory Disk CPU Memory Disk Storage Area Network Network Storage Pool Sub-pool 0 Sub-pool 1 Sub-pool 2 Sub-pool 3 Sub-pool 4 Solid State Sigle Disk RAID 5 Software Striped Disks RAID 3

90 Global File System (GFS) A symmetric shared file system Ð odes are idepedet ad act as peers relative to the storage devices: o file maager Ð like a SMP cliets are like processors etwork storage pool is the shared memory ay cliet ca execute ay portio of the kerel or ay applicatio Ð SMP architecture domiates todayõs multiprocessor desigs Most efficiet for load-balacig ad throughput efficiecy

91 GFS Cosistecy Ð similar to test-ad-set locks i memory Test-ad-set, Clear may locks per device: multiple lock accesses across parallel devices ca be made Ð lockig performed o the devices odes compete for pool of locks kept o the storage devices DLOCK SCSI commad developed to implemet fie-grai lockig SCSI RESERVE/RELEASE graularity is too high (whole device) SCSI DLOCKS provides a pool of 1000s of locks which the file system may the map to shared metadata as required

92 GFS Orgaizatio ad Structure Super block: cotais Mout iformatio ad static Resource Group Idex Multiple Resource Groups: Ð metadata partitioed ito groups for scalability ad load balacig Ð Resource Group Block bit maps for free data blocks data blocks Ð diodes (disk iodes) Ñ oe per file, dyamically allocated Ð metadata Ñ poiter blocks ad idirect poiter blocks Ð real data blocks

93 GFS Orgaizatio ad Architecture Super block Ð Maitais mout iformatio ad static file system attributes Resource Groups Ð Partitios ad distributes file system resources for parallel accesses Ð Allocated per subpool i the etwork storage pool Ð Cotais bitmaps used for block allocatio Ð Similar to Allocatio Groups i Silico GraphicsÕ XFS Device Lock 1 Resource Block - 0 Diode Bitmaps Data Block Bitmaps Diode - 0 Diode - w Data Block - 0 Data Block - q Device Lock 0 Super Block with Resource Idex 0 1 Resource Block - 1 Diode Bitmaps Data Block Bitmaps Diode - w+1 Diode - x Data Block - q+1 Data Block - r Device Lock 2 Resource Block - Diode Bitmaps Data Block Bitmaps Diode - y+1 Diode - z Data Block - s+1 Data Block - t Resource Group - 0 Resource Group - 1 Resource Group - Device Lock +1

94 GFS Orgaizatio ad Architecture GFS Diode Idirect Blocks Data Blocks Diode Number Resource Group Number File Type Mode Ower/Group Number of Liks Access Times Bookkeepig Ifo Dyamic Block Allocatio Ð Ð Available file system blocks may be freely allocated to directory or file diodes, poiter blocks, or data blocks Iode ad diode umbers based o storage pool address elimiatig lookup idirectio

95 Mappig Files to Resource Groups ad Subpools Network Storage Pool (NSP) Subpool 0 Subpool 1 Subpool 2 Subpool 3 Subpool 4 RG0 RG1 RG2 RG3 RG4 RG5 RG6 RG7 RG8 RG9 RG10 RG11 file6 file14 file10 file8 file1 file7 dir1 rootdir Resource Group 0 file7 file8 file1 dir1 dir3 file16 file2 file4 dir2 file6 file17 file10 dir3 file2 file16.1 Resource Group 6 file14 file6 dir2 Resource Group 10 file16.2 rootdir Directory Tree Resource Group 7

96 Compariso of GFS ad NFS Cotrol ad Data Paths NFS Cliet Server User Applicatio VFS VFS NFS Cliet NFS Server Local FS Memory XDR RPC Cliet TCP IP Network Device Drivers Network Adapter XDR RPC Server TCP IP Network Device Drivers Network Adapter Memory Buffer Cache Storage Device Drivers Host Adapter Cliet GFS User Applicatio VFS Memory GFS Cliet Storage Device Drivers Host Adapter Network Storage Pool

97 Diode Stuffig Directory ad file diodes occupy a etire file system block Ð As block size icreases header iformatio stays costat Ð Block utilizatio decreases decreases leadig to iteral fragmetatio Place user data i the uused diode space Ð Reduce iteral fragmetatio Ð Elimiate poiter idirectio Ð Elimiate a additioal read operatio Magic umber Diode umber / disk address Resource group umber Device lock umber File type File mode User ad group IDs Lik cout Ope cout Bytes ad blocks i file Time stamps Height of metadata tree Poiter[0] Poiter[1] Poiter[2] Poiter[3]... Poiter[-2] Poiter[-1]

98 GFS Cachig Some metadata is cached o the cliets Ð locks are polled to determie if metadata is stale or ot Ð callbacks i SCSI-3 would be ice Small files cached i cliet via diode stuffig Ð stuff data i same file system block as the diode (disk iode) We would like to do much more explicit data ad metadata cachig o the device Ð shared by all systems fast, low overhead

99 Performace Testig (May 1997) Test scalig as both cliets ad disk arrays were added 4-cliet, 4-array cofiguratio Ð Seagate disk drives, Ciprico 7000 arrays Ð SGI Challeges, Brocade switch, Prisa HAs Parameters tested Ð file trasfer time ad badwidth Ð file size ad request size are varied DLOCK performace: Ð about 1 millisecod for both Seagate ad Ciprico

100 Performace Tests: Cofiguratio Array A Array B Array C Array D Disks Loop Switch Port 0 Port 1 Port 0 Port 0 Port 0 Cliet W Cliet X Cliet Y Cliet Z

101 Performace Results: Sigle Host

102 Performace Results: Trasfer Rate Aggregate Trasfer Rate (MB/s) GFS 256 MB Create GFS 256 MB Read GFS 256 MB Write 0.0 1/2 2/3 3/4 Cliets/Devices

103 Performace Results: Speedup GFS 256 MB Create GFS 256 MB Read GFS 256 MB Write Scaled Speedup /2 2/3 3/4 Cliets/Devices

104 Performace Results: Trasfer Rate Aggregate Trasfer Rate (MB/s) GFS 256 MB Create GFS 256 MB Read GFS 256 MB Write 0.0 1/1 2/2 3/3 4/4 Cliets/Arrays

105 Performace Results: Speedup GFS 256 MB Create GFS 256 MB Read GFS 256 MB Write Scaled Speedup /1 2/2 3/3 4/4 Cliets/Arrays

106 Summary of Performace Results Scalig was good with separate root directory device Ð lockig ad dir searches isolated to oe device ot trasferrig data Scalig was bad without separate root dir device Ð resource groups were ot allocated uiformly across devices Ð DLOCK commads itermigled with SCSI data commads: this is bad DLOCK sittig behig a big read or write will hurt performace of urelated operatios

107 Summary of Performace Results Bottom lie: poor scalig caused by implemetatio ot the GFS architecture Ð DLOCKs ca ad will be made faster sice they are implemeted i hardware Ð more cachig where useful iteral to GFS cliet Ð miimize use of locks i GFS cliet code recetly reduced umber of locks used by factor of 3

108 Summary of Scalig Study Our results show the GFS approach to cluster file system desig is at least feasible Architecture scales well (so far) Ð 4 cliets ad 4 arrays: Power Challege Ð curretly testig 8-way scalability o a cluster of 8 SGI O2Õs Ð goal: scales to 32 or 64 cliets past 64 cliets will eed hierarchy of GFS ÒdomaisÓ both for performace ad to ease admiistratio

109 Summary of Scalig Study Lots of uaswered questios Ð What if you actually have decet cachig o the devices Ð Faster locks with richer parallel sematics (multiple readers sigle writer) Ð Head-of-queue taggig of DLOCK commads Ð More efficiet lock usage by GFS (reductio by factor of three of umber of locks ecessary) Ð Sigle cliet optimizatios Ð More aggressive cliet cachig Ð More sophisticated mappig of metadata to locks to reduce bottleecks

110 Badwidth Characterizatio Two parameter tests Ð Request size varied expoetially from 64 KB to 4 MB Ð Trasfer, or file, size varied expoetially from 64 KB to 512 MB Test cofiguratio Ð Sigle Silico Graphics O2 desktop workstatio Ð Prisa NetFX PCI-32 Fibre Chael host bus adapter Ð Sigle Ciprico Rimfire 7010 Fibre Chael RAID-3 Ð Brocade Silkworm 16-port Fibre Chael switch

111 Badwidth Characterizatio File System Buffer Cache regular User Space System Call Iterface raw direct Stream Head Module(s) Characterize the badwidth for the give test cofiguratio for each subsystem Quatify the amout of overhead icurred by each subsystem by examiig badwidth losses Device Drivers Hardware Layer

112 Host Adapter Badwidth Buffered Write Badwidth for Prisa Fibre Chael SCSI Driver w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Buffered Read Badwidth for Prisa Fibre Chael SCSI Driver w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Request Size (KB) Trasfer Size (KB) Badwidth (MB/s) Request Size (KB) Trasfer Size (KB) Badwidth (MB/s)

113 Network Storage Pool Badwidth Buffered Write Badwidth for Network Storage Pool Driver w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Buffered Read Badwidth for Network Storage Pool Driver w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Request Size (KB) Trasfer Size (KB) Badwidth (MB/s) Request Size (KB) Trasfer Size (KB) Badwidth (MB/s)

114 GFS Badwidth Buffered Read Badwidth for Global File System w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Request Size (KB) 512 Buffered Write Badwidth for Global File System 256 w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Direct Write Badwidth for Global File System w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Trasfer Size (KB) Badwidth (MB/s) Request Size (KB) Direct Read Badwidth for Global File System w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Trasfer Size (KB) Badwidth (MB/s) Badwidth (MB/s) Badwidth (MB/s) Request Size (KB) Trasfer Size (KB) Request Size (KB) Trasfer Size (KB)

115 XFS Buffered Read Badwidth for XFS w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Buffered Write Badwidth for XFS Request Size (KB) 256 w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Direct Write Badwidth for XFS w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Trasfer Size (KB) Badwidth (MB/s) Request Size (KB) Direct Read Badwidth for XFS w/ Silico Graphics O 2 ad Ciprico RF7010 RAID Trasfer Size (KB) Badwidth (MB/s) Badwidth (MB/s) Badwidth (MB/s) Request Size (KB) Trasfer Size (KB) Request Size (KB) Trasfer Size (KB)

116 Relative Subsystem Efficiecies Relative Efficiecy Prisa NetFX Driver Network Stora ge Pool Driver Global File Syste m XFS Buffered I/ O Writes Mea 100.0% 95.6% 24.9% 87.8% Stadard Deviatio Miimum 100.0% 90.4% 9.7% 43.7% Maximum 100.0% 100.5% 38.8% 475.0% Reads Mea 100.0% 97.2% 25.7% 64.5% Stadard Deviatio Miimum 100.0% 92.3% 14.7% 31.8% Maximum 100.0% 103.6% 68.8% 319.0% Direct I/O Writes Mea 29.6% 112.3% Stadard Deviatio Miimum 16.4% 94.6% Maximum 79.1% 375.8% Reads Mea 31.4% 115.4% Stadard Deviatio Miimum 16.9% 99.4% Maximum 74.5% 215.9% Overall Mea 100.0% 96.4% 27.6% 90.0% Stadard Deviatio Miimum 100.0% 90.4% 9.7% 31.8% Maximum 100.0% 103.6% 79.1% 475.0%

117 The SCSI Device Lock Commad Techical Descriptio Curret Implemetatio ad Proposed Improvemets Stadardizatio Collaboratio with Seagate

118 GFS: Future Plas NT port i collaboratio with idustry Ope Source versio o Liux IRIX versio give away to film ad video idustry, others

119 Overview ad Coclusios New ope, high-volume, relatively low-cost NAS iterfaces make widespread use of shared (disk) file systems potetially possible More research eeded: Ð Previous systems have ot succeeded i scalig past a small umber of cliets Ð Distributed coherece, cachig, ad recovery issues must be resolved ad the solutios stadardized by the idustry perform i cliet, file maager, etwork, or device Ð NASD keys i o two most importat issues objects ad security Ð Promise is better performace, better storage maagemet by the devices, ad ew applicatios i parallel clusters