Parallel NFS (pnfs) Garth Gibson CTO & co-founder, Panasas Assoc. Professor, Carnegie Mellon University.

Size: px

Start display at page:

Download "Parallel NFS (pnfs) Garth Gibson CTO & co-founder, Panasas Assoc. Professor, Carnegie Mellon University."

Amie Warren
5 years ago
Views:

1 Parallel NFS (pnfs) Garth Gibson CTO & co-founder, Panasas Assoc. Professor, Carnegie Mellon University

2 Panasas Parallel Storage: A Little Background

metadata mgmt & HA Quorum consensus state Legacy clustered NAS NFS and CIFS StorageBlades Wide striping &

3 Panasas Storage Cluster Architecture Object Based OSD/iSCSI at core of DF DirectFLOW client S/W Red Hat, Suse, Fedora all patch-free POSIX extensions for weak consistency & layout controls DirectorBlades DF metadata mgmt & HA Quorum consensus state Legacy clustered NAS NFS and CIFS StorageBlades Wide striping & smart prefetching & caching RAID across blades, inside each file independently Slide 3 March 21, 2007 Panasas, Inc.

4 Bladeserver of Simple PC Components Slide 4 March 21, 2007 Panasas, Inc.

5 Scalable Performance Scalable data throughput AND random I/O Slide 5 March 21, 2007 Panasas, Inc.

6 Petascale Bound, Traveling with LANL Roadrunner (1.4 PFLOPS) RFP specifies Panasas Storage Cluster RH Linux, IBM System x3755 & BladeCenter H with Cell B.E. Completion and acceptance anticipated in 2008 Slide 6 March 21, 2007 Panasas, Inc.

7 Shared, Scalable-BW Storage LINEAR: ~1 GB/s per TFLOPS Disk 2500 TeraBytes Balanced System Approach Computing Speed FLOP/s 10 Memory Year TeraBytes Parallel I/O GigaBytes/sec Network Speed Gigabits/sec Archival Storage Gigabytes/sec App Perfor mance Metadata Inserts/sec > 1 PB Panasas in many clusters GM: 5600 nodes, procs, Lightning, Bolt, Pink, TLC, Flash, Gordon IB: 1856 nodes, procs, Blue Steel, Coyote, & soon Roadrunner Disk Rich Supercomputer Past Generation ASCI Platform Disk Poor Clients cluster cluster cluster cluster Job Queue Site Backbone Archive Scalable OBFS Enterprise Global FS File System Gateways BackBone Disk Rich Supercomputer Object Archive Current Generation ASCI Platform Diskless Clients cluster cluster cluster cluster Slide 7 March 21, 2007 Panasas, Inc.

8 HPC Innovators choose PanFS Oil & Gas Life Sciences Government Finance Fluid Dynamics & dozens more Slide 8 March 21, 2007 Panasas, Inc.

9 Parallel NFS (pnfs)

10 Today s Ubiquitous NFS ADVANTAGES Familiar, stable & reliable Widely supported by vendors Competitive market LIMITATION Client moves all data and metadata for a sub-file system through one network endpoint (server) Clients File Servers Storage Net Disk arrays Host Net Exported Sub- File System Slide 10 March 21, 2007 Panasas, Inc.

11 Today s Ubiquitous NFS Doesn t Scale ADVANTAGES Familiar, stable & reliable Widely supported by vendors Competitive market DISADVANTAGES Capacity doesn t scale Bandwidth doesn t scale Cluster by customer-exposed namespace partitioning Clients File Servers Storage Net Disk arrays Host Net Exported Sub- File System Slide 11 March 21, 2007 Panasas, Inc.

12 Scale Out File Service w/ Out-of-Band Client sees many storage addresses, accesses in parallel Zero file servers in data path allows high bandwidth thru scalable networking A.K.A. SAN file systems and parallel file systems NOT NFS Clients Storage File Servers Slide 12 March 21, 2007 Panasas, Inc.

13 Out-of-Band Interoperability Issues ADVANTAGES Capacity scaling Faster bandwidth scaling DISADVANTAGES Requires client kernel addition Many non-interoperable solutions Not necessarily able to replace NFS Clients Storage Vendor X Kernel Patch/RPM Vendor X File Servers Slide 13 March 21, 2007 Panasas, Inc.

14 Scale Out: Cluster NFS Servers (1) Bind many file servers into single system image with forwarding Mount point binding less relevant, allows DNS-style balancing, more manageable Clients File Server Cluster Disk arrays Host Net Storage Net Slide 14 March 21, 2007 Panasas, Inc.

15 Scale Out: Cluster NFS Server (2) Single server does all data transfer in single system image Servers share access to all storage and hand off role of accessing storage Control and data traverse mount point path (in band) passing through one server Typically built on top of a SAN file system or parallel file system Clients File Server Cluster Disk arrays Host Net Storage Net Slide 15 March 21, 2007 Panasas, Inc.

16 Out-of-Band Standards? ADVANTAGES Capacity scaling Faster bandwidth scaling Clients Work to be done Get widespread agreement on semantics Build multiple reference implementations Test interoperability constantly Compete on SW, server implementations Storage Standard platforms embed into client code into all releases Vendor X File Servers Slide 16 March 21, 2007 Panasas, Inc.

Parallel NFS: Scalability for Mainstream IETF NFSv4.1 draft-ietf-nfsv4-minorversion1-10.txt Includes pnfs, sessions/rdma, directory delegations U.

17 Parallel NFS: Scalability for Mainstream IETF NFSv4.1 draft-ietf-nfsv4-minorversion1-10.txt Includes pnfs, sessions/rdma, directory delegations U.Mich/CITI impl g Linux client/server Three (or more) flavors of out-of-band metadata attributes: FILES: NFS/ONCRPC/TCP/IP/GE for files built on subfiles NetApp, Sun, IBM, U.Mich/CITI BLOCKS: SBC/FCP/FC or SBC/iSCSI for files built on blocks EMC, IBM (-pnfs-blocks-03.txt) OBJECTS: OSD/iSCSI/TCP/IP/GE for files built on objects Panasas (-pnfs-obj-03.txt) NFSv4 extended w/ orthogonal layout metadata attributes pnfs server Local Filesystem Client Apps pnfs IFS Layout driver Layout metadata grant & revoke 1. SBC (blocks) 2. OSD (objects) 3. NFS (files) Slide 17 March 21, 2007 Panasas, Inc.

18 Highlights of the History of pnfs Conversations with Gary Grider, LANL, & Lee Ward, Sandia, 2003 How to make HPC investment in High Performance File Systems persistent Workshop on NFS Extensions for Parallel Storage, Dec 2003, Ann Arbor Chaired by Peter Honeyman, CITI/U.Mich., & Garth Gibson, CMU Initial problem statement, operations proposal to IETF July & Nov 2004 Garth Gibson, Panasas, Peter Corbett, NetApp, Brent Welch, Panasas Standards development team in action Andy Adamson, CITI/U.Mich, David Black, EMC, Garth Goodson, NetApp, Tom Pisek, Sun, Benny Halevy, Panasas, Dave Noveck, NetApp, Spencer Shepler, Sun, Brian Pawlowski, NetApp, Marc Eshel, IBM, & many others Dean Hildebrand, CITI/U.Mich, with Lee Ward, did first prototype & paper IETF working group folded it into NFSv4.1 minorversion draft in Slide 18 March 21, 2007 Panasas, Inc.

19 SC06 High-Performance NFS Panelists Garth Gibson, CTO, Panasas Inc, & Prof., Carnegie Mellon Univ. Committed to pnfs products; prototype bandwidth near DirectFlow speeds Mike Kazar, VP & Chief Architect, Network Appliance GX is NAS switch ; pnfs eliminates switch overhead, faster single clients Paul Rutherford, Sr. Director, SW Engineering, Isilon High performance NFS = clustered NFSv3 + NFSv4 & pnfs Uday Gupta, CTO, NAS, EMC MPFSi will be standardized as pnfs Roger Haskin, Sr. Manager, File Systems, IBM Prototyping Linux pnfs on top of GPFS (GPFS client is pnfs data server) Weakness of pnfs: file level delegation, NFS consistency semantics Slide 19 March 21, 2007 Panasas, Inc.

20 pnfs Protocol Operations LAYOUTGET (filehandle, type, byte range) -> type-specific layout LAYOUTRETURN (filehandle, range) -> releases state for client LAYOUTCOMMIT (filehandle, byte range, updated attributes, layout-specific info) -> data visible to other clients Timestamps, end-of-file attributes updated CB_LAYOUTRECALL Server tells the client to stop using a layout CB_RECALLABLE_OBJ_AVAIL Delegation available for a file that was not previously available GETDEVICEINFO, GETDEVICELIST NFSv4 extended w/ orthogonal layout metadata attributes pnfs server Local Filesystem Map deviceid in layout to type-specific addressing information Client Apps pnfs IFS Layout driver Layout metadata grant & revoke 1. SBC (blocks) 2. OSD (objects) 3. NFS (files) Slide 20 March 21, 2007 Panasas, Inc.

21 pnfs READ LOOKUP+OPEN client to NFS server, returns file handle and state ids LAYOUTGET client to NFS server, returns layout Linux Compute Cluster READ client to storage devices, many reads in parallel LAYOUTRETURN client to NFS server Control Paths Clients can cache layouts for use with multiple READ and multiple LOOKUP+OPEN instances Metadata Manager READ Parallel Data Paths Server uses CB_LAYOUTRECALL when the layout is no longer valid Storage Devices Slide 21 March 21, 2007 Panasas, Inc.

pnfs WRITE LOOKUP+OPEN client to NFS server, returns file handle and state ids

client to storage devices LAYOUTCOMMIT client to NFS server publishes write

layout to reduce allocation overheads, avoid quota limits, etc.

22 pnfs WRITE LOOKUP+OPEN client to NFS server, returns file handle and state ids LAYOUTGET client to NFS server, returns layout Linux Compute Cluster WRITE client to storage devices LAYOUTCOMMIT client to NFS server publishes write LAYOUTRETURN client to NFS server Server may restrict byte range of write layout to reduce allocation overheads, avoid quota limits, etc. Metadata Manager Control Paths WRITE Parallel Data Paths Storage Devices Slide 22 March 21, 2007 Panasas, Inc.

23 IETF NFSv4 pushing pnfs NFSv4.1 closing Obsoletes v3 & v4 pnfs is principal Objects & blocks included w/ files Proposed standard is product threshold Friday was first round final comments deadline Good momentum behind this Slide 23 March 21, 2007 Panasas, Inc.

24 Current Status pnfs is part of the IETF NFSv4 minor version 1 standard draft Bi weekly conference calls to drive 4.1 draft to closure for Sept 07 completion Reference open source client done in CITI CITI owns NFSv4 Linux client and server Weekly pnfs implementers' conference call Callbacks not yet running; failure cases not sufficiently explored Participants: CITI (Files over PVFS2, Files over NFSv4) Netapp, Sun (Files over NFSv4) IBM (Files, based on GPFS) EMC (Blocks, based on HighRoad/MPFS) Panasas (Objects, based on Panasas ActiveScale Storage Cluster OSDs) Slide 24 March 21, 2007 Panasas, Inc.

25 References NFS Version 4 Minor Version 1 draft-ietf-nfsv4-minorversion1-10.txt draft-ietf-nfsv4-pnfs-obj-03.txt pnfs Problem Statement Garth Gibson (Panasas), Peter Corbett (Netapp), Internet-draft, July 2004, NFSv4 pnfs Extensions G. Goodson (Netapp), B. Welch, B. Halevy (Panasas), D. Black (EMC), A. Adamson (CITI), Internet-draft, October 2005, Linux pnfs Kernel Development CITI, University of Michigan Slide 25 March 21, 2007 Panasas, Inc.

26 Panasas pnfs strategy Slide 26 March 21, 2007 Panasas, Inc.

27 Multiprotocol DF & Clustered NAS Slide 27 March 21, 2007 Panasas, Inc.

28 Multiprotocol DF, pnfs & Clustered NAS CIFS & NFSv3 clustered, + NFSv4.1 (pnfs metadata service) futureproof DF investment: - pnfs is a software upgrade - concurrent use of DF & pnfs DirectFlow servers, + NFSv4.1 (pnfs object dataservers) Slide 28 March 21, 2007 Panasas, Inc.

1 server on DirectFlow/PanFS MDS Many parts of pnfs solution not yet done Iozone -c -e -r448k -s 5g -t

29 Panasas Committed to pnfs Promising preliminary results Built on U.Mich/CITI Linux client/server code base Layer NFSv4.1 server on DirectFlow/PanFS MDS Many parts of pnfs solution not yet done Iozone -c -e -r448k -s 5g -t #clients Client Apps pnfs IFS Objects Layout driver NFSv4 extended w/ orthogonal layout metadata attributes iscsi OSD (objects) pnfs server PanFS MDS 2 shelves (18 OSDs) Slide 29 March 21, 2007 Panasas, Inc.

30 Thank You!

PETASCALE DATA STORAGE INSTITUTE SciDAC university-led center of excellence 3 universities, 5 labs, 2006-2011, G.

edu/pdsi Community building: workshops, tutorials, courses APIs & standards: incubate, prototype, validate Failure data

31 PETASCALE DATA STORAGE INSTITUTE SciDAC university-led center of excellence 3 universities, 5 labs, , G. Gibson, PI PDSI: scientific computing storage issues Community building: workshops, tutorials, courses APIs & standards: incubate, prototype, validate Failure data collection, analysis & publication Performance trace collection & benchmark publication IT automation applied to HEC systems & problems Novel mechanisms for core HEC storage problems Slide 31 March 21, 2007 Panasas, Inc.

32 PDSI Example Projects: HEC POSIX Extensions HEC Failure Data Collection Slide 32 March 21, 2007 Panasas, Inc.

33 Disk drive failure data sources HPC1 HPC2 Type of drive 18GB 10K RPM SCSI 36GB 10K RPM SCSI 36GB 10K RPM SCSI Count Duration 3,400 5 yrs yrs Supercomputing X HPC3 15K RPM SCSI 15K RPM SCSI 7.2K RPM SATA 14,208 1 yr Various HPCs HPC4 250GB SATA 500GB SATA 400GB SATA 13,634 3 yrs COM1 COM2 10K RPM SCSI 15K RPM SCSI 26,734 39,039 1 month 1.5 yrs Internet services Y COM3 10K RPM FC-AL 10K RPM FC-AL 10K RPM FC-AL 10K RPM FC-AL Slide 33 March 21, 2007 Panasas, Inc. 3,700 1 yr

Annual replacement rate (ARR) in the field Datasheet MTTFs are 1,000,000 to 1,500,000 hours. => Expected annual replacement rate (ARR): 0.58-0.88 %.

34 Annual replacement rate (ARR) in the field Datasheet MTTFs are 1,000,000 to 1,500,000 hours. => Expected annual replacement rate (ARR): %. SATA 300K-400K MTT-replace ARR avrg = 3.0 ARR = 0.88 ARR = 0.58 No evidence that SATA disks exhibit higher replacement rates than SCSI or FC Slide 34 March 21, 2007 Panasas, Inc.

35 LANL Petascale (Red) FY08 NFS complex and other network services, WAN Site wide Shared Global Parallel File System TB GB/s (spans lanes) N gb/s N gb/s fail over 4 gb/s per 5-8 TB N gb/s FTA s N gb/s 20 gb/s per link Archive 156 I/O nodes, 1 10gbit link each, 195 GB/sec, planned for Accelerated Road Runner 1PF sustained 96 I/O nodes, 2 1gbit links each, 24 GB/sec IO Unit I B 4 X F a t T r e e M y r i n e t Compute Unit Compute Unit CU CU Road Runner Base, 70TF, 144 node units, 12 I/O nodes/unit, 4 socket dual core AMD nodes, 32 GB mem/node, full fat tree, 14 units, Acceleration to 1 PF sustained Bolt, 20TF, 2 socket, singe/dual core AMD, 256 node units, reduced fat tree, 1920 nodes Lane switches, 6 X 105 = 630 GB/s If more bandwidth is needed we just need to add more lanes and add more storage, scales nearly linearly, not N 2 like a fat tree Storage Area Network. Lane passthru switches, to connect legacy Lightning/Bolt 64 I/O nodes, 2 1gbit links each, 16 GB/sec IO Unit M y r i n e t CU Lightning, 14 TF, 2 socket single core AMD, 256 node units, full fat tree, 1608 nodes Slide 35 March 21, 2007 Panasas, CU Inc.

pnfs Update A standard for parallel file systems HPC Advisory Council Lugano, March 2011

pnfs Update A standard for parallel file systems HPC Advisory Council Lugano, March 2011 Brent Welch welch@panasas.com Panasas, Inc. 1 Why a Standard for Parallel I/O? NFS is the only network file system