Storage Aggregation for Performance & Availability:

Transcription

1 Storage Aggregation for Performance & Availability: The Path from Physical RAID to Virtual Objects Univ Minnesota, Third Intelligent Storage Workshop Garth Gibson CTO, Panasas, and Assoc Prof, CMU May 4, 2005

2 Cluster Computing: The New Wave Traditional Computing Monolithic Computers Cluster Computing Linux Compute Cluster Issues Complex Scaling Limited BW & I/O Islands of storage Inflexible Expensive Single data path Monolithic Storage? But Parallel data paths Scale to bigger box Scale: File & total BW File & total capacity Load & capacity balancing lower $ / Gbps Slide 2 April 26, 2005

3 Panasas: Next Gen Cluster Storage Scalable performance Parallel data paths to compute nodes Scale clients, network and capacity As capacity grows, performance grows Simplified and dynamic management Robust, shared file access by many clients Seamless growth within single namespace eliminates time-consuming admin tasks Integrated HW/SW solution Optimizes performance and manageability Ease of integration and support ActiveScale Storage Cluster Single Step: Perform job directly from high I/O Panasas Storage Cluster Control path Compute Cluster Parallel data paths Metadata Managers Object Storage Devices Slide 3 April 26, 2005

4 Rooted Firmly in RAID May 4, 2005

5 Birth of RAID ( ) Member of 4th Berkeley RISC CPU design team (SPUR: 84-89) Dave Patterson decides CPU design is a solved problem Sends me to figure out how storage plays in SYSTEM PERFORMANCE IBM 3380 disk is 4 arms in a 7.5 GB washing machine box SLED: Single Large Expensive Disk New PC industry demands cost effective 100 MB 3.5 disks Enabled by new SCSI embedded controller architecture Use many PC disks for parallelism SIGMOD88: A case for RAID PS. $10-20 per MB (~1000X now) 100 MB/arm (~1000X now) IO/sec/arm (5X now) Slide 5 April 26, 2005

6 But RAID is really about Availability Arrays have more Hard Disk Assemblies (HDAs) -- more failures Apply replication and/or error/erasure detection codes Mirroring wastes 50% space; RAID wastes 1/N Mirroring halves, RAID 5 quarters small write bandwidth Slide 6 April 26, 2005

7 Off to CMU & More Availability Parity Declustering spreads RAID groups to reduce MTTR Each parity disk block protects fewer than all data disk blocks (C) Virtualizing RAID group lessens recovery work Faster recovery or better user response time during recovery or mixture of both RAID over X? X = Independent fault domains Disk is easiest X Parity declustering is first step in RAID virtualization Slide 7 April 26, 2005

8 Network-Attached Secure Disks (NASD, 95-99) May 4, 2005

9 Storage Interconnect Evolution Outboard circuitry increases over time (VLSI density) Hardware (#hosts, #disks, #paths) sharing increases over time Logical (information) sharing limited by host SW 1995: Fibrechannel packetizes SCSI over a near general network Outboard circuitry increases over time (VLSI density) Hardware (#hosts, #disks, #paths) sharing increases over time Logical (information) sharing limited by host SW 1995: Fibrechannel packetizes SCSI over a near general network Packetized SCSI will go over any network (iscsi) What do we next do with device intelligence? Slide 9 April 26, 2005

10 Storage as First Class Network Citizen Direct transfer between client and storage Exploit scalable switched cluster area networking Split file service into: primitives (in drive) and policies (in manager) Slide 10 April 26, 2005

11 NASD Architecture Before NASD there was store&forward Server-Attached Disks (SAD) Move access control, consistency out-of-band and cache decisions Raise storage abstraction: encapsulate layout, offload data access Slide 11 April 26, 2005

12 Fine Grain Access Enforcement State of art is VPN of all out-of-band clients, all sharable data and metadata Accident prone & vulnerable to subverted client; analogy to single-address space computing File manager Private Communication NASD Integrity/Privacy Secret Key 2: CapArgs, CapKey 1: Request for access Object Storage uses a digitally signed, objectspecific capabilities on each request CapKey= MAC SecretKey (CapArgs) CapArgs= ObjID, Version, Rights, Expiry,... ReqMAC = MAC CapKey (Req,NonceIn) 3: CapArgs, Req, NonceIn, ReqMAC Client Secret Key NASD 4: Reply, NonceOut, ReplyMAC ReplyMAC = MAC CapKey (Reply,NonceOut) Slide 12 April 26, 2005

13 Scalable File System Taxonomy May 4, 2005

14 Today s Ubiquitous NFS ADVANTAGES Familiar, stable & reliable Widely supported by vendors Competitive market DISADVANTAGES Capacity doesn t scale Bandwidth doesn t scale Cluster by customer-exposed namespace partitioning Clients File Servers Storage Net Disk arrays Host Net Exported Sub- File System Slide 14 April 26, 2005

15 Scale Out 1: Forwarding NFS Servers Bind many file servers into single system image with forwarding Mount point binding less relevant, allows DNS-style balancing, more manageable Control and data traverse mount point path (in band) passing through two servers Single file and single file system bandwidth limited by backend server & storage Tricord, Spinnaker File Server Cluster Clients Disk arrays Host Net Storage Net Slide 15 April 26, 2005

16 Scale Out File Service w/ Out-of-Band Client sees many storage addresses, accesses in parallel Zero file servers in data path allows high bandwidth thru scalable networking Mostly built on block-based SANs where servers trust all clients E.g.: IBM SanFS, IBM GPFS, EMC HighRoad, Sun QFS, SGI CXFS, etc Beginning to be built on Object storage (SANs too) E.g. Panasas DirectFlow, Lustre, Clients Storage File Servers Slide 16 April 26, 2005

17 Object Storage Standards May 4, 2005

18 Object Storage Architecture An evolutionary improvement to standard SCSI storage interface (OSD) Offload most data path work from server to intelligent storage Finer granularity of security: protect & manage one file at a time Raises level of abstraction: Object is container for related data Operations: Read block Write block Storage understands how blocks of a file are related -> self-management Per Object Extensible Attributes key expansion of storage functionality Addressing: Block range Allocation: External Block Based Disk Security At Volume Level Slide 18 April 26, 2005 Operations: Create object Delete object Read object Write object Addressing: [object, byte range] Allocation: Internal Object Based Disk Security At Source: Intel Object Level

19 Why I Like Objects Objects are flexible, autonomous Variable length data with layout metadata encapsulated SNIA Shared Storage Model, 2001 Share one access control decision Amortize object metadata Defer allocation, smarter caching With extensible attributes E.g. size, timestamps, ACLs, + Some updated inline by device Extensible programming model Metadata server decisions are signed & cached at clients, enforced at device Rights and object map small relative to block allocation map Clients can be untrusted b/c limited exposure of bugs & attacks (only authorized object data) Cache decisions (& maps) replaced transparently -- dynamic remapping -- virtualization Slide 19 April 26, 2005

20 OSD is now an ANSI Standard Panasas NSIC NASD SNIA/T10 OSD OSD Standard CMU NASD Lustre Key Players: T10/SCSI ratifies OSD command set 9/27/04 Co-chaired by IBM and Seagate, protocol is a general framework (transport independent) & KEY: Resulting standard will build options for customers looking to deploy objects Slide 20 April 26, 2005

21 ActiveScale Storage Cluster May 4, 2005

22 Object Storage Systems Expect wide variety of Object Storage Devices Disk array subsystem Ie. LLNL with Lustre Smart disk for objects 2 SATA disks 240/500 GB Prototype Seagate OSD Highly integrated, single disk Orchestrates system activity Balances objects across OSDs Stores up to 5 TBs per shelf 16-Port GE Switch Blade 4 Gbps per shelf to cluster Slide 22 April 26, 2005

23 BladeServer Storage Cluster Integrated GE Switch Shelf Front 1 DB, 10 SB Battery Module (2 Power units) Shelf Rear DirectorBlade Midplane routes GE, power StorageBlade Slide 23 April 26, 2005

24 How Do Panasas Objects Scale? Scale capacity, bandwidth, reliability by striping according to small map File Comprised of: User Data Attributes Layout DATA Scalable Object Map 1. Purple OSD & Object 2. Gold OSD & Object 3. Red OSD & Object Plus stripe size, RAID level Slide 24 April 26, 2005

25 Object Storage Bandwidth Scalable Bandwidth demonstrated with GE switching GB/sec Object Storage Devices Lab results Slide 25 April 26, 2005

26 ActiveScale SW Architecture Mgr DB ActiveScale DirectorBlades Realm & Performance Mgrs + Web Mgmt Server Virtual Sub Mgrs File File File Mgr Mgr Mgr Quota Quota Stor Stor Stor Mgr Mgr Mgr Mgr Mgr Mgr RPC ActiveScale Protocol Servers NDMP Buffer Cache DirectFLOW Client RAID NFS NFS CIFS CIFS OSD / iscsi TCP/IP NTP + DHCP Server DirectFLOW DirectFLOW DirectFLOW StorageBlades Mgmt Agent DFLOW fs DFLOW fs RAID DFLOW 0 fs RAID 0 RAID 0 Zero NVRAM Zero Copy NVRAM Zero Copy Cache NVRAM Copy Cache Cache OSD / iscsi TCP/IP DirectFLOW DirectFLOW DirectFLOW NFS CIFS UNIX Windows POSIX App NT App Slide 26 April 26, 2005 Direct- FLOW RAID OSD / iscsi TCP/IP POSIX App Local Buffer Cache DirectFLOW Client Linux VFS

27 DirectFLOW Client Standard installable file system for Linux POSIX syscalls with common performance exceptions Reduce serialization & media updates (e.g. durable read timestamps) Add create parameters for file width, RAID level, stripe unit size Add open in concurrent write mode; where app handles consistency Mount options Local (NFS) mount anywhere in client, or Global (AFS) mount at /panfs on all clients Cache consistency and callbacks Need a callback on a file in order to cache any file state Exclusive, read-only, read-write (non cacheable), concurrent write Callbacks are really leases so AWOL clients lose promises Slide 27 April 26, 2005

28 Fault Tolerance Overall up/down state of blades Subset of managers track overall state with heartbeats Maintain identical state with quorum/consensus Per file RAID: no parity for unused capacity RAID level per file; small files mirror RAID5 for large files First step toward policy quality of storage associated w/ data Scalable rebuilt rate Large files striped on all blades Recon XOR on all managers Rebuilt files spread on all blades Slide 28 April 26, 2005

29 Manageable Storage Clusters Snapshots: consistency for copying, backing up Copy-on-write duplication of contents of objects Named as /.snapshot/juliansnaptimestamp/filename Snaps can be scheduled, auto-deleted Soft volumes: grow management without physical constraints Volumes can be quota bounded, unbounded, or just send on threshold Multiple volumes can share space of a set of shelves (double disk failure domain) Capacity and load balancing: seamless use of growing set of blades All blades track capacity & load; manager aggregates & ages utilization metrics Unbalanced systems influence allocation; can trigger moves Adding a blade simply makes a system unbalanced for awhile Slide 29 April 26, 2005

30 Out-of-band DF & Clustered NAS Slide 30 April 26, 2005

31 Scale Out 1b: Any NFS Server Will Do Single server does all data transfer in single system image Servers share access to all storage and hand off role of accessing storage Control and data traverse mount point path (in band) passing through one server Hand off cheap: difference between 100% & 0% colocated front & back end was about zero SFS97 NFS op/s and < 10% latency increase for Panasas NFS Allows single file & single filesystem capacity & bandwidth to scale with server cluster Clients File Server Cluster Disk arrays Host Net Storage Net Slide 31 April 26, 2005

32 Performance & Scalability for all Workloads Objects: breakthrough data throughput AND random I/O Slide 32 April 26, 2005 Source: SPEC.org & Panasas

33 ActiveScale In Practice May 4, 2005

34 Los Alamos Lightning 1400 nodes and 60TB (120 TB): Ability to deliver ~ 3 GB/s* * (~6 GB/s) Panasas 12 shelves Lightning 1400 nodes 4 4 Switch 4 Slide 34 April 26, 2005

35 Progress in the Field Seismic Processing Genomics Sequencing HPC Simulations Post-Production Rendering Slide 35 April 26, 2005

36 pnfs - Parallel NFS May 4, 2005

37 Out-of-Band Interoperability Issues ADVANTAGES Capacity scaling Faster bandwidth scaling DISADVANTAGES Requires client kernel addition Many non-interoperable solutions Not necessarily able to replace NFS EXAMPLE FEATURES POSIX Plus & Minus Global mount point Fault tolerant cache coherence RAID 0, 1, 5 & snapshots Distributed metadata and online growth, upgrade Clients Vendor X Kernel Patch/RPM Storage Vendor X File Servers Slide 37 April 26, 2005

38 Standards Efforts: Parallel NFS (pnfs) December 2003 U. Michigan workshop Whitepapers at U. Michigan CITI, EMC, IBM, Johns Hopkins, Network Appliance, Panasas, Spinnaker Networks, Veritas, ZForce NFSv4 extensions enabling clients to access data in parallel storage devices of multiple standard flavors Parallel NFS request routing / file virtualization Extend block or object layout information to clients, enabling parallel direct access IETF pnfs Documents: draft-gibson-pnfs-problem-statement-01.txt draft-gibson-pnfs-reqs-00.txt draft-welch-pnfs-ops-01.txt Slide 38 April 26, 2005

39 Multiple Data Server Protocols Inclusiveness favors success Three (or more) flavors of out-ofband metadata attributes: BLOCKS: SBC/FCP/FC or SBC/iSCSI for files built on blocks OBJECTS: OSD/iSCSI/TCP/IP/GE for files built on objects FILES: NFS/ONCRPC/TCP/IP/GE for files built on subfiles Inode-level encapsulation in server and client code NFSv4 extended w/ orthogonal disk metadata attributes pnfs server Local File system Client Apps pnfs IFS pnfs Disk driver disk metadata grant & revoke 1. SBC (blocks) 2. OSD (objects) 3. NFS (files) Slide 39 April 26, 2005

40 Object Storage Value Proposition NFS NFS CIFS DirectFLOW Linux Compute Cluster DirectFLOW Panasas Storage Cluster Large datasets, checkpointing Analysis & Simulation Results Gateway Scalable Shared Storage Value Proposition Larger surveys; more active analysis Applications leverage full bandwidth Simplified storage management Results gateway Slide 40 April 26, 2005 Analysis & Visualization Tools

41 Next Generation Network Storage Garth Gibson May 4, 2005