The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems

Transcription

1 The Storage Evolution: From Blocks, Files and s to Storage Systems Storagetechnology im Radisson SAS Hotel Schloss Fleesensee veranstaltet durch das Deutsche Zentrum für Luft- und Raumfahrt e.v. Christian Bandulet Principal Engineer Global Technical Advisory Team Sun Microsystems Inc.

2 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

3 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

4 The Data Access Taxonomy Data Access Block? SCSI, SAS, FCP, iscsi, SRP, ATA SATA?

5 The Block Paradigm SCSI, SAS, FCP, SRP, iscsi, ATA, SATA Client Interface Physical Blocks: e.g. 512 bytes

6 Logical Blocks & Physical Blocks Let's have a closer look... SCSI, SAS, FCP, SRP, iscsi, ATA, SATA Client Interface Logical Blocks Mapping Physical Blocks

7 Magnetic Hard Disk Drives Source:

8 Millipede ~ 10nm Human Hair ~80.000nm

9 Racetrack Spintronics Devices Research

10 Where do you want to go? An example from nature Information storage density 4 x 105 bits per π (50 x 10-9)2 m-2 -or- about 3x1016 bits/inch2 (~ 30 Pbit bits/inch2) A factor of 105 times today's state-of-the-art (x100,000) -or- equivalent to > 30 years of development Source:

11 Competing with Viruses (~ 30 Pbit bits/inch2) Source:

12 Disk to CPU Discontinuity Moore s Law is outstripping disk drive performance (rotational speed) As a result, servers and storage systems are hopelessly unbalanced between CPU/controller capability and storage pool performance The objective of modern systems design is to rebalance the CPUstorage ecosystem while optimizing both low $/GB and $/IOPS Source: MASS002 Intel Developer Forum 2008

13 Storage Tiers RAM DRAM SSD NAND-Flash SSD Enterprise HDD SATA Archive MAID Tape LOW Performance HIGH

14 I/O consolidation Processor Memory Processor Memory I/O I/O I/O I/O Subsystem Storage IPC LAN Storage IPC LAN

15 Ethernet vs. Fibre Channel Ethernet vs FibreChannel Gb/s 2011? MBit/s Gb/s 2011? Ethernet FibreChannel Year

16 Speed And Feeds Fibre Channel Roadmap *Base2 used throughout all applications for Fibre Channel infrastructure and devices. Each speed maintains backward compatibility at least two previous generations (I.e., 8GFC backward compatible to 4GFC and 2GFC) **Base10 is for ISLs, core connections, and other high speed applications demanding maximum bandwidth. Except for 100GFC (which follow Ethernet standards and compatibility guidelines), each Base10 speed is expected to be compatible at least one previous generation. ***FCoE: Fibre Channel over Ethernet tunnels FC through Ethernet and is compatible with all existing Fibre Channel fabric environments. FCoE ports follow Ethernet standards and compatibility guidelines.

17 I/O Consolidation Benefits FC HBA FC Traffic FC HBA FC Traffic NIC NIC Enet Traffic CNA CNA Enet Traffic Adaptor: NIC for Ethernet/IP, HCA for InfiniBand, Converged Network Adaptor (CNA) for FCoE > Customer Benefit: Fewer NIC s, HBA s and cables, lower CapEx, OpEx (power, cooling) > All traffic go over 10GE

18 FCoE Enablers 10 GbE Data Center Ethernet (Lossless Ethernet) Ethernet jumbo frames (2180 bytes) > Max FC frame payload (2112 bytes) Ethernet V2 Frame, Ethertype = FCoE=%x8906 FCS EOF FC Payload CRC FC Header FCoE Header Ethernet Header Same as a physical FC frame Control information: version, ordered sets (SOF, EOF)

19 I/O Consolidation: Benefits to Customers FC Traffic FC Traffic FCoE FCoE FCoE Enet Traffic Enet Traffic FCoE FCoE SAN Fewer NICs and Cables Same Management Model as Native FC Display FCoE FC HBA Adapter FC Storage FC Switch FCoE Switch No Gateway FCoE Adapter Server Less Power and Cooling SAN A SAN B

20 FCoE Switch Simplified Architecture FC port FC port FCF FCoE FC port FC port Ethernet Bridge Eth port Eth port Eth port Eth port Eth port Eth port *FCF: Fibre Channel Forwarder Eth port Eth port

21 I/O Consolidation Slower I/O Different service needs different fabrics No flexibility More ports to manage More power More space Higher TCO High bandwidth pipe for capacity provisioning D edicated I/O channels enable convergence > For Networking, Storage, Management > Application compatibility > QoS - differentiates different traffic types > Partitions logical fabrics, isolation Gateways - S hare remote Fibre C hannel and E th ports > Design based on average load across multiple servers > Scale incrementally add Ethernet/FC/Server blades > Scale independently

22 Upper Layer Protocols ULPs connect InfiniBand to common interfaces Supported on mainstream operating systems Clustering > MPI (Message Passing Interface) > RDS (Reliable Datagram Socket) Network > IPoIB (IP over InfiniBand) > SDP (Socket Direct Protocol) Storage > SRP (SCSI RDMA Protocol - > iser (iscsi Extensions for RDMA) IETF IP Storage WG > NFSoRDMA (NFS over RDMA) > FcoIB ISER, SRP and NFSoRDMA are part of mainline Linux kernel

23 Physical Layer

24 InfiniBand Resources InfiniBand software is developed under OpenFabrics Open source Alliance > > The mission of the OpenFabrics Alliance (OFA) is to develop, distribute and promote a unified, transport-independent, open-source software stack for RDMA-capable fabrics and networks, including InfiniBand and Ethernet. The software stack is developed for many hardware architectures and operating systems, including Linux and Windows. Upper level protocols in the stack support IP, Sockets, SAN, NAS, Clustered File Systems and Database application environments InfiniBand standard is developed by the InfiniBand Trade Association >

25 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

26 The Data Access Taxonomy Data Access Block SCSI, SAS, FCP, iscsi, SRP, ATA SATA? File Local FS, Distributed FS: (NFS, CIFS, WAFS) Shared FS: (Cluster FS, SAN FS) Global, Distributed & Parallel FS

27 Local File Systems Application Server File system structure (i.e. inodes) Data Mapping Physical Disk Logical Blocks on Disk Physical Blocks on Disk

28 Shared File Systems Shared Disk e.g. SAN FS or Cluster FS Dedicated Meta Data Server (MDS) optional Shared FS with Shared Device Shared FS Shared FS Shared FS Shared FS with integrated volume management MDS* Shared FS Shared FS Shared FS MDS* SAN Concatenate, RAID0, RAID1, RAID5,...

29 Distributed File Systems Application Server Application Server Application Server Application Server Application Server Application Server IP NAS Appliance e.g. NAS with NFS/CIFS protocol Might become a bottleneck hard to scale. Data SAN VDISK Disk Array Disk Array Disk Array Disk Array Physical Disk Physical Disk Physical Disk Physical Disk Physical Disk Physical Disk Physical Disk Physical Disk

30 NAS Aggregation/Virtualization Global Namespace Why should accessing and scaling file storage be any different in enterprises? N A S N A S

31 FS Virtualization - NAS Aggregation In-Band Solution Application Server Application Server Application Server Application Server Application Server Application Server IP Provides Global Namespace NAS Router Data NASAppliance Appliance NAS NAS Appliance NAS Appliance NAS Appliance NAS Appliance Data Data Data Data Data SAN SAN SAN SAN SAN Global Namespace

32 FS Virtualization NFS4.1 pnfs Out-of-Band Solution In-Band NAS: Application Server Application Server NFSv4 client Application Server Application Server NFSv4 client IP Out-of-Band NAS: Application Server Application Server NFSv4 client Application Server Application Server Application Server Application Server Application Server Application Server NFSv4.1 client with pnfs NFSv4.1 client with pnfs NFSv4.1 client with pnfs Storage Protocol: SCSI, NFS, OSD IP NAS Appliance with NFSv4.1 pnfs extensions NAS Appliance Data Data SAN SAN Storage Device Storage Device Scalable NAS - Loosely Coupled NAS Cluster

33 NAS Cluster NAS Grid (Scalable NAS) aka Tightly Coupled NAS Application Server Application Server Application Server Application Server Application Server Application Server VIP NAS Appliance NAS Appliance NAS Appliance NAS Appliance Data Data Data Data SAN Single Data Image Global Namespace

34 Knowledge Workers in the Information Age Information: > capture... > annotate... > store... > share... Ideas awarenesss exposure making sense of information, organize ideas, creativity... conversations, collaboration Communities Networks establishing and maintaining relations Individual

35 The New Challenge: Unstructured Data How to find one thing out of 100 million things?

36 Knowledge Workers in the Information Age Information: > capture... > annotate... > store... > share... Ideas awarenesss exposure making sense of information, organize ideas, creativity... conversations, collaboration Communities Networks establishing and maintaining relations Individual

37 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

38 The Data Access Taxonomy Data Access Block SCSI, SAS, FCP, iscsi, SRP, ATA SATA OSD CAS OSS File Local FS, Distributed FS: (NFS, CIFS, WAFS) Shared FS: (Cluster FS, SAN FS) Global, Distributed & Parallel FS

39 Data M N F N T O A I I O R

40 The Old Block Paradigm Write Block # Write Block # Write Block # Write Block # Write Block # Write Block # Write Block # Status Return 0 Read Block # Read Block # Read Block # Read Block # Read Block # Read Block # Read Block # Store Data Retrieve

41 The New Paradigm Data OID Store OID Data Retrieve

42 The New Paradigm (cont'd) WRITE 26,763 Bytes QoS = High Description = X-Ray Retention = 50 years Access Key = *&^%# Data Payload... Data OID Store Storage Responsibilities: Space Management Access Control (Identity Mgmt) QoS Management Cache, Backup Policy Migration, Retention

43 Inodes vs. s Host Host FileiSCSI System FileiSCSI System iscsiids Inodes: Block Pointers File Attributes s: Block Pointers Attributes Block Device Device

44 Autonomy Host Host FileiSCSI System FileiSCSI System iscsiids Inodes: Block Pointers File Attributes s: Block Pointers Attributes Block Device Device Storage becomes autonomous: > > > > > > > > > > > > > > capacity planning load balancing backup QoS, SLAs understand data/object grouping aggressive pre-fetching thin provisioning search compression/de-duplication, commonality factoring strong security, encryption compliance/retention/secure delete availability/replication Audit self-healing...

45 Data Sharing Homogeneous/Heterogeneous Host Host FileiSCSI System Host FileiSCSI System Inodes: Block Pointers File Attributes FileiSCSI System FileiSCSI System iscsiids iscsiids Inodes: Block Pointers File Attributes Cluster/SAN file system s: Block Pointers Attributes 1 5 Block Device Host Device

46 Additional Layer of Security Host iscsi Host Block Device Device strong security via external service > authentication > authorization > NIS, LDAP... fine granularity > per object

47 Living in a Flat Namespace File names / inodes s / OIDs Traditional Hierarchical Flat

48 Multiple Referenced s Metadata Record OID iscsi Z Metadata Record OID iscsi W Employeee iscsi ID Patient iscsiid Job iscsi Level Examination iscsi iscsi Date Salary iscsi Stock iscsi Options Diagnose iscsi Department iscsi Therapy iscsi LinkiSCSI to OID X LinkiSCSI to OID X Metadata Record OID iscsi Y Reference 2 Reference 1 Customer iscsi ID Preferences iscsi Next iscsi Store Revenue iscsi Reference 3 Credit iscsi History LinkiSCSI to OID X Data

49 Virtual View / Virtual File Systems (cont'd) Virtual View A Virtual View B File names / inodes s / OIDs Traditional Flat Virtual

50 Decomposition MDS OID OID:Layout OID:Layout OID:Layout OID:Layout OID:Layout OID:Layout OID:Layout Layout OSD OSD OSD OSD Component Component Component Component Layout

51 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

52 SCSI Standards Architecture Source:

53 SCSI Commands Source:

54 SCSI Transports Source:

55 SCSI OSD/OSD-2 Source:

56 SCSI OSD/OSD-2 over Fibre Channel Source:

57 SCSI OSD/OSD-2 over InfiniBand Source:

58 SCSI OSD/OSD-2 over Ethernet Source:

59 SCSI OSD/OSD-2 over SAS Source:

60 ANSI T10 OSD SCSI Targets Host Host FileiSCSI System FileiSCSI System iscsiids iscsiids SAS, FCP, iscsi, SRP, iser, SCSI,... Disk Array OSD SCSI Controller OSD SCSI Controller

61 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

62 Storage Server - OSS Host FileiSCSI System iscsiids Interface File System OS (BSD, Solaris, Linux, Windows..) Block Back-End Device Driver OSS could be a migration path to provide object technologies to legacy block devices

63 Distributed Parallel Cluster File System With Storage Server (OSS) Application Server Application Server Application Server Application Server Application Server Application Server Ethernet, IB, Myrinet, Quadrics... Storage Server Storage Server IP Storage Server SAN Storage Device Storage Device Storage Device Single Data Image MDS

64 Lustre Cluster File System World s Largest Network-Neutral Data Storage and Retrieval System The worlds most scalable parallel filesystem 10,000 s of clients Proven technology at major HPC installations: > Tokyo Tech, TACC (Sun), LANL, LLNL, Sandia, PNNL, NCSA, etc. 70% of Top10 run Lustre 50% of Top30 run Lustre 15% of Top500 run Lustre

65 Sun HPC Storage Solutions Long-Term Retention and Archive Compute Engine Data Cache Tier 2 Fixed Content Archive Computer Cluster IB Network Load Tier 2 Near-Line Archive Automated Migration Archive LAN and SAN Scalable Storage Cluster Data Movers Tier 1 Archive and Home Directories

66 Scale-Up Vertical Scaling

67

68 The Enterprise Dilemma While 70% or more of the digital universe is created, captured, or replicated by individuals consumers and desk and information workers toiling far away from the datacenter enterprises, at some point in time, have responsibility or liability for 85%. Responsibility includes: security, privacy protection, copyright protection, screening for obscenety, detecting fraud, archiving content, searching and retrieval, disposal...

69 THE WORLD NEEDS ONLY FIVE COMPUTERS Let's see, the Google grid is one. Microsoft's live.com is two. Yahoo!, Amazon.com, ebay, Salesforce.com are three, four, five and six. (Well, that's O(5) ;)) Of course there are many, many more service providers but they will almost all go the way of YouTube; they'll get eaten by one of the majors. And, I'm not placing any wagers that any of these six will be one of the Five Computers (nor that, per the above examples, they are all U.S. West Coast based --- I'll bet at least one, maybe the largest, will be the Great Computer of China). I'm just saying that there will be, more or less, five hyperscale, pan-global broadband computing services giants. There will be lots of regional players, of course; mostly, they will exist to meet national needs. Greg Papadopoulos, Chief Technology Officer and Executive Vice President of Research and Development Sun Microsystems blogs.sun.com/gregp/entry/the_world_needs_only_five

70 Storage in ipods ipod classic 2007 = 160GB Moore's law for storage ipods first weekend i.e. Xmas 2011: 270PB ipod storage on a single weekend 1Gbit wireless each... > 4 years = 48 month / 18 = 2.66 [((( %)+100%)+70%) ~ 1TB GByte Year

71 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

72 Content Addressable Storage - CAS OIDs are hash values derived from the objects' content Used as digital archive systems for long term fixed content data ECM applications used as data injection machines Proprietary API, XAM, WebDAV, NFS, FTP, CIFS... Database File system OS (BSD, Solaris, Linux...)

73 Data Placement Improved Resilience 1 Node Node Node Node 2 Node Node Node Node 3 Node Node Node Node Node Node Node Node = Data = Parity

74 RAIN Redundant Array of Inexpensive/Independent Nodes CAS CAS CAS Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) CAS CAS CAS Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) CAS CAS CAS Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) Database File system OS (BSD, Solaris, Linux...) Single Data Image

75 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

76 CAS: Content Addressable Storage ID Enterprise Content Management Injection Engine

77 CAS: Content Aware Storage ID Extensible Metadata MD/Content Search Trigger Applications Enterprise Content Management Injection Engine

78 Topics Block-Based Data Access File-Based Data Access -Based Data Access > -Based Storage Devices (OSD) > Storage Systems > Storage Server (OSS) > Content Addressable Storage (CAS) > Content Aware Storage (CAS) Intelligent Storage Nodes (ISN)

79 Traditional Storage Architecture (cont'd) IP Server CPU & Memory Complex Clients Server SAN Disk Controller Frame Disk Array FRAME HDD Box Logic-Box Up to 256 drives Controller & 128 drives AC-DC AC-DC Power Supply (750 mm) Power Supply (650 mm) (25.59 in.)

80 Traditional Storage Architecture (cont'd) IP Server CPU & Memory Complex Clients Server SAN HW RAID HW RAID SCSI Port to OS Controller Controller iscsi TCP/IP Adapter Driver

81 The New Challenges I/O Performance Availability Reliability Easiness to store, organize, retrieve and dispose Scale to billions of objects across PB of storage Horizontal scale (cloud storage, infinite disk,...) Complex data operations > aggregate, join, view, sort, convert, encrypt... Enhanced search operations (indexing) Flexibility to present data Customized storage behavior Maximize capacity through data reduction technologies (de-dup, classification,... Retain objects for the prescribed period, delete expired data securely Guard against data theft or loss, authentication, authorization Reduced administration costs Lengthy storage requirements can exceed normal life of media Policy managers Power savings (tape, MAID,...) Dynamic metadata: reputation management, access patterns, trending...

82 The Vertical Consolidation Storage and server Migration of data processing applications No I/O is best I/O Server Server Server Server Storver Storage Storage Storage Storage

83 The Storage Evolution: From Blocks, Files and s to Storage Systems Storagetechnology im Radisson SAS Hotel Schloss Fleesensee veranstaltet durch das Deutsche Zentrum für Luft- und Raumfahrt e.v. Christian Bandulet Principal Engineer Global Technical Advisory Team Sun Microsystems Inc.