Coming Changes in Storage Technology. Be Ready or Be Left Behind

Transcription

1 Coming Changes in Storage Technology Be Ready or Be Left Behind Henry Newman, CTO Instrumental Inc. Copyright 2008 Instrumental, Inc. 1of 32

2 The Future Will Be Different The storage landscape is headed for dramatic change, thanks to new technologies like Fibre Channel over Ethernet t (FCoE), NFV4.1 (pnfs), object-based storage and SAS that will impact everything from storage interconnects to disk drives and trickle down Copyright 2008 Instrumental, Inc. 2of 32

3 What is covered today New Interconnection technologies NFS v4.1 new data access method Future storage technologies Continuing storage challenges Copyright 2008 Instrumental, Inc. 3of 32

4 NEW INTERCONNECTION TECHNOLOGIES Fibre channel over Ethernet is likely going to change the world Copyright 2008 Instrumental, Inc. 4of 32

5 Why FCOE Will Be a Disruptive Technology Shift Commodity technologies are driving the storage market Take the case of enterprise disks (FC today) as compared to SATA Estimates are a factor of 10x different in volume for SATA FCOE combines commodity technology Ethernet with enterprise technology FC protocol The cost per Gbit/sec of Ethernet has been historically far lower than the cost of FC just given the volume Who does not have Gbit Ethernet on the motherboard? And who will not have 10GbE in a few years Copyright 2008 Instrumental, Inc. 5of 32

6 What Is the Impact for the Enterprise? Today, Ethernet networks are used for TCP/IP traffic and FC is used for data to RAID/disk and tape storage A small percentage use Infiniband (IB) for data traffic and communication Having a single fabric to manage switches, NICs and cables is appealing given the cost FCOE will likely require dedicated traffic to storage Currently 10 GbE is expensive but prices are dropping Recently, fairly dramatically Copyright 2008 Instrumental, Inc. 6of 32

7 Cost Differences Today Date Cost Per Gb 1 Cost Per Gb Cost Per Switch Cost Per Cost Per Gb 2 Cost Per Port port GbE 2 port GbE Port 4 port Blade Gb 1 port ports FC 16 Port Switch FC 10-Dec-07 $ $ $3, $ $ $ Dec-07 $ $ $3, $ $ $ Jan-08 $ $ $3, $ $ $ Jan-08 $ $ $3, $ $ $ Feb-08 $ $ $3, $ $ $ Feb-08 $ $ $3, $ $ $ Mar-08 $ $ $3, $ $ $ Apr-08 $ $ $3, $ $ $ Jun-08 $ $ $3, $ $ $ Jul-08 $ $ $3, $ $ $ Jul-08 $ $ $3, $ $ $ Aug-08 $ $ $1, $ $ $ Note the price drops for 10GbE and the cost per Gb is dropping far below FC. Source: PC Connection web site Copyright 2008 Instrumental, Inc. 7of 32

8 Enhanced Ethernet Lossless fabric for storage traffic on converged networks Required for FCoE Deterministic latency for High Performance Computing and Communications (HPCC) traffic To address IB connectivity Virtual links by traffic class, applying pause-per-class, and end-to-end congestion management and notification Determinist QOS Copyright 2008 Instrumental, Inc. 8of 32

9 NFS V4.1 NEW DATA ACCESS METHOD High performance data access via NFS will be a reality Copyright 2008 Instrumental, Inc. 9of 32

10 Currently NFS Does Not Scale Current NFS makes small requests to the file system and network Small requests Many metadata t requests A draft for changes to NFS v4, called NFS v4.1, exists and includes pnfs The draft can be found on the IETF web site and is called draft-ietf-nfsv4-minorversion txt NFS v4.1 is often referred to as pnfs as that was the original name when first proposed to the IETF Features needing to be added to the NFS v4 protocol include: Sessions/RDMA Directory delegations Copyright 2008 Instrumental, Inc. 10 of 32

11 What Will NFS v4.1 Look Like? Courtesy of Panasas Parallel access with DMA support and multiple access methods Copyright 2008 Instrumental, Inc. 11 of 32

12 Who Is Working on NFS v4.1 Three different access methods have been proposed FILES: NFS/ONCRPC/TCP/IP/GE for files built on subfiles NetApp, Sun, IBM, University of Michigan CITI BLOCKS: SBC/FCP/FC or SBC/iSCSI for files built on blocks EMC, IBM ( pnfs-blocks-03.txt) OBJECTS: OSD/iSCSI/TCP/IP/GE for files built on objects Panasas, Sun ( pnfs-obj-03.txt) Copyright 2008 Instrumental, Inc. 12 of 32

13 FUTURE STORAGE TECHNOLOGY What other technology could change our world Copyright 2008 Instrumental, Inc. 13 of 32

14 T10 Object Storage Device OSD allocation of space, however, splits the name space and allocation between the host side and the storage which handles the allocation This breaks the traditional file system name space model breaking allocation and Some vendors are developing OSD hardware for the 2010 timeframe OSD also provides an interface for device authentication and an encryption model In addition, OSD has other significant advantages I/O Hints could a big win for some applications OSD will the be the first major change in how file systems communicate Both Panasas (currently has this) and Sun (QFS working on this for the next release) Copyright 2008 Instrumental, Inc. 14 of 32

15 OSD is a Disruptive Technology Advance In today s block based world, fragmentation is becoming a significant issue for long term sustained performance With OSD, fragmentation is virtually eliminated Current RAID levels are fixed values with fixed allocations The OSD object manager will be able to select the RAID level based on the object size Mixing small and large files in a file system today and having high performance for both is difficult with fixed RAID allocations OSD addresses these problems along with enhancements for security and performance hints The problem is you cannot get at these enhancements in a standards based way from user space Copyright 2008 Instrumental, Inc. 15 of 32

16 What Will It Impact? OSD will change file and RAID controllers as well The adoption of OSD is in question because it presents a chicken and egg scenario Without OSD drives available, most vendors will not develop an OSD file system or an OSD storage manager Courtesy of Panasas Copyright 2008 Instrumental, Inc. 16 of 32

17 CONTINUING STORAGE CHALLENGES Storage Needs to scale and the datapath needs to change Copyright 2008 Instrumental, Inc. 17 of 32

18 Enterprise Disk Drive Density Is Not Scaling Year Capacity (GBytes) Overall Rate of Increase over last increase Rate of Increase Per Year Rate of Increase All Years Copyright 2008 Instrumental, Inc. 18 of 32

19 Tape Density Is Not Scaling Vendor Di Drive Media Introduced d Capacity GB Peak Xfer Rate Perf. BW per MB/sec MB/sec Increase per GB uncompressed uncompressed IBM 3420 Reel-to-Reel IBM IBM IBM 3490E IBM 3490E 3490E IBM E IBM STK SD-3 SD STK T9840A IBM 3590E 3590E IBM 3950E 3590E STK T9940A LTO LTO LTO STK T9840B STK T9940B IBM 3590H LTO LTO-II LTO IBM LTO LTO-III LTO IBM 3592J 3592J STK T10000 Titanium IBM TS JA/JW LTO LTO-IV LTO Sun T10000B Titanium IBM TS1130 JB/JX Copyright 2008 Instrumental, Inc. 19 of 32

20 Performance Is Not Scaling for HDD or Tape Uncompressed MB/sec Compressed Improvement since 1990 Uncompressed Improvement since Tape Compressed MB/sec Uncompressed MB/sec Compressed Uncompressed Inprovment Inprovment since 1990 since 1990 LTO T10000B TS Copyright 2008 Instrumental, Inc. 20 of 32

21 Latency Scaling Is Worse than Performance Year Approx. Seek +latency in milliseconds Improvement inch inch This trend will not change Spinning drives at 20K is not going to happen soon Copyright 2008 Instrumental, Inc. 21 of 32

22 Flash and the Enterprise There are three problems limiting flash in the enterprise: Wear leveling SMART (Self-Monitoring, i Analysis, and Reporting Technology) Monitoring Interface errors and performance Copyright 2008 Instrumental, Inc. 22 of 32

23 Wear leveling Most vendors quote absurd values for wear leveling Here is what Mtron says: Endurance Write Endurance Read Endurance Wearleveling algorithm >140 50GB write per day 50 GB per day is ~1.7 MB/sec Not really useful in most environments All vendors have the same marketing speak Wear leveling has not been proven in the enterprise What if you do RAID-1 and both drives fail at the same time? Copyright 2008 Instrumental, Inc. 23 of 32

24 SMART Monitoring SMART is a standard that was developed for disk drives, some of the error conditions found in flash likely do not fit within the framework for SMART Add to this that when integrated into a RAID device, which is doing predictive failure analysis to ensure high reliability, any RAID vendor must integrate the flash vendors' SMART implementation into their monitoring i and management framework. I believe that this is likely significant since there is no standard d for SMART statistics for flash New proposal to ANSI for flash but early stages Copyright 2008 Instrumental, Inc. 24 of 32

25 Interfaces Storage interconnect technology and Reed Solomon error encoding has not changed much since the late 1980s or early 1990s Error encoding was designed d for channels less than 1/10th speed of today and likely in environments with far fewer channels According to industry sources the undetectable error rate of the channel is: FC/SAS 10 E-21 bits SATA 10 E-17 bits The problem is not the drive but the interface to the drive True for both tape and disk Copyright 2008 Instrumental, Inc. 25 of 32

26 What the rate means to you Sustain Transfer Rate Per Second for a Year UDBER GB/sec 10 GB/sec 100 1TB/sec 10 TB/sec 100 TB/sec GB/sec GB/sec 1.E E E-19 1.E E E E E-15 1.E These are rates for a perfect world where the channels are operating at the specified rate of 10E -12 and corrected to10e -17/19 What happens when the world is not perfect? Copyright 2008 Instrumental, Inc. 26 of 32

27 Copyright 2008 Instrumental, Inc. 27 of 32 Increased reliability in datapath T10 Protection Information (PI) (was DIF) allows a checksum to be transmitted from the HBA and application to the disk drive Adds a HBA checksum and LBA check through the datapath Also adds an application checksum but: What standard framework exists to allow the application to do this? None is the answer as it is not supported in user space PI significantly improves reliability in the datapath No support for SATA or tapes

28 Latency Tolerance for CPUs (last 35 years) Hardware evolution based on memory component latencies have caused dramatic shifts in design Vectors started it all because of memory performance limitations Multiple l levels l of memory (L1, L2, L3, NUMA) Multi-threaded CPUs All of these changes are based on the need to hide latency when accessing memory as latency has increased as a function of CPU performance This latency trend is not going to change Copyright 2008 Instrumental, Inc. 28 of 32

29 Latency Tolerance for the Data Path (last 35 years) Application data path has not changed in a similar il way to address latency Storage latency has changed a small amount compared to CPU performance File systems do not pass topology to block devices to impact latency This has a major impact on RAID storage and read-ahead You cannot read-ahead if you do not know where the data is POSIX is old and decrepit and we are all bound by required POSIX semantics Copyright 2008 Instrumental, Inc. 29 of 32

30 It s All About Latency Application changes such as multithreading are the same techniques used to address latency issues in the late 70s and 80s with vector based computers Systems today are efficient if they hide memory latency I/O is efficient today if it can hide latency by multithreading I/O can be efficient by making large I/O requests Latency in the data path is growing as a function of computation True for memory True for I/O Copyright 2008 Instrumental, Inc. 30 of 32

31 Current Datapath Changes are Needed Current POSIX system calls open/read/write/aio. Limited communication with OS layer Application Application Future Needs ILM framework that applications can interface. Information is passed on through generations of systems. User searchable POSIX Atomic operations open/read/write/aio. No communication with physical layer Operating System Operating System ILM archival and backup policies based on per file information or defaults. Block based storage and limitations of 30+ year old technology Storage and Transport Storage and Transport More intelligent storage to address issues such as fragmentation, encryption, OSD and reliability to name a few Data path today is the Focus on end-to-end view of I/O same as the data path in whole data path. The benefits of this framework extend to the 20 years ago entire data path, not just ILM. Copyright 2008 Instrumental, Inc. 31 of 32

32 QUESTIONS? Thanks for Listening Copyright 2008 Instrumental, Inc. 32 of 32