Commodity Reliability And Practices or Building Reliable Systems with CRAP

Size: px

Start display at page:

Download "Commodity Reliability And Practices or Building Reliable Systems with CRAP"

Hannah Harvey
5 years ago
Views:

1 Commodity Reliability And Practices or Building Reliable Systems with CRAP Thomas M. Ruwart Chief Scientist University of Minnesota Digital Technology Center Minneapolis, MN October 8,

2 Why you are here To learn about a relatively new concept in the storage industry Concepts behind CRAP Related issues and problems Addressing those issues and problems 2

3 Orientation A bit of history How did we get to this point? Design requirements for enterprise-class storage Design requirements for consumer-class storage Principles of Commodity Reliability And Practices Why we need CRAP Conclusions 3

4 A bit of History What disk drives were like in the olden days in 10-year increments the view from someone who lived through it Future history were are we headed 4

5 1967 Disk drive platters were 30-inches in diameter Disk drives were the size of a large clothes washing machine A Million dollars per disk drive that held only a few MegaBytes Disk drives required daily maintenance of cleaning heads and platters Several people were crushed under the weight of their ipods which held only one song at a time 5

6 IBM 305 RAMAC 6

7 7

8 1977 Disk drive platters are now 14-inches in diameter Disk drive capacity up to about 315MB in a small washing-machine sized box Disk drives are still only used in computer centers Disk drives were just beginning to be sealed so that no physical maintenance was required for the heads or media Media maintenance for dealing with bad spots was required on a monthly basis ipods now carried by pack mules and elephants and as such, never caught on in the US 8

9 9

10 1987 The 8-inch disk drive form factor is the standard for enterpriseclass disk drives and uses the SMD or emerging IPI interface The PC revolution gives rise to consumer-class hard disk drives Consumer-class disk drives start with the small form factor inch full height ST506, ATA/IDE, SCSI, and ESDI are the new interfaces hard disk drives for consumer use Maintenance required to manage bad sectors but no other physical maintenance required RAID is just being invented again ipods become practical to the extent that they are smaller but battery life is about 3 seconds so the Walkman wins 10

11 11

12 1997 The 8-inch and 5.25-inch hard disk drive form factors give way to the 3.5- inch form factor (half-height) 3.5-inch form factor is dominant in both enterprise-class and consumerclass disk drives Parallel SCSI and Fibre Channel SCSI replace all other interfaces for enterprise-class disk drives IDE and EIDE are the standard consumer-class disk drive interface Capacities/Densities on both enterprise-class and consumer-class disk drives are equivalent No maintenance required bad sector management an integral part of each disk drive Zone-bit recording significantly increases disk drive capacities RAID Arrays are in wide use to increase data integrity and disk storage reliability as well as performance with minimal tradeoff in capacity ipods are deemed a waste of time because everything will be on minidisc or DAT 12

13 2007 The 3.5-inch, low profile (1-inch high) form factor is dominant in both enterprise-class and consumer-class storage The 2.5-inch form factor starting to make an appearance in the data center 2.5-inch (laptop drive) and 1.8-inch (ipod) disk drives are dominant in mobile devices Sub-1.8-inch disk drives being replaced by FLASH Consumer-class disk drives maintain a consistently higher bit density and drive capacity than Enterprise-class disk drives Enterprise-class disk drives maintain higher RPM and faster access times over consumer-class disk drives and 3-4x drive cost Enterprise-class disk drives giving way to consumer-class disk drives in previously Enterprise-class applications Concerns rise over use of Consumer-class storage in Enterprise-class applications ipods take over and begin to network themselves together to form another, higher intelligent life-form based solely on pop music. We call it: 13

14 2017 The 2.5-inch and 1.8-inch form factors are dominant in both enterprise-class and some consumer-class storage Data centers beginning to give up on disk drives and want to revert to using punch-cards Mobile devices are primarily FLASH-based Sub-1.8-inch disk drives being replaced by FLASH Consumer-class disk drives dominant in Enterpriseclass applications distinction between Consumerclass and Enterprise-class disk drives is narrow Concerns rise over the US President s demand to paint the White House purple 14

15 Where Drive Form Factors Come From

16 What s s inside a Video IPOD I knew it was too good to be true. 16

17 Let s s talk about the Lunatics High-End Computing (HEC) Community BIG data or LOTS of data, locally and widely distributed, high bandwidth access or high transaction rate, relatively few users, secure, short-term and long-term retention High Energy Physics (HEP) Fermilab, CERN, DESY BIG data, locally distributed, widely available, moderate number of users, sparse access, long-term retention DARPA Interagency High Productivity Computing Systems Design and build a peta-scale computer system that is usable for the year

18 HEP Fermilab and CMS The Compact Muon Solenoid (CMS - ) $750M Experiment being built at CERN in Switzerland Will be active in 2007 The Easy Part collecting the data Data rate from the detectors is ~1 PB/sec Data rate after filtering is a few GB/sec The Hard Part: Storing and Access Dataset for a single experiment is ~1PB Several experiments per year are run Must be made available to 5000 scientists all over the planet (Earth primarily) for the next years Dense dataset, sparse data access by any one scientist Access patterns are not deterministic 18

LHC Data Grid Hierarchy CMS as example, Atlas is similar human=2m Tier 1 ~PByte/sec CMS detector: 15m X 15m X 22m 12,500 tons, $700M. Online System Tier 0 +1 ~2.

Newman, CalTech and CERN analysis Institute Institute ~0.25TIPS Institute 100-1000 Mbits/sec Tier 4 Workstations Institute ~0.6-2.

19 LHC Data Grid Hierarchy CMS as example, Atlas is similar human=2m Tier 1 ~PByte/sec CMS detector: 15m X 15m X 22m 12,500 tons, $700M. Online System Tier 0 +1 ~2.5 Gbits/sec ~100 MBytes/sec event reconstruction event simulation French Regional Center German Regional Center Italian Center FermiLab, USA Regional Center Tier 3 Physics data cache Courtesy Harvey Newman, CalTech and CERN analysis Institute Institute ~0.25TIPS Institute Mbits/sec Tier 4 Workstations Institute ~ Gbps Tier2 Center Tier2 Center Tier2 Center Tier2 Center Tier2 Center Tier 2 ~ Gbps CERN/CMS data goes to 6-8 Tier 1 regional centers, and from each of these to 6-10 Tier 2 centers. Physicists work on analysis channels at 135 institutes. Each institute has ~10 physicists working on one or more channels physicists in 31 countries are involved in this 20-year experiment in which DOE is a major player.

20 20

21 21

22 What are the DARPA requirements? HEC Community The High Productivity Computing Systems (HPCS) from DARPA computations per second Peta-scale computing 1-10 trillion files in a single file system 100 s of thousands of processors Millions of process threads all needing and generating data TBytes/sec aggregate bandwidth to disk 30,000+ file creations per second Focus on ease of use, efficiency, and RAS 22

23 Lots of things have to scale File System Attributes Teraflops Memory size (TB) File system size (TB) ,000 20,000 Number of Client Tasks ,768 65,536 Number of Users 1,000 4,000 6,000 10,000 Number of Directories 5.0* * *10 7 2*10 8 Metadata Rates Data Rate 500/sec 1 mds 3 GB/sec 2000/sec 1 mds 30 GB/sec 20,000/sec n mds 100 GB/sec 50,000/sec n mds 400 GB/sec Number of Files 1.0* * * *

24 1TB/sec What are we getting ourselves into? 20,000 disk 50MB/sec/disk 10ms average access time 2 million 1TB/disk 20PB raw 25watts/disk (including cooling power) 500 KWatts 40,000 disk drives in an real design to include redundancy Space and power/cooling increase by 2x 1MWatt And that is just the beginning. 10TB/sec would be up to 400,000 disk drives 24

25 What does 1TB/sec really mean? To what? 1,000 1GB/sec each? 100,000 processes at 10MB/sec each? Assumes a process/processor can absorb/generate data at that rate Current data:instruction ratio is about 10:1 Therefore, 1TB/sec implies 100GFlops Thus 1PFlop implies a data rate of 100TB/sec opps. 25

26 Digging ourselves in deeper? 1 Trillion Files 30,000 file creations per second for 1 year = 1 trillion files 1PB of MetaData to describe 1Trillion files Finding any one file within 1 Trillion files Finding anything inside of the 1 Trillion files This is a major transactional problem not a bandwidth problem Traditional file systems and associated [POSIX] semantics break down at these scales need new/relaxed semantics Is the concept of a file still valid in this context? 26

27 The Growing Disk Drive Bottleneck Subsystem E 1 Increase Network I/O x Intel CPU x Storage Channel I/O x PCI x Intel Front Side Processor Bus x Random Disk IOPS x Random Disk IOPS per Gbyte 5, x Sequential Disk I/O x Sequential Disk BW/Gbyte x Notes: 1 Speed of subsystem in GBps 2 Ethernet 3 SCSI and Fibre Channel 4 IBM 3.5 inch drives internal data rate 5 IBM 3.5 inch drives se ek + rotational latency 6 Horison/Fred Moore 7 PCI versus 16xPCIe Source: "Evolution of Intel Microprocessors: 1971 to

28 Need more disks, not higher capacity ones Disk drive capacity improves faster than Data transfer rate Seek time Rotational Latency 28

29 Access Density 29

30 Serious Questions How do you package it? How do you maintain it? How do you connect it all together? How do you access/use a storage system with 250,000 disk drives? 30

31 How do you package this? Conservatively one hundred 3½ inch disks per rack with controllers 400 racks of disk drives and controllers 8,000 square feet 10TB/sec is 10 times this or about the size of two football fields (~100,000 sq ft) 31

32 How do you maintain it? Assume 40,000 disk configuration 2,000,000 hours MTBF per Enterprise-class disk 300,000 hours MTBF per Consumer-class disk ~4 disk failure per week for Enterprise-class disks ~20 failures per week for Consumer-class disks Continual rebuilds in progress 10TB/sec is 10 times this 32

33 How do you connect it all together? 10Gbit/sec/channel 1, % efficiency Implies a 2,000 channel non-blocking switch fabric What about transceiver failure rates When it breaks, how do you find the broken transceiver? 10TB/sec who on earth would want to do that? (don t ask) 33

34 How do you use this? Current file system technology is based on 30+ year-old designs and does not scale Disk I/O software stack is 30+ years old and does not scale Need lots of innovation in many areas Common shared file system interfaces Data Life Cycle Management and seamless integration into existing HEC environments Changes to standards that offer greater scalability without sacrificing data integrity Streaming I/O from zillions of single nodes Data alignment, small-block, large-block, and RAID issues File System Metadata Application Operating System Storage and Transport Application Operating System Storage and Transport 34

35 Commodity Reliability And Practices Processors, Networks, Graphics Engines have for the most part gone commodity Disk drives are still largely enterprise-class Significant pressure to move toward more use of commodity disk drives Requires a fundamental change in how we think about RAS for storage i.e. Fail-In-Place Assumes something is always in the process of breaking Must re-orient engineering to think about how to build reliable systems using unreliable components AKA How to build reliable systems using CRAP 35

36 What is CRAP? Successful systems are designed to be fault-tolerant faults are a normal occurrence Until recently, systems were designed and assumed to be highlyreliable and faults were an anomaly rather than a normal occurrence That was possible due to the relatively low part count in a given system In order to make computations go faster, significantly more parallelism is needed Parallelism implies a far greater part count A natural consequence of higher part count is a higher failure rate of some part in the system as a whole However, there is a great deal of duplication in highly parallel or high-part-count systems Need to take advantage of this parallelism 36

37 So, what is CRAP? High part-count systems must contain a large number of commodity components to be commercially viable Commodity components are less expensive and arguably less reliable in enterprise-class applications Thus, Systems should be designed with Commodity Reliability in mind Commodity Reliability means that at any point in time, something is in the process of failing within the system as a whole Given that something is always in the process of breaking, appropriate Engineering Practices must be employed to maintain: Data Access Data Integrity System Performance Requires a broader, systemic engineering view when designing with CRAP in order to ensure that things above and below a point of failure are minimally impacted software and hardware 37

38 What s s happening now? Areal Density is at about 150Gigabits per square inch 3.5-inch form factor is currently the standard 2.5-inch form factor is emerging in the enterprise SAS and SATA are getting significant traction OSD has been demonstrated and is in active development Consumer-grade storage is cheap cheap cheap Commodity interface speeds are up to 10Gigabits/sec Storage and Network processing engines are available New applications for storage are rapidly evolving Relaxed POSIX standards NFS V4 and Parallel NFS 38

39 Reaching Error Detection Code Limits Error Detection Codes are statistical in nature Current codes are rated at about 1 undetected error in bits transferred or about 1 bit in every 100 TeraBytes As storage capacities and transfer rates increase, the current error detection codes will no longer be sufficient to guard against silent data corruption Need stronger, non-intrusive error detection codes at multiple levels 39

40 Conclusions Need CRAP to design the next generation of scalable, high-performance systems System design is highly dependent on leveraging commodity components 40

41 Zone Bit Recording Commonly used technique to maximize use of media area More sectors recorded on the outer tracks than inner tracks Bit density remains constant but the data rate changes from zone to zone Typically zones on a disk varies from mfg to mfg, model to model 41

42 Example of Zones in ZBR Zone Map of the Fujitsu 60 GB 5400 RPM 2.5-inch SATA Disk Bandwidth in MB/sec Percent of disk from outer to inner tracks 42

43 Adaptive Formatting ZBR on steroids Each head is tuned to maximize the signal-to-noise ratio of each individual track Results in different effective data rate for each track Results in a variable number of sectors per track 43

44 Example of zones in AF Seagate 100 GB 5400 RPM Momentus Bandwidth in MB/sec Percent of disk from outer to inner tracks 44

Storage on the Lunatic Fringe

Storage on the Lunatic Fringe Thomas M. Ruwart Chief [Mad] Scientist tmruwart@atrato.com SNIA Developers Conference San Jose, CA September 25, 2008 Why you are here To learn that there are organizations