Cloud Storage and Parallel File Systems

Transcription

1 Cloud Storage and Parallel File Systems SNI Storage Developer Conference (SDC09), Sept 2009 Garth Gibson Carnegie Mellon University and Panasas Inc and

2 irth of RID ( ) Member of 4th erkeley RISC CPU design team (SPUR: 84-89) CPU design is a solved problem SYSTEM PERFORMNCE depends on data storage, so IM 3380 disk is 4 arms in a 7.5 G washing machine box SLED: Single Large Expensive Disk New PC industry demands cost effective 100 M 3.5 disks Enabled by new SCSI embedded controller architecture Use many PC disks for parallelism: CM SIGMOD 1988 RID paper 2 Garth Gibson, Sept 15, 2009

3 Object Storage (CMU NSD, 95-99) efore NSD there was store&forward Server-ttached Disks (SD) Move access control, consistency out-of-band and cache decisions Raise storage abstraction: encapsulate layout, offload data access Now NSI T10 SCSI command set standard (v2 emerging soon) 3 Garth Gibson, Sept 15, 2009

4 Fine Grain ccess Enforcement State of art is VPN of all out-of-band clients, all sharable data and metadata ccident prone & vulnerable to subverted client; analogy to single-address space computing File manager Private Communication NSD Integrity/Privacy Secret Key 2: Caprgs, CapKey 1: Request for access Object Storage uses a digitally signed, objectspecific capabilities on each request CapKey= MC SecretKey (Caprgs) Caprgs= ObjID, Version, Rights, Expiry,... ReqMC = MC CapKey (Req,NonceIn) 3: Caprgs, Req, NonceIn, ReqMC Client Secret Key NSD 4: Reply, NonceOut, ReplyMC ReplyMC = MC CapKey (Reply,NonceOut) 4 Garth Gibson, Sept 15, 2009

5 Panasas Inc. Spins Out (1999) Storage that accelerates the world s highest performance and most data-intensive applications - 10,000+ clients, 50+ G/s, 1,000+ storage nodes - Primary storage on first & fastest computer (Los lamos) Founded 1999, shipping solutions since Software innovation, packaged with industry-standard HW - Scalable RID over storage nodes, end-to-end check codes - Integrated SSD, extensive H, snapshot, async mirroring Record Q2 growth despite economy - 50% growth in revenue YoY vs -18% WW - Strong growth with partners, international, new accounts - Strong balance sheet 5

6 Panasas Storage Cluster Integrated 10GE Switch Shelf Front 1 D, 10 S attery Module (2 Power units) Shelf Rear Directorlade Midplane routes GE, power Storagelade 6

7 Leaders in HPC choose Panasas ENERGY SWIFTCOMPNY 7

8 World's Fastest Computer Los lamos RoadRunner Panasas Performance Scales # The world's first Linpak sustained 1.0 petaflops system. # #1 on TOP500 - chieved petaflops on May 25, 2008 # #3 on Green 500 # Time Magazine s 10 th Top Innovation for 2008 # een in development since 2002 # Open science phase in progress now # More than 4 P of Panasas # Greater than 50 G/s to apps # Computational chemistry at 369 Tflops Ref:

9 SciDC Petascale Data Storage Institute Eight organizations on the team Carnegie Mellon University, Garth Gibson, PI U. of California, Santa Cruz, Darrell Long U. of Michigan, nn rbor, Peter Honeyman Lawrence erkeley Nat. Lab, John Shalf Oak Ridge National Lab, Phil Roth Pacific Northwest National Lab, Evan Felix Los lamos National Lab, Gary Grider Sandia National Lab, Lee Ward Garth Gibson, 11/21/2008

10 Future is data-led Expert human translator LEU Score Usable translation Human-edittable translation Topic identification Useless Google ISI IM+CMU UMD JHU+CU Edinburgh NIST: translate 100 articles rabic-english competition 2005 outcome: Google wins! Qualitatively better on 1st entry rute force statistics with more data & compute!! 0.1 Systran Mitre 200M words from UN translations 1 billion words of English grammar 0.0 FSC 1000 processor cluster 10

11 Science of many types is data-led Contact Field Comments J Lopez, CSD strophysics SDSS digital sky survey including spectroscopy, 50T T Di Matteo, Physics strophysics igben HCosmo hydrodynamics (1 particles simulated), 30T F Gilman, Physics strophysics Large Synoptic Survey Telescope, LSST (2012) digital sky survey, 15T/day C Langmead,CSD iology Xray, NMR, CryoEM images; Sim d molecular dynamics trajectories J ielak, CE Earth sciences USGS sensor images; Sim d 4D earthquake wavefields >10T/run D rumley, ECE Cyber security Worldwide Malware rchive; 2T doubling each year O Mutlu, ECE Genomics 50G per compressed genome sequencing; expands to Ts to process Yu, ECE Neuroscience Neural recordings (electrodes, optical) for prosthetics; G each J Callan, LTI Info Retrieval ClueWeb09, 25T, 1 high rank web pages, 10 languages T Mitchell, MLD Machine Learning English sentences of ClueWeb for continuous automated reading (5T) M Herbert, RI Image Understanding Flickr archive (>4T); broadcast TV archive; street video; soldier video Y Sheikh, RI Virtual Reality Terascale VR sensor, 1000 camera+ 200 microphone, up to 5T/sec C Guestrin, CSD Machine Learning log update archives, 2T now + 2.7T/yr (about 500K blogs/day) C Faloutsos, CSD Data Mining Wikipedia change archive (1T), Fly embryo images (1.5T), links from Yahoo web S Vogel, LTI Machine Translation Pre-filtered N-gram language model based on statistics on word alignment, 100 T J aker, LTI Machine Translation Spoken language recording archive, many languages, many sources, up to 1P ecker, RI Computer Vision Social network image/video archive for training computer vision systems, 1-5T 11

12 Cloud as Home for Massive Data Moving massive data is not a good idea PanStarrs:1 T/day over nationally funded networks Large Synoptic Survey Telescope: 15 T/day planned South frican firm: homing pigeon faster than net Seismic survey firms use trucks and helicopters uild processing for massive data & share Share cost of storage, buy your own processing Private clouds share among units, college depts What semantics for storage in cloud? R. Wolski (Eucalytus): we haven t seen the cloud storage model yet 12

13 Why do I care about common semantics? Programmer productivity & ease of deployment HPC FSs are more mature, wider feature set In the comfort zone of programmers (vs cloud FS) High concurrent reads and writes Wide support, adoption, acceptance possible pnfs (NFS v4.1) working to be equivalent Netpp, EMC, Sun, IM, Panasas, Reuse standard data management tools ackup, disaster recovery, tiering, 13

14 Cloud HDFS & HPC PVFS Meta-data servers Store all file system metadata Handle all metadata operations Data servers Store actual file system data Handle all read and write operations Files are divided into chunks (objects) Chunks of a file are distributed across servers Designed for collocation of disks & compute HDFS is, PVFS is not (mazon-like) but can do it 14

15 HPC PVFS shim under Hadoop Hadoop applications Hadoop framework Extensible file system PI ~1,700 lines of code PVFS shim layer Readahead buffer HDFS client library HDFS client library HDFS servers HDFS servers PVFS shim layer PVFS shim layer Unmodified PVFS Unmodified PVFS client library (C) client library (C) Unmodified PVFS Unmodified serverspvfs servers File layout info Replication Client Client Server Server 15

16 It takes a little work, but PVFS gets there No changes in PVFS, just stdio -class shim library Out of the box, a big difference PVFS cache coherent by not prefetching; HDFS immutable Prefetching in the shim pretty much trivial but not enough HDFS exposes IP of each block s data server Hadoop schedules read on best node Exposing layout in PVFS simple, given an PI rchitecture is issue: where are disks? RID? In nodes/hdfs or not/s3 Or not using all nodes Rack awareness similar but smaller differences 16

17 closer look: reading a single file ggregate read throughput (M/ s) PVFS (no replication) HDFS (no replication) Completion Time (sec) Read (16G, 16 nodes) Number of Clients PVFS HDFS N clients, each reads 1/N of a single file (left) Round-robin file layout in PVFS avoids contention that occurs with random allocation in HDFS ut if Hadoop is given all work at once (right) Scheduling lots of work overcomes contention 17

18 ggregate write throughput (M/s) Writing has larger differences Number of Clients PVFS (no replication) HDFS (no replication) Writing: N clients to N files (left) HDFS writes locally first, so linear PVFS writes over net, shifting work to idle clients Writing: N clients to one file (right) HDFS does not support multiple concurrent writers Completion Time (sec) Parallel Copy (16G, 16 nodes) PVFS (16 writers) HDFS (1 writer) 18

19 Completion Time (sec) Grep (100G, 50 nodes) OpenCirrus/Yahoo M45 tests PVFS HDFS Completion Time (sec) Sort (100G, 50 nodes) PVFS & HDFS similar for reading as before PVFS HDFS PVFS(2 copies) HDFS non-striped local copy saves network traffic in network-limited write-intensive sort How fundamental is network-limited in the cloud? Sort (100G, 50 nodes) 1000 Cloud surge protection : data rarely local to small compute Network Traffic (G) PVFS HDFS PVFS(2 copies) 19

20 CMU testbeds: 10 GE networking Combined: 3 TF, 2.2 T, 142 nodes, 1.1K cores, ½ P 1 GE vs 10 GE Network vs disk bottleneck OpenCloud writes remote as fast a local EtherCLOS networks at SigCOMM No need for datacenter network to be big limit 20

21 Revisit HDFS Triplication GFS & HDFS triplicate every data block Triplication: one local + two remote copies 200% space overhead ut RID5 *is* simple? 21

22 Revisit HDFS Triplication GFS & HDFS triplicate every data block Triplication: one local + two remote copies 200% space overhead ut RID5 is simple? Can be done at scale Panasas does it with Object RID over servers >1PF, >50G/s, >10Kclients 22

23 Revisit HDFS Triplication GFS & HDFS triplicate every data block Triplication: one local + two remote copies 200% space overhead ut RID5 is simple? Panasas does it >PF, >50 G/s, >10K clients Can be done at scale ut sync error handling hard GFS & HDFS defer repair ackground task repairs copies Notably less scary to developers 23

24 DiskReduce: ackground Repair for RID Start the same: triplicate every data block Triplication: one local + two remote copies 200% space overhead 24

25 DiskReduce: ackground Repair for RID Start the same: triplicate every data block Triplication: one local + two remote copies 200% space overhead ackground encoding In coding terms: Data is, Check is,, f(,)=+ + 25

26 DiskReduce: ackground Repair for RID Start the same: triplicate every data block Triplication: one local + two remote copies 200% space overhead ackground encoding In coding terms: Data is, Check is,, f(,)=+ Std single failure recovery + 26

27 DiskReduce: ackground Repair for RID Start the same: triplicate every data block Triplication: one local + two remote copies 200% space overhead ackground encoding In coding terms: Data is, Check is,, f(,)=+ Std single failure recovery Double failure recovery uses parity and related data blocks Then applies standard single failure recovery + 27

28 Preliminary Evaluation Pre-OpenCirrus testbed 16 nodes, PentiumD dual-core 3.00GHz 4G memory, 7200 rpm ST 160G disk Gigabit Ethernet Implementation specification: Hadoop/HDFS version Test conditions File size distribution from Yahoo! M45 sample enchmarks modeled on Google FS paper enchmark input after all parity groups are encoded enchmark output has encoding in background No failures during tests 28

29 The Obvious Little Degradation Completion Time Sort with 7.5G data Original Hadoop 3 replicas Parity Group Size 8 Parity Group Size Completion Time Grep with 45G data Original Hadoop 3 replicas Parity Group Size 8 Parity Group Size 16 29

30 DiskReduce v2.0 Only 33% saving with v1.0 Stopped work on v1.0 b/c encoding didn t scale Started v2.0 model based on codeword selection Select a RID group before creating data Retain random distribution Worry more about cleaning post block delete Return of Small Write bottleneck of RID 5 & 6 Select blocks in group to promote co-deletion We ll get to performance degradation. ackup jobs like having a second copy 30

31 Closing: Clouds & Parallel File Systems ig similarity between cloud & parallel file systems Similar scale 1000s of nodes Two layer implementation, object-based PVFS can match HDFS performance Datacenter network bottleneck & colocation overplayed 10GE & fat trees coming; surge protected = non-local Differences stem from maturity & simplicity Simpler semantics works sooner ut users want familiar semantics Writing, directories, concurrency, low overhead,. Integrates into broader infrastructure seamlessly My view: layer Parallel NFS under cloud PIs Exposing it as customers ask for richer semantics 31

32 Thank you 32

33 INCST: TCP Throughput Collapse Cluster Environment 1Gbps Ethernet 100us Delay 200ms RTO S50 Switch 1M lock Size [Nagle04] called this Incast; used app-level & switch selection workaround [Fast08] Cause of collapse: 200ms TCP timeouts on loss of whole window Is reducing timeout effective, practical and safe in the real world? 33

34 Mitigating Incast Cluster Environment 1Gbps Ethernet 100us Delay S50 Switch 1M lock Size Eliminating RTO bound: 5ms timeouts - clock granularity limit Single line, server-only change is effective at medium scales Eliminating RTO bound + Microsecond TCP timeouts avoids Incast Exploits High resolution timers to avoid high interrupt overhead 34

35 The Need for Microsecond Timeouts Simulation Environment 10Gbps Ethernet 20us Delay 40M lock Size Future datacenters: More bandwidth, less delay, more servers Retransmission timeouts should not be bounded below ccepted into SIGCOMM