BUILDING HIGH AVAILABILITY SSD

Size: px

Start display at page:

Download "BUILDING HIGH AVAILABILITY SSD"

Esmond Thomas
6 years ago
Views:

1 BUILDING HIGH AVAILABILITY SSD Company overview Architecture & Performance Reliability Maximizing SSD Q&A Adam Chunn 15 March 2012 Lugano- Switzerland 1-56

financial trade Installed and in production in over 34 countries Shopped online Used pre-paid wireless

2 Select RamSan Facts The largest SSD installations in production in the world Currently operating in 10 major financial exchanges worldwide Used today by 7 out of 11 of the world s largest telecoms Conducted a financial trade Installed and in production in over 34 countries Shopped online Used pre-paid wireless Gamed online Sent a text message Placed online bet Booked a cruise or flight Used an ATM RamSan is Everywhere 2-56

Used today by 7 out of 11 of the world s largest telecoms Conducted a

Shopped online Used pre-paid wireless Gamed online Sent a text message

3 Select RamSan Facts The largest SSD installations in production in the world Currently operating in 10 major financial exchanges worldwide Used today by 7 out of 11 of the world s largest telecoms Conducted a financial trade Installed and in production in over 35 countries Shopped online Used pre-paid wireless Gamed online Sent a text message Placed online bet Booked a cruise or flight Used an ATM RamSan is Everywhere 3-56

patents granted and pending; many trade secrets Growing enterprise customer base in over 34 countries Strong Financial

4 Background on TMS Solid State Storage Leader Deep Domain Expertise Global Enterprise Customers Industry s highest performance, highest reliability, lowest latency, lowest power SSD solutions 33 years experience designing SSDs; 30+ patents granted and pending; many trade secrets Growing enterprise customer base in over 34 countries Strong Financial Performance No Venture Capital/Long Term Debt World Class Team Strong management and engineering teams Over 400 man-years of SSD experience 4-56

5 Key References See all of these and more in the Success Stories section of our web site at

6 ARCHITECTURE & PERFORMANCE 6-56

7 L = λw The long-term average number of customers in a stable systemlis equal to the long-term average effective arrival rate,λ, multiplied by the average time a customer spends in the system,w 1 Above is Little s Law which is just a fancy way to say that performance is based on Latency and Parallelism 1 Paraphrased from Little s Law, John D.C. Little and Stephen C. Graves, MIT 7-56

8 Flash Controller Design Basics Lookup Tables Flash Controller FPGA FLASH Media Write Buffer I/O Interface CPU CPU RAM Each controller handles 10 flash chips The Lookup Tables and Write Buffer is RAM accessible from the controller only. The I/O Interface and controller are both separate FPGAs The CPU is an embedded processor that handles all out-of-band operations DMAs are all processed completely in FPGA hardware 8-56

9 DMAs are hardware only FLASH Media Lookup Tables Flash Controller FPGA Write Buffer I/O Interface CPU CPU RAM DMAs are all processed completely in FPGA hardware 9-56

10 Decreasing Latency Lookup Tables Flash Controller FPGA FLASH Media Write Buffer I/O Interface CPU CPU RAM The Embedded CPU Remove from the DMA path, all non-critical flash memory book-keeping Write setup Garbage collection Error handling Health calculation Wear Leveling Statistics collection Formatting Backup/Restore Key Generation 10-56

11 Increasing Parallelism Lookup Tables Flash Controller FPGA FLASH Media Write Buffer I/O Interface CPU CPU RAM Increasing the number of flash chips that can run concurrently Which is done by increasing the number of flash chip controllers Each TMS flash chip controller can do 36 4KB DMAs in parallel (40 if you include the background chip RAID, or VSR, operations) A RamSan-70 has 8 controllers, so it can do 288 4KB operations simultaneously A RamSan-810 has 40 controllers, so it can do KB operations simultaneously 11-56

12 L = λw So, what else effects Latency and Parallelism? 12-56

13 L = λw What else effects Latency? CPU Speed not number of cores not number of chips Bus architecture North/south bridges PCIe hierarchy PCIe controller CPU Usage (so in a convoluted way, cores and chip counts do matter) 13-56

14 L = λw What else effects Latency? Operating system and file system OSes and file systems optimized for disks tend to count on slow data access to hide processing Modern OSes and file systems are now written to maximize SSD The driver, the bridge between the OS and the hardware It must be thin or else adds latency Linux, Windows, Solaris, VMWare, OSX, AIX We are actively trying to push the driver into the Linux kernel If measuring at the application layer, middleware (for example, databases) can inject latency 14-56

15 L = λw What else effects Parallelism? Large Blocks RamSan products break apart large block DMAs into multiple, parallel DMAs For example, a 64kB DMA is converted into 16 parallel 4kB DMAs A single application can be written to either have multiple threads of synchronous I/O or a single thread that allows multiple outstanding asynchronous I/O Most high-performance middleware does just this (such as Microsoft SQL, Oracle, et cetera) Running multiple applications can provide the same effect as a single application running multiple threads CPU becomes more and more of a bottleneck, however 15-56

16 CSCS Benchmark CSCS = Swiss National Computing Centre Independent evaluation of PCIe SSDs RamSan-70 results: by far the best IOPS result we have ever measured (300K+ random 4K IOPS) Unlike the FusionIO and Virident TachIOn devices, the bandwidth is almost independent of block size 16-56

RamSan Flash Product Portfolio RamSan-70 RamSan-710/810 RamSan-720/820

12/24TB 10TB 1.2M IOPS 400K/320K IOPS 500K/450K IOPS 1M IOPS 2.

0 Single Server Apps; Distributed filesystems 1U rackmount, 4x IB or FC

17 RamSan Flash Product Portfolio RamSan-70 RamSan-710/810 RamSan-720/820 RamSan-630 SLC Flash SLC/eMLC Flash SLC/eMLC Flash SLC Flash 900GB 5/10TB 12/24TB 10TB 1.2M IOPS 400K/320K IOPS 500K/450K IOPS 1M IOPS 2.5GB/s 5/4GB/s 5/4GB/s 10GB/s Full-height, halflength PCIe x8 2.0 Single Server Apps; Distributed filesystems 1U rackmount, 4x IB or FC ports 3U rackmount, 10x IB or FC ports Clustered Server Apps; Shared-storage filesystems (GPFS, GFS2, etc) 17-56

18 SPC Price/Performance Leader SPC-1 IOPS / Total TSC Price (USD) 1,60 1,40 1,20 1,00 0,80 0,60 0,40 0,20 Top 10 SPC-1 IOPS TMS RamSan- 400 TMS RamSan-630 0, SPC-1 IOPS SPC-2 MBPS x1k / Total TSC Price (USD) Top 10 SPC-2 MBPS TMS RamSan SPC-2 MBPS 18-56

19 Keys to Performance Hardware-only Data Path FPGA & Hardware Logic Faster than software-shared memory Software cannot add performance Virtualization is a software overhead to utilizing additional hardware QoS is a software overhead to give applications priority over another on shared hardware 19-56

20 RELIABILITY 20-56

Flash Quality Flash type matters! SLC in most RamSans Enterprise MLC (emlc) in RamSan-8x0 SLC is best but most expensive/least dense emlc chips last 10x longer vs.

21 Flash Quality Flash type matters! SLC in most RamSans Enterprise MLC (emlc) in RamSan-8x0 SLC is best but most expensive/least dense emlc chips last 10x longer vs. normal MLC TMS technologies like Variable Stripe RAID lengthen system life P/E Cycles (Thousands) Typical Chip Endurance MLC emlc SLC Flash Type 21-56

22 Combat Endurance Endurance of system is calculated: Flash Capacity Flash Quality Media Write Bandwidth 22-56

23 Combat Endurance 5TB RamSan-710 (SLC Flash) 5TB 100,000 1GBps = 15.8 Years Endurance 10TB RamSan-810 (emlc Flash) 10TB 30,000 1GBps = 9.5 Years Endurance 23-56

24 Combat Endurance Fight endurance with increased capacity emlc has 2x Capacity for same cost 2/3 rd endurance of SLC MLC is 3000 Writes where emlc is Writes MLC is ~1/4 th price of emlc storage Sustained writes do not make sense for MLC MLC will last less than a year from sustained writes at same cost and half the write workload 1TB 3, MBps = Less than a year 24-56

25 Flash Problems and TMS Solutions Problem Limited write-erase cycles Bit errors Block/plane/device failures Disturb errors (read, write, erase) Erases need big blocks and take a long time Solution Wear leveling ECC Block remapping, RAID, Variable Stripe RAID Voltage and timing adjustments Overprovisioning 25-56

26 Four Layers of Data Correction Layer System-level RAID 5 managed by centralized RAID controllers Module-level Variable Stripe RAID managed by each module across its chips Module-level RAID 5 managed by each module across its chips Chip-level ECC managed by each module using its chips Protection Module failure RamSan-720/820 only Sub-chip failure, System Longevity Chip failure Bit and block errors RamSan-720/820 introduce System-Level RAID 5 across Flash modules, plus the other mechanisms found on all RamSan Flash storage systems

27 Variable Stripe RAID (VSR) Patented VSR allows RAID stripe sizes to vary. If one die fails in a ten-chip stripe, only the failed die is bypassed, and then data is restriped across the remaining nine chips. 10 Chips 16 Planes FAIL 27-56

28 2D Flash RAID (RS-720/820) External Interfaces (FC, IB) Interface Interface RAID Controllers RAID Controller RAID Controller RAID 5 within Flash Modules (9 data + 1 parity) TMS 2D Flash RAID RAID 5 across Flash Modules (10 data + 1 parity + 1 hot spare) 28-56

29 RamSan-70 Overview 1. PCIe 2.0 x PowerPC CPU 3. Xilinx FPGAs GB usable SLC Flash (1374GB raw) GB DRAM 6. Super-Capacitors 7. Half-length card 1 7 Usable GB 650,000 4K IOPS 2.5 GB/s Bandwidth 30 µs sustained 4K Write Latency / 100us 4K Read Latency 10 Years Life Expectancy Series-7 Flash Controller 29-56

30 MAXIMIZING SSD 30-56

31 Segregation of Workload Metadata, Working Data, Archived Data Metadata is typically accessed the most, but takes up the least space Archived Date is accessed the least, but takes up the most space Moving high-access data into a high-performance medium has the greatest impact But the question is, what data makes sense to store on SSD? 31-56

32 Performance per Capacity Historically, TMS has designed DRAM-based SSD devices that performed GB/sec per GB of storage [Metadata] Our flash-based SSD devices perform GB/sec per TB of storage [Metadata, Working Data] Disk-based products typically grossly underperform SSD, but economical performance at >>TB of storage [Archive, Large Working Data] 32-56

33 Algorithm Matrix Low CPU Utilization + Low I/O Wait Low CPU Utilization + High I/O Wait High CPU Utilization + Low I/O Wait High CPU Utilization + High I/O Wait = Algorithm needs to provide more work = Great fit for SSD!! = In-memory work = Using Asynchronous I/O Add disks for growing capacity Add SSD for same size capacity 33-56

34 Q & A 34-56

RamSan-440 Overview 128-512 GB capacity 600,000 IOPS 4.

35 RamSan-440 Overview GB capacity 600,000 IOPS 4.5 GB/s throughput Latency 15 µs 2-8 FC Ports Industry Firsts: 512GB Non-volatile RAM storage RAM SSD with Flash backup RAID protected RAM and Flash modules TMS patented IO 2 Instant-on Input-Output option

Channel or InfiniBand Interfaces Hot Swappable Redundant Power

36 RamSan-440 Architecture RAID Protected RAM Boards Redundant Batteries Management Control Processor 4 Dual-ported Fibre Channel or InfiniBand Interfaces Hot Swappable Redundant Power Supplies 4U Chassis RAID Protected Backup Flash Redundant Fans 36-56

37 Series-7 Flash Controller Design Lookup Tables 4 GB RAM Cache Write Buffer Best Performance: 4K aligned I/O CPU (out of Primary Data Path) Write setup, Garbage collection, Error handling Out of the data path activities Flash Controller FPGA (Process all of the IN DATA activities) 4 GB RAM Cache I/O Interface Super Capacitors Memory Backup 37-56

RamSan-630 Overview 1-10TB capacity 1 Million IOPS 10 GB/s throughput Latency 80-250 µs Highest density SLC Flash SSD system available.

38 RamSan-630 Overview 1-10TB capacity 1 Million IOPS 10 GB/s throughput Latency µs Highest density SLC Flash SSD system available. Leverages proven flash core from the RamSan-20 and RamSan-620 Easily shared and multipathed through ten 8 Gbit Fibre Channel ports or QDR InfiniBand ports Enterprise Reliability Single Layer Cell (SLC) Flash Fault Tolerant Flash (FTF) Architecture Active Spare Flash 38-56

39 RamSan-630 Architecture 5 Dual-ported FC or IB Interfaces 1-10TB of SLC Flash Boards Management Control Processor 3U Chassis Redundant Fans Redundant Power Supplies 39-56

40 Embedded PowerPC RamSan-630 Flash Board RAID-5 Protected Flash 480 GB usable, 640 GB RAW On Board RAM ECC Protected Gateway FPGA 4 Flash Controllers Super capacitors 40-56

Highest density SLC Flash SSD system available in a 1U Series-7 Flash Controller Four 8 Gbit

41 RamSan-710 Overview 1-5 TB Usable capacity (6.8 TB Raw) 400,000 IOPS 5 GB/s throughput µs latency 150K+ Write/Erase Cycles per Cell Highest density SLC Flash SSD system available in a 1U Series-7 Flash Controller Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports Enterprise reliability Single Layer Cell (SLC) Flash Variable Stripe RAID (VSR) Active Spare 41-56

42 RamSan-710 Overview 4-20 Flash modules + 1 Active Spare 2 dual-ported 8Gb FC or QDR IB interfaces management control processor 1U chassis redundant power supplies redundant fans N+1 batteries 42-56

43 RamSan-710 Overview 43-56

RamSan-810 Overview 2-10 TB Usable capacity (13.7 TB Raw) 320,000 IOPS 5 GB/s throughput 70-225 µs latency (est.

44 RamSan-810 Overview 2-10 TB Usable capacity (13.7 TB Raw) 320,000 IOPS 5 GB/s throughput µs latency (est.) 30K+ Write/Erase Cycles per Cell Highest density emlc Flash SSD system available in a 1U Series-7 Flash Controller Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports Enterprise reliability enterprise Multi-Level-Cell (emlc) Flash Variable Stripe RAID (VSR) Active Spare 44-56

45 RamSan-810 Architecture 4-20 Flash modules + 1 Active Spare 1-2 interface modules management control processor 1U chassis redundant power supplies redundant fans N+1 batteries 45-56

46 Motherboard 46-56

47 Toshiba emlc Flash Series-7 Flash Controller FPGAs Gateway FPGA DDR DRAM PowerPC 400 MHz 47-56

48 Applications Suited for emlc Data Warehousing Web Content Hosting Low Bandwidth Log Files READ Intensive, Low WRITE Application For Users Writing at 600 MB/s, the Lifetime of the emlc RamSan-810 is rated at 10 years*. *2TB =10TB WRITES per Day *6TB =30TB WRITES per Day *10TB =50TB WRITES per Day 48-56

49 RamSan-70 Overview 49-56

RamSan-70 Architecture 1. PCIe 2.0 x8 5 3 5 4 2 2.

50 RamSan-70 Architecture 1. PCIe 2.0 x GB usable SLC Flash (1374GB raw) 3. Power PC CPU 333 mhz Xilinx FPGAs 5. 4GB DRAM 6. Super-Capacitors 1 7. Half-length card GB IOPS (4K) 2,5GB/s Bandwidth 30 µs Write Latency 10 Years Life Expectancy (25% writes) Series-7 Flash Controller 50-56

51 RamSan-720 Overview 6 or 12 TB Usable capacity (~ 7.8 or ~15.6 TB Raw) 500,000 IOPS (4K) 5 GB/s throughput <100µs latency No Single Point of Failure (nspof) Hot Swappable Flash Cards Highest density SLC Flash SSD system available in a 1U Series-7 Flash Controller Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports High Enterprise reliability Single-Level-Cell (SLC) Flash Variable Stripe RAID (VSR) 2D Flash RAID 51-56

52 RamSan-820 Overview 12 or 24TB Usable capacity (~ 15.6 or ~31.2 TB Raw) 450,000 IOPS (4K) 5 GB/s throughput <100µs latency No Single Point of Failure (nspof) Hot Swappable Flash Cards Highest density emlc Flash SSD system available in a 1U Series-7 Flash Controller Four 8 Gbit Fibre Channel ports or QDR InfiniBand ports High Enterprise reliability enterprise Multi-Level-Cell (emlc) Flash Variable Stripe RAID (VSR) 2D Flash RAID 52-56

53 RamSan-Green IT 53-56

54 Bandwidth Latency Speed & IOPS H A H N S T Ä T TE N M Ü N C H E N CONFIDENTIAL COPYRIGHT BY PSP

55 World s Fastest Storage Since 1978 Pure SSD-Racepower H A H N S T Ä T TE N M Ü N C H E N CONFIDENTIAL COPYRIGHT BY PSP

56 Thanks for your attention 56-56

Flash In the Data Center

Flash In the Data Center Enterprise-grade Morgan Littlewood: VP Marketing and BD Violin Memory, Inc. Email: littlewo@violin-memory.com Mobile: +1.650.714.7694 7/12/2009 1 Flash in the Data Center Nothing