Choosing Storage for MySQL. Peter Zaitsev CEO, Percona Inc Percona Live, Washington,DC 11 January 2012

Transcription

1 Choosing Storage for MySQL Peter Zaitsev CEO, Percona Inc Percona Live, Washington,DC 11 January 2012

2 Storage for MySQL Storage vs Memory Aspects of Choosing Storage for MySQL Directly Attaches Storage SAN NAS Cloud Storage SSD Hybrid Storage Storage configuration from OS Side

3 Storage vs Memory The goal is not Fast Storage The goal is great Performance Which can be achieved with Fast Storage large amount of memory Memory beats even fastest SSD Partially because database can access rows in memory while it has to read/write whole pages to the disk 50 us Per IO is great latency though it is just IO per second while you may access millions of short rows at same time

4 Memory is not Persistent Writes need to happen to permanent storage Especially problem for write heavy applications Memory Contents is lost on Crash/Restart Performance with empty cache and full cache can differ 100x or more So system is effectively down until it is warmed up Need to Plan for Warmup

5 Can't Beat Memory Latency Some worloads need data in memory Alternative can be expensive distributed application Imagine you need response time of 1 sec for query which traverses 100K rows to deliver results. Storage has 1ms latency (as example) With 99% hit rate we will still need 1000 IOPS to storage Which will use all our time budget already Even 99% hit ratio might not be enough

6 So how much memory I need? How much memory is needed for 99% hit ratio? It is workload related So much there is no useful Ratio I can say When you hear about recommendation to maintain 1:5 memory to data ratio it relates to certain class of application only If you're serious to understand it you need to experiment/benchmark with your workload and data Make sure not to focus on average case Being able to handle outliers well often will be limiting factor

7 What is Cost Efficient? It depends on workload a lot Real Case: Increasing memory from 16G to 64G increased capacity 6x It would cost a lot more to get storage system allowing to deliver 6x IOPS In other cases we find investing in storage is better than memory

8 Example Memory Fit Graph

9 Another Graph

10 Graph for Different Storage

11 Aspects of Choosign Storage Performance Reliability Extra Features

12 Performance Response time Always Important Throughput Important for high concurrency Apps Performance for Reads and Writes Ability to buffer-merge writes is very important Random IO Most typical concern for database applications Sequential IO Important for Analytics and Maintenance

13 Reliability Loosing Previously stored Data Think hard drive failure Loosing data for acknowledged write Writeback cache without BBU Longivity Especially concern for some of SSDs

14 Extra Features Mostly for SAN/NAS storage Or provided by Operating System Encryption Backups Replication Snapshots Ability to Rollback to previous state of data

15 Directly Attached Storage Internal Storage Built in Server Case. Up to 24 hard drives for some models Directly Attached as drive shelf Controller is in Server, drive shelf is case extension Conventional RAID typically used RAID10, RAID5, RAID6, RAID1 Safe WriteBack cache is typically available Very improtant for Durable Performance Great Latency for Conventional Storage The most typical Deployment for MySQL

16 RAID Levels RAID10 Good Default RAID Level for Database RAID5 Good for data having mostly Reads. Non critical recovery For example when you will switch to slave if hard drive failed in master RAID6 Even slower than RAID5 for write but handles 2 disks failed RAID1 Good Entry Level. Operating System, Log disk etc.

17 Stripe Sizes For Random IO large stripe size is best For Sequential IO medium sizes are good Partition,File system alignment very important (especially for smaller sizes) 256K (per drive) is good Beware of Controller Cache Sometimes it may try to cache whole stripes

18 RAID Caching You want RAID with Write Back Cache Write Back Cache preserved on Power Loss BBU Capacitor + Flash Based BBU May need learning every few months Getting array to go in Write Through cache mode Monitor RAID Cache State Read Caching on RAID side is not needed Read-Ahead on RAID typically not needed

19 MySQL and SAN Typically used NOT for Performance reasons Extra features; Default setup in organization; manageability Fiber Channel SAN and Ethernet (iscsi) SAN Complex Piece of Hardware A lot of Downtime is caused by bad SAN maintenance SAN can fail (though it really tries not to) SAN is Expensive for MB or IOP Do not relay on SAN caches Limited Transparency Sharing Resources adds complication to Performance Troubleshooting

20 MySQL and NAS A lot of Hardware can be accessed as SAN or NAS NAS is typically with NFS Have been improving over years Many Success stories running MySQL on NFS Yet Many Client and Server variations So test catefully Watch for file meta data access operations might require network IO High Performance and Reliable network is needed Same concerns in Transparency and Resource Sharing as NAS

21 Cloud Storage Different Cloud Solutions offer different storage Typically some form of very large scale SAN/NAS Amazon EBS is the most common example Performance is rather slow and unpredictable A lot of work on getting better storage in cloud Flash is likely to be common in the Cloud Soon

22 Flash Storage SSD PCI-E External Flash Storage

23 Flash Technology Most SSD storage uses NAND technology SLC Many writes per cell (up to 100K). Faster. Expensive MLC Fewer writes per cell (hundreds to 10K). Slower. Inexpensive emlc MLC targeted for Enterprise Application Flash Is Tricky Can't Overwrite. Erase Cycle is needed (slow, large blocks) Wear Leveling needed to deal with limited number of updates

24 Space and TRIM Flash often handles writes faster if not completely used Not using 100% of space might help Performance Support for TRIM operation (marking blocks free when they are not used any more) is important Yet make sure device supports it well too

25 SSD Space Performance

26 SSD Consumer Grade (Laptops) and Server Grade Performance and amount of writes differences Typically 2.5 SATA interface Make sure RAID controllers support SSD Older controllers had a lot of problems handling SSD Common Choice for entry level flash deployments Many Vendors on the market Intel, OCZ, Unigen, Samsung etc Unlike Hard Drives SSDs are really different

27 PCI-E Card Inserted in PCI-E Slot Typically uses host CPU and Memory for operation Both SLC and MLC models available Up to 2TB in size per card (MLC) Advanced Internal Data Protection Typically no need for additional RAID A lot higher performance compared to SSD FusionIO, Virident most well known vendors

28 External Flash Storage Storage available in external box Can be connected as SAN or External PCI-E Application of high IO rate and data sizes which internal cards can't provide Violin Memory, Nimbus Data are common providers

29 Hybrid Storage What if storing everything in Flash does not make sense for me? Applications with lots of data and only some of it frequently accessed Data for current year accessed frequently and for former years very rarely Basic Solutions: Store Flash and non Flash data in Different Systems Store Flash and non Flash data in different tables (partitions) which are stored on different devices.

30 Real Hybrid Storage SAN can use Flash as cache Hybrid Hard Drives available on market RAID controllers being able to use SSD as Cache FusionIO Flash Caching Solution Flash Cache support by Filesystem (ZFS) Flash Cache support on Linux Device Mapper level FlashCache project by Facebook.

31 Difficulties with Flash Cache Non uniform access times which can be hard to understand Some technologies are complicated to use Long cache Warmup might be needed

32 OS Configuration Volume Management Alignment File System Layout

33 Volume Management Use LVM on Linux if using simple storage Or other Volume management technology with other operating systems Snapshots are very helpful for backups, testing etc.

34 Alignment On Hardware RAID you want to make sure files are aligned to Stripes Or single Innodb page read may hit 2 stripes On SSD flash block size is often different from classical sector size (512 bytes) Both Partition should be aligned and file system created properly. Check out MPB Article:

35 Filesystem Issues Beware of Inode locking on EXT3 Reduced performance at high concurrency XFS is generally good choice on Linux EXT4 also showing good results Note dioread_nolock mount option Consider mounting MySQL data directory with noatime Enable TRIM support if using Flash Assuming Device is stable and supports it with good performance.

36 MySQL Layout Consider separate partition for MySQL data Low end use all available drives for single RAID volume Storing Innodb logs on separate device might make sense if many drives are available Though consider your Backup Strategy Binary logs on separate RAID volume increase data protection If having both conventional storage and SSD in the system consider placing binary log, double write buffer, logs on conventional storage (if has BBU)

37 Summary Think whenever Memory or Storage investment is optimal. Understand how MySQL uses storage. Consider Flash if you have not yet done so

38 Percona Live DC Sponsors

39 MySQL Conference & Expo 2012 Presented by Percona Live The Hyatt Regency Santa Clara & Santa Clara Convention Center April 10th-12th, 2012 Tutorials & Initial Breakout Sessions Announced Featured Speakers Mark Callaghan (Facebook), Jeremy Zawodny (Craigslist), Marten Mickos (Eucalyptus Systems) Sarah Novotny (Blue Gecko), Peter Zaitsev (Percona), Baron Schwartz (Percona) Learn More at /live/mysql-conference-2012/

40 Want More MySQL Training? Percona Training in Washington DC Next Week January 16th-19th, 2012 MySQL Workshops MySQL Developers Training - Monday MySQL DBA Training - Tuesday MySQL InnoDB / XtraDB - Wednesday MySQL Operations Thursday Use Discount code DCPL30 to save 30% Visit

41 Upcoming Percona MySQL Training Washington, DC January 16, 2012 Vancouver, Canada - February 6, 2012 Frankfurt, Germany - February 13, 2012 Irvine, CA February 14, 2012 Denver, CO - February 20, 2012 San Francisco, CA - March 12, 2012 Austin, TX - March 19, 2012 Visit

42 Peter Zaitsev We're Hiring! /about-us/careers/

43 /live