Moving to SANless for your Database Storage

Transcription

1 Moving to SANless for your Database Storage Back to DASD Tony Rogerson, SQL Server

2

3 Transition Back (SAN to DASD) Node1 Node2 Node1 Node2 Shared Storage DASD DASD Disk Replication Shared Storage Node2 DASD

4 Agenda Kit Connectivity Protocols Mechanical Drives Solid State Storage SANless SQL Server Semi-Random GUID V Incremental ID Comparison Clustered V Heap Bottleneck shift (CPU, Latching) TempDB Transaction Logs Scale out approach Availability Groups Hybrid Tiered Storage Approach Futures Massively Parallel Computing (GPGPU)

5 Getting to your device CONNECTIVITY PROTOCOLS

6 Connectivity Protocols Cable attached SAS and SATA 3Gb/s (300MiB/s) SAS and SATA 6Gb/s (600MiB/s) SATA (3/6Gbit/s) InfiniBand (GiBytes/s) Fibre Channel (GiBytes/s) External PCIe (GiBytes/s) USB 2.0, 3.0 (ok, MiBytes/s but latency shocking) Directly Attached PCIe (GiBytes/s)

7 Point to Point (SAS/SATA) SAS 600 / SATA 3 550MiBytes/sec SAS 600 / SATA 3 550MiBytes/sec SAS 600 / SATA 3 550MiBytes/sec PCIe SAS 600 / SATA 3 550MiBytes/sec Each channel has full throughput (not shared)

8 Port Expanders/Multipliers SAS 600 / SATA 3 550MiBytes/sec (Multiplexed) PCIe Devices share the bandwidth (same as Ultra SCSI multi-drop bus architecture). OK for HDD, not for SSD

9 Throughput Limiters 10 x SAS 600 / SATA 3 10 x 550MiBytes/sec SSD populated SAN 4Gbit/sec HBA SERVER Bottleneck is HBA, multiple HBA s required more cost! SSD easily run at 500MiB/sec on sequential and random loads. 4Gbit = approx 500MiB/sec 5.5GiBytes/s into 500MiB/s does not go

10 Rotating disk/spindle/hard disk whatever it has moving parts! MECHANICAL DRIVES

11 Mechanical Drives Stuck at 15Krpm for over a decade Density not improving (150GB per platter for enterprise 15Krpm, 1TB per platter for commodity 7,2Krpm) though helium filled coming Performance (latency and throughput) variance - outer edge better than inner. Poor level of IOps for a semi-random database workload. Ok for sequential. Power and heat high compared to Flash

12 Disk Throughput 100% Sequential 100% Random Strip Size: 64KiB; Windows OS allocation 8KiB; Transfer Size 8KiB 40GiB test file created on freshly formatted drive 146GB partition Many worker threads required to get throughput Drive is a HP EH0146FARWD 146GB K

13 Mechanics of a Hard Drive Multiple Platters per disk Platters double sided Two heads per platter Move together More sectors on outer than inner

14 Disk Geometry Effects 100% sequential read 64KiB transfer 3223IOps 0.30ms latency 1914IOps 0.52ms latency

15 Disk Geometry Visualisation Drop of 55,000 Kbytes per second from outer to inner edge Faster decline in performance as you approach the centre * HDDScan utility used

16 Ringing out the Performance (Mechanical drives) Increase platters [2 heads per platter], max 4 platters on K, 2 on K Hybrid add flash and keep hot data on the SSD Shortstroking RAID (striping a form of shortstroking!) Disk writes are smoothed through Write Cache (have plenty of it!)

17 Shortstroke the disk 3223IOps 0.30ms latency 1914IOps 0.52ms latency

18 Shortstroking Use only the sweet spot (outer edge) More sectors can be read in a single revolution Less head movement (jump tracks). 600GiB disk may only use 50GiB!

19 Effect of RAID MiBytes per second Latency (ms) # disks # disks More disks, less data per disk (implicit short stroking) Sequential workload saturates at 330MiB/sec regardless of number of disks port multiplexing? TEST TEST TEST your hardware before installing anything!

20 Master File Table Position FREE SPACE!!! Files will start to the right of the MFT so you end up with the sweet spot of the disk being free space 3.1GiB not used.

21 Like me when hung-over, it has no moving parts SOLID STATE STORAGE

22 Solid State Storage Game changer because of single device capability (IOps [random and sequential], Throughput, Latency, Power and heat. Commodity MLC, TLC Enterprise emlc, SLC, TLC, MLC Future? Density of flash is finite because of leakage between cells. SLC Single Level Cell: 1 bit per cell MLC Multi-Level Cell: 2 bits per cell TLC Triple Level Cell: 3 bits per cell

23 SSD Geometry (SQL Server Serial Plan) Consistent Throughput and Latency Query time 8mins 37secs Notes: Clustered Index on table; Table Scan to calculate MAX( CHECKSUM() ) on a char(7900) column;

24 SSD Geometry Effects (SQL Server Parallelism) Consistent Throughput and Latency Query time 3mins 7secs Notes: Clustered Index on table; Table Scan to calculate MAX( CHECKSUM() ) on a char(7900) column;

25 Disk Geometry Effects (SQL Server Parallelism) Latency increase gives Throughput decrease Query time 2hrs 50mins Notes: Clustered Index on table; Table Scan to calculate MAX( CHECKSUM() ) on a char(7900) column;

26 Disk Geometry Effects (SQL Server Serial Plan) Latency increase gives Throughput decrease Query time 2hrs 50mins Notes: Clustered Index on table; Table Scan to calculate MAX( CHECKSUM() ) on a char(7900) column;

27 NOLOCK effect When Nolock used it follows the page chain of the leaf level of the clustered index. Otherwise, it follows the B-Tree.

28 Performance P/E Cycle Cells cannot be overwritten Update = Erase then write Background processes perform erase Write goes to another cell (unlike a Hard Disk) TRIM works for the file system not internally for SQL Files which remain constant.

29 Reliability? Wear Levelling through Over Provisioning Smart algorithms Errors mostly occur on Write RAISE (redundant array independent silicon elements) Long endurance ratings (e.g. Intel 710) 1.8PB to 3PB for 300GiB drive

30 Form Factors SAS or SATA device 3.5 or 2.5 form factor Limited to protocol throughput (max 550MiB/sec) per channel Sizes up-to 2TB per drive PCIe Full or half height Limited to bus lane speed and capacity (16 lanes PCIe 2.0 gives approx 6.7GBytes/sec) Servers: each CPU socket has own PCI bus for performance more lanes, more throughput! Sizes up-to 10.24TB per card

31 Requirement: Cost Comparison I need to store 40GiBytes of data in a SQL Server table, the query I want to perform needs to return quickly, is highly random in terms of reading, anticipated requirement of 4,000 64KByte IOps at a read latency of less than 4 milliseconds per IO. Memory is limited to 8GiBytes.

32 6 x 15Krpm drives RAID 0 Cost Sequential Workload Single OCZ IBIS (PCIe) Random Workload Price per IO 304 * 6 / 4093 = 0.44 per IO Price per GB 304 * 6 / 40 = per GB Price per IO 442 / = per IO Price per GB 442 / 40 = per GB Requirement: Data is 40GiBytes, requirement is 4000 IOps for a single worker on a highly random query.

33 Full circle, but they re calling it Appliance SANLESS

34 Trend: [move Process TO data] not [move data TO process] Programs operate close to the data thus reducing latency and throughput bottlenecks Latency! Latency! Latency! (and throughput) Results PROCESS SAN (Network) DATA Results PCIe PROCESS DATA

35 What is a SAN? Storage Server connected via a Storage Area Network. Servers connected via the Storage Area Network using one or more HBA, multipathing for redundancy etc.. Storage Server OS is specifically tailored to the job of Storage.

36 Why SANs? Consolidate storage in one place solves the proliferation of DASD hard disks {jobs for the boys proprietary} Management easier provisioning of extra storage {nothing wrong with that!} Performance benefits IOps and throughput {now cheaper alternatives + I question the cost to get good IOps with low latency} SQL Server Clustering and the write Cache problem

37 Appliances? The big virtual appliance the cloud doesn t use them! Appliances all based on DASD PDW/Fast Track based around SAN but device dedicated to a single server node approach. Two prongs: Server based appliances with DASD replace SAN SAN based appliances (with SQL Server running inside) replace servers?? Does that not become DASD?

38 HA/DR? No more central point of failure the SAN Scale out multiple copies of the data kept in sync. SQL Server Availability Groups CITRIX NetScaler DataStream Other Scale out products HADOOP Cassandra VoltDB Gizzzzziiiiillliannnnn more.

39 Shared Storage or Scale out DASD Node1 Node2 Node1 Node2 Shared Storage DASD DASD Node2 Disk Replication Shared Storage DASD

40 Some considerations/thoughts SQL SERVER

41 SQL Server yes that old war GUID V INCREMENTAL-ID

42 The GUID war SSD s are fantastic at high IOp random IO at a low latency Inserts into HEAPS are blisteringly quick Do we care about fragmentation? For IO (sort of): Fragmentation = more IO because data isn t contiguous More memory usage More latching No right answer depends what you are doing!

43 SQL Server another religion to rethink COMPARISON CLUSTERED V HEAP

44 Data Benefits of clustering Leaf level of the index is the data Data is grouped in terms of locality by the clustering key (subject to extent fragmentation) No RID look up required when using clustering key

45 Data Benefits of Heap No B-tree maintenance causing page splits on un-ordered inserting Non-Clustered indexes narrow and smaller in size Insert speed rocks

46 Sample Lookup Data 50K rows randomly picked (using ORDER BY NEWID()) Fairly even pick of rows across the big table of 63,419,340 rows

47 Comparison DEMO

48 SQL Server BOTTLENECK SHIFT (CPU, LATCHING)

49 New bottleneck Driving the CPU harder data arrives earlier Exposes weakness in the application hidden by slow IO Instead of wait for IO now more latching in memory Helps reduce deadlocks because transactions are faster

50 SQL Server TEMPDB

51 TEMPDB on SSD? TEMPDB usage Version Store Work Tables for Group By, Order By etc.. User temporary tables and table variables How big is your tempdb requirement? Doable on SSD but provision based on expected throughput rate and expected ware rate

52 SQL Server TRANSACTION LOGS

53 Any point? Sequential writes to the outer portion of a disk is good already 12K IOP s at 0.08ms latency 96MiB/sec What s your transaction throughput rate? Can you really keep the transaction log file on the outer edge of the disk and the head in the correct track? Anyway Write cache should soak that up and smooth out

54 Reducing Transaction Log Perf on Mechanical Drives Single RAID array stick all the transaction logs from all the databases on it Disk head movement! Adding Latency Balanced by write cache? TEST TEST TEST

55 SQL Server SCALE OUT APPROACH (AVAILABILITY GROUPS)

56 Scale out HA/DR approach SERV_A DASD SERV_C Disaster Recovery DASD SERV_D DASD SERV_B DASD Cluster (Availability Group Listener) Readers (ReadOnly intent) (via Listener) Writers (via Listener) Readers

57 SQL Server HYBRID TIERED STORAGE

58 Scale Up (single box) Tier 0 File Group Tier 1 File Group Tier 2 File Group Tier 3 File Group PCIe Enterprise PCIe Commodity SAS Drives SATA Drives Table Placement Partitioning of current to historical data across file groups

59 From the few to the many GPGPU (MASSIVELY PARALLEL COMPUTING)

60 GPGPU v CPU comparison 192 logical proc s (GPGPU) v 8 (CPU) Host to Device (GPU) GiBytes/s depending on lanes + PCIe 2.0 or 3.0 Internal GPU memory speed astonishing

61 Summary Mechanical drives do they still have a place depends what you are doing High IOps use PCIe based cards Scale Out rather than Up Be fooled no longer by marketing BS Be aware what protocol version you are using PCIe 2.0 v 3.0, SATA 3 v 6, SAS 3 v 6. Finally: cut away all those components and go scale-out, design from the offset.

62 Remember: It s lunchtime, its sunny and you are now being held up. Tony Rogerson, SQL Server tonyrogerson@torver.net

63

64