Cisco HyperFlex Design, Performance and Benchmarking for All-Flash and NVMe Clusters. Silvo Lipovše, sistemski inženjer

Transcription

1

2 Cisco HyperFlex Design, Performance and Benchmarking for All-Flash and Ne Clusters Silvo Lipovše, sistemski inženjer

3 Agenda What s New All-Flash System Design Benchmarking Tools Performance Testing Conclusion March 2018 Cisco Connect Rovinj, Croatia 3

4 What s New in HyperFlex 4

5 What s New With Cisco HyperFlex 2.6? M5 Generation Server Models All-Flash Now Supports Ne Caching Disks Native Snapshot Based Replication Cisco Intersight Cloud- Based Management Mixed Clusters of M4 and M5 Servers, Higher Capacities and Larger Scale Support for Self Encrypting Disks (SEDs) Expanded Compute-Only Node Support HyperFlex Connect Native HTML5 GUI with Role Based Access Control March 2018 Cisco Connect Rovinj, Croatia 5

6 New Disk Options Non-Volatile Memory Express (Ne) Replaces SAS write caching disk More efficient streamlined protocol Uses PCI-Express instead of legacy serial bus Multi-queue, more commands per queue, high parallelism Higher performance, lower latency Self Encrypting Disk (SED) Used for write caching disk and capacity disks Disk hardware encrypts all data written to the device Can use UCSM derived local encryption keys, or keys from a centralized KMS via KMIP Secure erase, re-key March 2018 Cisco Connect Rovinj, Croatia 6

7 Cisco HyperFlex All Flash Configurations HXAF220c Cluster HXAF240c Cluster Expanded Clusters + Standard UCS 2 & 4 Socket Servers Smallest Footprint 3 16 Node Cluster Per-Node Caching Device: 1 x 800 GB SSD, 400 GB Ne, or 800 GB SED Capacity Disks: 6 x 960 GB, 3.8 TB SSD, or 800 GB SED Boot/Housekeeping: M.2 SSD + 120GB or 240GB SSD Capacity-Optimized 3 16 Node Cluster Per-Node Caching Device: 1 x 800 GB SSD, 400 GB Ne, or 800 GB SED Capacity Disks: up to 23 x 960 GB, 3.8 TB SSD, or 800 GB SED Boot/Housekeeping: M.2 SSD + 120GB or 240GB SSD Compute-heavy Hybrid (Compute Bound Apps/VDI) 3-16 Node HXAF220 or HXAF240 Cluster Up to 16 Compute Nodes Blade or Rack B200 M3/M4, B260 M4, B420 M4, B460 M4, C220 M3/M4, C240 M3/M4, and C460 M4 Local Disk, SD Card or SAN Boot March 2018 Cisco Connect Rovinj, Croatia 7

8 HyperFlex Native Data Protection and Security Protect against Site Failure Data-At-Rest Loss Prevention Secure Platform Regulatory Compliance Native Replication of Snapshots across Sites Data-At-Rest Encryption thru Self-Encrypting Drives Vulnerability Assessments and Hardening Certifications starting with FIPS, Common Criteria Disaster Recovery Protection Enterprise Key Management Support Role-based access control and Auditing Compliance for Data Privacy (GDPR, HIPAA, PCI-DSS etc.) March 2018 Cisco Connect Rovinj, Croatia 8

9 Cisco Intersight Cloud-Based Management March 2018 Cisco Connect Rovinj, Croatia 9

10 Native HyperFlex Connect HTML5 GUI March 2018 Cisco Connect Rovinj, Croatia 10

11 All-Flash System Design 11

12 Building on the Right Foundation Cisco HX Data Platform Unique Architecture Distributed File system Local Disks Local Disks Local Disks Local Disks Built From the Ground Up for Hyperconvergence Distributed Log- Structured File System Designed for Scale-out, Distributed Storage Advanced Data Services (Snapshots, Clones, Replication) and Data Optimization (Inline Dedupe, Compression) Without Trade-offs Better Flash Endurance and Disk Performance Computing, Storage, Networking, and Hypervisor Integration No Reliance on Legacy Filesystems or Technology March 2018 Cisco Connect Rovinj, Croatia 12

13 Software Modules Inside a Server DATASTORE/VOLUME HYPERVISOR IO Visor VAAI sthyp CONTROLLER HDD HDD SDD SDD SSD SSD SSD SSD Controller Has Direct Control of Drives IO Visor Module Presents NFS to ESX and Stripes IO VAAI Plugin Offloads Snapshots and Clone Operations sthypervisorsvc Coordinates Native Recovery Snapshots and Replication March 2018 Cisco Connect Rovinj, Croatia 13

14 Optimizing for All Flash: Read Path L1 L2 Hybrid Read DRAM Read Cache SSD L1 (DRAM) Cache: Metadata and data Helps with small bursts L2 (SSD) Cache: No L2 (SSD) Cache: De-duplicated Read No flash Cache on flash advantage Use Use aggregate performance performance of of all all SSDs Read Cache SSDs All Flash Read L1 DRAM Read Cache Distributed Objects Log FS SC RAM: Unified capacity disk index cache Improvements by avoiding additional index reads Distributed Objects Log FS March 2018 Cisco Connect Rovinj, Croatia 14

15 Technical Improvements in HX 2.5+ Updated Linux kernel in the controller s Gains of 20-30% in read performance Optimized write cache destaging process More consistent performance with less variability Metadata Optimization Less space consumption, fewer updates required, faster index lookups Cleaner Improvements Lower performance impact, less variability Write Log Enhancements In-line compression with no latency penalty, makes effective write log size larger March 2018 Cisco Connect Rovinj, Croatia 15

16 Data Distribution and Non-Disruptive Operations HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER DATASTORE DATASTORE File.vmdk A B C D E A 1 B 2 C 2 D 3 E 1 B 1 A 2 C 3 E 3 D 1 C 1 A 3 D 2 E 2 B 3 D 1 E 1 B 3 Stripes blocks of virtual disk files across servers using a hashing algorithm Replicate one or two additional copies to other servers Handle entire server or disk failures Restore back to original number of copies Rebalance s and data post replacement Rolling one-click software upgrades March 2018 Cisco Connect Rovinj, Croatia 16

17 Life of a Write IO: Write Log ACK A CONTROLLER CONTROLLER CONTROLLER ESXi DS IOvisor ESXi DS IOvisor ESXi DS IOvisor Write Log Capacity Used Size: 8GB or 32GB Hybrid / 60GB AF A A A B B B C C C D D D 1. Application writes data (A) to OS 2. writes to DK on datastore 3. Write intercepted by IOvisor and sent to primary controller for that DK segment and offset 4. Primary data is compressed and committed to WL SSD 5. Simultaneously, data replicated to two other nodes (RF3) and committed to their WL SSD 6. Acknowledgement sent to 7. Data is successfully committed and protected March 2018 Cisco Connect Rovinj, Croatia 17

18 Life of a Write IO: Destage CONTROLLER CONTROLLER CONTROLLER ESXi DS IOvisor ESXi DS IOvisor ESXi DS IOvisor Write Log Capacity Used B A A A A A B B B C C C D D D C D C D C D 1. When the active write log of a node is full, the active log becomes the secondary log and destages 2. Primary data copies are deduplicated 3. Data is then three way written (RF3) to SSD / HDD on 3 separate nodes 4. After all data is successfully committed, data is purged from the secondary log on the caching SSD 5. In hybrid systems, recent writes are added to the read cache area of the SSD during destage March 2018 Cisco Connect Rovinj, Croatia 18

19 Life of a Read IO Read A CONTROLLER CONTROLLER CONTROLLER ESXi DS IOvisor ESXi DS IOvisor ESXi DS IOvisor Active WL Passive WL L1 SSD (AF) / HDD (Hybrid) L2 1. Application reads data from OS filesystem 2. reads from DK on datastore 3. Read intercepted by IOvisor and sent to primary controller for that DK segment and offset a) First check active write log b) Next check passive write log (the duplicated write log of the other nodes) c) Look in L1 (DRAM) cache d) Look in L2 (SSD) cache [Not in AF models] e) Retrieve from SSD / HDD copied to L2 [Not in AF models] and L1, decompress & returned to IOvisor 4. Return data to March 2018 Cisco Connect Rovinj, Croatia 19

20 Dynamic Data Distribution Systems Built on Conventional File Systems Write Locally, Then Replicate, Creating Performance Hotspots HX Data Platform stripes data across all nodes simultaneously, leveraging cache across all SSDs for fast writes, and HDDs for capacity, via the low-latency UCS internal network Balanced space utilization, no data migration required following a migration, failure tolerance, and no hotspots HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HX Data Platform March 2018 Cisco Connect Rovinj, Croatia 20

21 Independent Scaling of Compute and Capacity IOVisor IOVisor Scale Compute IOVisor IOVisor IOVisor IOVisor IOVisor IOVisor IOVisor IOVisor HYPERVISOR CONTROLLER HYPERVISOR CONTROLLER HYPERVISOR HX Data Platform HX Data Platform CONTROLLER HYPERVISOR CONTROLLER Non- HyperFlex Hosts Can Connect to Storage with IOVisor Scale Capacity Within Nodes Add Nodes March 2018 Cisco Connect Rovinj, Croatia 21

22 HyperFlex Capacity Considerations SSD reliability is significantly better than HDD; from 2-10 times lower annual replacement rate in studies by major online providers. HyperFlex installer no longer defaults to RF3 for an all-flash system, forcing the end user to make a decision. Consider RF2 for all-flash systems that do not require triple data copies or multiple component redundancy. RF2 shows 15-40% reduced write latencies in lab testing. RF2 can provide clusters up to 591 TiB maximum capacity. Keep overall cluster capacity consumed below 70% for best performance March 2018 Cisco Connect Rovinj, Croatia 22

23 Benchmarking Tools 23

24 Iometer Intel project now developed as an open source project No active development in 3+ years Primarily a Windows GUI tool, but ports exist for Linux and Mac OS X Easy configuration of an access profile Not a primary tool for our team any longer Issues: Not very easy to set up server/client multi node tests Difficult to script runs on Windows No configurable dedupable or compressible data patterns Default patterns are highly dedupable which can give false results March 2018 Cisco Connect Rovinj, Croatia 24

25 FIO Written by Jens Axobe, the current Linux kernel block layer maintainer Compiled binary install for most Linux distributions Configurations to test are written in text based job files Highly customizable FIO can be configured to run at startup, then a master machine can start tests remotely Issues: FIO must be installed on all testing machines If FIO is not already running, you cannot start a multi-node test Output can quickly get cluttered and hard to read March 2018 Cisco Connect Rovinj, Croatia 25

26 Vdbench Written by Henk Vandenbergh at Oracle Java application that can run on almost all platforms with a JRE Configurations to test are written in text based job files Highly customizable Easy server/client job execution via SSH or RSH Easy to read output in HTML files Has become the standard tool for the Cisco HX BU TME team Issues: Thread and J count can make calculating the OIO per disk difficult to calculate March 2018 Cisco Connect Rovinj, Croatia 26

27 Vdbench Output March 2018 Cisco Connect Rovinj, Croatia 27

28 Cisco HX Bench Cisco developed tool to automate many facets of benchmark testing Download at (CCO login required) Deploys a master test controller from an OVA file on your HX cluster HTML GUI makes deployment of s and test configuration easy Uses vdbench for the testing Use built in test scenarios or build your own Automatically deploys lightweight Ubuntu s as the test load generators Test results and charts are customizable and even easier to view than the default HTML output from vdbench March 2018 Cisco Connect Rovinj, Croatia 28

29 Cisco HX Bench User Interface March 2018 Cisco Connect Rovinj, Croatia 29

30 Others HCIBench ware automation wrapper that executes vdbench on Photon s Results are identical to vdbench, somewhat easier to use but not as flexible HammerDB Used internally for database workload qualifications SQLIO, SLOB, Jetstress, etc. Application specific tests not used for general performance characterization Mixed Workload Tools Examples: Weathervane, mark March 2018 Cisco Connect Rovinj, Croatia 30

31 Vdbench Example Job Files Vdbench dedupunit=8k dedupratio=1.25 compratio=1.4 hd=default,user=root,shell=ssh,jvms=1 hd=vdbench1,system= hd=vdbench2,system= hd=vdbench3,system= hd=vdbench4,system= sd=default,openflags=o_direct,threads=16 sd=sd1,host=vdbench*,lun=/dev/sdb wd=wd1,sd=sd*,xfersize=8k,rdpct=70,seekpct=100 rd=rd1,wd=wd1,iorate=max,elapsed=5m,interval= March 2018 Cisco Connect Rovinj, Croatia 33

32 Vdbench Example Job Execution Ensure each load generator has the vdbench java executable in the same location, and copy SSH keys so SSH logins happen without a password. Since the vdbench job file specifies the clients as well, just run the test: vdbench f 7030_4k.vdb o /var/www/html/ Easy to loop multiple runs in sequence: for i in {1..3} ; do vdbench f 7030_4k.vdb o /var/www/html/ && sleep 3600 ; done Simple scheduling of runs with Linux at: # at 1am tomorrow at> vdbench -f 7030_4k.vdb -o /var/www/html/output/7030_4k.tod at> <EOT> (Press CTRL+d to end the job creation) March 2018 Cisco Connect Rovinj, Croatia 34

33 Performance Test Results 35

34 HX Hybrid to All-Flash Performance Comparison Performance Multiple 7,0 6,0 5,0 4,0 3,0 2,0 1,0 0,0 Performance With Varying Working Set All Flash Performance Multiple HX240 hybrid read cache 1.2 TB per node 0.4GB 100GB 800GB 2TB Effective Working Set Size per Node Tests done on 4 x HX240 AF offers performance improvements and lower latency even at small working sets AF offers huge performance and latency improvements for working sets that do not fit in the hybrid model read cache March 2018 Cisco Connect Rovinj, Croatia 36

35 Base Configurations for Vdbench Tests HX All-Flash HX All-Flash + 40GbE HX All-Flash + 40GbE and Ne Version HX 2.0.1a HX 2.0.1a HX 2.5.1a HX 2.6.1b HX All-Flash M5 HW Configuration Benchmark Setup 8 x HXAF240 M4, each with: WL: 800G SSD Data: 10 x 960G VIC 1227 E5-2690v4 512 GB RAM 8 x HXAF240 M4, each with: WL: 800G SSD Data: 10 x 960G VIC 1387 E5-2690v4 512 GB RAM 8 x HXAF240 M4, each with: WL: 400G Ne Data: 10 x 960G VIC 1387 E5-2690v4 512 GB RAM Vdbench Linux s per host (64 total) Each has 1 x 200GB virtual disk, HX filesystem ~60% full (12.4 TB working set) Priming: 64K writes 100% sequential Tests run 3 times and averaged 8 x HXAF240 M5, each with: WL: 400G Ne Data: 10 x 960G VIC 1387 Xeon GB RAM March 2018 Cisco Connect Rovinj, Croatia 37

36 Benchmark 1 Performance Results Using Vdbench Tests 8K 100% Sequential Write Workload IOPs Maximum Sequential Write IOPs Latency , , ,87 7, GbE GbE 2.6 M5 20,00 18,00 16,00 14,00 12,00 10,00 8,00 6,00 4,00 2,00 0,00 Latency Target Write Latency < 10ms 16 OIO Per Disk 27% Latency Reduction and IOPs Gain with 40 GbE 6% Latency Reduction and IOPs Gain on Ne 8% Latency Reduction and IOPs Gain with HX 2.6 on M March 2018 Cisco Connect Rovinj, Croatia 38

37 Benchmark 2 Performance Results Using Vdbench Tests 8K 100% Random Read Workload IOPs Maximum Random Read IOPs Latency ,26 4,93 4,00 3, GbE GbE 2.6 M5 20,00 18,00 16,00 14,00 12,00 10,00 8,00 6,00 4,00 2,00 0,00 Latency Target Read Latency < 5ms 32 OIO Per Disk 7% Latency Reduction and IOPs Gain with 40 GbE 23% Latency Reduction and IOPs Gain on Ne 3% Latency Reduction and IOPs Gain with HX 2.6 on M March 2018 Cisco Connect Rovinj, Croatia 39

38 Benchmark 3 Performance Results Using Vdbench Tests 100% Random 70/30% R/W with Varying Block Sizes 4K 8K 16K 64K 256K IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency , , , , , GbE 152, , , , , GbE 161, , , , , M5 180, , , , , Target Read Latency < 5ms and Write Latency < 10ms for 4-16K blocks 8 OIO Per Disk ~6% Latency Reduction and IOPs Gain with 40 GbE at 4-8K, % Gains for Larger Blocks 10-22% Latency Reductions and IOPs Gains with HyperFlex 2.5 and Ne Caching Disks 3-11% Latency Reductions and IOPs Gains with HyperFlex 2.6 on M5 Generation Servers March 2018 Cisco Connect Rovinj, Croatia 40

39 Benchmark 4 Performance Results Using Vdbench Tests 8K 100% Random 70/30% R/W IOPs Curve 100% 20% 40% 60% 80% IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency , % , % , % , % , % GbE 151, % , % , % , % , % GbE 169, % , % , % , % , % M5 173, % , % , % , % , % 2.38 Curve test measures IOPs stability by calculating the standard deviation of IOPs as a percentage of the final average result, at various performance levels, starting with a 100% unthrottled test, then again at a percentage of that maximum Maximum speed, unthrottled tests will always have more variability due to more aggressive background tasks 20% through 60% values are typically very low because the system is not being pushed very hard 65% improvement in IOPs stability with 40 GbE at 80% of maximum HX 2.5 shows a further 78% improvement HX 2.5 also shows a 54% improvement of IOPs stability in the 100% unthrottled test HX 2.6 on M5 generation servers shows improvements in all tests, including 28% at 100%, and 45% at 80% March 2018 Cisco Connect Rovinj, Croatia 42

40 19-21 March 2018 Cisco Connect Rovinj, Croatia 43

41 Benchmark 5 Performance Results Using Vdbench Tests I/O Blender 100% Random 70/30% R/W IOPs Curve 100% 20% 40% 60% 80% IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency IOPs IOPs STDV Avg Latency , % , % , % , % , % GbE 114, % , % , % , % , % GbE 137, % , % , % , % , % M5 137, % , % , % , % , % 2.89 Test measures IOPs stability by calculating the standard deviation of IOPs as a percentage of the final average result, at various performance levels, starting with a 100% unthrottled test, then again at a percentage of that maximum Block Size Mix: 4K 40%, 8K 40%,16K 10%, 64K 10% 20% through 60% values are typically very low because the system is not being pushed very hard 30% improvement in IOPs stability with 40 GbE at 80% of maximum, HX 2.5 shows a further 84% improvement HX 2.5 also shows a 39% improvement of IOPs stability in the 100% unthrottled test HX 2.6 on M5 generation servers shows improvements in all tests, including 11% at 100%, and 20% at 80% March 2018 Cisco Connect Rovinj, Croatia 45

42 Per Test Results Variation IOPS in a 70/30 Workload 1 Hour Duration HX Vendor Y Intervals Intervals March 2018 Cisco Connect Rovinj, Croatia 48

43 HyperFlex Performance Summary HyperFlex shows large gains in performance by transitioning to 40 GbE and 3 rd Gen FIs and VICs HyperFlex GbE offers significant gains in IOPs, latency and stability for all test workloads HyperFlex 2.6 on M5 shows further improvements in IOPs, latency and more consistent performance HX All Flash, unlike competition, shows highly consistent performance across all s March 2018 Cisco Connect Rovinj, Croatia 49

44 50

45