Analyzing Performance in the Cloud

Transcription

1 Analyzing Performance in the Cloud solving an elastic problem with a scientific approach Nicholas Wakou (Dell EMC), Alex Krzos (Red Hat) Thursday, October 27, 2016 Barcelona Openstack Summit 2016

2 Presenters Nicholas Wakou is a Principal Performance Engineer with the Dell EMC Open Source Solutions Nicholas_Wakou@dell.com Alex Krzos is a Senior Performance Engineer at Red Hat working on Openstack akrzos@redhat.com lyzing-performance-in-the-cloud-solving-an-elastic-problem-with-a-scientific-approach

3 Agenda CLOUD DEFINITION & CHARACTERISTICS PERFORMANCE MEASURING TOOLS SPEC CLOUD IaaS 2016 BENCHMARK PERFORMANCE MONITORING TOOLS PERFORMANCE CHARACTERIZATION TUNING TIPS

4 CLOUD DEFINITION & CHARACTERISTICS

5 DEFINING A CLOUD NIST SPECIAL PUBLICATION Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

6 CLOUD CHARACTERISTICS

7 PERFORMANCE MEASURING TOOLS

8 RALLY OpenStack Benchmarking Tool as-an-app and as-a-service Verification Benchmarking Profiling Reports SLAs for Benchmarks Many plugins Source: What is Rally?,

9 PERFKIT BENCHMARKER OpenSource Living Benchmarking Framework containing a set of Benchmarks used to compare cloud offerings/environments 10+ Cloud Providers/Environments 34+ Benchmarks Large Community Involvement Capture Cloud Elasticity with Benchmark Results Use Cloud/Environment CLI Tooling Publish Results to BigQuery for Comparison Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

10 PERFKIT EXPLORER Dashboarding and Performance Analysis Tool for PerfKitBenchmarker Results Multiple Chart Options Uses BigQuery as backend data-store Hosted in Google App Engine Source:

11 CLOUDBENCH Framework that automates cloud-scale evaluation and benchmarking Benchmark Harness Requests the Cloud Manager to create an instance(s) Submit configuration plan and steps to the Cloud Manager on how the test will be performed At the end of the test, collect and log applicable performance data and logs Destroy Instances no longer needed for the test.

12 HARNESS AND WORKLOAD CONTROL Benchmark Harness Benchmark Harness. It comprises of Cloud Bench (CBTOOL) and baseline/elasticity drivers, and report generators. For white-box clouds the benchmark harness is outside the SUT. For black-box clouds, it can be in the same location or campus. Cloud SUT Group of boxes represents an application instance

13 BROWBEAT Orchestration tool for existing OpenStack Workloads Combines Workloads, Metrics, and Results into single tool Runs Performance Workloads: Rally - Control Plane Rally Plugins & Rally+pBench Plugins - Control+Data Plane Shaker - Network Data Plane PerfKitBenchmarker - Data Plane + Cloud Elasticity Provides Performance Infrastructure Installation and Configuration for Carbon/Graphite/Grafana Collectd ELK FluentD Provides dashboards for Visualizing and Comparing Results and System Performance Metrics

14 BROWBEAT - RESULTS

15 BROWBEAT - Metrics

16 SPEC CLOUD IAAS 2016 BENCHMARK

17 SPEC CLOUD IAAS 2016 BENCHMARK Measures performance of Infrastructure-as-a-Service (IaaS) Clouds Measures both control and data plane Control: management operations, e.g., Instance provisioning time Data: virtualization, network performance, runtime performance Uses workloads that resemble real customer applications benchmarks the cloud, not the application Produces metrics ( elasticity, scalability, provisioning time ) which allow comparison SPEC Cloud IaaS Benchmarking : Dell Leads the Way

18 Scalability and Elasticity Analogy Climbing a mountain inf init ely hig hm ou nta in IDEAL Scalability Mountain: Keep on climbing Cloud: keep on adding load without errors Elasticity Mountain: Each step takes identical time Cloud: performance within limits as load increases an Elasticity time for each step ng { on qu eri { c { bili ty { Sc ala { { { { c c 18

19 WHAT IS MEASURED? Measures the number of AIs that can be loaded onto a Cluster before SLA violations occur Measures the scalability and elasticity of the Cloud under Test (CuT) Not a measure of Instance density SPEC Cloud workloads can individually be used to stress the CuT: KMeans CPU/Memory YCSB - IO

20 SPEC CLOUD BENCHMARK PHASES Baseline Phase Determine the results for a single application instance of a workload AI = stream of 5 runs Elasticity Phase Determine cloud elasticity and scalability results when multiple workloads are run KMeans baseline AI YCSB baseline AI

21 BENCHMARK STOPPING CONDITIONS 20% AIs fail to provision 10% AIs have errors in any run Max number of AIs set by Cloud Provider 50% AIs have QoS violations KMeans completion time 3.33x Baseline phase YCSB Throughput Baselinethroughput / 3 YCSB Read Response Time 20 x BaselineReadResponse Time YCSB Insert Response Time 20 x BaselineInsertResponse Time

22 HIGH LEVEL REPORT SUMMARY

23 PUBLISHED RESULTS WEBSITE

24 PERFORMANCE MONITORING TOOLS

25 CEILOMETER Another familiar OpenStack project Goal is to efficiently collect, normalize and transform data produced by OpenStack services Interacts directly with the OpenStack services through defined interfaces Applications can leverage Ceilometer to gather OpenStack performance data Source:

26 COLLECTD/GRAPHITE/GRAFANA Collectd Daemon to collect System Performance Statistics Plugins for CPU, Memory, Disk, Network, Process, Graphite/Carbon Carbon receives metrics, and flushes them to whisper database files Graphite is webapp frontend to Carbon Grafana Visualize metrics from multiple backends.

27 GANGLIA Ganglia is a scalable, distributed monitoring system for high-performance computing systems such as Server Nodes, Clusters and Grids. - Relatively easy to setup - Tracks a lot hardware-centric metrics - Low operational burden

28 PERFORMANCE CHARACTERIZATION

29 PROVISIONING TIME: SPEC CLOUD The time needed to bring up a new instance, or add more resources (like CPU or storage) to an existing instance Instance: Time FROM request to create a new instance TO time when the instance responds to a netcat probe on port 22. Application instance: Time FROM request to create a new instance TO time when the AI reports readiness to accept client requests. Provisioning Time Characterization using Baseline phase Increase number of VMs (vary YCSB seeds KMeans and/or Hadoop slaves) and note impact on provisioning time. vary instance configuration (flavor)

30 IO LIMITS PCI-E Limits For PCI-E Gen-3 capable slots. SAS Limit An LSI whitepaper, Switched SAS: Sharable, Scalable SAS Infrastructure pdf

31 NETWORK/IO CHARACTERIZATION SPEC Cloud YCSB Baseline tests Throughput (ops/s) Vary number of Seeds Increase number of YCSB records and operations Increase number of YCSB threads CloudBench fio CloudBench Netperf Understand network utilization under load Management networks Data networks (Neutron tenant) Monitor with Ganglia, collectd, Linux tools (vmstat, iostat etc)

32 CPU CHARACTERIZATION Understand CPU utilization under load Monitor with Ganglia, collectd, graphana Linux tools (top, vmstat), SPEC Cloud, Kmeans Note: CPU user time CPU system time CPU iowait time CPU irq time Use SPEC Cloud Baseline tests for CPU Characterization Vary number of hadoopslaves Increase sample size, number of dimensions, number of clusters

33 SCALABILITY/ELASTICITY Understand Scalability/Elasticity of the CuT SPEC Cloud Elasticity phase Vary number of AIs Monitor with FDR html report

34 TUNING TIPS

35 HARDWARE/OS TUNING Latest BIOS and Firmware revs Appropriate BIOS settings RAID/JBOD Disk controller NIC driver- Interrupt coalescing and affinitization NIC bonding NIC jumbo frames OS configuration settings

36 CLOUD TUNING HW/OS Tuning Cloud Configs/Settings Workload tuning

37 INSTANCE CONFIGURATION Performance is impacted by Instance type (flavor) Number of Instances

38 OVER-SUBSCRIPTION Beware of over-subscription!!!

39 LOCAL STORAGE Use of local storage instead of shared storage (like Ceph) could improve performance by over 50%... depending on Ceph replication. Source: OpenStack: Install and configure a storage node - OpenStackkilo. (2015)

40 NUMA NODES Pinning instance CPU to physical CPUs (NUMA nodes) on local storage further improves performance. Source: Red Hat: Cpu pinning and numa topology awareness in OpenStackcompute. (2015)

41 DISK PINNING Disk Pinning shows a 15% performance improvement Source: OpenStack: OpenStack cinder multibackend. (2015)

42 WORKER COUNT CONFIGURATION Allow Services to use available resources with higher concurrency Examples: Keystone Process Count Neutron Workers Glance Workers Gnocchi API Workers

43 UNEVEN CONTROLLER USAGE One controller had more cores available than the other two and ended up with all the jobs. This scenario was identified easily because the correct dashboarding was in place.

44 HEAT MEMORY USAGE About 1GB of memory used by Heat for every 10 compute nodes deployed. Size your controller memory appropriately.

45 DEPLOYMENT TIMINGS OSPD 9 Overcloud Deployment

46 CONCLUSION

47 CONCLUSION Define what you are trying to measure Define a cloud Define what metrics are important Use the correct tools Rally PerfKitBenchmarker Cloudbench SPEC Cloud IaaS 2016 Benchmark Ceilometer Collectd/Graphite/Grafana Ganglia Browbeat Gather and analyze data Apply tuning tips based on the data PARTICIPATE!

48 ADDITIONAL INFORMATION Guidelines and Considerations for Performance and Scaling your Red Hat Enterprise Linux OpenStack Platform 6 Cloud Guidelines and Considerations for Performance and Scaling your Red Hat Enterprise Linux OpenStack Platform 7 Cloud Red Hat OpenStack Blog Red Hat Developer Blog Red Hat Enterprise Linux Blog

49

50 Rally Source:

51 Rally Rally is a familiar OpenStack project An automated benchmark tool for OpenStack Benchmarking Multiple use cases Development and QA DevOps CI/CD

52 PERFKIT BENCHMARKER Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

53 PERFKIT BENCHMARKER Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

54 PERFKIT BENCHMARKER Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

55 PERFKIT BENCHMARKER Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

56 BROWBEAT Scale and Performance automation Ansible playbooks for automation Provides automation wrapper around existing tooling Rally - Control plane tests Shaker - Data plane network tests Perfkit - Data plane tests Leverages existing upstream test frameworks rather than replacing them Performance Monitoring Collectd/Graphite/Grafana Results Capture/Storage/Analytics ELK stack Allows for results comparison

57 BROWBEAT

58 BROWBEAT - RESULTS

59 COLLECTD/GRAPHITE/GRAFANA Example Grafana dashboards

60 UNEVEN CONTROLLER USAGE About 1GB of memory used by Heat for every 10 compute nodes deployed. Size your controller memory appropriately.

61 HEAT MEMORY USAGE About 1GB of memory used by Heat for every 10 compute nodes deployed. Size your controller memory appropriately.

62 DEPLOYMENT TIMINGS Saw many instance reschedules with default scheduler. Deployment time dropped dramatically by setting up assignments via ironic.

63 DEFINING A CLOUD Cloud = Private Cloud = OpenStack Cloud = Rain Cloud = Cumulus Cloud = Public Cloud = Funnel Cloud = OpenShift Cloud = Community Cloud = Cirrus Ten different people will probably give you ten different answers

64 DEFINING A CLOUD NIST SPECIAL PUBLICATION Private cloud The cloud infrastructure is provisioned for exclusive use by a single organization comprising multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises. Public cloud The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider. Hybrid cloud The cloud infrastructure is a composition of two or more distinct cloud infrastructures (private, community, or public) that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).

65 DEFINING A CLOUD SPEC OSG Cloud Subcommittee Glossary Blackbox Cloud A cloud-provider provides a general specification of the SUT, usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider. Whitebox Cloud The SUT s exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds. Source:

66 DEFINING A CLOUD The focus of this presentation will be predominantly on white box private cloud environments Primary example is OpenStack Many of the tools and methodologies are usable in the other cloud environments as well

67 CLOUD CHARACTERISTICS

68 CLOUD CHARACTERISTICS SPEC RESEARCH GROUP - CLOUD WORKING GROUP group.html READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS rg_cloud/endorsed_publications/spec-rg _cloudmetrics.pdf

69 ELASTICITY THE DEGREE TO WHICH A SYSTEM IS ABLE TO ADAPT TO WORKLOAD CHANGES BY PROVISIONING AND DEPROVISIONING RESOURCES IN AN AUTONOMIC MANNER, SUCH THAT AT EACH POINT IN TIME THE AVAILABLE RESOURCES MATCH THE CURRENT DEMAND AS CLOSELY AS POSSIBLE Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud Working Group

70 ELASTICITY THE DEGREE TO WHICH A SYSTEM IS ABLE TO ADAPT TO WORKLOAD CHANGES BY PROVISIONING AND DEPROVISIONING RESOURCES IN AN AUTONOMIC MANNER, SUCH THAT AT EACH POINT IN TIME THE AVAILABLE RESOURCES MATCH THE CURRENT DEMAND AS CLOSELY AS POSSIBLE Source: READY FOR RAIN? A VIEW FROM SPEC RESEARCH ON THE FUTURE OF CLOUD METRICS, SPEC RG Cloud Working Group

71 ELASTICITY

72 ELASTICITY Source: HOW FAR WILL HE STRETCH? WILL HE BREAK WHEN STRETCHED? AS YOU STRETCH HIM DOES IT GET HARDER TO STRETCH HIM MORE? WHEN I LET GO DOES HE RETURN TO HIS ORIGINAL SHAPE? HOW LONG DOES HE TAKE TO RETURN TO HIS NORMAL SHAPE?

73 RESULTS COMPARED # Submissions Submission ID Cloud Type 2 Dell_12g Private / White box Hardware Platform 12g, 7xR720 Compute nodes Job Date 03/05/2016 Metrics Scalability@AIs 10.3@10 Scalability per AI 1.03 Elasticity 63.0% Inst. Prov. Time 163 (s) AI Prov. Success 100% AI Run Success 100% Total Instances 65 Baseline Phase YCSB Throughput 13,082.6 KMeans Job time (s) Elasticity Phase YCSB Throughput 9,480.9 KMeans Job time (s) Dell_13g Comment Private / White box 13g, 9xR630 Compute nodes 06/08/ @ % 135 Higher is better Higher is better Higher is better Lower is better 86.96% 100% 131 Higher is better Higher is better Higher is better 17, Higher is better Lower is better 14, Higher is better Lower is better

74 PERFKIT BENCHMARKER Source: Introduction to Perfkit Benchmark and How to Extend it, Tech-Talks

75 BENCHMARK HARNESS

76 SURPORTED WORKLOADS

77 BROWBEAT

78 BROWBEAT REPEATABLE AUTOMATED TESTING

79 PROVISIONING TIME: RALLY Automated VM provisioning Nova Success rate Source: measuring the Cloud Using Rally & CloudBench, Douglas Shakshober, Red Hat Inc. 79

80 IO LIMITS PCI-E Limits For PCI-E Gen-3 capable slots. ( Gen-3 is defined at 8 Gbps; this gives a bandwidth of 8.0 Gb/s (Scrambling +128b/130b encoding instead of 8b/10b encoding) per lane, so for example a PCI-E Gen-3 x8 link delivers an aggregate bandwidth of 8 GB/s SAS Limit An LSI whitepaper, Switched SAS: Sharable, Scalable SAS Infrastructure ( shows how to calculate the SAS limit of an 8 lane controller port with a SAS bandwidth of 6Gbps: Vary number of Seeds 6Gb/s x 8 lanes = 48Gb/s per x8 port 48Gb/s (8b/10b encoding) = 4.8GB/sec per port (per node) 4.8GB/s per port x 88.33% (arbitration delays and additional framing) = 4320MB/s per port

81 CEILOMETER: High-level Architecture Source:

82 CEILOMETER Another familiar OpenStack project Goal is to efficiently collect, normalize and transform data produced by OpenStack services Interacts directly with the OpenStack services through defined interfaces Many tools utilize Ceilometer to gather OpenStack performance data

83 COLLECTD/GRAPHITE/GRAFANA Collectd Daemon to collect System Performance Statistics CPU, Memory, Disk, Network, Process, MariaDB, Load, Logged errors and more Graphite/Carbon Carbon receives metrics, and flushes them to whisper database files Graphite is webapp frontend to carbon Grafana Visualize metrics from multiple backends.

84 SPEC CLOUD WORKLOADS YCSB Framework used by a common set of workloads for evaluating performance of different key-value and cloud serving stores. KMeans -Hadoop-based CPU intensive workload -Chose Intel HiBench implementation