A Resource Contention Analysis Framework for Diagnosis of Application Performance Anomalies in Consolidated Cloud Environments

Transcription

1 A Resource Contention Analysis Framework for Diagnosis of Application Performance Anomalies in Consolidated Cloud Environments Tatsuma Matsuki, Naoki Matsuoka Fujitsu Laboratories LTD. ICPE Copyright 206 FUJITSU LIMITED

2 Performance Problems in Cloud Physical resources in cloud are shared within VMs and applications. Anomalous workloads of applications create bottlenecks in cloud Application Application Application Application Application VM VM2 VM3 VM4 VM5 Use Server resource (e.g., Server CPU) Resource contention (Bottleneck) Cloud service Storage resource infrastructure (e.g., Storage devices) Network resource (e.g., Network Switch) Copyright 206 FUJITSU LIMITED

3 Performance Diagnosis Diagnose the performance anomalies with Anomalous response etc. Application Virtual CPU usage (%) Virtual NIC throughput etc. VM VM2 VM3 VM4 VM5 Virtual Disk IOPS etc. Server resources (e.g., Server CPU) PercentProcessorTime (%) Interrupts/sec etc. Storage resource (e.g., Storage devices) IO/sec Read,WriteBytes/sec CacheHitRate (%) etc. Network resource (e.g., Network Switch) Packets/sec DroppedPackets/sec QueueLength etc. Administrator 2 Copyright 206 FUJITSU LIMITED

4 How much does the diagnosis cost? Without any diagnosis tools All available N have to be investigated Large number of N Diagnosis cost: X = αn Administrator With a diagnosis tool Diagnosis tool can automatically select causal Large number of N Diagnosis tool Selected Diagnosis cost: X = 0? 3 Copyright 206 FUJITSU LIMITED

5 The Remaining Diagnosis Cost A diagnosis tool is not perfect! The diagnosis tool leaves some diagnosis costs for users. Large number of N False negatives Diagnosis tool Precision: p Recall: r Small number of N The Remaining Diagnosis Cost is estimated as: False positives X = αn p + α(n N )( r) Selected include false positives with probability, p Additional investigation is required with probability, r 4 Copyright 206 FUJITSU LIMITED

6 The Remaining Diagnosis Cost Transform the formula as follows X = αn p + α N N = X αn d d r + d p r N = dn, d Gain of the diagnosis tool Recall is more important than precision! When d, the gain is approaching αnr and p makes no contributions. 5 Copyright 206 FUJITSU LIMITED

7 Our Framework Approach Objective of our framework Select causal and VMs with better recall Large number of Our framework Causal Causal VMs Metrics labels have VM names Statistical correlation analysis Evaluate Pearson correlation between time-series Mining Association Rules in time-series data => reduce false negatives and achieve better recall! 6 Copyright 206 FUJITSU LIMITED

8 application performance Why the false negatives occur? Temporary correlated lead to false negatives Application performance (e.g., response time) A causal metric in cloud time unit Not correlated period Correlated period Statistical Pearson correlation is not sufficient! Correlation coefficient is in this case 7 Copyright 206 FUJITSU LIMITED

9 Temporary Correlated Metrics Why does the temporal correlation occur? Use VM2 Application Bottleneck on shared resource VM3 VM4 Temporary correlated Highly correlated Several causes When several causes exist, the temporary correlations occur! 8 Copyright 206 FUJITSU LIMITED

10 Association Rule Analysis Widely used in data-mining area Discovering relations between variables in databases Original Association Item set A Item set B A event with a item set A is likely to bring about a event with a item set B Our framework Association Metric Application performance A anomalous metric behavior is likely to bring about a anomalous application performance 9 Copyright 206 FUJITSU LIMITED

11 Association Rule in time-series application performance Calculate the confidence and support measures time unit confidence B A = support B A = A t B t 0 0 t A t B t t B t t A t B t t A t = 5 5 = = 5 0 = 0.5 Association rule A valid association rule exits? How strong the association? 0 Copyright 206 FUJITSU LIMITED

12 Association Rule Calculation app performance app performance Application performance discretizing Example: User-defined threshold time unit Example2: User-defined anomalous period time unit Copyright 206 FUJITSU LIMITED

13 Association Rule Calculation Metrics discretizing Investigate the optimal discretizing threshold for each metric 3.E+07 2.E+07 2.E+07.E+07 5.E+06 0.E+00 Largest support 0.5 within a constraint, confidence time unit t m(t) Candidates of the optimal threshold Optimal threshold = arg max support B m t m t A s. t. confidence B m t A 0.8 Discretized time series with threshold m(t) Given the discretized application performance Copyright 206 FUJITSU LIMITED

14 Our Framework Overview Applying the association rule mining Input Application performance Input All performance in cloud Input Discretized application performance Evaluate Pearson correlation Low correlated Highly correlated Evaluate association rules Discretize performance Metrics with a valid association rule Output Hypothesis testing of Pearson correlation with two-level significance level, 0. and Copyright 206 FUJITSU LIMITED

15 Experimental Evaluation Experimental setup 2 host servers, a physical switch Windows performance counters: 6,0 Applications: a typical web application, virtual desktop CPU, Disk, Network bottlenecks are injected Web access Network load Disk load VDI WEB, AP LB Load DB DB DB Load 3 VM 0 VM VM VM VM VM VM VM CPU load Host Windows Hyper-V Host 2 Windows Hyper-V External Physical switch 4 Copyright 206 FUJITSU LIMITED

16 precision, recall narrow down: d Experimental Results One of the main evaluation results precision: p recall: r narrow down: d correlation correlation + association rule significance level: Normalized remaining diagnosis cost correlation correlation + association rule Tradeoffs between precision and recall 2 Larger number of are selected by our framework Our framework reduce the expected cost Quantify the tradeoffs 2 We expected that our framework reduces the diagnosis cost by > 80% in this case. 5 Copyright 206 FUJITSU LIMITED

17 Read Bytes / sec app performance Read Bytes / sec app performance Read Bytes / sec app performance Testing on a Real Data Set Real data set from a Virtual Desktop Infrastructure (VDI) About 300 VMs are running on Windows Hyper-V Windows performance counters: about 35,000 Application performance is obtained from benchmark results Performance problems caused by storage resource bottlenecks. Examples of the selected causal Support threshold is set to 0.2 Our framework can capture these! VM application performance VM2 application performance VM3 application performance 2.E+06 2.E+06.E+06.E+06.E+06 8.E+05 6.E+05 4.E+05 2.E+05 0.E time unit E+07.E+07 8.E+06 6.E+06 4.E+06 2.E+06 0.E time unit E+06 3.E+06 2.E+06 2.E+06.E+06 5.E+05 0.E time unit Copyright 206 FUJITSU LIMITED

18 Conclusion We proposed a performance diagnosis framework Our framework selects that cause application performance problem Our framework can capture temporary correlated with application performance => use association rule mining technique => reduce false negatives We evaluated our framework From the perspective of the remaining diagnosis cost Verification of the accuracy of the remaining diagnosis cost is future work. We tested our framework on a real data set Temporary correlated actually exit Our framework can capture those Questions? 7 Copyright 206 FUJITSU LIMITED