Resource Usage Monitoring for Web Systems Using Real-time Statistical Analysis of Log Data

Size: px

Start display at page:

Download "Resource Usage Monitoring for Web Systems Using Real-time Statistical Analysis of Log Data"

Vernon Phelps
6 years ago
Views:

1 Resource Usage Monitoring for Web Systems Using Real- Statistical Analysis of Log Data MATSUKI YOSHINO, ATSURO HANDA Software Division, Hitachi Ltd. 53, Totsuka-cho, Totsuka-ku, Yokohama, JAPAN {matsuki.yoshino.pw, NORIHISA KOMODA Graduate School of Information Science and Technology Osaka University 2-1 Yamadaoka, Suita, Osaka, JAPAN MICHIKO OBA The School of Systems Information Science Future University Hakodate Kamedanakano-cho, Hakodate, Hokkaido, JAPAN Abstract: - For Web-based systems accessible from the Internet, it is difficult to estimate workloads precisely. Precise estimation of resources necessary for the system is important for effective utilization of resources in a datacenter. Therefore, capacity planning to forecast the amount of IT resources necessary for a system is important. In capacity planning, the amount of resources necessary for a system is calculated based upon numbers determined by the architecture and business requirements of the system. An example of a number determined by the architecture is the amount of memory required by a single user. An example of a number determined by business requirements is the estimated maximum number of simultaneous users. By multiplying these two numbers, a maximum memory requirement can be calculated. Usually, system memory consumption and the number of simultaneous users are monitored during operation, and if either value exceeds a threshold, an alarm is sent to operators. The authors propose a method to monitor memory consumption per user from memory consumption data and the number of users, and perform statistical significance testing in real by applying a stream database. The window size used in a CQL statement for the test affects the precision of the test and memory consumption of the stream database. Through experimentation, the authors propose an optimal window size. Key-Words: - Web systems, Resource usage, Statistical testing, Real- analysis, Stream database, Capacity planning 1 Introduction Currently, most information systems provided to enterprises and by enterprises to customers and end users are Web based. For Web-based systems, it is common that the number of accesses varies dramatically and rapidly. To maintain stable system performance, it is necessary to prepare sufficient resources. Capacity planning is widely conducted in order to estimate the amount of resources necessary for information systems. A method for capacity planning applied in a real-world rapidly expanding service is published in [1]. Even with capacity planning, resource shortages can occur due to various reasons. Resource shortages occur if assumptions of capacity planning do not hold. There are two salient assumptions in capacity planning. One is the number of expected ISBN:

2 accesses, and the other is the amount of resources necessary for a single access. The total amount of resources is estimated by the product of these numbers. Traditionally, the amount of resources dedicated to a system is determined by the results of capacity planning that consider peak usage, or the system and safety coefficient based upon experience. Therefore, considering an entire enterprise, too many resources are deployed. To avoid this over-deployment of resources, cloud technologies have been introduced to assign resources to applications adequately in order to optimize resource usage. In a cloud environment, if a resource shortage occurs in a system, additional resources are deployed to the system from a resource pool. Nevertheless, because the additional resources are costly and may not available, monitoring the resource usage of systems is still very important, even in a cloud environment. In order to detect resource shortages in advance and prevent the system from going down, resource usage monitoring by using thresholds is widely implemented. Because monitoring the total amount resource usage means monitoring a product of two factors of capacity planning, it is difficult to analyze the real cause of unexpected usage of resources. The authors propose a method to monitor each factor by analyzing log data in real by using a stream database and to detect unexpected resource usage by applying statistical testing. 2 Limitations of Threshold Monitoring 2.1 Limitations of Total Resource Monitoring In capacity planning, the amount of resources necessary for processing a unit of work, such as a transaction or a user, is determined by considering the application framework used for the applications and experiences of similar applications. In this paper, this value is called the design base value. The total amount of resources is calculated by multiplying the design base values by the estimated number of maximum number of units of work to be processed simultaneously. Let r be the design base value, F be the amount of resources necessary without any work units, and q be the number of work units to be processed. The total amount of necessary resources R is thus given as follows: R F qr (1) It is common to set two levels of thresholds for monitoring total resource usage. The first threshold is a warning level, and the second one is an emergency level. Figure 1 shows an example of these two thresholds. In this case, the warning threshold is set to 5%, and the emergency threshold is set to 8%. Both graphs shown in Fig.1 and Fig.2 are example cases in which total resource monitoring does not work well. Fig.1 shows a case in which resource usage rises too rapidly, and there is little interval between exceeding the warning threshold and then exceeding the emergency threshold. Fig.2 shows a case where three incidents exceeded the warning threshold, but the system returns to normal status immediately; therefore, these warnings are false alarms. CPU Usage[%] emergency threshold warning 11:23:2 11:46:4 12:1: 12:33:2 12:56:4 13:2: 13:43:2 14:6:4 14:3: 14:53:2 15:16:4 15:4: 16:3:2 16:26:4 16:5: 17:13:2 17:36:4 Fig.1 An example of threshold monitoring (case 1) CPU Usage[%] emergency threshold warning threshold 11:26:2 11:52:4 12:19: 12:45:2 13:11:4 13:38: 14:4:2 14:3:4 14:57: 15:23:2 15:49:4 16:16: 16:42:2 17:8:4 17:35: Fig.2 An example of threshold monitoring (case 2) ISBN:

3 2.2 Method of Monitoring the Run Base Value, and Its Limitation In total resource monitoring, only the result of expression (1) is taken into account. In order to take account of the meaning of expression (1), by transforming expression (1), consider the following expression: R F r (2) q Obtaining total amount of resource usage R m and a number of units of work q for example, a number of simultaneous accesses from log data the amount of resources consumed by a unit of work during run r m can be calculated by following expression: Rm F rm (3) q The value calculated by expression (3) in run is called the run base value. By comparing the run base value and design base value during run, it is possible to verify whether the system consumes resources as planned in capacity planning. In preliminary experimentation, comparison between these two values is performed by evaluating the difference of these values by using a simple threshold. In a real Web system, the variance of the run base value is large. A threshold that is too large causes a delay in detecting abnormal behavior of the system, and a threshold that is too small causes too many false alarms. σ 2 /n. In large scale Web systems, the number of units of work for example, the number of users simultaneously accessing the system can be in the hundreds and regarded as large enough to apply the central limit theorem. Furthermore, the interval for calculating the run base value is larger than the duration of each transaction in the Web system, and each run base value has no correlation and can be considered independent. By these observations, the distribution of run base values can be assumed as a normal distribution, and a t-test can be applied. In the following discussion, let m be the design base value and consider the test of equivalence of m and the mean of the distribution of run base values. In this case, the variance of the distribution of run base values is unknown. Then, by a twosided t-test of significance level α, the condition to conclude that a mean of run base values is equivalent to m is as follows, where is a mean of run base values and n is the number of run base values to be tested: In expression (4), α (4) is an α/2 percentile of the t-distribution of degree of freedom n-1, and u 2 is an unbiased variance, which is given by the following expression, where x 1, x 2,..., x n are run base values: 3 Monitoring Resource Usage by Statistical Test In order to overcome the limitation shown above, the authors propose a method to evaluate the deviation of the run base value from the design base value quantitatively. Assuming that a distribution of population is normal, there are several statistical testing methods to test equivalence of means of populations with significance level α. In this paper, a t-test is applied. Because the run base value is calculated by expression (3), run base value is a mean of the amount of resources consumed by each unit of work. It is safe to assume that distributions of the amount of resources consumed by each unit of work are the same. Assuming that the mean of the distribution is m and the variance of that is σ 2, by the central limit theorem, if n is large enough, a mean of n probability distributions X i (i = 1, n) is approximated by a normal distribution of mean m and variance 4 An Implementation of Statistical Testing Using A Stream Database 4.1 Overview of Stream Databases A stream database is a CEP (complex event processing) tool and is used for analyzing continuously created data in real. A stream is defined as an infinite series data sequence, to which a datum is appended upon occurrence of some event. By applying some condition, a finite set of data can be extracted from a stream. The extracted finite set of data can be treated like a table in a relational database. In a stream database engine, CQL (Continuous Query Language) [2], which has similar syntax and semantics as SQL (Structured Query Language), can be used to perform queries to extracted finite sets of data. Among well known ISBN:

4 examples of stream databases, there are reports on monitoring packet traffic through networks [3] and monitoring messages issued in a large scale datacenter to detect abnormalities in the datacenter [4]. Data Input stream Sliding window: A CQL statement processes data within a sliding window. Output stream The window size is determined either by interval or number of data. In this case, the size is determined by the number of data. 4.2 Determining the Window Size for a CQL Statement for Statistical Testing In Chapter 3, it is assumed that the number of run base values used for a statistical test is sufficiently large. In a CQL statement for statistical testing, the number of run base values used for a statistical test is expressed as a window size of the CQL statement. A window size is the width of a sliding window to extract a finite set of data from a stream. The concept of sliding windows and window size is explained in Fig.3. A sliding window can be defined either by the number of data in a sliding window or by the amount of elapsed after data enters the window. For the purpose of statistical testing in this paper, the size of a sliding window is determined by the number of data. In order to realize real- processing, stream database engines handle all the data in memory. Therefore, to attain scalability of a stream database engine, it is important to reduce memory usage of the stream database engine. For example, in the stream database engine that the authors applied in the experiments mentioned in Chapter 5, 2sW + 1 (KB) of memory is consumed for each system to be monitored, where W is the window size of a CQL statement for statistical testing, and the data size of an element of the input stream is s. In the case of the authors' experiments, s is 1.25 KB. On the other hand, 2 MB of memory is consumed for registration of each CQL statement. For the experiment, 7 CQL statements have to be registered; therefore, 14 MB of memory is consumed by only the registration of CQL statements for statistical testing. In this situation, if W is less than 1, memory consumption proportional to window size is less than 2.5% of memory consumption used by registration of CQL statements, and can be considered negligible. However, if the window size is over 1,, then memory consumption proportional to window size is about 2% of memory consumption used by registration of CQL statements. This has a significant impact on overall memory usage of the stream database engine. In designing CQL statements for statistical testing, it is important to find an adequate window size to realize both early detection of abnormalities in resource Fig.3 Sliding window and window size usage and minimization of memory usage of a stream database engine used for statistical testing. 4.3 Determining the Significance Level of Statistical Testing Besides the window size for CQL statements, the significance level of statistical testing is also a critical parameter for checking conditions discussed in Chapter 3. From the definition of the significance level, it is obvious that the larger the significance level is set, the more false alarms occur, but the possibility of problems being detected increases. When an alarm is issued, it is necessary to examine whether some actions need to be taken, which causes an increase in operation costs. Balancing operation costs and the range of possible abnormal situations to detect in a datacenter can be quantitatively defined by the significance level of the statistical test. This quantification can be used as a parameter of a service level agreement with managed service providers or cloud service providers of PaaS (Platform as a Service). Service providers can demand more expensive service fees for monitoring services with larger significance levels for statistical testing. Therefore, it is important to determine an adequate significance level to set. 5 EXPERIMENTS 5.1 Environment of Experiments In order to determine the adequate window size setting and significance level, the authors conducted experiments using data simulating the environment shown in Fig.4. In the environment assumed, a Web application runs on three servers, and from each server, information about the number of users simultaneously accessing the server and the total CPU usage are periodically output to respective log files. Data that simulates typical problems in Web applications (discussed in document [5]) is created and used as test data in the experiment. ISBN:

Capacity planning tool Stream database Users Run environment Merged log info log info log info Load balancer log info Fig.4 Environment assumed in experiments Fig.

5 Capacity planning tool Stream database Users Run environment Merged log info log info log info Load balancer log info Fig.4 Environment assumed in experiments Fig.5 shows the number of accesses for the situation shown in Fig.1. Both information shown in Fig.1 and Fig.5 are provided directly by the logs. Fig.6 shows information on CPU usage per access, that is, an example of run base values. The CQL statements used for statistical testing merge two pieces of log information, calculate the information shown in Fig.6, and perform statistical tests against the design base value, which is.1% in this test case. Fig.7 shows the number of simultaneous accesses, and Fig.8 shows run base values of the situation shown in Fig.2. Although there are three spikes in Fig.2, there are only two spikes in Fig.7. That means that only the spike around 13:3 in Fig.2 is caused by unexpected behavior of the system, and the other two spikes are due to an abrupt increase in the number of accesses and is considered to be normal behavior. Fig.8 shows the run base values for the data simulating a normal state. In all cases, the design base value is.1%. All the log data is read every 1 seconds, and new data is added to streams. Number of Access CPU Usage per Access[%] :24:5 11:49:4 12:14:3 12:39:2 13:4:1 13:29: 13:53:5 14:18:4 14:43:3 15:8:2 15:33:1 15:58: 16:22:5 16:47:4 17:12:3 17:37:2 Fig.5 Number of accesses (case 1) 11:23:2 11:46:4 12:1: 12:33:2 12:56:4 13:2: 13:43:2 14:6:4 14:3: 14:53:2 15:16:4 15:4: 16:3:2 16:26:4 16:5: 17:13:2 17:36:4 Fig.6 CPU usage per access (case 1) 5.2 Results of Experiments Experiments are performed by using CQL statements that issue an alarm when the condition discussed in Chapter 3 is not met, with various combinations of window size N and significance level α. Table 1 shows the s when deviations of a run base value from the design base value were detected. Compared with Fig.1, wherein a warning alarm is issued around 14:4, deviation of the run base values from the design base value, which causes unexpected resource usage behavior in the system, is detected well in advance. Table 2 shows the results of whether the spike at 13:3:3 is detected for the data of Fig.2 and Fig. 8. Table 3 shows number of false alarms for the norml case data shown in Fig. 9. Number of Access :22:1 11:44:2 12:6:3 12:28:4 12:5:5 13:13: 13:35:1 13:57:2 14:19:3 14:41:4 15:3:5 15:26: 15:48:1 16:1:2 16:32:3 16:54:4 17:16:5 17:39: Fig.7 Number of accesses (case 2) ISBN:

6 CPU Usage per Access[%] CPU Usage per Access[%] :23:2 11:46:4 12:1: 12:33:2 12:56:4 13:2: 13:43:2 14:6:4 14:3: 14:53:2 15:16:4 15:4: 16:3:2 16:26:4 16:5: 17:13:2 17:36:4 Fig.8 CPU usage per access (case 2) : 11:3 12: 12:3 13: 13:3 14: 14:3 15: 15:3 16: 16:3 17: 17:3 Time Fig.9 CPU usage per access (normal case) As shown in Table 1, the larger N and α are, the sooner the detection of deviations of run base values from the design base value. However, as shown in Table 3, the larger N is, the more false alarms occur. Considering the argument of memory consumption of the steam database engine in 4.2, an appropriate value of N is approximately 4, and an appropriate range of α is from.1 to.5. Compared with total resource monitoring shown in Fig.1, by using this proposed method, unexpected resource usage is detected sooner. Compared with the graph of Fig.2, by using this proposed method, false alarms can be avoided by properly setting parameters N and α. Table 1. Times of detection of abnormality α N :44:2 13:14: 13:7:1 13:5:5 13:6:1 13:6:.5 14:13:2 13:2: 13:8:5 13:7: 13:6:5 13:6: :33:2 13:26:3 13:11:1 13:8:1 13:7:2 13:7:2.1 13:35:3 13:13:1 13:1: 13:8:1 13:8:1.5 14:33: 13:49:4 13:27:4 13:8:5 13:14:5 Table 2. Detection of the spike at 13:3:3 α N Table 3. Number of false alarms α N CONCLUSION For detecting unexpected resource usage behavior of Web systems, the authors propose a method of monitoring deviations of run base values from the design base value using statistical testing. Effectiveness of the method is proven by experiment using the implementation of statistical testing using a stream database. References: [1] J. Allspaw. The Art of Capacity Planning: Scaling Web Resources, O'Reilly Media, 28. [2] A. Arasu, S. Babu, and J. Widom, The CQL Continuous Query Language: Semantic Foundations and Query Execution, Technical report, Stanford University ( 23. [3] C. Cranor, T. Johnson, O. Spataschek, and V. Shkapenyuk, Gigascope: A stream database for network applications, Proc. of the 23 ACM SIGMOD Intl. Conf. on Management of Data, 23, pp [4] M. Yoshino, M. Oba, N. Komoda, T. Yamade, and S. Nakamichi, Message analysis method based on a stream database for information system management, Proc. of the 4th International Conference on Research Challenges in Information Science 21 (RCIS 21), 21, pp [5] K. Kobori, S. Moro, M. Satou, H. Ishigaki, and T. Iiyama. Practical Debugging and Troubleshooting (Java Edition), Shoeisha, 21. (in Japanese) ISBN:

On a Method for Describing System Management Operations and its Use in Evaluating Energy Saving Operations

DVNCS in DT NTWORKS, COMMUNICTIONS, COMPUTRS On a Method for Describing System Management s and its Use in valuating nergy Saving s Matsuki Yoshino *, Norihisa Komoda**, Michiko Oba *** * Software Division