Analysis of Server Resource Consumption of Meteorological Satellite Application System Based on Contour Curve

Transcription

1 Advaces i Computer, Sigals ad Systems (2018) 2: Clausius Scietific Press, Caada Aalysis of Server Resource Cosumptio of Meteorological Satellite Applicatio System Based o Cotour Curve Xiagag Zhao a, Cuqu Fa b, *, Mayu Li, Lizi Xie, Xi Zhag Natioal Satellite Meteorological Cetre, Beijig, Chia a xgzhao@cma.gov.c, b facq@cma.gov.c *correspodig author Keywords: meteorological satellite; server resource; cosumptio aalysis; cotour curve Abstract: I order to effectively aalyze the system server resource cosumptio, this paper proposed a server resource cosumptio aalysis method based cotour curve. Firstly, the defiitio of resource cosumptio aalysis was give, ad the several aalysis methods of resource cosumptio were give, icludig: applicatio resource cosumptio profile, applicatio resource cosumptio profile stadard deviatio, applicatio resource cosumptio frequecy, applicatio resource cosumptio mootoicity ad so o. Fially, based o this, the total cosumptio, cosumptio mea, kurtosis, skewess, reload, resource load ad so o were aalyzed. The method proposed i this paper ca effectively aalyze the server resource cosumptio i depth, improve the server resource utilizatio ad reduce the server hardware loss to a certai extet. 1. Itroductio Meteorological satellite groud applicatio system has a large amout of data traffic. The groud system is maily resposible for receivig, processig ad trasmittig data. This requires a large amout of server resources to support. Due to limited server resources, how to properly aalyze the cosumptio of server resources will be further optimizatio of server resources has a critical impact. Therefore, this paper studies the aalysis of server resource cosumptio. Some research has bee doe o the aalysis of server resource cosumptio. Paper [1] proposed a liear regressio method ad a maximum likelihood techique for estimatig the service demads of requests based o measuremet of their respose times istead of their CPU utilizatio. The paper [2] aims to address this problem by combiig kowledge of database query processig with statistical models. Paper [3] experimetally ivestigates the relatioship betwee the power cosumptio ad resource utilizatio of a multimedia server cluster whe differet load balacig policies are used to distribute a workload. I paper [4], they study the problem of idetifyig a assigmet scheme betwee the iteractig compoets of a applicatio, such as processes ad virtual machies, ad the computig odes of a cloud system, such that the total amout of resources cosumed by the respective applicatio is miimized. The paper [5] tries to establish 19

2 relatio betwee the power usage ad software agig. Paper [6] give a experimetal case study about resource cosumptio parameters ad workload parameters i a Iteret Iformatio Services. They use fitted resource cosumptio parameter to lear the relatioship betwee workload parameters ad resource cosumptio parameters through visual observatio ad calculatio. 2. Related Defiitio Resource cosumptio aalysis is to extract valuable statistical features ad quatitative idicators for importat parameters of each type of resources (CPU/memory/IO/etwork/NFS) of a applicatio through a certai statistical method, so as to preset the overall Apply a statistical patter of behaviour durig the ru. Each type of resource cotais a variety of parameters. If you aalyze each parameter, it will actually brig a higher amout of calculatio, ad i fact, some parameters are ot meaigful eve if they are aalyzed. After aalysis ad screeig, the applicatio resource cosumptio aalysis uses the followig importat parameter sets to collaborate o the aalysis object: a) CPU parameters: CPU_All_Sys, CPU_All_User, CPU_All_Idle, all i percetage. b) Memory parameters: Mem_All_MemRatio, Mem_All_SwapRatio, all i percetage. c) IO parameters: Disk_All_Read, Disk_All_Write, Disk_All_Svctm, i bytes (Bytes). d) Network parameters: Net_All_Recv, Net_All_Sed, i bytes (Bytes). e) NFS parameters: NFS_All_Read, NFS_All_Write, i bytes (Bytes). 3. The Method of Resource Cosumptio Aalysis 3.1 Applicatio Resource Cosumptio Profile The resource cosumptio profile is used to collectively show the overall level of occupyig a resource whe the target applicatio is scheduled multiple times. For example, for a CPU resource cosumptio profile curve, the legth of the curve represets the average time that the applicatio rus whe it is scheduled multiple times, ad the poit o the curve represets the mea value of CPU resources occupied by the applicatio i the case of multiple schedulig at the curret time poit. For the resource S of the target applicatio X, whe the applicatio is scheduled N times, ad the schedulig time of the i-th time is T i, the total legth of the resource cosumptio cotour correspodig to the resource S is defied as T=mi(T 1, T 2,, T N ), ad the resource cosumptio cotour value of the j-th poit. X + X + + X Yj = N 1j 2 j Nj Where X ij represets the value of the S resource parameter of the target applicatio X at the j-th time poit of the i-th schedulig. For CPU resources, this parameter is the CPU active total CPU_All_Active = CPU_All_Sys + CPU_All_User; for memory resources, physical memory footprit Mem_All_MemRatio ad swap area memory footprit Mem_All_SwapRatio will calculate the resource cosumptio profile; for IO, Parameters such as etwork ad NFS are calculated by usig the correspodig data read/write or etwork trasmissio ad receptio. (1) 20

3 3.2 Applicatio Resource Cosumptio Profile Stadard Deviatio The stadard deviatio of the applicatio resource cosumptio profile is used to show the fluctuatio of the correspodig resources occupied by the target applicatio at each time poit durig multiple schedulig. For a certai poit i time, if the applicatio is scheduled N times, whe the stadard deviatio of the cosumptio profile correspodig to the time poit is large, it idicates that the amout of the specified resource occupied by the N times of the applicatio at the time poit is discrete. The distributio may be due to ustable applicatio implemetatio ad large data chages. For the resource parameter S of the target applicatio X, whe the applicatio is scheduled N times, ad the i-th schedulig time is T i, the total legth of the resource cosumptio profile is defied as T=mi(T 1, T 2,..., T N ), ad the j poit resource cosumptio profile stadard deviatio. Y= DX (, X,, X ) (2) j 1j 2 j Nj Where X ij represets the value of the resource parameter S of the target applicatio X at the j-th time poit of the i-th schedulig, ad D(X 1, X 2,...) represets the stadard deviatio of the data. 3.3 Applicatio Resource Cosumptio Frequecy The frequecy of applicatio resource cosumptio is based o the resource cosumptio profile, ad aalyzes the fluctuatio patter of the applicatio resource cosumptio profile curve. By performig frequecy domai aalysis o the resource cosumptio profile curve, iformatio such as the fluctuatio amplitude ad phase of the cotour curve cosumed at each frequecy ca be obtaied. Whe the cosumptio profile curve has a small fluctuatio amplitude at all frequecies, it idicates that the hardware resources occupied by the target applicatio durig the etire workflow flow show a stable tred, which may be that the resource is occupied or occupied for a log time; The fluctuatio amplitude o the frequecy is much larger tha other frequecies, idicatig that the target applicatio occupies a ustable state of resources. For the target applicatio X, if the resource cosumptio profile curve of the correspodig resource S is Y, the frequecy of cosumptio of the resource S is rfft(y), where rfft is a real sequece fast Fourier trasform. 3.4 Applicatio Resource Cosumptio Mootoicity The CPU resource cosumptio mootoicity curve is similar to the CPU resource cosumptio profile stadard deviatio. It is also a measure of the CPU resources occupied by the target applicatio whe it is scheduled multiple times at a certai poit i time. Each poit o the mootoic curve idicates whether the CPU usage of the target applicatio presets a mootoous tred durig multiple schedulig at that poit i time. I theory, i the case of multiple schedulig, the CPU usage at each poit i time should preset a discrete radom distributio, ad should ot exhibit sigificat mootoicity. If the CPU usage of the multiple schedulig at a certai poit i time is obviously mootoous, there may be problems, such as busy garbage collectio tasks ad more space allocatio. For the target applicatio X, whe the applicatio is scheduled N times ad the i-th schedulig time is T i, the total legth of the CPU resource cosumptio profile is defied as T=mi(T 1, T 2,..., T N ), ad the j-th poit The value of the CPU resource cosumptio mootoicity curve Y j =MK(X 1j, X 2j,..., X Nj ), where X ij represets the value of the CPU occupacy of the target applicatio X at the j-th time poit of the i-th schedulig, MK(X, Y,...) represets the mootoic tred of discrete sequeces calculated usig the Ma-Kedall algorithm. 21

4 The Ma-Kedall algorithm is described i detail as follows. (1) For time series X = ( X1, X2, X ), the statistics for the test are as follows. 1 S( X)= sg( xj xk) (3) k= 1 j= k+ 1 1, x > 0 Where sg( x)= 0, x= 0. 1, x < 0 (2) Whe is large eough, it ca be proved that S(X) obeys a ormal distributio with a mea of 0 ad a variace of. Whe >10, the stadard ormal system variable calculatio expressio is as follows. S 1, S > 0 var ( S ) Z= 0, S = 0 S + 1, S < 0 var ( S ) The positive or egative of Z idicates that the origial time series is i a mootoous icreasig tred or a mootoous decreasig tred; the size of Z represets the credibility of the mootoic tred. Whe Z >0.8, the reliability of the mootoic tred of the origial time series is 80%, ad the origial data is cosidered to be mootoous. 4. Resource Type Profilig The resource type profile, which is a further statistical aalysis based o the cosumptio profile curve of a resource, is a overview. For each type of resource, the calculated statistical characteristics iclude total resource cosumptio, mea, peak, cocetratio, reload, resource cosumptio load, etc., ad specific statistical characteristics, such as throughput, for certai resources, occupacy, etc. Resource type profiles iclude total cosumptio, cosumptio mea, kurtosis, skewess, reload, resource load, ad so o. 4.1 Total Resource Cosumptio For the cosumptio profile X of a resource, the total cosumptio S is S=SUM(X 1, X 2,..., X ). 4.2 Mea Value of Resource Cosumptio For the cosumptio profile X of a resource, the cosumptio mea E is E=AVE(X 1, X 2,..., X ), ad AVE is a average fuctio. 4.3 Resource Cosumptio Kurtosis Kurtosis is also kow as the kurtosis coefficiet. Characterizatio of the peak height of the probability desity distributio curve at the mea. Cosiderig the resource cosumptio profile as (4) 22

5 a distributio i probability space, with the ormal distributio as the referece, the kurtosis ca represet the cocetratio of the resource cosumptio distributio i a short time; uder the same stadard deviatio, the greater the kurtosis coefficiet The distributio has more extreme values, the the rest of the values must be more cocetrated aroud the mode, ad its distributio is boud to be steeper, that is, there is a situatio i which the resource cosumptio is excessively cocetrated i a short period of time. For the cosumptio profile X of a resource, the cosumptio kurtosis is as follows. 1 4 ( x ) 1 i u i= K= 1 ( i 1 ( xi u = )) Where u is the mea of resource cosumptio ad is the legth of the cosumptio profile X. 4.4 Resource Cosumptio Skewess The skewess coefficiet is a feature umber that describes the degree to which the distributio deviates from the symmetry. The resource cosumptio profile is regarded as a distributio o the probability space, with the ormal distributio as the referece. Whe the distributio is bilaterally symmetric, the skewess coefficiet is 0. Whe the skewess coefficiet is greater tha 0, that is, whe the heavy tail is o the right side, the distributio is right-biased. Whe the skewess coefficiet is less tha 0, that is, whe the heavy tail is o the left side, the distributio is left-biased. Ituitively, whe the skewess coefficiet is less tha 0, the heavy tail is o the left side, idicatig that the overall resource cosumptio is i the form of quick ad slow ; whe the skewess is greater tha 0, the heavy tail is o the right, idicatig that the overall resource cosumptio is preseted. The form of "slow first ad the urget". For a cosumptio profile X of a resource, the resource cosumptio skewess is as follows x u W= ( ) (5) i 3 (6) i= 1 σ Where u is the mea of resource cosumptio, σ is the stadard deviatio of the cosumptio profile X, ad is the legth of the cosumptio profile X. 4.5 Resource Cosumptio Cocetratio ad Reload The cocetratio of resource cosumptio idicates the proportio of time occupied by the high cosumptio of resources i the overall time; the resource cosumptio overload degree is similar to the cocetratio degree, which is the total cosumptio of resources i the overall time. For a class of resources, whe the job is ruig i the whole process, it is i a high-occupacy ad high-cosumptio mode. This idicates that the correspodig hardware resources are uder great pressure, ad whe the time is too log, there may be a risk of dowtime. For the cosumptio profile curve X of a resource, the resource cosumptio overload degree is as follows. H= xi I{ xi Th} (7) i= 1 The resource cosumptio cocetratio is as follows. 23

6 1 C= I{ xi Th} (8) i = 1 Where is the legth of the cosumptio profile X ad I{} is the sig fuctio. 1, f = true I{} = 0, f = false Th is a threshold, meaig that whe the occupacy rate of the resource parameter is higher tha the threshold, it is cosidered to be high occupacy; whe the resource rate of the resource parameter is lower tha the threshold, it is ot cosidered to be high occupacy. The curretly set threshold Th = 80%, called 80% time cocetratio C 80%. I geeral, the value of the threshold Th ca be set to correspod to differet time cocetratios. 4.6 Resource Load The resource cosumptio load is a idicator that quatifies the resource occupacy curve ad measures the load of a resource per uit time durig the ruig of the target job. For the occupacy curve of a certai type of resources, the resource cosumptio load aalyzes the value of the curve at each time poit. Based o the cosideratio of reducig the overall system load, the high occupacy is give a higher weight, ad the lower occupacy is give a lower value. The weight value, such that the value calculated i the form of a weighted average is called the resource cosumptio load. For the cosumptio profile curve X of a resource, ad T is the legth of the curve X, the detailed algorithm for applyig the resource cosumptio load is as follows. (1) Set the upper boud, lower boud ad icremet times of the algorithm accordig to the rage of values of the resource parameters. For a percetage parameter such as CPU_All_Idle, Mem_All_MemRatio, the upper limit is geerally set to 100, the lower boud is 0, ad the icremet is set to 100. For a parameter with a relatively large value of Net_All_Recv, the logarithm of the parameter value is geerally first, ad the The upper boud is 32, the lower boud is 0, ad the icremet is set to 100. (2) Accordig to the umber of icremets, the lower boud of the algorithm is uiformly icreased to the upper boud of the algorithm. For each icremet, the icremetal threshold is calculated, ad how may parameter values i the time period are higher tha the curret threshold. For example, if the curret threshold is 50, you eed to cout the percetage of the parameter value greater tha 50 i the curret time period as a percetage of the total time. This percetage is the weight correspodig to the threshold. The weight is the threshold obtaied i step (2), ad the icremetal threshold is weighted. If the parameter value higher tha 50 accouts for 20% of the total time, ad the parameter value higher tha 60 accouts for 10% of the total time, the weighted sum is 20%*50+10%*60=16, which is the resource load value. The resource cosumptio aalysis is cetered o the resource cosumptio profile curve, ad the correspodig statistical features are added. The job is cosidered to be relatively ormal whe the cotour curve exhibits the followig patter. 1) The CPU resource cosumptio profile average is less tha 60%, the CPU load is less tha 2000; the memory resource cosumptio profile average is less tha 75%, ad the memory load is less tha ) The CPU/memory cotour stadard deviatio curve mea value is 0, idicatig that the resource cosumptio occupied by each schedulig of the job is cosistet. (9) 24

7 3) CPU/memory/IO resource cosumptio cocetratio is less tha 70% 5. Coclusios I order to aalyze the server resource cosumptio of satellite groud applicatio system, this paper proposes a server resource cosumptio aalysis method based o cotour curve. Through the defiitio of resource cosumptio aalysis, several aalysis methods of resource cosumptio are give, icludig: applicatio resource cosumptio profile, applicatio resource cosumptio profile stadard deviatio, applicatio resource cosumptio frequecy, applicatio resource cosumptio mootoicity ad so o. O this basis, the total cosumptio, cosumptio mea, kurtosis, skewess, reload, resource load, etc. are aalyzed. The method proposed i this paper ca effectively aalyze the server resource cosumptio i depth, thereby improvig server resource utilizatio ad reducig server hardware loss to a certai extet. Ackowledgemets The work preseted i this study is supported by Natioal Key R&D Program of Chia (2018YFB ), Natioal Key R&D Program of Chia (2018YFB ), The Upgrade Program of New Satellite Data Receivig ad Processig System (201601JJ01). Refereces [1] Dawso S, Dawso S, Dawso S, et al. Estimatig service resource cosumptio from respose time measuremets[c]. Iteratioal ICST Coferece o PERFORMANCE Evaluatio Methodologies ad TOOLS. ICST (Istitute for Computer Scieces, Social-Iformatics ad Telecommuicatios Egieerig), 2009:48. [2] Li J, Narasayya V, Chaudhuri S. Robust estimatio of resource cosumptio for SQL queries usig statistical techiques[j]. Proceedigs of the Vldb Edowmet, 2012, 5(11): [3] Dargie W, Schill A. Aalysis of the Power ad Hardware Resource Cosumptio of Servers uder Differet Load Balacig Policies[C]. IEEE, Iteratioal Coferece o Cloud Computig. 2012: [4] Tziritas N, Kha S U, Xu C Z, et al. O miimizig the resource cosumptio of cloud applicatios usig process migratios[j]. Joural of Parallel & Distributed Computig, 2013, 73(12): [5] Moha B R, Reddy G R M. Resource Cosumptio Aalysis Usig ARIMA-ARCH ad ARIMA-GARCH Models for Virtualized Server Cosolidatio System[J]. Iteratioal Joural of Software Egieerig & Its Applicatios, 2017, 11(6):1-14. [6] Ya Y, Guo P, Cheg B, et al. A experimetal case study o the relatioship betwee workload ad resource cosumptio i a commercial web server[j]. Joural of Computatioal Sciece, 2018, 25(03):