Utilization vs. SLO-Based Control for Dynamic Sizing of Resource Partitions

Transcription

1 Utlzaton vs. SLO-Based Control for Dynamc Szng of Resource Parttons Zhku Wang, Xaoyun Zhu, Sharad Snghal HP Laboratores Palo Alto HPL-5-6(R.) January 3, 6* server vrtualzaton, resource partton, system dentfcaton, adaptve control In ths paper we deal wth a shared server envronment where the server s dvded nto a number of resource parttons and used to host multple applcatons at the same tme. In a case study where the HP-UX Process Resource Manager s taken as the server parttonng technology, we nvestgate the techncal challenges n performng automated szng of a resource partton usng a feedback control approach, where certan nput varable such as the CPU enttlement for the partton s dynamcally tuned to regulate output metrcs such as the CPU utlzaton or SLObased applcaton performance metrc. We demonstrate the mportance of obtanng proper models to characterze both the statc and dynamc nput-output relatons, dentfy the nonlnear and bmodal propertes of the models across dfferent operatng regons, and dscuss ther mplcatons for the desgn of the control loop. We then present varous controller desgns for ether the relatve utlzaton or the mean response tme, evaluate the performance of the closed-loop systems whle varyng certan operatng condtons, and dscuss ther advantages and ssues. Fnally, we present an adaptve controller that combnes the CPU enttlement and utlzaton nformaton and acheves more robust performance than pror solutons. * Internal Accesson Date Only Publshed n and presented at the 6 th IFIP/IEEE Dstrbuted Systems: Operatons and Management (DSOM 5), 4-6 October 5, Barcelona, Span Approved for External Publcaton Copyrght 5 IEEE

2 Utlzaton vs. SLO-Based Control for Dynamc Szng of Resource Parttons Zhku Wang Xaoyun Zhu Sharad Snghal Hewlett Packard Laboratores 5 Page Mll Rd, Palo Alto, CA 9434 {zhku.wang, xaoyun.zhu, sharad.snghal}@hp.com Abstract In ths paper we deal wth a shared server envronment where the server s dvded nto a number of resource parttons and used to host multple applcatons at the same tme. In a case study where the HP-UX Process Resource Manager s taken as the server parttonng technology, we nvestgate the techncal challenges n performng automated szng of a resource partton usng a feedback control approach, where certan nput varable such as the CPU enttlement for the partton s dynamcally tuned to regulate output metrcs such as the CPU utlzaton or SLObased applcaton performance metrc. We demonstrate the mportance of obtanng proper models to characterze both the statc and dynamc nput-output relatons, dentfy the nonlnear and bmodal propertes of the models across dfferent operatng regons, and dscuss ther mplcatons for the desgn of the control loop. We then present varous controller desgns for ether the relatve utlzaton or the mean response tme, evaluate the performance of the closedloop systems whle varyng certan operatng condtons, and dscuss ther advantages and ssues. Fnally, we present an adaptve controller that combnes the CPU enttlement and utlzaton nformaton and acheves more robust performance than pror solutons. Keywords: server vrtualzaton, resource partton, system dentfcaton, adaptve control. Introducton Resource parttonng s a type of vrtualzaton technology that enables multple applcatons to share the system resources on a sngle server [3][5][9] whle mantanng performance solaton and dfferentaton among them. On most current systems, partton szes are predetermned and allocated to applcatons by system admnstrators, posng a challengng confguraton problem. On the one hand, each partton has to be provded wth enough resources to meet servce level objectves (SLOs) of the applcatons hosted wthn t n spte of changes n workloads and the underlyng system. On the other hand, excessve over-provsonng makes neffcent use of resources on the system. Offlne capacty plannng or calendar-based schedulng usng profles of past applcaton resource usage are not always accurate or up-to-date and cannot handle unexpected short-term spkes n demand. To ease management of shared server envronments, and to provde capacty on demand to enterprse applcatons, our work ams to develop formal control-theory based technques to automatcally sze a resource partton based on ts CPU utlzaton, the SLO and the tme-varyng workload of ts hosted applcatons. Ths paper s organzed as follows. In secton, we descrbe the technology, the overall archtecture and the test bed n our case study. Related work s revewed n Secton 3. Secton 4 descrbes how the nput-output behavor of the system was modeled, and dscusses mplcatons of the varous models for control desgns. Secton 5 presents a number of dfferent controller desgns and ther performance evaluaton usng our test bed. Fnally, we summarze our results and conclusons, along wth drectons for future work n Secton 6.

3 . A Case Study usng A Feedback Control Approach We conducted a case study where we used the HP-UX Process Resource Manager (PRM) [3] as an example of the resource parttonng technology. PRM s a resource management tool that can be used to partton a server nto multple PRM groups, where each PRM group s a collecton of users and applcatons that are joned together and allocated certan amounts of system resources, such as CPU, memory, and dsk bandwdth. Under overload condtons, PRM guarantees a mnmum enttlement of system resources to each PRM group. Optonally, f CPU or memory cappng s enabled, PRM ensures that each PRM group s usage of CPU or memory does not exceed the cap regardless of whether the system s fully utlzed. In ths paper, we focus on szng of a PRM group n terms of CPU allocaton (wth cappng enabled). In general, a PRM group can be ether a PSET PRM group that s assgned a subset of the system s processors, or an FSS PRM group that s assgned a percentage of the CPU cycles by specfyng a number of shares. Here we consder only the FSS PRM group and refer to the CPU percentage allocated as ts CPU enttlement. Because ths percentage s enforced by the Far Share Scheduler (FSS) n the HP- UX kernel, t can be changed at any tme thereby enablng dynamc szng of the PRM group. Ths s partcularly useful for dealng wth uncertantes n the resource demands of enterprse applcatons. It also allows a feedback control approach to be used for automated resource allocaton to these resource parttons.. System archtecture Fgure llustrates a shared server that has m resource parttons, where each partton can be a PRM group. We consder the scenaro where each PRM group s used to host one applcaton. The resource controller nteracts wth each partton through two modules, A and S, where S s the sensor that perodcally measures the performance metrc for applcaton, and A s the actuator that dynamcally sets the CPU enttlement for partton accordng to the output of the resource controller. The tmng of the controller s based on the noton of a samplng nterval. At the begnnng of each samplng nterval, the controller collects from S the measured performance for the last samplng nterval, compares t to the desred performance, computes the needed CPU enttlement for the current samplng nterval usng certan control algorthms and passes t to A for actuaton. In the remander of ths paper, we focus on resource control for one such partton. The results should be extensble to controllng multple parttons wth multple resource types usng any parttonng technology. Shared Server Partton (applcaton ) A S CPU enttlement Partton (applcaton ) A S Resource Controller Desred performance Partton m (applcaton m) A m S m Measured performance Fgure. CPU enttlement control system archtecture

4 . Test bed setup In the case study, we took the Apache Web server as an example of the hosted applcatons. We set up an FSS PRM group on an HP-UX server to host an Apache Web server of verson..5. We refer to ths PRM group as the Web server partton. We used a modfed verson of httperf.8 (ftp://ftp.hpl.hp.com/pub/httperf) on a Lnux clent to contnuously send HTTP requests to the Web server and to log the response tme of every request. We developed a sensor module that parses the httperf log and computes the mean response tme () of all the requests completed durng each samplng nterval. It s assumed that the Web server has a target value for the based on ts SLO. We also used a PRM provded utlty prmmontor to measure the average CPU utlzaton of a partton for every nterval. The CPU enttlement (wth cappng enabled) for the Web server partton can be adjusted at the begnnng of every nterval to bound the percentage of CPU cycles used by the Web server n that nterval. We chose the smplest possble workload where a sngle statc page was repeatedly fetched from the Web server at a fxed rate. Ths settng ensured that memory and network bandwdth utlzaton was low and only mnmum dsk actvty was nvolved. It means CPU was the only potental bottleneck n the system as the workload ntensty vared. Therefore, we focus on the problem of dynamcally determnng the CPU enttlement for the Web server partton usng feedback control. 3. Related Work Our approach dffers from pror work on operatng systems support for server resource reservaton and enforcement [7][7][6][7] or schedulng [][3] n that t s more generc and can be used on any commodty operatng system that supports a resource parttonng technology, and any applcaton that can be hosted nsde a partton. In addton, our control loop utlzes a feedback mechansm to determne an applcaton s CPU enttlement n real-tme, whch s useful n dealng wth uncertantes n the resource demands of typcal enterprse applcatons. In [8] a feedback-drven adaptve scheduler was presented to allocate a percentage of CPU cycles to a thread over a perod of tme. In contrast, our controller allocates a percentage of CPU cycles to a whole applcaton so that the assgned CPU enttlement can be ted drectly to the applcaton s SLO. Although the proposed feedback loop s already n use n some exstng workload management tools [4][6], our approach s dstnct n that we rely on classcal control theory to gude the desgn of the control algorthms. Feedback control theory has been appled to solve a number of performance or qualty of servce (QoS) problems n computer systems and servces n recent years. (See [][] and the references theren.) Ths approach allows feedback loops to be desgned n a systematc fashon and results n well-behavng closed-loop systems wth provable propertes such as stablty and responsveness. On the other hand, the robustness of these propertes depends heavly on the ftness of the mathematcal models used to characterze the dynamc behavor of the systems beng controlled and how these models are obtaned. Computer systems n general lack common mathematcal models that accurately descrbe ther complex, hghly-customzed and ever-evolvng behavor. As a result, much pror work that apples control theory employs a black-box approach and uses nput-output models to capture the dynamc relaton between control knobs (nputs) and performance metrcs (outputs). Although much of computer systems behavor exhbts nonlnearty, most researchers choose lnear models to represent the nput-output relaton for smplcty and tractablty. However, a sngle lnear model s often nsuffcent to unformly capture a system s behavor under all operatng condtons for at least two reasons. Frst, each lnear model may be only a local approxmaton of the system s nonlnear behavor around an operatng pont. Second, the system beng controlled may face constant changes n workloads and other condtons that lead to changes n the nputoutput relaton. Therefore, a model estmated under one operatng condton may not explan the 3

5 system s behavor under another condton. Snce models are used as the bass for controller desgn, mproper choce of models can lead to nstablty or poor performance of the closed-loop system. More recent work that apples adaptve control theory to computer systems addresses ths ssue by allowng the model to automatcally adapt to changes n operatng condtons usng onlne system dentfcaton [9][][][4]. However, these papers typcally focus on the controller desgn and do not offer n-depth dscusson of the modelng phase of the desgn cycle. In ths paper, we present a detaled quanttatve analyss of model ftness and varablty through our case study. Performance control of Web servers has been studed extensvely n the lterature. For nstance, applcaton-level mechansms or admsson-control schemes were proposed n [3][][8] to provde dfferent levels of servce to requests of dfferent classes. Whle these approaches were manly based on heurstcs or queung models, other work has appled classcal control theory to manage Web server delay or server resource utlzaton usng admsson control and content adaptaton [][8], connecton schedulng and process reallocaton [3], or applcaton parameter tunng [9]. All of these methods requre modfcaton to the server applcaton software (wth the excepton of [8]), whch may not be feasble for other enterprse applcatons. Snce our focus s not controllng Web server performance n partcular, but rather provdng a general approach for dynamc szng of any resource parttons, t s mportant that our approach s applcable to any applcatons that can be hosted nsde such parttons. On the other hand, we do use Apache Web server as a test case to nvestgate the challenges n both modelng and controller desgns. In [] the authors offered nsghts nto how to obtan approprate models for the actuator, sensor, and the controlled system usng benchmarkng and lnear regresson based estmaton technques, whlng usng CPU utlzaton as the output. The resultng relaton between the adaptaton level and the CPU utlzaton s a tme-varyng statc gan wth no dynamcs but wth a tme delay. In comparson, we use ether CPU utlzaton or response tme as the system output, and evaluate both the statc and dynamc nput-output relatons usng expermental data collected at both long and short tme scales. We found that both low-order dynamcs and nonlnearty are useful n capturng certan characterstcs n the nputoutput relatons. Ths work s the contnuaton of our earler work n [] where we desgned and mplemented an adaptve PI controller that regulates the around a target value and self-tunes ts gan parameters based on onlne estmaton of the dynamc model. In the next secton, we descrbe a new set of modelng experments and analyss and demonstrate how the system s nput-output relaton changes along wth varous operatng condtons of the system. As a result, we show that controllng the usng the CPU enttlement alone s only effectve when the Web server partton s CPU utlzaton s close to ts CPU enttlement, but may not work well when the applcaton s underutlzng ts enttled CPU. In ths paper, we present alternatve controller desgns such as controllng the relatve utlzaton of the partton, or ncorporatng CPU utlzaton nformaton nto the control of SLO-based metrcs such as the so that the closedloop system acheves more robust performance across dfferent operatng regons. 4. Modelng of the Input-Output Relaton 4. Statc nput-output relaton Frst, to understand the system s long-term average behavor n the whole operatng range, we vared the CPU enttlement (denoted by u) for the Web server partton from. to.9, at.5 ncrements. At each settng, the Web server was loaded for 6 seconds wth a fxed workload, whle the average CPU utlzaton (denoted by v) of the Web server partton was observed and the of all requests returned durng ths perod was computed. Fgure shows the statc relaton between the CPU enttlement, the (absolute and relatve) CPU utlzaton, and the for dfferent workload ntenstes rangng from to requests/second (or r/s). Note that 4

6 each data pont s the average of samples obtaned from repeated experments. In addton to u and v, let y denote the nverse of (/), and r denote the relatve CPU utlzaton of the partton,.e., r = v / u. CPU Utlzaton (v) Rate= Rate=4 Rate=6 Rate=7 Rate=9 Rate= CPU Enttlement (u) (a) CPU utlzaton vs. CPU enttlement Rate= Rate=4 Rate=6 Rate=7 Rate=9 Rate= CPU Enttlement (u) (b) vs. CPU enttlement.4 CPU Relatve Utlzaton (r) Rate= Rate=4 Rate=6 Rate=7 Rate=9 Rate= CPU Enttlement (u) / (y) (c) Relatve CPU utlzaton vs. CPU enttlement..8 Rate=.6 Rate=4 Rate=6 Rate=7.4 Rate=9 Rate= CPU Enttlement (u) (d) / vs. CPU enttlement Fgure. Long-term relaton between CPU enttlement, CPU utlzaton and Our key observatons from these fgures follow: As shown n Fgure (a), for any gven request rate, as the CPU enttlement vares, the CPU utlzaton demonstrates a clear bmodal behavor that can be approxmated usng the followng equaton: u, f u < V, v = () V, f u >= V, where V s the maxmum porton of CPU needed for a gven workload. For nstance, a workload of 6 r/s requres.4 CPU. When the CPU enttlement s below.4, the system s n the overload regon where the allocated CPU cycles are fully utlzed, therefore makng the utlzaton equal to the enttlement; when the CPU enttlement s above.4, however, the system s n the underload regon where the allocated CPU cycles are underutlzed. In the latter case, the CPU utlzaton stays relatvely constant. Fgure (c) shows a dfferent vsualzaton of the same behavor through the relatve CPU utlzaton. The followng equaton s equvalent to (), except expressng the CPU utlzaton n a relatve term: 5

7 , f u < V, r = () V / u, f u >= V. Ths nonlnear equaton wll be useful n the analyss of the controllers n the next secton. Smlarly, the same bmodal behavor s observed n the relaton between the and the CPU enttlement n Fgure (b). Snce the s clearly a nonlnear functon of the CPU enttlement, we plot / vs. CPU enttlement n Fgure (d) to better llustrate the relaton. As we can see, when the system s overloaded (r = n Fgure (c)), there exsts a lnear mappng from the CPU enttlement to /, and ts slope s ndependent of the request rate. However, when the system s underloaded (r < n Fgure (c)), / ncreases rapdly wth ncreasng CPU enttlement, ndcatng a sharp drop n the. The lnear mappng between the CPU enttlement and / for the overload regon mples that a lnear nput-output model s plausble for ths regon f / s chosen as the system output. When the system s reasonably underloaded (r <.8), the becomes ndependent of the CPU enttlement settng. Therefore, the s uncontrollable usng the CPU enttlement n ths regon. Fgure 3 shows the mean, maxmum and mnmum values of the relatve CPU utlzaton and from the experments for a request rate of 9 r/s. The relatve utlzaton n Fgure 3(a) shows a relatvely small varaton throughout the whole operatng regon, wth the varaton beng slghtly hgher n the underload regon. For the n Fgure 3(b), n the overload regon where CPU enttlement s n [*.,.55], the has a small varance. However, large devaton can be found n the underload regon where the CPU enttlement s n [.65.9], whch means that may not be taken as a stable metrc n ths regon. Relatve Utlzaton (r) Mean Max Mn CPU Enttlement (u) (a) Relatve CPU utlzaton Mean Max Mn CPU Enttlement (u) (b) Mean Response Tme Fgure 3. Propertes of relatve utlzaton and as metrcs when rate s 9 Next, we descrbe how to obtan a lnear dynamc model for controllng through CPU enttlement n the overload regon usng standard system dentfcaton technques. 4. Dynamc lnear model dentfcaton Smlar to pror work, we chose the followng lnear auto-regressve model as the potental model to represent the dynamc relaton between the CPU enttlement and the nverse of : n m y( = a y( k ) + b u( k d j) + ε (, (3) = j= 6

8 where the parameters a, b j, the orders m, n, and the delay d characterze the dynamc behavor of the system. y( s the nverse of for samplng nterval k, u( s the CPU enttlement for samplng nterval k, and ε ( s the resdual term. For convenence, we refer to such a model as ARXmnd n the followng dscusson. We performed two seres of system dentfcaton experments for varous request rates and samplng ntervals, where the CPU enttlement was randomly vared n [.,.8], and the resultng / was calculated. The model n (3) wth dfferent structure and parameters was estmated offlne usng least-squares based methods [] n the Matlab System ID Toolbox [5] to ft the nput-output data. The models are evaluated usng the r metrc defned n Matlab as a goodness-of-ft measure. In general, the r value ndcates the percentage of varaton n the output captured by the model. In the frst seres, the samplng nterval was fxed as 5 seconds whle the rate was vared from r/s to r/s, and the experment was repeated for each rate. The results are shown n Tables (a), (b) and (c). In the second seres, the rate was fxed at 9 r/s but the samplng nterval was vared from 3 seconds to seconds. The r values (n %) of the resultng models are shown n Table (d). Table. Model ftness and parameter values for the ARX nput-output model Model Rate (r/s) ARX ARX (a) r of frst-order models under dfferent workloads Param. Rate (r/s) a b b/(-a) (c) Parameters of ARX under dfferent workloads (d) The observatons we made from the above tables are the followng: Rate (r/s) Model ARX ARX ARX33 ARX (b) r of models wth dfferent orders Model Samplng Interval (seconds) ARX ARX r of models under dfferent samplng ntervals The lnear ARX model fts the nput-output data for heaver workloads, not lghter workloads. When the system s sgnfcantly underloaded, a smple lnear model does not ft the nput-output data as shown n Table (a) by the r value for ARX (frst-order model wth no delay) for a rate below or equal to 6 r/s. Ths s consstent wth our earler observaton from Fgure (d). In contrast, when the request rate s above 6 r/s, the ARX model fts qute well, provdng a good bass for controller desgn. Moreover, ARX (frst-order model wth one-step delay) does not explan the system behavor for any request rate, showng that no sgnfcant delay s observed n the system dynamcs for a samplng nterval of 5 seconds. Under all condtons where an ARX model s a good ft, a frst-order model s suffcent. Table (b) shows, usng rates of 9 and r/s as examples, that ncreasng the order of the ARX model does not ncrease ts ftness. Addtonally, experence has shown that smpler models offer better robustness for the resultng control system. Under overload condtons, the parameter values of the dynamc model changes wth the workload ntensty. The evoluton of the parameters a and b values of the ARX model as shown n Table (c) reveals that, wth a heaver workload, the system contans more nerta and responds slower to changes n the CPU enttlement. However, the steady state 7

9 gan, b /( a), remans relatvely constant at around.4 (except for 7 r/s). Ths s consstent wth the constant slope n Fgure (d). Models wth dfferent delay values should be chosen for dfferent samplng ntervals. Table (d) shows how the ftness of the ARX (no delay) or ARX (one-step delay) model changes as we change the samplng nterval T s, under a workload of 9 r/s. For a short samplng nterval (T s =3 seconds), the ARX model provdes a much better ft ( r = 59.% ) than the ARX model ( r = 5.4% ), ndcatng a delay from the nput to the output of around 3 seconds. It s manly caused by the actuaton delay n PRM. For longer samplng ntervals (T s >5 seconds), ths delay can be gnored, and an ARX model provdes a farly good ft ( r > 7% ). For a samplng nterval of 5 seconds, nether model provdes a good enough ft. Because smaller T s results n noser data, and larger T s causes slower response n the controller, both the samplng nterval and the delay n the model should be chosen carefully before the controller desgn. The above experments and analyss were repeated for a dfferent server, and the same qualtatve results were observed. Our man concluson s that, due to the exstence of frst-order ARX models for the dynamc relaton between the CPU enttlement and / when the system s overloaded, the should be controllable usng smple controllers such as the adaptve PI controller used n []. On the other hand, t wll be qute challengng to regulate the n the underload regon because our observatons from the modelng exercse suggest that the s smply uncontrollable usng the CPU enttlement as the only nput. Because varaton n the workload ntensty may frequently move a resource partton between the two regons, t s desrable to desgn a controller that can manage the partton sze across both operatng regons. From Fgure, we know the essental reason that s uncontrollable when the Web server partton s underutlzed s that the s no longer correlated wth the CPU enttlement. However, the should always be dependent upon the real CPU utlzaton of the Web server process. Ths was confrmed from the followng exercse, where offlne dentfcaton experments were repeated when the CPU enttlement was randomly vared n two regons, as shown n Table. Under a fxed workload of 9 r/s, the system worked n the overload or underload regon when the CPU enttlement was vared n [.,.5] or [.5,.8], as can be seen from Fgure (b). Consder the ARX models between the CPU enttlement and /, the CPU utlzaton and /, and the CPU enttlement and utlzaton. In the underload case where the enttlement range s [.5,.8], / s only weakly correlated wth the CPU enttlement wth r =6%. However, the r value of the models between the CPU utlzaton and / s always much hgher. Therefore, t may be helpful to ntroduce the CPU utlzaton nto the control loop for the such that more robust desgns can be acheved. Table : r (n %) of ARX models between dfferent nput-output pars for dfferent regons Range of Enttlement [.,.5] [.5,.8] Enttlement --> / Utllzaton --> / Enttlement --> Utlzaton Controller Desgn and Performance Evaluaton The CPU utlzaton of a Web server s a common metrc that s montored to determne whether more or less CPU resource should be allocated to the server. The usage-based operaton mode n the HP-UX Workload Manager [4] allows the relatve CPU utlzaton of a PRM group to be controlled wthn a user-specfed range, e.g., 5%-75%. Compared to SLO-based metrcs such as response tmes, the relatve utlzaton of the resource partton s easer to measure on the server 8

10 sde, more drectly related to the CPU enttlement and ts control s more ntutve. The downsde les n that the relaton between a gven relatve utlzaton level and the clent-perceved servce level vares wth the demand of the workload. Therefore, no guarantees can be gven to metrcs such as the for an arbtrary workload when only the relatve utlzaton s beng controlled. Ths s n contrast to usng the as the controlled output that s more drectly related to the SLO but ts relaton wth the CPU enttlement s rather complex. In addton, regulatng the usng the CPU enttlement s more dffcult n the underload regon, as explaned n Secton 4. In ths secton, we present controller desgns for dynamc szng of the Web server partton usng both output metrcs, and dscuss possble ways to combne these two metrcs to provde more effectve control across the whole operatng regon. 5. Control of relatve utlzaton We frst consder dynamc szng of the Web server partton usng ts relatve utlzaton, r(, as the output and the CPU enttlement, u(, as the nput. The goal s to mantan dynamcally the relatve utlzaton at a reference value, r ref. Ths value can be chosen hgher for more predctable workloads, and lower for more varable workloads. Pror work on applyng control theory to regulatng Web server CPU utlzaton used dfferent nput varables, such as the content adaptaton level [] or the applcaton parameters [9]. From offlne dentfcaton experments, we observed that r( responds quckly to changes n u( wth neglgble delay and nerta when the samplng nterval s set at 5 seconds. Therefore, the nonlnear statc model () can be used to represent the nput-output relaton..e.,, f u( < V ; r ( = (*) V / u(, f u( >= V. Defne the trackng error at samplng nterval k as e( = rref r(. We can then use the classcal ntegral (I) controller to dynamcally tune the CPU enttlement based on the trackng error: u( = u( k ) K e( k ). (4) In theory, ntegral control ensures zero steady state error,.e., the measured relatve utlzaton should converge to r ref, and the ntegral gan K determnes the aggressveness of the tunng. Note that e( s a non-decreasng functon of u(. The mnus sgn n (4) s requred to ensure a negatve feedback loop for a postve K. The man challenge here s to choose the rght gan parameter K such that the closed-loop system s stable, and the relatve utlzaton tracks the reference value as quckly as possble. Ths can be llustrated usng an example. Let r ref =8%. We started from a workload of 5 r/s, changed t to 8 r/s at the th samplng nterval, and back to 5 r/s at the 4 th nterval. Fgure 4 demonstrates the behavor of the closed-loop system wth a K value of. and., respectvely. The two top fgures show both the measured relatve utlzaton and ts reference value. The two bottom fgures show the allocated CPU enttlement and the measured (absolute) CPU utlzaton. As we can see, when the gan s too small as n Fgure 4(a), the response to the workload change s very sluggsh; when t s large, however, the response s unstable as shown n Fgure 4(b). In ths experment and all those followed, the CPU enttlement was upper-bounded by.8. The x- axes n the fgures on the control system performance are labeled n the number of samplng nterval. 9

11 .5 Reference (r ref ) Relatve Utlzaton (r) Reference (r ref ) Relatve Utlzaton (r) Enttlement (u) Utlzaton (v) Enttlement (u) Utlzaton (v) (a) K =. (b) K =. Fgure 4. Performance of fxed I Controller from CPU enttlement to relatve utlzaton Although an optmum K value may be chosen carefully for certan workload, t may not be applcable to a dfferent workload or a dfferent operatng regon. Therefore, t should be desrable to desgn an adaptve controller wth an adjustable gan parameter. Snce the process s nonlnear, the controller desgn approach for lnear systems such as pole placement cannot be appled. Instead, we can perform the followng analyss. It does not solve the problem completely, but we can have some analytcal drecton on the parameter confguraton. We rely on the statc models defned n () and (*). Assume that at steady state, u ( ) = u, v( ) = V. Consder the problem n the two operatng regons respectvely. In the overload case, u( k ) V, v ( k ) = u( k ), then e ( k ) = r, and u = V / rref u( k ) / r ref. A safe but conservatve opton for K ( could be u ( k ) / rref such that u( = u( k ) / rref u. Note that ths adaptve gan wll lead to exponental ncrement of the CPU enttlement n the overload regon snce r <. In the underload case, u ( k ) > V, and u( = u( k ) K ( r v( k ) / u( k )). Let us consder the local stablty of ref ths nonlnear system. Lnearzng the equaton around u, we derve that K V / u < s requred for the local stablty. Ths s a necessary condton. But we can stll consder one canddate for K n ths regon as K k ) = λ v( k ) r, wth some < λ <. Fgure 5(a) ( / ref ref ref shows the performance of such a nonlnear adaptve controller where λ =.5. The relatve utlzaton s mantaned around the reference, and the CPU enttlement tracks the changes n the workload farly quckly. Because the nonlnearty n equaton (*) n the underload regon, the analyss of the global stablty for the controller n (4) s challengng. We perform the followng transformaton to lnearze the system here. Let ~ r ( = / r( k ) be the output of the system. Then r ~ ( = u( / V when u ( >= V. Now defne the trackng error at tme k as e ~ ( = ~ r ~ ref r (, and use the followng negatve feedback ntegral controller: ~ u( = u( k ) + K ~ e ( k ). (5) ~ We may set the gan K ( k ) adaptve to the workload as follows: ~ u( k ), when u( k ) = v( k ), K ( = (6) λ v( k ), when u( k ) > v( k ).

12 Then n the overload and underload cases, respectvely, we can stll have the exponental change of the enttlement when the workload changes. Moreover, global stablty can be guaranteed n the latter case snce the closed-loop system s lnear. One example for such a controller wth λ =.5 s shown n Fgure 5(b), whose performance s smlar to that n Fgure 5(a)..5 Relatve Utlzaton (r) Reference (r ref ). Enttlement (u) Utlzaton (v) (a) Nonlnear adaptve controller.5 Relatve Utlzaton (r) Reference (r ref ). Enttlement (u) Utlzaton (v) (b) Lnear adaptve controller Fgure 5. Performance of adaptve I Controller from CPU enttlement to relatve utlzaton 5. Control of mean response tme In ths secton, we demonstrate the challenges n controllng the mean response tme compared to controllng the relatve utlzaton. We frst consder both a fxed PI controller and the adaptve PI controller presented n [] that dynamcally adjust the CPU enttlement for the Web server partton to meet ts SLO-based target. Usng examples, we show that ths adaptve controller s effectve n handlng changes n the target performance by onlne estmaton of the dynamc model, but may not work well when a sudden change n the workload pushes the system nto the underload regon. We then descrbe a new controller desgn by ntroducng the CPU utlzaton measurement nto the control loop. Based on the analyss n Secton 3, we consder the followng ARX model to represent the dynamc relaton between the CPU enttlement (u) and the nverse of (y): y( = ay( k ) + bu( + ε (. (7) Defne the trackng error e( = yref ( y(, where y ref ( s the target value for /. Then a PI controller mplements the followng algorthm: u( = u( k ) + ( K p + K ) e( k ) K pe( k ). (8) The closed-loop system s of second order and the gan parameters, K p and K, can be chosen usng the pole placement algorthm accordng to desgn specfcatons such as overshoot, rsng tme and settlng tme [5]. We appled both the fxed PI controller and the adaptve PI controller to regulate the around ts target. For the fxed PI controller, we used the model dentfed offlne under a fxed rate 9 r/s wth (a, b) = (.,.4), from whch the best gan parameters were chosen. For the adaptve PI controller, the model parameters were estmated onlne and the gan parameters were also computed onlne to mantan the desred closed-loop poles. In the frst experment, a fxed workload wth 9 r/s rate was sent to the Web server for a duraton of 6 samplng ntervals (5 mnutes), whle the target of was changed from seconds to second at the 3 th samplng nterval. From Fgure (b), we can see that these two steady states are located n the overload and underload regons, respectvely. Fgure 6(a) and 6(b) show the performance of the closed-loop system for both controllers for the duraton of the experment. The top fgures compare the measured to ts target value, whle the bottom fgures show both the CPU enttlement actuated and the measured CPU utlzaton. Both controllers worked well when the system was overloaded. However, as n Fgure 6(a), the CPU enttlement was oscllatng heavly (httng the upper bound perodcally) when the model

13 parameters were fxed. Ths s because the gan parameters for the PI controller were chosen usng the model estmated over the whole nput regon, and were too aggressve when the system became underloaded. The adaptve PI controller fxes ths msmatch of the parameters and acheves more stable results as shown n Fgure 6(b). In the second experment, the target was fxed at.5 seconds, but the rate of the workload was changed from 9 r/s to 5 r/s at the 3 th samplng nterval, whch pushes the system suddenly nto the underload regon. Fgure 6(c) and 6(d) show the performance of the closedloop system for both controllers. We can see that both the CPU enttlement and the resultng became unstable because of the over-tunng actons of the controller. In ths scenaro, the adaptve controller only mproves the performance very margnally. 3 Reference Enttlement (u) Utlzaton (v) (a) Fxed PI controller upon change of reference 3 Reference Enttlement (u) Utlzaton (v) (b) Adaptve PI controller upon change of reference 4 Reference 4 Reference Enttlement (u).5 Utlzaton (v) Workload Enttlement (u).5 Utlzaton (v) Workload (c) Fxed PI controller upon change of workload (d) Adaptve PI controller upon change of workload Fgure 6. Performance of PI controllers from CPU enttlement to In summary, n the overload regon, both the fxed and adaptve PI controllers work reasonably well, whle the adaptve controller offers better performance n the slghtly underload regon (as n Fgure 6(b)) by self-tunng of ts parameters. However, nether controllers are applcable when the system s sgnfcantly underloaded (as n Fgure 6(c) and 6(d)), where the loss of controllablty of the by the CPU enttlement leads to over-provsonng of the CPU resource. Ths s also consstent wth our observaton from Fgure 3(b) that the s not a stable metrc n the underload regon. Gven the suggeston of Fgure 3(a) that the CPU utlzaton s a more stable metrc, we propose one desgn that attempts to ncorporate the measured CPU utlzaton nto the control loop to extend the controllable regon, as llustrated n Fgure 7.

14 Specfcaton Desgn (a,b) Estmaton (K p, K ) / ref e PI Controller Ent G Utl G / Fgure 7. Block dagram of an adaptve control loop wth ncorporaton of measured CPU utlzaton From Secton 3., we know that the CPU utlzaton has a tghter relaton wth the than the CPU enttlement does n the underload regon. In the followng desgn, the ARX model was estmated onlne between the measured CPU utlzaton (v(, as the nput) and / (y(, as the output). Moreover, the term u(k-) n the controller (8) s replaced by v(k-) as follows: u( = v( k ) + ( K p + K ) e( k ) K pe( k ). (9) The parameters were chosen accordng to the same specfcaton as n the pror desgns. The prevous experments wth varyng target or workload ntensty were repeated usng the new controller n (9) and the closed-loop performance s shown n Fgure 8. In both cases, the measured s kept near the reference values. In the second case, even wth a sgnfcantly reduced workload, the stablty of the system s mantaned. In ths control desgn, ntroducng the CPU utlzaton nto the model estmaton leads to a more truthful and stable model. Wth the controller n (9), over-tunng of the CPU enttlement can be avoded snce t s based on the measured utlzaton. However, one mplct assumpton n ths soluton s that the utlzaton measurement tracks the enttlement mmedately, that s, G =. Ths s satsfed n the overload regon where v ( = u(. When the system works close to the overload regon, t s an approxmaton of the relaton between the enttlement and the utlzaton. Error exsts n the underload regon between the expected value of the utlzaton and ts measurement. That s why, as shown n Fgure 8, the measured s above the target value when the system s underloaded. Ths steady-state error can be estmated approxmately as follows. Assume that v( = βu( wth β < and the closed loop s stable. Then the transfer functon s, by omttng the estmaton and desgn blocks, K p βb( K p + K )( z ) K p + K Gc ( z) =. () K p ( z β )( z a) + βb( K p + K )( z ) K + K Accordng to the Fnal Value Theorem, the DC gan of G c (z) s G c ( ) < snce β <. Note that the output s the nverse of the. The n steady state s larger than ts reference on average. It s possble to elmnate ths average error, for nstance, by settng the enttlement approprately larger than the expected utlzaton, or settng the reference proportonal to / G () p c to compensate the steady state error. However, t s challengng to choose the rght rato β due to varatons n the workload and measured utlzaton. Fgure 9 shows the performance of the controller n the two scenaros when β s assumed to be a fxed value of.9, and the output of the controller s multpled by /. 9 so that G = s satsfed. In the frst case when the reference value changes durng the process, the steady-state error n the underload regon s almost removed as shown n Fgure 9(a). In the second case as shown n Fgure 9(b), the error n the underload 3

15 regon s decreased, but the output s noser than before. In the overload regon, new errors appear n both cases. Ths s because β s actually n ths regon. These errors may be removed f the operatng regons can be dentfed. However, there s no obvous boundary between the two regons and t s dffcult to fnd the exact value for β. Therefore, the proposed soluton can mprove the robustness of the controller (8) sgnfcantly only n or close to the overload regon. 3 4 Reference Reference Enttlement (u) Utlzaton (v) (a) Varyng target Enttlement (u) Utlzaton (v) Workload (b) Varyng workload ntensty Fgure 8. Performance of the adaptve PI controller wth ncorporaton of measured CPU utlzaton 3 Reference Enttlement (u) Utlzaton (v) (a) Varyng target 4 Reference Enttlement (u) Utlzaton (v) Workload (b) Varyng workload ntensty Fgure 9. Performance of the adaptve PI controller wth ncorporaton of measured CPU utlzaton 6. Conclusons β s assumed to be.9 and u s set to /. 9 of the controller output. Ths paper dentfes challenges n applyng control theory to dynamc szng of a resource partton usng CPU enttlement as the nput and the mean response tme or the relatve CPU utlzaton as the output metrc. We emphasze that the nput-output relaton has to be accurately modeled for the theoretcal propertes of the closed-loop control system to be meanngful. We recognze that ths relaton vares sgnfcantly as the resource partton moves between the overload and the underload regons, whch has a notceable mpact on the performance of any controller desgn. We evaluate the performance of the closed-loop system usng ether relatve utlzaton or the mean response tme as the controlled output, and dscuss ther respectve advantages and ssues. Fnally, we present a new adaptve controller desgn for regulatng the 4

16 mean response tme that ncorporates nformaton on measured CPU utlzaton and mproves the robustness of pror adaptve algorthms. To make the system work well across all operatng regons, we need to respect the bmodal behavor of system and develop a better way to ntegrate the control of relatve utlzaton (usng controller (4-5)) and the response tme (usng controller (8)) n possbly dfferent regons. Note that the targets on the relatve utlzaton and the may not be tracked smultaneously n most regons. To make the combnaton possble, one scheme s to gve the regulaton of the relatve utlzaton a hgher prorty than that for the regulaton of the, and to desgn an ntellgent swtchng scheme between these two controllers as the system state moves across dfferent operatng regons. It could be appled to the control of multple resource contaners, where the objectves on relatve utlzaton can be guaranteed when there s no contenton for resource among the contaners, and then the controller for comes nto play when the system s overloaded. Ths s one topc of our ongong work. Another nterestng drecton s to apply the same approach to dynamc szng of a resource partton n terms of ts physcal memory allocaton. The dstnct nteracton between applcaton performance and ts memory may make t much more challengng to desgn a sensble controller that works under all operatng condtons. References [] T.F. Abdelzaher, Y. Lu, R. Zhang, and D. Henrksson, ``Practcal applcaton of control theory to Web servces,'' nvted paper, Amercan Control Conference, June 4. [] T.F. Abdelzaher, K.G. Shn, and N. Bhatt, Performance guarantees for Web server end-systems: A control-theoretcal approach, IEEE Transactons on Parallel and Dstrbuted Systems, vol. 3,. [3] J. Almeda, M. Dabu, A. Mankutty and P. Cao (998), Provdng dfferentated levels of servce n Web content hostng, SIGMETRICS Workshop on Internet Server Performance, June 998. [4] Apache Web server, [5] K. Astrom and T. Hagglund, PID Controllers: Theory, Desgn, and Tunng ( nd Edton), Instrument Socety of Amerca, 995. [6] K.J. Astrom and B. Wttenmark, Adaptve Control (nd Edton), Prentce Hall, 994. [7] G. Banga, P. Druschel, and J.C. Mogul, Resource Contaners: A new faclty for resource management n server systems, 3 rd USENIX Symposum on Operatng Systems Desgn and Implementaton, Feb [8] P. Bhoj, S Ramanathan, and S. Snghal, WebK: Brngng QoS to Web servers, HP Labs Techncal Report, HPL--6, May. [9] Y. Dao, N. Gandh, J.L. Hellersten, S. Parekh, and D.M. Tlbury, MIMO control of an Apache Web server: Modelng and controller desgn, Amercan Control Conference,. [] L. Eggert and J. Hedemann, Applcaton-Level dfferentated servces for Web servers, World Wde Web Journal, Vol. 3, No., pp. 33-4, March, 999. [] P. Goyal, X. Guo, and H. Vn, A herarchcal CPU scheduler for multmeda operatng systems, nd USENIX Symposum on Operatng System Desgn and Implementaton, October, 996. [] J.L. Hellersten, Y. Dao, S. Parekh, and D. Tlbury, Feedback Control of Computng Systems, Wley-Interscence, 4. [3] HP Process Resource Manager, [4] HP-UX Workload Manager, [5] IBM Applcaton Workload Manager, [6] IBM Enterprse Workload Manager, [7] M.B. Jones, D. Rosu, and M.-C. Rosu, CPU reservatons and tme constrants: Effcent, predctable schedulng of ndependent actvtes, 6 th ACM Symposum on Operatng Systems Prncples, 997. [8] V. Kanoda and E. Knghtly, Mult-Class latency-bounded Web servces, 8 th IEEE Internatonal Workshop on Qualty of Servce, June,. [9] M. Karlsson, C. Karamanols, and X. Zhu, Trage: Performance solaton and dfferentaton for storage systems, th IEEE Internatonal Workshop on Qualty of Servce, 4. 5

17 [] A. Kamra, V. Msra, and E.M. Nahum, Yaksha: A self-tunng controller for managng the performance of 3-tered web stes, IEEE Internatonal Workshop on Qualty of Servce, June, 4. [] X. Lu, X. Zhu, S. Snghal, and M. Arltt, Adaptve enttlement control of resource parttons on shared servers, 9 th Internatonal Symposum on Integrated Network Management, May, 5. [] L. Ljung, System Identfcaton: Theory for the User (nd Edton), Prentce Hall, 999. [3] C. Lu, T.F. Abdelzaher, J. Stankovc, and S. Son, A feedback control approach for guaranteeng relatve delays n Web servers, IEEE Real-Tme Technology and Applcatons Symposum,. [4] Y. Lu, C. Lu, T. Abdelzaher, and G. Tao, An adaptve control framework for QoS guarantees and ts applcaton to dfferentated cachng servces, IEEE Internatonal Workshop on Qualty of Servce, May,. [5] Matlab System Identfcaton Toolbox, [6] C.W. Mercer, S. Savage, and H. Tokuda, Processor Capacty Reserves: Operatng system support for multmeda applcatons, Internatonal Conference on Multmeda Computng and Systems, 994. [7] R. Rajkumar, K. Juvva, A. Molano, and S. Okawa, Resource Kernels: A resource-centrc approach to real-tme and multmeda systems, ACM Conference on Multmeda Computng and Networkng, 998. [8] D.C. Steere, et al., A feedback-drven proporton allocator for real-rate schedulng, 3 rd USENIX Symposum on Operatng System Desgn and Implementaton, 999. [9] SUN Solars Resource Manager, [3] C. Waldspurger and W. Wehl, Lottery Schedulng: Flexble proportonal-share resource management, st USENIX Symposum on Operatng System Desgn and Implementaton,