Automated Control for Elastic Storage

Transcription

1 Automated Control for Elastic Storage Summarized by Matthew Jablonski George Mason University October 26, 2015 Lim, H. C. and Babu, S. and Chase, J. S. (2010) Automated Control for Elastic Storage The 7th ACM International Conference on Autonomic Computing (ICAC). Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

2 Overview 1 Introduction 2 System Overview Controller Controlling Elastic Storage 3 Components of the Controller Horizontal Scale Controller (HSC) Data Rebalance Controller (DRC) State Machine 4 Implementation 5 Evaluation 6 Discussion and Future Work 7 Conclusion Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

3 Background Problem: Web 2.0 workloads often suffer from flash cloud phenomenon Service demands change over time Towards a solution: Growing commercial interest in automating surge protection and adaptive resource provisioning for dynamic service loads The key elements for wide deployments of infrastructure are now in place through elastic scaling opportunities afforded by cloud computing Components can run in clusters at a range of scales where servers can be added/removed with negligible interuption of service Customers charged for resources as they are utilized Note this is Infrastructure-as-a-Service (IAAS) Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

4 Contributions of This Work Begins by noting that multi-tier Web services generally have common dynamic request demand profiles (see figure in next slide) Provides a method for scaling the storage tier in a data-intensive cluster-based multi-tier service Focuses on policies for elastic scaling based on automated control Uses Proportional Thresholding - integral control technique used to continuously modulate the number of discrete server instances in a cluster (more on this later) Addresses two new challenges on delay and cost of redistributing stored data to set of active instances in storage tier: 1 Actuator lag 2 Interference Implements and evaluates a prototype controller for elastic storage system using Cloudstone generator Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

5 System Overview Target Environment: A multi-tier application service (guest) hosted on virtual server instances rented from an elastic cloud provider. An automated controller uses cloud APIs to acquire and release instances for the guest as needed to serve a dynamic workload. Cloud guest presumed to have SLO based on some performance metric Performance is a function of the resources granted by cloud provider Work targets controlled elasticity to grow and shrink resources, as needed, to meet performance SLOs Grow: Service a growing load more efficiently with more resources Shrink: Use resources more efficiently and save money Focus is on elastic control of the storage tier Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

6 Controller Implement elastic controller process to run on behalf of guest Controller leverages sensors to measure performance from guest and/or cloud provider Based on this info, manages actuators to add storage server to group (join) or remove arbitrary server from group (leave) Actions combine to form control policies that should handle unanticipated workloads and meet SLO Controller can run outside cloud provider to manage control policies across multiple cloud providers Multiple cloud providers present discrete actuator problem Paper provides proportional thresholding technique Each tier exports control API to add/remove instances, which proportional thresholding uses for elastic control of storage tier Solution assumes a mechanism which balances load across servers within tier so request capacity scales Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

7 Controlling Elastic Storage Assumptions about architecture and features of storage tier: 1 Stores data across servers in a way that balances load for reasonable access patterns, and can rebalance in response to join and leave events 2 Replicates data internally for robust availability 3 Storage and IO capacity of system scales linearly with size of active server set Elastic control for cloud infrastructure presents new challenges: 1 Data Rebalancing: Adding a new storage node does not give immediate performance benefits as it does not have persistent data to serve 2 Interference to Guest Service: Data rebalancing consumes resources (CPU, bandwidth, etc.) that can otherwise be used to serve clients 3 Actuator Delays: once rebalancing commands issue, there is a delay before performance improvements observed The Hadoop Distributed File System (HDFS) is used for experiments Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

8 Components of the Controller The elastic controller has three parts: 1 Horizontal Scale Controller (HSC): Grows and shrinks number of storage nodes based on sensor information 2 Data Rebalance Controller (DRC): Controls data transfer for storage tier as it grows and shrinks 3 State Machine: Coordinates actions between HSC and DRC Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

9 Components of HSC Horizontal Scale Controller (HSC) is comprised of: Actuators: Cloud APIs used by HSC to grow/shrink number of storage nodes; each node runs on a separate instance Sensors: HSC bases elastic sizing choices on feedback incorporating one or more performance metric Paper prefers CPU utilization (see diagram on right for why) CPU is easy to measure accurately, is reasonably stable, and correlates to measure of level of service Other metrics could be used Control Policy: Classical integral control used as starting point; it is self-correcting, provably stable, and assumes actuator is continuous (next slide) Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

10 HSC Control Policy Definition of Control Policy u k+1 = u k + K i (y ref y k ) u k and u k+1 : current and new actuator values K i : Integral gain parameter y k andy ref : Current and desired sensor measurements Integral control: adjusts actuator value from previous time step proportionally to the deviation between the current and desired values of the sensor variable in the current time step Actuator u is a continuous variable (discrete actuators can cause instability due to oscillation) Avoiding oscillation: Make y ref a target range, with a pair of high y h and low y l sensor measurements HSC will react if: y h < y k : under-provisioned system y l > y k : over-provisioned system Setting y h and y l statically can lead to resource inefficiency or instability Proportional thresholding combines the above equation with dynamic setting of the target range... Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

11 Proportional Thresholding Source: Lim et. al., Automated Control in Cloud Computing: Challenges and Opportunities, 2009, URL: dcm/teaching/cop6938-cloudcomputing/studentpresentations/automaticcontrol- RamyaPradhan.pptx Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

12 HSC Dynamic Control Policy Property I: Dynamic target range, where y h = y ref and vary y l to vary range Performance impact of adding/removing a node becomes smaller size N of system increases Thus, Property I defines the property range to ensure efficient use of resources: lim N y l = y ref = y h Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

13 workload l = workload h N 1 N y l = f (workload l ) = f (f 1 (y h ) N 1 N ) Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24 HSC Dynamic Control Policy Property II: If system shrinks by one node as y k < y l, then y k+1 y h To capture these properties, we model the relationship between Cloudstone workload and average CPU utilization in storage tier: Empirical Model CPU = f (workload); thus, workload = f 1 (CPU) workload h = f 1 (y h ) represents a workload corresponding to y h = y ref if workload k > workload h, increase node by 1 if workload k < workload l, decrease node by 1 To ensure the property holds, we parameterize the trigger condition for various load levels:

14 Components of DRC - Data Rebalance Controller (DRC) uses rebalancing utility that comes with HDFS to rebalance data as nodes are added/removed - Rebalancing can cause actuator delay and interference - Actuators: Knobs control bandwidth b allocated to the rebalancer this is max and min bandwidth allocated to rebalancer b is selected to control tradeoff between lag and interference interference is proportional to b lag is s/b, where s is amount of data to be copied Issue with research: HDFS is insensitive to b over 3Mb/s, so paper uses it as is and adapts to behavior with DRC control policy Impact is the difference between the average response time with and without rebalancer running See Figure 3 (bandwidth pipeline) and 4 (performance impact) for more discussion (next slide) Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

15 DRC Actuator Figures Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

16 Components of DRC - Sensors and Control Policy: Experiments conducted to model the following relationships using multi-variate regression 1 Time = f t(b, s): Time to completion of rebalancing (Time) as a function of the bandwidth throttle (b) and size of data to be moved (s) 2 Impact = f i (b, l): Impact of rebalancing on service response time (Impact) as a function of bandwidth throttle (b) and per-node workload (l) Goodness of fit (R ) for both models s and l are sensor measurements b is a tuning knob whose value is TBD, represents a tradeoff between Time and Impact - b as cost-based optimization function Cost = f c(time, Impact) = f c(f t(b, s), f i (b, l)) Choose b to optimize Cost given s and l Cost is a weighted sum: Cost = α Time + β Impact α β can be specified by guest as a relative preference of Time over Impact Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

17 State Machine - Mutual dependencies of HSC and DRC must be managed: If HSC adds/removes node, DRC actuator lag results in delay before rebalancing is complete Noise introduced in sensor measurements from one controller while the actions of the other controller are being applied - Elasticity controller has state machine to manage HSC and DRC interactions to prevent bad things (oscillation): Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

18 Implementation Cloudstone Guest Application Front-end application server tier = Glassfish Database tier = PostgreSQL Storage tier = HDFS HDFS distributes files across nodes, HDFS does not ensure storage nodes are request balance but Cloudstone s workload generator accesses them in a uniform distribution that naturally balances requests Modified HDFS to expose bandwidth throttle b; notifications between nodes over RPC Cloud Provider Local ORCA (Duke University) cluster as cloud IAAS provider Provides leasing interface to allow guests to lease resources from Xen-based provider Elasticity Controller Java-based controller used to implement state machine Allows guests to use resource leasing mechanisms via control plug-in module Handler manages adding/removing server instances for storage nodes Gathers CPU utilization metrics; information shared between controllers via RPC; sensor collects every 10 seconds and applies control policy Has two subcontroller modules for HSC and DRC Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

19 Controller Effectiveness - Exp. 1 - First Experiment Interested in evaluating effectiveness and adaptivity of controller to unexpected workload changes - Cloudstone used to subject HDFS to dynamic workloads Small workload (load factor = 1.0) at initialization, increased by factor of 10 at 600 seconds Target response time = 3 seconds which corresponds to 20% CPU utilization Start with 3 HDFS nodes Comparison of static provisioning vs. elastic controller in Figure 6 (on right) By t = 7800 seconds, rebalancing finished and average response time and CPU dropped below target limit due to addition of new HDFS nodes Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

20 Controller Effectiveness - Exp. 2 - Second Experiment Interested in evaluating effectiveness and adaptivity of controller to small workload increase Workload factor of 7.0 at initialization, increased by small factor of 35% at 600 seconds Target response time = 3 seconds which corresponds to 20% CPU utilization Start with 10 HDFS nodes Comparison of static provisioning vs. elastic controller in Figure 7 (on right) One storage node added at 700 th second By t = 2450 seconds, rebalancing finished and average response time and CPU dropped below target limit due to addition of new HDFS nodes Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

21 Resource Efficiency Experiment - Third Experiment Interested in evaluating effectiveness and adaptivity of controller to small workload DECREASE Workload factor of 5.0 at initialization, decreased to factor of 3.5 at 370 seconds Target response time = 3 seconds which corresponds to 20% CPU utilization Initial provisioning with enough nodes to handle workload without SLO violation Comparison of static provisioning vs. elastic controller in Figure 8 (on right) One storage node removed at 420 th second Even with decrease - no SLO violations; static policy incurs unnecessary resource costs Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

22 Comparison of Rebalance Policies - Comparison of rebalance policy vs aggressive and conservative policies Aggressive policy allocates as much bandwidth as possible to data rebalancer (set α to 0) Conservative policy allocates minimal bandwidth for minimal impact on response time during rebalancing (set β to 0) This experiment drives a heavy workload and lets the controller allocate storage nodes based on the policy Aggressive policy finishes rebalancing 5 minutes faster, but impacts response time Conservative policy takes twice as long to complete - not good because it prolongs period of SLO violations Fig 9 (on right) compares policies Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

23 Discussion and Future Work Other Cloud Computer Models: Paper focused on IAAS model, could use SAAS In SAAS, control problem moves to cloud provider Solution for SAAS cloud provider would be used and shared by multiple guests - each with their own performance objectives and data Data Rebalancing: Current HDFS rebalancer limits performance of elasticity controller Dealing with Multiple Actuators: Hierarchical coordination scheme used in this paper to balance two actuators, but there are limitations Investigating alternative policies for coordination between actuators, e.g. concurrent Adapting to Expected Load Changes: Controller is currently reactive in addressing workloads Future work could implement predictive rebalancing Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24

24 Conclusion Presented an automated controller for elastic storage systems in the context of cloud computing To manage discrete actuators, solution uses proportional thresholding to determine size of storage cluster Controller uses a cost-optimization model to manage bandwidth allocated for rebalancing as nodes are added/removed Cost function provided to determine tradeoff of Time vs. Impact for bandwidth allocation Provided an evaluation that showed the controller can dynamically provision coarse-grained resources under unanticipated changes in client workload Controller maintains client SLOs while being resource efficient Summarized by Matthew Jablonski (GMU) Elastic Storage October 26, / 24