Problems for Resource Brokering in Large and Dynamic Grid Environments Cătălin L. Dumitrescu Computer Science Department The University of Chicago cldumitr@cs.uchicago.edu (currently at TU Delft) Kindly presented by Ana Lucia Varbanescu (TU Delft)
Introduction Grids provide a means for harnessing the computational and storage power of widely distributed collections of resources Resources are provided under agreements that appear and vanish with a high frequency due to the environment's scale We distinguish between two types of participating entities: Providers: want to express uslas about resource availabilities Consumers: want to interpret uslas published by providers Usage Service Level Agreements (uslas): sharing rules about how a resource is used after access was granted C. Dumitrescu, M. Wilde, I. Foster, A Model for Usage Policy based Resource Sharing in VOs, Policy Workshop 2005, Stockholm, Sweden
What Is Grid Brokering? Resources in Grids are characterized by: independence from global control support for attribute based search availability governed by various local uslas Grid Brokering: (a) the automated discovery and negotiation of usage agreements based on the resource attributes and local uslas (b) the identification of the extended execution environment based on a Grid scheduler Our focus: the identification of requirements to support usla expression, publication, discovery, interpretation, enforcement, and verification provisioning the design ingredients for building a scalable distributed brokering service
Presentation Overview Introduction / What is Grid Brokering? An Overview of the Brokering Key Requirements Environment Examples The Three Requirements Illustration in a Concrete Case Some Background DI GRUBER Framework Enhanced DI GRUBER Description Performance Results Metrics Accuracy with Brokering Network Mesh Connectivity DP Scheduling Comparison with a P2P solution Conclusions
Environment Examples Open Science Grid (OSG, previously known as Grid3): multi VO environment that sustains production level services comprises more than 30 sites and 4500 CPUs, over 1300 simultaneous jobs and more than 2 TB/day aggregate data traffic participating sites are the main resource providers under various uslas LHC Computing Project (LCG): data storage and analysis infrastructure for the entire high energy physics community that will use the LHC (Large Hadron Collider) >100,000 CPUs required for data processing >5000 scientists in approx 500 research institutes worldwide focuses on developing and deploying computing services based on a Grid model requires the management of acquisition, installation, and capacity planning for a large number of commodity hardware components
Identified Brokering Requirements Support for brokering of dynamic and numerous resources Adequate accuracy independent of the infrastructure Fault tolerance
Dynamic and numerous resources Communities, providers and VOs might join a Grid environment for short time intervals (days to weeks) to quickly solve problems When the environment is large (> 1k providers and 10k consumers), changes may occur every hour Thus, rapid propagation is required for information about available resources new administrative policies about how resources are available The brokering infrastructure must be: distributed and not rely on a single decision point (DP) scalable, due to the size of the environment
Adequate Accuracy For a distributed infrastructure, several operations have to be considered, such as propagation, reconciliation and removal These operations may occur whenever new decisions are performed and new resources join or leave the environment The entire brokering infrastructure must become aware of these changes in a timely fashion manner
Fault Tolerance A client expects to perform scheduling operations over the Grid even when some failures occur When many clients perform queries, the brokering infrastructure must be able to cope adequately with the increased request load The employment of an adequate fault tolerance strategy is important to the client
Presentation Overview Introduction / What is Grid Brokering? An Overview of the Brokering Key Requirements Environment Examples The Three Requirements Illustration in a Concrete Case Some Background DI GRUBER Framework Enhanced DI GRUBER Description Performance Results Metrics Accuracy with Brokering Network Mesh Connectivity DP Scheduling Comparison with a P2P solution Conclusions
Illustration Settings What brokers? GRUBER and DI GRUBER grid brokers Where? Planet Lab nodes What size? Brokering testing for a grid 10x bigger than today s OSG/Grid3
GRUBER A Bit of History Started in Grid3 context as a monitoring tool Evolved in a site recommendation engine Later got enhanced with capabilities for: Enforcement components (Queue Managers) Complex uslas and specification interfaces Distributed capabilities (the start of DI GRUBER)
GRUBER: A Grid Broker Implements the brokering functionalities required for steering workloads in a distributed environment based on uslas Implemented as a Grid Web Service using Globus technology Does not perform job submission by itself, but can be used in conjunction with various grid job submission infrastructures Interfaced with Euryale and Pegasus (largely used on OSG/Grid3) for job execution C. Dumitrescu, I. Foster, GRUBER: A Grid Resource usla-based Broker, Euro-Par 2005, Lisboa, Portugal
GRUBER in a Nutshell C. Dumitrescu, I. Foster, GRUBER: A Grid Resource usla-based Broker, Euro-Par 2005, Lisboa, Portugal
DI GRUBER Framework A single usla management decision point providing brokering decisions over hundreds/thousands of jobs and sites => bottleneck Distributed GRUBER (DI GRUBER): extends GRUBER with support for distributed brokering provides a scalable management service with the same functionalities as GRUBER but in a distributed approach allows multiple decision points to coexist and cooperate in real time implemented as a two layer resource brokering service C. Dumitrescu, I. Raicu, I. Foster, DI-GRUBER: A Distributed Brokering Infrastructure, Super-Computing 2006, Seattle, USA
DI GRUBER Operational Mode Decision points (DPs / PEPs) are responsible for executing uslas: gather monitoring metrics and other information relevant to their operations use this information to steer resource allocations as specified by the uslas Two types of PEPs: Site PEPs VO PEPs C. Dumitrescu, I. Raicu, I. Foster, DI-GRUBER: A Distributed Brokering Infrastructure, Super-Computing 2006, Seattle, USA
DI GRUBER Novelty Capable to handle: Sites with RMs & VOs Multiple submission hosts based on GTx technology Model usage allocations (uslas) at several levels Large grids with many resources and users Dynamic VO bootstrap Capacity to: Collect monitoring metrics from a grid Make various decisions based on this information Enforce complex uslas by various means
DI GRUBER Enhancements Transparent Decision Point (DP) Bootstrapping: DPs register with WS Index Service at startup and unregister when leave Transparent Client Scheduling: DPs and clients use the registry to discover the infrastructure clients are scheduled to DPs based on a least used (LU) strategy whenever a DP stops responding, its clients are re scheduled Failure Handling: based on a fault signaling mechanism: every time a client fails to communicate with a DP, it sends a request fault based on faults and a specific policy new DP are started (WS GRAM invocation)
Presentation Overview Introduction / What is Grid Brokering? An Overview of the Brokering Key Requirements Environment Examples The Three Requirements Illustration in a Concrete Case Some Background DI GRUBER Framework Enhanced DI GRUBER Description Performance Results Metrics Accuracy with Brokering Network Mesh Connectivity DP Scheduling Comparison with a P2P solution Conclusions
Metrics for Performance Analysis Response (RTi = individual response for job i; N = number of jobs): Response = Σ i=1..n RT i / N Throughput: defined as the number of requests completed successfully by the service per time unit (second) Accuracy (SAi = ratio of free resources at the selected site for the job i to total free resources in the grid): Accuracy = Σ i=1..n (SA i ) / N
Experimental Settings Emulated environment: 300 sites, approx 40,000 nodes (a grid 10 x today s OSG/Grid3) based on OSG/Grid3 configuration in terms of CPU counts and network connectivity 120 submission hosts Synthetic workloads in which jobs are: arriving with a rate of 1 job/s at each submission host submitted by a submission host to a site, but queued or held running at a site and completed Composite workloads that overlay work for 60VOs, 10 groups/vo A submission hosts maintains connection with a DP, selected w/ a given policy (LU / random) The experiment duration is 3600s (1 hour)
Accuracy vs. Mesh Connectivity DP connectivity: 100% connectivity 50% connectivity 25% connectivity Accuracy drops linearly with connectivity degree of each DP Utilization is low because jobs do not start all in the beginning Table. DI GRUBER Accuracy Performance with Mesh Connectivity Connectivity Util Accuracy All 35% 75% One half 27% 62% One fourth 20% 55% Requests Handled by GRUBER Total Request All 41% 68% One half 30% 60% One fourth 21% 50%
Throughput (random vs. LU) 20 18 Throughput (run10) Throughput (run3) Throughput (queries / sec 10 9 8 7 6 5 Throughput (10DP) Throughput (3DP) Throughput (1DP) Throughput (queries / sec) 16 14 12 10 8 6 4 3 2 1 0 0 10 20 30 40 50 60 70 80 90 100 110 120 Load (# of concurent clients) 4 2 0 0 10 20 30 40 50 60 70 80 90 100
Response (random vs. LU) Response Time (sec) 100 90 80 70 Response Time (1DP) Response Time (3DP) Response Time (10DP) 60 50 40 Response Time (se 60 50 40 Response Time (run3) Response Time (run10) 30 30 20 20 10 10 0 0 10 20 30 40 50 60 70 80 90 100 110 120 0 0 10 20 30 40 50 60 70 80 90 100 Load (# of concurent clients)
DP Scheduling and Gains On average, we find: modest improvements for 3 decision points (19% higher throughput and 8% lower response time) significant improvements for 10 decision points (68% higher throughput and 70% lower response times).
Comparison with a P2P System PAST free distributed lookup service (based on PASTRY) Response: 90% smaller compared to the 3 DP DI GRUBER 50% smaller compared to the 10 DP DI GRUBER P2P has higher variance in the beginning (the stabilization of the P2P network) Throughput: 7 times higher compared to the 3 DP DI GRUBER Only 1.6 times higher compared to the 10 DP DI GRUBER The message lost rate for P2P network is much higher compared with the DI GRUBER's one
Conclusions What are the key requirements an already existing management infrastructure should meet in order to support large and dynamic Grid environments? Our experimental results show in a concrete case the importance of this question: the brokering accuracy decreases almost linearly with the loss of connectivity for a single decision point instance the performance of the system almost doubles in the 10 decision points case due to the better repartition of the clients
Summary We have identified three essential requirements a brokering infrastructure must meet when deployed in large and dynamic Grids We have analysed their feasibility and performance improvements in two novel metrics for DI GRUBER: brokering accuracy as a function of infrastructure components connectivity performance gains when using automated decision point scheduling for the clients We have compared the performance of this brokering solution with a P2P based system for file management
Acknowledgements Ian Foster Michael Wilde Jens S. Vöckler Yong Zhao Ioan Raicu Luiz Meyer ivdgl / GriPhyN Teams
The end Thank you for your attention! For more in depth details, I (the speaker) recommend: Read the paper (that s what I did ) Ask me I will try to answer I will FWD your questions to Mr. Dumitrescu Skip the speaker, send an e mail: c.l.dumitrescu@tudelft.nl