Sphinx: A Scheduling Middleware for Data Intensive Applications on a Grid

Transcription

1 Sphinx: A Scheduling Middleware for Data Intensive Applications on a Grid Richard Cavanaugh University of Florida Collaborators: Janguk In, Sanjay Ranka, Paul Avery, Laukik Chitnis, Gregory Graham (FNAL), Pradeep Padala, Rajendra Vippagunta, Xing Yan 1

2 The Problem of Grid Scheduling o Decentralised ownership o No one controls the grid o Heterogeneous composition o Difficult to guarantee execution environments o Dynamic availability of resources o Ubiquitous monitoring infrastructure needed o Complex policies o Issues of trust o Lack of accounting infrastructure o May change with time o Information gathering and processing is critical! 2

3 A Real Life Example o Merge two grids into a single multi-vo inter-grid o How to ensure that o neither VO is harmed? o both VOs actually benefit? o there are answers to questions like: LBL Caltech UW UC UI ANL o With what probability will my job be scheduled and complete before my conference deadline? UCSD UTA OU SMU Rice IU UM FNAL BU MIT BNL UF o Clear need for a scheduling middleware! 3

4 Some Requirements for Effective Grid Scheduling o Information requirements o Past & future dependencies of the application o Persistent storage of workflows o Resource usage estimation o Policies o Expected to vary slowly over time o Global views of job descriptions o Request Tracking and Usage Statistics o State information important o Resource Properties and Status o Expected to vary slowly with time o Grid weather o Latency measurement important o Replica management o System requirements o Distributed, fault-tolerant scheduling o Customisability o Interoperability with other scheduling systems o Quality of Service 4

5 Incorporate Requirements into a Framework VDT Client o Assume the GriPhyN Virtual Data Toolkit: o Client (request/job submission) o Globus clients o Condor-G/DAGMan o Chimera Virtual Data System o Server (resource gatekeeper) o Globus services o RLS (Replica Location Service) o MonALISA Monitoring Service o etc VDT Server??? VDT Server VDT Server 5

6 Incorporate Requirements o o into a Framework Framework design principles: o Information driven o Flexible client-server model o General, but pragmatic and simple o Implement now; learn; extend over time o Avoid adding middleware requirements on grid resources o Take what is offered! Assume the GriPhyN Virtual Data Toolkit: o Client (request/job submission) o Globus clients o Condor-G/DAGMan o Chimera Virtual Data System o Server (resource gatekeeper) o Globus services o RLS (Replica Location Service) o MonALISA Monitoring Service o etc VDT Client VDT Server? VDT Server Scheduler VDT Server 6

7 The Sphinx Framework Sphinx Client Chimera Virtual Data System Data Warehouse Request Processing Data Management Condor-G/DAGMan VDT Client Globus Resource Sphinx Server Information Gathering Replica Location Service MonALISA Monitoring Service VDT Server Site 7

8 Sphinx Scheduling Server o Functions as the Nerve Centre o Data Warehouse o Policies, Account Information, Grid Weather, Resource Properties and Status, Request Tracking, Workflows, etc o Control Process o Finite State Machine o Different modules modify jobs, graphs, workflows, etc and change their state o Flexible o Extensible Data Warehouse Sphinx Server Control Process Message Interface Graph Reducer Job Predictor Graph Predictor Job Admission Control Graph Admission Control Graph Data Planner Job Execution Planner Graph Tracker Data Management Information Gatherer 8

9 Policy Constraints o Defined by Resource Providers o Actual grid sites (resource centres) o VO management o Applied to Request Submitters o VO, group, user, or even a proxy request (e.g. workflow) o Valid over a Period of Time o Can be dynamic (e.g. periodic) or constant o Global accounting and book-keeping is necessary 9

10 Quality of Service o For grid computing to become economically viable, a Quality of Service is needed o Can the grid possibly handle my request within my required time window? o If not, why not? When might it be able to accommodate such a request? o If yes, with what probability? o But, grid computing today typically: o Relies on a greedy job placement strategies o Works well in a resource rich (user poor) environment o Assumes no correlation between job placement choices o Provides no QoS 10

11 Quality of Service o As a grid becomes resource limited, o QoS becomes even more important! o greedy strategies may not be a good choice o Strong correlation between job placement choices o Sphinx is designed to provide QoS through time dependent, global views of o Requests (workflows, jobs, allocation, etc) o Policies o Resources 11

12 Resource Usage Estimation o User Requirements o Upper limits on CPU, memory, storage, bandwidth usage o Domain Specific Knowledge o Applications are often known to depend logarithmically, linearly, etc on certain input parameters, data size or type o Historical Estimates o Record the performance of all applications o Statistically estimate resource usage within some confidence level 12

13 Data Management o Smart Replication: o Graph based o Examine and insert replication nodes to minimise overall completion time o Distribute and collect required data o Particularly useful in data parallelism o Hot Spot based o Monitor current and historical data access patterns and replicate to optimise future access 13

14 Data Management o Smart Replication: o Graph based o Examine and insert replication nodes to minimise overall completion time o Distribute and collect required data o Particularly useful in data parallelism o Hot Spot based o Monitor current and historical data access patterns and replicate to optimise future access 14

15 Early Sphinx Prototype Test Results o Simple sanity checks o 120 canonical virtual data workflows submitted to US-CMS Grid o Round-robin strategy o Equally distribute work to all sites o Upper-limit strategy o Makes use of global information (site capacity) o Throttle jobs using just-in-time planning o 40% better throughput (given grid topology) Distribution of jobs across the US-CM S Grid Testbed DGT IGT UCSD CALTECH Sites Average time to complete a job 3250 Round Robin Upper Limit Round Robi n Upper Limit o Conclusion: Prototype is working! DGT IGT UCSD CALTECH Sites 15

16 Some Current and Future Activities o Policy Based Scheduling oquality of Service o Graph Partitioning o Data Parallelism o Prediction Module o Useful Views and Fusion of Monitoring Data 16

17 Conclusions o Scheduling on a grid has unique requirements o Information o System o Decisions based on global views providing a Quality of Service are important o Particularly in a resource limited environment o Sphinx is an extensible, flexible grid middleware which o Already implements many required features for effective global scheduling o Provides an excellent workbench for future activities! 17