Integrating Apache Mesos with Science Gateways via Apache Airavata

Transcription

1 Integrating Apache Mesos with Science Gateways via Apache Airavata Organization: Apache Software Foundation Abstract: Science Gateways federate resources from multiple organizations. Most gateways solve the problem of scheduling across multiple organizations in an ad hoc, per-gateway fashion. HPC environments statically partition their resources to avoid interference between multiple applications. Static partitioning is known to lead to low utilization. This project will evaluate and provide recommendations for how Apache Mesos can be used with Apache Airavata for efficient scheduling of gateway jobs. Apache Mesos has gained significant traction in the last eighteen months and is currently used as the resource manager by several Big Data companies. Mesos enables dynamic partitioning and removes the need to isolate frameworks on separate resources. Mesos combines cluster resources (such as CPUs, memory, and storage) into a shared pool, and efficiently allocates pool components to competing applications based on their fine-grained resource needs. Mesos applications are called frameworks that view the Mesos layer as a cluster-wide, highly-available, fault-tolerant, distributed operating system. This project will investigate the use of Apache Mesos as a gateway level job scheduler with the aim of integrating it with Apache Airavata, which is software that supports general purpose science gateway services. Integration with Apache Mesos will allow Science Gateways to schedule by taking into account a composite view of the distributed resources. The default mapping of jobs to resources in Apache Mesos is based on Dominant Resource Fairness (DRF), which is a generalization of weighted max-min fairness but for multiple resources. The idea driving DRF is that the allocation for a given user should be based on its dominant share (the maximum share that the user has

2 been allocated for any resource). The key benefits of this approach for Big Data applications is that it can support several resource allocation policies such as priority, reservation, deadline based allocation. This project will collect the requirements in terms of priority, reservation, and deadlines, for Science Gateway applications and evaluate to what extent Mesos meets the requirements. Goals This project will investigate how metadata from the various distributed Gateway resources can be collected and provided to Mesos so that it can function as a gateway-level job scheduler. This project will study the characteristics of the typical jobs run by Science Gateways and determine the effect on the queue waiting times with Mesos when the default fairness algorithm is used. A comparison study will be conducted to determine which scientific applications can directly work with Mesos, and which ones may require the Dominant Resource Fairness (DRF) algorithm to be modified. It is to be noted that currently the science gateways do not allow applications to be coscheduled, while Mesos DRF algorithm is designed to allow efficient coscheduling of jobs. Timeline May 1st - May 22nd: Community Bonding Period Project Development Starting from May 25th 1.5 Weeks: Install and understand how Apache Mesos works using a small local cluster. 2 Weeks: Understand and run some applications using Airavata. Visit and meet the Airavata team in Bloomington, Indiana. 1 weeks: collect requirements of applications run via Apache Airavata and match them to how Resource Offers work with Mesos.

3 2 weeks: Develop Framework Allocator, Framework Scheduler and Framework Filter for one class of scientific applications run via Airavata. Choose the applications in consultation with the Airavata team. 1 Week:Mid semester evaluation meetings and demos. 2 weeks: Develop Framework Allocator, Framework Scheduler, and Framework Filter for another class of applications, in consultation with the Airavata team. 2 weeks: Integrate Mesos with a forked-off version of Airavata code base. 1.5 weeks: documentation, testing, and release of the developed software. Involvement After the GSOC project, I continue to work with the Airavata team to focus my Masters project to enhancing Airavata. Personal Information Name: Pankaj Saha University: State University of New York at Binghamton Major: Pursuing Masters in Computer Science (2nd semester) Expected Graduation: May 2015 Technical Skills: Hands on in depth working experience with Java, C, C++, Javascript, Jquery, CSS, HTML, XML,XSL, Oracle and MySQL database. Experience: I have worked on the following academic projects: Genetic algorithms: Conducted experiments on gene pool information to predict the effect of several genetic events such as crossover (single & double) and mutation.

4 Parallel Processing: Analysis of several data parallel and control parallel approaches on a multiprocessor based IBM server(p60 and p90). I have worked in following companies on software maintenance and product development: HCL Technologies(india) - (Aug 2007 to Dec 2009) WIPRO Technologies(india) - (Dec Dec 2010) Oracle(india) - (Dec 2010 to Oct 2013) Cognizant Technologies(india)-(Dec 2013 to June 2014) Contact Information: pankajandsaha@gmail.com, psaha4@binghamton.edu Skype ID: pankajandsaha Availability I am available during the entire GSoC 2015 period. I have no other commitment during this period Typical Working Hours: 11 am to 1 pm, 2pm to 8 pm (5 days a week, amounting to 40 hours per week) Other Commitments: None Any time in Summer where you will be away for more than 2 days: None

5 References Apache Airavarta: Apache Mesos: