CHAPTER 2 LITERATURE REVIEW AND BACKGROUND

Transcription

1 8 CHAPTER 2 LITERATURE REVIEW AND BACKGROUND 2.1 LITERATURE REVIEW Several researches have been carried out in Grid Resource Management and some of the existing research works closely related to this thesis have been discussed. These works have been organised in different subsections based on the techniques they discuss Matchmaking System in Grid Gridway (Huedo et al. 2005) is a Grid metascheduler built over globus middleware. It supports job scheduling across globus based Grid resources, aggregates resource information, and performs keyword based matchmaking to discover suitable resources that matches application requirements. The Monitoring and Discovery System (MDS) implemented in Globus middleware (Foster 2006) is designed to provide a standard interface for publishing and discovering Grid resource status and configuration information in GLUE schema as well as its own schema called MDS schema. The representation of the information aggregated by MDS is used by Grid schedulers and it supports keyword based matchmaking of resource requirements against the available resources.

2 9 Condor (Imamagic et al. 2006) is a cluster manager that supports job submission across several clusters. The Workload Management System (WMS) implemented in glite middleware uses condor for matchmaking user s application requirements against the available resources in the Grid. Condor specifies a standard for describing jobs, workstations, and other resources. However, with this representation, it is not possible to understand the semantic relationship between the available resource information and the requested ones. Gridbus broker (Venugopal 2006) is a resource broker designed to support scheduling of both computational and data Grid applications. However, the resource discovery module implemented in the Gridbus broker does not support semantic description and discovery of Grid resources, and it uses the Globus Grid Index Information Services (GIIS) (Foster 2006) or Grid Market Directory (GMD) (Yu 2006) to gather Grid resource information Semantic Grid Projects The content based information search and retrieval always poses a great challenge in terms of accuracy of results and amount of computations need to be performed. Chunqiang et al (2003) presented a decentralized peer to peer information retrieval system. It distributes document indices through the peer to peer network based on document semantics generated by Latent Semantic Indexing. Rinaldi, A.M. (2009) proposed a novel metric to measure the semantic relatedness between words. This approach is based on ontologies represented using a general knowledge base for dynamically building a semantic network. With this approach, it is possible to obtain an efficient strategy to rank digital documents from the Internet according to the user s interest domain.

3 10 Generally Ontology mapping is to find semantic correspondences between similar elements of different ontologies. Ming M (2007) proposed a generic and scalable ontology mapping approach based on propagation theory, information retrieval and artificial intelligence model. The approach utilizes both linguistic and structural information, measures the similarity of different elements of ontologies in a vector space model, and deals with constraints using the interactive activation network. In all the above works, the focus is solely on similar information retrieval in clusters of documents or documents present across Internet. The approach differs when it comes to Grid environment where information is often computational resource metadata. In the following sections, appropriate research works already carried out in the field of semantic Grid has been highlighted. A reference architecture that extends OGSA to support the explicit handling of semantics is proposed in the literature (Corcho et al. 2006). It defines the associated knowledge services to support a spectrum of service capabilities, and a model, the capabilities and the mechanisms for the semantic Grid. It extends the capabilities of Grid middleware to include semantic provisioning services and semantically aware Grid services. An extensive survey on semantic web and semantic Grid technologies is presented by Murphy et al (2008). It explores the potential and predicted impact of emerging technologies on collaborative industrial design, and outlines a reference architecture that can be implemented to meet specific requirements of industrial business. Wroe et al. (2004) used ontologies to describe and select web-based services used in the Life Sciences whereas Chen et al (2003) used ontologies to guide aeronautical engineers to select and configure Matlab scripts. Mirza Pahlevi Said and Isao Kojima (2009) proposed Semantic Monitoring and

4 11 Discovery System (S-MDS) built on the top of WS-Resource Framework. Here they extended MDS4 of globus middleware to support RDF/OWL descriptions and accept query from SPARQL inference engine. Further, Said and Kojima (2008) provided extension to S-MDS by incorporating rule based approach in conjunction with inference capability. However, the S-MDS approach is tightly coupled with underlying globus middleware, and the authors did not provide information about support for other middleware. Harth et al. (2004) provided an ontology based Matchmaker Service that supports dynamic resource discovery and resource descriptions. However, the request is expressed using request ontology and hence there is a need to compile the user request as ontology descriptions. Pernas and Dantas (2005) used ontology for describing Grid resources thereby enabling semantic discovery. They define concept-classification tree to create resource ontology. It uses protégé axioms to discover resources semantically. Similarly, Tangmunarunkit et al. (2003) exploited semantic web technology for resource matching in the Grid. They also utilized background ontology and matchmaking rules to perform matchmaking of the resources. Zhang and Wang (2008) proposed a semantic Grid infrastructure for e-governance applications. They stressed the necessity of such infrastructure for management of e-governance resources in the form of services across virtual government agencies. Yin et al. (2009) presented an agent based semantic Grid for collaboration across virtual enterprises by forming a super peer network over the semantic Grid. Hartung et al. (2008) provided a platform and a metamodel that allows the user to create and edit Grid related metadata present across the Grid infrastructure. They also implemented this platform in German D-Grid. However, our focus is to devise a semantic system to complement Grid scheduling decisions and hence must be adaptable to integrate with Grid schedulers. Further, it requires a different approach to represent and manage distributed computational Grid resources.

5 12 Dumitrescu et al. (2005) proposed a model for facilitating resource usage policy based allocation in Grids. They implemented the model and integrated it with maui scheduling mechanism. Feng et al. (2007) proposed a mechanism to express resource usage policy and its enforcement in Grid. It uses request response paradigm based on XACML and introduces relevant attributes to express and enforce Grid resource usage policies Virtualization in Grid Keahey et al. (2005 and 2006) introduced the concept virtual workspace (VW) that aims to provide a customizable and controllable remote job execution environment for Grid. Virtual Workspace supports unmanned installation of legacy applications, which can effectively reduce the deployment time. By allocating and enforcing resources required by jobs with different priorities, virtual machine can realize fine-grained resource provision, including CPU, memory and network band Freeman et al. (2006) addressed the management issues arising from division of labor. The abstractions and tools that allow clients to dynamically configure deploy and manage required execution environments in application-independent ways as well as to negotiate enforceable resource allocations for the execution of these environments. However, virtual workspace does not support standard Grid authentication and authorization mechanisms, and it uses pre-configured virtual machine images for workspace creation. Zhang et al. (2006) extended the virtual workspace to encompass the notation of a cluster. They described the extensions needed for workspace definition, protocols and other services to create a virtual cluster for authorized Grid clients. They have conducted several experiments to analyze time complexity for virtual cluster creation and deployment of required execution environment. Sotomayor et al. (2008) developed a model in which a virtual workspace is associated with a well-defined resource allocation

6 13 strategy to enable accurate and efficient creation or timely creation of virtual workspaces. It allows a remote client to create virtual resource securely using standard web service based protocols and services. Further, Sotomayor (2007) addressed methods to minimize the overhead introduced by having to deploy virtual machine images before the start of a resource leasing Grid Interoperability Initiatives Grid Interoperation Now (GIN) (Reidel 2008) Community Group (CG) of the Open Grid Forum (OGF) is a group of representatives from world-wide Grid, e-science infrastructure and other national Grid initiatives. Its aim is to organize and manage interoperation efforts among different production Grid infrastructure. GIN-CG is working on different areas of Grid middleware through different working groups like GIN-INFO, GIN-DATA and GIN-AUTH and it has practically demonstrated Grid interoperation between Grid infrastructure like EGEE, OSG, NAREGI, and DEISA. OMII-Europe (2006) project has been funded by European Union to explore interoperability among several heterogeneous Grid infrastructure. OMII-Europe follows the standard based approach to achieve interoperability. It has selected specific components like Basic Execution Service (BES), Accounting Service, Database service and VOMS services and makes them available across middleware platforms. In EU-CHINA (2006) project, adapter based approach is implemented to achieve interoperability between China and European Grid. China Grid is using Grid Operating System (GOS) middleware and European Grid is using glite as its middleware. Different adapters have been used to interoperate various components of middleware, like JSDL to JDL and JDL to JSDL adapter for job submission.

7 14 The Australian Research Council has funded the GRIDS lab of the University of Melbourne towards Gridbus project. The GRIDS lab has developed Gridbus Broker which schedules jobs to Grid resources across middleware such as Globus 4.0, Globus 2.4 and also to local schedulers such as SGE, PBS etc. Marcos Dias et al. (2007) described the InterGrid project that focuses on internetworking heterogeneous Grids across the globe. This project proposes an InterGrid gateway peering arrangement at every Grid. Also, it deals with the formation and management of virtual organization across the Grids. The glite middleware distribution developed by the Enabling Grids for E-sciencE (EGEE) project of Europe is now deployed by about 80 percent of the sites connected to the EGEE infrastructure, making it the main middleware distribution used in production. glite makes use of components from other Grid middleware projects and is designed as a modular system to allow users to tailor the system to their specific needs by deploying the services they require, rather than having to use the system as a whole (glite 2006). Gridway Metascheduler, developed by Huedo et al. (2005), achieves interoperability between glite and Globus middleware. Initially, it was developed as a metascheduler for job scheduling across Globus Grids. Exploiting the Globus 2.4 characteristics of glite, it is also possible to schedule jobs across glite based computing elements but one has to bypass WMS component of glite middleware. Commodity Grid (CoG) Kit (Gregor 2001) enables Grid users, Grid application developers, and Grid administrators for rapid Grid application development. It provides the implementation of Java-based GSI, GridFTP,

8 15 myproxy, GRAM client implementations and capable of submitting job across several versions of Globus middleware but not other middleware Summary In all the above middleware and metascheduler, the matchmaking is carried out based on the keywords present in the request. Further, several approaches have been proposed for semantic based Grid resource management, an extensive analysis in devising a mechanism for resource representation, selection of reasoning engine and integration of such component with Grid scheduler is still missing. The work presented in this thesis can bypass the conventional matchmaking system and performs semantic based resource discovery. It can also be easily integrated with Grid metaschedulers and two such integrations are described in this thesis. The semantic component has also been extended to consider resource usage policies and their representation in the ontology. The semantic component has been further extended to determine potential resources that support virtualization in them. 2.2 BACKGROUND According to Foster et al. (2001), Grid Technologies support the sharing and coordinated use of diverse resources in dynamic virtual organization that is the creation, from geographically and organizationally distributed components, of virtual computing systems that are sufficiently integrated to deliver desired QoS. Foster et al. (2002) defined the essential properties of Grids and also introduced the key requirements for protocols and services, distinguishing among connectivity protocols concerned with communication and authentication, resource protocols concerned with negotiating access to individual resources and collective protocols and services concerned with the coordinated use of multiple resources. Grid

9 16 Concepts and technologies were first developed to enable resource sharing within far-flung scientific collaborations. The establishment, management, and exploitation of such dynamic, cross-organizational sharing relationships require new technology and architecture. Hence, the Grid Architecture proposed by Foster et al. (2001) identifies fundamental system components, specifies the purpose and function of these components and indicates how these components interact with one another Grid Standards Grid is a complex system in which many services are collaborating to solve a complex scientific application and hence they interact more frequently. For example, the job management service might consult the resource discovery service to find computational resources that match the job s requirements. With so many services and so many interactions between them, there exists the potential for chaos. It would be very difficult to get all the different pieces of software to work together. The solution is Standardization: define a common interface for each type of service. The Open Grid Services Architecture (OGSA), developed by The Global Grid Forum (GGF), aims to define a common, standard, and open architecture for Grid-based applications. The goal of OGSA is to standardize practically all the services one commonly finds in a Grid application (job management services, resource management services, security services etc.) by specifying a set of standard interfaces for these services. While creating this new architecture, the developer realized that they needed to choose some sort of distributed middleware on which to base the architecture and hence Web Service technology was chosen as base for OGSA. However, although the Web Services Architecture was certainly the best option, it still did not meet one of OGSA s most important requirements; the underlying middleware had to be stateful. In theory, Web Services can be

10 17 either stateless or stateful, they are usually stateless and there is no standard way of making them stateful. To overcome this limitation, OASIS developed a specification called Web Services Resource Framework (WSRF) that specifies how a Web Services can be made a stateful service. WSRF is a joint effort by the Grid and Web Services communities and hence fits well inside the whole Web Services Architecture. WSRF provides the stateful services that OGSA needs. In other words, while OGSA is the architecture, WSRF is the infrastructure on which that architecture is built on (Sotomayor et al. 2005) Grid Middleware Grid is an infrastructure that involves the integrated and collaborative use of computers, networks, databases and scientific instruments owned and managed by multiple organizations. Grid applications often involve large amount of data and/or computing resources that require secure resource sharing across organizational boundaries. This makes Grid application management and deployment a complex undertaking. Grid middleware provides users with seamless computing ability and uniform access to resources in the heterogeneous Grid environment. Several software toolkits and systems have been developed, most of which are results of academic research projects, all over the world. In order to provide users with a seamless computing environment, the Grid middleware systems need to solve several challenges originating from the inherent features of the Grid. One of the main challenges is the heterogeneity in Grid environments, which results from the multiplicity of heterogeneous resources and the vast range of technologies encompassed by Grid. Another challenge involves the multiple administrative domains and autonomy issues because of geographically distributed Grid resources across multiple administrative domains and owned by different organizations. Other

11 18 challenges include scalability (problem of performance degradation as the size of Grids increases) and dynamicity/ adaptability (problem of resource failing is high). Middleware systems must tailor their behaviour dynamically and use the available resources and services efficiently and effectively. A tremendous amount of efforts has been spent in the design and development of middleware system for enabling computational Grids. Globus Alliance, a community of organizations and individuals developing fundamental technologies behind the Grid, along with many other communities, developed Globus Toolkit. It is an open source software toolkit used for building Grid systems and applications. Currently, many organizations and research institutes are using this middleware for their Grid operations. The glite middleware distribution developed by the Enabling Grids for E-sciencE (EGEE) project of Europe has now been deployed by more than 80 percent of the sites connected to the EGEE infrastructure, making it the main middleware distribution used in production. glite makes use of components from Globus middleware and is designed as a modular system to tailor the system to their specific needs by deploying the services they require, rather than having to use the system as a whole. Subsequently, several middleware came up such as Unicore, Gridbus Broker, etc and a very good discussion and comparison of popular Grid middleware can be found in the literature (Asadzadeh P et al. 2005) Semantic Web Technologies for Grid The emergence of Semantic Web concepts offers sophisticated tools and mechanisms to describe a particular domain of interest, and querying and rule based languages for information retrieval. They provided a new dimension to describing web content for efficient information retrieval and solve interoperability issues in collaborative computing. Semantic web, proposed by Berners Lee et al. (2001), is a collection of information described

12 19 in a hierarchical manner so that they can easily understandable by the computers thereby bringing human and computer, a step be closer. Such explicit specification of conceptualization is called Ontology. Ontologies facilitate the interchange and integration of heterogeneous information. They are used to capture knowledge about some domain of interest. Ontology describes the concepts in the domain and also the relationships that hold between those concepts a.k.a. classes (OWL 2004). It also allows to model real time entities of a domain as instances of classes. The ontology together with instances forms knowledge base of that domain. Semantic web uses Resource Description Framework (RDF 2004) over XML based representation of data as a standard data interchange format to create ontologies. Unlike traditional XML, RDF schemas allow to provide semantic relationship between the two concepts such as Cluster is a type of computingresource where Cluster and computingresource are referred to as classes and a type of is referred to as property. With the wide adoption of RDF by the semantic web communities, RDF continued to grow. The most recent development in standard ontology language is Web Ontology Language (OWL) from the World Wide Web Consortium. OWL is developed as a vocabulary extension to RDF and is capable of representing several relationships such as disjoint and cardinality. Classes and Properties described using ontology languages are identified as URIs, and can easily be accessible through internet which enables flexible sharing and reusing them. (Chen et al 2003). To facilitate creation of ontologies, several ontology editors were developed. Protégé (Matthew Horridge 2004) is one such editor that supports creation of ontology using OWL language and is widely used for developing ontology based applications. An ontology description of a particular domain supports semantic or context based information retrieval especially in Artificial Intelligence (AI) applications where semantic relationships between

13 20 concepts are established using description logic. There are inference engines such as Algernon (Michael Hewitt 2005) that provides versatile DL based querying mechanism for knowledge retrieval from ontology knowledge base. Further, using rule languages like SWRL (SWRL 2004), allows even more sophisticated information retrieval mechanisms. Such developments in semantic web technologies tempted Grid researchers to integrate them with Grid environment to address, especially Grid resource and service management. This initiative has led to the notion of Semantic Grid that suggests machine processable descriptions of particular domain to maximize the potential for sharing and reuse, and thus better enabling computers and people to work in cooperation (Murphy 2008). In computational Grid environment, users and software agents should be able to discover, invoke, compose and monitor Grid nodes offering required services and possess particular properties. Describing these resources and services using the concepts of semantic web technology supports understanding the meaning of the request. This would help the Grid scheduler to identify computational resources that are not exactly matching with the application requirements but can still execute the application. For instance, an application has been compiled for Linux operating system and submits a request to the Grid scheduler for such resources in the Grid. If such resources are not available, the Grid scheduler can suggest resources with Fedora operating system for executing this application. This is for the simplest case of scheduling decision. In real environment, there would be huge amount of metadata that need to be processed for making versatile scheduling decisions. In such situation, conventional description of metadata supported by current middleware components offers limited flexibility with respect to knowledge retrieval. For instance, a Globus 2.4 middleware based Grid aggregates Grid

14 21 resource information and stores in volatile LDAP server. The description of resource is found as object and corresponding attribute. In the advanced versions of Globus such as Globus 4.x, information is stored as XML documents. High level services such as Resource management services, and Resource brokering services uses such conventional description of resources and make appropriate scheduling decision such as resource discovery, verifying resource usage policies and application scheduling to selected resources. They can compare the characteristics requested by the applications against the available resource configuration based on keyword present in the request. Hence, such discovery offers limited flexibility and restricted amount of information retrieval. However, if the Grid metadata are described using the concepts of ontology, the Grid scheduler can employ semantic based information retrieval mechanism using appropriate inference engine. Though OWL increases the level of expressiveness with a richer vocabulary, it is primarily used to describe content. Thus, it is very well suitable for describing semantic relationship between different types of computational resources and not describing service. In order to describe the semantics of services to improve their platform- and organizationindependent interoperability over the Internet, an upper ontology for the description of web services named OWL for Services (OWL-S) has been introduced in the literature (OWL-S 2004). The service descriptions using OWL-S can increase the ability of computer systems to find eligible services autonomously. Basically a service provider describes his services in an OWL- S compliant ontology and a service requester queries for services expressing their requirements. In this scenario, matching service descriptions of advertisements with requirements has the purpose to select a suitable service among a set of available ones (Micheal et al. 2004).

15 Policy Management System Large scale Grid environments are generally complex as they involve multi institutions and the resources in the Grid are autonomous in nature. Hence, it is not possible to set a single usage policy across the Grid. Further, the resources belong to different organizations and have their own policies. Managing these resources and coordinating them in solving a typical Grid application is a challenging task. For example, in Garuda Grid (India Grid), different organizations including academic institutions contribute resources following their own policies. They have their own restrictions with respect to, for example, number of hours they can contribute their resource to the Grid. In such an environment, we need a mechanism that allows the resource provider to express and enforce such usage policies. Also, the Grid scheduler or Grid coordinator must be able to verify these policies against the user s demands before submitting the job. Resource policies generally comprise access policies and usage policies. Access policy is used to express authentication and authorization of users submitting job to the particular resource. It controls the accessibility level of the user so that he will be allowed to utilize the resources for what he is eligible for. The resource usage policy deals with the explicit policy requirements such as duration of availability of the resources, quality of services offered by the resources, and also the capability of the resources with respect to specific job. With these observations and inferences, the development of knowledge layer that supports description and discovery of Grid services and Grid computational resources required different approach as the way it is handled and discovered by conventional Grid middleware is different. Further, the policy matching system proposed in this thesis would address the

16 23 expression of resource usage policies, and its implementation with a grid scheduler. In the next chapter, a knowledge layer component that supports description as well as discovery of Grid Services is explained. Grid Service providers register their services and their corresponding WSDL documents are then converted to OWL-S descriptions using WSDL2OWLS utility. These descriptions are maintained in a semantic service repository. The Service matchmaking system obtains requested capabilities of a service from the user. It then compares the capabilities against the available service descriptions based on the semantics present in the request and the advertisements. This approach eliminates unwanted and /or irrelevant services from the resultant services.