Grid Middleware and Globus Toolkit Architecture

Transcription

1 Grid Middleware and Globus Toolkit Architecture Lisa Childers Argonne National Laboratory University of Chicago

2 2 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach

3 3 A fundamental problem of Grid Computing: supporting virtual organizations

4 4 Virtual Organizations

5 5 Virtual Organizations Distributed resources and people

6 6 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains

7 7 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals VO-A VO-B

8 8 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals Dynamic VO-A VO-B

9 9 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals Dynamic Fault tolerant VO-A VO-B

10 10 movie time! (a short tour of the accelerator at CEN)

11 11 10,000 Encyclopedia Britannica's per second

12 12 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach

13 Support for Heterogeneous Environments 13 esource virtualization Common management capabilities Attribute-based resource discovery Standard protocols and schemas

14 Cross-Organizational esource Sharing 14 Global namespace Metadata services Site autonomy esource usage data

15 15 Optimization of esource Usage Dynamic resource allocation (supply-side management) Dynamic workload prioritization (demand-side management)

16 16 Quality of Service (QoS) Assurance Service-level agreement and attainment Migration

17 17 Administration Policy-based management mechanisms Problem-determination mechanisms Scalable management architecture

18 18 High Availability Disaster recovery mechanisms Mechanisms for fault management

19 19 Job Execution Support for a variety of remote job types emote job management Job scheduling esource provisioning

20 20 Data Services Abstractions that provide uniform access and integration to various types of data Mechanisms to keep data consistent across replicas and caches Mechanisms to persist data Mechanisms for data location management

21 21 Security Authentication and authorization mechanisms Support for multiple security infrastructures Perimeter security mechanisms that support local infrastructure Isolation Delegation of access rights Support for dynamic negotiation of security policies Monitoring in support of intrusion detection and secure logging

22 22 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach

23 Grid Infrastructure: a Conceptual View 23

24 24 Execution Services To instantiate and manage units of work Behavior includes Finding and selecting execution candidate locations Preparing for execution Initiating execution Managing execution

25 25 Data Services To move, access and manage data resources Behavior includes Move data Manage replicated copies un queries and updates Transform data into new formats Maintain metadata

26 26 esource Management Services Management of low-level resources Monitoring, setup and control, discovery Management of the capabilities Functional interface management (e.g. create and destroy jobs) Domain-independent management System management of the many services on the Grid (e.g., system-wide backups)

27 27 Security Services To facilitate the enforcement of securityrelated policy within Virtual Organizations Behavior includes Verifying proof of an asserted identity Identity mapping Policy-based access control decision-making Audit and secure logging Privacy

28 28 Self-Management Services To reduce the cost and complexity of owning and operating IT infrastructure Behavior includes Negotiating mechanisms for resource usage

29 29 Information Services To efficiently access information about applications, resources and services Behavior includes Monitor services and resources Log system activities Cache and publish metadata

30 30 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach

31 31 The ole of the Globus Toolkit The Globus Toolkit is a collection of solutions to problems commonly found in collaborative distributed applications Heterogeneity A focus, in particular, on wrapping heterogeneity for application developers Abstractions Supporting general-case patterns and interactions, not specific to a particular application domain Standards We capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF) GT also includes reference implementations of new/proposed standards in these organizations

32 32 Layers in the Grid

33 33 Application Developer Off the Shelf Globus Toolkit Web Browser Grid Community Users work with client applications Without the Globus Toolkit Web Portal Certificate authority Chat Tool Credential epository Simulation Tool Data Viewer Tool Application services organize VOs & enable access to other services egistration Service Telepresence Monitor Data Catalog Collective services aggregate &/or virtualize resources A B C D E Compute Server Compute Server Camera Camera Database service Database service Database service esources implement standard access & management interfaces

34 34 Application Developer Off the Shelf Globus Toolkit Web Browser Grid Community Users work with client applications With the Globus Toolkit CHEF Certificate Authority Data Viewer Tool CHEF Chat Teamlet MyProxy Simulation Tool Application services organize VOs & enable access to other services Globus Index Service Telepresence Monitor Globus LS Collective services aggregate &/or virtualize resources Globus GAM Globus GAM Globus DAI Globus DAI Globus DAI Compute Server Compute Server Camera Camera Database service Database service Database service esources implement standard access & management interfaces

35 The Globus Toolkit: Standard Plumbing for the Grid Today the majority of the GT public interfaces are usable by application developers and system integrators elatively few end-user interfaces In general, not intended for direct use by end users (scientists, engineers, marketing specialists) Not turnkey solutions, but building blocks & tools for application developers & system integrators Some components (e.g., file transfer) go farther than others (e.g., remote job submission) toward end-user relevance Better to reuse than reinvent! Plenty of interesting unsolved problems to work on Compatibility with other Grid systems comes for free 35

36 Sidebar: The Globus Commitment to Open Source 36 To allow for inspection for consideration in standardization processes To encourage adoption in pursuit of ubiquity and interoperability To encourage contributions harness the expertise of the community

37 Globus Toolkit by Domain Areas 37 Core runtime Infrastructure for building new services Security Apply uniform policy across distinct systems Execution management Provision, deploy, & manage services Data management Discover, transfer, & access large data Information services Discover & monitor dynamic services

38 Globus Toolkit by Protocol Type 38 Web service protocols WSDL, SOAP WS Addressing, WSF, WSN WS Security, SAML, XACML WS-Interoperability profile Non Web service protocols Standards-based, such as GridFTP Custom

39 Globus Toolkit version 2 (GT2) Web Services Components Pre-WS Authentication Authorization GridFTP Grid esource Alloc. Mgmt (GAM) Monitoring & Discovery (MDS) C Common Libraries Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime

40 Globus Toolkit version 3 (GT3) Community Authorization WS Authentication Authorization OGSA-DAI eliable File Transfer Grid esource Alloc. Mgmt (WS GAM) MDS3 Java WS Core Web Services Components Pre-WS Authentication Authorization GridFTP eplica Location Grid esource Alloc. Mgmt (GAM) Monitoring & Discovery (MDS) C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime

41 Globus Toolkit version 4 (GT4) Delegation Data eplication Grid Telecontrol Protocol Community Scheduling Framework WebMDS Python WS Core Deprecated Community Authorization Authentication Authorization OGSA-DAI eliable File Transfer Workspace Management Grid esource Allocation & Management Trigger Index C WS Core Java WS Core Web Services Components Pre-WS Authentication Authorization Credential Mgmt GridFTP eplica Location Pre-WS Grid esource Alloc. & Mgmt Pre-WS Monitoring & Discovery C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime

42 Globus Toolkit: Common Capabilities 42 Credential Mgmt Delegation Community Authorization Data eplication eplica Location OGSA-DAI eliable File Transfer Grid Telecontrol Protocol Community Scheduling Framework Workspace Management Globus Toolkit v4 WebMDS Trigger Python untime C untime Authentication Authorization GridFTP Grid esource Allocation & Management Index Java untime Security Data Mgmt Execution Mgmt Info Services Common untime

43 43 GT4 Common untime Providing the common interfaces and capabilities for the toolkit as a whole Supports GT services (GAM, FT, Delegation, etc.) as well as user-developed services Leverages existing WS standards WS-I Basic Profile: WSDL, SOAP, etc. WS-Security, WS-Addressing Adds support for emerging WS standards WS-esource Framework, WS-Notification Java, Python, & C hosting environments

44 FileTransferService (without WSF) 44 FileTransfer Service move state whathappen tellmewhen cancel move (A to B) : transferid Client Developer reinvents wheel for each new service Custom management and identification of state: transferid Custom operations to inspect state synchronously (whathappen) and asynchronously (tellmewhen) Custom lifetime operation (cancel)

45 Uniform Interface for Common Interaction Patterns 45 EP EP EP Service esource Ps GetP GetMultPs SetP QueryPs Subscribe SetTermTime Destroy State representation esource esource Property State identification Endpoint eference State Interfaces GetP, QueryPs, GetMultiplePs, SetP Notification Interfaces Subscribe Notify Lifetime Interfaces SetTerminationTime ImmediateDestruction ServiceGroups

46 46 FileTransferService (with WSF) FileTransferService createesource createesource (A to B) : EP Client Transfer Ps getp queryps destroy Developer specifies custom method to createesource and leaves the rest to WSF standards: State exposed as esource + esource Properties and identified by Endpoint eference (EP) State inspected by standard interfaces (GetP, QueryPs) Lifetime management by standard interfaces (Destroy)

47 Uniform Interfaces for Common equirements Naming and bindings (basis for virtualization) Every resource can be uniquely referenced, and has one or more associated services for interacting with it Lifecycle (basis for fault resilient state mgmt) esources created by services following factory pattern esources destroyed immediately or scheduled Information model (basis for monitoring, discovery) esource properties associated with resources Operations for querying and setting this info Asynchronous notification of changes to properties Service groups (basis for registries, collective svcs) Group membership rules & membership management Base Fault type 47

48 Globus Toolkit version 4 (GT4) Delegation Data eplication Grid Telecontrol Protocol Community Scheduling Framework WebMDS Python WS Core Deprecated Community Authorization Authentication Authorization OGSA-DAI eliable File Transfer Workspace Management Grid esource Allocation & Management Trigger Index C WS Core Java WS Core Web Services Components Pre-WS Authentication Authorization Credential Mgmt GridFTP eplica Location Pre-WS Grid esource Alloc. & Mgmt Pre-WS Monitoring & Discovery C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime

49 49 Wrap-up Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach