Grid Middleware and Globus Toolkit Architecture Lisa Childers Argonne National Laboratory University of Chicago
2 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach
3 A fundamental problem of Grid Computing: supporting virtual organizations
4 Virtual Organizations
5 Virtual Organizations Distributed resources and people
6 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains
7 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals VO-A VO-B
8 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals Dynamic VO-A VO-B
9 Virtual Organizations Distributed resources and people Linked by networks, crossing admin domains Sharing resources, common goals Dynamic Fault tolerant VO-A VO-B
10 movie time! (a short tour of the accelerator at CEN)
11 10,000 Encyclopedia Britannica's per second
12 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach
Support for Heterogeneous Environments 13 esource virtualization Common management capabilities Attribute-based resource discovery Standard protocols and schemas
Cross-Organizational esource Sharing 14 Global namespace Metadata services Site autonomy esource usage data
15 Optimization of esource Usage Dynamic resource allocation (supply-side management) Dynamic workload prioritization (demand-side management)
16 Quality of Service (QoS) Assurance Service-level agreement and attainment Migration
17 Administration Policy-based management mechanisms Problem-determination mechanisms Scalable management architecture
18 High Availability Disaster recovery mechanisms Mechanisms for fault management
19 Job Execution Support for a variety of remote job types emote job management Job scheduling esource provisioning
20 Data Services Abstractions that provide uniform access and integration to various types of data Mechanisms to keep data consistent across replicas and caches Mechanisms to persist data Mechanisms for data location management
21 Security Authentication and authorization mechanisms Support for multiple security infrastructures Perimeter security mechanisms that support local infrastructure Isolation Delegation of access rights Support for dynamic negotiation of security policies Monitoring in support of intrusion detection and secure logging
22 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach
Grid Infrastructure: a Conceptual View 23
24 Execution Services To instantiate and manage units of work Behavior includes Finding and selecting execution candidate locations Preparing for execution Initiating execution Managing execution
25 Data Services To move, access and manage data resources Behavior includes Move data Manage replicated copies un queries and updates Transform data into new formats Maintain metadata
26 esource Management Services Management of low-level resources Monitoring, setup and control, discovery Management of the capabilities Functional interface management (e.g. create and destroy jobs) Domain-independent management System management of the many services on the Grid (e.g., system-wide backups)
27 Security Services To facilitate the enforcement of securityrelated policy within Virtual Organizations Behavior includes Verifying proof of an asserted identity Identity mapping Policy-based access control decision-making Audit and secure logging Privacy
28 Self-Management Services To reduce the cost and complexity of owning and operating IT infrastructure Behavior includes Negotiating mechanisms for resource usage
29 Information Services To efficiently access information about applications, resources and services Behavior includes Monitor services and resources Log system activities Cache and publish metadata
30 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach
31 The ole of the Globus Toolkit The Globus Toolkit is a collection of solutions to problems commonly found in collaborative distributed applications Heterogeneity A focus, in particular, on wrapping heterogeneity for application developers Abstractions Supporting general-case patterns and interactions, not specific to a particular application domain Standards We capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF) GT also includes reference implementations of new/proposed standards in these organizations
32 Layers in the Grid
33 Application Developer Off the Shelf Globus Toolkit Web Browser Grid Community Users work with client applications 10 12 0 0 Without the Globus Toolkit Web Portal Certificate authority Chat Tool Credential epository Simulation Tool Data Viewer Tool Application services organize VOs & enable access to other services egistration Service Telepresence Monitor Data Catalog Collective services aggregate &/or virtualize resources A B C D E Compute Server Compute Server Camera Camera Database service Database service Database service esources implement standard access & management interfaces
34 Application Developer Off the Shelf Globus Toolkit Web Browser Grid Community Users work with client applications 2 9 4 4 With the Globus Toolkit CHEF Certificate Authority Data Viewer Tool CHEF Chat Teamlet MyProxy Simulation Tool Application services organize VOs & enable access to other services Globus Index Service Telepresence Monitor Globus LS Collective services aggregate &/or virtualize resources Globus GAM Globus GAM Globus DAI Globus DAI Globus DAI Compute Server Compute Server Camera Camera Database service Database service Database service esources implement standard access & management interfaces
The Globus Toolkit: Standard Plumbing for the Grid Today the majority of the GT public interfaces are usable by application developers and system integrators elatively few end-user interfaces In general, not intended for direct use by end users (scientists, engineers, marketing specialists) Not turnkey solutions, but building blocks & tools for application developers & system integrators Some components (e.g., file transfer) go farther than others (e.g., remote job submission) toward end-user relevance Better to reuse than reinvent! Plenty of interesting unsolved problems to work on Compatibility with other Grid systems comes for free 35
Sidebar: The Globus Commitment to Open Source 36 To allow for inspection for consideration in standardization processes To encourage adoption in pursuit of ubiquity and interoperability To encourage contributions harness the expertise of the community http://dev.globus.org
Globus Toolkit by Domain Areas 37 Core runtime Infrastructure for building new services Security Apply uniform policy across distinct systems Execution management Provision, deploy, & manage services Data management Discover, transfer, & access large data Information services Discover & monitor dynamic services
Globus Toolkit by Protocol Type 38 Web service protocols WSDL, SOAP WS Addressing, WSF, WSN WS Security, SAML, XACML WS-Interoperability profile Non Web service protocols Standards-based, such as GridFTP Custom
Globus Toolkit version 2 (GT2) Web Services Components Pre-WS Authentication Authorization GridFTP Grid esource Alloc. Mgmt (GAM) Monitoring & Discovery (MDS) C Common Libraries Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime
Globus Toolkit version 3 (GT3) Community Authorization WS Authentication Authorization OGSA-DAI eliable File Transfer Grid esource Alloc. Mgmt (WS GAM) MDS3 Java WS Core Web Services Components Pre-WS Authentication Authorization GridFTP eplica Location Grid esource Alloc. Mgmt (GAM) Monitoring & Discovery (MDS) C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime
Globus Toolkit version 4 (GT4) Delegation Data eplication Grid Telecontrol Protocol Community Scheduling Framework WebMDS Python WS Core Deprecated Community Authorization Authentication Authorization OGSA-DAI eliable File Transfer Workspace Management Grid esource Allocation & Management Trigger Index C WS Core Java WS Core Web Services Components Pre-WS Authentication Authorization Credential Mgmt GridFTP eplica Location Pre-WS Grid esource Alloc. & Mgmt Pre-WS Monitoring & Discovery www.globus.org C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime
Globus Toolkit: Common Capabilities 42 Credential Mgmt Delegation Community Authorization Data eplication eplica Location OGSA-DAI eliable File Transfer Grid Telecontrol Protocol Community Scheduling Framework Workspace Management Globus Toolkit v4 www.globus.org WebMDS Trigger Python untime C untime Authentication Authorization GridFTP Grid esource Allocation & Management Index Java untime Security Data Mgmt Execution Mgmt Info Services Common untime
43 GT4 Common untime Providing the common interfaces and capabilities for the toolkit as a whole Supports GT services (GAM, FT, Delegation, etc.) as well as user-developed services Leverages existing WS standards WS-I Basic Profile: WSDL, SOAP, etc. WS-Security, WS-Addressing Adds support for emerging WS standards WS-esource Framework, WS-Notification Java, Python, & C hosting environments
FileTransferService (without WSF) 44 FileTransfer Service move state whathappen tellmewhen cancel move (A to B) : transferid Client Developer reinvents wheel for each new service Custom management and identification of state: transferid Custom operations to inspect state synchronously (whathappen) and asynchronously (tellmewhen) Custom lifetime operation (cancel)
Uniform Interface for Common Interaction Patterns 45 EP EP EP Service esource Ps GetP GetMultPs SetP QueryPs Subscribe SetTermTime Destroy State representation esource esource Property State identification Endpoint eference State Interfaces GetP, QueryPs, GetMultiplePs, SetP Notification Interfaces Subscribe Notify Lifetime Interfaces SetTerminationTime ImmediateDestruction ServiceGroups
46 FileTransferService (with WSF) FileTransferService createesource createesource (A to B) : EP Client Transfer Ps getp queryps destroy Developer specifies custom method to createesource and leaves the rest to WSF standards: State exposed as esource + esource Properties and identified by Endpoint eference (EP) State inspected by standard interfaces (GetP, QueryPs) Lifetime management by standard interfaces (Destroy)
Uniform Interfaces for Common equirements Naming and bindings (basis for virtualization) Every resource can be uniquely referenced, and has one or more associated services for interacting with it Lifecycle (basis for fault resilient state mgmt) esources created by services following factory pattern esources destroyed immediately or scheduled Information model (basis for monitoring, discovery) esource properties associated with resources Operations for querying and setting this info Asynchronous notification of changes to properties Service groups (basis for registries, collective svcs) Group membership rules & membership management Base Fault type 47
Globus Toolkit version 4 (GT4) Delegation Data eplication Grid Telecontrol Protocol Community Scheduling Framework WebMDS Python WS Core Deprecated Community Authorization Authentication Authorization OGSA-DAI eliable File Transfer Workspace Management Grid esource Allocation & Management Trigger Index C WS Core Java WS Core Web Services Components Pre-WS Authentication Authorization Credential Mgmt GridFTP eplica Location Pre-WS Grid esource Alloc. & Mgmt Pre-WS Monitoring & Discovery www.globus.org C Common Libraries extensible IO (XIO) Non-WS Components Security Data Mgmt Execution Mgmt Info Services Common untime
49 Wrap-up Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities The Globus Approach