A Gentle Introduction to Globus Otto Sievert May 26, 1999 Parallel Computation, CSE160/CSE260, Spring 1999 Abstract As computers become better connect

Transcription

1 A Gentle Introduction to Globus Otto Sievert May 26, 1999 Parallel Computation, CSE160/CSE260, Spring 1999 Abstract As computers become better connected across the globe, opportunities to create new programs utilizing geographically distant resources increase. New applications not possible without this computational grid include distributed supercomputing and intelligent sensing applications. However, diculties implementing this new class of applications exist: the computers on a grid are unreliably connected and not beholden to any single management policy. It is dicult to achieve performance on the grid. Users are currently requiredtohave extraordinary knowledge about every element inthe grid to make informed choices about resource allocation/management and programming paradigms. The Globus project provides infrastructure to addresses these issues. Globus is a complex entity; this document isa simple introduction to Globus and its architecture. 1 Introduction The terms computational grid and metacomputing refer to the utilization of a collection of distributed computing resources as if it were a single entity. A computational grid may include personal computers, remote instruments, workstations and MPPs, all connected via awideareanetwork. The fundamental idea of a computational grid is that the local view of all computing resources, communication, and security should be uniform. The autonomous nature of individual systems makes ecient implementation of a grid dicult. A grid attempts to unify what are, by denition, distinct systems. This tension leads directly to the most fundamental problems unique to grid computing: Dierent Administrative Domains Security and authentication methods vary from system to system. Resource Allocation Selecting the most suitable resources among a pool of simply dierent resources is a non-trivial task, even ignoring the fact that resource availability and usage may change radically in a short period of time. Resource Management Even when appropriate resources have been allocated, grid application schedulers are often in competition with local job and process schedulers. Heterogeneous Computing Resources Performance is more dicult to guarantee when individual systems have radically dierent architectures and communicate over non-uniform network connections. Communication Often, communication is relatively slow compared to clusters or other computing collections. Reliability is also an issue, as is quality of service, because grids may share communication bandwidth with others. Though dicult, it is important to address these problems because of the potential for new and exciting computing applications. For example, a grid facilitates computation on a scale never before available. Imagine connecting all the computing power of NCSA, NASA, NPACI, etc., together at one time, running one program. Imagine connecting a remote sensing device like a scanning electron microscope to a remote, highperformance visualization engine, producing real-time three-dimensional imagery of the subject and allowing better instrument control and deeper understanding. Imagine a researcher controlling this system from an of- ce computer thousands of miles away. One motivation of distributed computing is economic; if high-performance and specialized equipment were not prohibitively expensive, everyone would have 1

2 local access to these resources. Every scanning electron microscope would sit next to a graphics visualization engine; each student would sit in school with a fully interactive, educational supercomputer. But economic realities and an ever-increasing demand for computational power have created a situation where powerful or specialized resources are distributed. Computational grids are a strategy to increase eective computing capability without incurring the high cost of new equipment. 2 Globus Opportunities beckon; problems exist. The Globus project, a joint eort of the Argonne National Laboratory and the University of Southern California Information Sciences Institute, addresses some of these issues. Globus is many things; at it's core the Globus team performs basic infrastructure development for applications that integrate geographically distributed systems. Globus has developed a prototype computational grid, GUSTO, and has helped develop grid-enabled software applications to test this infrastructure. The most concrete result of the Globus project is the Globus Toolkit. This toolkit is a symbiotic set of basic grid services, including resource managers, security protocols, information services, communication services, fault tolerance services, and remote data access facilities. The Toolkit has several interesting characteristics. It adopts a \bag of services" model, which allows developers to use as few or as many of these stand-alone services as they want, and does not impose any single programming paradigm. Globus tools may replace or co-exist with mechanisms provided by commodity computing. All of this has been done with careful thought given to enabling high performance. In the rest of this paper various aspects of the Globus Toolkit are discussed in terms of how they address the fundamental grid problems mentioned above. Treatment is given to resource management, communication, security, information services, fault tolerance, and remote le access. A Globus installation, GUSTO, is then discussed, as is an example Globus application, SF- Express. 3 Resource Management Service The Globus Resource Allocation Manager (GRAM) provides a uniform interface to the wide range of local resource management tools. GRAM translates generic resource requests into commands that are specic to a particular local system. GRAM is the lowest level of Globus resource management, and interacts directly with local schedulers[4]. The user interface to GRAM is through a gatekeeper. A gatekeeperruns on every Globus machine. Globus job requests are submitted to the gatekeeper on the machine whose resources are requested. This gatekeeper authenticates the user (while the user mutually authenticates the gatekeeper), then matches the user to an account local to the machine it is necessary to have a local account on all machines whose resources are requested. After a remote user is determined, the gatekeeper starts a job manager process running under the remote user's login. The job manager is then solely responsible for that single Globus job. The job manager is responsible for communicating with the (local) user, allocating resources, executing the job, and deallocating resources after job conclusion. Another logical part of the Globus resource management system is the co-allocator. The co-allocator is responsible for coordinating resource requests to multiple machines on possibly multiple sites to synchronize program execution. Currently, all Globus co-allocation is done by the application. However, this function could be spun o into another Globus tool. Resource requests are phrased in the Globus Resource Specication Language (RSL). RSL is an extensible language used to communicate resource requests. The language is simple: a list of resource expressions logically combined with a handful of logical operators. With these operations resources may be explicitly requested or implicitly dened. For example, a concrete RSL request to run the program myprogfor 24 nodes on a specic computer at Cal Tech might look like this: &(executable=myprog) (count=24) (resourcemanager=neptune.caltech.edu:755) while an abstract resource request to execute this program on either 24 nodes of a computer with 64 MB of memory per node or 10 nodes on a computer with 128 MB of memory per node might look like: &(executable=myprog) ( (&(count=24)(memory>=64)) ((count=10)(memory>=128))) RSL allows the user to communicate intents, which paves the way for automated resource selection and allocation through the use of brokers. Automated brokers, which do not exist currently, could accept an abstract RSL request and specialize it into a concrete RSL that species exact resources. RSL parameters should be of three types: a Distinguished Name (DN) from the Metacomputing Directory Service (MDS, discussed later); a parameter specic to the local resource manager; 2

3 a language-extension parameter. Any language extension, typically created for convenience, must be distilled into one of the other two RSL parameter types before reaching the job scheduler. 4 Communication Service The Nexus communication library provides fast communication primitives to Globus. Nexus allows multi-mode communication conversations (communication over several physical networks and/or communication protocols). It supports unicast and multicast communication methods. Nexus is not designed as a user communication protocol; rather, friendly and ecient user communications are built using Nexus. Nexus abstracts ve communication objects: nodes, contexts, threads, communication links, and remote service requests[10]. A program consists of one or more address spaces or contexts, eachcontaining one or more threads of execution. Threads communicate via links between communication nodes by sending and receiving messages and executing remote service requests to other contexts. Messages contain control information, but may also contain global addresses which point to context-invariant data structures. Remote service requests allow threads to execute functions in another address space. Basic communication is constructed piece by piece. Nodes are created in two contexts[11]. One node becomes a communication start-point, the other an endpoint. One-waycommunication is then obtained by linking the start-point tothe endpoint. Two-way communication requires repeating this process with the startpoint and endpoint in the other contexts. Broadcast communication is obtained by linking one start-point to more than one endpoint. Similarly, a gather communication is obtained when more than one start-point is linked to one endpoint. 5 Security Service The Globus Security Service (GSS) provides a common interface for integrating diverse local security solutions. Features of this system include a single user sign-on, meaning that subsequent authentication requests are automatically handled by proxies. All resource use is authenticated, maintaining system security and controlling resource usage[9]. Globus security uses the ITU X.509 authentication protocol. This standard species the use of certicates, which are credentials that prove identity. Certicates are exchanged and compared to reference certicates for validity. The reference certicates are typically not identical to the oered certicate, but contain sucient information to prove the sender's identity. Certicate-based security relies exclusively on the trust of a Certicate Authority (CA). The CA is trusted by all,andprovides each entity with its own certicate. The basic rule used in certicate-based authentication is the transitive property of trust: if entity 1 trusts the CA, and entity 2hasshown that the CA trusts them, then entity 1 trusts entity 2. Each entity must initially register with the CA (and in return receive a certicate) to be able to participate in certicate-based authentication schemes. For example, assume three entities: a CA, a system administrator, and a user who wishes to use the system. By exchanging certicates, the system administrator can verify that the user is registered (trusted) by the CA and approve the access. Additionally, by obtaining the system's certicate the user can verify that they are communicating with the true system, and not an imposter. After secure authentication, Globus communication is performed using the Secure Socket Layer (SSL) protocol, which additionally encrypts communication exchanges using public/private key encryption. One additional aspect of the Globus security system is worth mentioning. A user can set up a proxy that will automatically authenticate them. Each Globus security proxy exists only for a specic time period dened by theuser. The advantage of a proxy is convenience: a user who has created a proxy does not have to authenticate himself/herself as long as the proxy is active. Additionally, proxies allow unattended batch jobs to be submitted to Globus (for example if a user wants to perform three Globus runs overnight). 6 Information Service A directory is like a database; directories hold records of information. Unlike databases, though, directories are read from more than they are written to. Globus provides the Metacomputing Directory Service (MDS) that contains resource information about grid elements[5]. System conguration information like how many processors the computer has, and what operating system is running. Usage information like the number of unallocated processors, and the system load. Management information like the name and contact information for the owning organization. Conguration info like theversion of Globus running on the machine, or the type of local resource scheduler. MDS information is automatically updated regularly but infrequently (updates occur on the order of hours) from GRAMs running on each local machine. Access to the MDS is obtained through the use of the Lightweight Directory Access Protocol (LDAP). LDAP 3

4 species the network protocols used to access directories and denes the form and character of the information as a directory information tree. This tree gives hierarchical information about specic resources[12]. The MDS is generated with and queried using LDAP. The MDS allows two kinds of directory service queries. A white pages request provides information regarding a specic directory entry. For example, the MDS may be queried to nd the IP address of a specic machine. A yellow pages request provides a list of directory entries with a specic property. For example, the MDS may be asked to provide a list of all machines with more than 48 processors. Today, the information held in the MDS allows grid users to understand the resources available and to formulate concrete resource requests. The yellow pages access mode of MDS enables future automated resource selection. 7 Fault Tolerance Service In addition to some fault tolerance integral to Nexus, Globus provides the Globus Heartbeat Monitor (HBM). The HBM performs periodic inquiry to detect process failures. The HBM does not provide system information, only process information. It additionally only provides an answer to the question "is process X still active?". To use the HBM, the user process or heartbeat client (HBC) makes a function call to register with the HBM daemon running on the local machine. This registration noties the daemon to actively probe the process, species the probing frequency, and species the data collector that receives probe data[13]. In this manner a process' health can be remotely monitored. 8 Remote Data Access Service The Global Access to Secondary Storage (GASS) provides primitive access to remote data. By accessing all les via a Universal Resource Locator (URL) the actual physical location of the le does not matter. GASS allows four types of remote le access[1]: shared read-only access to an entire le; shared write access where the last write dominates; shared append-only access to a le; unshared unrestricted read/write access. These le access modes are designed to provide commonly needed features while requiring very simple coherency rules. 9 GUSTO A prototype computational grid running Globus has been constructed. Called the Globus Ubiquitous Supercomputing Testbed (GUSTO), this system includes computing facilities from the U.S., Japan, Europe, and Australia[7]. Currently GUSTO has access to more than 3600 processors on more 300 computers at seventeen locations[8]. These machines are connected primarily via the Internet, with some connections provided by ATM networks. The aggregate peak performance of GUSTO is more than 2 Tops (2 trillion oating point operations per second)[7]. 10 SF-Express One of the most dramatic demonstrations of Globus' potential was the execution of a 55,000 entity battleeld simulation program. The U.S. Government Department of Defense maintains a virtual battleeld program to simulate wartime conditions. This application, Modular Semi-automated Forces (ModSAF), is an example of the Distributed Interactive Simulation (DIS) class of programs. In this program, troops, tanks, aircraft, and other entities interact with each other and with user input for testing and training purposes[3]. The program treats each entity uniquely and utilizes a cluster of workstations connected via a Local Area Network (LAN). Each workstation hosts a modest number of entities (30-100), and is responsible for handling interactions between entities residing on other workstations. Entity conditions and interactions are communicated in Protocol Data Unit (PDU) messages. In ModSAF, each machine broadcasts its entities' PDUs at regular intervals. All machines receive and parse all PDUs. One can see the scalability problem with this architecture as the number of entities increases: with the amount of communication scaling with the number ofentities, communication bandwidth is quickly exceeded. Barely-manageable runs of Mod- SAF involved 5400 entities. The Department of Defense, in conjunction with the Defense Advanced Research Projects Agency (DARPA) issued a challenge: they wanted to perform a ModSAF simulation with 10,000 entities. A new program, called Simulated Forces Express (SF-Express), performed the same actions as ModSAF, but was created specically for Globus. This program altered the entity communication patterns to be more scalable. Distant entities do not aect each other, allowing many communication messages to be ignored. 4

5 SF-Express dedicated one or more processors per machine to the task of discarding unneeded messages. SF- Express also used dedicated data servers, through which all access to secondary storage was routed. Agiven machine was decomposed into a number of simulation nodes that actually performed work, a number of interest management nodes that pruned the message stream, a number of data server nodes that managed secondary storage, and a router node that coordinated communication with other machines. Communication within a machine was handled via the Message Passing Interface (MPI), while communication between machines used sockets. These improvements, along with the raw power of GUSTO, allowed SF-Express to perform a run with more than 55,000 entities in August, This was avery successful demonstration of Globus and GUSTO. However, this display of computing power required signicant manual coordination between the six sites involved. The machines at these sites were dedicated entirely to SF-Express, and coordination of execution was painfully managed by hand[2]. 11 Conclusion The Globus Toolkit provides infrastructure supporting grid computing, enabling next-generation applications to be run on a widely distributed collection of computing resources. Globus is very much a work in progress. As new techniques addressing the fundamental grid computing problems are discovered, advances in this infrastructure ease the pain of distributed computing by increasing performance and eliminating extensive user knowledge about the underlying system. More detail can be found in the references or via the World Wide Web: 12 Review Questions 1. A telephone book is most like what Globus service? 2. What is RSL? How does Globus use it? 3. Why would you use a computational grid? Why would you not? 4. Name three fundamental problems in grid computing. 5. Does Globus provide the ability for multiple processes to simultaneously access a remote le? Under what conditions? 6. What two Globus communication primatives enable threads to communicate? References [1] Bester, J., Foster, I., Kesselman, C., Tedesco, J., and Tuecke, S. Gass: A data movement and access service for wide area computing systems. In 7 (1997), p [2] Brunett, S., Czajkowski, K., Foster, I., Fitzgerald, S., Johnson, A., Kesselman, C., Leigh, J., and Tuecke, S. Application experiences with the globus toolkit. In Proc. 7th IEEE Symp. on High Performance Distributed Computing (July 1998), IEEE Computer Society Press. To appear. [3] Brunett, S., Davis, D., Gottschalk, T., Messina, P., and Kesselman, C. Implementing distributed synthetic forces simulations in metacomputing environments. In Proceedings of the Heterogeneous Computing Workshop (Mar. 1998). [4] Czajkowski, K., Foster, I., Karonis, N., Kesselman, C., Martin, S., Smith, W., and Tuecke, S. A resource management architecture for metacomputing systems. In The 4th Workshop on Job Scheduling Strategies for Parallel Processing (Mar. 1998), IEEE-P, pp. 4{18. [5] Fitzgerald, S., Foster, I., Kesselman, C., von Laszewski, G., Smith, W., and Tuecke, S. A directory service for conguring highperformance distributed computations. In Proc. 6th IEEE Symp. on High Performance Distributed Computing (1997), IEEE Computer Society Press, pp. 365{375. [6] Foster, I., Karonis, N. T., Kesselman, C., and Tuecke, S. Managing security in highperformance distributed computations. Cluster Computing 1, 1 (1998), 95{107. [7] Foster, I., and Kesselman, C. Globus: A metacomputing infrastructure toolkit. International Journal of Supercomputer Applications 11, 2 (1997), 115{128. [8] Foster, I., and Kesselman, C. The Globus project: A progress report. In Proceedings of the Heterogeneous Computing Workshop (Mar. 1998). [9] Foster, I., Kesselman, C., and Tsudick, S. T. G. A security architecture for computational grids. In Proc. of the 5th ACM Conference on Computer and Communication Security (Nov. 1998), ACM Press. To appear. 5

6 [10] Foster, I., Kesselman, C., and Tuecke, S. The Nexus task-parallel runtime system. In Proc. 1st Intl Workshop on Parallel Processing. Tata Mc- Graw Hill, 1994, pp. 457{462. [11] Foster, I., and Tuecke, S. Nexus: Runtime support for task-parallel programming languages. Tech. rep., ANL, [12] Foster, I., and von Laszewski, G. Usage of ldap in globus. Tech. rep., ANL, [13] Stelling, P., Foster, I., Kesselman, C., Lee, C., and von Laszewski, G. A fault detection service for wide area distributed computations. In Proc. 7th IEEE Symp. on High Performance Distributed Computing (July 1998), IEEE Computer Society Press. To appear. 6