A Multi-agent System Architecture for End-User Level Grid Monitoring Using Geographic Information Systems (MAGGIS): Architecture and Implementation

Size: px

Start display at page:

Download "A Multi-agent System Architecture for End-User Level Grid Monitoring Using Geographic Information Systems (MAGGIS): Architecture and Implementation"

Annabelle Barber
6 years ago
Views:

1 A Multi-agent System Architecture for End-User Level Grid Monitoring Using Geographic Information Systems (MAGGIS): Architecture and Implementation Shaowen Wang 1, Anand Padmanabhan 1, Yan Liu 1, ansom Briggs 1, Jun Ni 1, Tao He 1, Boyd M. Knosp 1, and Yasar Onel 2 1 Academic Technologies-esearch Services of Information Technology Services, The University of Iowa, Iowa City, IA 52242, USA {shaowen-wang, anand-padmanabhan-1, yan-liu-1, ransom-briggs, jun-ni, tao-he, boyd-knosp}@uiowa.edu 2 Department of Physics and Astronomy, The University of Iowa, Iowa City, IA 52242, USA yonel@newton.physics.uiowa.edu Abstract. This paper illustrates a Multi-Agent system architecture for end-user level Grid monitoring using Geographical Information Systems (MAGGIS). The purpose of this research is to investigate MAGGIS architecture and implementation issues, and to verify the following two hypotheses: 1.) multiagent systems provide an effective and scalable architecture to synthesize various Grid information providers for monitoring Grid resources; and 2.) geographic information systems (GIS) provide an ideal solution to organizing and managing the geographic aspect of Grid resource information as well as to providing an effective user interface for monitoring Grid status. The MAGGIS framework is implemented in a Grid portal environment based on the open Grid service architecture. It is observed that the MAGGIS not only helps end-users monitor the status of Grid resources, but also provides quick and comprehensive information for resource scheduling and management on behalf of user applications. 1 Introduction Grid technologies enable large-scale coordinated sharing of distributed computing resources within Virtual Organizations (VO) [1, 2]. Grid monitoring solutions are required to provide information to determine the source of performance problems, tune Grids and their applications to optimal performance, detect faults and execute recovery mechanisms, and predict performance and schedule computational tasks [3]. ecent active research in Grid monitoring has covered a broad scope of research topics [4]. These topics mainly include Grid monitoring architectures [3], monitoring information modeling [5], query methods for Grid information services [6], and performance study of monitoring and information services for distributed systems [7]. Grid monitoring can be classified as two types based on its purposes: end-user level monitoring and system level monitoring. Although some researchers, e.g., M. Li et al. (Eds.): GCC 2003, LNCS 3032, pp , Springer-Verlag Berlin Heidelberg 2004

2 A Multi-agent System Architecture for End-User Level Grid Monitoring 537 Laszewski et al., have conducted research on the end-user level Grid monitoring in a service-oriented way [8], most current research has been focusing on system level monitoring. However, end-user level monitoring must be designed to meet the needs of user applications as opposed to the emphasis of the system level monitoring has on helping manage Grid resources. This paper demonstrates a Multi-Agent system architecture for end-user level Grid monitoring using Geographical Information Systems (MAGGIS) and its prototype implementation. Agents can be defined to be autonomous, problem-solving computational entities capable of effective operation in dynamic and open environments [9]. Agents are often deployed in a multi-agent system in which they interact, and maybe cooperate, with other agents that have possibly conflicting aims [10]. Agent-based approach has been applied to address issues in Grid computing such as load balancing [11] and system level monitoring [12]. However, it is advantageous to apply the multi-agent system approach to end-user level monitoring for the following two reasons: 1.) knowledge about Grid resource information can be transferred from Grid information services to user applications in a consistent way through agent communication mechanisms such as the Knowledge Query and Manipulation Language (KQML) [13]; 2.) agents, on behalf of users and user applications, can represent the preferences and goals of monitoring resources, which potentially leads to high efficiency and optimal performance of the MAGGIS. In MAGGIS, geographical information systems (GIS) are used to handle the geographical aspect of Grid information. GIS is defined as an information system that is used to input, store, retrieve, manipulate, analyze, and output geographically referenced data or geospatial data [14]. Map-based geographic referencing [15] has been used to monitor Grid resources in several large European Grid projects. However, this type of research effort needs to be extended to fully address the needs of monitoring the Grids that span across multi-scale and dynamic VOs. In rest of this paper, section 2 articulates the MAGGIS multi-agent architecture. Section 3 explains how GIS is used to handle the geographic aspect of Grid information based on a spatial-temporal data model. Section 4 provides a prototype implementation for MAGGIS in a Grid portal [16] context based on the Open Grid Service Architecture (OGSA) [17]. Finally, section 5 draws several conclusions, based on which some future research directions are pointed out. 2 Architecture In principle, the MAGGIS multi-agent system adopts the classical multi-agent architecture used in distributed artificial intelligence [18]. 2.1 Functions The MAGGIS multi-agent system architecture includes two logic layers: a data collection layer and a knowledge layer. The data collection layer is comprised of monitoring and synthesizing components while the knowledge layer incorporates data representation, modeling, communication, and analysis components. This two-layer

3 538 S. Wang et al. architecture can be translated into a functional view (Fig. 1) in which the data collection layer is equivalent to monitoring agents while the knowledge layer is equivalent to user agents. Consequently, the monitoring agents have capabilities for monitoring and synthesizing information as well as for modeling data. The user agents are primarily responsible for analyzing Grid information from monitoring agents and presenting the aggregated information to a user client. In addition, the user agents handle user requests and maintain user profiles through the use of the services provided by the monitoring agents. The communication between user agents and monitoring agents is implemented using the KQML [13]. The monitoring agents store information that is based on a spatial-temporal data model described in section 3. : esource (unning Sensor) e.g. Ganglia GIP: Grid Information Provider e.g. MDS DB: Database (MySql in this case) MA: Monitoring Agents UA: User Agents Serving User equest UA User Agent Communication MA MA MA DB DB GIP VO1 GIP GIP VO2 GIP Fig. 1. Multi-agent system architecture of MAGGIS 2.2 Scalability The main advantage of using this multi-agent system approach to Grid monitoring is that the multi-agent system architecture by its inherent nature is scalable and capable of aggregating information from disparate monitoring data sources (e.g., MDS [1], Ganglia [19], NWS [20], and local job managers). In the present architecture, databases are associated with VOs. The number of databases per VO can be determined based on the VO size. There is a monitoring agent process at the database level that manages the registration information of Grid resources. This process will

4 A Multi-agent System Architecture for End-User Level Grid Monitoring 539 also be employed to dynamically balance monitoring loads among available monitoring agents. 2.3 Methods The methods used to implement the multi-agent system architecture are mainly reflected in the following two tasks agents perform autonomously. 1. Collecting monitoring information: When a monitoring agent is instantiated, it acquires the information about particular Grid resources it is supposed to monitor. The monitoring agent implements a multi-threaded model that allows the agent to independently monitor various Grid resources throughout its lifetime. The resource information can be collected from different information providers and stored in databases. 2. Servicing user agents: A non-blocking multi-threaded mechanism was implemented to handle requests from multiple users. The communication between user and monitoring agents is realized through the use of KQML. The format of a KQML message sent from a user agent to a monitoring agent along with an illustrative example is provided as follows: Performative: Sender: eceiver: Message Content Example: Ask-All: UserAgent1: MonitoirngAgent1: Provide information about CPU utilization of machine XYZ for the past hour The monitoring agent responds to the user agent using tell or sorry performative. A sorry performative is used when the monitoring agent is unable to comply with the user requests. A tell performative is used to send the requested information to the user agent. Moreover, all agents are autonomous and a failure of one agent or failure of one thread within an agent does not affect the other agents. 3 Geographic Information Integration The integration of geographic information has been addressed in the past research of Grid monitoring to emphasize the needs of visualizing the geographic distribution of computing resources [21]. However, no existing data model for monitoring information has specifically taken geographic information into account. It is necessary to develop a generic spatial-temporal data model to handle the monitoring data that includes geographic attributes. 3.1 Spatial-Temporal Data Model Our spatial-temporal data model adopts the conceptual pyramid model [22]. At the knowledge level, it is integrated with the MAGGIS multi-agent system architecture that is independent from the implementation of a particular Grid information service. Consequently, the model can be developed independently from the implementation aspect of data models in Grid information services such as the relational data model [5], the directory service based on the Lightweight Directory Access Protocol (LDAP)

540 S. Wang et al. [1], or XML. In addition, our spatial-temporal data model is able to be incorporated in the agent communication through an event-based mechanism [23]. Fig. 2.

5 540 S. Wang et al. [1], or XML. In addition, our spatial-temporal data model is able to be incorporated in the agent communication through an event-based mechanism [23]. Fig. 2. A MAGGIS user interface 3.2 Data Model Implementation An implementation of our spatial-temporal data model is based on an extended relational data model implemented in a popular GIS software solution ArcGIS [24]. In this data model, the attribute information that may or may not be location-sensitive is stored and managed using a relational database. The association between geographic information and attribute information is established through the use of an indexing method for geometric objects. GeoTools [25] is used to handle the geographic knowledge aspect of the spatial-temporal data model in an object-oriented way, which meets the needs of translating the spatial-temporal data to specific knowledge in the multi-agent system architecture. For example, the left part of Fig. 2 shows an applet-based interface that provides map-based resource visualization, browsing, selection, and spatial information query functions. 4 MAGGIS Implementation MAGGIS was prototyped as a Grid service in a Grid portal that is called Grid esearch & education IoWa (GOW) portal. The GOW Grid portal was developed using Jetspeed [26] as a Grid portal server and development toolkit. The relationship between Jetspeed and other technologies used is illustrated in Fig. 3.

6 A Multi-agent System Architecture for End-User Level Grid Monitoring MAGGIS Grid Service MAGGIS Grid service was implemented based on Globus Toolkit 3.0 [17]. It includes three major portlets: GeoTools portlet, user-agent portlet, and visualization portlet. GeoTools portlet was developed using GeoTools and it directly interacts with user-agent portlet to manage geographic aspect of monitoring information. User-agent portlet provides the capability for aggregating information based on user preferences through the communication with monitoring agents. Visualization portlet displays the aggregated information collected from user-agent portlet. These three portlets are integrated together as a Grid service that is portable to other Grid-service-based portal environment. VO esources MAGGIS Grid service JavaCOG Grid middleware OGSA services MyProxy Monitoring Grid service GeoTools Query Visualization Xportlets OGSA User mgmt Security Portlet mgmt Jetspeed portlet engine T o m c a t User Fig. 3. Grid portal technologies employed 4.2 Case Study MAGGIS Grid service is coupled with the GOW portal that uses Java CoG Kit [27] and Sun s Java XML for developing OGSA-compliant Grid services. MyProxy [28] is integrated with Jetspeed to provide a Web-based Grid security solution. The MAGGIS service in the context of GOW portal is deployed to a prototype campus Grid at the University of Iowa, several resources of which are belong to a few national Grid testbeds. Fig. 2 shows an end-user level Grid monitoring scenario in which the MAGGIS Grid service in the GOW portal was used. This scenario took place after a user selected a VO from a VO list. All the resources of this selected VO are displayed on a map. Also, the user can select particular resources in the VO, examine their dynamic performance, and visualize aggregated and summary information. Our case study has demonstrated that MAGGIS Grid service provides a user-friendly environment in which monitoring information is presented in an effective way.

7 542 S. Wang et al. 5 Concluding Discussions End-user level Grid monitoring is critical to ensure that Grid resources are useful for user applications. The MAGGIS multi-agent system is designed to achieve scalable monitoring for multiple VOs of Grid resources from the perspective of user applications. The system interacts with heterogeneous Grid information providers through the drivers of monitoring agents. User agents are characterized to work on behalf of user clients to pull preferred information out of MAGGIS. The information in the MAGGIS multi-agent system is represented using a spatialtemporal data model and communicated using KQML. The data model is implemented using GIS software to manage geographic aspect of Grid information. It is found that GIS provides an effective solution to releasing the cognitive load for users to understand and manage the information of Grid resources organized in a VO fashion. The MAGGIS is prototyped in a Grid portal environment to emulate the situation in which user-level Grid monitoring is a part of problem solving environments for applications. Future research will focus on evaluating performance and developing fault tolerance mechanisms in MAGGIS multi-agent system. Several identified failure scenarios will be addressed. These failures mainly include monitoring agent failure (e.g., fail to collect monitoring data, or fail to respond to user agents), database failure, and user agent failure (e.g., not responding to user requests). Acknowledgements. A subcontract with National Science Foundation and Department of Energy of U.S.A., The University of Iowa Informatics Initiative, and the Information Technology Services of The University of Iowa funded this research. The authors would like to thank Frederick M. Noth for his helpful suggestions. eferences 1. Czajkowski, K., Fitzgerald, S., Foster, I., Kesselman, C.: Grid Information Services for Distributed esource Sharing. Proceedings of the Tenth IEEE International Symposium on High-Performance Distributed Computing (HPDC-10), IEEE Press (2001) 2. Foster, I., Kesselman, C., Tuecke, S.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal of Supercomputer Applications, 15 (2001) 3. Tierney, B., Aydt,., Gunter, D., Smith, W., Taylor, V., Wolski,., Swany, M.: A Grid Monitoring Architecture. The Global Grid Forum GWD-GP-16-2, January (2002) 4. Casanova, H.: Distributed Computing esearch Issues in Grid Computing. Quarterly Newsletter for the ACM Special Interest Group on Algorithms and Computation Theory (SIGACT News), 33 (2002) 5. Fisher, S.: elational Model for Information and Monitoring. Technical eport GWD- Perf-7-1, GGF (2001) 6. Plale, B., Schwan, K.: Dynamic Querying of Streaming Data with the dquob System, IEEE Transactions on Parallel and Distributed Systems, 14 (2003)

8 A Multi-agent System Architecture for End-User Level Grid Monitoring Zhang, X., Freschl, J. L., Schopf, J. M.: A performance Study of Monitoring and Information Services for Distributed Systems. Proceedings of HPDC-12 (2003) 8. Laszewski, G., Gawor, J., Pe na, C. J., Foster, I.: InfoGram: A Peer-to-Peer Information and Job Submission Service. Proceedings of the 11th Symposium on High Performance Distributed Computing (2002) 9. Luck, M., McBurney, P., Preist, C.: Agent Technology: Enabling Next Generation Computing A oadmap for Agent-based Computing Version 1.0. Available at: (2003) 10. Jennings, N., Sycara, K., Wooldridge. M.: A roadmap for agent research and development. 1 (1):7-38 (1998) 11. Shen, W., Li, Y., Ghenniwa, H., Wang, C.: Adaptive Negotiation for Agent-Based Grid Computing. Proceedings of AAMAS2002 Workshop on Agentcities: Challenges in Open Agent Environments, Bologna, Italy, pp (2002) 12. Newman, H. B., Legrand, I. C., Bunn, J. J.: A Distributed Agent-based Architecture for Dynamic Services. CHEP, Beijing, China (2001) 13. Finin, T., Labrou, Y., Mayfield, J.: KQML as an Agent Communication Language. Software Agents, AAAI/MIT Press (1994) 14. Goodchild, M. F.: Geographical information science. International Journal of Geographical Information Systems, 6, (2003) 15. Map Center Project. Available at: (2003) 16. Fox, G., Pierce, M., Gannon, D., Thomas, M.: Overview of Grid Computing Environments. Technical eport GGF-GCE-OVEVIEW2, GGF (2003) 17. Foster, I., Kesselman, C., Nick, J., Tuecke. S.: Grid Services for Distributed System Integration. Computer, 35 (6) (2002) 18. Hartvigsen, G., Johansen, D.: Co-operation in a Distributed Artificial Intelligence Environment the StormCast Application. Pergamon Press, Oxford, England (1990) 19. Ganglia, a Distributed Monitoring and Execution system. Available at: (2003) 20. Network Weather Service (NWS). Available at: (2003) 21. Baker, M. A., Smith, G.: A Prototype Grid-site Monitoring System, Version 1, DSG Technical eport, January (2002) 22. Mennis, J. L., Peuquet, D. J., Qian, L.: A Conceptual Framework for Incorporating Cognitive Principles into Geographical Database epresentation. International Journal of Geographical Information Science, 14 (6), (2000) 23. Peuquet, D. J., Duan, N.: An Event-based Spatialtemporal Data Model (ESTDM) for Temporal Analysis of Geographical Data. International Journal of Geographical Information Science, 9 (1), 7-24 (1996) 24. ArcGIS Software. Available at: (2003) 25. GeoTools Open Source Project. Available at: (2003) 26. Jetspeed Open Source Project. Available at: (2003) 27. Java Cog Kit Open Source Project. Available at: (2003) 28. Novotny, J., Tuecke, S., S. Welch. V.: An Online Credential epository for the Grid: MyProxy. Proceedings of the Tenth International Symposium on High Performance Distributed Computing (HPDC-10), IEEE Press (2001)

AGARM: An Adaptive Grid Application and Resource Monitor Framework

AGARM: An Adaptive Grid Application and Resource Monitor Framework Wenju Zhang, Shudong Chen, Liang Zhang, Shui Yu, and Fanyuan Ma Shanghai Jiaotong University, Shanghai, P.R.China, 200030 {zwj03, chenshudong,