An Overview of Grid Computing Day 2 August 06, 2004 (Friday) By CDAC Experts Contact :vcvrao@cdacindia.com; betatest@cdacindia.com URL : http://www.cs.umn.edu/~vcvrao 1
Betatesting Group,NPSF, C-DAC,Pune Dr.VCV.Rao (vcvrao@cdacindia.com) Mr.Ravi Kumar (ravi@cdacindia.com) Mr.Subba Ramanna (subba@cdacindia.com) Mr.Sridhar Padala (spadala@cdacindia.com) Ms. B.S.V.S. Sudha (sudhas@cdacindia.com) NPSF System Admin Group, C-DAC,Pune Dr. Sandeep K. Joshi (skjoshi@cdacindia.com) 2
! Grid Programming: an overview of tools, Environments and projects Part II Middleware Frameworks o Cactus Problem Solving Environments Portals Interoperability of data parallel libraries Towards a Global Compilation system 3
! Open source Modular Structure facilitate parallel computation CACTUS is a freely available, modular portable and manageable environment for collaboratively developing parallel, high-performance multidimensional simulations 4
Cactus name derives from the design of central core ( flesh ) which connects to application or infrastructure modules ( thorns ) through an extensible interface Flesh provides code infrastructure Parameter passing variable, scheduling, database, error handling, API s, make, parameter parsing etc Thorns are plug in and swappable modules or collections of subroutines providing both computation infrastructure and communications. 5
% & ' '( "! )) # $ *! ++ * ++! " 6
" # $ % Cactus acts a the main routine of your code Everything Cactus does is contained in thorns (modules). It provides basic API Cactus originated in 1997 as a code for numerical relativity Cactus 4.0 beta : Total rewrite and redesign of code, learning from experiences with previous versions (Tom Goodale) 7
&,!!" # $%& '()*+ )*+,-. (/ '* 0(1 %"2 3 0(1. 2 (0$-4. 3# 54 6 1 6 '*7!2*#4 *(3# * *0 *(*( )6 * # 8* *9' &' 858787. 1*857. 3# 6 8
+&&0 : 1 9
Generalization Grid Programming An Overview of Tool s & Environment :Part-II ' Meta-code that can be applied to any numerical system Abstraction Identify key concepts that can be abstracted Encapsulation Modularize, protect thorn developers from each other Extension Prepare for future concepts and technologies 10
Abstract Flesh API for Grid Programming An Overview of Tool s & Environment :Part-II Driver functions ( storage, communication) Reduction IO, Checkpointing etc ( In general thorns overload or register their capabilities with Flesh, agreeing to provide a function with the correct interface 11
Thorns A thorn usually implements a single specific task Object Orientated Feature Grid Variable, Functions o Private, protected, public Parameters o Private, restricted, global Implements o What does this thorn do? Provides relationships between thorns Inherits o Get all public variables, functions from an implementation an its ancestors 12
) Thorns provide all the real functionality, to add something new just write a new thorn Everything in the flesh can be overloaded and/or extended by thorns 13
* "1 0 " ; =0 ;& 0 * *10 =0 8 8 >> 8 + < 6 ; 14
+ * Each thorn provides 3 configuration files, detailing its interface with the Flesh and with other thorns CCL : Cactus Configuration Language Interface.ccl Implementation, this thorn s variables and variables used from other thorns, functions provided or used from other thorns Param.ccl This thorn s parameters, parameters used and extended from other thorns Schedule.ccl When and how this thorn s routines should be executed, optionally with respect to routines form other thorns 15
* &?0@ 1&( & A %1@0 ;0%$=&=' 6 000@ A %1@1 1 A %@%1& ;(B0 0000; & 0=&= A (0 "0 A 8 0@%" 0 " ) 3 9. 2 00( 39. ' )--- 16
, ((0 "0 (0 "0 ((0 "0 (0 "0 "CD$E 01 *" 17
* ) ) %' 3 0' ) 9 + 0( ' *" ) 1 )1+6. ""+. ""+2* 2 (0. 00 0 01; + (" ; * (0' @"%7+FF @"%+FF 2 00( 200)2002"*200*' 8 39. 39. 39. ( 39. 839. *0B 39. ' " @ ' 0' '. +4'. +430 ' ' *. +4 * ; " %" " 07" 0 " (0" ( "8 +*0 @+*0 @ +*0 @## @) %* @" % @"%## @"%FF @"%G @"%G 2 0 0',(&B 18
* - ;H +6 )*; 0 *(6 +6 (0& +6 6 9& 0+ H +* (0+0%). +)00). +)00+ + @ 2@0H +6 2@0* (06 0 0%H +6 +6 0% 2. " 9 C 0 '.. + ' 11@0B0; 1 &( @0 2=&=;; 2E 2 &2 * (02 2"1 +6 D)**! )6 D200( )6 D-( B). +). - +' )00). 8&+). - 19
: 9 20
+./ E ;B0&1 &00B@; 1&1 ( B; &B% 0(0&? E ((0; @0@0& ; (& 1& &?0 B0% 10B%%I B ;E 21
0 >9+* A. @0(1J" /0K (%; / ;; ((0 & 8 / 1(((0 ((06 &/ >9 B A 0;! 08 0@% 22
1 A 6 0(B0; ;(000 0 0 0 A B%0& L 00$;01M & @ L0 0$1M& 0 1@0? & A & ;&" 0. A ;%; 1( B&&;&;;; A ( @ N@B & 23
&( ; ;&((0 ; 1&10%(B0 (0% % 119 1 90B16. 9 " 1; 1 @& 11 0&11; 1((0&BB0 9 @ = 24
Grid Portals, IDEs & Debuggers 25
A grid portal is a web-based interface to a grid system. Users interact with the portal through a web browser from which they can view files, submit and monitor runs, and view accounting information. 26
Gridscape ) SDSC Gridport Toolkit GRB Portal GENIUS portal for the European Data Grid The Alliance Portal NEESgrid Portal IeSe(Integrated e-science Environment Gridsphere project in Germany NSF NMI Open Grid Computing Environment project in the US 27
*1 1*% Application Specific NorduGrid Grid Monitor GRIDView monitoring tool for the US ATLAS Grid Testbed Non-application specific GridPort GPDK Legion Portal Gridscape 28
It is a tool that enables the rapid creation of interactive and dynamic testbed portals No programming effort required. Gridscape primarily aims to provide a solution for those users who need to be able to create a grid portal but do not necessarily have the time or resources to build a system of their own from scratch. Gridscape itself consists of three individual components a web application a related administrating tool Interface to grid information service They are implemented in Java by following MVC (Model-View- Controller) based, Model-2 type architecture. 29
30
Mr.Rajkumar Buyya, who is an ex-cdacian, is one of the persons behind Gridscape Gridscape has already been used by a number of organisations to create their Grid testbed portals for visualising and monitor resources. Australian Virtual Observatory and UK AstroGrid Collaboration Belle Data Analysis Grid University of Melbourne s World-Wide Grid (WWG) testbed The Gridscape software has been released under GPL license As can be seen, Gridscape uses Globus 31
&& * It is a set of libraries/interfaces that can be utilized for the development of a web portal Gridport toolkit libraries/interfaces have been utilised in the development of NPACI HotPage portal Requires explicit programming effort in order to create a portal Perl based APIs Available as open source Based on Globus Developed by San Diego Supercomputer Center(SDSC) and NPACI 32
The Grid Resource Broker (GRB) is one of the current Globus projects of the HPC Lab of the University of Lecce, Italy, done in collaboration with CACR,California Institute of Technology GRB is a grid portal that allows users to create and handle computational grids using a GUI Based on Globus 21 In order to use GRB, a user must apply to the HPC Lab to get an account 33
Why do we need a Grid IDE? Grid programming environment is critical to the adoption of end users Lack of powerful grid programming environment is an obstacle for wide usage of grid technology Various grid computing tools need to be integrated so that they can be used conveniently 34
*% / Basic programming environment to generate grid applications Tools to search and browse the available grid resources Tools to submit grid jobs Tools to monitor grid job execution and fetch the results 35
It was developed by APSTC Singapore, established by Sun Microsystems'Education & Research LAB The GriDE project aims to provide a flexible,portable and convenient integrated development environment for grid applications 36
37
In addition to the basic functionalities provided in a traditional IDE, the GriDE provides: Script Editor - GUI based resource allocation - Automatically generates RSL Resource Browser - Easy to explore Grid resources - Realtime discovery and monitoring of Grid resources Job Submit - Conveniently submit jobs to any authorized remote location Job Monitoring - Retrieve and monitor detailed job execution information status - User-friendly interface for managing user jobs 38
' Semantic layers in the Grid - Network-level debugging - Middleware-level debugging - Application-level debugging Error definitions - What types of errors make sense at the different levels 39
''-33 Developed by NAS Parallel Tools Group A debugging system for parallel and distributed programs. Works with gdb as its debugging agent. Runs on any system having gdb P2D2 has been demonstrated to be controlling a Globus computation that had processes on 6 machines (some of them geographically distant) where there were: 6 different executables 5 different OS s 4 architectures 40
! 2! *-! % * Verification of MPI standard conformance The MPI calls are traced on each node throughout the whole application Detects deadlocks in MPI programs When detecting a deadlock, the last few calls (as configured by the user) can be traced back on each node. 41
! 2! * 42
!2!* MARMOT uses the profiling interface to intercept the MPI calls of the application for automatic analysis at runtime before they are passed onto the "real" MPI library. It adds a supplementary MPI process working as debug server, for example to handle deadlock detection or the output of errors and warnings. Information between the MPI processes and this additional debug process are transferred using MPI 43
Dimemas is a performance prediction tool for MPI applications for Grid environments Programmers perform predictions of the execution of MPI programs without requiring the use of the target platform. Dimemas reconstructs the time behavior of a parallel application on that target architecture by modeling it with a set of performance parameters. Dimemas is useful in two phases of the life of an MPI application: during its development, to perform analysis of the effect of different parameters in the performance without requiring the use of the target architectures before the production phase, to select the best Grid target!! &-! ' architecture to run the application 44
!!& The output of Dimemas can be simply a text output with the prediction of the elapsed time that will be obtained when the MPI application is run on the target architecture or a visualization tracefile. The output of Dimemas can be used with other performance analysis tools such as Paraver and Vampir. These tools enable the user to examine any performance problems indicated by a simulator run 45
' % Interoperability of Data Parallel libraries and Metachaos Runtime library used to exchange data between data parallel programs written using High Performance Fortran, the Chaos and Multiblock Parti libraries developed at Maryland for handling various types of unstructured problems, and the runtime library for pc++, a data parallel version of C++ from Indiana University Utility : computational aerodynamics problem Computational fluid dynamics (CFD) flow solvers often use different types of meshes to represent different physical structures 46
' % Interoperability of Data Parallel libraries and Metachaos Communicating between two libraries within the same program 47
' % Interoperability of Data Parallel libraries and Metachaos Communicating between two libraries in two different programs 48
' % Interoperability of Data Parallel libraries Unstructured Mesh 0 1 2 Metachaos Structured Mesh 0 1 2 3 49
' % Towards a global compilation system To compile and optimize a Grid-aware application to a target Grid configuration Grid Application Development Software (GrADS) Adapt application to changes in Grid resources Application is encapsulated as a configurable object program, optimized rapidly for execution on specific collection of Grid resources 50
' % Grid Application Development Software GrADS Program Preparation & Execution Architecture 51
Conclusions Grid Programming An Overview of Tool s & Environment :Part-II Whatever toolkit/tool/programming paradigm was designed because there has been a requirement First, problem, second an application, third a Grid, and last tools used. Seeing what most of the researchers need, the tool development took place 52
2 1. Cactus, www.cactuscode.org 2. Cactus and Grid, www.gridlab.org 3. Grid Portals, www.gridcomputing.com 4. Gridscape, http://www.gridbus.org/gridscape/ 5. Gridport, https://gridport.npaci.edu/ 6. GRB Portal, http://sara.unile.it/grb/ 7. GridE, http://apstc.sun.com.sg/gride/ 8. p2d2 A portable Distributed Debugger, science.nas.nasa.gov/groups/tools/projects/p2d2/ 9. MARMOT - tool for analysing and checking MPI-programs, www.hlrs.de/people/mueller/projects/marmot/ 10. DIMEMAS, http://www.cepba.upc.es/dimemas/ 11. G. Edjlali, A. Sussman, J. Saltz, Interoperability of data parallel runtime libraries, in: Proceedings of the Eleventh International Parallel Processing Symposium, IEEE Computer Society Press, 1997. 12. GrADS-Grid Application Development Software Project, http://www.hipersoft.rice.edu/grads/ 53
54