Enabling Legacy Applications on Service Grids. Asvija B, C-DAC, Bangalore

Transcription

1 Enabling Legacy Applications on Service Grids Asvija B, C-DAC, 1

2 Legacy Applications Service enablement of existing legacy applications is difficult No source code available Proprietary code fragments Production applications Effort Wastage in rewriting A possible option is to give Service Frontends 2

3 Service Wrappers Easiest possible approach to enable many legacy applications as Grid Services However not a panacea for all applications Each application requires detailed analysis and study before service enablement Outline of writing service wrappers with MM5 as case study application. 3

4 What is MM5? MM5 - Mesoscale Meteorological Model Version 5 Atmospheric Model to simulate or predict mesoscale atmospheric circulation Mesoscale Study ( Typical spatial scales between 10 and 1000 km ) Examples of Mesoscale phenomena Thunderstorms, gap winds, downslope windstorms, land-sea breezes, and squall lines 4

5 5

6 What is MM5? Developed at Penn State and NCAR (National Centre for Atmospheric Research) USA as a community mesoscale model with contributions from users worldwide Currently MM5 software is freely provided and supported by the Mesoscale Prediction Group in the Mesoscale and Microscale Meteorology Division, NCAR 6

7 7

8 Model Details Input Data Required - Topography and landuse (in categories); Gridded atmospheric data that have at least these variables: sea-level pressure, wind, temperature, relative humidity and geopotential height; and at these pressure levels: surface, 1000, 850, 700, 500, 400, 300, 250, 200, 150, 100 mb; Observation data that contains soundings and surface reports. 8

9 Model Details Mostly Fortran programs (Major portion of code) + Some code in C F77 and F90 code Requires Intel s ifort compilers on Linux Machines 9

10 Why Does MM5 require Grid? Huge Input and Output Data Sizes (in the order of GBs) Large Execution Time (typically in hours even for smaller data sets) Real Time Simulation and Predictions are required Data Sets are distributed worldwide and require Online download and Processing Application ideally suited to be packaged as a Grid Workflow 10

11 MM5 I/P and O/P file sizes Size File Name BDYOUT_DOMAIN BDYOUT_DOMAIN LOWBDY_DOMAIN LOWBDY_DOMAIN MMINPUT_DOMAIN MMINPUT_DOMAIN2 11

12 MM5 I/P and O/P file sizes Size File Name TERRAIN_DOMAIN TERRAIN_DOMAIN MMOUT_DOMAIN MMOUT_DOMAIN2 12

13 Steps in building the wrapper Step A - Identify the actual components of the application Selecting the different components of your application that you wish to make available as a service Identifying the Input / Output for each of the components. Identify the procedure for invoking each of these components. Identify the sequence in which these components have to be invoked Draw a flow chart of the sequence of operations 13

14 Steps in building the wrapper Step B - Implement the core workflow From the abstract flowchart of the model, we shall write shell scripts which perform these individual actions on an identified Cluster system. We shall also make use of the Grid specific commands for doing data transfers from our workflow system. 14

15 Our Planned Workflow Input Data MM5 (MPI Version) GRAPH Visualization Utilities Visual Output 15

16 Planning Data Staging Input Data staging Use the Grid file transfer commands to stage in data to the remote machine Multiple input files can be archived into a single tar archive and copied & later expanded Output Data staging Output data and the visualization data have to be staged back to the end user submit node. 16

17 Planning Core Workflow Start Step 1. Create a temporary working directory based on random number input and copy the executables to it. Step 2. Do a Grid File transfer for staging in (copying) the Input Data to the current working directory & Expand the tar archive Step 3. Run the MM5 Model (MM5 Version) Input Data Staging Running the Model 17

18 Planning Core Workflow Step 4. Run the GRAPH Utility Step 5. Run the CTRANS Utility to get a Postscript file Step 6. Run the PS2PDF Utility to convert into a PDF file Step 7. Do a Grid File transfer for staging out (copying) the Output PDF file to the user specified location Post Process Output Data Output Data Staging Stop 18

19 Implementing Core Workflow Write a shell script for the workflow /usr/local/mpich-intel/bin/mpirun -np $no_of_nodes -machinefile hostfile mm5.mpp # Run the GRAPH Utility./graph.csh 1 1 MMOUT_DOMAIN1 &> /dev/null.. 19

20 Service Wrapper Design Inputs for the Service Input Host name Name of the host where the input files are located Input Data File name - Absolute path of the input tar archive on the Input host Output Host name Name of the host where the output files are to be copied Output Data Dir - Absolute path of the directory on Output host where the output data has to be copied. 20

21 WSDL for input type <xsd:element name="run"> <xsd:complextype> <xsd:sequence> <xsd:element name="inputhost" type="xsd:string" /> <xsd:element name="inputfile" type="xsd:string" /> <xsd:element name="outputhost" type="xsd:string" /> <xsd:element name="outputdir" type="xsd:string" /> </xsd:sequence> </xsd:complextype> </xsd:element> 21

22 Implementing the server StringBuffer commandbuf = new StringBuffer(); commandbuf.append("/usr/local/mm5/mm5.sh "); commandbuf.append(randint); commandbuf.append(" ").append(inputhost); commandbuf.append(" ").append(inputfile); commandbuf.append(" ").append(outputhost); commandbuf.append(" ").append(outputdir); String command = commandbuf.tostring(); Process p = Runtime.getRuntime().exec(command); // Run the command 22

23 Building & Deploying Building and deploying./globus-build-service.sh MM5 globus-deploy-gar in_cdac_mm5_services.gar 23

24 Accessing the Service Service can be accessed using the URL similar to this : srf/services/mm5/mm5service (hyd01.hardware.cdac.ernet.in is the server name) 24

25 How to invoke Grid Services? 25

26 Deployed Service Model 26

27 Service Details Service URL - MM5/MM5Service MM5 Distributed Memory (DM) Parallel Option (MPP) Runs using MPICH v Deployed at resources in CDAC-Hyderabad 8 Nodes running on Linux Red Hat EL 4 Intel Xeon Dual Core CPUs GHz 27

28 Service Details Deployed as a WSRF Service Deployed in the standard GT4 Container Stdout and StdErr of the application are exposed as WSRF ResourceProperties Implements the WS-Notification (WSN) for StdOut and StdErr changes Clients can Subscribe for streaming StdOut and StdErr Clients can also Subscribe for Program Exit Event 28

29 Service Details Service Input No. of Nodes to Use Input Host and Input File Path Output Host and Output File Path I/P Files : Model initial condition file(s): MMINPUT_DOMAINx Lateral and lower boundary condition files for the coarsest domain: BDYOUT_DOMAIN1, LOWBDY_DOMAINx Nest terrain file(s) from program TERRAIN: TERRAIN_DOMAIN2, 3, etc O/P Files : MMOUT_DOMAIN files Post-Processed files from GRAPH-CTRANS gmeta.pdf 29

30 30