MPI Applications with the Grid Engine

Transcription

1 The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) MPI Applications with the Grid Engine Riccardo Rotondo Joint CHAIN/EUMEDGRID-Support/EPIKH School to Science Gateways Amman, Jordan

2 Outline What is MPI? MPI Challenges on GRID MPI Interface MPI embedded in your GRID Engine MPI Portlet Development 2 2

3 Message Passing Interface What is MPI? A standard defining a syntax and semantics useful for righting message-passing appications. Why MPI? Heavy usage of CPU power in HPC; Development of portable and scalable large parallel applications. 3 3

4 MPI in GRID There is no standard way of starting an MPI application No common syntax for mpirun, mpiexec support optional The cluster where the MPI job is supposed to run doesn't have a shared file system How to distribute the binary and input files? How to gather the output? Different clusters over the Grid are managed by different Local Resource Management Systems (PBS, LSF, SGE, ) Where is the list of machines that the job can use? What is the correct format for this list? How to compile MPI program? How can a physicist working on Windows workstation compile his code for/with an Itanium MPI implementation? 4 4

5 MPI-Start Specifies a unique interface to the upper layer to run a MPI job Allow the support of new MPI implementations without modifications in the Grid middleware Support of simple file distribution Provide some support for the user to help manage his data Grid Middleware MPI-START MPI Resources 5 5

6 MPI-Start mpi-start is a recommended solution to hide the implementation details for jobs submission. The design of mpi-start was focused in making the MPI job submission as transparent as possible from the cluster details! Using the mpi-start system requires the user to define a wrapper script that set the environment variables and a set of hooks. 6 6

7 MPI-Start design goals Portable The program must be able to run under any supported operating system Modular and extensible architecture Plugin/Component architecture Relocatable Must be independent of absolute path, to adapt to different site configurations Remote injection of mpi-start along with the job Remote debugging features 7 7

8 MPI-Start Architecture CORE Scheduler Execution Hooks PBS/Torque SGE LSF Open MPI MPICH2 LAM PACX Local User Compiler File Dist. 8 8

9 MPI-Start Flow NO NO START Scheduler Plugin Do we have a scheduler plugin for the current environment? Ask Scheduler plugin for a machinefile in default format Execu2on Plugin Do we have a plugin for the selected MPI? Activate MPI Plugin Prepare mpirun Trigger pre-run hooks Start mpirun Dump Env Trigger post-run hooks EXIT Hooks Plugins 9 9

10 Using MPI-Start Interface with environment variables: I2G_MPI_APPLICATION: The executable I2G_MPI_APPLICATION_ARGS: The parameters to be passed to the executable I2G_MPI_TYPE: The MPI implementation to use (e.g openmpi,...) I2G_MPI_VERSION: Specifies which version of the the MPI implementation to use. If not defined the default version will be used 10 10

11 Using MPI-Start More variables: I2G_MPI_PRECOMMAND Specifies a command that is prepended to the mpirun (e.g. time). I2G_MPI_PRE_RUN_HOOK Points to a shell script that must contain a pre_run_hook function. This function will be called before the parallel application is started (usage: compilation of the executable) I2G_MPI_POST_RUN_HOOK Like I2G_MPI_PRE_RUN_HOOK, but the script must define a post_run_hook that is called after the parallel application finished (usage: upload of results)

12 Using MPI-Start " The script: ~]$ cat test2mpistart.sh #!/bin/sh # This is a script to show how mpi-start is called # Set environment variables needed by mpistart export I2G_MPI_APPLICATION=/bin/ hostname export I2G_MPI_APPLICATION_ARGS= export I2G_MPI_NP=2 export I2G_MPI_TYPE=openmpi export I2G_MPI_FLAVOUR=openmpi export I2G_MPI_JOB_NUMBER=0 export I2G_MPI_STARTUP_INFO=/home/ imain179 export I2G_MPI_PRECOMMAND=time export I2G_MPI_RELAY= export I2G_MPI_START=/opt/i2g/bin/mpi-start " The submission (in SGE): [imain179@i2g-ce01 ~]$ qsub -S /bin/bash -pe openmpi 2 -l allow_slots_egee=0./test2mpistart.sh " The StdOut: [imain179@i2g-ce01 ~]$ cat test2mpistart.sh.o Scientific Linux CERN SLC release 4.5 (Beryllium) Scientific Linux CERN SLC release 4.5 (Beryllium) lflip30.lip.pt lflip31.lip.pt " The StdErr: [lflip31] /home/imain179 > cat test2mpistart.sh.e Scientific Linux CERN SLC release 4.5 (Beryllium) Scientific Linux CERN SLC release 4.5 (Beryllium) real 0m0.731s user 0m0.021s sys 0m0.013s # Execute mpi-start $I2G_MPI_START MPI commands are transparent to the user No explicit mpiexec/mpirun instruc2on Start the script via normal LRMS submission 12 12

13 Wrapper Script for MPI Start #!/bin/bash # Pull in the arguments. MY_EXECUTABLE=`pwd`/$1 MPI_FLAVOR=$2 # Convert flavor to lowercase for passing to mpi-start. MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR tr '[:upper:]' '[:lower:]'` # Pull out the correct paths for the requested flavor. eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH` # Ensure the prefix is correctly set. Don't rely on the defaults. eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH export I2G_${MPI_FLAVOR}_PREFIX # Touch the executable. #It exist must for the shared file system check. # If it does not, then mpi-start may try to distribute the executable # when it shouldn't. touch $MY_EXECUTABLE # Setup for mpi-start. export I2G_MPI_APPLICATION=$MY_EXECUTABLE export I2G_MPI_APPLICATION_ARGS= export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh The script takes 2 argument the executable name and the MPI flavor The script refers to the Hooks scripts # If these are set then you will get more debugging information. export I2G_MPI_START_VERBOSE=1 #export I2G_MPI_START_DEBUG=1 # Invoke mpi-start. $I2G_MPI_START 13 13

14 Wrapper Script for MPI Start #!/bin/bash # Pull in the arguments. MY_EXECUTABLE=`pwd`/$1 MPI_FLAVOR=$2 # Convert flavor to lowercase for passing to mpi-start. MPI_FLAVOR_LOWER=`echo $MPI_FLAVOR tr '[:upper:]' '[:lower:]'` # Pull out the correct paths for the requested flavor. eval MPI_PATH=`printenv MPI_${MPI_FLAVOR}_PATH` # Ensure the prefix is correctly set. Don't rely on the defaults. eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH export I2G_${MPI_FLAVOR}_PREFIX # Touch the executable. #It exist must for the shared file system check. # If it does not, then mpi-start may try to distribute the executable # when it shouldn't. touch $MY_EXECUTABLE # Setup for mpi-start. export I2G_MPI_APPLICATION=$MY_EXECUTABLE export I2G_MPI_APPLICATION_ARGS= export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh The script takes 2 argument the executable name and the MPI flavor The script refers to the Hooks scripts # If these are set then you will get more debugging information. export I2G_MPI_START_VERBOSE=1 #export I2G_MPI_START_DEBUG=1 # Invoke mpi-start. $I2G_MPI_START 14 14

15 Outline Resource Manager worker node 15 15

16 MPI Portlet MPI Portlet Develop 16 16

17 cpunumber in mpi_portlet.java A global variable representing the number or the cpu can be set by user String cpunumber; // Number of cpu will execute the mpi script in parallel Reading the CPU number from the user request public void getinputform(actionrequest request) { List items = upload.parserequest(request); cpunumber=item.getstring(); // or cpunumber=(string)request.getparameter("cpunumber"); } 17 17

18 cpunumber in mpi_portlet.java Using the cpunumber submitting the job: public void submitjob() { String arguments="mpi-start-wrapper.sh cpi mpich2"; JSagaJobSubmission tmpjsaga = new JSagaJobSubmission(); tmpjsaga.settotalcpucount(cpunumber); } 18 18

19 cpunumber input.jsp Using the cpunumber submitting the job: <form enctype="multipart/form-data" action="<portlet:actionurl portletmode="view"><portlet:param name="portletstatus" value="action_submit"/></portlet:actionurl>" method="post"> <p>insert <b>cpu number</b></p> <textarea id="cpunumber" rows="1" cols="60%" name="cpunumber">4</textarea> </form> 19 19

20 mpi-start, mpi-hooks, mpi-app Setting the location of the mpi scripts and mpi application. public void submitjob() { String executable="/bin/sh"; String arguments="mpi-start-wrapper.sh cpi mpich2"; String inputsandbox= appserverpath+"/web-inf/job/ pilot_script.sh +","+appserverpath+"/web-inf/job/cpi.c" +","+appserverpath+"/web-inf/job/mpi-hooks.sh" +","+appserverpath+"/web-inf/job/mpi-start-wrapper.sh" +","+inputsandbox_inputfile; } 20 20

21 Grid Option in portlet.xml In portlet.xml can set some default preferences that can be recall in the java code: <portlet> <portlet-name>mpi-portlet</portlet-name> <display-name>mpi-portlet</display-name> <portlet-class>it.infn.ct.mpi_portlet</ portlet-class> <init-param> <name>init_jobrequirements</name> <value> Member("MPI-START", other.gluehostapplicationsoftwareruntimeenvironment); Member("MPICH", other.gluehostapplicationsoftwareruntimeenvironment) </value> </init-param> </portlet> 21 21

22 Grid Option in mpi-portlet.java First we need to get the value from portlet preferences: private void getpreferences( ActionRequest actionrequest, RenderRequestrenderRequest) { pref_jobrequirements=prefs.getvalue("pref_jobrequirement s",init_jobrequirements); } 22 22

23 Grid Option in mpi-portlet.java Now set the Job Requirements in java code to submit the job: public void submitjob() { String jdlrequirements[] = pref_jobrequirements.split(";"); int numrequirements=0; for(int i=0; i<jdlrequirements.length; i++) { if(!jdlrequirements[i].equals("")) { jdlrequirements[numrequirements] = "JDLRequirements=("+jdlRequirements[i]+")"; numrequirements++; tmpjsaga.setjdlrequirements(jdlrequirements); } 23 23

24 References References Gilda Trainin Material: Science Gateway developer page: %20Gateway%20Developer%20Pages MPI Standalone wiki: GridEngineMPIStandaloneCode 24 24

25 Thank you for your attention Special thanks to Mario Reale 25 25