Parallel programming in Matlab environment on CRESCO cluster, interactive and batch mode Authors: G. Guarnieri a, S. Migliori b, S. Podda c a ENEA-FIM, Portici Research Center, Via Vecchio Macello - Loc. Granatello - 80055 Portici (Naples) b ENEA-FIM, Enea-Sede, Lungotevere Thaon di Revel n. 76, 00196 Roma, c ENEA-FIM, Frascati Research Center, Via E. Fermi 45, 00044 Frascati (Roma) Introduction Matlab (MATrix LABoratory) provides an environment for numeric elaboration and an interpreted programming language widely used in science and technology. The package is equipped with specialized libraries (toolboxes) which can solve problems in several fields. For some years the implementation of parallel programming in Matlab environment has been under study. Mathworks, the producer of Matlab, distributes toolboxes with this target, but there are also third party projects with the same aim. Among them is the MatlabMPI library developed by Lincoln Laboratories of the Massachusetts Institute of Technology. ENEA GRID, by the introduction of CRESCO machines, has achieved a very high computational power. So, the possibility of parallel programming in Matlab on this structure becomes a meaningful aim. We have chosen to modify and install the MatlabMPI library for Linux machines of ENEA GRID and particularly for CRESCO cluster. This functions library, written in pure Matlab language, implements a subset of the functions defined in the MPI standard and allows to run, at the same time, several Matlab processes both on multicore machines and computer clusters. Data exchange among the processes does not happen by real message sending and receiving, but via file system access operations. Some code changes were required to install MatlabMPI on ENEA GRID and to adapt the use of the library to CRESCO submission mode. To enable the launching of Matlab jobs on remote machines in interactive such as in batch mode we have resorted to the ENEA GRID submission tool. The use of this library along with the multithreading programming mode on multicore nodes, enabled in latest versions of Matlab, also permit to enforce a hybrid programming mode on CRESCO cluster. The user can exploit the parallelism assured by multithreading on multicore nodes and implement the communication among the nodes with MatlabMPI. MatlabMPI. ("Parallel Programming with MatlabMPI", Jeremy Kepner, Proceedings of the High Performance Embedded Computing (HPEC 2001) workshop, 25-27 Sep 2001, MIT Lincoln Laboratory, Lexington, MA). MatlabMPI library contains the following items: MPI_Run MPI_Init MPI_Comm_size MPI_Comm_rank MPI_Send MPI_Recv MPI_Finalize MPI_Abort MPI_Bcast MPI_Probe MPI_cc MatMPI_Comm_dir MatMPI_Save_messages MatMPI_Delete_all MatMPI_Comm_settings MatMPI_Buffer_file MatMPI_Lock_file MatMPI_Commands MatMPI_Comm_init MatlabMPI: Message exchange. The send operation consists of storing the data to send in a file in a common directory that all processes can access. The name of this file contains the information about the message to exchange, the sending process and the receiving one. The receive operation, instead, consists of detecting the file from which to load the message by means of the information contained in its name, of waiting for the end of send operation and finally of loading the data with a reading operation. Fig.2. How to launch in interactive mode a session with MatlabMPI on ENEA GRID machines. MatlabMPI on CRESCO cluster. CRESCO HPC system consists of more than 2700 cores, divided into three main sections. This cluster has been integrated in ENEA GRID. The main software components are the LSF scheduler, the openafs distributed file system and the integrated Kerberos 5 authentication. There are eight front-end nodes by which users can access the system and submit their jobs on computational nodes. In order to adapt MatlabMPI library to submit jobs on CRESCO cluster in batch mode it was necessary to further modify the code. We have created some scripts to interface the library with the scheduler environment. Sender Process Receiver Process Send operations Receive operations MPI_Send MPI_Recv Data File Variable Save operation Common file system Load operation Variable Fig.3. How to launch in batch mode a session with MatlabMPI on CRESCO cluster from a front-end command line. We submit with bsub command the script matmpilauncher.sh on the default queue small_10m. Fig.1. Graphic description of a message exchange between two parallel Matlab processes using MatlabMPI functions MPI_Send and MPI_Recv. matmpilaucher.sh LSF Scheduled Resources Since the library consists of pure Matlab code, it s possibile to use it involving heterogeneous sets of platforms as long as they support Matlab environment. On a shared memory system a single licence is enough to use all available cores, whereas on distributed memory systems the user needs a licence for every engaged node. However it is possibile to compile Matlab scripts with the mcc command, in this way only one Matlab licence is enough on distributed memory systems too. MatlabMPI on ENEA GRID. ENEA GRID is based on AFS distributed fyle system, which allows users to access their home directory from every point of the grid. This supports the use of MatlabMPI because the parallel Matlab processes can simply share a common working directory. The library is presently installed on ENEA GRID and it s available for all Linux machines and for the following Matlab releases: 1) 7.0.1 24704 (R14) 2) R2007a 3) R2008a We have modified in some points the original code to adapt the use of MatlabMPI to our computational grid. Primarily, to enable the launching of Matlab jobs on remote machines, allowing them to access the AFS file system without losing reading and writing permissions, or to be more precise to ensure token preservation for the user, we have resorted to the ENEA-GRID submission tool. Moreover, in ENEA there are a limited number of Matlab licences and therefore at run time the number of available ones may be insufficient to launch all required Matlab processes, this could cause an error in the elaboration that MatlabMPI cannot detect. We have introduced a shell script that checks the number of available Matlab licences; if this number is insufficient the program exits with an appropriate error message. This script is called by MPI_Run before the stage of processes spawning. MPI_Run function was also modified in order to detect the version of Matlab that is in use and to check the number of available licences for the appropriate version. In this way the user can avoid the risk of creating pending processes with subsequent deadlock errors. References. http://www.mathworks.com http://www.ll.mit.edu/matlabmpi/ http://www.ll.mit.edu/pmatlab/ MATLAB matmpiscript.m MatlabMPI + blaunch.sh Fig.4. The script matmpilauncher.sh reads some LSF environmental variables in which are contained the scheduled resources for the request expressed in the command line of Fig.3 and launch Matlab with the M-File matmpiscript.m. The latter passes the scheduled resources as argument to MatlabMPI functions. In batch mode the launch of the parallel Matlab jobs happens by means of blaunch command. Hybrid programming mode on CRESCO cluster - MatlabMPI+. MatlabMPI Fig.5. As the CRESCO architecture is distributed with multicore nodes the best parallel programming style is the hybrid one. This technique combines together multithreading on shared memory machines and message passing for comunication among nodes. The latest Matlab versions enable multithreading on a single multicore node thus the MatlabMPI library fully exploits the CRESCO capabilities.
Fig.3 Benefits of multicluster queue.
ENEA-FIM, C.R. Portici
References http://www.cresco.enea.it/la1/cresco_ sp12_graf3d/