Running LAMMPS on CC servers at IITM

Transcription

1 Running LAMMPS on CC servers at IITM Srihari Sundar September 9, 2016 This tutorial assumes prior knowledge about LAMMPS [2, 1] and deals with running LAMMPS scripts on the compute servers at the computer center in IIT Madras. But, the procedure for building and using the serial and parallel version should work on any system with small modifications here and there. Further, it assumes basic experience using a command line interface, especially bash on a Linux terminal. Installing LAMMPS As a first step download the LAMMPS tarbal from the downloads page on the website and transfer it to the home directory of your account on the server, using winscp from windows or sftp from a Linux terminal. Once this is done login to your account. You should find the file you just transferred if you do an ls. Next, extract the contents of the tarbal using the following command. tar xvf lammps-stable.tar The stable version at the time of writing is from 30th July and hence you should a new folder named lammps-30jul16 or something like lammps-date in general. Move into the src folder within this folder using the following: cd lammps-30jul16/src The src folder has the source code for LAMMPS written in C++ (mostly). The following command gives a bare minimum serialized version of the code: make serial make also works in parallel and you can use multiple cores on the CPU to do this, using: make -j 16 serial To get the parallel version of LAMMPS with MPI, execute this command on the same folder: make -j 16 mpi This requires openmpi or MPICH already installed and configured for your compiler. You can execute the above on GNR/VIRGO out of the box. Next, create a soft link of the executables to a bin folder in your home directory. Add the bin folder to your bash search path ($PATH). Make this permanent by adding a line to the.bashrc file in the home directory which sets the environment for bash. Execute the following commands one by one on the terminal to do this. 1

2 mkdir $HOME/bin/ ln -s `readlink -f lmp_serial` $HOME/bin/ ln -s `readlink -f lmp_mpi` $HOME/bin/ echo "export PATH=\$PATH:\$HOME/bin/" >> $HOME/.bashrc source $HOME/.bashrc NOTE 1. The readlink -f... part of the command should be within backticks (The symbol below tilde on the key below Esc key on standard keyboards). On shell, a part of a command which is within backticks is executed by the shell and the output from this part is substituted in the whole command. If the backtick does not get copied from the pdf file, type the command out 2. Run all this from the src folder Now, executing lmp_serial from any folder should print this on the terminal : LAMMPS (30 Jul 2016) This shows that the build process was successful. Kill (ctrl-c) this process for now. Executing LAMMPS scripts on head node From the src directory we move to a directory with an example file for simulating crack propagation. You can use any LAMMPS input file in any directory (within your home folder) for this and replace/modify the commands appropriately. cd../examples/crack/ Listing the files should show the presence of a file named in.crack. Run this simulation using the serial version of LAMMPS using: lmp_serial < in.crack This should run and output some details of the simulation, with the last line being: Total wall time: 0:00:06 The time may vary according to the load on the system. This shows that the simulation completed successfully. Now, in order to use the parallel version of LAMMPS we need to setup some environment variables for getting the libraries right. For this open the.bashrc file from the home directory in vi. vi $HOME/.bashrc If using GNR, paste the following line at the end of the file and save it. source /Apps/intel-2016-up3/bin/compilervars.sh intel64 2

3 If using VIRGO, paste the following line at the end of the file and save it. source /IITM_GPFS_FS1/sware/intel2016/bin/compilervars.sh intel64 Source the environment file so that the changes are reflected in the current session too. source $HOME/.bashrc Now executing the next statement will give a similar output as using lmp_serial. mpirun -np 16 lmp_mpi < in.crack But, there should be a significant reduction in the total wall time. The previous statement indicates that 16 threads should be used to run the simulation. Job submission to a compute node on GNR While you can use the compute node to get your input script error free, it is never advisable to actually run your jobs on it. For this you should queue your jobs on the scheduler. For this purpose, GNR uses the PBS job scheduling software. So, you should write a job submission script and provide it to the software. Serial execution: Create and open a file called job serial.cmd using vim vi job_serial.cmd Paste the following in that file and save it. #! /bin/bash #PBS -o logfile.log #PBS -e errorfile.err #PBS -l cput=10:00:00 #PBS -l select=1:ncpus=1 tpdir=`echo $PBS_JOBID cut -f 1 -d.` tempdir=$home/work/job$tpdir mkdir -p $tempdir cd $tempdir cp -R $PBS_O_WORKDIR/*. lmp_serial < in.crack mv../job$tpdir $PBS_O_WORKDIR/. From the shell execute this: qsub job_serial.cmd This should give you an output of the form number.gnr. Running qstat should show that the job is running against that number. Once the job finishes the system moves the directory where the job executed, named jobnumber to the current folder along with two other files. The errorfile.err displays any error with execution and the logfile.log shows the stdout when the LAMMPS script executed. The folder should have all your outputs. 3

4 Parallel execution: For parallel execution the procedure is very similar except for small modifications to the job submission scripts. Create a file named job parallel.cmd and paste the following in the file: #! /bin/bash #PBS -o logfile.log #PBS -e errorfile.err #PBS -l cput=10:00:00 #PBS -l select=1:ncpus=16 tpdir=`echo $PBS_JOBID cut -f 1 -d.` tempdir=$home/work/job$tpdir mkdir -p $tempdir cd $tempdir cp -R $PBS_O_WORKDIR/*. mpirun -np 16 lmp_mpi < nve.in mv../job$tpdir $PBS_O_WORKDIR/. Execute with: qsub job_parallel.cmd You should see nearly same output, but in a shorter time. Job submission to a compute node on VIRGO While GNR uses PBS for job scheduling, VIRGO uses Load Leveler for this and we need to write LL batch job scripts. Serial execution: Create and open a file called job serial.cmd using vim vi job_serial.cmd Paste the following in that file and save it. #!/bin/bash #@ output= test.out #@ error= test.err #@ job_type= serial #@ class= Medium #@ environment = COPY_ALL #@ queue Jobid=`echo $LOADL_STEP_ID cut -f 6 -d.` tmpdir=$home/scratch/job$jobid mkdir -p $tmpdir; cd $tmpdir cp -R $LOADL_STEP_INITDIR/* $tmpdir lmp_serial < in.crack mv../job$jobid $LOADL_STEP_INITDIR 4

5 From the shell execute this: llsubmit job_serial.cmd This should give you an output of the form c1hn1.job-id Running llq job-id should show that the job is running against that number. Once the job finishes the system moves the directory where the job executed, named jobjob-id to the current folder. Parallel execution: For parallel execution the procedure is very similar except for small modifications to the job submission scripts. Create a file named job parallel.cmd and paste the following in the file: #!/bin/bash #@ output= test.out #@ error= test.err #@ job_type= MPICH #@ class= Medium #@ node = 1 #@ tasks_per_node = 16 #@ environment = COPY_ALL #@ queue Jobid=`echo $LOADL_STEP_ID cut -f 6 -d.` tmpdir=$home/scratch/job$jobid mkdir -p $tmpdir; cd $tmpdir cp -R $LOADL_STEP_INITDIR/* $tmpdir mpirun -np 16 lmp_mpi < in.crack mv../job$jobid $LOADL_STEP_INITDIR Execute with: llsubmit job_parallel.cmd Logging into the CC website ( and looking at the Virgo info under the hpce tab in the right side bar will give you more options you can use with the Load Leveller. Useful tools Some tools which will make life easier for you (and others): 1. Google: Straight forward. Any error you see, google it up. Someone would have encountered the same error and posted on stack overflow or on the forums for the software. A person with more experience or the software developers would have solved it. Google indexes all of this and so you have you solution then and there. This works most of the times. 2. GNU Screen 3. Bash scripting 5

6 There are numerous tutorials for 2 and 3 online and should be easily accessible with minimal Linux experience. Bash scripting especially will help your productivity. You can have one script for creating input scripts, executing it and post processing data. Go figure! Acknowledgment While I ve wanted to make this tutorial to make life easier for myself (as a teaching assistant), I was spurred on to get it done by Hari Haran. A big portion of this tutorial comes from the chat I had with him and also he provided me access to his GNR account as mine expired. Also thanks to Darshan for letting me use his VIRGO account. References [1] LAMMPS Molecular Dynamics Simulator. [2] S. Plimpton, P. Crozier, and A. Thompson. Lammps-large-scale atomic/molecular massively parallel simulator. Sandia National Laboratories, 18, Disclaimer: The author takes no responsibility for any part of this tutorial failing to work for you. Do not try this on the day before project deadline. Corrections: I ll be grateful if you can mail any corrections and suggestions to sriharisundar95 at gmail dot com. 6