How to compile Fortran program on application server

Transcription

1 How to compile Fortran program on application server Center for Computational Materials Science, Institute for Materials Research, Tohoku University version 1.0

2 Contents 1. Compile How to use GNU Fortran How to use Intel Fortran How to use PGI Fortran Library MKL ACML IMSL Fortran FFTW... 9

3 Summary of changes version Date Changes version /10/2015 Initial version

4 How to compile Fortran program on application server We explain how to compile Fortran program on IBM BladeCenter HS22 (hereafter called the application server ). 1. Compile GNU Fortran, Intel Fortran and PGI Fortran are available on application server. The compile command and option depend on the parallelization scheme. We describe how to compile a source program parallelized by the following scheme. Automatic parallelization: Parallelize using auto-threading function of compiler. OpenMP:Parallelize using OpenMP directives. MPI:Parallelize using MPI. MPI and automatic parallelization: Parallelize using MPI and each process is multi-threaded by auto-threading function of compiler. MPI and OpenMP:Parallelize using MPI and each process is multi-threaded by OpenMP directives. You can compile a source program on the front-end node (app1.imr.edu or app2.imr.edu) or TSS node (app3t.imr.edu or app4t.imr.edu). The CPU time limitation of the front-end node is 1 hour and that of the TSS node is 30 hours. If it takes more than 1 hour to compile one source program, you must compile it on the TSS node or create a script file and submit it from the front-end node. 1.1 How to use GNU Fortran Compile commands of GNU Fortran: Command Purpose gfortran Compiles serial source program. /usr/bin/mpif77 Compiles MPI source program written in FORTRAN 77. /usr/bin/mpif90 Compiles MPI source program written in Fortran 90. File types: Suffix Format f, for Fixed format f90, f95 Free format Followings are some examples of compiling a source program on app1.imr.edu. You can also compile a source program on TSS node. (Example) Compile a serial source program. [user01@app1~]$ gfortran sample.f (Example) Compile a serial source program using auto-threading function. [user01@app1~]$ gfortran -O3 -march=native -mtune=native -ftree-parallelize-loops=4 sample_auto.f The number of parallel loop can be specified by the ftree-parallelize-loops option. 1

5 (Example) Compile a serial source program using OpenMP. [user01@app1~]$ gfortran -fopenmp sample_omp.f (Example) Compile a MPI source program. [user01@app1~]$ source /export/common/environment/app_mpi_environment_mpich2 [user01@app1~]$ /usr/bin/mpif90 sample_mpi.f (Example) Compile a MPI source program and each process is multi-threaded by OpenMP. [user01@app1~]$ source /export/common/environment/app_mpi_environment_mpich2 [user01@app1~]$ /usr/bin/mpif90 -fopenmp sample_mpi_omp.f The following is an example how to submit a job which compiles a source program. Create a job script file (example: script.sh) using text editor such as vi command first. gfortran sample.f Submit the job script file to ASB queue using the submit command. See the about the syntax of the submit command and for the details of queues in How to execute a program on application server. [user01@app1 ~]$ submit ASB general -exec script.sh You can omit the -exec option if you create the job script file named COMMANDLINE. [user01@app1 ~]$ submit ASB general Main options: Option Description -o outfile Specifies the load module file name. If this is not specified, the load module file name is a.out. -L<dir> Specifies the path name of the directory containing the library you want to retrieve. -l<lib name> Specifies the library to link. -I<dir> Specifies the directory that contains the source statement library. -O0 -O1 -O2 -O3 Specifies the optimizing level. If this is not specified, the system assumes that -O0 has been specified. -ftree-parallelize-loops=n Enables parallelization of source program. n is the number of parallelized loop. -fopenmp Enables parallelization of source program by using OpenMP directives. -ffree-form Indicates that the source program is in free format. -ffixed-form Indicates that the source program is in fixed format. 2

6 1.2 How to use Intel Fortran Compile commands of Intel Fortran: Command purpose ifort Compiles serial source program. mpiifort Compiles MPI source program. File types: Suffix Format f, for, ftn, i Fixed format f90, i90 Free format Followings are some examples of compiling a source program on app1.imr.edu. You can also compile a source program on TSS node. (Example) Compile a serial source program. [user01@app1~]$ ifort sample.f (Example) Compile a serial source program using auto-threading function. [user01@app1~]$ ifort -parallel sample_auto.f (Example) Compile a serial source program using OpenMP. [user01@app1~]$ ifort -openmp sample_auto_omp.f (Example) Compile a MPI source program. [user01@app1~]$ source /opt/intel/impi/ /bin64/mpivars.sh [user01@app1~]$ mpiifort sample_mpi.f (Example) Compile a MPI source program and each process is multi-threaded by auto-threading function. [user01@app1~]$ source /opt/intel/impi/ /bin64/mpivars.sh [user01@app1~]$ mpiifort -parallel sample_mpi_auto.f (Example) Compile a MPI source program and each process is multi-threaded by OpenMP. [user01@app1~]$ source /opt/intel/impi/ /bin64/mpivars.sh [user01@app1~]$ mpiifort -openmp sample_mpi_omp.f The following is an example how to submit a job which compiles a source program. Create a job script file (example: script.sh) using text editor such as vi command first. ifort sample.f 3

7 Submit the job script file to ASB queue using the submit command. See the about the syntax of the submit command and for the details of queues in How to execute a program on application server. [user01@app1 ~]$ submit ASB general -exec script.sh You can omit the -exec option if you create the job script file named COMMANDLINE. [user01@app1 ~]$ submit ASB general Main options: Option Description -o outfile Specifies the load module file name. If this is not specified, the load module file name is a.out. -L<dir> Specifies the path name of the directory containing the library you want to retrieve. -l<lib name> Specifies the library to link. -I<dir> Specifies the directory that contains the source statement library. -O -O0 -O1 -O2 -O3 -Ofast Specifies the optimizing level. If this is not specified, the system assumes that -O2 has been specified. -parallel Enables parallelization of source program. -openmp Enables parallelization of source program by using OpenMP directives. -free Indicates that the source program is in the free format. -fixed Indicates that the source program is in the fixed format. -xhost Generates instructions for the highest instruction set available on the host processor. 1.3 How to use PGI Fortran Compile commands of Intel Fortran: Command purpose pgf77 Compiles serial source program written in FORTRAN 77. pgfortran Compiles serial source program. mpif77 Compiles MPI source program written in FORTRAN 77. mpif90 Compiles MPI source program written in Fortran 90. File types: Suffix Format f, for Fixed format f90 Free format Followings are some examples of compiling a source program on app1.imr.edu. You can also compile a source program on TSS node. (Example) Compile a serial source program. [user01@app1~]$ pgfortran sample.f 4

8 (Example) Compile a serial source program using auto-threading function. [user01@app1~]$ pgfortran -Mconcur sample_auto.f (Example) Compile a serial source program using OpenMP. [user01@app1~]$ pgfortran -mp sample_auto_omp.f (Example) Compile a MPI source program. [user01@app1~]$ source /export/common/environment/app_mpi_environment_pgi [user01@app1~]$ mpif90 sample_mpi.f (Example) Compile a MPI source program and each process is multi-threaded by auto-threading function. [user01@app1~]$ source /export/common/environment/app_mpi_environment_pgi [user01@app1~]$ mpif90 -Mconcur sample_mpi_auto.f (Example) Compile a MPI source program and each process is multi-threaded by OpenMP. [user01@app1~]$ source /export/common/environment/app_mpi_environment_pgi [user01@app1~]$ mpif90 -mp sample_mpi_omp.f The following is an example how to submit a job which compiles a source program. Create a job script file (example: script.sh) using text editor such as vi command first. pgfortran sample.f Submit the job script file to ASB queue using the submit command. See the about the syntax of the submit command and for the details of queues in How to execute a program on application server. [user01@app1 ~]$ submit ASB general -exec script.sh You can omit the -exec option if you create the job script file named COMMANDLINE. [user01@app1 ~]$ submit ASB general 5

9 Main options: Option Description -o outfile Specifies the load module file name. If this is not specified, the load module file name is a.out. -L<dir> Specifies the path name of the directory containing the library you want to retrieve. -l<lib name> Specifies the library to link. -I<dir> Specifies the directory that contains the source statement library. -O0 -O1 -O -O2 -O3 -O4 Specifies the optimizing level. If this is not specified, the system assumes that -O1 has been specified. -Mconcur Enables parallelization of source program. -mp Enables parallelization of source program by using OpenMP directives. -Mfree Indicates that the source program is in the free format. -Mfixed Indicates that the source program is in the fixed format. -fastsse Chooses generally optimal flags for a processor that supports to vectorize for the SSE/AVX instructions. 6

10 2. Library The following libraries are available on application server. Library Description MKL Libraries for linear algebra, fast fourier transform and statistics functions provided by Intel. ACML Libraries for optimized and threaded math routines. IMSL Fortran Libraries for mathematics and statistics provided by Rogue Wave. FFTW Libraries for computing the discrete fourier transform (DFT). 2.1 MKL The following site provides advice of link options for using MKL. If you use MKL, please execute the following command. In case of 4 bytes integer [user01@app1~]$ source /opt/intel/mkl/bin/mklvars.sh intel64 lp64 In case of 8 bytes integer [user01@app1~]$ source /opt/intel/mkl/bin/mklvars.sh intel64 ilp64 (Example) Use 4 bytes integer and link MKL BLAS to your serial source program. [user01@app1~]$ ifort -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include blas_sample.f90 ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread lm Please note that it is a single command-line string. (Example) Use 4 bytes integer and link MKL BLAS to your serial source program using OpenMP. [user01@app1~]$ ifort -openmp -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include blas_omp_sample.f90 ${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm Please note that it is a single command-line string. (Example) Use 4 bytes integer and link MKL LAPACK to your serial source program. [user01@app1~]$ ifort -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include lapack_sample.f90 ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a -Wl,--end-group -lpthread -lm Please note that it is a single command-line string. 7

11 (Example) Use 4 bytes integer and link MKL LAPACK to your serial source program using OpenMP. [user01@app1~]$ ifort -openmp -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include lapack_omp_sample.f90 ${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm Please note that it is a single command-line string. (Example) Use 4 bytes integer and link MKL ScaLAPACK to your MPI source program. [user01@app1~]$ mpiifort -I${MKLROOT}/include/intel64/lp64 -I${MKLROOT}/include scalapack_mpi_sample.f90 ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_sequential.a -Wl,--end-group ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm Please note that it is a single command-line string. 2.2 ACML OpenMP libraries and scalar ones (1 CPU version) are available. This is available only in PGI fortran. Description Library OpenMP libraries libacml_mp.a Scalar libraries libacml.a (Example) Link scalar libraries [user01@app1~]$ pgfortran sample.f -lacml (Example) Link OpenMP libraries [user01@app1~]$ pgfortran -mp sample_openmp.f -lacml_mp 2.3 IMSL Fortran MPI libraries and scalar ones (1 CPU version) are available. This is available only in Intel fortran. If you use IMSL Fortran, please set the environment variables by the following command. (Example) Link scalar libraries [user01@app1~]$ source /opt/intel/bin/compilervars.sh intel64 [user01@app1~]$ source /export/app/imsl/imsl/fnl700/rdhin111e64/bin/fnlsetup.sh 8

12 (Example) Link MPI libraries source /export/common/environment/app_mpi_environment_imsl source /export/app/imsl/imsl/fnl700/rdhin111e64/bin/fnlsetup.sh (Example) Link MKL libraries dynamically $FFLAGS imsl.f $LINK_FNL (Example) Link MKL libraries statically $FFLAGS imsl.f $LINK_FNL_STATIC (Example) Link IMSL Fortran libraries dynamically $FFLAGS imsl.f $LINK_FNL_IMSL (Example) Link IMSL Fortran libraries statically $FFLAGS imsl.f $LINK_FNL_STATIC_IMSL (Example) Link MKL libraries $FFLAGS imsl_mpi.f $LINK_MPI (Example) Link IMSL Fortran libraries $FFLAGS imsl_mpi.f $LINK_MPI_IMSL 2.4 FFTW MPI libraries, OpenMP ones and scalar ones (1 CPU version) are available on /opt/fft/fftw-3.3.4/lib64. Library path Description Library MPI libraries libfftw3_mpi.a /opt/fft/fftw-3.3.4/lib64 OpenMP libraries libfftw3_omp.a Scalar libraries libfftw3.a (Example) Link MPI libraries source /opt/intel/impi/ /bin64/mpivars.sh mpiifort sample_mpi.f -I/opt/fft/fftw-3.3.4/include -L/opt/fft/fftw-3.3.4/lib64 -lfftw3_mpi -lfftw3 (Example) Link OpenMP libraries ifort -openmp sample_omp.f -I/opt/fft/fftw-3.3.4/include -L/opt/fft/fftw-3.3.4/lib64 -lfftw3_omp (Example) Link scalar libraries ifort sample.f -I/opt/fft/fftw-3.3.4/include -L/opt/fft/fftw-3.3.4/lib64 -lfftw3 9

13 You have to set the following environment variable when you run program which is linked FFTW libraries dynamically. export LD_LIBRARY_PATH=/opt/fft/fftw-3.3.4/lib64:${LD_LIBRARY_PATH} 10