Step 3: Access the HPC machine you will be using to run WRF: ocelote. Step 4: transfer downloaded WRF tar files to your home directory

Transcription

1 Step 1: download WRF packages Get WRF tar file from WRF users page, Version Also get WPS Version (preprocessor) Store on your local machine Step 2: Login to UA HPC system ssh Password is UAnetid password Step 3: Access the HPC machine you will be using to run WRF: ocelote Once logged in, type ocelote Step 4: transfer downloaded WRF tar files to your home directory Once on ocelote, put tar file for WRF (Version 3.8.1) and WPS on main directory (/home/u*/uanetid). Use sftp to access your local machine. Step 5: Create temporary directory xdisk c create m 200 d 45 [This will create a temporary directory under /xdisk/uanetid. The command above will set up your xdisk directory valid for 45 days with a memory size of 200 Gb.] If you need to extend the amount of time before the 45 day expiration (e.g. to request an additional 45 days), do the following command: xdisk c expire d 45 Step 6: Move WRF tar file to temporary directory /xdisk/uanetid Step 7: unzip and untar the WRF files gunzip tar file (tar.gz)

2 tar xvf tarfile (tar) Repeat for WPS After this process you should have directories for WPS and WRFV3 Step 8: Load modules needed to run WRF [you have to do this everytime you log back on, or add the module load commands in your.bash_profile] Module avail will show all modules available for your use Module load (enter all directory below as separate commands) module load hdf5_18 module load netcdf/fortran module load perl module load intel/compiler/64 module load intel/mkl/64 module load intel/mpi/64 module load pbspro/ [remove, can't check queue if this module is used] module load gcc/5.2.0 Verify these modules installed by doing module list Step 9: compile WRF cd to WRFV3!!!! Important!!!! module unload netcdf/intel [this command is to bypass the compiling issue, just need to do it when compiling, not needed after WRF is compiled] export WRFIO_NCD_LARGE_FILE_SUPPORT=1 [for creating large WRF files] export HDF5=/cm/shared/apps/hdf5_18/./configure

3 [Because you unloaded netcdf library, you will be asked to set the library mannually during compilation] ** WARNING: No path to NETCDF and environment variable NETCDF not set. ** would you like me to try to fix? [y] Enter full path to NetCDF include directory on your system /cm/shared/uaapps/netcdf/fortran/intel/4.4.4/include Enter full path to NetCDF library directory on your system /cm/shared/uaapps/netcdf/fortran/intel/4.4.4/lib [You will also be prompted to select an compiler] Please select from among the following Linux x86_64 options: Enter selection [1-75] : 20 Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 1 Configuration successful! -----To compile 2-D Idealized cases Please select from among the following Linux x86_64 options: Enter selection [1-75] : 18 Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 0 [You will be running the cases in serial using./***.exe] [Now you are ready to compile WRF]./compile em_real > & log.compile [this step will take 45 mins to an hour, if successful, you will find executables created in main and run directories, real.exe, wrf.exe, ndown.exe, tc.exe]

4 Step 10: Compile WPS Go to your WPS directory export WRFIO_NCD_LARGE_FILE_SUPPORT=1 export HDF5=/cm/shared/apps/hdf5_18/ export NETCDF=/cm/shared/uaapps/netcdf/fortran/intel/4.4.4./configure [again, you will be asked to set the netcdf directory /cm/shared/uaapps/netcdf/fortran/intel/4.4.4 ] Please select from among the following supported platforms. 17. Linux x86_64, Intel compiler (serial) ## choose 17 Configuration successful. To build the WPS, type: compile [Using the version of netcdf library used in this exercise, modify the configure.wps file by: ] remove -lnetcdf in WRF_LIB, the command should look like this: -L$(NETCDF)/lib -lnetcdff./compile wps [if successful, you will create geogrid.exe, ungrib.exe and metgrid.exe]!!! YOU ARE DONE COMPILING WRF AND WPS!!! [EXTRA ] (1) create.mpd.conf file in your home directory [only need to do this once] the file will contain only one line: PD_SECRETWORD= oneword (you pick) (2) while configuring, I need to reset the library path:

5 ./clean -a [clean up the previous environment setting, no need to start from module load, as long as you did not log off, then start from./configure again] (3)./clean -a is not letting me to type the library directory again It could come from the command below [Note I used in when compiling WPS, but not in WRF. With or without the following command doesn't change the compiling outcome. You can check with netcdf is been used by reading the printed message on your screen carefully] export NETCDF=/cm/shared/uaapps/netcdf/fortran/intel/4.4.4 (4).bash_profile Make sure your existing.bash*** files doesnot contradict the one you copied from mine. A good exercise will be to check what is in each of those files and removed overlapped commands. [HPC documentation] (1) Script sample to submit parallel job: (2) submit parallel job: [four type of queues: priority, standard, winfall, debug] qsub ***.csh (3) chech your job status: qstat -q UAnetid (4) delet a job you already submitted: qdel job#.head1 (5) check how much walltime left: va ----create a ***.csh file using the following script #!/bin/bash #PBS -N wrf #PBS -q standard #PBS -l walltime=1:00:00 #PBS -l cput=56:00:00

6 #PBS -l select=2:ncpus=28:mem=168gb:pcmem=6gb #PBS -l place=free #PBS -W group_list=clcastro export PATH=/opt/ScaleMP/mpich2/1.0.8/bin:$PATH # If application binary is MPICH1 and dynamically linked, uncomment this # export USE_MPI_WRAPPER=MPICH1 # # Bind MPI processes on consecutive cores # # Adding numabind for dynamic placement (according to VSMP_PLACEMENT) # #export VSMP_PLACEMENT=PACKED # #export VSMP_VERBOSE=YES # #export VSMP_MEM_PIN=YES ulimit -s unlimited # #vsmputil unpinall cd /xdisk/hchang05/wrfv3/run [recommend change to the test directory you selected] printenv rm rsl.error.* rm rsl.out.* rm wrfout* #set groupio time mpirun -np 56./wrf.exe > wrf.out