Mechanics of running HWRF

Transcription

1 HWRF v3.5b Tutorial Taipei, Taiwan, May 22, 2014 Mechanics of running HWRF Ligia Bernardet NOAA ESRL Global Systems Division, Boulder CO University of Colorado CIRES, Boulder CO 1

2 Overview of this lecture Datasets and fix files needed for running HWRF Location of source code and compiled executables Location and overview of scripts Tasks involved in a HWRF run Scripts, functions and wrappers Definition of global variables to be used by scripts Paths to input and output, initialization time, storm etc. Namelists and tables Location of output files Submission of jobs in TTFRI machine 2

3 Datasets and fix files Dataset Subdirectory File name TC Vitals Tcvitals syndat_tcvitals.$yyyy A decks abdecks a$basin$sid$yyyy.dat B decks abdecks b$basin$sid$yyyy.dat GFS anal & forec GFS/gridded/$YYYY$MM$DD$HH gfs.$yyyy$mm$dd$hh.pgrbf $fff GFS track GFS/gridded/$YYYY$MM$DD$HH avn.$yyyy$mm$dd $HH.cyclone.trackatcfunix UPP fix fix/upp * UPP coefficients fix/upp/hwrfspccoeff fix/upp/hwrftaucoeff * WPS fix wps_geog landuse_2m, topo_2m etc Location: /lfs/tutorial/hwrf/datasets 3

4 Structure of Compiled source WPSV3 geogrid.exe ungrib.exe metgrid.exe WPSV3 WRFV3_Idealized UPP hwrfutilities main main bin exec real_nmm.exe wrf.exe ideal.exe wrf.exe copygb.exe ndate.exe unipost.exe Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc gfdlvortextracker trk_exec diffwrf_3dvar.exe grbindex.exe hwrf_gettrk.exe hwrf_anl_4x_step2.exe hwrf_tave.exe hwrf_anl_bogus_10m.exe hwrf_anl_cs_10m.exe hwrf_vint.exe hwrf_bin_io.exe hwrf_create_nest_1x_10m.exe hwrf_create_trak_fnl.exe hwrf_create_trak_guess.exe hwrf_data_flag.exe hwrf_inter_2to1.exe hwrf_inter_2to2.exe hwrf_inter_2to6.exe hwrf_inter_4to2.exe hwrf_inter_4to6.exe hwrf_merge_nest_4x_step12_3n.exe hwrf_pert_ct1.exe hwrf_prep.exe hwrf_readtdrstmid.exe hwrf_readtdrtime.exe hwrf_split1.exe hwrf_swcorner_dynamic.exe hwrf_wrfout_newtime.exe wgrib.exe 4

5 Scripts, Wrappers, and Functions The scripts (ksh93) to run HWRF are located in hwrf-utilities/scripts hwrf-utilities The scripts use a library of functions wrappers_scripts scripts funcs Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc/ hwrf-utilities The scripts need to receive global variables with information about initialization date, which storm to run etc. Those will be passed by the wrappers A generic automation system could be use to drive these script (automation not covered in this tutorial) 5

6 Example of a wrapper: hwrf_wrapper Every wrapper is a ksh93 script that Sets global variables (e.g., storm, initialization time, dir for input/output) Sets local variables, which are specific to the script that will be called Calls a script #!/usr/bin/env ksh93!! # # Global definitions of environment variables! # PATH_TO_GLOBAL/global_vars.ksh! # # Local definitions of environment variables! # ! export WRF_MODE=main! export WRF_CORES=${HWRF_COUPLED_CORES}!! # export DEBUG=1! ${HWRF_SCRIPTS}/wrf.ksh! 6

7 Example of a script: hwrfdomain.ksh # Main! function main {! typeset errno! # Init the function library!. ${HWRF_SCRIPTS}/funcs/init! set -e! # Check to see if all the variables are set! check_vars DOMAIN_DATA START_TIME!! # Create a working directory (and cd into it)! create_work_dir ${DOMAIN_DATA}/messages!! # Create the tcvital and tcvitals.as files! lat_lon=( $( tcvitals ) )!! # Create the storm centre file! storm_centre ${lat_lon[@]}! Functions used only once are located in script that uses them storm_centre Functions used more than once are located in a library of functions check_vars and create_work_dir 7

8 Tasks, wrappers, and scripts I Task Wrapper Script Determine domain location based on TC Vitals hwrfdomain_wrapper hwrfdomain.ksh Run WPS geogrid geogrid_wrapper geogrid.ksh Run WPS ungrib ungrib_wrapper ungrib.ksh Run WPS metgrid metgrid_wrapper metgrid.ksh Run real real_wrapper real.ksh Run WRF Analysis (90s) wrfanalysis_wrapper wrf.ksh Run WRF Ghost (90s) wrfghost_wrapper wrf.ksh Run postprocessor and tracker on WRF Analysis track_analysis_wrapper output to find location of vortex in GFS track_analysis.ksh Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc/hwrf-utilities 8

9 Tasks, wrappers, and scripts II Task Wrapper Script relocate1_wrapper relocate_stage1_3d.ksh relocate2_wrapper relocate_stage2_3d.ksh relocate3_wrapper relocate_stage3_3d.ksh Merge information on various domains to create IC for WRF merge_wrapper merge.ksh Run main forecast hwrf_wrapper wrf.ksh Run postprocessor unipost_wrapper unipost.ksh Run external vortex tracker tracker_wrappe tracker.ksh Create plots rungrads_wrapper run_grads Run the vortex initialization Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc/hwrf-utilities 9

10 Defining global variables for scripts I Wrappers source global_vars.ksh to pass global variables to scripts Soulik 07W 2013 START_TIME Initialization time START_TIME_MINUS Initialization time minus 6 hr FCST_LENGTH 12 Length of forecast (in hrs) FCST_INTERVAL 6 Time between consecutive HWRF cycles STORM_NAME SOULIK Storm name SID 07W Storm ID L=Atl; E=E Pac; W=W Pac BASIN WP Basin AL, EP, WP, IO Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc/hwrf-utilities/wrapper_scripts/global_vars.ksh 10

11 Defining global variables for scripts II Soulik 07W 2013 RUN_PREP_HYB F T= Use global model in binary format; F=Use global model in GRIB format UPP_PROD_SAT T T= UPP outputs synthetic satellite fields BKG_MODE GFS Select GFS or GDAS for initial conditions RUN_GSI F RUN_GSI_WRFINPUT F RUN_GSI_WRFGHOST F INNER_CORE_DA Disable data assimilation 0 Full description of global_vars.ksh can be found in Appendix of HWRF v3.5a Users Guide 11

12 Defining global variables for scripts III Soulik 07W 2013 HWRF_SRC_DIR /lfs/tutorial/$user/hwrf_v3.5b/sorc Path to HWRF source code HWRF_DATA_DIR /lfs/tutorial/hwrf/datasets Path to HWRF input datasets CYCLE_DATA ${HWRF_OUTPUT_DIR}/${SID}/$ {START_TIME_MINUS6} Path to previous cycle output DOMAIN_DATA ${HWRF_OUTPUT_DIR}/${SID}/$ {START_TIME} Path to current cycle output GFS_DIR ${HWRF_DATA_DIR}/GFS Path to toplevel GFS input GFS_GRIDDED_DIR ${GFS_DIR}/gridded Path to GFS gridded data TCVITALS ${HWRF_DATA_DIR}/Tcvitals Path to TC Vitals files GEOG_DATA_PATH ${HWRF_DATA_DIR}/wps_geog Path to geographical fixdata GRADS_BIN /lfs/tutorial/hwrf/bin Path to GrADS binaries GADDIR /lfs/tutorial/hwrf/grads/data Path to grads tables 12

13 Defining global variables for scripts IV Soulik 07W 2013 MPIRUN mpiexec Command to run parallel code WRF_ANAL_CORES 12 Number of cores to run WRF Analysis WRF_GHOST_CORES 12 Number of cores to run WRF Ghost HWRF_FCST_CORES 202 Number of cores to run HWRF forecast GEOGRID_CORES 12 Number of cores to run geogrid METGRID_CORES 12 Number of cores to run metgrid REAL_CORES 1 Number of cores to run real UNI_CORES 12 Number of cores to run UPP ATCFNAME HCOM ATCF identifier for HWRF forecast 13

14 Namelists and tables hwrf_cntrl.* Namelists for UPP hwrf_namelist.input Template namelist for WRF runs (ghost, analysis, main) hwrf_namelist.wps Template namelist for WPS (geogrid, ungrib, metgrid) hwrf_co2_trans hwrf_eta_micro_lookup.dat hwrf_etampnew_data hwrf_etampnew_data_dbl hwrf_genparm.tbl hwrf_landuse.tbl hwrf_rrtm_data hwrf_rrtm_data_dbl hwrf_soilparm.tbl hwrf_tr49t67 hwrf_tr49t85 hwrf_tr67t85 hwrf_vegparm.tbl Tables used for WRF runs hwrf_grib_reduce.parms List of variables used to thin the input GFS file toexpedite processing hwrf_storm_* Tables used for vortex relocation Location: /lfs/tutorial/$user/hwrf_v3.5b/sorc/hwrf-utilities/parm 14

15 WRF namelist template @[start:0:4],! @[start:4:2],! @[start:6:2],!! num_soil_layers = 4,! mp_physics = 85, 85, 85,! ra_lw_physics = 98, 98, 98,! ra_sw_physics = 98, 98, 98,! Some variables are directly defined in template namelist Other variables in front) are populated when scripts run Scripts Read hwrf-utilities/parm/hwrf_namelist.input Modify namelist according to global variables defined Write namelist in working directory 15

16 Location: /lfs/tutorial/${user}/hwrf_v3.5b/results/$sid/$yyyymmddhh Output directory structure Subdirectories Creator script messages hwrfdomain.ksh geoprd geogrid.ksh ungribprd/$yyyymmddhh ungrib.ksh metgridprd/$yyyymmddhh metgrid.ksh realprd_gfs real.ksh wrfghostprd/$yyyymmddhh wrf.ksh run in Ghost mode wrfanalysisprd/$yyyymmddhh wrf.ksh run in Analysis mode trkanalysisprd/$yyyymmddhh track_analysis.ksh relocateprd/$yyyymmddhh relocate*.ksh mergeprd merge.ksh wrfprd wrf.ksh in main mode postprd run_unipost gvtprd tracker.ksh Some subdirectories have a date string on them. For this tutorial, it is the same as initialization time. This is done because for some data assimilation configurations it is necessary to process data for other valid time as well 16

17 What you learned in this lecture The source code has many components WPS, WRF, UPP, hwrf-utilities, GFDL vortex tracker Several tasks are needed to do an end-to-end HWRF run Hwrf-utilities scripts, functions, wrappers are used to run HWRF Wrappers Source global_vars.ksh (paths, initialization time, storm etc.) Define local variables Call scripts File structure for input and output files Next: how to submit a job in TTFRI machine 17

18 How to log in to the TTFRI computer Open 2 login windows on your laptop: editing and running Linux: select open terminal from the right mouse button menu. OS X: start the Terminal and XQuartz applications from the /Applications/Utilities folder. Windows: start the application XSession from the Start menu and then Putty Open an X-enabled window to a login node Log in to the supercomputer : ssh -X user_name@ ! At the prompt, enter your temporary password Practical session instructions 18

19 How to submit wrapper scripts Define your wrapper directory setenv WRAPPER_DIR /lfs/tutorial/$user//hwrf_v3.5b/hwrf-utilities/wrapper_scripts! or!! export WRAPPER_DIR =/lfs/tutorial/$user/hwrf_v3.5b/hwrf-utilities/wrapper_scripts! Set the options for the job bsub_options='-l procs=16,walltime=6:00:00,vmem=30g -N hwrf -o hwrf.out -e hwrf.err -S /bin/bash -k oe -z! Submit the job bsub $bsub_options ${WRAPPER_DIR}/geogrid_wrapper! Descriptions of batch system options -l resources needed to run script -N job name -o standard out file name -e standard err name -S use bash -k keep standard error and output -Z suppresses the job ID from being written to standard output 19

20 ! t s e r e t n i r u o y r o f u o y k an Th For more information, you can Ask questions during the tutorial Contact me later: ligia.bernardet@noaa.gov Reach our user helpdesk: wrfhelp@ucar.edu Visit our website: HWRF v3.5a Users Guide ( HWRF_v3.5a_Users_Guide.pdf) HWRF v3.5a Scientific Documentation ( scientific_documents/hwrfv3.5a_scientificdoc.pdf) WRF-NMM V3 Users Guide ( users_guide_nmm_chap1-7upp.pdf) 20