RegCM-ROMS Tutorial: Coupling RegCM-ROMS

Transcription

1 RegCM-ROMS Tutorial: Coupling RegCM-ROMS Ufuk Utku Turuncoglu ICTP (International Center for Theoretical Physics) Earth System Physics Section

2 - Outline Outline Information about coupling and ESMF Installation and setup Coupled model (RegCM+ROMS) Design, Limitations and Pros/Cons Installation Configuration Running and Restarting Debugging Future Plans

3 - Information about ESMF (1/2) Why ESMF? Different ways for online coupling Creating subroutinized sub-components (like sub-grid parameterizations or current version of the CLM coupling) Most of the existing code use this approach. It is not generic solution and not flexible. Think about changing version of CLM in current version of RegCM model. Using existing communication protocols such as (MPI, InterComm) Implementation is not easy. It is not generic and efficient solution. Using coupling libraries or frameworks (i.e. ESMF, MCT) Flexible, efficient, scalable and generic solution. It needs time to convert existing code into components. Using a coupler (OASIS, MPCCI etc.) Flexible, efficient, scalable and generic solution. It also needs time to implement into existing codes. Increasing needs to such tools like ESMF, MCT and OASIS!!! The current models evolve to a multi-component Earth System Models (ESM).

4 - Information about ESMF (2/2) Basics ESMF The complexity of the regular tasks to create a coupled modeling system (synchronization of the model components, exchanging coupling fields among modeling components, interpolation between deferent grids etc.) can be simplified. ESMF can perform different interpolation types (bilinear, conservative etc.) over exchange fields (i.e. sea surface temperature, heat and momentum fluxes) in a parallel way without using external grid weight file generated by SCRIP. Supports both C/C++ or Fortran programming language. Highly optimized and portable (the library is tested regularly across different computing platforms, compilers (and also versions), mpi implementations etc.) library Scalable beyond few thousand processor (if you can feed the CPUs) Supports different technologies such as parallel IO, XML, web services (like OpenMI).

5 - Installation of ESMF (1/3) Steps to install ESMF library Get ESMF library source code Latest public release 5.2.0rp2 ESMF_5_2_0rp2/reg/ESMF_Framework_Reg.html Fill simple form and download the source code Installation information for different platforms can be found in Install third-party libraries NetCDF Parallel NetCDF (required for coupled model debugging) Xerces XML Parser (required for exporting component, field level metadata in XML format) Install ESMF and Setup Environment Variables Third party libraries must be installed first!!! Same compiler and MPI implementation must be used!!!

6 Example installation environment on ARGO - Installation of ESMF (2/3) third-party libraries Install both optimized and debug versions ESMF library related definitions Used for RegCM+ROMS installation

7 - Installation of ESMF (3/3) Commands To install optimized version mkdir ~/progs; cd progs tar -zxvf esmf_5_2_0rp2_src.tar.gz; mv esmf esmf_5_2_0rp2 cd esmf_5_2_0rp2 make info (it shows the configuration. important for solving installation problems) make >& make_o.log make install To install debug version make clean export ESMF_BOPT=g (for tcsh setenv ESMF_BOPT g) make >&make_g.log make install Notes To test the installation, the ESMF test suite can be installed

8 - Design (1/5) General structure (1) Psurf, T2m, Q2m, Rain Swrad, Lwrad, U10m, V10m SST Hice Ocean / Lake ATM à Surface atmospheric conditions, heat fluxes and rain. The atmospheric variables is used to calculate fluxes in the ROMS. OCN à SST and Ice Thickness (if the ice model exist). The SST is used to calculate sensible heat flux, evaporation, ground temperature etc. Ice Thickness is used to change land type and updates the evaporation flux over ice The interaction or coupling time step (in second) can be defined using RegCM namelist file

9 - Design (2/5) General structure (2) MPI Comm. World Run Init 0 1 N-1 Number of PETs: N N N+1 2N-1 Total Number of PETs: 2N Finalize Init Run Finalize RegCM Atmosphere Model Number of PETs: N ROMS Ocean Model Import States Export States COUPLER Import States Export States Initialize Run Forward Backward Initialize Run Finalize Finalize ATM to OCN OCN to ATM COUPLER: Online Interpolation,Vector Rotation, Unit Conversion

10 - Design (3/5) General structure (2) Time Axis Time Axis ATM BATS MTX ZENG LAKE UPD CPL FWD OCN CPL BCK T = 0 update forcing variables in ROMS send ocean variables (SST, ICE etc.) to RegCM T = Tcpl update SST and ICE in RegCM update forcing variables in ROMS

11 - Design (4/5) Design details (1) Supports It can be used together with Zeng Ocean and one-dimensional Lake Model. ESMF online re-gridding capability and conservative type interpolation is used for flux variables (bilinear type interpolation for others) Works with ROMS 3.5 tagged version and also ROMS-Ice branch. Different number of processor can be assigned to gridded components (coupler component uses all processors) Limitations It only interacts with BATS (no support CLM or BAND) The ocean model still needs forcing files (RegCM model updates them when coupled mode is activated). Current version only supports Gregorian calendar Ocean model must be configured to write daily restart file and this adds extra overhead into coupled model run. There is a bug in ROMS (PERFECT_RESTART+LcycleRST == F)

12 - Design (5/5) Design details (2) Pros The version of ocean component (ROMS) can be changed easily by porting patch (it may also requires minor change in the RegCM component code). This is tested before is tested before, 3.4 à 3.5 à 3.5_ice. The coupled model provides better representation of oceans, lakes and inland waters (If the ROMS model is tuned well). New components (like CLM) can be added by modifying coupler interface. Cons It needs experience also in ocean modeling, which the configuration of realistic application is not straightforward (create ocean model grid, initial, boundary and forcing files). Adds additional computational and messaging overhead into the model User needs to tune the ocean model like atmospheric model. This must be done in standalone mode first!!!

13 - Installation (1/3) Applying patch to ROMS and install Patch includes following modifications Updates forcing variables with RegCM provided ones Enables to use two namelist (*.in for RegCM and ROMS) file in the job submission step Fixes caldate subroutine in (ROMS/Utils/get_date.F90). The changes are activated by new pre-processor flag (#define REGCM_COUPLING) which must be set in *.h header file. Without this option./build.sh compiles original version of ROMS. Applying patch Creating and installing new ROMS case (before compiling RegCM!!!)

14 - Installation (2/3) Configure and install RegCM./configure --help options for coupling Example command to configure RegCM in coupled model mode

15 - Installation (3/3) Additional information about installation User may need to install newer version of autoconf (>2.68) and automake (>1.11.1) tools to run bootstrap.sh script. Script --with-esmf option looks for ESMF_LIB environment variable. The ROMS Build path must be given as an absolute path into configure script using --with-roms option!!! After./configure step, the RegCM code can be compiled with conventional way using make install command. The ROMS ice branch is automatically identified by configure script and required code is added into RegCM (it just check the seaice.o file in the ROMS Build directory). The patched version of ROMS (with REGCM_COUPLING option) and also ESMF library must be installed before RegCM installation!!! The ROMS does not need ESMF library. So, do not install it with existing ESMF_COUPLING and/or ESMF_LIB header definitions. It is also gives error with these options because it is not fully implemented yet.

16 - Configuration (1/2) New parameters (*.in) physicsparam iocncpl cplparam 0 no coupling (default) 1 activates ROMS coupling cpldt controls the coupling time step (in second, 21600) cpldbglevel specify the debug level for coupled model. (1) 0 no debugging 1 only informative print statement (array and pointer dimensions etc.). 2 previous option + write grid information in VTK (The Visualization Toolkit) format which can be visualized by using such tools like Visit. 3 previous option + write exchange fields in NetCDF format 4 previous option + write exchange fields in ASCII format

17 - Configuration (2/2) New parameters (regcm.rc) The new namelist file (regcm.rc) is created to pass parameters to the coupled model (sample file is in the Testing/CPL_FILES) In the current implementation, the new parameter file is used just to set number of processor for each gridded (M, N) component. If the file does not exist, then the total number of processor (M+N) divided to two and assigned to each gridded component (M, N). In this case N+M can be divided evenly or not. We plan to move all the parameters (coupling time step, debug level etc.) to the new parameter file (regcm.rc). The example for the PETs parameter: PETs: In this case atmospheric model (RegCM) will use 32 (M) processor and ocean model (ROMS) will use 28 (N) and the coupler component uses all the available CPUs (32+28 = 60) to perform interpolation between grids. The M must be evenly divided to the jx and placed under case directory like regcm.in

18 - Running (1/4) Job submission under PBS regcm.job path of ESMF shared libraries definition of namelist files The same ESMF path must be defined with./configuration step. The ESMF must be compiled with same version of the compiler and ESMF implementation The total number of processor must be consistent. In the example script regcm.rc (M+N = = 60 = 12*5 in regcm.job) cas.in (NtileI*NtileJ = 4*7 = N = 28)

19 - Running (2/4) Restarting (1) Requirements User must configure ROMS to write restart data each day without cycling. To do that set NRST to 86400/DT and LcycleRST to F. The date of the records in the restart file might be checked by CDO. cdo sinfo ocean_rst.nc. Steps in ROMS side In namelist file, set NRREC to -1 (indicates that the run is restart) Split desired date from ocean_rst.nc file (this date must be same with the date of latest RegCM restart file - SAV). Point created file as a restart file in namelist file. To do that set ININAME as output/ocean_rst_splited.nc (or point the correct directory). Steps in RegCM side Set ifrest as.true. Set mdate1 as a date of the latest SAV file Run RegCM model

20 - Running (3/4) Restarting (2) Notes Restating of the coupled model can be little bit confusing Both model must have restart information in same date RegCM normally creates restart file at the beginning of the each month but you can not define monthly time interval for ROMS restart files. As a result we choose to write ROMS restart each day. By this way, ROMS will have restart information at the beginning of the month. This can be potential performance bottleneck and waste of disk usage In theory, it is possible to set date for RegCM model to write intermediate weekly restart file (TMPSAV) by using savfrq parameter. In this case, ROMS model can be configured to write restart file with the same interval (i.e. weekly). It is not tested!!! There could be time lag between restart files created by RegCM and ROMS. To simplify the current restart mechanism, we created a restart script (Tools/CPL_FILES/restart.sh) that automatize the steps in the previous slide.

21 - Running (4/4) Restarting (3) restart.sh Required modules (CDO and NCO). definition of RegCM and ROMS namelist files User may need to modify this script. It uses Module (user environment manager) to setup paths for NCO and CDO tools. It is written in bash shell and uses awk and sed. User need to check the RegCM and ROMS namelist files after using this script. The backup is created for namelist and restart file.

22 - Debugging (1/2) VTK (The Visualization Toolkit) and Visit Used to see the grid representation of each gridded component Created for each grid stencil Files with *.vtk extension for each processor can be visualize in Visit

23 - Debugging (2/2) Notes If you have problem with the coupled model, check following before doing anything Check grid definitions using Visit (see previous page ) Check import and export fields (can be written in NetCDF and ASCII format). The ATM/OCN export files must match with OCN/ATM import files. The debug level of the coupled model must be >2. Use debug version of ESMF library (compiled with ESMF_BOPT=g). The ESMF can give some information in the PET* files. Use debug version of the model components. The ROMS can be compiled in debug mode by modifying./build.sh script (set USE_DEBUG as on). The RegCM can be compiled in debug mode by running configure script with --enable-debug option. In the stdout file you can see the actual line number of the source file that triggers the error. Check input files and the configuration of the model components (RegCM and ROMS).

24 - Future Plans More work Add support for identical mesh/grid. No need to interpolation between grid. The possible candidate to test this configuration is the BAND version of the RegCM. The tool to create ROMS grid from RegCM domain file is almost ready. It is written in NCL. Move all coupled model configuration options to regcm.rc Add support for boundary smoothing. It is important for the ocean or sea centric applications (i.e. Mediterranean Sea, Indian Ocean). Add support for selective interpolation types for each variable. User can change from conservative type re-gridding to bilinear or others. Improve the air-sea interaction interface. In the current implementation, ocean send SST and ICE thickness to atmospheric model. The aerosol related variables, surface roughness etc. can be added into list. Benchmarking is needed to see the actual performance of the model. It also helps to find possible performance bottleneck sources of the current implementation.

25 - News RegCM is in the component list of the ESMF J