Observational data structure of the HIRLAM variational system Kristian S. Mogensen ksm@dmi.dk March 27, 2002 1 Introduction The original data structure of the HIRLAM variational code was based on ECMWF's CMA (Central Memory Array) format (ECMWF, 2000), where all observational data were packed report by report into a single 64 bit oating-point array. This format was used both for external les produced by the OBSPROC software and as the internal data structure. There have been two main problems with using CMA as the internal data structure: 1) in the ongoing development of the HIRLAM variational system, we have onseveral occasions seen that some users have had diculty in understanding the CMA format and 2) several parts of the code operate on only a limited amount of information from the observations (e.g. the fraction of land check for ships only needs to read the fraction of land and to write various ags for rejected observations) so several routines existed which extracted information from CMA, did some calculation, then stored the results in CMA. On modern computers where memory access can be the limiting factor for performance, this can degrade performance. In addition ECMWF and MeteoFrance are currently in the process of moving from CMA to ODB, so it is possible that HIRLAM will end up with a data format we have to maintain and support ourselves. This was the basis for a discussion at the 2001 HIRLAM all-sta meeting (Jarvinen et al., 2001), where we agreed on preparing for a new external data format by changing the internal structure but keeping CMA as the external le format. A proposal for a new internal structure was made shortly after (Mogensen, 2001). This newsletter describes the implemented version of the new structure with a short description of the basis for the design decisions made. 2 Typical data access patterns in the variational code An example of typical data access patterns in the variational code is as follows: do i=1,nobs norerr=obfg(i)**2/(obserr(i)**2+bgerr(i)**2) if (norerr.lt.chklim(1)) then iflag=0 98
elseif (norerr.lt.chklim(2)) then iflag=1 endif if (iflag.gt.1) then staflg(i)=flrejected endif enddo This chunk of code is taken from the rst guess check and illustrates that we typically only need a few variables from all observations in each step of the code. In the example we need the observation error (obserr), the rst guess error (bgerr) and the dierence between the rst guess and the observations (obfg) from the observational data in order to calculate the normalized error used for checking against specied limits. If some limits are exceeded the status ag (stag) are set to rejected. Typically only 5 to 10 variables are needed from each observation type in the calculations of the HIRLAM variational system. 3 Design of a new observational data structure Based on the discussions at the 2001 all-sta meeting the following points were drawn up: In physical les the data is often stored report by report. Atypical memory access pattern is to access a small number of values (e.g. position, time, observed value) from an observation type. It should be possible to use the original external observation le format (CMA). New or other existing external data formats should also be possible. 4D-VAR and FGAT should be taken into account in the design phase. The code should be easy to port and should be understandable for Fortran-77 programmers. The code should be designed with performance in mind. Based on these statements the following design decisions were made: Data should be stored in Fortran-90 allocatable arrays to allow dynamic memory allocation. The allocatable arrays should be collected in Fortran-90 modules in order to allow adding of new variables without changing the calling tree. By storing each variable of a report (e.g. observation status ag) in a global allocatable array for all reports of a given observation type, ecient memory access is possible both on scalar and vector machines. 99
The ag structure of the CMA format should be used since it is well documented. OpenMP parallelization of the observation handling should be implemented as an option. 4 Examples on observation modules In order to relate some of the design decisions to real code we willnow present some of the observation modules implemented. Each observation module is organised in a series of sections. The rst section is pure bookkeeping and contains information about how many observations currently can be stored in the module, how many observations are currently stored in the module and where each time window (for 4D-VAR and FGAT) starts and ends. The second section contains information about the station making the observation and contains things like position and observation time. For each observed variable a section follows which contains the observed value, the dierence between observed value and the background, ags etc. 4.1 Single level data The simplest observation module is for single level data such as surface data, aircraft data and satellite wind data. An example of parts of such a module are given below for SYNOP's data: module synopdata integer :: max_synop_number! Max no. of Synops integer :: synop_number! Current no. of Synops integer,allocatable :: synop_winstart(:)! Start of each time slot integer,allocatable :: synop_seqno(:)! Sequence number integer,allocatable :: synop_obsno(:)! Obs. number integer,allocatable :: synop_date(:)! Obs. date integer,allocatable :: synop_time(:)! Obs. time real,allocatable :: synop_fispres(:)! Pressure for fis obs real,allocatable :: synop_fisobs(:)! Observed fis real,allocatable :: synop_fisobfg(:)! Fgs - observed fis integer,allocatable :: synop_fisstaflag(:)! Fis status flag end module synopdata 5 Multi level data For some multi level observation types there is the complication that the number of levels for individual reports varies. An example of such observation types is radiosondes. In the present code we choose to get around this problem by using one dimensional arrays for observed values and have a start and an end pointer for each observed value for each prole. For radiosondes the structure looks like the following: 100
module tempdata integer :: max_temp_number! Max no. of Temps integer :: temp_number! Current no. of Temps integer,allocatable :: temp_winstart(:)! Start of each time slot integer :: max_temp_ttotalobsnumber! Max total number of T obs integer :: temp_ttotalobsnumber! Total number of T obs integer,allocatable :: temp_tstart(:)! Start of each T profile integer,allocatable :: temp_tend(:)! Start of each T profile real,allocatable :: temp_tpres(:)! Pressure for T obs real,allocatable :: temp_tobs(:)! Observed T real,allocatable :: temp_toban(:)! Analysed - observed T real,allocatable :: temp_tobfg(:)! Fgs - observed T integer,allocatable :: temp_tstaflag(:)! T status flag end module tempdata A complication with this set-up is that the start and end pointers have to be carried over into subroutine calls. An example of such a call is the call to vertical interpolation of radiosonde data which looks like the following: call vint_p_ml_mod_bg(nlev,temp_totalobsnumber, x temp_tstart,temp_tend,indext_vint_mod, x wgtt_vint_mod,t_bg,temp_tbg) where the vertical interpolation then uses the start and end pointer information when doing the vertical interpolation. An alternative would be to have a two dimensional array with dimensions given by the maximum levelsinany report and the total number of observations, but that would waste memory if large dierences between number of levels exist. 6 Data ow The data ow with the new structure is illustrated on gure 1. The observations in BUFR are converted to CMA by the OBSPROC program and read into the variational analysis code where the information is converted into the module structure described above. During the screening the modules are updated with information from the screening (SMOD on the gure). For the minimization a separate observation module for each observation type is then allocated. Originally it was planned to use only one module for each observation type, but in order to archive unit-stride memory access in the minimization a second module (CMOD in the gure) was created for each observation type which only contains observations accepted by the screening and only the information needed for the calculation of the cost function and its gradient. This module is updated with variational quality control ags and the dierences between the analysis and the observations during the minimization. After the minimization this information is stored in the full modules (now labelled AMOD) and the minimization modules deallocated. The last step is to convert the observation modules back into CMA for writing to disk and subsequent diagnostics. 101
FILES BUFR CMA BUFR ACMA PLOTS PROGRAM OBSPROC HIRVDA(C2M) HIRVDA(SCR) HIRVDA(MCO) HIRVDA(MIN) HIRVDA(MDCO) HIRVDA(M2C) CMASTAT DATA MOD SMOD CMOD MMOD AMOD 7 Conclusions Figure 1: Data ow with the new data structure. The new structure has been implemented and tested without diculties on many dierent computers at various HIRLAM member institutes. The performance of the new code is similar or better than the original code. The variational code currently still reads CMA as input, but any external format can be used as long as it can be converted into the format of the modules. For some new initiatives, such as Quikscat data, input data are read directly from BUFR and converted into observation modules. In general the users seem quite happy with the new structure. With the new structure we are not dependent on having the same internal stucture as the external le format and new developments such as either implementing ECMWF's ODB format or implementing direct reading of BUFR data is possible. 8 People involved in this work. The coding of the observation modules was done together with the following people: Nils Gustafsson (SMHI), Xiang-Yu Huang (DMI), Magnus Lindskog (SMHI), Frank Tveter (DNMI) and Ole Vignes (DNMI). In addition the following people to part in the dis- 102
cussions of the structure: Gerard Cats (KNMI), Heikki Jarvinen (FMI) and Per Unden (SMHI). References ECMWF. 2000. ECMWF. IFS Documeentation. Part I: Observation Processing (CY21R4). Jarvinen, H., Mogensen, K., and Saarinen, S. 2001. ASM-2001 discussion on Observation handling. HIRLAM newsletter, 189{192. Mogensen, K. 2001. Proposal for a new internal observational data structure in the HIRLAM variational data assmilation code. HIRLAM newsletter, 193{202. 103