EUMETCast Reception System in AEMET

Transcription

1 EUMETCast Reception System in AEMET Miguel A. Martinez, Nieves Clemente, Esteban Rodriguez, Ana Sanchez State Meteorological Agency of Spain (AEMET), Spain Abstract The paper describes the current reception and processing system of satellite data disseminated by EUMETCast in the AEMET (known internally as AEMETCast). AEMET has twenty systems (Linux PC based) dedicated to EUMETCAST transmission s reception. At headquarters (HQ) there are three of them and the rest are spread all over the Delegations that AEMET has in Spain. There are also other two sytems Windows PC based at headquarters. INTRODUCTION AEMET (former INM) launched in 2005 an open tender to get a system to receive the data from EUMETCast dissemination, to manage them, to produce images and products automatically in real time and to archive them with the AEMET policy. The system must run over Linux PCs, using McIDAS secondary servers as the mechanism to make the displaying and to have the opportunity to not only be accessed as graphics files but using McIDAS. Thus, any user with any computer inside the intranet network can get access to the satellite data through web pages or through the McIDAS ADDE protocol. Reception systems must be installed not only in AEMET s HQ but in other 17 sites located in different cities in Spain. For the sites others than headquarters (HQ), a redundancy system was required. Another important feature was that AEMET is the leader entity of the NWC SAF and that the system should be able to supply the HRIT SEVIRI files to the NWC SAF Reference System machine and to manage and produce the images for the NWC SAF web pages. Since in that time the official and only available displaying system was McIDAS all the system should be compatible with McIDAS software. The winner was a consortium made up of two Spanish enterprises. The antennas, hardware, system configuration, the management of the decompression of the files and the redundancy system were developed by PRODETEL enterprise. The McIDAS secondary servers and the software of the management of the files were developed by GMV enterprise. AEMET is the proprietary of the developed software. SYSTEM DESCRIPTION Each AEMET EUMETCast reception system is divided in two parts. The first one (receiving PC) is dedicated to the reception, decoding, decompression, dispatching and monitoring of reception. In the second one (server PC) images and products are generated for forecasting, research and development departments, AEMET s web servers and for an increasing number of users. For the automatic image generation, the SSEC McIDAS ( is the software most commonly used. Another important feature is that the second PC has installed McIDAS servers, McIDAS secondary servers for HRIT data, for Foreing Satellite (FD) products and format converters for MPEF products and for NWC SAF products. Thus, users can access to the data also using any ADDE protocol client as McIDAS-X client, McIDAS-V client or IDV client. One of the AEMET EUMETCast reception systems in HQ is dedicated to supply HRIT data to the NWC SAF (Nowcasting Satellite Application Facility) for product generation in its Reference System, and to the new Ninjo system. The NWC SAF products files are dispatched to the server PC and it generates the image files for the NWCSAF web page. The second PC has also one apache web server and it has several web pages in order that users can have a quick-look of the received images and the administrators of the AEMET reception systems to monitor the performance of the reception.

2 Eventually the data, images and products obtained in HQ are stored in the archiving system according to the AEMET archiving policy. The whole system is built on configurable files in order to allow each AEMET s meteorological regional centre could be able to adapt and manage the reception system according to their own needs. As example the central coordinates of the images used by Canary Island regional centres are different to the ones used by north Spain regional. In Fig. 1, geographical distribution of AEMETCAST stations is represented. This distribution allows every AEMET s site with a EUMETCast Recepction System installed to have access to data and products that they need. Figure 1. AEMETCast Stations distribution. An AEMETCast Reception System is made up of two PC Workstation (one for reception, and one for processing) with the next features: In both PCs CPU=Intel Pentium D 3.2 GHz (Dual Core), Memory 2 GB, 250 Gb of HD, 2 Ethernets ports, USB 2.1 ports Operating System: Red Hat Enterprise Linux 3 (update 7); Kernel: ELsmp File Manager: DB_GEST File Dispatcher: EFTS_Push_Ftp Agent v2.0 Web Browser: Mozilla Java: JRE 1.5 Web Server: Apache 2.0 Monitoring and accounting HRIT/EUMETCast reception: HMC Redundancy System: SDR (in AEMET s Delegations) Receiving PC DVB Reception Card: Technisat Skystar 2 rev 2.6 Driver DVB: Linuxtv Reception Card Tunning: Dvbtune-0.5 y szap ftp client/server system: Tellicast v2.4 Multicast Package routing: smcroute Data Decoder: EKU (Eumetsat Key Unit), Pcscd y Etokend de Aladin File Decompressor: XRITDecompress PC Server Data processor & Image server: converters Image WebServer McIDAS v2005b plus Secondary Servers & format Web_Catalog In AEMET headquarters there are two redundant systems with identical configuration, securing at this way the availability of the services (Figure 2), and furthermore it s possible to use one of the reception systems to accomplish tests and temporal changes when they are required, without compromise the operational services.

3 1.2 m. Antenna Eurobird 9 Ku Band WAN 1.2 m. Antenna Eurobird 9 Ku Band WAN McIDAS network in site McIDAS network in site INMCast Receptor (receptor+server) INMCast Receptor1 INMCast Receptor2 McIDAS Client McIDAS Client (McIDAS o IDV) INMCast Server Web Client Linux or Solaris or Web Client (McIDAS o IDV) INMCast Server1 INMCast Server2 Linux or Solaris or (receptor+server) WINDOWS WINDOWS Figure 2. Left) EUMETCast reception at AEMET s Delegations. Right) EUMETCast reception at AEMET s Headquarters. As can be seen in Figure 2 all the systems are interconnected through the internal AEMET wide area network (WAN). As the McIDAS ADDE protocol is based on client-server architecture any client in the WAN can access to any server inside the WAN. For AEMET authorised users is possible to connect to AEMET McIDAS server outside through INTERNET; as example, NASA access to the McIDAS servers when shuttle is launched because Zaragoza and Moron airports can be used in case of problems at launching. AEMET reception system functions and services The main functions and services are: - db_gest: this software manages (files delivery, maintenance, decompression and scripts execution associated to an event) EUMETCast received data and products, by using the information contained in a configuration file. - McIDAS plus secondary servers and format converters developed in AEMET, for images and products generation from EUMETCast data reception. - An ensemble of shell-scripts executed automatically provides files and images to support AEMET public and AEMET intranet webservices. - Web_catalog: application developed to provide clients with an interface web for generated products. - EUMETCast Raw data delivery to other meteorological applications (SAFNWC package, Ninjo, numerical weather prediction models, etc). - HMC: is a Monitoring and accounting HRIT/EUMETCast reception tool that allow report generation about Seviri-HRIT files received and processed by the tool db_gest. The reception control is done by comparing the files processed by db_gest with the information available in the DAILY_LOG file disseminated daily by EUMETCast about the files that have been disseminated without problems by EUMETSAT. -SDR, Redundancy tool: EUMETCast reception system in AEMET s Delegations are made up of 2 PC s with the redundancy tool SDR. This application periodically keep a check on both PC s in such a way that if there is some problem in one of them, the other one can execute the main functions and services of both until problems can be resolved and then to come back to the nominal situation. It will also be necessary that both PCs have similar hardware and software features. A diagram of these main functions and services can be seen in the Figure 4. db_gest file manager and automatic Shell-scripts execution One key part of the AEMET reception system is the powerful combination of the db_gest service and the automatic execution of specific shell-script executions (including McIDAS commands) launched in real time after the reception of one file. A software tool named DB GESTOR was developed to dispatch the raw data, according to filters, to specific folders in a directory structure, to delete/remove files and release disk space and to execute programs if an event happens. All the activity of this application is registered in a daily log file.

4 The db_gest tool also allows the management of the files. After reception of a new file, according to a configurable ASCII file (db_gest configuration file), one of several rules is applied. The file processing depends on the file s name but a typical process should be the movement of the file to a specific directory for this type of file (there is one directory for files with unknown type) and the execution of a Unix shell-script that receives as parameters: the name of the file, the nominal hour, the nominal day, etc. These parameters are built by the db_gest by extracting them from the name of the file. Reception (Tellique EUMETSAT SW) AEMETCast Services Compressed segments Wavelet decompress (EUMETSAT SW) HRIT & LRIT JPG&GIF generation McIDAS Secondary Server DB_GEST (msgt Linux) JPG & GIF McIDAS dataset (AREAs) Webserver (Web_catalog) McIDAS HRIT client McIDAS client WWW client Figure 4. Functions and Services Since all McIDAS-X command can be used as Unix command then a combination of Unix commands, McIDAS-X commands, or other programs can be joined in different shell-scripts which will be executed automatically when a new file is received. Thus, in this way the graphic image files can be generated in real time using McIDAS-X commands and dispatched to the web servers using the standard transfer Unix commands. DISPLAYING SATELLITE DATA AND PRODUCTS In order to display satellite data and products, it was decided to use the McIDAS (Man computer Interactive Data Access System) software, adapted to AEMET enviroment. Taking into account the big number of available products disseminated by EUMETCast, so that the different data format (GRIB, BUFR, HDF5, etc ), a suite of McIDAS secondary servers and format converters had to be developed in order to the system could display all of these products. A secondary server provides an interface to display data in different formats. The idea of the McIDAS secondary server is very simple and it is based on transform the request received from one McIDAS client in the read of the specific file and to build an answer to the client using the ADDE protocol. The process can be seen in the Figure 5 for the MSG SEVIRI images received on the EUMETCast receiving PC on HRIT files format. The McIDAS secondary server manages the request received from one McIDAS client by reading the requested file in native format, generating the file in McIdas format and building the answer for the client using the ADDE protocol. ADDE (Abstract Data Distribution Environment) is the client/server mechanism for distributing data in McIDAS. In ADDE, an application sends a request for data through an API routine to a server through the network infrastructure (intranet or internet). The server is the software running on a machine in a distributed system that stores data and supplies it to the client upon request. The server interpret the data request from the client, retrieve the requested data from disk using the specific library (as example for HRIT one with routines similar to the NWC SAF library), arrange the data into the proper format for the client and send the data back to the application using the McIDAS convention.

5 Forecasters can in real time using the McIDAS client-server to plot any combination of data and SEVIRI images or products using the ADDE clients (McIDAS-X, McIDAS-V or IDV). McIDAS Client It send a request to the server Secondary Server: XXXX Mcidas server analyses the request Group, identify the type of data It get the answer (directory, calibration, navegation, data) McIDAS Secondary Server It read the file in native format. HRIT/LRIT files received by EUMETCast It generates the file in McIdas format Figure 5. McIdas Client-Server configuration An OVERVIEW of the operative procedure for SEVIRI HRIT files The SEVIRI HRIT files received, decrypted and wavelet decompressed by the receiving PC are dispatching by db_gest tool to the specific directory from where they will transfer to the server PC. The db_gest tool in the server PC checks the file incoming on the specific directory where have been put by the FTP transfer, moves the HRIT files following the rules in the db_gest configuration file and when the EPIlogue HRIT file is received the db_gest move it to the SEVIRI HRIT directory and the script in the line-hrit of the db_gest configuration file is executed. This script triggers the execution of the conversion to AREA using the IMGCOPY for some regions as Spain, Europe, etc. The regions more requested by users are converted to McIDAS AREA format every 15 minutes in the operation chain in order to avoid the conversion on the fly for every user that asks for the same data. It is also possible getting images with polar stereographic projection, etc,. The visible channels are also normalized as if the sun were in the zenith over the pixel (see Martinez 2009) with a McIDAS command developed by J Ruiz, a former AEMET fellow. The normalization is made automatically inside the script for HRIT files and the normalized dataset are written in McIDAS datasets. From the McIdas datasets are generated graphic files (GIF or JPEG); some of the images are reduced using ImageMagick commands to thumbnails for displaying-icons. The graphic files are subsequently sent to the different web servers: AEMET web server, AEMET intranet web server, Web-Catalog web server and NWC-SAF web server through copy, remote copy, ftp or secure copy mechanisms depending on the web server. One the greatest advantage of the McDAS is the navigation and the remapping possibilities. The McIDAS on the fly is able to made the process to convert line element to longitude and latitude (and viceversa) using one navigation module (a Fortran routine executed on real time as a dynamic library), through this mechanism the task to convert from SEVIRI (satellite) projection to other can be made automatically. Nowadays there are datasets created for polar stereographic projection every 3 hours for forecasting department; one example can be seen in the Figure 7. It s also possible to overlay and plot other data over the SEVIRI (satellite) projection. In the Figure 6 the lightning strokes data are plotted in real time over the SEVIRI images and written as GIF files for

6 intranet or internet web pages. Another example is provided in Figure 6 where the NWC SAF winds are plotted over the IR10.8 channel image. High Resolutioin Winds (NWCSAF) HRV SEVIRI+ Lightning strokes Figure 6, left) NWC SAF winds plotted over satellite image and right) lightning strokes data plotted over satellite image The RGB images are built using the McIDAS command: IMGCOPY, IMGOPER and IMGDISP and the use of COMBINE option. One example of airmass RGB can be seen in the Figure 7. MSG-2 Airmass MSG-2 HRVIS Analysis RGB IR PS Figure7, left) not normalized HRV image; center: Airmass Analysis RGB image; right) IR10.8 image on PS projection The Figure 8 shows the RGB combination image used on the Internet AEMET web page. It was created by Jose A. Quirantes (AEMET McIDAS Unit) for public not familiar with satellite images interpretation to get that snow and high ice phase clouds were displayed in white instead blue. This Quirantes s RGB is made using as red component the maximum of VIS0.8 and NIR1.6 channels (instead the NIR1.6 channel used in the natural RGB). In the Figure 8 the natural RGB and natural modified RGB has been generated from the normalized visible datasets. Figure 8, left) natural RGB using normalized visible radiances; right) modified natural RGB image using the normalized visible radiances red component is maximum of 0.8 and 1.6 channels

7 MCIDAS SECONDARY SERVERS AND FORMAT CONVERTERS IN AEMET Together with the MSGT secondary server that allows McIDAS clients accessing MSG data in HRIT native format (as they are disseminated by EUMETCast) other McIDAS secondary servers has been developed for AEMET. The McIDAS secondary server McFSD and McMSAT allows the accessing to Foreign Satellite Data (FSD) GOES-11, GOES-13, MTSAT-2, and METEOSAT-7 data in LRIT native format (as they are disseminated by EUMETCast). The maintenance of these servers has high priority due to the international activities of some AEMET units. Examples of images can be seen on image 9 an 10. Meteosat 7 IR GOES 13 IR GOES 11 WV Figure 9, Foreign satellite images examples: left) METEOSAT-7, center: GOES-13, right) GOES-11. As other example of the McIDAS remapping capabilities the global IR image combining IR images from several geostationary satellites in Mercator projection is shown on Figure 10. It is available on the AEMET Internet web page on real time. GOES, MSG, MSAT and MTSAT Composite Figure 10, Mercator global IR image in AEMET Web Server composed from IR images of GOES, MSG, METEOSAT-7 and MTSAT satellites. Other McIDAS secondary servers are the ones for MPEF products (McMPEF), Land SAF products (McLSAF) and OSI SAF products. These servers allow reading products that are in hdf5, GRIB2 and BUFR formats and convert them to McIDAS format. Some examples can be seen on Fig. 11 and 12. MPEF Cloud Mask MPEF MPEF Cloud Height Total ozone Figure 11, MPEF products images in AEMET Web Server

8 LAND SAF Albedo LAND SAF DSL FLuxes Figure 12, LSA-SAF products images in AEMET Web Server The NWC SAF products are converted to McIDAS AREA files with a format converter (McSAFN converter) and displayed using MCIDAS for GIF files writing. They are available on real time on Internet in the NWC SAF web page Convective Rain Rate (NWCSAF) Layer Precip. Water med.lvl. (NWCSAF) Total Precipitable Water (NWCSAF) Figure 13, NWC-SAF products images examples. FUTURE WORKS After five years running 365 days 24 hours from 2006 one renovation of the Linux PCs is foreseen and with the new hardware one improving in the capabilities is expected. Another area of interest is migration of some scripts from McIDAS-X to McIDAS-V or IDV possibilities and displaying 3D images, volumes, etc. on real time. ACKNOWLEDGMENTS To all AEMET people who have helped to make this AEMETCast system possible. Especially: Marcelino Manso and the colleagues working in Satellites and McIdas units in AEMET. Thanks to Jose A. Quirantes and J. Ruiz. REFERENCES McIDAS programmer manual. M A Martínez, J Ruiz and E Cuevas (2009). Use of SEVIRI images and derived products in a WMO Sand and Dust Storm Warning System Institute of Physics (IOP) Conference Series: Earth and Environmental Science.