Scalable solution: cost effective LITE version full-featured CDMS* with variety of optional modules available cloud solution High Availability Option Manual or Automatic Data Sources Highly Configurable Quality Control Metadata Management Export to Standard Formats Mathematical / Statistical Packages Advanced Graphical Presentation Tool and Map Analysis Universal & Reliable Climate Database Management System *CDMS - Climate Data Management System Climatological database is a critical component of any modern meteorological institute of today. The main utilization of such system is storage of all collected meteorological data in one unified structure to avoid data inconsistencies and discrepancies and to enable standard comfortable data access for all users and other software systems. Within the Climatological database (CLDB), the above-mentioned unified structure is based on SQL Database Server and the standard data access is based on SQL language. The guarantee of data storage quality is the industry-proven Oracle Database Server, the world leader in database technologies. Great alternative for smaller systems is based on an open source solution offered by PosgreSQL. customization. The end user can specify additional non-standard input and output modules. Modules can be easily implemented and added to any existing or future installation. One of the most interesting extensions offered is upgrade to environmental database (integration of AWS, radar, satellite, profiler, historical observations, marine data, radiation and air pollution monitoring and more). To provide high availability of the database we use Oracle Real Application Cluster - the database solution provides fault tolerance and high performance for mission-critical applications. CLDB is based on WMO 1 recommended practices for climatological data processing (WMO Guide No. 100). It follows the WMO suggestion of a RDBMS (Relational Database Management System) application with wide use in climatology (World Climate Program efforts concerning new Climate Data Management Systems - CDMSs). Great advantage of CLDB is its modular architecture, which gives the end user the possibility of detailed 1 World Meteorological Organization Customized CLDB Menu 1
Connection and Data Input The CLDB can hold textual and numerical data, graphical information and animations. The database is capable of receiving, decoding and storing of the following data types from different data sources: 1. Data electronically imported from third party database system (e.g CLICOM) 2. Data manually entered or sent by SMS from manned stations with regular or irregular observing schedule. 3. Data from meteorological messages received via GTS2 or other distribution system: Surface (SYNOP, METAR) Upper air (TEMP, PILOT) Marine data (BUOY) binary messages (BUFR, GRIB, NetCDF, HRIT) special (AIREP, AMDAR) 2 Global Telecomunication System 4. Data collected from automated weather stations (AWS); distributed in various formats, including: Comma Separated Value (csv) Other Delimiter Separated Values Text Files (txt) MS Excel 2000-2016 (xls) Open Document Format xml (schema/dtd/raw) 5. Remote sensing data: radar and satellite images: Satellite IR, WV, VIS channels, colored RGB composites Radar products: Intensity, Rainfall Rate, Velocity Lightning positions Wind profiles 6. Model predictions (NWP outputs) 7. Other numerical, textual, binary or graphical data according user requirements can be stored in CLDB after proper analysis GTS Messages Marine Data Upper Air Sounding Historical Archive Manual data entry Satellite Radar and GRIB Data AWS Data Hydrological Data External Data Storages Synoptic station data Agromet station data Rainfall station data Sunshine and solar radation data Tropical cyclone data Severe weather data Astronomical handbook Data Collection Decoding module Data Convertor Syntax Verification Conversion Utilities Data Repair and Corrections Terminal Input Automatic Data Import Manual Data Import Quality Control Elimination of Duplicates Internal Consistency Validity Temporal Consistency Spatial Consistency Seasonal Control Database and Intranet Web Portal Metadata Storage Data Storage Access Control Local Security Data provision Distribution Publication Products Tabular Reports Data Charts Tropical Cyclone Tracks Realtime Data Maps Map Analysis Windrose Charts Upper Air Diagrams Radar Images Satelite Images Forecast Views Scheme of functions 2
Data Processing and Quality Control Climatological data in the CLDB system are processed by different modules before presentation to the end user. The data are collected from different sources by the respective UDCS (Unified Data Collection System) module. The module decodes the received data and passes them to the database as raw and unchecked values. The decoding mechanism is customizable to client message types. Periodical quality check monitors data validity (elements limits), internal consistency (elements relationships), temporal consistency (rate of change), standard deviation, spatial consistency and other more complex relationships between three elements. The quality checks are fully configurable by the database administrator in a very user-friendly interface. Manually entered data are checked simultaneously with the key entry process. A more advanced quality check is performed periodically at a later time in the database. During quality check processing, CLDB calculates derived values (e.g. sea level pressure, vapor pressure). The calculated elements and derivation formulas are customized according to customer s requirements. Emphasis on Metadata Spatial QC configuration Detailed description of observing stations and their history are inevitable part of climate data itself. CLDB extends CLICOM metadata into a powerful textual and graphical metadata management tool. In addition to the standard descriptive values, system administrator can define new metadata entries. Metadata cover these areas: Station Instruments module contains searchable database entries for sensor parameters traceable to individual sensor piece with assigned serial number Textual and numerical information is accompanied by copies of datasheets, photos of sensor and environment and calibration certificates with possible e-mail alerts announcing recalibration need Station Observation describes observational program, setting and environment complete with panorama photos and maps Full history of instrumentation and applied corrections is tracked Station Maintenance module keeps records of regular and special maintenance Interfaces are fully web based which creates possibility to access the system from the field via mobile networks System can ingest all kinds of metadata: WMO required and recommended, nationally developed, special metadata of individual scientific projects and inter-comparisons, Leroy siting classification. 3
Reports and Statistics The data stored in the database can be viewed and exported in many ways. 1. MicroStep-MIS selection tools End user applications are provided by MicroStep- MIS for retrieving data from the database. Maximally flexible export from CLDB to user-defined tables is possible with user-friendly CLDB Select tool. When selecting the data, user can set element(s), variable(s), region(s), station(s), observation terms or time periods as a retrieval criteria. User then selects the form of output - all raw data, aggregated data (minima, maxima, means, sums, counts, standard deviations), or a long year statistics over selected month. The resulting data is in the form of a graph or a table. The table could be saved as a spreadsheet or a text file (comma separated values, space separated values, ). The output files are directly usable by standard mathematical/statistical software products or data mining tools. Example of a report generated by the Unified Meteorological Information System developed for the Philippines Atmospheric, Geophysical and Astronomical Services Administration (PAGASA), Republic of the Philippines. 2. User s standard desktop tools End user can work with the database within standard desktop publishing, statistical and graphical tools which are capable of retrieving data using standard interfaces like ODBC or contain tools for direct connection to Oracle Database. The user can also export files to Spread Sheets (e.g. with Microsoft Excel ) with simple few clicks. Tabular reports: climatic elements summaries (means, extremes, counts) long-term means and extremes percentage frequencies of occurrence of concurrent wind direction and speed frequencies of element occurrence below/above the specified thresholds return periods, cumulative frequencies characteristics of distribution (measures of central tendency mean, mode, median, fractiles (percentiles), vector mean; measures of dispersion, coefficient of variation, ) Upper Air summaries counts of missing data customized tables and many others 3. Reports - predefined CLDB statistics The CLDB Reports application provides an easy to use interface for generating tabular and graphical reports (daily, monthly, annual, over normal periods). Reports are generated directly in printable form (.pdf) or as Excel worksheets. Graphical outputs: wind roses (user definable colors, scale, 4, 8, 12, 16 sectors) elements mean, max, min graphs rainfall bar customized graphs and many other graphs 4
Aditional Tools Map Analysis Map Analysis is a flexible computer graphics system designed for professional usage, which provides spatial visualization of climatological information and helps users by creation of more comprehensive views of historical and currently received data: map analysis of current data, historical data and normals comparison analysis of actual data and normal (selectable normal) comparison analysis of actual data and historical data (selectable any historical period) comparison analysis of selected historical data with selected normal selectable percentile values for all meteorological variables in any period Standard Precipitation Index, other indices 5
Aditional Tools Radar and Satellite Data Application This application enables to display, manage, integrate into composites and archive radar data from various sites and manufacturers as well as satellite data. (c) Eumetsat 2016 6
7
Aditional Tools Numerical Weather Prediction Products Application The application serves for displaying, managing and archiving of Numerical Weather Prediction products. 8
Aditional Tools Map Server Map server enables publication of the spatial data using open standards and the implementation of the Web Map Service. The maps could be created in variety of formats. Connection to the traditional GIS architecture such as ESRI ArcGIS Topographic/geographic data Station data layer Meteo data layer spatial climatic analyzes Movie mode 9
Technologies and Standards Multi-platform product The server and client can run on: Microsoft Windows Linux Presentation Tier: Web interface: jquery, Mozilla, Chrome, Internet Explorer Mobile applications: Android, ios Industry proven technologies: SQL, Oracle Java XML HTML5, Java Script, CSS App screens Java Web Services Documentation CLDB comes with extensive user s and administrator s documentation. User s guide contains examples and how-to instructions. Administrator s guide is especially focused on installation, database model, quality control settings and electronic data import. Additional guidance is covered by YouTube videos. IMS CLDB Web Screen for Intranet and Internet Mobile App 10
Modules and Licensing Module / functionality UDCS/CLDB Lite UDCS/CLDB EnviDB.Cloud Order Code Web interface (English) ü ü ü Web interface (other languages) extra option extra option ü Data collection from MicroStep-MIS stations (station number may be limited in UDCS/CLDB LITE if licensing per station is selected) Data collection from 3rd party stations Migration of the legacy / historical data ü ü ü extra option extra option various extra option UDCS/Hist Import of FM-12 SYNOP codes ü Import of FM-15 METAR codes ü Binary data collection (radar, satellite, NWP) using standard protocols (WMO, file based) ü ü Camera images support extra option ü ü UDCS/Lite/Camera Camera image visibility assessment extra option VIS/Camera Station and variable metadata ü ü ü Advanced metadata module extra option CLDB/Meta AWS Network Status map ü ü Database statistics ü ü ü Select application ü ü ü Manual data entry application (standard forms) Manual data entry application (customized forms) Reports application (basic reports, no customization) ü ü ü extra option ü ü ü Customized reports extra option CLDB/Reports/Custom Quality control - ruled based QC ü ü ü Quality control - spatial QC extra option CLDB/QC/Spatial Quality control - NWP QC module extra option CLDB/QC/NWP Basic text current data screen ü ü ü Graphical current data screen extra option ü UDCS/Lite/Current Complex WIS compatible metadata management extra option CLDB/WIS Wind roses ü Database of normals and extremes ü Map analysis basic extra option IMS4/Maps/Basic Map analysis advanced (difference from normals, from long-term averages, SPI index) extra option IMS4/Maps/Advanced 11
Modules and Licensing Module / functionality UDCS/CLDB Lite UDCS/CLDB EnviDB.Cloud Order Code FM-12 SYNOP (TAC) ü FM-15 METAR, FM-16 SPECI (TAC) ü FM-71 CLIMAT (TAC) ü TDCF (FM-94 BUFR, FM-95 CREX) code creation support extra option CLDB/TDCF National code forms extra option CLDB/Codes Homogenization toolset extra option CLDB/Homog ETCCDI Climate Indices extra option CLDB/ETCCDI High availability configuration extra option ü UDCS/D, CLDB/D Upper air data import (TEMP), processing, aerological diagtram extra option CLDB/TEMP Hydrological database module extra option extra option IMS4/Hydro Radiation data database module extra option extra option IMS4/Rad Tropical cyclone data module extra option IMS4/Cyclone Satellite weather data extra option IMS4/SAT Radar studio extra option IMS4/RADAR Lightning studio extra option IMS4/LTNG AMDAR module extra option IMS4/AMDAR Multimedia data database module extra option IMS4/Media ModelSuite database - forecast data integration extra option ü IMS/MSDB Warning suite - base, CAP extra option IMS4/Wrng/Base Warning suite - meteorology module extra option IMS4/Wrng/MET Warning suite - hydrology module extra option IMS4/Wrng/Hydro Warning suite - lightning warning module extra option IMS4/Wrng/Ltng Map Server (zoomable layered maps) extra option ü IMS/MapsDB + modules PDF documentation ü ü ü Printed documentation extra option 12