Time Series Data Management in Air Quality Monitoring. AQE 2013, Telford/UK. Edgar Wetzel, 14 March 2013

Transcription

1 Time Series Data Management in Air Quality Monitoring AQE 2013, Telford/UK Edgar Wetzel, 14 March 2013 Brief company presentation KISTERS AG software company with core competence: RDBMS-based time-series management B2B and G2B customers world-wide +500 employees in offices all around the globe (4 continents) 50 years of experience since the company started back in 1963 KISTERS AG 14/03/

2 Measurement data: short term use monitoring station permanent = continuous in time fixed= stable in space monitoring station temporary = discontinuous in time mobile = moving in space measurement data <site><date+time><value><flags> multiple software packages from distinct datalogger and instrument manufacturers data user multiple measurement data a time series! validation + analysis regulatory reporting archiving KISTERS AG 14/03/ agency Measurement data: quantity and variety data are measured all over the place PM 10 meteorology NO 2 ambient air quality NO X CEMS deposition PM 2,5 CO 2 O 3 concentration quantity C KISTERS AG 14/03/ wind speed + direction 2

3 Measurement data: quantity monitoring networks consist of numerous stations each stations monitors several meteorological and air quality parameters large-scale trends can be determined using trend analysis on long-term data sets station type: background traffic industry KISTERS AG 14/03/ What happens to the measured data? it is processedand ends up in regulatory reports KISTERS AG 14/03/

4 What happens to the measured data? next it is archivedand more often than not forgotten further use is limited due to... limited availability difficult accessibility incompatible file formats inconsistencies unstructured archiving unrelated data sets KISTERS AG 14/03/ Each data points has an economic value! -1- measuring data costs money equipment human resources maintenance archiving data costs money albeit prices for data storage are steadily declining thus price is no limitation to long-term archiving of mass data! KISTERS AG 14/03/

5 Each data points has an economic value! -2- there is a scientific and economic value in the information potentially derived from the data information must be derived from the data data from different disciplines must be correlated communication and warning: defects preventive maintenance disaster early detection of critical situations transparent information policy information quality increases by means of... tried & tested validation procedures optimized analysis/evaluation tools data correlation algorithms KISTERS AG 14/03/ Long-term data analysis each new measurement point increases simultaneously both the economic value of the time series the statistical significance of the data pool BUT... using many software makes evaluation difficult data piles don t have any statistical or economic value if they are not properly organized/structured/managed/documented(meta data) accessible available quality assured Data piles need to be managed by a single enterprise solution software to maximize return on investment! KISTERS AG 14/03/

6 How to improve? Time-Series Data Management Systems Enterprise Solution... RDBMS-based advanced... time-series data management time-series data analysis multi-tier architecture manage in one DB both current data (measured & calculated) legacy data (measured & calculated) assure all of the following functionality: centralization, validation, aggregation, processing, alerting, reporting, dissemination, and public information. KISTERS AG 14/03/ GUI applications application server data base RDBMS AquisNet in brief Field-proven software built on 15 years of experience in managing huge amounts of current and legacy AQM data! AquisNet == Air quality information system NETworks software family providing modules for the 2 main blocks of an AQM network: Data Center: centralization, validation, aggregation, processing, alerting, reporting & dissemination RDBMS-based advanced time-series data management and analysis DAS: optional data acquisition system, i.e. the monitoring station software sample data from analogue, digital and smart sensors/analyzers run function tests and calibration cycles plausibility testing and local data storage KISTERS AG 14/03/

7 Air Quality Information System: core functions Configuration & Definitions system wide catalogues/ key list regulatory compliance rules (EC, US EPA, UBA...) predefined and userdefinable data groups data access permissions & data views Operational Meta Data Management site station (measurement location) parameter (channel) time series definition Time Series Data Management data feeds (import/export) data validation and editing calculation of derived time series data publishing Time Series Data Analysis statistics and air quality indicators manual or automated reporting graphics and numeric tables Performance Reporting (KPI) KISTERS AG 14/03/ What s a time series for KISTERS? any data that is linked to a location and has a date/time stamp KISTERS will increase the value of the TS data by means of... data centralization and subsequent validation appending quality and other information to the measured value creating an auditable track of data versions providing an organized data source for other uses providing an organized data sink for other data providers current state of production time series transaction history KISTERS AG 14/03/ data quality state, changes Y/N +comments 7

8 Typical time series data management workflow measurement measure <date+time><value> raw data data transmission plausibility check <date+time><value><quality flags> 1 st level QA checked data centralization and storage receive from station and record in RDBMS station meta data -> sequence (<date+time><value><quality flags>) original measurement data increase quality and set additional QA flags accordingly validation station meta data -> sequence (<date+time><value><quality flags>) 2 nd... xthlevel QA checked production data aggregate, analyze processing station meta data -> sequence (<date+time><calculated value><quality flags>) derived data: temporary or permanent reporting KISTERS AG 14/03/2013 produce report chosen report format report Data centralization and storage AquisNet SODA - KISTERS telemetry unit Data collection and communication on all channels Internet, phone, GSM & GRPS, radio communication, satellite, web services supports both push and pull modes! Central soft-and hardware for all data Consistent planning and monitoring Data security & user management Powerful and scalable 8

9 Data validation AquisNet supports both... fully automated algorithm-based validation visual inspection and validation All validation steps are... documented in data quality flags in a versioning system allowing traceability & rollback commented AquisNet DMO Graph/Table Editor time series explorer interlinked table + graph views multiple time series + scales/legends + graph types KISTERS AG 14/03/

10 Data processing data security by means of controlled redundancy all incoming data are write-protected and will remain untouched a copy will be produced for validation and further processing automated data aggregation derived time series (mean, max, min, percentiles,...) are calculated automatically as new data comes in on the fly calculation of air quality indicators statistical indicators temporary time-series for ad-hoc analysis AquisNet time series analysis regression calculation: graph view + regression equation programmable equations: transform input time series into derived time series KISTERS AG 14/03/

11 Data reporting reports can be... pre-defined by the user executed automatically following a pre-determined schedule executed manually on demand in the ownership of the system (all users), a single user, a user group numeric tables or graphs or a combination of both data formats for reports... national or international standard formats (EC, US EPA, UBA, etc.) standard file formats (csv, pdf, jpg,...) user-defined formats (optional) If it s in the DB (monitoring & station data, other data)it can be in a report! All tables & graphs on the screen can be part of a report! Reporting can be manual or automated! AquisNet REP time series graphs time series line graphs: multiple series, w/o numeric table 11

12 AquisNet REP sample line/bar graphs XY graphs with labels line + step chart box-whisker plots scatter plot + linear regression tiled bar graphs with labels tiled line graphs with sum bars AquisNet polar plots (w/o table) pollution rose wind rose 12

13 3/14/2013 AquisNet WEB (option) web site content generation AquisNet Web generates dynamic content that you can insert into the existing framework of your own web site design! air pollution network Air Quality Index synoptic map KISTERS AG 14/03/ AquisNet Dashboard (option) desktop + mobile app 13

14 ... mobile apps... Why KISTERS is involved in interoperability... data hub: search and retrieve the data you need... WISKI Other wheredo I find the datai needformy climate change research? Web Services send request receive data catalogue send specific request receive TS data Web Services KISTERS DATA HUB Web Services Other Other Other WISKI WISKI WISKI Web Services INTERNET 14

15 AquisNet hosting options customer premises AquisNet can be hosted at the customer s or remotely in the cloud! AquisNet Data Center - future proof investment modern technology: flexible, scalable and customizable multi-tier software architecture: RDBMS Business Layer Rich Client RDBMS Business Layer Web Server Browser Client single or multi-user with full-featured user-administration easy to integrate in existing IT environments: choice of RDBMS s (Oracle, MS SQL) & operating systems (MS Windows, Linux) independent of any instrument or data logger manufacturer KISTERS AG 14/03/

16 KISTERS + AquisNet advantages in a nutshell AquisNet: vertical solution for the air quality community especially developed for AQM and related applications built on KISTERS time-series technology: solid, proven, future-proof +15 years of AquisNet KISTERS: 50 years of existence in the world-wide market solid business, markets and technological experience presence in all 4 corners of the world: close to the customers multi-lingual staff & intercultural experience dynamic + enthusiastic staff: solid mix of experienced and young employees +25 years of experience in time series data management close supervision, testing and implementation of new technology secured RDBMS-based technology with built-in redundancy (backup) transaction log and full rollback truly field-proven in various environments: B2B/B2G customers KISTERS AG 14/03/ Thank you for your attention. Nous vous remercions de votre attention. Vielen Dank für Ihre Aufmerksamkeit. KISTERS AG Charlottenburger Allee 5 Tel info@kisters.de D Aachen Fax