ARMADA (Army RDT&E Meteorological Architecture for Data Archival. 4DWX Forecasters Training 27 February 2008

Transcription

1 ARMADA (Army RDT&E Meteorological Architecture for Data Archival 4DWX Forecasters Training 27 February 2008

2 Outline ARMADA ARMADA at the Range Break ARMADA ETL Demonstration ARMADA Feedback Session

3 The significant problems we have cannot be solved at the same level of thinking with which we c r e a t e d t h e m. - A l b e r t E i n s t e i n

4 Data Warehouse 1 A data warehouse is an application with a computer d a t a b a s e t h a t c o l l e c t s, i n t e g r a t e s a n d s t o r e s a n organization's data with the aim of producing accurate and timely management of information and supporting data analysis. The practice of data warehousing includes the storage of virtually all transactional data, master data (customer, material), and meta data 2 at a very detailed level. 1 From Wikipedia 2 Meta Data data about the data

5 Data Archival Systems ARMADA 1 (2007) Data Volume Human (1800) Paradigms Strip Charts (1950) Database (MySQL) (2002) Digital (ASCII) (1987) 1 Data Warehouse Time

6 Meteorological Observing Platforms at Dugway Proving Ground Platform Observing Unit Ceilometers Spatial Horizontal Vertical (Units) (Levels) 3 20 Frequency (Seconds) 15 Observing Parameters (Count) 1 Observation Records Per Day 345,600 Electric Field Meters ,419,200 SAMS 1 (Mesonet) ,488 Present Weather ,040 Wind Profiler ,880 PWIDS ,320 SODAR ,520 Towers (32M) ,200 Sonics (3D) ,200,000 Rawinsonde 5 >1000 N/A 8 N/A Tethersonde ,400 Yellow Continuous permanent Orange Continuous mobile Light Blue Manual mobile 1 SAMS Surface Atmospheric Measuring System 2 PWIDS - Portable Weather Instrumentation Data System

7 Pre ARMADA Data Collection Data collection was developed around the observing platform Databases: Microsoft Access, Microsoft SQL Server, and MySQL AB. Text files: delimited, fixed column, and terse ASCII Other media: magnetic tape, CDs, and paper Examples of method and media used for archiving SAMS MySQL database RAID 1 Sonics delimited or terse ASCII CDs PWIDS delimited ASCII on CDs Rawinsonde text files on personal desktop Metadata Typically archived in spreadsheets Metadata often archived on different computers 1 RAID - Redundant Arrays of Independent Drives

8 Current SAMS Range Database Problem Areas Quality Assurance Range Data None or minimal! Not included in SAMS DB Rawindsonde SODARs Problem PWIDS Tower Solution Create a quality assurance program Archive ALL range data Not archived Metadata Site History Location Sensors Calibration Archive ALL Metadata Data Ingesters Sensor Location SAMS only! Create an application to extract, transport, and load multiple data sets

9 Current Requirements Any meteorological data and/or metadata collected must be: Accessible Available Usable Data integration (modeling) Standardization Labels, units, and time

10 Centralizing the Data Goal of ARMADA is to centrally archive ALL data and metadata Relation database management system (RDBMS) Data warehouse Controlled by a MySQL AB. database server Single point input and output access Applying proper information assurance procedures (Army Regulation 25-2) can reduce single point failures Standardization of labels and units is applied to all components in the archive Database and table names follow predefined naming convention Column names follow Climate Forecast (CF) standard names and associated units (typically SI)

11 ARMADA System Concept SODAR SAMS PWIDS 1 Archive (Databases) 4DWX 4 Quality Assurance Tower Sounding Other/ Historical Data 2 3 Data Ingest Data Output

12 Field Data ARMADA (Army RDT&E Meteorological Architecture for Data Archival) ETL 1 Applications Serial Ports TCP/IP ARCHIVE End User Applications Web GUIs Text Files CLI 2 Web Portal (Metadata) 1 Extract, Transform, Load 2 Command Line Interface Quality Assurance 3 Army RDT&E Four-Dimensional Weather 4DWX 3

13 Standardization of Archive (1) Self Describing Standard naming conventions Database/Table/Column Column/Variable (Climate Forecast Compliant) International System of Units (SI) Applies to all archive components

14 Archive Databases (1) Platforms SAMS 1 PWIDS 1 Tower 1 META Sensor Fielded Non-Fielded Calibration 1 Note site metadata history table resides in platform database

15 Example of a PWIDS Database Tables_in_pwids pwids_2007_02 pwids_2007_03 pwids_2007_04 pwids_2007_05 pwids_month pwids_site-metadata -metadata Archive Table Real Time Archive Table Site History Table

16 Example of a PWIDS Table Field date_time unit_id site_id wind_from_direction wind_speed eastward_wind northward_wind wind_from_direction_qc wind_speed_qc eastward_wind_qc northward_wind_qc last_update_date_time Type datetime char(10) char(10) double double double double tinyint tinyint tinyint tinyint timestam p Purpose Date time (UTC) of observation Is a unique unit id of a platform (i.e., SAMS) The id of the site location and is subject to change Raw data fields that utilize SI units Note: Data types are not restricted to doubles Raw data quality control flag Note: flag notation has yet to be determined A time stamp (UTC) record of insertion or modification Note: temperature, moisture, and pressure have been left off

17 4/15/2007 WSMR Boundaries 12 2/2/2008 Unit ID = 12 2/12/ Site ID = Y 12 Unit ID = 12 Site ID = X Unit ID = 12 Site ID = Z

18 Example of a PWIDS Site Meta Table Fields unique_site_id unit_id site_id date_time_start date_time_stop longitude latitude surface_altitude above_ground_level site_name notes Type int char(10) char(10) datetime datetime double double double double char(50) tinytext Purpose Define a unique number to guarantee no duplicity (auto inc) Is a unique unit id of a platform (i.e., SAMS) The id of the site location and is subject to change Date time start for validity of record Date time end for validity of record Geodetic longitude (Note western hemisphere is negative) Geodetic latitude (Note northern hemisphere is positive) Altitude at the surface above mean sea level Arbitrary altitude of sensors above ground level Optional name for the site location Optional notes about the site location

19 Metadata Web Interface Sensor Sensor Location Calibration Site Location Users Options 1. Web portal and archive at DPG 2. Web portal and archive at each range

20 MEMORIZE Because table names use a - (dash), these are considered special characters. To escape this character use the back tick ` found on the left of the number 1 key. Note the back tick can be applied to any special character. Example: select * from `pwids_site-metadata`;

21 Ingest Applications (2)

22 Turn on/off process buttons Background colors in last data updated Yellow - idle Green processing Red auto update off Orange an output process is off Magenta no new data Current system clock date/time Configurable Title Last data record ingested site id and date + time Turn off/on Auto update Check box is on Enlarged visual Indicator of on()/off() processes buttons Button to manually force an ingest process (runs only once)

23 Configuration File Provides instructions to File Ingester to: 1. Extract Location and architecture of data file 2. Transform Converts raw data to ARMADA architecture 3. Load Writes or populates the data into ARMADA Coded in XML (extensible Markup Language) Readable and/or self describing Rigid Configuration editor is in the plans for FY08

24 Configuration file Broken down into two groups Global parameters Effects all data Applies to: Database connections Base directory Time of data processing Site or Unit parameters Effects on individual observations Applies to: Interpolation Calculations Upload location within ARMADA

25 Example <?xml version="1.0"?> <FILEINGESTER> <BASEDIRECTORY Dir="c:/fm" /> <DISPLAYTITLES Title="FIELD MILL" SiteTitle="FM ID" DateTitle="COOL" TimeTitle="TIME UTC"/> <TIMERGROUP TimeDataIO="60" TimeProcess="5" TimeAlarm="60" TimeStationUpdate="45"/> <DATABASECONNECT Source=" " Username="remote" Password="remote" Database="field-mills" Port="3307"/> <STARTPROCESS Auto="true" Database="true" Output="false" /> <GLOBALSITEINFO SiteTable="`field-mills_site-metadata`" SingleOBFileCount="28" MultiOBFileCount="0"> <SINGLEOBFILE FileName="c:/fm_data/EFM001_Tab1sec" StringSplitter="," EndOfLine="\\n" CharsAllowed=" :," -AppendFilePrefix="DCP0Z_" AppendFileDate="day" OverWrite="True" RemoveFile="True" SkipHeader="4"> <SITEID SiteID="1" Unit_ID="1" Site_ID="1" ColumnCount="13" VariableCount="6" EquationCount="0" SpecialEquations="0" TableName="`field-mills_month_test`" AppendFilePrefix="EFM001_Tab1sec" AppendFileDate="none" OverWrite="True" RemoveFile="True"> <DATETIME MySQLDT="-1" Year4="0" Year2="-1" Month="1" DayOfYear="-1" DayOfMonth="2" HourMinute="-1" Hours="3" Minutes="4" Seconds="5" UTCTimeOffSet="0" SystemTime="false" /> <INSTANCE> <COLUMNVARIABLE ColumnNumber="7" ColumnName="surface_electric_field" Type="Constant" Unit="NONE" /> <COLUMNVARIABLE ColumnNumber="8" ColumnName="status" Type="Constant" Unit="NONE" /> <COLUMNVARIABLE ColumnNumber="9" ColumnName="leakage_current" Type="Constant" Unit="NONE" /> <COLUMNVARIABLE ColumnNumber="10" ColumnName="panel_temperature" Type="Temperature" Unit="C" /> <COLUMNVARIABLE ColumnNumber="11" ColumnName="battery_voltage" Type="Constant" Unit="NONE" /> <COLUMNVARIABLE ColumnNumber="12" ColumnName="internal_relative_humidity" Type="Moisture" Unit="%" /> </INSTANCE> </SITEID> </SINGLEOBFILE>

26 Output (3) Commercial or open source applications MySQL Query Browser Microsoft Excel Custom applications Excel (Macros) Times Series Climatologies SAMS Report HPAC GUI Data Getter PWIDS Display Field Meter Display

27

28 PWIDS Display (Portable Weather Instrumentation Data System)

29 2D Electric Field (V/m) Contour Plot

30 Quality Assurance (4) Quality Control Installation ARMADA Measurements Data Collection Communication

31 ARMADA QC/QA Current Quality Control Some data tests PWIDS Sonics Wind Profiler (NIMA) Data are only QC ed for customer requested data Not archived in QC format Goal Quality Assurance 1. QA applied to all data 2. Create an archive environment to support QA (ARMADA) 3. Develop a QA program to a. Automate process b. Manual inspection c. Process in near real time d. Rigorous follow on tests

32 Comparison Current Manual Higher Labor Costs 1 Range Test + Visual Single Application Per Data Set Post Analysis Goal Automated Lower Labor Costs 1 Multiple Test + Visual Single Application For Multiple Data Sets Real Time Lower Confidence 1,2 Higher Confidence 1,2 1 Per datum 2 End user

33 ARMADA QA Flow Raw QA Flags GOLD Standard Quality Control Server Temporary Data Storage Fail 1 Tests Pass Fail Do Nothing Pass All Tests QCS: Developed in Python

34 QA Flag The net results of QA tests! Every variable in ARMADA is assigned a QA flag Results of every test are archived 1. All tests are packed into a type 2. A type can be archived as a single element in the database 3. Initially only pass/fail results will be archived 4. Possible to archive more test result information QA Flag is composed of 0 and 1 s that describe the results of the tests

35 MySQL Types Type Bits Bytes Possible Integers Bit Representation Bit 0 or 1 1/8 2 1 Nibble 1 4 1/ Byte Tiny INT Small INT , Medium INT * INT * Big INT * Not a MySQL type

36 Packing Results via Bits Pass Fail Not Tested P 0 F 1 Above Increasing info per test N 00 P 01 A 10 B 11 Below Validity Test A 1010 B 1011 C 1100 D 1101 E 1110 F Base 4-Base 8-Base 16-Base N-Base Bit Oct Hex

37 Human/DB/Computer (RELATIONSHIP) Validity Check Human Not Tested Pass Above Below MySQL (Integer) Computer

38 QC Approach Raw QA Flags Quality Control Server Config File (XML) Temperature_QC = int (Flag) Manual Validity Tests 1 Persistence Buddy 1 All tests are performed

39 ARAMADA Implementation Pass/Fail A few test on SAMS Manual Validity Persistence Step test Buddy Check Will take years to implement through out ARMADA!

40 ARMADA SUMMARY All data and metadata will be archived in a central location Repository will be standardized Naming conventions SI Units Quality Assure all data 4DWX access Questions?

41 ARMADA at the RANGE

42 SAMS DB Migration to ARMADA SAMS Ingest will be replaced by File Ingester Allow non-fixed number of elements in SAMS data stream Interpret numbers and strings Capable of archiving additional derived variables (See hand out) SAMS DB will conform to ARMADA standardization SAMS DB will be reconfigured around range needs SAMS DB will be rebuilt during range installation SAMS DB and ARMADA will run simultaneously indefinitely (suggested 3-6 months)

43 ARMADA Installation Software 1. Fileingester SAMS Rawindsonde 2. MySQL 5.x 3. Range QC 4. Configurator? Setup 1. Software install + database setup 2. SAMS + possible other data sets 3. Training 4. Upload old SAMS DB to ARMADA One week installation (4-5 days) Starting FY08, except CRTC August 07 Run old SAMS DB simultaneously for 3-6 months

44 Hardware Desktop Server RAID Current Hardware Current Hardware (Data Ingest) Quality Assurance Database + Output (Web) Storage (4TB) Data Ingest Quality Assurance Output (Web) Database Storage (8TB) FY 08 FY FY Acceptable Recommended Optimal

45 Backups/Redundancy All raw data will be archived 1 Ingest software will be able to Write raw data to local files Read raw data files and populate into ARMADA Database Tables Updating tables daily or weekly Archive tables once or has been updated Will be automatic Master archive for all ranges located at DPG Need to get Port 3306 open through the firewall! 1 At a minimum raw data should be archived on long term transferable media such as CD or DVD

46 Information Assurance Adhere to Army Regulation 25-2 Information Assurance and comply with local DIACAP (DoD Information Assurance Certfication and Accreditation Process) requirements Tighten up access to ARMADA No more global IP s No remote root access 1 Metadata web portal will be password protected Sensitive data can be archived in its own database, and is not limited to the master archive 1 Not possible on PC based systems

47 Maintenance/Administrative Responsibilities 0-2 Years 1 Range DPG 1. Manage raw data archives 2. Load archived raw data 1. Primary MySQL admin 2. Backup database tables 3. Assist in backup and recovery 4. Maintain metadata 5. Monitor data flow 3. Master repository 4. ARMADA development 5. Customer support 1 Negotiable

48 Out Years Objectives FY Fall Winter Spring Summer 07 Install ARMADA at CRTC 08 Install ARMADA at ATC, EPG, NVL, RTTC, WSMR, and YPG Master repository at DPG Replace Legacy Programs (SAMS Report) Quality Assurance Program Version 1 Upgrading File Ingester for SODAR/Profiler Metadata Web Server operational Decommission SAMS DB Update Hardware (rack mounted system) 2. Web interface or GUI tools to ARMADA, climatology, time series, etc 3. Expand ingest capabilities beyond File Ingester 4. Primary range data source to 4DWX 1. QA Program Version 2 2. All range data + historical data in ARMADA 3. Develop data mining tools Migrate all capabilities to Linux? 2. Include non-meteorological data (Test reports, HPAC output, etc.)

49 Summary ARMADA is coming! CRTC August All other Ranges Fall 07 to Winter 08 Plan on a full week ARMADA will replace SAMS DB New hardware is not required for initial install ARMADA will comply with the local DIACAP Questions?