Data Management. Updated Guidelines for Atmospheric Trace Gas Data Management GGMT 2017

Size: px

Start display at page:

Download "Data Management. Updated Guidelines for Atmospheric Trace Gas Data Management GGMT 2017"

Brittany Wilkins
5 years ago
Views:

1 Data Management Updated Guidelines for Atmospheric Trace Gas Data Management GGMT 2017

2 The Guidelines are referenced in the GAW recommendations Original guidelines were written in 1998 Intended to help programs new to the field avoid common pitfalls Updated to reflect current design and management strategies

3 Data integrity Accessibility Recoverability Traceability

5 Why Do We Need A Strategy? Long term data storage is difficult NOAA has almost 50 years of measurements and we still struggle with this Data should be permanent and permanently useful

6 Topics The case for a RDBMS Goals of a successful data management strategy How they are met in the Carbon Cycle Group

7 Advantages Of A RDBMS Practical advantages Designed to share large datasets with multiple users Centralized repository that manages data access and modifications Managed service handles difficult technical issues like concurrent access and security

8 Advantages Of A RDBMS Logical advantages - Relational data model Separates storage details from presentation - adaptable design Normalized design minimizes redundancy and inconsistencies Allows complex relationships to be stored Coded data allows program logic to make decisions based on the data entered

9 Coded dictionaries allow easy translation to various standards

10 How Data Gets Stored is Important Relational databases are difficult to implement and maintain - Carbon Cycle spends ~25% on data management and QA Ease of implementation should be weighed against long term goals of the program Well architected data storage provides the structure that the rest of the data management strategy is built on The benefits of a managed repository outweigh the initial costs of implementation

11 Data integrity Accessibility Recoverability Traceability

12 Data Integrity Ensure the long term accuracy and consistency of stored data Carbon Cycle group has several custom software programs to manage with our data Policy and procedures are important - Operations Manager Operations Manager handles logistics for over 60 sites, 5 labs It has many software checks to validate entries, policy and procedures are required to make sure they are correct

13 Flexible Architecture Allows Adaptable Design Documenting and managing issues with the data is an important part of quality control The 3 character positionally significant flag is very good at conveying information that users want, but not flexible enough for internal users to manage and track issues Only character per position Issues can t be related to more than one measurement Limited ability to document How you store data doesn t have to be the same as how it s presented

14 Data Tagger Data Tagger is a multi year effort to address shortcomings Multiple issues (tags) can be associated with a range of samples or measurements Issues can be fully documented to provide context Improves communication along the measurement path All entries are logged to provide traceability Issues are coded which allows them to be combined into a 3 character external flag

15 Accessibility Ensure data are available External Availability 50 years of sampling, ~100 million measurements. Much is publicly available, but not all All data sets are available by request BEDI project will make all data available via Obspack, some still preliminary Very configurable table driven meta data Download and usage statistics for data providers, communication channel to users

16 Accessibility Ensure data are available Internal Availability Internally, ability to view, plot and analyze data from all of our different methods, platforms and lab programs is widely available for research and QA Example; Aircraft group uses elevated H1211, a halon compound used in fire extinguishers, to flag suspected cabin air leaks Possible because HATS data is integrated into our centralized database that can easily retrieve related data We now put extinguishers in all our planes In the process of automating this logic

17 Recoverability Ensure data are recoverable in the event of catastrophe Must have a plan, should be automated What is acceptable data loss? Range of options from weekly offsite snapshots to HA failover As acceptable data loss goes down, cost and complexity go up Carbon Cycle has a fairly complete strategy that allows recovery to multiple points in time, but under certain scenarios up to 6 months of data loss is possible Local, offsite and archive (WDCGG) backups

18 Traceability requires that every measurement must be traceable to primary standards through an unbroken hierarchical chain of direct comparisons to successively higher order standards.

19 Traceability Ensures data are comparable across time and programs Changes to assigned values must be anticipated and planned for Affected raw instrument output must be able to be reprocessed Our process is fairly automated, although some datasets are difficult to update Documentation is key As the reference lab, we also need to be able to test scale changes Test environments to simulate scale updates Discrepancies can be sorted and filtered to find problems

20 Summary The repository used to store data forms the structure on which the rest of the data management strategy is built, this requires resources A comprehensive strategy must: Ensure data integrity Provide ready access to data Enable recovery options Plan for updates to scales and standards

Data Storage, Recovery and Backup Checklists for Public Health Laboratories

Data Storage, Recovery and Backup Checklists for Public Health Laboratories DECEMBER 2018 Introduction Data play a critical role in the operation of a laboratory information management system (LIMS) and