HUMIT Interactive Data Integration in a Data Lake System for the Life Sciences

Size: px

Start display at page:

Download "HUMIT Interactive Data Integration in a Data Lake System for the Life Sciences"

Harry Richards
5 years ago
Views:

1 HUMIT Interactive Data Integration in a Data Lake System for the Life Sciences PD Dr. Christoph Quix Fraunhofer-Institut für Angewandte Informationstechnik FIT Life Science Informatics Abteilungsleiter High Content Analysis & Information-intensive Instruments christoph.quix@fit.fraunhofer.de Vertretungsprofessur Data Science Leiter der Forschungsgruppe Big Data & Model Management RWTH Aachen University

2 Funding period: , funded by BMBF Use Case Partners Regulation requirements & Quality assurance Coordinator / Technology Partner

3 High Content Screening Automatic analysis by substructure Systematic variation in parameters, e.g. by compound or sequence

data extraction from multimedia data Data management not

4 Big Data in Life Sciences High-Content-Analysis Systematic Analysis of huge image sets Automated image analysis Meta data extraction from multimedia data Data management not only in life sciences Scientific Data Management Workflow integration

5 Zeta: Application Specific Platform Plugins Plugin Toolbar View Component Overlays Image Galeries Directory Tree Time Line Animation

6 Example Configuration Cell-Cycle Analysis Registration FB Detection Segmentation Tracking Classification Evaluation Result Cell-ID Position[x,y] Mother-ID Time-ID Size MeanIntensity TotalIntensity G phase Mitosis ImageName Well Site Wavelength SR100702Live_G12_w1_s1_t171.t 1 29,35-1 t if G12 s1 w1 SR100702Live_G12_w1_s1_t171.t 2 44,82-1 t if G12 s1 w1 3 63,465-1 t SR100702Live_G12_w1_s1_t171.t if G12 s1 w1 SR100702Live_G12_w1_s1_t171.t 4 97,363-1 t if G12 s1 w1

7 Metadata and data is managed files and filenames! is an inhibitor of Assay: cell cycle inhibition trichostatin A Histone deacetylase 1 File name: TSA_HDAC1_2.png Table name 7

8 Agenda 1. Motivation: Data Management in the Life Sciences 2. Requirements for Scientific Data Management 3. Data Lake Architecture in HUMIT 4. Summary

9 Scientific Data Data collected during the work of scientist Measuring results, test data, reports, analysis, Various file formats Excel, CSV, images/audio/video, text, XML, proprietary formats, Heterogeneous semantics Test vs. Result data, own vs. other data, timeframe, Idea Proposal Experiment Result Report

Traceability and reproducability is difficult because of manual

10 Heterogeneity is unavoidable Islands of data in separate projects and applications Integrated data analysis requires huge manual effort Traceability and reproducability is difficult because of manual processes Goal: From isolated data islands to (partially) integrated data landscapes

Requirements for Scientific Data Management Integration: Combined analysis of different data sources Traceability: Reproducability of research results

11 Requirements for Scientific Data Management Integration: Combined analysis of different data sources Traceability: Reproducability of research results Evidence in lawsuits: IP protection Reusability: Acccessibility for future usage Flexibility: Adapt to changes in the research processes Documentation Semantics Models

12 Agenda 1. Motivation: Data Management in the Life Sciences 2. Requirements for Scientific Data Management 3. Data Lake Architecture in HUMIT 4. Summary

Data Lakes If you think of a datamart as a store of bottled water cleansed and packaged and structured for easy consumption the data lake is a large body of water in a more natural state.

13 Data Lakes If you think of a datamart as a store of bottled water cleansed and packaged and structured for easy consumption the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples. Maintain source data in its original structure Postpone (semantic) integration tasks Manage metadata about sources, mappings, and data quality Provide interfaces for uniform querying and interactive exploration of the data lake James Dixon (Pentaho)

mappings, and data visualization Separation of data storage and data processing/transformation; raw data is stored with

14 HUMIT: Data Integration for High-Content Analysis Integration based on Pay-as-you-go Idea Incremental extraction and integration of data Interactive tools for exploration and querying of data, definition of semantic relationships and mappings, and data visualization Separation of data storage and data processing/transformation; raw data is stored with metadata in a Data Lake, thereby immediately available for data analysis; data integration and mapping done later (ELT instead of ETL)

15 Proposal for a Data Lake Architecture

Ingestion Layer Low Effort for loading data (ELT instead of ETL) Support for the extraction of metadata and data Degree of automatization (especially for metadata extraction)?

16 Ingestion Layer Low Effort for loading data (ELT instead of ETL) Support for the extraction of metadata and data Degree of automatization (especially for metadata extraction)? Schema extraction for semi-structured data (JSON, XML) Schema-on-Read Lazy Loading Data quality control Specify minimal requirements for ingested data Complement and annotate extracted metadata

17 Storage Layer Choice of data storage HDFS? NoSQL? RDBMS? A hybride solution is required, but A uniform interface for data access A uniform query language ( query rewriting and data transformation) Metadata Repository and Metadata Model Manage schemata, mappings, data quality information and data lineage Close integration of data and metadata Data quality management Monitor data quality of data stores Semantic enrichment of metadata Prepare data marts for specific data sets

Interaction Layer Explore & Search in data repository Less direct queries (SQL), more Google-like queries Query for metadata and data User interaction should be captured as metadata Definition of

18 Interaction Layer Explore & Search in data repository Less direct queries (SQL), more Google-like queries Query for metadata and data User interaction should be captured as metadata Definition of exact queries Identification of new data relationships Metadata & Data Quality Management Exploration of the data lake (what kind of information is available) Capture semantic annotations of users Provide data quality information to users & collect feedback

Data Quality Comprehensive data quality mgmt for a data lake is necessary Data quality management is more than just data cleaning goals, metrics, measurements, analysis, improvements Data quality

19 Data Quality Comprehensive data quality mgmt for a data lake is necessary Data quality management is more than just data cleaning goals, metrics, measurements, analysis, improvements Data quality needs to be checked already for ingested data Minimal requirements for data sources (e.g., provide metadata or certain data items such as identifiers) Manage data quality information in metadata repository and make it available to data users

20 Agenda 1. Motivation: Data Management in the Life Sciences 2. Requirements for Scientific Data Management 3. Data Lake Architecture in HUMIT 4. Summary

Summary Data management in life sciences is often file-based which limits reuse and reproducability of experiments Making the data available in a data lake system provides query, search and

21 Summary Data management in life sciences is often file-based which limits reuse and reproducability of experiments Making the data available in a data lake system provides query, search and exploration features to the users Data lake is in early concept and requires more research Within the HUMIT project, we are developing several components and the framework for a data lake system Metadata extraction ( CAiSE Forum 2016) Constance Data Lake Framework ( SIGMOD 2016) Data quality management ( QDB workshop at VLDB 2016) User interaction and data visualization

Data Lakes: A Solution or a newchallenge for Big Data Integration. Christoph Quix, DATA 2016

Data Lakes: A Solution or a newchallenge for Big Data Integration Christoph Quix, DATA 2016 christoph.quix@fit.fraunhofer.de FrequentProblems ofa Big Data Project Which data sources are available? WhereisthedatawhichI