Discovery Net : A UK e-science Pilot Project for Grid-based Knowledge Discovery Services. Patrick Wendel Imperial College, London

Transcription

1 Discovery Net : A UK e-science Pilot Project for Grid-based Knowledge Discovery Services Patrick Wendel Imperial College, London Data Mining and Exploration Middleware for Distributed and Grid Computing, September 18-19, 2003

2 Why Discovery Net? Data Challenge: Distributed, heterogeneous & large scale data sets Novel and real-time data sources Resource Challenge Novel specialised data analysis components/services continually being published/made available Computational resources provided Information Challenge: Data cleaning, normalisation & calibration New data needs to be related to existing data Knowledge Challenge: Collaborative, interactive & people-intensive Result interpretation & validation in relation to existing knowledge Knowledge sharing is key

3 What is Discovery Net Goal : Construct an Infrastructure for Global wide Knowledge Discovery Services Key Technologies: Grid and Distributed Computing Workflow and service composition Data Mining & Visualisation. Data Access & Information Structuring. High Throughput Screening Devices: real-time.

4 Discovery Net: Unifying the World s Knowledge Data Integration: Dynamic Real Time Construction of Data Grids Application Integration: Component and Service-based Integration People Integration: Global-wide Discovery Groupware Knowledge Integration: Multi-subjects and Multi-modality Integrative Analysis to Cross Validate and Annotate Related Discovery Work

5 What is Discovery Net Information Scientific Discovery Scientific Literature Real Time Integration Workflow Construction Databases Operational Data Images Dynamic Application Interactive Visual Integration Analysis Using Distributed Resources Instrument Data

6 Discovery Net Layer Model (Life Science Application) D-Net Clients: End-user applications and user interface allowing scientists to construct and drive knowledge discovery activities Deployment Web/Grid Services OGSA D-Net Middleware: Provides execution logic for distributed knowledge discovery and access to distributed resources Computation & Data Resources: Distributed databases, compute servers and scientific devices. High Performance and Grid-enabled Transfer Protocol (GSI-FTP, DSTP..) Grid-enabled Infrastructure (GSI)

7 Goal: Plug & Play DNet server DNet Server A Knowledge Grid based on D-Net Servers Data Sources, Analysis Components and Knowledge Discovery Processes Knowledge discovery services InfoGrid Data access & Storage Components Computation Deployment DNet API DNet server DNet Server DNet participating client DNet Client DNet Server XML DPML Internet DNet client DNet Client Web client WWW RDBMS Data sources Computational services Several types of clients for different usage (from thin web client to participating client) Current implmentation based on Java distributed objects (EJB), moving towards Web/Grid service But deployment and API access through standard Web/Grid service

8 Discovery Process Management Workflow based service composition Data-flow approach fits Knowledge Discovery process Allows scientists to develop processes. Towards a Standard Workflow Representation for Discovery Informatics: Discovery Process Markup Language (DPML): Contains component data-flow graphs, but also Records collaboration information (user, changes) Records execution constraints (location, parameterisation) Becomes a key intellectual property: Discovery Processes can be stored, reused, audited, refined and deployed in various forms D-Net Workflow for Genome Annotation : 16 services executing across Internet

9 InfoGrid: Dynamic Data Integration Dynamic Data Integration = On-demand access to heterogeneous data sources + information structuring Towards a Dynamic Information Integration Methodology: Specialised Information Source Access: InfoGrid allows users to register, locate and connect to various specialised information sources. On the-fly Integration: InfoGrid allows Journals Project Reports Patents Structures Libraries Catalogues Trails Patients Clinical Journals Integrative Analysis Chemistry Gene Biological Screening Protein / Targets Sequence Activity Protocols Toxicology Metabolic Pathways Sequence Structure Location users to build their own integration Synthetic Expression Function structure on the fly (Worst case: pathways Function proprietary protocol/format, best case JDBC/HTTP-XML-XPath/Web Service). Easy Maintenance: Wrappers/Drivers to new data sources can be added through a clean API

10 Dynamic Application Integration Services Dynamic Application Integration = Ondemand access and composition of remote analysis components Clustering Classification Towards a Dynamic Component Integration: Component service: allow users to register, locate and remotely execute components (Java component interface or Web Service port type). Execution service: allow users to control the execution of components distributed environments Easy Maintenance: New components can be added through a clean API Regression Promoter Prediction D-NET API Gene function perdition Homology Search

11 Discovery Deployment Discovery Deployment = On-demand rapid application construction and publishing Towards a Dynamic Deployment of Knowledge Discovery Procedures: Deployment Engine : allows users to build and publish applications based on DPML code coordinating remotely execute components, as Web Page, Web/Grid Service, command line tool. Easy Maintenance: New discovery procedures described in DPML, a Standardised representation of composed discovery procedures Discovery Component Discovery Service Discovery Process in DPML Report Batch processing Storage & Reporting Servers: allow users to share DPML procedures and to generate workflow audit reports.

12 Knowledge Integration & Interpretation Dynamic Knowledge Interpretation = cross-reference and verify analysis results against background knowledge Towards a Knowledge Integration Framework: Multi-subject data analysis Specialised Client Interfaces: Interactive Analysis and dynamic component interaction Text Mining Sequence Genetic Analysis Pathway Result Annotation, Structuring and Storage: Information source query, result browsing, sharing and markup Analysis Analysis Life science example application

13 Workflow execution Component execution location resolution User list of known resources A component can require explicitly to be executed on a particular resource A component can choose from a set of resources proposed (and could use Grid resource information systems and network weather information to determine where to go) For unconstrained components, simple near the data execution policy: If single input data location then execute there Otherwise fallback to original execution location Allows usual DPKD workflows to be designed Handles data management and transfer (serialisation, Java based, FTP based)

14 Discovery Net and Grid technologies Cluster/Campus Grid level: Partial or complete workflow execution on Condor / SGE Task farming on subset of the workflow Global Grid: GSI integration (Java Cog Kit) GSI-FTP transfer functionality (Java Cog Kit) OGSA Grid Service access to functionalities (GT3) Potential use of GRIS or NWS in component implementation Globus scheduler? Unicore? SRB?

15 Discovery Net Application Testbeds GUSTO UNITS with wireless connectivity Life Science Testbed: Gene sequencing, Protein Chips High Throughput real-time genome annotation testbed: analyse and interpret new sequences using existing distributed bioinformatics tools and databases Environmental Modelling Pollution Sensors (GUSTO): SO 2, Benzene,.. High Throughput real-time pollution monitoring testbed: analyse, interpret time-resolved correlations among remote stations, and with other environmental data sets Geo-hazard Prediction Multi-spectral, multi-temporal, Satellite imagery Real-time geo-hazard prediction testbed: analyse, interpret satellite images with other data sets to generate thematic knowledge

16 Case Study: SC2002 HPC Challenge High Throughput Sequencers Identify Organism Chromosomes Identify Organism s DNA D-Net based Global Collaborative Real- Time Genome Annotation Nucleotide-level Annotation Genes Gene markers Regulatory Segmental Regions Duplication Literature References trnas, rrnas Non-translated Repetitive RNAs Elements SNP Variations.. EMBL TIGR NCBI SNP genscan grail E-PCR blast Repeat Masker genscan Protein-level Annotation Identify Proteins Functional Characteisation Domain Fold Prediction Literature References Classify into Protein Families Homologues 3-D Structure Secondary structure.. Inter Pro SMART Inter SWISS Pro PROT blast PFAM predator 3D-PSSM Motif Search DSC Genome Annotation Process-level Annotation Relate Cell Cycle Drugs Cell death Literature References Metabolism Biological Process.. Embryogenesis.. Pathway Ontologies Maps GO CSNDB AmiGO GeneMaps virtual KEGG GK GenNav chip 15 DBs 21 Applications

17 500 3 Nucleotide Annotation Workflows How It Works Interactive Editor & Visualisation Download sequence from Reference Server Inter Pro EMBL SMART NCBI KEGG SWISS PROT Save to Distributed Annotation Server TIGR SNP GO 1800 clicks Web access 200 copy/paste weeks work in 1 workflow and few second execution Execute distributed annotation workflow

18 Conclusion and Future works Towards an open integration platform that enables scientists to conduct their KD activities Several levels of integration required Enable use of available resources Evolution towards cost model integration (performance, value, QoS) Semantic based service retrieval and composition Other useful standards? (OGSA-DAI?)