Service-oriented architecture for integration of bioinformatic data and applications

Transcription

1 Service-oriented architecture for integration of bioinformatic data and applications Xiaorong Xiang Department of Computer Science and Engineering University of Notre Dame 4/6/07 Ph.D defense 1

2 Contributions Survey of research issues and challenges in service-oriented computing (Chapter 2) Built a SOA based system for supporting bioinformatics research (Chapter 3) Explored the deep phylogeny of the plastid with the system (Chapter 4) Enhanced the system with semantic web technology and a novel approach of reuse workflows (Chapters 5 & 6) 4/6/07 Ph.D defense 2

3 Outline Introduction to SOA MoG project and MoGServ Ontological data and service representation model Knowledge and workflow reuse 4/6/07 Ph.D defense 3

4 SOA an architectural style of distributed computing Discovery Service Broker Service Requester Publish interface Invoke 4 Service Provider Why SOA Reusability Interoperability Security Maintenance Save cost when integrating applications Adoption of SOA e-business e-science e-government 4/6/07 Ph.D defense 4

5 Web services one realization of SOA Additional WS* Standards Business Process Execution BPEL4WS, WFML, WSFL, BizTalk, Service Publishing & Discovery UDDI Services Description WSDL Services Communication SOAP Transactions Management Security Universal Description, Discovery and Integration Web Service Description Language Simple Object Access Protocol Meta Language XML Network Transport Protocols TCP/IP, HTTP, SMTP, FTP, etc 4/6/07 Ph.D defense 5

6 SOA research orientations Semantic Web Semantic Web Service Semantic Grid 2 Service-oriented Architecture (Web Service) 1 3 Grid Computing Semantic Grid Service Open Grid Service Architecture (OGSA) The P2P technology plays an important role of increasing the scalability and reliability in Service discovery and workflow execution process 4/6/07 Ph.D defense 6

7 Bioinformatics today From the article Genome Sequencing vs. Moore s Law: Cyber Challenges for the Next Decade by Folker Meyer in journal CTWatch Quarterly August, 2006 volume 2 number 3 Rapidly accumulating data: DNA sequences, contigs, expression data, ontologies, annotations, etc. Non-standard independently developed heterogeneous data sources Data sharing, data integration, and security 4/6/07 Ph.D defense 7

8 SOA in Bioinformatics Large public database Middleware projects Recent exposure of data & analysis tools as services MORE Community efforts needed to provide more shared and reliable services More demonstration projects needed => best practices, measured utility, feedback to middleware projects, etc. Provide infrastructure To compose, manage, Execute, connect the Distributed services 4/6/07 Ph.D defense 8

9 Outline Introduction to SOA MoG project and MoGServ Ontological data and service representation model Knowledge and workflow reuse 4/6/07 Ph.D defense 9

10 Mother of Green (MoG) project Biological science In collaboration with Prof. Jeanne Romero-Severson, Biological Sciences, University of Notre Dame. Study the deep phylogeny of plastid Computer science Provide an environment to support scientists investigations A case study of using SOA for data and application integration A prototype for future research in service-oriented architecture domain 4/6/07 Ph.D defense 10

11 MoG project one motivation Malaria causes million deaths every year 3,000 children under age five die of malaria every day Plasmodium falciparum (P. falciparum) causes human malaria Targeted drug design through phylogenomics P. falciparum has three genomes: nuclear, mitochondrial, plastid (apicoplast) Find the ancestors of the apicoplast, better understanding of the evolution of plastid Identify genes in the ancestors Determine gene function Apicoplast in P. falciparum P. falciparum 4/6/07 Ph.D defense 11

12 A typical in-silico investigation Data driven research workflow A: Query complete genome sequences given a taxon B: Query protein coding genes for each genome sequence E: Phylogenetic analysis D: Sequences alignment C: Eliminate vector sequences 4/6/07 Ph.D defense 12

13 Challenges (Time consuming manual webbased operations) Data collection and information gathering Rapid accumulation of raw sequence information Rate of accumulation is increasing Information accumulates faster than analyses finish Information in forms not readily accessible Analysis tool usage Experimental data recording Repetitive experiments for scientific discovery 4/6/07 Ph.D defense 13

14 MoGServ System Architecture Web Interface Data Access Services Data Analysis Services Application Server Local Data Storage Job Manager Job Launcher NCBI DDBJ EMBL Applications Service/Workflow Registry Metadata Search Workflow/Soap Engines Services Others Services Access Client MoGServ Middle Layer Data/Services Providers

15 Data storage and access services Local database Integrating data from multiple data sources with scientists interests Supporting repetitive investigations against several subsets of sequences Avoiding network traffic and service failure when retrieving data on-the-fly from public data sources Accessing the data in the local database by services 4/6/07 Ph.D defense 15

16 Service and workflow registry A table-based description with necessary properties Text description Service location Input/output Provider Version Algorithm Invocation method Not intended for supporting service discovery or composition at current stage A repository of service and workflow used for local application developers 4/6/07 Ph.D defense 16

17 Indexing and querying metadata Metadata Service and workflow description Description of sequence data in order to track the origination of data Experimental data output, input, and intermediate data Indexing and querying with keyword Lucene Implemented as services 4/6/07 Ph.D defense 17

18 Service and workflow enactment Service/Workflow Registry INPUT Find the service/workflow definition using the task name Parameters Job Launcher Task Name Job Manager Form a Job Description Job Information Timer Output Job ID Instances of Workflow/Service Engines 4/6/07 Ph.D defense 18

19 Implementation Development and deployment J2EE, JSP, XSLT Tomcat / Axis 1_2RC2 Database PostgresSQL 8.1 Index and search of metadata Apache Lucene library Service implementation Java2WSDL Wrap command line applications with JLaunch library Workflow Taverna workbench, part of mygrid project Freefluo workflow engine 4/6/07 Ph.D defense 19

20 4/6/07 Ph.D defense 20

21 Taverna workbench 4/6/07 Ph.D defense 21

22 A more complex workflow 4/6/07 Ph.D defense 22

23 Issues with the first prototype Meta data description Solution Index-based (keyword syntactic search) Capture most properties to support the end-users requirement Support data provenance Limitation Similar to most services in the bioinformatics community Lack of semantic description (goal => semantic search) Failure tolerance and recovery Solution Statically encode alternative services in the workflow to prevent service failure Record status of the service and workflow execution into the database for possible recovery strategy Multiple workflow engines deployment to prevent the hardware or network failure Limitation No dynamic service selection (more semantic description support) during execution time No fine grained resource management and monitoring Security 4/6/07 Ph.D defense 23

24 Outline Introduction to SOA MoG project and MoGServ Ontological data and service representation model Knowledge and workflow reuse 4/6/07 Ph.D defense 24

25 Semantic web Semantic web vision Giving meaning (semantics) to web-based information Machine-understandable such that software agents can autonomously process them Two standards: OWL & RDF The Web Ontology Language (OWL) Defines common vocabularies for specifying the concepts and relationship among concepts Resource Description Framework (RDF) Formal format for encoding web content using defined vocabularies Semantic web for Bioinformatics UniProt RDF project Semantic web for SOA Automated service discovery, composition 4/6/07 Ph.D defense 25

26 Resource Description Framework (RDF) MoG is a project #hastextdescription #foundation #bioinformatics #hasfundedby #hasresearchtopic A graph model of statements, a set of triples: Predicate (Subject, Object) Representations: #hascreator RDF/XML #gmadey N-triples Turtle #hasfullname Gregory Madey #haspersonalsite #hastitle #professor A standard format to connect web information Literal Resource # URI provided the definition of these vocabularies 4/6/07 Ph.D defense 26

27 Ontological modules used for semantic description of data, services & workflows MoGServ application Domain Ontology (MoGServ) Service Domain Ontology (mygrid) Generic Service Description Ontology (mygrid/feta model) Software components for annotation RDF Store Data Services Workflows 4/6/07 Ph.D defense 27

28 MoGServ Application Domain Ontology To better track the data origination To support the automation of workflow creation To better share the data on the web in the future Example concepts and properties defined in MoGServ properties invokedby isparentof isinstanceof hassetname domain Job Set Job Set range User Set Service XML:String Ontological modules Number of Concepts Number of properties Object Datatype MoGServ mygrid mygrid/feta model /6/07 Ph.D defense 28

29 Sample data annotation metadata from MoG local database Displayed by Rdf-Gravity 4/6/07 Ph.D defense 29

30 Sample service/workflow annotation Question: Which service has an operation that accepts nucleotide_sequence as a parameter Answer: Uri: /alignment:blastn_ncbi OperationName: Run Displayed by Rdf-Gravity 4/6/07 Ph.D defense 30

31 Implementation of annotation and query components for data, services & workflows SeRQL Query templates Sesame library Query Components Supports files, RDBMS, SeRQL Sesame RDF store result Annotation components Select Y, W, X from {Y} mg:hasoperation{w} mg:inputparameter {X} rdf:type {mog:set} using namespace rdf = < mg = < mog = < Annotation Templates (Service) Annotation Templates (Data) Service: axis/services/clustalw?wsdl Operation: runclustalwdf inputparameter: setid 4/6/07 Ph.D defense 31

32 Limitations The MoGServ ontology is not complete Contains a small portion of necessary concepts used for tracking the data provenance Service domain ontology is not complete Needs more concepts as more services are published Challenges of using semantic web in general Ontology creation, never complete Data and service annotation accuracy, efficiency Ontology integration 4/6/07 Ph.D defense 32

33 Outline Introduction to SOA MoG project and MoGServ Ontological data and service representation model Knowledge and workflow reuse 4/6/07 Ph.D defense 33

34 Three user-defined workflows from different views Question: are gene genealogies for ATP subunitαβ γ different? Retrieving Aligning Workflow A defined by a less experienced user using the functional definition of services querygene querygene clustalw querygene setids setfilter clustalw querygene clustalw clustalw Workflow B defined by an intermediate user with executable services Workflow C defined by an expert user with two extra executable services to ensure the accurate output of the biological process 4/6/07 Ph.D defense 34

35 Limitations of current workflow management systems Existing workflow management system and bioinformatics middleware Taverna, Kepler, Triana, Pegasus Design, execute, monitor, re-run Support ad-hoc, semi-automated and automated service discovery and composition from scratch Our approach: reuse the verified knowledge and workflow Increase the correctness over time Provide more accurate guidelines 4/6/07 Ph.D defense 35

36 Workflow execution engine concrete workflow Workflow composer (software agent/experienced users) Knowledge discovery Knowledge base management Collect and manage information about data origination Data provenance management Semantics enabled service discovery Service matchmaking Find appropriate service Semantics enabled service registry Abstract workflow User Create abstract workflow using ontology DL reasoner Ontology Annotate services using ontology Service Annotator Enhanced workflow system 4/6/07 Ph.D defense 36

37 Task A Task B Abstract workflow Encode, convert the High level definition To low-level executable F F T f i l e a Abstract Workflow Service A Service B M o v e f i l e a f r o m h o s t 1 : / / h o m e / f i l e a Concrete workflow Service C Service D Replace individual Services with their optimal alternatives /usr/local/bin/fft /home/file1 Concrete Workflow t o h o s t 2 : / /h o m e / f i l e 1 DataTransfer Service A Service B Service D Service C Optimal workflow Invoke a workflow with Specific input data and Record the data Provenance and Performance of services, workflows. Data Registration Pegasus workflow structure Service A input Service B Service D Service C Workflow instance output Our hierarchical workflow structure 4/6/07 Ph.D defense 37

38 Reusable knowledge Connectivity Helps to convert from abstract workflow to concrete workflow Alternatives and quality-of-service profiles Helps to convert from concrete workflow to optimal workflow Mapping of abstract workflow and concrete workflow Helps to choose reusable workflows 4/6/07 Ph.D defense 38

39 Connectivity identification (Match detection) Service: QueryLocal Operation: createset performtask: mygrid:retrieving inputpara: Settype(String, mog:gene) Queryterm(String, null) outputpara: Setid(string, mog:geneset) useresource: MoG Service: ClustalW Operation: runclustalwdf performtask: mygrid:aligning inputpara: Setid(String, mog:set ) Sequencetype(String, mog:sequence) outputpara: filen(string, mygrid:sequence _alignment_report) useresource: EBI Service: FormatConversion Operation: convert performtask: mygrid: translating inputpara: filen(string, mygrid:sequence _alignment_report ) outputpara: Out(String, mygrid:nexus _paup_format) useresource: MoG Parameter (data type, semantic type) Matching rule: opertation ij operation mn if exist parameter k is output parameter of operation ij and exist parameter o is input parameter of operation mn and data type (parameter o ) = data type (parameter k ) and semantic type (parameter o ) = semantic type(parameter k ) 4/6/07 Ph.D defense 39

40 Need for verified service connectivity The mismatching problem Accurate annotation Match detection output No Inaccurate annotation Lack of semantic annotation Inaccurate reasoning DDBJ-XML Out: sequence data record Self-defined format FP X Yes NCBI blast In: sequence data record fasta format Yes TP FN Real match No FP TN GenBankService Out:GenBank record Inaccurate annotation Lack semantic annotation Inaccurate reasoning May be detected by expertise at design time or after run Accurate annotation Can be detected automatically TN X Mediator, adaptor, shim Blastp In: protein sequence 4/6/07 Ph.D defense 40

41 Connectivity Graph Implementation Registration process registry Automatically Identify the connectivity Workflow Translation / Service composition process Store the connectivity Refine, update, decompose the workflow Knowledge base connect (service a, operation ai, parameter c, service b, operation bi, parameter d ) identifyconnect (Single service, rdf repository) Search at syntactic level: search path between two nodes search next available service automatic composition base on input, output Implementation: shortest path algorithm Dijkstra 4/6/07 Ph.D defense 41

42 Experiment Used 418 concepts from domain ontology for semantic type, defined 10 concepts for data type. Randomly generate service annotation. 1 input, 1 output 1000 services connectivity graph (right side) Intel Pentium mobile 1.5GZ Number of services Number of Matched pair Load RDF repository (milliseconds) Number of nodes Number of arcs Average path search time (milliseconds) Connectivity graph load time (milliseconds) Average time of match detection per single service (milliseconds) Less than Length 0 = 724, length 1= 587, length 2=448, length 3= 281, Length 4=114, length 5=71 Length 6 =28, length 7=16 Length 8 = 4, length 9 = 2 Conclusion: Feasible solution. 4/6/07 Ph.D defense 42

43 Reuse of workflows Reuse of abstract workflows Reuse of concrete workflows Compare structural similarity of two workflows Implementation: SUBDUE algorithm query_term Graph view hasparameter input hasinput task performtask hasnext retrieving task hasoutput performtask output aligning hasparameter SUBDUE input format v 1 input v 2 output v 3 task v 4 task v 5 query_term v 6 retrieving v 7 aligning v 8 multiple_aligning_report e 3 4 hasnext e 3 1 hasinput e 4 2 hasoutput e 3 6 performtask e 4 7 performtask e 1 5 hasparameter e 2 8 hasparameter multiple_alignment_report 4/6/07 Ph.D defense 43

44 Pro and Con Pro Increase the correctness of the formed workflow over time Avoid the incorrect, inaccurate semantic annotations Take advantage of verified knowledge Avoid the ontological reasoning process Better support for semi-automated and automated service composition over time Provide more accurate guideline to users over time Con The connectivity graph can be big Number of parameters Number of services Search the connectivity of a service when a service is registered in the system may take relative long time More complex matching rule Number of parameters May not have high accuracy at the beginning 4/6/07 Ph.D defense 44

45 Summary Described the design and implementation of MoGServ Explored the ontological representation of data and services Described new approach for reuse of workflows and connectivity of services 4/6/07 Ph.D defense 45

46 Future work Integrate the GridSam into the MoGServ for execution, monitoring Integrate the Grid computing technology for resource allocation Refine the MoGServ application domain ontology Create interface for end-user workflow creation Create interface for individual workspace Evaluate the scalability, accuracy of connectivity graph approach and the graph matching approach with large number real workflows and services 4/6/07 Ph.D defense 46

47 Acknowledgements Dr. Madey Dr. Romero-Severson Dr. Flynn Dr. Striegel Dr. Chaudhary Dr. Collins Mr. Eric Morgan Dr. Jean-Christophe Ducom Partially supported by the Indiana Center for Insect Genomics (ICIG) with funding from the Indiana 21 st Century fund 4/6/07 Ph.D defense 47

48 Publications X. Xiang, G. Madey and J. Romero-Severson, A Service-oriented Data Integration and Analysis Environment for In-Silico Experiments and Bioinformatics Research, Proceedings of the 40th Annual Hawaii International Conference on System Sciences (CD-ROM), January , Computer Society Press. Xiaorong Xiang and Greg Madey, "A Semantic Web Services Enabled Web Portal Architecture", IEEE International Conference on Web Services (ICWS 2004), San Diego, July 2004 Xiaorong Xiang and Greg Madey, Improving the reuse of scientific workflows and their by-products. In International Conference on Web Services (ICWS2007). Under review. Xiaorong Xiang and Eric Lease Morgan, Exploiting "Light-weight" Protocols and Open Source Tools to Implement Digital Library Collections and Services. D-Lib Magazine, October 2005, Volume 11 Number 10 4/6/07 Ph.D defense 48

49 Publications planned One journal paper for BMC Bioinformatics Chapter 3, chapter 4, chapter 5 Future IEEE ICWS proceedings Chapter 6 Biology journal TBD Results from using MoGServ 4/6/07 Ph.D defense 49

50 Thank you 4/6/07 Ph.D defense 50