Enterprise Content Integration Services

Transcription

1 Enterprise Content Integration Services Jacques Conan ECI Product Manager Pierre-Yves Pitch Chevalier ECI Software Development Manager EMC Software Group 2004 EMC Corporation. All rights reserved. 1 During this session you will discover the ECI Services that Documentum has released as part of the Virtual Repository initiative. We will demonstrate in details the end-user features available with this new content integration framework as well as how to configure adapters on enterprise contents. You will also discover how to create an adapter on external web information sources. The target audience ranges from IT administrators to developers. 1

2 Information Overload Information Overload Workers spend up to 50% their time searching for actionable information. IDC Difficulty to locate high value information 80% of the information accumulated by companies is never used. KM World 2004 EMC Corporation. All rights reserved. 2 2

3 Enterprise Content Integration Solution Documentum Enterprise Content Integration Services Provides access to the global content in just one query Enhances existing solutions in place without imposing change Adapts to changes of the information space Coexists with any kind of IT infrastructure Find, assimilate, synthesize, share and integrate information 2004 EMC Corporation. All rights reserved. 3 3

4 Documentum ECI status ECI Services Version 4 First release of EMC Documentum Enterprise Content Integration solution, Released on July 24th 2004 for EF languages ECI Services Version 4 SP1 Localization for IGSJK, new version of linguistic engine Planned for Nov 2004 ECI 5.3 DCTM 5.3 to embed ECI capabilities ECI Services Version EMC Corporation. All rights reserved. 4 4

5 Distributed Search Clustering Vehicle Adapter MotorcycleAutomobile Truck SUV Sports Car Minivan Sedan Convertible HardtopHatchback Adapter Adapter Adapter Adapter Adapter Adapter Adapter Public Web Oracle Notes OpenText Other Contents 2004 EMC Corporation. All rights reserved. 5 5

6 The Indexing Barrier Static Web Index Analysis 20-30% of total document size Cannot crawl constantly Index becomes out-of-date Dynamic Web (forms, login) X Cannot build Cannot go through permanent URLs Login forms (session expires) Unlikely to preserve structured information Proprietary Applications Duplicate internal indexes Databases, Content repositories 2004 EMC Corporation. All rights reserved. 6 Peter Serrao below details limitations faced by most full-text search engines that are based on indexing technologies. askonce is an enterprise content integration toolkit that allows you to search across all internet, intranet, and deep web sources (both structured and unstructured.) It does not crawl and re-index any data from any source but relies upon the native indexes of each source. In as much it has the ability to search across both metadata and full-text indexes. One argument is that any such search is only as good as the indexing capability of the source. That is true, however it depends upon the nature of the source. I do not doubt that google and verity do a better job of full-text indexing as opposed to tools like SiteServer or SPS. But none of the tools can do better jobs of indexing complex data sources such as Documentum, Livelink, Docushare, Lotus Notes, or RDBMS systems. Here the ability of the native system to provide attribute based indexes, can result in more relevant results. Many of these systems also incorporate full-text indexes. Besides, looking at infrastructure issues, why do you need to maintain large indexes for these sources when they already maintain indexes themselves. One customer I spoke too had 100 terabytes of Documentum and Livelink data. If you used google or verity to re-index these, I have no doubt that the resultant index files would be in the terabyte range. If you look at the server, disk and other infrastructure costs to support a new index, on data that's already indexed by the native source, and given that the fact that the index won't support metadata, then does it really make sense to do that. Its a benefit to cost analysis. The ability of SOS or google to search across these external structured data sources is questionable. Gartner in a brief on the google 6

7 Live Query Static Web Existing index, categorization Index Analysis Dynamic Web (forms, login) Dynamic queries to up-to-date content Pass through login requests Build permanent URLs (across sessions) Preserve structure and meta-data Proprietary Applications Flexible Leverage internal indexes Databases, Content repositories 2004 EMC Corporation. All rights reserved. 7 7

8 Search Domains 2004 EMC Corporation. All rights reserved. 8 8

9 Single Sign-On List of user credentials for sources requiring authentication Managed by each user individually Submitted with each query automatically 2004 EMC Corporation. All rights reserved. 9 9

10 Sophisticated Query Language Name CONTAINS EQUALS < <= = >= > AFTER BEFORE Semantic attribute contains a sub-string that matches the value (default) attribute is equal to the string attribute is less than the numerical value attribute is less than or equal to the numerical value attribute is equal to the numerical value attribute is greater than or equal to the numerical value attribute is superior to the numerical value< date is after day date is before day Usage title:contains:chaco canyon music title:chaco canyon music author:equals:balzac sum:<:29.29 sum:<=:29.29 sum:=:29.29 sum:>=:29.29 sum:>:29.29 date:after: date:before: Examples General, Library author:equals:flaubert or stendhal / title:contains:(madame AND bovary) OR (chartreuse AND parme) body:contains:flaubert or stendhal / title:contains:(madame AND bovary) OR (chartreuse AND parme) General title:contains:"find good paying jobs" / date:after: Shopping title:contains:tolkien or (Lord and rings) / price:<=: EMC Corporation. All rights reserved

11 Search Status 2004 EMC Corporation. All rights reserved

12 Query Statistics 2004 EMC Corporation. All rights reserved

13 Quick Skimming of Results Floating window shows a short abstract by just moving the mouse over a particular result 2004 EMC Corporation. All rights reserved

14 Scheduled Queries 2004 EMC Corporation. All rights reserved

15 The Problem with Metadata 2004 EMC Corporation. All rights reserved

16 Metadata Extraction and Normalization 2004 EMC Corporation. All rights reserved

17 Document Snapshots Content Extract Determine quickly content relevance No need to open actual document Metadata Any metadata found Organized for easy comprehension 2004 EMC Corporation. All rights reserved

18 Dynamic Linguistic Contextual Clustering Clustering Designed for state of the art type searches Allows to quickly view a large set of results by topics of interest Dynamic Organizes and groups search results automatically Complements existing pre-defined taxonomies Linguistic Extraction of terms (multi word expression), stemming Contextual Selection of most interesting groups based on search context Helps to refine search criteria 2004 EMC Corporation. All rights reserved

19 Personal Relevancy Ranking 2004 EMC Corporation. All rights reserved

20 Cross-Lingual Content Retrieval Multi-Lingual Search On the fly query translation Content comprehension aid Build-in dictionary for words and expressions 2004 EMC Corporation. All rights reserved

21 Content Export 2004 EMC Corporation. All rights reserved

22 DEMO 2004 EMC Corporation. All rights reserved

23 Adapter Technology Interface to any application Creates structure out of un-structured information Meta-data extraction and attributes based filtering Intelligent components enhancing source querying capabilities Unique framework for rapid adapter production and maintenance 2004 EMC Corporation. All rights reserved

24 Adapter Library Content Providers Factiva Lexis Nexis World Wide Web (General) AltaVista Google Open Directory Yahoo! Apple Sherlock plug-ins Full-text search engines Verity K2, PortalOne, Search97 Verity Ultraseek Google Search Appliance Enterprise Repositories Documentum ECM 4.x, 5.x Documentum eroom 6, 7 Documentum AX 4.6, 5.0 DocuShare 1.5, 2.0, 2.1, 3.0 Lotus Notes R4.6 Lotus Domino R5, R6 Microsoft SiteServer 3.0 Microsoft Exchange Oracle 7.2, Oracle 8i JDBC/ODBC Z39.50 (optional) Bundles Pharma Bundle (35 adapters) Sciences Bundle (15 adapters) 2004 EMC Corporation. All rights reserved

25 Anatomy of an Adapter User Query Query mapping Query submission Filter Function Service Description Information Source Responses Post Filtering Meta-Data Extraction 2004 EMC Corporation. All rights reserved. 25 The role of an adapter is to send the user query to various information sources, gather and filter the results, and present them to the end user. Adapters hide the particularities of each individual source by fully managing the interaction between a request and the sources. Service Description Information stored by the adapter configuration determines how to contact the source (address of the server, type of information source) and what is expected in terms of the query format (attributes supported to formulate queries, query language supported, date format, name of attributes etc.). This service description is used during the various phases of the adapter s workflow. Query Mapping The objective of the query-mapping phase is to transform the user query into a query that is understood by the content source. The query language is specific for each source and depends on the source or service description. Let s suppose a user formulates a query like Author CONTAINS Ellroy AND Title CONTAINS road OR (street NEAR light). The underlying source may not support attributes; also, it may not support the proximity operator NEAR, but the AND operator instead. In this case, the user query will be transformed into Ellroy AND Road OR Street AND light. Let s take another example of a source that supports attribute queries but names the attribute author as creator and names the title as reference and doesn t support the proximity operator NEAR but supports the AND operator instead. In this case, the query will be transformed into Creator CONTAINS Ellroy AND Reference CONTAINS road OR (street AND light). Also if a source doesn t support the OR operator, this query will be split into two queries and results lists merged in return. Filter Function 25

26 Query Mapping S E A R C H full-text:contains:betaferon full-text:contains:betaferon OR OR Interferon Interferon / / title:contains:approved title:contains:approved A D A P T E R DCTM: -Fielded search -Boolean operators Post-filter: 4 results valid out of 4 received PubMed: -Full-text search -Boolean operators Post-filter: 2 results valid out of 50 received FDA: -Full-text search -Internet-style -No Boolean Post-filter: 8 results valid out of 100 received N A T I V E select r_object_id from select r_object_id from dm_document search topic dm_document search topic ( Betaferon* or ( Betaferon* or Interferon* ) where Interferon* ) where folder( Drugs', descend) and folder( Drugs', descend) and (lower(title) like (lower(title) like '%approved%' or '%approved%' or lower(object_name) like lower(object_name) like '%approved%') '%approved%') (Betaferon OR Interferon) AND (Betaferon OR Interferon) AND approved approved +Betaferon +approved +Betaferon +approved +Interferon +approved +Interferon +approved S O U R C E Products content repository PubMed web site FDA CDER web site 2004 EMC Corporation. All rights reserved

27 Metadata retrieval for HTML Pages UNSTRUCTURED CONTENT STRUCTURED CONTENT 2004 EMC Corporation. All rights reserved. 27 The adapter analyzes the unstructured HTML results pages on the fly. Metadata is extracted for each result and is fed into the ECI structured result set. Additional metadata is obtained from other related HTML pages and appended to the result in the structure. The adapter s metadata extraction handles variations on the pages: for example, if on some results the date is sometimes replaced by a link, it will be detected and handled by the adapter. 27

28 Metadata extraction: learning phase HTML Results page Sample1 HTML Results page Sample2 learning phase learning phase HTML Results page SampleN learning phase Extraction Agent Version 1 update Extraction Agent Version 2 update Extraction Agent Final Version EMC Corporation. All rights reserved

29 Metadata extraction: self-repairing Structure change Content change Context change When change is detected: find out trusted fragments; apply one or more recovery routines; repairing can be complete or partial; update grammar if recovery threshold is satisfied Conceptual shift assumption Context change: change in the page mark-up, like putting title in bold font Content change: CIKM Conf. Information Knowledge Management Structural change: label addition/removal, order permutation, etc EMC Corporation. All rights reserved

30 Adapter Builder Unique framework for rapid adapter production, testing and maintenance DEMO 2004 EMC Corporation. All rights reserved

31 Adapter Configuration: setup Admin Center: Web-based configuration Wizard for enterprise adapters: DCTM, Domino, eroom, 2004 EMC Corporation. All rights reserved

32 Adapter Configuration: diagnostic 2004 EMC Corporation. All rights reserved

33 Adapter Exchange Site Adapters.html Repository to publish and to exchange sample adapters (source-code or binary delivery) InternetArchive InvisibleWeb Java Developer Connection SourceForge Dedicated forum to discuss tips & issues related to adapter development Go & contribute! 2004 EMC Corporation. All rights reserved

34 ECI 5.3 Architecture Webtop Portal ECI Client 5.3 ECI Portal 5.3 WDK 5.3 DFC 5.3 Access to external contents ECI Services 5.3 WWW Docbases Lotus Domino Lexis Nexis Google 2004 EMC Corporation. All rights reserved

35 WDK Search Components Single box search Multi-docbases Results and Status Wait screen Results Status Status refreshed Status stopped Enter credentials from status No results Advanced and Revise Advanced from results (revise a search) Advanced from browsing Advanced cleared Changing Sources Change sources Check box selected Authenticate repository Repository added to list Navigate into repository and breadcrumb Navigate into a cabinet and breadcrumb Saving a Search My saved searches Properties of a saved search All saved searches Preferences Search locations - favorite repositories Search locations - selected specifically Favorite repositories Favorite repositories (admin) 2004 EMC Corporation. All rights reserved

36 DCTM 5.3 Extended Search Query Building in DFC import com.documentum.fc.client.search.*; // Initialization IDfSearchService searchservice = client.newsearchservice(sessionmgr); IDfMetadataMgr metadatamgr = searchservice.newmetadatamgr(); IDfQueryMgr querymgr = searchservice.newquerymgr(metadatamgr); // Creation of the query IDfQueryBuilder querybuilder = querymgr.newquerybuilder(); querybuilder.addselectedsource( dm_notes ); querybuilder.addselectedsource( AmazonBooks ); // Definition of the constraints IDfExpressionSet exprset = querybuilder.getrootexpressionset(); exprset.addfulltextexpression( singleton Java ); exprset.addsimpleattrexpression( title", IDfValue.DF_STRING, IDfSimpleAttrExpression.SEARCH_OP_CONTAINS, false, false, design patterns ); 2004 EMC Corporation. All rights reserved

37 DCTM 5.3 Extended Search Query Execution in DFC // Synchronous execution IDfQueryProcessor syncprocessor = searchservice.newqueryprocessor(querybuilder); IDfResultsSet results = syncprocessor.blockingsearch(-1); // -1 == no timeout manipulator = service.newresultsmanipulator(metadatamgr); IDfResultsSet sortedresults = manipulator.sortby(results, "date", true); 2004 EMC Corporation. All rights reserved

38 Customers Using Documentum ECI Services European Patent Office 2004 EMC Corporation. All rights reserved. 38 Conclusion slide 38

39 Conclusion Get ECIS and ADK! Adapter Exchange html 2004 EMC Corporation. All rights reserved

40 Uniting The World Through Content 2004 EMC Corporation. All rights reserved