Migrating from FAST to EMC Documentum xplore: What To Do and Why You'll Love It. Ed Bueché EMC Distinguished Engineer and xplore Architect

Transcription

1

2 Migrating from FAST to EMC Documentum xplore: What To Do and Why You'll Love It Ed Bueché EMC Distinguished Engineer and xplore Architect

3 Agenda Introduction to xplore xplore 1.2 new capabilities FAST-to-xPlore Migration bestpractices

4 Documentum xplore at-a-glance Documentum xplore is the next generation search for Documentum Replaces FAST: By end-of-year 2011 support will end for the FAST version of Search within Documentum Technology Foundation: EMC xdb (native XML database) and Lucene D6.5 SP2 (and later) compatible Indexing system D6.5 SP2/3, D6.6, D6.7 Client's supported by D6.5 SP2 will work without change (with small corner-case exceptions) Dual mode migration: xplore & FAST both active (index & query) on the same repository

5 Migrate to xplore Support for search within Documentum based on FAST ends Dec 31, 2011 Feature is replaced by Documentum xplore xplore 1.0 since Oct 2010 xplore 1.1 since April 2011 xplore 1.2 ~ Nov 2011 No additional license cost for xplore! Hundreds of deployments have already converted and hundreds more are in progress

6 Other Sessions on xplore at Momentum 2011 Optimizing EMC Documentum: Performance and Scalability Wednesday, 2nd November 2011, 2:00pm 2:45pm Covers xplore scalability and tuning best practices Optimizing EMC Documentum: Best Practices for Deployment Wednesday, 2nd November 2011, 3:00pm 3:45pm Covers additional Documentum scalability improvements as well as HA / DR best practices for xplore

7 What is new in xplore 1.2 (Nov 2011) Thesaurus support o o Synonyms, alternate spellings, acronyms Based on SKOS structure Customizable Natural Language Processing Pipeline o UIMA support and custom text extraction Query-based subscriptions o Ability to subscribe to a query executed on an interval and be notified of the results Indexing and query performance o o o Multiple CPS-per-xPlore instance Wildcard performance improvements Automatic query warmup New Languages supported o Administration o o Russian, Arabic, Hebrew, and Brazilian Portuguese Improved deployment and automation (CLI) Silent installer (local & remote)

8 xplore Features At-a-Glance

9 xplore 1.2: Thesaurus Improve the findability (or recall) of the content Allows for customer defined business thesaurus Thesaurus support allows you to query for one name and get hits in documents that have the related names Example: in the Pharma industry a drug is known by: Scientific name Internal code name Marketing name

10 xplore 1.2 Thesaurus Feature Notes Simple Knowledge Organizational System (SKOS) Simple Knowledge Organizational System (SKOS) Standard representation for Thesaurus and Categorizations, etc XML format (RDF) Able to Represent synonyms, concepts Case and Space insensitive format Ability to store multiple Thesauri per Docbase Ability to set default thesaurus Can override thesaurus per query Can specify multiple thesauri per query (clause specific) Support in DFC Search Service & DQL from D6.5 SP2 and later (in latest patches)

11 Thesaurus: cross-language term normalization and other use-cases Commission SKOS formatted thesaurus allows for cross language terminology mapping Use-case: Ability to search for content in one language and get hits in others Not a full translation mechanism but useful for domain specific cross language terms Only one language is lemmatized, so most useful for names Also possible to create other relationships aside from synonyms Επιτροπή комисия

12 Example: searching for acetaminofén (in Spanish) With no thesaurus only spanish documents are found With Thesaurus alternate synonyms in multiple languages are found

13 xplore Thesaurus Administration Easy import mechanism Can define multiple thesaurus per docbase

14 xplore 1.2 : Customizable Natural Language Processing Pipeline xplore 1.2 opens the Natural Language Processing (NLP) pipeline to customization Allows customers to go beyond the base linguistic analysis Able to inject standard UIMA compliant customizations Use cases include Entity extraction Classification ID normalization Custom Text Extraction

15 Functional view of NLP in xplore In coming doc event Content Fetch Text extraction Lang Identification Linguistic analysis CF TE LI LA Store in index

16 Custom Text Extraction for xplore NLP in 1.2 In coming doc CF LI LA Store in TE index Text extraction customization Post-Linguistic analysis UIMA extensions Text Extraction customization based on Mime-type Plugin-customization code can be defined for: o Pre-Text extraction o Text Extraction phase o Post-Text extraction Plugins can be Java or C/C++

17 UIMA customizations supported post-linguistic Analysisfor xplore NLP in 1.2 In coming doc CF TE LI LA Store in index Unstructured Information Management Architecture (UIMA) Apache Standard Architecture for Natural Language Processing customizations xplore 1.2 allows for UIMA components to process and annotate document elements Enables annotation of DFTXML without adding relational columns in the RDBMS Post-Linguistic analysis UIMA extensions

18 Potential UIMA customization use-cases ID normalization Official Company ID s: ABC-1234-D567-EFG Users want to query on: D567EFG, because ABC-1234 is not selective Pipeline step can create the alternative ID formulation Classification Examine text of document and automatically tag it with additional metadata based on a taxonomy CIS-based classification documented as an xcellerator Entity Extraction Extract entities with 3 rd party entity extractor

19 Advanced Customizations: Remote UIMA & Custom Thesaurus In coming doc CF TE LI LA Store in index Remote UIMA Query Processor Custom Thesaurus access Get additional terms 3 rd Party external component Query with additional terms

20 Query-based Subscriptions (QBS) User to subscribe to the periodic execution of stored query Notified of any new results since the last execution Queries execute on defined intervals Hour, day, week, or month Result notification or the initiation of an xcp-defined business process

21 Query-by-subscription Overview Subscription? Query executed automatically Query executed? Stored queries? User subscribes to query Results fed to business process results Query defined and stored User notified xcp Business Process

22 Subscribing to a saved query

23 Users, Subscriptions, Queries, Results Some Relationships User A User B Subscription to query #1 from User A Subscription to query #2 from User A Results: user A query #2 Subscription to query #2 from User B?? Results: user B query #2 Query #1 Query #2

24 QBS user activity report Provides information on each user s subscription activity

25 QBS activity report by subscription ID

26 Query-based Subscriptions: Delivery Notes Supported only with D6.7 SP1 and later DFC and Content Servers TaskSpace components delivered as xcellerator To be posted when D6.7 ships in Nov API available for custom UI s, this includes Stored query definition (dm_smartlist) Subscription definition and management

27 Additional xplore 1.2 Enhancements Multiple CPS processes on single xplore Instance Significantly simplifies content processing scaling Improved wildcard query performance New Language Certifications Russian, Arabic, Hebrew, and Brazilian Portuguese

28 Agenda Introduction to xplore xplore 1.2 new capabilities FAST-to-xPlore Migration bestpractices

29 FAST-to-xPlore Migration Best-practices at-a-glance Stay current with software RTO: Backup / Restore Plan and Test scale with larger environments Convert Legacy DQL Apps to DFC Search Service SAN s Provide best performance

30 Stay Current with Software xplore 1.1 shipped with DCTM 6.7 in April 2011 Why would you start your deployment with xplore 1.0? Patch Roll up releases available each month Available Sept 30: xplore 1.1 P03 and xplore 1.0 P13 Available Oct 30: xplore 1.1 P04 Some important items covered Snapshot-too-old consistency fixes Improved diagnostics and repair for index inconsistencies Fix for result inconsistency due to updates

31 RTO: Backup / Restore Recovery Time Objective The target time to restore the system back to service after some sort of failure Usually a target set by business users Example characterization of hardware failure in Google s data centers: In cluster (of 1800 machines), 1,000 will fail somehow in first year of service Thousands of hard drives will fail 50% chance that rack will overheat xplore migrations typically involve new hardware in new operating environments Human & Environment failures will be higher than normal Time to recovery varies Dual xplore systems provide fastest (but most expensive) RTO Sometimes (not always) data failure can be rectified with xplore repair tools Restore from backup is next fastest Re-feed from Documentum is the slowest To be discussed in more detail on Wed, Nov 2 in Optimizing EMC Documentum: Best Practices for Deployment at 3pm

32 Convert Legacy DQL Apps to DFC Search Service API Options for Documentum Search Applications IDfQuery and DQL Legacy compatibility DFC Search Service & automatically generated XQuery Foundation for Advanced Search since D6.6 IDfXQuery and custom defined Xquery Used primary for Zone Search of XML For most uses, the DFC Search Service is the best choice Best performance: Pulls the least amount of data per bounded result Native facets supported Not part of DQL avoids huge result set ingestion More efficient date range query processing

33 Plan and Test scale with larger environments xplore provides great out-of-box support for Documentum However, some aspects might require index tuning If index tuning not done, then re-feed or index re-build might be required Important to find these in larger test environments than production environments Items to watch for Special character tuning Date and integer values for custom DCTM object types Metadata wildcard optimization options Native Facet values Leverage Free xplore tools on EDN!

34 SAN s provide best performance xplore Supports SAN s, NAS, and local disk Services and functionality varies across disk hardware All-else-equal: SAN volumes offer best performance and are recommended However, some capabilities not available Basic xplore host sparing Simple inter-host data movement If leveraging NAS, please review latest guidelines on configuration

35 EMC Symmetrix: Nondisruptive Mobility Virtual LUN VP Mobility Flash 400 GB RAID 5 Fibre Channel 600 GB 15K RAID 1 SATA 2 TB RAID 6 Virtual Pools Fast, efficient mobility Maintains replication and quality of service during relocations Supports up to thousands of concurrent VP LUN migrations Recommendation: work with storage technicians to ensure backend storage has sufficient I/O

36 Questions? Comments? See xplore on EMC Developer Network:

37 THANK YOU This presentation is also available at password: spree