TRANSFORMATION GATEWAY

Size: px

Start display at page:

Download "TRANSFORMATION GATEWAY"

Sylvia Sharleen Scott
5 years ago
Views:

1 TRANSFORMATION GATEWAY

2 Optimizing EMC Documentum: Performance and Scalability Ed Bueché EMC Distinguished Engineer TRANSFORMATION GATEWAY

3 Agenda: Top xplore Performance Tips Tip #1: Leverage Sizing tools Tip #2: VMware can speed time-to-deploy Tip #3: SAN storage provides best-performance Tip #4: Install latest patches Tip #5: Pre-Index tuning should be evaluated Tip #6: Migration-specific tuning is available Tip #7: Convert Legacy Apps to DFC Search Service Tip #8: Leverage Dual Mode & pre-test production Tip #9: Batch Ingestion Optimization with Collections

4 Tip #1: Leverage Sizing tools Calculates resources for xplore based on workload input Is separate from Documentum sizing spreadsheet Can be found on Powerlink Used for more detailed sizing of xplore

5 Sizing for Experts: Tool Supports Estimating Day Forward vs. Backfile ingestion Sizing for backfile migration Setting a target migration time and obtaining hardware to achieve this Drawback: In many situations, the day-forward rate will be significantly smaller than the backfile Sizing for day-forward ingestion Sizing hardware for max number of documents per day, total size of system, plus the query rate Defines the length of time a migration will take

index-to-cache You estimate how much memory system will have (vs.

6 Other sizing tips for Experts: Items to take into account for very large systems Query scope percentage Used to account for savings when using xplore optimized collection routed queries Percentage of index-to-cache You estimate how much memory system will have (vs. index size) and Tool calculates Disk I/O Percentage of cold index data Used to account for I/O savings when separating older (cold) data from newer (hot) data

7 Collection directed queries consume fewer resources than default (which cover all collections) Typical use-cases require collection-directed indexing 1. User data or departmental data directed to specific collections 2. Newer data (with higher business value) directed to specific collections Option #1: DQL IN COLLECTION clause Example: select r_object_id from dm_document search document contains benchmark in collection( default ) Option #2: DQL FDS_QUERY_COLLECTION hint Configurable through dfcdqlhints.xml Example: select r_object_id from dm_document search document contains benchmark enable(fds_query_collection_default) Option #3: DFC Search Service: addpartitionscope() method call Option #4: IDfXQuery collection() function call within xquery definition

8 Collection Directed Queries Big impact on highly selective security filtering of unselective queries Scenario: User only has access to a small subset of documents These could be stored with all other documents or in a separate collection (e.g., col-1) In either case, a query across all data will lead to many results being discarded returned Security filter discarded Security filter Col-1 Col-1

9 Example of highly under privileged user Example of an extremely under privileged user (allowed to see only 52 hits out of 139,000)

10 Collection Directed Queries Big impact on security filtering of unselective queries If the query is scoped to that collection then the unselective query will run significantly faster Security filter Excluded from query Security filter

xplore needs Don t just accept standard corporate issue Use xplore Sizing tool Confirm that physical hardware

11 Tip #2: VMware can speed time-to-deploy Hardware acquisition proven to be significantly faster with Vmware Virtual resources can be increased and scale very quickly However, important to ensure that vm s are sized to xplore needs Don t just accept standard corporate issue Use xplore Sizing tool Confirm that physical hardware is not over subscribed OS tools can spot disk over-subscription VCenter ready-counter can spot CPU over subscription

12 Some Vmware statistics Ready metric Generated by Vcenter and represents the number of cycles (across all CPUs) in which VM was denied CPU Generated in milliseconds and real-time sample happens at best every 20 secs For interactive apps: As a percentage of offered capacity > 10% is considered worrisome Pages-in, Pages-out Can indicate over subscription of memory 12

13 Sample %Ready for a production VM with xplore deployment for an entire week 16% 14% 12% 10% 8% 6% 4% 2% 0% In this case Avg resp time doubled and max resp time grew by 5x official area that Indicates pain 13

14 Actual Ready samples during several hour period 2500 Ready samples (# of millisecs VM denied CPU in 20 sec intervals)

15 Some Subtleties with Interactive CPU denial The Ready metric represents denial upon demand Interactive workloads can be bursty If no demand, then Ready counter will be low Poor user response encourages less usage Like walking on a broken leg Causing less Ready samples Denial spike 20 sec interval 15

Tip #3: SAN storage provides best-performance All-Else-Being-Equal SAN storage offers best performance for xplore Storage Type SAN Local disk NAS Perf.

16 Tip #3: SAN storage provides best-performance All-Else-Being-Equal SAN storage offers best performance for xplore Storage Type SAN Local disk NAS Perf. Multi-node HA Options Best overall, least risk Low latency, but runs out of I/O capacity High network traffic, need to disable some optimizations Yes, hosts should crossmount volumes Yes, but hosts should cross mount volumes Yes Active / Passive Active / Active (dual system) Spare node Active / Passive Active / Active (dual system) Spare node Active / Passive Active / Active (dual system) Spare node

17 Tip #4: Upgrade to latest releases and/or patches Issue or Improvement Improvement use-case Recommended xplore Upgrade Slow ftintegrity RDBMS query with row-based processing off Ftintegrity slow to run 1.1 P05 or later or 1.2 Final merge improved optimization of lucene segments CPS recycle to not cause objects to be dropped CPS with address space limitations Wildcard query slow other queries possibly slow Improves CPS memory resource consumption Improves CPS stability 1.1 P05 or later or P05 or later or or later with 64 bit

Best of xplore 1.1 and 1.2 items that will not be backported to earlier releases Earliest xplore Release Improvement Use-case that would see improvement 1.1 64 bit CPS Large files 1.

18 Best of xplore 1.1 and 1.2 items that will not be backported to earlier releases Earliest xplore Release Improvement Use-case that would see improvement bit CPS Large files 1.1 Blacklist management Deletion of large number of docs 1.2 Improved query response with automatic warmup 1.2 Wildcard fragment search dictionary term limit restart Highly unselective wildcard queries

Tip #5: Pre-Index tuning should be evaluated xplore automatically indexes most Documentum

is necessary Items to look for: Type tuning Non-string types may need special configuration

defines certain attributes for facets Customer application may desire additional ones that

explicitly for wildcard attributes Facet compression tuning Should be disabled for unique

19 Tip #5: Pre-Index tuning should be evaluated xplore automatically indexes most Documentum data models But some scenarios require additional attention up-front else an online re-index is necessary Items to look for: Type tuning Non-string types may need special configuration Common case: date range queries on custom date attribute Facet tuning By default the system defines certain attributes for facets Customer application may desire additional ones that need to be reflected Wildcard tuning Metadata fields can at times benefit from tuning explicitly for wildcard attributes Facet compression tuning Should be disabled for unique facet attributes Example: disable acl_name facet compression when docbase has more than 300,000+ unique ACLS

20 Type Tuning scenario at-a-glance Apply change to indexserverconfig.xml and run online index rebuild <sub-path leading-wildcard="false" compress="false" boost-value="1.0" includedescendants="false" returning-contents="false" value-comparison="true" fulltext-search="true" enumerate-repeating-elements="false" type="datetime" path="dmftmetadata//customer_date" /> Index can be queried and be changed during rebuild

21 Tip #6: Migration-specific tuning is available Migration of existing (backfile) content can require special tuning Day-forward ingest loads can be significantly smaller than those needed by target backfile migration time Explore has several tools and techniques to aid this Multiple CPS processes Multiple collections Delay of final merge (with query penalty) Temporary multiple instances Ingestion Filters

22 Single Instance xplore Migration Tuning: multiple CPS and multiple collections Multi-threaded CPS (xplore 1.0+ ) 64 bit CPS (xplore 1.1+) CPS daemon CPS daemon CPS daemon Multiple CPS processes per xplore instance (1.2) DSS routing Index Agent xdb StatusDB CPS xdb / Lucene xdb statusdb 64 bit xplore instance (since 1.0) Multiple collections (xplore 1.0+ )

23 Delay of final merge (with query penalty) xplore reflects changes to underlying index through new (small) indexes Queries are made efficient through the index consolidation (merging) Once some index data has been sync d to disk it is clean The process of merging to the final index can involve much I/O if the index is large By default (in 1.0 GA and 1.1 GA) this merge happens every 4 hours if needed This could be too frequent during a migration

24 Sample I/O load on a large (100 s of GB) index due to too-frequent of a final merge I/O load vs. warmup response time default final merge activity with an index that is 100's of GB hrs between starts of high i/o I/O's per sec resp (secs) tps resp

25 Tuning the final merge time with finalmerginginterval Lengthing the finalmerginginterval during migration can lower total I/O Determines the time between final merge starts Set in xdb.properties (default = 4 hours) Example: xdb.lucene.finalmerginginterval = Merge will happen once every 24 hours Available since 1.0 GA However, it does this at expense of query Possibility of reaching per-process file descriptor limit on Linux Query response time grows with many extra indexes Should be turned back to lower value after migration

26 Indexing Filters xplore greatly expands ability to flexibly filter out certain documents By Cabinet, Folder, and Types This can greatly reduce the indexing of objects that don t need to be full text indexed However, this introduces a form of acceptable inconsistency It is acceptable because the application doesn t need to full text query for these items Needs to be done with care, applications should test that queries function properly after filter applied Example: folder-descend() queries may not work properly if folder objects are excluded

27 Tip #7: Convert Legacy Apps to DFC Search Service API Options for Documentum Search Applications IDfQuery and DQL Legacy compatibility DFC Search Service & automatically generated XQuery Foundation for Advanced Search since D6.6 IDfXQuery and custom defined Xquery Used primary for Zone Search of XML For most uses, the DFC Search Service is the best choice Best performance: Pulls the least amount of data per bounded result However, DQL performance can be improved by setting return_top_results_row_based = false in server.ini Native facets supported Not part of DQL avoids huge result set ingestion More efficient date range query processing

Tip #8: Leverage Dual Mode & Pre-Test Production Dual mode option Advantages Disadvantages Separate active content server instances (loading either FAST or xplore query plugin) Content Server

28 Tip #8: Leverage Dual Mode & Pre-Test Production Dual mode option Advantages Disadvantages Separate active content server instances (loading either FAST or xplore query plugin) Content Server instance loading either FAST or xplore but not both on same docbase at the same time Allows for simultaneous query to FAST or xplore Can migrate groups of users at a time Simpler to setup No need for extra app servers or content server instances The most complex to set up Have to test & tune xplore environment in off-hours Have to reboot content server instances when switching between servers

Pre-Production testing best practices Functional test on small scaled system prior to full production load Easier to correct issues Typically also a best practice to smaller docbases into production

29 Pre-Production testing best practices Functional test on small scaled system prior to full production load Easier to correct issues Typically also a best practice to smaller docbases into production before attempting larger ones See xplore tools on EDN: Query warmer available on EDN for 1.1 and 1.0 deployments default part of product for 1.2 xplore query driver tool (free on EDN!) Use finalmergingblackout (in xdb.properties) to keep large final merges from happening during peak query times Increase xdb buffer cache Set xhive-cache-pages to at least 25% of JVM memory in indexserverconfig.xml Use xplore native facets & small result windows instead of large result sets for interactive users

Tip #9: Batch Ingestion Optimization with Collections xplore has been optimized out-of-the-box for low latency Updates / inserts reflected as soon-as-possible Will temporarily create blacklisted

30 Tip #9: Batch Ingestion Optimization with Collections xplore has been optimized out-of-the-box for low latency Updates / inserts reflected as soon-as-possible Will temporarily create blacklisted items in existing index and will create new temporary small indexes Normal merge activity consolidates these for good response time However, large batch ingest scenarios could, by default, hurt online user response time temporarily Could lead to situation where batches have to be slowly ingested over time to minimize impact xplore s collections can be used in several ways to improve response time Ingestion into index-only collections Sub-collection adoption Dedicated batch index agent

Dedicated Batch Index Agent Purpose: to maintain index latency quality-of-service for interactive users in presence of large day-forward batch ingest rates Setup Outline: Setup additional normal mode

31 Dedicated Batch Index Agent Purpose: to maintain index latency quality-of-service for interactive users in presence of large day-forward batch ingest rates Setup Outline: Setup additional normal mode index agent with separate Ft user Unregister all events to that FT user Use id file loading technique or application created queue entries to feed the index agent events Batch events Interactive user events Batch index Agent Interactive index Agent

32 Ingestion into Index-Only collections Batch events Interactive user events Custom routing logic Custom routing logic Index-only collections are skipped by default query processing, but can be queried directly xplore Query When large merge is complete then collection state can be changed to index & search

33 Example: col-3 is set to index-only col-3 now in index only state

34 Testing that index-only data is not found in cross collection query So, although there are documents in the collection, a search across all of the does not find the col-3 docs. Cross collection query Document in col-3 doesn t show up in results

35 Ingestion into Index-Only collections Interactive user events Custom routing logic When large merge is complete then collection state can be changed to index & search xplore Query

36 Set back to index & search Putting the collection back into index & search state ensures it gets queried

Sub-collection Adoption Col-1 Indexes rolled up to col-2 col2 Adopt col-1 to col-2 Col-2 Col-1 now a subcollection of col- 2 With globally scoped queries many collections can degrade query

37 Sub-collection Adoption Col-1 Indexes rolled up to col-2 col2 Adopt col-1 to col-2 Col-2 Col-1 now a subcollection of col- 2 With globally scoped queries many collections can degrade query performance With high day-forward batch ingestion an alternative optimization is sub-collection adoption Collection is adopted as subcollection Indexes are rolled up to parent Restrictions apply Default facet compression needs to be disabled (for each facet field) else poor adoption performance will result Will lead to some I/O spikes Pre-production load testing strongly recommended Col-1

38 Example of sub-collection adoption Col-2 a top level collection Move button used for adoption Col-2 now a subcollection

39 Upcoming Next: Optimizing EMC Documentum: Best Practices for Deployment Wednesday, 2nd November 2011, 3:00pm 3:45pm Covers additional Documentum scalability improvements as well as HA / DR best practices for xplore

40 THANK YOU This presentation is also available at password: spree

Migrating from FAST to EMC Documentum xplore: What To Do and Why You'll Love It. Ed Bueché EMC Distinguished Engineer and xplore Architect

Migrating from FAST to EMC Documentum xplore: What To Do and Why You'll Love It Ed Bueché EMC Distinguished Engineer and xplore Architect Agenda Introduction to xplore xplore 1.2 new capabilities FAST-to-xPlore