EMC Documentum Search

Transcription

1 EMC Documentum Search Version 6.5 Development Guide P/N A01 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright EMC Corporation. All rights reserved. Published July 2008 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up to date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners.

3 Table of Contents Preface... 7 Chapter 1 Fulltext Indexing Overview... 9 Introducing fulltext indexing... 9 Indexing by format How content file size determines what is indexed Which languages are indexed How particular characters are handled Choosing parts of speech to index (grammatical normalization) Overview of the indexing process What comprises a fulltext indexing installation Content Server Index agent Index server and fulltext index The fulltext index Repository objects and properties supporting fulltext indexing Indexing of object attributes Querying a fulltext index Chapter 2 DQL Queries Database, fulltext, and standard searches Deciding whether to use a fulltext or a database query Content Server security and queries FTDQL syntax FTDQL hints How fulltext index queries are processed What causes inconsistent results Configured maximum results for indexer exceeded Indexing latency issues Grammatical normalization differences Differences in duplicate handling Query timeouts Case sensitivity differences What causes query timeouts Searching on property values that are too common Searching on property values using wildcards Using a wildcard in a SEARCH DOCUMENT CONTAINS clause Including the FOLDER(DESCEND) clause in a fulltext query Specifying a date range in the query Adding special characters to a query Searching on a grammatically normalized string Including count(*) in the selected values list Chapter 3 Configuring and Customizing the Webtop Search UI EMC Documentum Version 6.5 Search Development Guide 3

4 Table of Contents Configuration and customization points Understanding search in WDK Configuring search controls Fulltext and attributes only searches Configuring the basic search 6.0 component Configuring the advanced search component Configuring the savesearch component Configuring search results and tuning performance Making custom attributes available in search results Configuring Webtop extended search clustering Customizing search in Webtop applications Modifying the search JSP pages Modifying the search component query Hiding the customization from query editing Programmatic search value assistance Chapter 4 Customizing Search with DFC Configuring search in dfc.properties Configuring search relevancy Generating queries in DFC Simple queries in DFC DFC query builder APIs Creating DFC search applications Hello World search application Sample DFC application Using the visitor API Using a DQL hints file Hints file location Hints file elements Hints file examples DQL hints and Webtop search components Optimizing query batch size Chapter 5 Troubleshooting Queries Speeding up queries in WDK based applications Testing, logging, and debugging queries Tracing fulltext query operations Viewing the query smartlist Appendix A DQL Hints File DTD Appendix B DFC Search Service API Overview EMC Documentum Version 6.5 Search Development Guide

5 Table of Contents List of Figures Figure 1. FTDQL query algorithm Figure 2. Configuration and customization points in Webtop applications Figure 3. Search size custom drop down list Figure 4. Limiting the selectable types and subtypes Figure 5. Limiting the selectable types without subtypes Figure 6. Conditional value assistance UI Figure 7. Attribute selection drop down Figure 8. Specific attributes as search criteria Figure 9. Custom attributes as search criteria Figure 10. Fulltext search box removed from UI Figure 11. Simple search without added clause Figure 12. Simple search with added clause Figure 13. Asynchronous query processing Figure 14. Simple DFC search UI Figure 15. Simple DFC search application structure Figure 16. Data model EMC Documentum Version 6.5 Search Development Guide 5

6 Table of Contents List of Tables Table 1. Relevancy ranking configuration elements Table 2. DQL hints file elements EMC Documentum Version 6.5 Search Development Guide

7 Preface This document summarizes information of interest to developers customizing search in their applications. When you customize search in a WDK based application or a web service, you often need information about topics in several different products: Content Server, index server, DQL, DFC, and WDK. Consequently, the information in this document is drawn from the following sources: Content Server Administration Guide, Content Server DQL Reference Manual, DFC javadocs, WDK Development Guide, WDK Reference Guide, and internal information sources. Refer to Documentum Foundation Services 6.5 Development Guide for information about DFS search services. When you become familiar with the Content Server data model and indexing, you can design queries and search customizations and troubleshoot query performance. Web Development Kit (WDK) and Web Services Framework (WSF) provide you with tools to display query generating pages and results pages in Web accessible applications. DFC allows you to access query building functionality within a client application. DQL allows you to construct queries to use within your WDK, WSF, or DFC based application. The following chapters discuss various tools and procedures that will help you develop or tune search customizations: Chapter 1, Fulltext Indexing Overview Chapter 3, Configuring and Customizing the Webtop Search UI Chapter 4, Customizing Search with DFC Chapter 2, DQL Queries Chapter 5, Troubleshooting Queries This document does not cover administration details such as how to set up and configure an index server or ECIS server. For information on installing and configuring an index server and index agent, and information on maintaining indexes, refer to Content Server Fulltext Indexing System Installation and Administration Guide. For complete information on DQL and DQL hints, refer to Content Server DQL Reference Manual. This document also does not cover how to install ECIS adapters or how to develop an ECIS server adapter, because that information does not share any aspects with the remainder of this document. ECIS server and ECIS adapters are required for federated search (searches against external sources). For information on developing an ECIS adapter, refer to Enterprise Content Integration Services Adapter Development Guide. If you need assistance in implementing your customizations, contact Documentum Professional Services or Documentum Developer support. EMC Documentum Version 6.5 Search Development Guide 7

8 Preface Intended Audience This guide is directed to Java developers who are developing custom DFC, Web Services, or WDK based clients of the Content Server. The following skills are required for various customization tasks: Java JSP XML JavaScript DQL Conventions This manual uses the following conventions in the syntax descriptions and examples. Syntax conventions Convention Identifies italics A variable for which you must provide a value [ ] square brackets An optional argument that may be included only once { } curly braces An optional argument that may be included multiple times Revision History The following changes have been made to this document. Revision History Revision Date July 2008 Description Initial release for 6.5:added ORDER BY support in IDfQueryBuilder and reference to DFS Development Guide. 8 EMC Documentum Version 6.5 Search Development Guide

9 Chapter 1 Fulltext Indexing Overview This chapter describes the fulltext indexing process and the software components that perform indexing and searching, as well as the indexes themselves. For more information on index related objects and instructions for the maintenance of fulltext indexes, refer to Content Server Fulltext Indexing System Installation and Administration Guide. This chapter contains the following topics: Introducing fulltext indexing, page 9 Overview of the indexing process, page 14 What comprises a fulltext indexing installation, page 15 Indexing of object attributes, page 17 Querying a fulltext index, page 18 Content Server security and queries, page 21 Introducing fulltext indexing A fulltext index is an index on the properties and content files associated with objects of SysObjects and SysObject subtypes. Searching the index allows the rapid retrieval of objects whose properties or associated content files contain the values for which you are searching. All properties of SysObject and SysObject subtype objects are indexed automatically. That cannot be turned off. If any indexed object has an associated content file, the content file is also indexed, if the a_full_text property of the object is set to TRUE and the format of the content file is indexable. If the a_full_text property is set to FALSE, the content file is not indexed. Content files in all storage areas are indexed. If you are using distributed content, all content is copied to the distributed store component on the primary site for indexing. The drive on which the primary content store resides must have sufficient space for the primary content store plus the content copied from remote stores for indexing. During Content Server installation, you are asked to designate those languages for which grammatical normalization is enabled. Grammatical normalization ensures that all forms of a word are indexed and that a search for one form of a word also returns other forms of a word. Choosing parts of speech to EMC Documentum Version 6.5 Search Development Guide 9

10 Fulltext Indexing Overview index (grammatical normalization), page 12, and the Content Server Fulltext Indexing Installation Guide contain additional information about grammatical normalization. Fulltext indexing is enabled in the repository by default when the repository is created or upgraded to this Content Server version. However, Content Server itself does not create or maintain the fulltext index. You must install the fulltext indexing software components, which create and maintain the index. For instructions on installing the software and creating the index, refer to the Content Server Fulltext Indexing Installation Guide. Phonetic searching is not supported. Indexing by format Properties of the format object determine which formats are indexable and which content files in indexable formats are indexed. If the value of the can_index property of a content file s format object is set to TRUE, the content file is indexable. If the primary content of an object is not in an indexable format, you can ensure that the content file is indexed by creating a rendition in an indexable format. The format_class property of the format object may be set to values that determine which formats are indexed: ftalways All renditions in formats whose format_class property is set to ftalways are indexed. For example, if a document has renditions in Microsoft Word and PDF formats and the format_class property for both formats is set to ftalways, both renditions are indexed. ftpreferred If a document has multiple renditions in indexable formats and one is in a format whose format_class property is set to ftpreferred, the rendition in that format is indexed rather than any renditions in other formats, with the exception that any formats whose format_class property is set to ftalways are also indexed. If a document has more than one rendition whose format_class property is set to ftpreferred, the first rendition processed for indexing is indexed and the other renditions are not. Which rendition is processed for indexing cannot be determined in advance. It is recommended that for any document, only one rendition is in a format whose format_class property is set to ftpreferred. If a document has renditions in four different formats, of which the format_class of one is set to ftpreferred and the format_class of the other three is set to ftalways, all four renditions are indexed. There is no default value for the format_class property. You must set it manually to designate whether a format is always indexed or is the preferred format for indexing. By default, the first content file in a format whose can_index property is set to true is indexed. Other renditions of the object are not indexed. If the primary content of an object is not in an indexable format, create a rendition in an indexable format. Refer to Content Server Fulltext Indexing System Installation and Administration Guide for a complete list of indexable formats. 10 EMC Documentum Version 6.5 Search Development Guide

11 Fulltext Indexing Overview If the content file associated with a SysObject exists in a non indexable format, its properties are still indexed. To index the content, create a rendition of the SysObject in an indexable format. Use Documentum Content Transformation Services or third party client applications to create the rendition. Some formats found in the appendix are not be represented in the repository by a format object. The formats.cvs file, which is located in $DM_HOME/install/tools, contains a complete list of supported mime_types and the formats with which they are associated. If a supported mime_type is not represented by a format object, create a format object in the repository and map the supported mime_type to the format. How content file size determines what is indexed A content file consists of properties, indexable content (words), and non indexable content such as pictures, binary content. If the sum of the metadata size and indexable content size is greater than 10 MB, the content and properties are not indexed. If you are indexing very large files, refer to Content Server Fulltext Indexing System Installation and Administration Guide for important configuration recommendations. Which languages are indexed Content files and properties in all supported languages are indexed by default. All standard Unicode character sets are supported. No special configuration is necessary. Refer to Content Server Fulltext Indexing System Installation and Administration Guide for a complete list of supported languages. Two right to left languages are supported for fulltext indexing with certain limitations: Hebrew and Arabic. Other right to left languages cannot be indexed. For non binary formats, only logical text representation is supported; visual text representation is not supported. For those binary formats listed in Content Server Fulltext Indexing System Installation and Administration Guide that support right to left text in the native format, support is provided for indexing Hebrew and Arabic text, with the exception that PDF files cannot be indexed. How particular characters are handled The following Unicode characters are indexed and are searchable: Alphabetic characters Numeric characters EMC Documentum Version 6.5 Search Development Guide 11

12 Fulltext Indexing Overview Extender characters Extender characters extend the value or shape of a preceding alphabetic character. These are typically length and iteration marks. Custom characters enclosing Chinese, Japanese, and Korean letters and months These are derived from a number of custom character ranges that have bidirectional properties, falling in the FF range. The specific character ranges are: B 327F 32B0 32C0 32CB 32D0 32FE Other characters, including punctuation, accent, and diacritical marks, and characters such as and #, are not indexed or searched. Such nonsearchable characters are removed from the indexed text and treated as if they are blank spaces. The index server treats the following characters as white space:!@#$%^_,.&;:() +=< When these characters appear in indexable content, they are replaced by white space. For example, when the address MyName@company.com is indexed, it appears as as ʺMyName company comʺ in the index. The text is treated as three words. Documents returned by a search for MyName@company.com are treated as if they contain the words ʺMyName company com.ʺ If a special character is included in a query, it is removed. For example, querying on Richard+Dodd would return a document containing the text Richard=Dodd because the + and = signs are both replaced by a blank space. If a search term includes an accent or diacritical mark, the search returns all matching words with or without the accent or diacritical mark. Choosing parts of speech to index (grammatical normalization) The fulltext engine can perform grammatical normalization, also called lemmatization, while indexing and for specific queries. When you install the fulltext components, you can designate the parts of speech to be normalized. For example, a search for ʺcarsʺ also returns ʺcarʺ if nouns are normalized. The following combinations of parts of speech can be normalized: Nouns Nouns and adjectives Nouns, adjectives, and verbs 12 EMC Documentum Version 6.5 Search Development Guide

13 Fulltext Indexing Overview Nouns and verbs If you choose to install grammatical normalization, you can then choose whether to normalize nouns, verbs, adjectives, or combinations of these. The choice applies to both indexing and querying normalization operations. For example, if you chose to normalize nouns and verbs, the fulltext system will normalize nouns and verbs during indexing and while processing query search conditions. The default choice during installation, recommended for better performance and disk space usage, is to normalize only nouns. Internally, within the index, normalized entries are stored separately from the exactly matching entries. When a document is indexed under grammatical normalization, it will have entries in both portions of the index. When a grammatically normalized search is conducted, the search is conducted against the normalized entries. If a search is conducted with grammatical normalization turned off, the search is conducted against the portion of the index containing the exactly matching entries. The fulltext engine uses reductive grammatical normalization for both indexing and query normalization. For example, suppose a document is submitted for indexing that contains the noun ʺcatsʺ. The word is reduced to its grammatical base, cat for indexing purposes. When indexing is complete, the document has entries in the index for ʺcatʺ as well as ʺcatsʺ. Similar reductive grammatical normalization is performed on terms during searches. For example, if a search is conducted on ʺcatsʺ, the term in the query is reduced to its grammatical base before the index is searched for a match. To illustrate, suppose a user issues the following query to find documents containing ʺcatsʺ: SELECT r_object_id,owner_name FROM dm_document SEARCH DOCUMENTUM CONTAINS 'cats' The fulltext engine grammatically normalizes ʺcatsʺ before conducting the search and searches the index for any normalized entries that contain ʺcatʺ. As a consequence of this normalization for indexing and searching, the query returns documents that contain ʺcatʺ as well as those that contain ʺcatsʺ. Note: For information on the effects of grammatical normalization on search results, refer to Searching on a grammatically normalized string, page 30. Grammatical normalization is enabled automatically for Chinese, Japanese, and Korean. Grammatical normalization can be optionally enabled for the following languages: German English Spanish French Hungarian Italian Norwegian Polish Portuguese Russian EMC Documentum Version 6.5 Search Development Guide 13

14 Fulltext Indexing Overview Overview of the indexing process Fulltext indexing is controlled by three software components: Content Server manages the objects in a repository, generates the events that trigger fulltext indexing operations, queries the fulltext indexes, and returns query results to client applications. The Content Server uses a query plug in to send fulltext queries to the index server. The index agent processes index queue items generated by Content Server and prepares SysObjects for indexing. The index server creates fulltext indexes and responds to fulltext queries from Content Server. The Content Server Installation Guide contains information on installing Content Server. The Content Server Fulltext Indexing Installation Guide contains information on installing the index agent and index server. The indexing process is: 1. A Save, Checkin, Destroy, Readonlysave, or MoveContent operation is performed on a SysObject in the repository. 2. The event generates a queue item. 3. The queue item is sent to the fulltext user s work queue. The fulltext user, dm_fulltext_index_user, is a Superuser created when a repository is created or when an existing repository is upgraded. 4. The index agent acquires the queue item. 5. The index agent retrieves the object associated with the queue item from the repository and, using its index plug in, creates a DFTXML representation of the object that can be understood by the index server. DFTXML is an XML format that contains the object s properties and the location of the object s content file, if any. If you are using distributed content and a content file is located at a remote component of the distributed store, the content file is copied to the distributed store component located at the primary site. 6. The DFTXML representation of the object is sent to the index server. 7. The index server retrieves the content file, if any, and creates its own representation of the content file and properties, called FIXML. 8. The index server notifies the index agent that the FIXML is created and the object will be indexed. 9. The index agent destroys the queue item for the object. 10. The index server indexes the content file and its properties. The object is now searchable, but note that the index server does not provide any indication that an object is searchable. 14 EMC Documentum Version 6.5 Search Development Guide

15 Fulltext Indexing Overview The fulltext index is updated on a continuous basis, provided that all of the software components are running. No special administrative tasks must be performed to ensure that the index is updated and current. What comprises a fulltext indexing installation A fulltext indexing installation consists of the software components (Content Server, the index agent, and the index server), the fulltext index, the repository objects required to support the indexing process, and initialization files and initialization file parameters that support fulltext indexing. Content Server The Content Server generates queue items that are read by the index agent, issues queries to the index server, and returns query results to client applications. The Content Server uses a query plug in to translate native Documentum fulltext queries to the index server s query language. Index agent The index agent is a multithreaded Java application running in the Apache Tomcat servlet container. It is installed on the Content Server host or a separate host. An index agent may run in one of several operational modes: Normal mode is for event driven indexing operations. An index agent associated with a new repository is automatically created in normal mode. In normal mode, the index agent processes index queue items and prepares the SysObjects associated with the queue items for indexing. Migration mode is for creating indexes for existing content in an upgraded 5.3 or 6 repository. In migration mode, the index agent processes all SysObjects in a repository sequentially in r_object_id order and prepares them for indexing. A single queue item, the high water mark queue item, is used to mark the index agent s progress in the repository. For complete information on running the index agent in migration mode and creating new indexes, refer to the Content Server Fulltext Indexing Installation Guide. File mode is used to submit a list of objects IDs to the index agent when a new index is created and index verification determines which objects are missing from the index. Because file mode is used following a migration, its usage is described in the Content Server Fulltext Indexing Installation Guide. EMC Documentum Version 6.5 Search Development Guide 15

16 Fulltext Indexing Overview An index agent in normal mode is represented by an ft index agent config object. The properties of the ft index agent config object primarily record status information about the index agent, including the mode in which the index agent is running and when the index agent began processing queue items. The properties also record configuration information about the index agent, such as the number of queue items processed in a single batch, the number of exporter threads, and the time interval at which the index agent polls the repository for queue items. This information may be viewed using Documentum Administrator. For more information about the ft index agent config object, refer to the EMC Documentum Object Reference Manual. An index agent in migration mode is represented by an XML configuration file, indexagent.xml, on the index agent host. Do not modify the parameters in the configuration file unless you are enabling file store mapping. (Mapping file stores for improved indexing performance is documented in the Content Server ffulltext Indexing Installation Guide.) Index server and fulltext index The index server is a third party server product that creates and maintains the fulltext index for a repository. The index server also receives fulltext queries from Content Server and responds to those queries. The index server may be installed on the Content Server host or on a different host. For performance reasons, it is strongly recommended that the index server is installed on a different host. The index server is installed to $DOCUMENTUM/fulltext on UNIX and %DOCUMENTUM%\fulltext on Windows. For complete information on installing and running the index server, refer to the Content Server Fulltext Indexing Installation Guide. The index server is represented in the repository by the ft engine config object (dm_ftengine_config). There is one instance of the ft engine config object for each fulltext index object. The index server receives DFTXML from the index agent and creates its own representation of the SysObject to be indexed, called FIXML. If the SysObject has an associated content file, the index server retrieves the content file from the repository. The FIXML is then used to add the SysObject s content and properties to the index. If the index agent indicates that an object was deleted from the repository, the index server deletes the object from the index. The index server logs may be viewed from the JSP interface during migration or from Documentum Administrator when the index server is running against a 5.3 or later repository and the index agent is in normal mode. The fulltext index The fulltext index is stored on the index server host, in the $DOCUMENTUM/data/fulltext (%DOCUMENTUM\data\fulltext on UNIX), in two directories: 16 EMC Documentum Version 6.5 Search Development Guide

17 Fulltext Indexing Overview fixml, which contains the raw data from which the index is created index, which contains the fulltext index in a group of subdirectories Repository objects and properties supporting fulltext indexing Fulltext indexing is supported in the repository by objects representing the fulltext index, the index server, and the index agent. Properties of the server config object, SysObject, and queue item also support fulltext indexing. The following properties of other objects support fulltext indexing: a_full_text fulltext_location Indexing by format, page 10 contains information on how the setting of the format_class property of the format object controls which renditions are indexed. The a_full_text property is defined for the SysObject type and is inherited by all SysObject subtypes. It is a Boolean property that controls whether an object s associated content files are indexed. When a_full_text is TRUE, content files are indexed whenever a Save, Checkin, Destroy, Readonlysave, or MoveContent operation generates an index queue item for the object. Any changes to the object s content are added to the index. The a_full_text property is set to TRUE whenever a SysObject is created. Users with Sysadmin or Superuser privileges can change the a_full_text setting. Users without Sysadmin or Superuser privileges cannot change the a_full_text setting. Indexing of object attributes Specific attributes of the server config object, SysObject, and queue item support fulltext indexing. Use Documentum Application Builder to make object types and attributes searchable or not searchable by clients and to set allowed operators, default operator, and default operator argument. Making an object type searchable You can set the is_searchable attribute on an object type to allow or prevent searches for objects of that type. This attribute must be read by the client; it is not used by the indexing process. Webtop does not present object types or attributes for selection in the search UI if is_searchable is set to false. Making searchable attributes and setting operators You can set the is_searchable attribute on an attribute to allow or prevent searches on the attribute value. You can set allowed_search_ops to set the allowed search operators and default_search_op to set the default operator. The default_search_arg EMC Documentum Version 6.5 Search Development Guide 17

18 Fulltext Indexing Overview attribute sets a default argument to be used with the default operator. These attributes must be read by the client; they are not used by the indexing process. Webtop honors these attributes. Making content searchable The a_full_text attribute is defined for the SysObject type and is inherited by all SysObject subtypes. It is a Boolean attribute that controls whether an object s associated content files are indexed. When a_full_text is TRUE, content files are indexed whenever a Save, Saveasnew, Checkin, Destroy, Branch, or Prune operation is performed on the object. Any changes to the object s content are added to the index. The a_full_text attribute is set to TRUE whenever a SysObject is created. Users with Sysadmin or Superuser privileges can change the a_full_text setting. Querying a fulltext index Indexes are queried using a subset of DQL called FTDQL. Refer to FTDQL syntax, page 21 for information on FTDQL syntax. FTDQL is fully described in Content Server DQL Reference Manual. All index searching is case insensitive. 18 EMC Documentum Version 6.5 Search Development Guide

19 Chapter 2 DQL Queries To retrieve data dictionary information using DQL, you use a query against the object types that contain the published information. These types are dd common info, dd type info, and dd attr info. For example, the following query returns the labels for dm_document attributes in the English locale: SELECT ʺlabel_textʺ FROM ʺdmi_dd_attr_infoʺ WHERE ʺtype_nameʺ= dm_document AND ʺnls_keyʺ= en. For full information on using DQL, refer to Content Server DQL Reference Manual. For full documentation on objects and attributes on which you can base a query, refer to EMC Documentum Object Reference Manual. The following topics describe fulltext queries: Database, fulltext, and standard searches, page 19 Deciding whether to use a fulltext or a database query, page 20 Content Server security and queries, page 21 FTDQL syntax, page 21 FTDQL hints, page 21 How fulltext index queries are processed, page 22 What causes inconsistent results, page 24 What causes query timeouts, page 26 Optimizing query batch size, page 78 Database, fulltext, and standard searches Depending on the structure of a query, the query is executed against the database (the repository metadata), the fulltext index, or both. A query that executes solely against the database is called a database query. A query that executes solely against the fulltext index is called a fulltext query or FTDQL. A query that is executed against both the database and the index is called a standard query. EMC Documentum Version 6.5 Search Development Guide 19

20 DQL Queries A database query searches on values recorded in properties. It does not have a SEARCH DOCUMENT CONTAINS clause, nor any other hint or option that would cause Content Server to direct the query to the fulltext index. For example, the following query is a database query: SELECT object_name,object_owner,title FROM dm_document WHERE subject 'Engine Additives' A fulltext index query searches the index for the objects that satisfy the search criteria. The query may search on indexed property values, content, or both. A fulltext query conforms to the rules of FTDQL query syntax. For example, the following query is a fulltext query: SELECT object_name,object_owner,title FROM dm_document SEARCH DOCUMENT CONTAINS 'Engine Additives' Note: The CONTAINS operator in Content Server 6 searches only for the word, not a fragment of a word. For example, if the user searches for a title attribute containing the string ʺwdk on the advanced search page of Webtop, only titles containing the word ʺwdk would be returned and not titles such as ʺwdkindexing or ʺsearch_wdk. For information on how to revert to the CONTAINS fragment behavior, refer to Enabling the wildcard CONTAINS operator for string property searches, page 41. A standard query queries both the fulltext index and the database. Such a query typically contains a SEARCH clause and a WHERE clause. If the WHERE clause is not compliant with the rules of FTDQL or the query contains an explicit ENABLE(NOFTDQL), Content Server executes the SEARCH clause against the fulltext index and the WHERE clause against the database and then returns the intersection of the results. Note: For information about how to customize search component queries in WDK based applications, refer to Customizing search in Webtop applications, page 46. If you run a query as a fulltext query and then rerun it as a database or standard query, you may obtain results that are inconsistent. The number of results from one execution may differ from the number of results in the second execution. What causes inconsistent results, page 24, discusses some causes of inconsistent results. Deciding whether to use a fulltext or a database query Fulltext queries have more capability for natural language and free text querying than DQL queries. They are not the query type of choice for transactional semantics or ID based searches using wildcards. For example, DOCUMENT CONTAINS FOO scales well with few occurrences of FOO in the repository metadata but with many hits in the fulltext index. In contrast, DOCUMENT CONTAINS ID does not scale well because the repository metadata has many hits for ID. For an attributes only query, a NOFTDQL can perform better. Attribute queries must be evaluated for whether they are likely to return many hits. For attributes that can return many results, such as format, you may consider using the NOFTDQL hint. A selection in the Webtop UI labelled Include recently modified properties will search for attribute values in the database instead of the fulltext 20 EMC Documentum Version 6.5 Search Development Guide

21 DQL Queries index (performs NOFTDQL search on attributes). This option is not enabled out of the box and requires configuration. Refer to Enabling the wildcard CONTAINS operator for string property searches, page 41 for instructions. Note: With fulltext indexing, all attributes are indexed. For attributes that will be queried against the database, you must create an index in the database. In WDK clients, all search component queries are fulltext queries of the fulltext index unless a DQL hints file is in place. The hints file allows you to specify certain conditions under which a database or standard query will be done in place of a fulltext query. For information on this, refer to Using a DQL hints file, page 72. Content Server security and queries When the fulltext engine runs a query, Content Server permissions are applied in order to restrict content that can be accessed in the query results. Permissions are applied by the server, not through clients such as WDK or DFC. If a user has only Browse permission on an object, the object can still appear in a result list from a fulltext search, and summaries for browse only objects are displayed, but the user will not be able to access the content object if required permissions are not present. FTDQL syntax An FTDQL statement is a SELECT statement whose syntax conforms to a particular set of rules that allow the query to be executed directly against the fulltext index. If the statement conforms to FTDQL syntax, the statement is run solely against the fulltext index; the repository is not queried. This feature provides enhanced performance for the query. The general syntax accepted for an FTDQL query is shown in the formal syntax descriptions in Content Server DQL Reference Manual.. Following is an example of FTDQL: SELECT owner_name,r_creation_date FROM dm_document SEARCH DOCUMENT CONTAINS 'budget' For full information on FTDQL syntax, refer to Content Server DQL Reference Manual. FTDQL hints DQL hints can be added to a query to change query behavior. For information on all DQL hints, refer to Content Server DQL Reference Manual. The ENABLE(FTDQL) hint causes the Content Server to attempt to execute the query as an FTDQL query. If the remaining syntax in the query conforms to the required syntax for an FTDQL query, EMC Documentum Version 6.5 Search Development Guide 21

22 DQL Queries the query is executed as an FTDQL query. If the syntax does not conform to FTDQL query rules, an error is returned. Note: Some DQL statements qualify as FTDQL only when they are explicitly marked as FTDQL. For example, the following statement is not FTDQL unless the ENABLE(FTDQL) hint is used: select r_object_id from dm_document where any authors like '%ziff%' The statement does not have at least one of the features required for FTDQL: search clause, SCORE or SUMMARY in the selected values list, or the DQL hint ENABLE(FTDQL). Without the hint, this statement would be silently executed as normal DQL against the repository metadata. With the hint, it is executed by the fulltext search engine, and the term following the ʺlikeʺ will be converted to a fulltext query. The TRY_FTDQL_FIRST hint is added to all queries that are built with the DFC query builder package. This hint handles timeouts and resource exceptions returned from the index server by querying the attributes portion of a query against the repository s database. You can turn off FTDQL for the attribute portion of a query with the hint ENABLE(NOFTDQL), similar to the following query: Select r_object_id from dm_document SEARCH DOCUMENT CONTAINS foo WHERE object_name = bar ENABLE(NOFTDQL) All WDK based search components issue FTDQL queries with the TRY_FTDQL_FIRST hint. You can disable FTDQL querying under certain conditions, or cause the query to be issued as a standard query, using a DQL hints file. Refer to Using a DQL hints file, page 72. How fulltext index queries are processed FTDQL is a subset of Document Query Language (DQL). FTDQL is used for querying fulltext indexes. DQL and FTDQL are fully documented in Content Server DQL Reference Manual. Figure 1, page 23 diagrams the process of a query that meets FTDQL requirements. All WDK based search components use FTDQL queries. 22 EMC Documentum Version 6.5 Search Development Guide

23 DQL Queries Figure 1. FTDQL query algorithm In the FTDQL query algorithm, the SEARCH and WHERE clauses are converted to a fulltext query against the fulltext index and the SELECT list values are queried against the database. The Content Server checks security on the results returned from the fulltext index and merges the results from the database and index. FTDQL is a performance enhancement for queries, so most queries should be FTDQL. Application writers should design interfaces that favor FTDQL queries. Fulltext querying by the DFC query builder can be turned off globally in dfc.properties on any DFC installation host, including a Web application server host, with the following setting: dfc.search.fulltext.enabled=false You can disable FTDQL querying under certain conditions, or cause the query to be issued as a standard query, using a DQL hints file. Refer to Using a DQL hints file, page 72. Note: A consistency error can occur with FTDQL: When an object is saved to the repository, it may not be immediately indexed at the same time that the transaction is committed. EMC Documentum Version 6.5 Search Development Guide 23

24 DQL Queries What causes inconsistent results Some queries generate different results when they are executed as a fulltext query than when they are executed as a database or standard query. Possible reasons for this are discussed in the following topics. Configured maximum results for indexer exceeded XXXXXXXXX question to CS team 6/25/08 (Mike Lee and Angie Elliott to pass it on), as I could not find any configuration setting that governs this. Note from Bruno M.: ʺYou can specify at the indexer level, a maximum number of returned results. If a query can potentially returns more results than this number, the query is unselective and the CS returns 0 results without any error messages. Ask the CS team about his propertyʺxxxxxxxxxxxxx Indexing latency issues Latency is the time interval between two events. In the context of searching, latency caused by a number of situations may cause inconsistent results. For example, the following situations may generate latency periods that result in inconsistent results: An object was deleted in the repository but that deletion is not yet reflected in the index In this case, a query against the index returns a result, whereas the same query against the repository does not. An object was added to the repository but is not yet added to the index In this case, a query against the repository returns the result, whereas the same query against the index does not. Additionally, a query that includes a count(*) in the selected values list may return an inflated number if the query is run against the fulltext index. During some phases of index management by the fulltext engine, there may be two entries for an object in the index. A query containing a count(*) issued at that time will count both entries. When Content Server processes results, it removes duplicates as much as possible. However, it may not be able to remove all duplicates, and the count may be inflated as a result. Grammatical normalization differences The fulltext engine uses grammatical normalization when conducting a search. Database searches do not support grammatical normalization. Content Server only returns exact matches. This means 24 EMC Documentum Version 6.5 Search Development Guide

25 DQL Queries that the same query, run against the index and run again against the database may return different numbers of results. For more information about grammatical normalization and how to control its use, to turn it on or off, refer to Searching on a grammatically normalized string, page 30. Differences in duplicate handling The fulltext engine may return duplicates for some results. When this occurs, Content Server attempts to remove the duplicates but is not always successful. Consequently, a query executed in one manner may return results that differ from the same query executed in another manner. The DFC search package (query builder), which is used by the Webtop search components, removes duplicate results so no duplicates are presented in the results UI. Query timeouts If a fulltext query times out, it returns an error and no results. If the same query is executed against the database and does not time out but returns results, users experience an apparent difference in returns. The users may be unaware that the difference in returned results is due to a query timeout. Queries against an indexed Server that are constructed through the DFC QueryBuilder package will automatically include the TRY_FTDQL_FIRST DQL hint. This hint helps avoid query timeouts by running the query against the database if the query times out when run against the fulltext index. The hint can be overridden by explicit NOFTDQL hints. For information on using a DQL hints file for NOFTDQL hints, refer to Using a DQL hints file, page 72. For information about how to formulate queries to avoid timeouts, refer to What causes query timeouts, page 26. Case sensitivity differences Searches on the fulltext index are not case sensitive. Searches in the database are case sensitive by default. This difference can cause queries to return different numbers of results. For example, suppose you issue the following query: SELECT object_name,object_owner,title FROM dm_document WHERE subject = 'bread' ENABLE(FTDQL) This query runs as a fulltext query and returns all objects whose subject is bread, Bread, bread, or any other combination of upper and lowercase letters that spell bread. If the query is run without the ENABLE(FTDQL) hint, it runs against the database. In that case, the query returns only those objects whose subject is bread, all lowercase. EMC Documentum Version 6.5 Search Development Guide 25

26 DQL Queries If you want to run that query against the database and in a case insensitive manner, you could use the upper (or lower) function: SELECT object_name,object_owner,title FROM dm_document WHERE UPPER(subject) = UPPER('bread') Note: Queries that include the UPPER or LOWER functions perform better if the database includes functional indexes on the queried properties. What causes query timeouts This section discusses query timeouts and how to avoid them. A query timeout represents a situation in which the fulltext engine could not process a query to completion because resource limits or time limits were exceeded. There are a variety of reasons that a query might timeout. The following topics describe the most common reasons and provides some guidelines for avoiding each. Searching on property values that are too common A query may be an unselective query because it bases a search on a property value that is very common among the objects in the repository. For example, the following query may be such a query if the specified property value is common: SELECT object_name, object_owner FROM dm_sysobject WHERE a_storage_type = "engrfilestore" ENABLE(FTDQL) If engrfilestore is the default filestore for SysObjects, it is very possible that this query would find a large number of objects and may time out. Searches with the ʺcontainsʺ, ʺbegins withʺ, and ʺends withʺ operator is a wildcard search that is also likely to return a large number of results. The index server has a limit for wildcard results in the file fsearch.addon, located in the directory $DOCUMENTUM\fulltext\IndexServer\etc\config_data\RTSerch\webcluster. The value of the wildcardcutoff is set to When this limit is exceeded by a search, an error message ʺQRServerError (1012): Resource limit exceeded is returned and the query does not return any hits. This value is configurable. Avoiding timeout To avoid this sort of timeout, use a query that is more selective or execute the query against the database. To make the query more selective, query against a more selective property value. For example: SELECT object_name, object_owner FROM dm_sysobject WHERE subject = "proposal" ENABLE(FTDQL) 26 EMC Documentum Version 6.5 Search Development Guide

27 DQL Queries To execute the query against the database, remove the ENABLE clause. If the query is generated through a Webtop search component, you can use the DFC dqlhints.xml file to remove the clause. For instructions on configuring that file, refer to Using a DQL hints file, page 72. Searching on property values using wildcards Fulltext queries that include a LIKE predicate that is too unselective due to a wildcard may time out. For example, the following query is such a query: SELECT object_name,bug_reporter FROM bugreport WHERE object_name LIKE 'Bug *' ENABLE(FTDQL) Depending on the number of bug reports in the system, this query could return thousands of objects. The query is unselective because it causes the Index Server to look up all Bug numbers starting with 1. Avoiding timeout To avoid this type of timeout, either use a more selective query or query the database instead of querying the index. If it is possible, provide a more selective LIKE predictate: SELECT object_name,bug_reporter FROM bugreport WHERE object_name LIKE 'Bug 150%' ENABLE(FTDQL) If providing a more selective predicate is not possible or is not effective, query the database instead of executing the query as a fulltext query. For example: SELECT object_name,bug_reporter FROM bugreport WHERE object_name LIKE 'Bug 1%' Without ENABLE(FTDQL), Content Server executes the query against the database. If the queries are generated through the Webtop search components, use the DFC dqlhints.xml file to define rules about when to drop the ENABLE(FTDQL) clause. For information about that, refer to Using a DQL hints file, page 72. In Webtop applications, the default operator for an attribute is read from the data dictionary. Use Documentum Application Builder to change the default operator for commonly queried attributes from ʺcontains, which generates a starting and ending wildcard, to ʺbegins with. Using a wildcard in a SEARCH DOCUMENT CONTAINS clause A SEARCH DOCUMENT CONTAINS clause in a query automatically directs Content Server to run the query against the fulltext index to find those results that satisfy the clause. If that clause contains a search condition that specifies an unselective search, the query may time out. For example, the following query is such a query: SELECT object_name,bug_reporter FROM bugreport SEARCH DOCUMENT CONTAINS 'Bug 1%' EMC Documentum Version 6.5 Search Development Guide 27

28 DQL Queries Avoiding timeout You cannot issue a query that contains only a SEARCH DOCUMENT CONTAINS clause against the database. That clause automatically sends the query against the index. However, to make the query less likely to time out, you can: Add more search terms to the query Use a longer query string before the wildcard, making that search more selective Use the DQL hint ft_contain_word, which will search for whole words only and not sentence fragments. For more information, refer to Using a DQL hints file, page 72. If the string you are searching for represents a property value, remove the SEARCH DOCUMENT CONTAINS clause and run the query against the database instead. For example: SELECT object_name,bug_reporter FROM bugreport WHERE object_name LIKE 'Bug 1%' Including the FOLDER(DESCEND) clause in a fulltext query A fulltext query that includes a FOLDER(DESCEND) clause may time out if there are too many folder in the DESCEND path. The query plug in translates FOLDER(DESCEND) clauses to a list of the folders in the DESCEND path and appends that list to the generated query. The plug in has a hard coded limit on the number of folders that the query can include. If the folders in the path exceed that number, the query returns an error. However, if the number is less than the limit but is still very high, the query may time out. For example, assuming that /Engineering is a top level cabinet and contains hundreds of folders, the following query might result in a timeout: SELECT r_object_id,title FROM dm_document SEARCH DOCUMENT CONTAINS 'specification' WHERE FOLDER('/Engineering',DESCEND) Avoiding timeout You can avoid this problem by executing the FOLDER clause against the database, rather than against the index. To do so, rewrite the query to execute as a standard query. For example: SELECT r_object_id,title FROM dm_document SEARCH DOCUMENT CONTAINS 'specification' AND FOLDER('/Engineering',DESCEND) ENABLE(TRY_FTDQL_FIRST) Adding the ENABLE(TRY_FTDQL_FIRST) hint directs Content Server to first execute the query as a fulltext query, but if that times out, to then retry the query against the database and retry the fulltext string against the index. The hint is added automatically to an FTDQL query against an indexed Content Servers if the query is generated using the DFC QueryBuilder package. If you do not use that package to generate the query, you can use the DFC dqlhints.xml file to define rules about when to add the hint. You can override the TRY_FTDQL_FIRST hint in query builder queries by using the ENABLE(NOFTDQL) hint. This will query attributes against the database only. 28 EMC Documentum Version 6.5 Search Development Guide

29 DQL Queries The DFC hints file also supports turning off FTDQL for specific locations. For more information, refer to FTDQL hints, page 21. Specifying a date range in the query Specifying a date range in a query is common, but queries that include them often perform poorly when executed as fulltext queries. For example, the following query includes a date range and is executed against the fulltext index: SELECT r_object_id,object_name FROM dm_document SEARCH DOCUMENT CONTAINS 'proposal' WHERE r_creation_date>=date('01/01/05','mm/dd/yy') AND r_modify_date<=date('01/01/06','mm/dd/yy') ENABLE(FTDQL) Avoiding timeout To avoid a timeout, execute the query as a standard query, so that the WHERE clause is executed against the database, instead of against the fulltext index. For example, the following query executes as a standard query: SELECT r_object_id,object_name FROM dm_document SEARCH DOCUMENT CONTAINS 'proposal' WHERE r_creation_date>=date('01/01/05','mm/dd/yy') AND r_modify_date<=date('01/01/06','mm/dd/yy') ENABLE(NOFTDQL) Another option is using the ENABLE(TRY_FTDQL_FIRST) hint instead of the ENABLE(FTDQL) hint. The TRY_FTDQL_FIRST hint directs Content Server to first execute the query a fulltext query, but if that times out, to then retry the attributes portion of the query against the database. This hint is added automatically to all queries generated using the DFC QueryBuilder package. Adding special characters to a query The fulltext engine treats some special characters as white space. For example, the term ʺSection 1.0.9ʺ would be indexed as four separate terms (Section, 1, 0, and 9) because the fulltext engine treats dashes and periods as white space when indexing. If a user searches on a phrase or word that contains these characters, the search engine will generate a list of the occurrences of these ʺindividualʺ terms and then check their positions in the documents where the occurrences happen to check that the terms are exactly next to each other. The search may cause a timeout due to the complexity of the search processing. Refer to How particular characters are handled, page 11 for a list of the characters that the fulltext engine treats as special characters and a description of how the characters are handled during indexing. Avoiding timeout To avoid this kind of timeout, rephrase the query to search on a value that does not include a special character. EMC Documentum Version 6.5 Search Development Guide 29

30 DQL Queries Searching on a grammatically normalized string Grammatical normalization is enabled or disabled during installation of fulltext indexing. For more information on this setup, refer to Choosing parts of speech to index (grammatical normalization), page 12. Because normalization increases the size of the index and the number of possible results for a search, normalization can contribute to timeouts or resource (index server) overload. Grammatical normalization is automatically disabled in the certain circumstances to avoid query timeouts. Normalization is not used in the following FTDQL queries: A search on a phrase of more than one word A search condition that contains an asterisk (*) or question mark (?), which denote a wildcard A DQL hint that turns off normalization Because special characters are treated as whitespace, searches on words that contain special characters are treated as multi word phrase searches. For those words, grammatical normalization is not used. Avoiding timeout To avoid this issue, you can turn off grammatical normalization. You can turn it off globally for all queries or on a per query basis. The DFC query builder, which is used by the Webtop search components, turns off normalization for wildcard and phrase searches. Turning off normalization globally To control the feature globally, for all searches, set the fulltext engine configuration parameter fds_override_lemmatize. The parameter is set in the dm_ftengine_config object. The parameter name is set in the param_name property and the value (true or false) is set in the param_value property. Setting the parameter to true turns off grammatical normalization for all searches. Setting the parameter to false turns on normalization for all searches. The properties param_name and param_value are repeating. Consequently, you must ensure that the name and value are set at the same index position in the properties. You must reinitialize Content Server after setting the parameter. Turning off normalization for specific queries If you wish to execute a query in a grammatical normalization mode that is not in agreement with the globally set mode, you can use a hint to the query plug in. This hint is ENABLE(dm_fulltext( qtf_lemmatize=0 1 )). If grammatical normalization is turned on globally, but you want to execute one query without using that feature, add the following to the end of the query: ENABLE(dm_fulltext('qtf_lemmatize=0')) For example: SELECT r_object_id, object_owner FROM dm_document SEARCH DOCUMENT CONTAINS 'specification' ENABLE(dm_fulltext('qtf_lemmatize=0')) Setting the hint to 0 directs the query plug in not to perform a grammatically normalized search, but only return results that match the search term or phrase exactly. 30 EMC Documentum Version 6.5 Search Development Guide

31 DQL Queries If grammatical normalization is turned off globally, but you want to execute a query using that feature, add the following to the end of the query: ENABLE(dm_fulltext('qtf_lemmatize=1')) For example: SELECT r_object_id, object_owner FROM dm_document SEARCH DOCUMENT CONTAINS 'specification' ENABLE(dm_fulltext('qtf_lemmatize=1')) Setting the hint to 1 turns on the feature for that query. Like other hints, you can use this hint in the dqlhints.xml file to turn normalization on or off for particular query conditions. For information about using a dqlhints.xml file, refer to Using a DQL hints file, page 72. Including count(*) in the selected values list Queries that include count(*) in the selected values list are often run to determine how many of a particular object are indexed or found in the repository. However, these queries can often return very large numbers of results. Because queries that reference an aggregate function such as count() cannot run as FTDQL queries, to process those results, the system populates a temporary table, which can be resource and time consuming. For example, the following query may exceed available resources or cause a timeout: SELECT count(*) FROM dm_sysobject SEARCH DOCUMENT CONTAINS 'dm' Avoiding timeout Use ESTIMATE_SEARCH, an administration method, to execute the query. This method returns the exact number of matches that satisfy the SEARCH condition if the user running the method is a Superuser or if there are more than 25 matches. To use this method, the use_estimate_search key in the server.ini file must be set to true. For complete information about ESTIMATE_SEARCH and its syntax, refer to its reference description in the DQL Reference Manual. EMC Documentum Version 6.5 Search Development Guide 31

32 DQL Queries 32 EMC Documentum Version 6.5 Search Development Guide

33 Configuring and Customizing the Webtop Search UI Chapter 3 The following topics describe search configuration and customization for WDK based applications: Configuration and customization points, page 33 Understanding search in WDK, page 35 Configuring search controls, page 37 Fulltext and attributes only searches, page 38 Configuring the basic search 6.0 component, page 38 Configuring the advanced search component, page 39 Configuring the savesearch component, page 42 Configuring search results and tuning performance, page 43 Making custom attributes available in search results, page 44 Configuring Webtop extended search clustering, page 45 Customizing search in Webtop applications, page 46 Programmatic search value assistance, page 55 Configuration and customization points Figure 2, page 34 shows points at which you can configure or customize search component presentation and behavior in Webtop applications. EMC Documentum Version 6.5 Search Development Guide 33

34 Configuring and Customizing the Webtop Search UI Figure 2. Configuration and customization points in Webtop applications Key 1. Refer to Configuring search controls, page 37, Configuring the basic search 6.0 component, page 38, Configuring the advanced search component, page 39, Configuring search results and tuning performance, page 43, and Making custom attributes available in search results, page EMC Documentum Version 6.5 Search Development Guide

35 Configuring and Customizing the Webtop Search UI 2. Refer to Customizing search in Webtop applications, page 46 and Generating queries in DFC, page Refer to Creating DFC search applications, page Refer to Using a DQL hints file, page Refer to Testing, logging, and debugging queries, page 79. Understanding search in WDK Following is a brief general description of the WDK customization model. Information on individual search controls and components is contained in the comprehensive reference guide, Web Development Kit and Webtop Reference Guide. WDK and Webtop search components can search multiple repositories and external sources. The search components are versioned, so if a request is made for a search component, the new component is returned by default. If you customized a previous version of a Webtop search component and extended it, and the version is still supported, your customization will be used in place of the new search components. Search sources Multiple repositories can be added to the user s search preferences. If ECI Services is installed, the user can select external sources for search and import results into the current repository. Included files within HTML or XML documents are not imported. Search on attribute values All attributes are indexed, so a query for attribute criteria is run against the fulltext index by default. The attributes for search criteria are supplied by the data dictionary of the selected repository. If value assistance is defined in the data dictionary, the values are supplied for ʺisʺ and ʺis notʺ search criteria. Verity operators such as ʺnotʺ or ʺbetweenʺ are not supported.. Tip: You can specify in the data dictionary the default operator to be used for an attribute, such as ʺisʺ or ʺis notʺ. Saving searches Searches are saved as smartlist objects. Saved searches save the display configuration as well as the query, and the user has the option of saving query results with the query. Users can revise a saved search using the advanced search component. Smartlists created with Documentum Desktop can be executed or edited in the advanced search UI but will no longer be usable in Desktop. Smartlists that are created in WDK applications cannot be used or edited in Desktop. Fulltext search Simple and advanced searches query the fulltext index by default. You can run a fulltext query in advanced search using the Contains field. The the Contains field or the simple search text box can contain a string within quotations marks to search for the exact string, for example, ʺthis stringʺ. The box also supports the operators AND and OR operators. The following rules apply: EMC Documentum Version 6.5 Search Development Guide 35

36 Configuring and Customizing the Webtop Search UI Either operator may be appended with NOT. The operators are not case sensitive. Punctuation, accents, and other special characters are ignored (replaced with a space). The AND operator has priority over the OR operator. For example, if you type knowledge AND management OR discovery, the results will contain both knowledge and management or that contain discovery. Parentheses override the priority of operators. For example, if you type knowledge AND (management OR discovery), the results must contain knowledge and must also contain either management or discovery. The NOT operator cannot be used to qualify an expression within parentheses, for example, NOT (a and b), but it can be used within parentheses, for example a OR (b and NOT c). If no operators are used between words, multiple words are treated with the ACCRUE operator, which is an OR operator with a result ranking that gives a higher score to results that contain all words. The NOT operator is processed differently. The wildcard operator * allows 0 or more characters added to the string in the text box, for indexed repositories only, and the operator? allows a single character. Value assistance Value assistance as defined within a doc app is supported. The assistance within the doc app should provide a union of values for a type across lifecycles. For information on supporting conditional value assistance in JSP pages, refer to Providing conditional value assistance, page 40. Limitations: Not all values in value assistance may be available across repositories in a logical OR operation. (This is not a limitation for the AND operation.) Locale based assistance values are not available in a search of repositories on multiple locales unless the assistance values are present in each locale s data dictionary. Presets for search on object type In the Webtop presets editor, you can create a preset that limits the searchable object types. This preset will override the <includetypes> setting in the advanced search component definition. Webtop Extended Search Webtop extended search can be installed on a Content Server to provide search results clustering, search templates, and search monitoring. The clustering and search monitoring DAR file must be deployed to a global registry repository of version 6.x. The search templates DAR file must be deployed to each repository (version 5.3.x or 6.x) in which you wish to store search templates. Use Documentum Composer to deploy these DAR files to the repositories. Instructions for deploying the Webtop Extended Search DocApps are in the Web Development Kit and Webtop Deployment Guide. 36 EMC Documentum Version 6.5 Search Development Guide

37 Configuring and Customizing the Webtop Search UI Configuring search controls Refer to Web Development Kit and Webtop Reference Guide for details on each control s configuration. You can globally configure all instances of certain advanced search controls by modifying the control configuration definitions on wdk/config/advsearchex.xml. The following controls can be configured: searchattribute controls, match case attribute (does not apply to searches of the index) searchsizeattribute control searchdateattributecontrol search clusters The following example changes the size range dropdown selections. It modifies advsearchex.xml in a modification file located in custom/config with the following content: <config version='1.0'> <scope type='dm_sysobject'> <searchsizeattributerange modifies="searchsizeattributerange: wdk/config/advsearchex.xml"> <insert> <option> <label>any old size</label> <operator>lt</operator> <value> 1</value> <unit>kb</unit> </option> </insert> </searchsizeattributerange> </scope> </config> The resulting UI shows the new values for size attribute range: Figure 3. Search size custom drop down list Case sensitivity Search on fulltext strings or attributes against an repository is not case sensitive. If the repository is not indexed, queries are case sensitive by default. Case sensitivity for non indexed repositories can be turned on or off in wdk/config/advsearchex.xml, as the value of the <defaultmatchcase> element. If you turn off case sensitivity, create functional indexes on the attributes that will be queried. EMC Documentum Version 6.5 Search Development Guide 37

38 Configuring and Customizing the Webtop Search UI If you use a ENABLE(NOFTDQL) hint in a DQL hints file to send attribute queries against the database rather than the fulltext index, your query will be case sensitive if you set the value of <defaultmatchcase> to true. For better performance, set case sensitivity to true, or set it to false and create a functional index on the queried attribute columns. Fulltext and attributes only searches The default search is for a string query type in a fulltext search. If the Content Server is indexed, the query is performed against the fulltext index including all metadata. If the Content Server is not configured to create a fulltext index, the query is transformed into constraints against object_name, title, and subject with an OR operator. If you wish the query to include attributes, those attributes must be indexed. Users can search attributes only and not fulltext in the following ways: (Advanced search only) Add a checkbox for Include recently modified properties on the advanced search page. Attributes will be queried against the metadata and not the index. To do this, uncomment the following lines your custom advanced search JSP page (a copy of the webcomponent advanced search JSP page): <! <tr class="leftalignment" valign=top> <td class="leftalignment" valign=top nowrap> <dmfxs:searchscopecheckbox name='<%=advsearchex.database_search_scopecheckbox_control%>' scopename='<%=repositorysearch.database_search_property%>' checkedvalue='true' uncheckedvalue='false' nlsid='msg_database_search' tooltipnlsid="msg_database_search_tip"/> </td> </tr> > Use the dql query type for a custom search component and pass the query string in the query parameter. Turn off FTDQL (queries against index) using a DQL hints file. You can disable index queries for attributes without affecting the fulltext string portion of a query. For more information, refer to Using a DQL hints file, page 72. Set dfc.search.fulltext.enable to false in the application s dfc.properties, which is located in WEB INF/classes. Configuring the basic search 6.0 component Basic search searches all sysobjects in the current repository for the user supplied string in the fulltext index of content and attributes. The default base type for the search can be configured in the search component definition. The default preferred sources can also be specified in the component definition. These sources can include multiple repositories and external sources if ECI Services is installed. 38 EMC Documentum Version 6.5 Search Development Guide

39 Configuring and Customizing the Webtop Search UI Note: The list of object types and their attributes comes from the reference repository. The reference repository is the first repository selected by the user. If external sources only are selected, then the list of object types in the current repository is used. Configuring the advanced search component The data dictionary provides the following data to the search UI: The default and other searchable attributes for a given object type The list of searchable types, which are then filtered by the presets or configuration file The default and other search operators for a given type and attribute Value assistance values for ʺ=ʺ and ʺ< >ʺ search operations, if defined in the data dictionary The WDK search UI contains search controls. To make changes to control attribute values, you must extend a search component and modify your custom search JSP page. Example 3 1. Setting the search type drop down list The available search types list is configured by the includetypes element in the advsearch component definition. The includetypes list is comma delimited. The descend attribute specifies whether subtypes or included or not. Create your modification definition in custom/config. The following example will display dm_folder and all of its subtypes including custom types that subtype dm_folder: <component modifies="advsearch:webcomponent/config/library/search/searchex/ advsearch_component.xml"> <replace path="includetypes"> <includetypes descend="true">dm_folder</includetypes>... </replace></component> Figure 4, page 40 shows the type selection list set by includetypes with descend set to true: EMC Documentum Version 6.5 Search Development Guide 39

40 Configuring and Customizing the Webtop Search UI Figure 4. Limiting the selectable types and subtypes The following example will display only two selections, because the descend parameter is set to false: <includetypes descend="false"> dm_folder, my_type </includetypes> Figure 5, page 40 shows the type selection list set by includetypes with descend set to false: Figure 5. Limiting the selectable types without subtypes Example 3 2. Providing conditional value assistance Use individual searchattribute control tags to provide conditional value assistance. The default value assistance must have no dependency on another attribute. Conditional value assistance depends on the display order of the constraints in the JSP page, so you must display the controls in the dependency 40 EMC Documentum Version 6.5 Search Development Guide

41 Configuring and Customizing the Webtop Search UI order. The searchattributegroup tag provides only simple attribute assistance unless the constraints are entered in the correct order. The lists of conditional values are set in Documentum Application Builder. Query value assistance can use a reference ($value(attribute)), for example: SELECT "MyDocbase"."MyTable"."MyColumn1" FROM "MyDocbase"."MyTable" WHERE "MyDocbase"."MyTable"."MyColumn2" = '$value(myattribute)' The following example lists four attributes, three of which have conditional value assistance lists that were set up in Documentum Application Builder. The drop down list for Make determines the list available for Model. The drop down lists Fuel and Year both depend on Model. Figure 6. Conditional value assistance UI This UI was generated from the following set of controls in the JSP page: <tr> <td>make:</td> <td><dmfxs:searchattribute name='make' attribute="make"/></td> </tr> <tr> <td>model:</td> <td><dmfxs:searchattribute name='model' attribute="model"/></td> </tr> <tr> <td>year:</td> <td><dmfxs:searchattribute name='year' attribute="year"/><td> </tr> <tr> <td>fuel:</td> <td><dmfxs:searchattribute name='fuel' attribute="fuel"/></td> </tr> Example 3 3. Enabling the wildcard CONTAINS operator for string property searches The Content Server version 6 treats the CONTAINS operator as CONTAINS WORD to avoid wildcard query timeouts against the fulltext index. If the user searches on a string value in a property on the advanced search page, for example, ʺwdk, this string is treated as a string ʺ wdk ʺ (note the spaces), so that documents named ʺwdkindex or ʺsearch_wdk would not be returned, but documents named ʺwdk treats or ʺAbout WDK would be returned. Most queries that users make are for whole words, not parts of words. EMC Documentum Version 6.5 Search Development Guide 41

42 Configuring and Customizing the Webtop Search UI If grammatical normalization was enabled when the fulltext index server was installed, additional words will be returned. For example, if the query term is ʺtest, the results could contain ʺtested, ʺtester, and ʺtest but not ʺtestreport. You can change the behavior of the CONTAINS operator behavior by enabling the searchscope checkbox in the advanced search JSP page. This checkbox can serve the following purposes: Retrieve objects with recently modified properties that have not yet been indexed Perform case sensitive queries against the database Note: You can also force queries against the database globally in dfc.properties (set dfc.search.fulltext.enable to false) or with thedql hint ft_contain_fragment Revert search behavior to CONTAINS Note: The checkbox is not documented in the user help because it is not enabled. To enable the checkbox, remove the JSP comment tags around tine following tag in the advancedsearchex.jsp page: <! <tr class="leftalignment" valign=top> <td class="leftalignment" valign=top nowrap> <dmfxs:searchscopecheckbox name='<%=advsearchex.database_search_scopecheckbox_control%>' scopename='<%=repositorysearch.database_search_property%>' checkedvalue='true' uncheckedvalue='false' nlsid='msg_database_search' tooltipnlsid="msg_database_search_tip"/> </td> </tr> > Configuring the savesearch component The savesearch component displays checkboxes that allow the user to save search results with a search and to make the saved search public. These two features can be removed by setting the value of the configuration element <enablesavingsearchresults> to false. The following example in a modification file removes these two checkboxes: <component modifies=" savesearch:webcomponent/config/library/savesearch/savesearchex/savesearch_component.xml"> <replace path="enablesavingsearchresults"> <enablesavingsearchresults>false</enablesavingsearchresults> </replace></component>... The default for whether to save results with a search is set by the configuration element <includeresults>. 42 EMC Documentum Version 6.5 Search Development Guide

43 Configuring and Customizing the Webtop Search UI Configuring search results and tuning performance You can configure the maximum number of search results and turn off term hit highlighting. After you have made custom types and their attributes available for search, you can configure the display of custom attributes in the search results. You can configure the display_preferences component to allow users to configure their preferences for displaying custom attributes. Maximum number of search results The maximum number of search results, globally and per source, is configured in dfc.properties. The maximum number of search results is specified as the value of dfc.search.maxresults (was maxresults_per_source in 5.3.x). The maximum number of results per source is specified as the value of dfc.search.max_results_per_source. For example, if you have specified a maximum of 1000 results and a maximum per source of 500, results will be accumulated from each source until the source maximum of 500 is reached or until the global maximum of 1000 is reached. Note: These settings can affect performance. Setting the value too high can overload the index server, and setting it too low can frustrate users. You must evaluate the best settings for your environment. Term hit highlighting Term hit highlighting (highlighting of the search term in the results) can be set as a user preference. The default value is set as the value of the element <highlight_matching_terms> in the search component definition, which is located in webcomponent/config/library/search/searchex. If you are customizing Webtop or an application that extends Webtop, you must add a <highlight_matching_terms> element to the top level search component definition. Folder path display is disabled for faster performance The display of the results folder path is turned off in order to enhance query performance. The value of <displayresultspath> in webcomponent/config/library/search/searchex/search60_component.xml is set to false. Turning off summary calculation for faster performance The summary column is calculated, which can add to query overhead. Turn off the summary column by extending the Webtop search component searchex_component.xml, which is located in webtop/config. Copy the <columns_xxx> elements (<columns_drilldown>, <columns_list>, and <columns_saved_search>) from the parent configuration file search60_component.xml in webcomponent/config/libarary/search/searchex. In each of the <columns> elements, set the value of <column>.<attribute>.<visible> for the summary attribute to false. Set the value of <columns_xxx>.<loadinvisibleattribute> to false to ensure that the column is not calculated. Configuring the display of attributes in search results Default search result columns are configured as <column> elements in the basic search configuration file search60_component.xml in webcomponent/config/library/search/searchex. Only attributes marked as searchable in the data dictionary can be specified as columns. Users can set a preference for search results columns in the display_preferences component, which will then override the default settings in the configuration file. EMC Documentum Version 6.5 Search Development Guide 43

44 Configuring and Customizing the Webtop Search UI Your custom search component definition must specify a scope for the custom type in order to define default visible columns for custom attributes. For example, if the user selects a custom type for the advanced search, the columns specified in your scoped basic search component will be displayed in the results. Details of the columns configuration can be found in Web Development Kit Reference Guide In the following simple configuration, the definition extends the WDK search component definition and adds some custom attribute columns: <config version='1.0'> <scope type='technical_publications_web'> <component modifies=" search:webcomponent/config/library/search/searchex/search60_component.xml"> <insert path='columns_list'> <column> <attribute>tp_edition</attribute> <label>edition</label> <visible>true</visible> </column> <column> <attribute>tp_web_viewable</attribute> <label>ok to display</label> <visible>true</visible> </column> </insert> </component> </scope> </config> Making custom attributes available in search results The user can select attributes for display in search results, which overrides the default display. The preferences UI allows users to specify the attributes that will be displayed for specific object types. If the user configures different display columns, the query will not be reissued, so data may not be displayed in the new columns until the search is performed again. For example, calculated columns such as score or summary will not display any values unless they are selected before the query is run. You must modify the definition for the display_preferences component to make columns of your custom type available to users for display. To make a custom type available in preferences 1. Modify the display_preferences component in your custom/config directory: : <component modifies=" display_preferences:webtop/config/display_preferences_ex_component.xml"> 2. Add your custom type to the <display_docbase_types> element. For example: <insert path='preferences.display_docbase_types'> <docbase_type> 44 EMC Documentum Version 6.5 Search Development Guide

45 Configuring and Customizing the Webtop Search UI <value>my_custom_type</value> <label>my type</label> </docbase_type> </insert> 3. Save this file and refresh the configuration files on the application server by navigating to wdk/refresh.jsp. To make a calculated attribute available in search results 1. Extend the Search60 class in the package com.documentum.webtop.webcomponent.search. 2. Override the initattributes method and add your computed attribute. The following example adds ʺmyComputed attribute: protected void initattributes() { List<String> mandatoryattrs = getattributesmanager().getmandatory(); mandatoryattrs.add("mycomputed"); getattributesmanager().setmandatory(mandatoryattrs); super.initattributes(); } 3. Extend the search component definition to use your custom class, and scope it to your custom type. Set the class to use the custom class. Configuring Webtop extended search clustering The Webtop extended search clustering DAR file can be installed in the global registry to support clustering of search results in groups based on their attribute values. You define the strategies including default strategies in clusterstrategies_config.xml, which is located in the wdk/config of the WDK based application. Each cluster strategy is defined by a <clusterstrategy> element, which contains one or more attributes specified as the value of the <criterion> child element. The display is governed by the <clustertree> element. Its child elements <primary> and <secondary> have values that correspond to the IDs of strategies.. Tokenizers split attribute strings into chunks that are then used as clusters. Only one tokenizer is associated with an attribute. The default tokenizer is text, and other tokenizers are defined to tokenize on number, author and date. Tokenizers are part of the clustering SBO. Using the WDK configuration syntax, you can add, remove, or change a strategy definition or add, remove, or change which strategies are displayed in the default cluster tree. Users can change these defaults in their search preferences. To add a strategy definition 1. Create a file clusterstrategies_modifications.xml in custom/config. EMC Documentum Version 6.5 Search Development Guide 45

46 Configuring and Customizing the Webtop Search UI 2. Add the opening and closing declarations: <?xml version="1.0" encoding="utf 8" standalone="no"?> <config> <scope> </scope> </config> 3. Within the <scope> element, add the following element that specifies the primary element you are modifying and the file in which it exists: <clusterstrategies modifies="clusterstrategies:wdk/config/clusterstrategies_config.xml"> </clusterstrategies> 4. Within the <clusterstrategies> element, insert the new strategy that will cluster results for a certain attribute. This example creates a cluster for the keywords attribute: <insert> <clusterstrategy id="keywords" nlsid="msg_keywords" icon="cluster/ranking.gif" threshold="5"> <criterion>keywords</criterion> </clusterstrategy> </insert> Note that if you provide an nlsid value, you must have a corresponding string in clusterstrategiesnlsprop.properties. The icon path is relative to the theme folder icons/browsertree directory in the application. The threshold specifies the minimum number of documents for which to display the cluster. 5. You must refresh the configurations in memory by navigating to wdk/refresh.jsp or restart the application server. To display a new strategy in the default cluster tree 1. In the modifications file you created that contains the new strategy, add the following child element to <scope> (sibling to <clusterstrategy>): <clustertreegroup modifies="clustertreegroup:wdk/config/clusterstrategies_config.xml"> <insert> <clustertree> <primary>keywords</primary> <secondary>topic</secondary> </clustertree> </insert> Customizing search in Webtop applications The following topics describe some customizations to Webtop search components. 46 EMC Documentum Version 6.5 Search Development Guide

47 Configuring and Customizing the Webtop Search UI Modifying the search JSP pages Changes to JSP pages are considered to be customizations. The following examples extend Webtop search component definitions and specify a custom JSP page in which to make customizations. Example 3 4. Setting the default search type To set the default search type, supply your preferred type in the JavaScript function that calls the advanced search container. In Webtop, this is the file titlebar.jsp, so extend the titlebar component and provide the following postcomponentnestevent calls in the onclickadvancedsearch JavaScript function. Substitute your custom type (in quotation marks) for custom_type: postcomponentnestevent(null, "advsearchcontainer", "content", "component", " advsearch", "type", custom_type, "usepreviousinput", "false", "query", strvalue);... postcomponentnestevent(null, "advsearchcontainer","content","component"," advsearch", "type", custom_type, "usepreviousinput", "true"); Example 3 5. Displaying specific attributes for search You can specify attributes for your search rather than allowing them to be generated by the searchattributegroup control. In the following example of a custom advsearch component, specific attribute controls have replaced the searchattributegroup control in the JSP page:... <dmfxs:searchobjecttypedropdownlist name='objecttypectrl'.../> </td> </tr> <tr><td colspan='2' class='spacer' height='10'> </td></tr> <tr> <td align=right valign=top nowrap><dmf:label label='name' cssclass=" fieldlabel"/></td> <td align=left valign=top nowrap> <dmfxs:searchattribute name='searchname' attribute='object_name' andorvisible="false" removable="false"> </dmfxs:searchattribute> </td> </tr> <tr> <td align=right valign=top nowrap><dmf:label label='type' cssclass=" fieldlabel"/></td> <td align=left valign=top nowrap> <dmfxs:searchattribute name='searchtype' attribute='r_object_type' andorvisible="false" removable="false"> </dmfxs:searchattribute> </td> </tr>... Note: Set the andorvisible and removable attributes to false on the searchattribute control. Before this customization, the user must select properties from a dropdown: EMC Documentum Version 6.5 Search Development Guide 47

48 Configuring and Customizing the Webtop Search UI Figure 7. Attribute selection drop down After customization, the UI shows the individual attributes ʺNameʺ and ʺTypeʺ as search criteria: Figure 8. Specific attributes as search criteria Example 3 6. Displaying custom attributes as search criteria To display specific custom attributes as individual search criteria, extend the advanced search component, scope the definition to your custom type, and provide a custom JSP page. In that page, add attribute controls for your attributes. When the user selects the custom type, the scoped definition will be read by the configuration service and the custom JSP page will be displayed. The custom attributes will be displayed similar to the following: 48 EMC Documentum Version 6.5 Search Development Guide

49 Configuring and Customizing the Webtop Search UI Figure 9. Custom attributes as search criteria Example 3 7. Enabling search on string fragments The Content Server 6 treats the CONTAINS operator as CONTAINS (WORD) to avoid wildcard query timeouts against the fulltext index. Following are two examples of the old CONTAINS and the new CONTAINS (WORD) queries: If the user searches on a property string in the advanced search page, such as ʺwdk in the object name, this string is treated as a word with spaces before and after: ʺ wdk ʺ. The documents named ʺwdkindex or ʺsearch_wdk would not be returned, but the documents named ʺwdk treats or ʺAbout WDK would be returned. If the user searches on a property string such as ʺran in the old CONTAINS mode, ʺRestaurants.txt, ʺInsurance.pdf, ʺRan by Kurosawa.doc or On the Run.pdf would be returned. (Note that the last example requires lemmatization configured on the index server.) In the new CONTAINS (WORD) mode, only the last two examples would be returned. Most queries that users issue are for whole words, not parts of words. With the new CONTAINS (WORD) operator, the query will return the same results for string properties entered into the fulltext search box or a property box. With the old CONTAINS string fragment search, lemmatization contributed to timeout errors by multiplying the number of results, but with CONTAINS (WORD), lemmatization becomes helpful. There are four ways to revert to the wildcard CONTAINS search for a string fragment: Globally Add fds_contain_fragment to the parameter list in dm_ftengine_config object and set it to true. For certain query conditions Using the DQL hint ft_contain_fragment to a DQL hints file. EMC Documentum Version 6.5 Search Development Guide 49

50 Configuring and Customizing the Webtop Search UI In the Webtop UI Uncomment the checkbox in the advanced search JSP page advsearchex.jsp located in webcomponent/library/advancedsearch: <! <tr class="leftalignment" valign=top> <td class="leftalignment" valign=top nowrap> <dmfxs:searchscopecheckbox name='<%=advsearchex.database_search_scopecheckbox_control %>' scopename='<%=repositorysearch.database_search_property%>' checkedvalue='true' uncheckedvalue='false' nlsid='msg_database_search' tooltipnlsid="msg_database_search_tip" /> </td> </tr> > In the Webtop UI Use a wildcard (* or?). For example, if the user enters ʺ*ran for object name, the query would return ʺrestaurants or ʺinsurance. Example 3 8. Removing fulltext search from the UI This example modifies the advanced search component definition to display a custom page with specific attributes for search. To remove the fulltext search or other portions of the UI, modify the search component definition that contains the JSP page specification. The top layer search component is in webtop/config, but that definition does not contain a JSP <pages> element. It extends the webcomponent advanced search definition, which does contain a <pages> element, so you must modify that definition. <component modifies="advsearch:webcomponent/config/library/search/searchex/ advsearch_component.xml"> Modify the <pages> element to specify a custom JSP directory, similar to the following: <replace path='pages.start'> <start>/custom/advancedsearch/advsearchex.jsp</start> </replace> Copy the Webtop advanced search JSP page advsearchex.jsp from webcomponent/library/ advancedsearch to custom/advancedsearch and remove the table row that contains the fulltext search control: <tr> <td scope='row' class="rightalignment" valign=top nowrap> <dmf:label nlsid='msg_containing_words' cssclass="fieldlabel"/></td> <td class="leftalignment" valign=top nowrap> <dmfxs:searchfulltext name='fulltextctrl' size="49"/></td></tr> To remove the fulltext search box for a certain object type or role, scope the definition to the desired condition. In the following example, a custom JSP page displays no fulltext search box when the custom type is selected. This requires an advanced search component definition that is scoped to the custom type and that references the custom JSP page. Figure 10, page 51 displays the results when the fulltext search box is removed from the advanced search page: 50 EMC Documentum Version 6.5 Search Development Guide

51 Configuring and Customizing the Webtop Search UI Figure 10. Fulltext search box removed from UI Modifying the search component query The following methods in the basic search component class Search60 provide customization points: initsearch(arg): Override to modify queries before execution initcontrols(arg): Override to update custom controls initattributes(): Override to perform specific treatment for columns. Use getattributesmanager() to manipulate columns and query attributes initresultsset(): Override to manipulate the results that are fed to the datagrid initsearchexecution(): Start the actual query execution You can access a query before it is submitted and modify it in various ways. The query is accessible by overriding the initsearch() method of the Search60 class. Your custom class should extend the Webtop version of either the Search60 or AdvSearchEx component class. Note: Custom modifications to the query will be visible to the users when they edit the query after running it. Example 3 9. Adding a WHERE clause to simple search To add a WHERE clause to the query in simple search, extend Search60 in the package com.documentum.webtop.webcomponent.search. You can add criteria other than keywords to the initsearch method. If you override buildquery, you may break smartlist usage. The following example adds an AND clause to the query that requires a specific string in the name of the object, in addition to any criteria entered into the simple search text box by the user. First, create your extended search component definition in custom/config as follows: <?xml version="1.0" encoding="utf 8" standalone="no"?> <config version='1.0'> <scope> <component modifies="search:webtop/config/search60_component.xml"> <replace path='class'> <class>com.mycompany.searchex</class> </replace> </component> EMC Documentum Version 6.5 Search Development Guide 51

52 Configuring and Customizing the Webtop Search UI </scope> </config> Next, create your custom class that extends Search60 and overrides initsearch(): package com.mycompany; import com.documentum.fc.client.search.idfexpressionset; import com.documentum.fc.client.search.idfquerybuilder; import com.documentum.fc.client.search.idfsimpleattrexpression; import com.documentum.fc.common.idfvalue; import com.documentum.web.common.argumentlist; import com.documentum.webcomponent.library.search.searchinfo; public class SearchEx extends com.documentum.webtop.webcomponent.search.search60 { protected void initsearch (ArgumentList args) { super.initsearch(args); String querytype = args.get(arg_query_type); if ((querytype == null) (querytype.length() == 0) (querytype.equals("string"))) { SearchInfo info = getsearchinfo(); IDfQueryBuilder qb = info.getquerybuilder(); IDfExpressionSet rootset = qb.getrootexpressionset(); IDfExpressionSet setand = rootset.addexpressionset (IDfExpressionSet.LOGICAL_OP_AND); setand.addsimpleattrexpression("r_modifier", IDfValue.DF_STRING, IDfSimpleAttrExpression.SEARCH_OP_CONTAINS, true, false, "tuser"); } } } This example adds an AND criterion in which the modifier attribute must contain the user name ʺtuserʺ. Before the customization, a search on the string ʺTargetʺ in the simple search box returns 3 results as shown Figure 11, page 52: Figure 11. Simple search without added clause After customization, only a single result in which the object name contains ʺTargetʺ and the user name contains ʺtuserʺ returned. (User name is displayed in the second column, as ʺModifier.ʺ) 52 EMC Documentum Version 6.5 Search Development Guide

53 Configuring and Customizing the Webtop Search UI Figure 12. Simple search with added clause With IDfExpressionSet, you can add the following operators: LOGICAL_OP_AND, LOGICAL_OP_DEFAULT (default operator in data dictionary), and LOGICAL_OP_OR. The following expressions, also called predicates, are available for IDfSimpleAttrExpression (names are self explanatory): SEARCH_OP_BEGINS_WITH SEARCH_OP_CONTAINS SEARCH_OP_DOES_NOT_CONTAIN SEARCH_OP_ENDS_WITH SEARCH_OP_EQUAL SEARCH_OP_GREATER_EQUAL SEARCH_OP_GREATER_THAN SEARCH_OP_IS_NOT_NULL SEARCH_OP_IS_NULL SEARCH_OP_LESS_EQUAL SEARCH_OP_LESS_THAN SEARCH_OP_NOT_EQUAL The following expression is available for IDfValueRangeAttrExpression: SEARCH_OP_BETWEEN The following expressions can be used with IDfValueListAttrExpression: SEARCH_OP_IN SEARCH_OP_NOT_IN Example Adding a WHERE clause to advanced search In advanced search, you override buildquery to access the user s query. The search class is as follows: package com.mycompany; import com.documentum.fc.common.idfvalue; import com.documentum.fc.client.search.idfsimpleattrexpression; import com.documentum.fc.client.search.idfexpressionset; import com.documentum.fc.client.search.idfquerybuilder; public class AdvSearchEx extends com.documentum.webtop.webcomponent.advsearch.advsearchex { protected IDfQueryBuilder buildquery() throws Exception EMC Documentum Version 6.5 Search Development Guide 53

54 Configuring and Customizing the Webtop Search UI { IDfQueryBuilder qb = super.buildquery(); IDfExpressionSet rootset = qb.getrootexpressionset(); IDfExpressionSet setand = rootset.addexpressionset (IDfExpressionSet.LOGICAL_OP_AND); setand.addsimpleattrexpression("object_name", IDfValue.DF_STRING, IDfSimpleAttrExpression.SEARCH_OP_CONTAINS, true, false, "xpath"); return qb; } } Example Changing the query source You can change the location, including the source and folder path in the repository with query builder APIs. The following example adds a source repository to IDfQueryBuilder instance and sets a path within the repository for the query. The examples for basic and advanced search show you how to get the query builder instance (variable qb in this example): qb.clearselectedsources(); qb.addselectedsource("dm_notes"); // set source, path, descend flag qb.addlocationscope("dm_notes", "/Temp", false); The resulting query is similar to the following: SELECT r_object_id,text,object_name,from dm_document SEARCH DOCUMENT CONTAINS testing WHERE (object_name LIKE %testing% ESCAPE \) AND FOLDER(/Temp) AND (a_is_hidden = FALSE) Hiding the customization from query editing If you have intercepted and modified a query after form submit, the hidden query processing will be displayed when the user tries to modify the query. To hide the custom modification, add the userpreviousinput parameter in the call to the advanced search component. Modify the titlebar component definition to use your own titlebar.jsp page as follows: <component modifies="titlebar:webtop/config/titlebar_component.xml"> <replace path="pages.start"> <start>/custom/titlebar/titlebar.jsp</start> </replace></component> In your custom titlebar JSP page, change the call to the advanced search component to set usepreviousinput to false: postcomponentnestevent(null, "advsearchcontainer","content","advsearch", "type", "dm_sysobject", "usepreviousinput", "false")' 54 EMC Documentum Version 6.5 Search Development Guide

55 Configuring and Customizing the Webtop Search UI Programmatic search value assistance Data dictionary value assistance is available in advanced search. If you have not defined value assistance for an attribute in the repository data dictionary, you can add value assistance programmatically. You must define a custom tag handler to render the value assistance values. The tag handler is specified in the search configuration file advsearchex.xml as follows: <searchvalueassistance> <attribute_type_name> fully_qualified_class_name </attribute_type_name> </searchvalueassistance> When the user selects an attribute for search, the values in the criteria dropdownlist control will be populated by the custom tag class. To add your own custom tag class, copy the file wdk/advsearchex.xml to custom/config and add your handlers to the <searchvalueassistance> element. Your tag handler must implement ISearchAttributeValueTag. Note: Do not delete the Documentum value assistance handlers, because the entire contents of the <searchvalueassistance> will override the contents of the element in the WDK version of this file. The following tag handlers render values for certain attributes. The handler classes are in com.documentum.web.formext.control.docbase.search. BooleanVATag Provides values for any Boolean attribute ContentTypeVATag Provides valid a_content_type (dm_format) names and descriptions ExistingValueVATag Uncomment this tag and specify an attribute for which to populate the drop down list with all existing values for the selected object type ObjectTypeVATag Populates the search object type drop down list with available object types PermissionVATag Provides possible permission values (none, browse, read, relate, version, write, delete) for setting world_permit, group_permit, and owner_permit attributes SearchMetaDataVATag Gets attribute names, default value, and description for each attribute. This handler is for internal use only. Your tag class should extend the abstract class SearchVADropDownListTag and implement ISearchAttributeValueTag. For example, the BooleanVATag class implements populatevaluedropdownlist to provide the two boolean values: protected void populatevaluedropdownlist(searchdropdownlist ddlist) EMC Documentum Version 6.5 Search Development Guide 55

56 Configuring and Customizing the Webtop Search UI { } Option optiontrue = new Option(); optiontrue.setvalue("1"); optiontrue.setlabel(searchcontrol.getstring("msg_true", ddlist)); ddlist.addoption(optiontrue); Option optionfalse = new Option(); optionfalse.setvalue("0"); optionfalse.setlabel(searchcontrol.getstring("msg_false", ddlist)); ddlist.addoption(optionfalse); 56 EMC Documentum Version 6.5 Search Development Guide

57 Customizing Search with DFC Chapter 4 DFC search APIs are briefly described below. Use these APIs to customize search component behavior within WDK client applications. The following topics describe search support in DFC: Configuring search in dfc.properties, page 57 Configuring search relevancy, page 59 Generating queries in DFC, page 62 Creating DFC search applications, page 65 Using the visitor API, page 70 Using a DQL hints file, page 72 Optimizing query batch size, page 78 Configuring search in dfc.properties The following options in dfc.properties configure search behavior in DFC and DFC clients such as WDK and Webtop. This file is located in the Documentum home config directory as specified by the dfc_data environment variable, for example, C:\Documentum\config or /tmp/documentum/config. This file includes settings to enable ECIS and provide the ECIS host and password, a keystore and keystore password for ECIS authentication, and the backup ECIS server for failover. # Searching Options # ================= # DFC enables you to use Enterprise Content Integration Services (ECIS) to # search for information in Documentum and other repositories. Refer to the ECIS # documentation for information about the following properties. Values after the # equal signs are default values. # # The following properties control the way ECIS uses brokers, which are threads # in the DFC process space that pull queries from a common queue and execute # them. # # Number of broker threads supporting execution of the Documentum repository EMC Documentum Version 6.5 Search Development Guide 57

58 Customizing Search with DFC # part of a query. One broker supports execution of the query for each # repository selected for this query. # min value: 0, max value: 1000 dfc.search.docbase.broker_count = 20 # Subdomain containing the source available to DFC. By default, DFC uses the # default domain of the standalone ECI WEB client dfc.search.ecis.adapter.domain = JSP # RMI registry host to connect to the backup ECI Server. The ECIS Development # Guide chapter on the application SDK explains the RMI registry. dfc.search.ecis.backup.host = localhost # RMI registry port to connect to the backup ECI Server. The ECIS Development # Guide chapter on the application SDK explains the RMI registry. # min value: 0, max value: dfc.search.ecis.backup.port = 3005 # Number of broker threads supporting execution of the ECIS part of a query. One # broker supports the execution of the query for all external sources selected # for this query. # min value: 0, max value: 1000 dfc.search.ecis.broker_count = 30 # Use ECIS. Setting this to True tells DFC to use ECIS in addition to Content # Server's basic search facilities. dfc.search.ecis.enable = false # RMI registry host to connect to ECI Server. The ECIS Development Guide chapter # on the application SDK explains the RMI registry. dfc.search.ecis.host = localhost # Default credentials to connect to ECI server as guest. dfc.search.ecis.password = askonce # RMI registry port to connect to ECI Server. The ECIS Development Guide chapter # on the application SDK explains the RMI registry. # min value: 0, max value: dfc.search.ecis.port = 3005 # Time in milliseconds to wait for answer from ECIS server. # min value: 0, max value: dfc.search.ecis.request_timeout = # Time in milliseconds before retrying to connect to the main ECIS server (after # having switch to the backup ECIS server). # min value: 0, max value: dfc.search.ecis.retry.period = # RMI Registry symbolic name associated with ECIS API. dfc.search.ecis.rmi_name = xtrim.rmiapi # Enable encryption of results and content sent from the ECI server to the DFC # client. dfc.search.ecis.ssl.enable = false # Define a keystore where to find DFC client certificate and keys and ECI Server 58 EMC Documentum Version 6.5 Search Development Guide

59 Customizing Search with DFC # trusted certificate.this keystore is a file available locally on the machine # where the DFC resides. dfc.search.ecis.ssl.keystore = # Define the password for the keystore file used for communication with the ECIS # server. dfc.search.ecis.ssl.keystore_password = # Default credentials to connect to ECI server as guest. dfc.search.ecis.username = guest # Time in milliseconds between refreshes of the cache of docbase formats. # min value: 0, max value: dfc.search.formatcache.refresh_interval = # Use the Content Server fulltext engine (for example, FAST). If you set this # to False, DFC replaces DQL fulltext clauses by LIKE clauses on the following # attributes: object_name, title, subject. dfc.search.fulltext.enable = true # Maximum number of results to retrieve by a query search. # min value: 1, max value: dfc.search.max_results = 1000 # Maximum number of results to retrieve per source by a query search. # min value: 1, max value: dfc.search.max_results_per_source = 350 # Time in milliseconds between refreshes of the search source map cache. # min value: 0, max value: dfc.search.sourcecache.refresh_interval = # Time in milliseconds between refreshes of the cache of type information. # min value: 0, max value: dfc.search.typecache.refresh_interval = Configuring search relevancy You can configure the weighting of criteria used for ranking the relevancy of search results. A weight is a numerical value that increases or decreases the importance of a search source or set of sources. Scores for sources are combined by an algorithm to produce a relevancy ranking that is used to display the most relevant results first. Weights for relevancy ranking are configured in a file named dfc searchranking.xml, located in the Documentum home /config directory, for example, C:\Documentum\config. In WDK based applications, the Documentum home directory is under the application server executable directory. Add this file to the Documentum/config subdirectory of the binary directory, for example, CATALINA_HOME/bin/Documentum/config. You can specify an alternate location as the value of a Java system property named dfc.searchranking.file. EMC Documentum Version 6.5 Search Development Guide 59

60 Customizing Search with DFC Table 1, page 60 describes the elements that configure relevancy ranking. All elements are contained within the root element <SearchRanking>. Table 1. Relevancy ranking configuration elements Element <SourceBonus> <AttributeQuery> <FullTextQuery> <RankConfidence > <FullText> <Attribute> Description Specifies a specific source or set of sources for which to provide bonus ranking. Contains <AttributeQuery>, <FullTextQuery>, or both. The source attribute value is a source name or a regular expression that defines the source. The type attribute value can be used to restrict the source type to either docbase or external. Specifies a separate bonus for attributes. The source bonus is within [ 1,1] Specifies a separate bonus for fulltext. The source bonus is within [ 1,1] Decreases confidence ranking for specific source or set of sources. The value is within [0,1]. The source attribute value is a source name or a regular expression that defines the source. The type attribute value can be used to restrict the source type to either docbase or external. Specifies a set of attributes to be added to the computation of the fulltext factor. By default, as a partial representation of the fulltext score for a specific document, the computation uses the concatenation of Dublin Core Metatdata Elements. You can set one or more attributes to be used for the computation. Contains one or more <Attribute> elements. Specifies an attribute to be weighted with the fulltext score. The value is an attribute or a regular expression that resolves one or more attributes. 60 EMC Documentum Version 6.5 Search Development Guide

61 Customizing Search with DFC Element <AttributeWeight> <RatingWeight> Description Adds weight for a specific attribute value or values that match a regular expression. The weight of an attribute is a positive number, relative to the other attributes weight. A weight of 0 negates the effect of the attribute. The neutral weight is 1. The attribute attribute specifies the attribute or a regular expression that resolves one or more attributes. The value attribute specifies a value or a regular expression that resolves one or more values. The value is within [0+]. Specifies the relative weight of the score from specific source types compared to the relevancy ranking score (this last one is assigned a neutral weight of 1). With a weight of 0 the score from the specific source is not taken into account; with a weight of 100 or greater the relevancy ranking score is ignored (not computed). The source attribute value is a source name or a regular expression that defines the source. The type attribute value can be used to restrict the source type to either docbase or external. The rating weight is within [0+]. Note: Regular expression substitution is supported. For example, attrribute=ʺ.*format.*ʺ resolves any attribute with the substring format in the name. The declaration <Attribute>abstract.* summary</ Attribute> resolves any attribute starting with abstract, or the summary attribute. DTD for this file: <!ELEMENT SearchRanking (SourceBonus*, RankConfidence*, FullText?, AttributeWeight*, RatingWeight*)> <!ELEMENT SourceBonus (AttributeQuery?, FullTextQuery?)> <!ATTLIST SourceBonus source CDATA #IMPLIED> <!ATTLIST SourceBonus type (any docbase external) "any"> <!ELEMENT AttributeQuery (#PCDATA)> <!ELEMENT FullTextQuery (#PCDATA)> <!ELEMENT RankConfidence (#PCDATA)> <!ATTLIST RankConfidence source CDATA #IMPLIED> <!ATTLIST RankConfidence type (any docbase external) "any"> <!ELEMENT FullText (Attribute*)> <!ELEMENT Attribute (#PCDATA)> <!ELEMENT AttributeWeight (#PCDATA)> <!ATTLIST AttributeWeight attribute CDATA #REQUIRED> <!ATTLIST AttributeWeight value CDATA #IMPLIED> <!ELEMENT RatingWeight (#PCDATA)> <!ATTLIST RatingWeight source CDATA #IMPLIED> <!ATTLIST RatingWeight type (any docbase external) "any"> EMC Documentum Version 6.5 Search Development Guide 61

62 Customizing Search with DFC Example 4 1. Using the FAST score for 5.3 repositories The following example retains the FAST index score computed for 5.3 repositories. (These repositories in the example contain the substring ʺ53 in their name) instead of the unified ranking score. For all other sources, the computed source score is merged with the unified ranking score. <RatingWeight source=".*53.*" type="docbase">100</ratingweight> Example 4 2. Adding a bonus for a specific source The unified ranking score takes only into account the results metadata. You can give a bonus for a specific source when you know that the source returns relevant results. In the following sample, a 0.3 bonus is added to the score of all results returned by the source named ʺgood_source. <SourceBonus source="good_source"> <AttributeQuery>0.3</AttributeQuery> <FullTextQuery>0.3</FullTextQuery> </SourceBonus> Example 4 3. Improving the score for a specific document type or format You can weight the score for a specific document type or format or based on some other attribute value. The following example improves the score of results that match the constraints and that contain the format attribute with the value of pdf. <SearchRanking> <AttributeWeight attribute="format" value="pdf">10</attributeweight> </SearchRanking> Generating queries in DFC There are two ways to execute a query in DFC: Simple query using IDfQuery Complex query using IDfQueryManager (query builder) Simple queries in DFC You can use the IDfQuery interface, which is not part of the search service, for simple queries. Refer to the javadocs for the com.documentum.fc.client.search package for a description of how to use this capability. Executing a simple query with IDfQuery The following example from the WDK GroupAttributes class executes a simple query and gets the results as an IDfCollection: StringBuffer query = new StringBuffer(512); query.append( "SELECT group_name FROM dm_group where ANY i_all_users_names = '"); query.append(loginusername); 62 EMC Documentum Version 6.5 Search Development Guide

63 Customizing Search with DFC query.append("'"); IDfQuery queryobject = DfcUtils.getClientX().getQuery(); queryobject.setdql(query.tostring()); IDfCollection collection = queryobject.execute( getdfsession(), IDfQuery.DF_READ_QUERY); DFC query builder APIs Building a query definition with IDfQueryBuilder With IDfQueryBuilder, you can use DQL syntax to query one or more fulltext indexed or non indexed Servers. In conjunction with Enterprise Content Integration (ECI) Services product, you can query external sources and the client desktop as well. IDfQueryBuilder provides a programmatic interface to change the query structure, support external sources, support asynchronous operations, change display attributes, and perform concurrent query execution in a federation. IDfQueryBuilder allows you to build queries that provides the following information: Data to build the query Source list (required) Max result count attribute Container of source names Transient search metadata manager bound to the query Transient query validation flag Attributes to order the results by The search interfaces are described below. IDfQueryManager IDfQueryManager is an object oriented interface to build a query. This interface does not manipulate a String representation. It is internally responsible for translating the query to different language and language levels: DQL, FTDQL, ECI Query Language. In Webtop search components, use IDfQueryBuilder to access and manipulate queries. IDfQueryProcessor Synchronous and asynchronous queries are queued and executed by pools of brokers: one queue for repositories, one queue for external sources. Each broker is a thread running in DFC that executes a query on a single source. For example, a broker can execute an IDfQuery on a repository. Brokers for external sources connect to ECI Services brokers for repositories. The number of brokers can be configured in dfc.properties as shown in the following example: dfc.search.ecis.brokers=30 dfc.search.docbase.brokers=20 Results and events such as progress or errors are returned as soon as they arrive. This process is diagrammed below: EMC Documentum Version 6.5 Search Development Guide 63

64 Customizing Search with DFC Figure 13. Asynchronous query processing IDfSourceMap Maps the available repositories and external sources and their capabilities. (The latter requires ECI Services.) IDfSearchMetadataMgr Determines for the query builder what metadata is available from the selected sources, such as available object types and data dictionary information about the types. External sources are managed by the ECIS server store and administration tools. The Search Metadata Manager communicates with the ECIS server to assemble a list of available sources. External custom types can be searched if they are defined in the ECIS configuration file as a value of client.dfc.types. Additionally, dm_sysobject and dm_document types are queried in external sources, but not all attributes of these types will be available in the external sources. For multi repository searches, the first repository in a client search list is used as the metadata model server and is used to retrieve all data dictionary information. The search metadata manager has methods to get types and their attributes from each source. IDfResultObjectManager Gets the object content. 64 EMC Documentum Version 6.5 Search Development Guide

65 Customizing Search with DFC Creating DFC search applications You can create DFC search applications based on servlets and JSP pages and the DFC Search Service. For information on the DFC query builder service, refer to Generating queries in DFC, page 62 and the javadocs for the package com.documentum.fc.client.search. Hello World search application The following example takes a search input string and searches all available sources known to the search service: /** * Search the web based on the search string and stores it in the Hashmap */ private void saveecisearchresults() { System.out.println("ECISearch Method :SaveECISearchResults: Start"); IDfSearchSourceMap srcmap = null; IDfClient localclient = null; IDfQueryManager querymgr = null; IDfQueryBuilder querybldr = null; IDfQueryProcessor idfqueryprocessor = null; IDfResultsSet resultsset = null; IDfResultObjectManager idfresultobjmgr = null; ArrayList arrexternalsources = new ArrayList(20); mmap = new HashMap(); int c = 0; try { IDfClient client = m_clientx.getlocalclient(); /* * sessionmanager A session manager to be used for authentication * against search sources * defaultmetadatadocbase The default repository from which to pick * type metadata. Can be safely set to null if the search service is * configured to search only repositories and not on external sources. * Must not be null if external sources are configured in the search * service. The session manager must have login info for the repository */ IDfSearchService searchservice = client.newsearchservice( m_sessionmanager, m_docbasename); srcmap = searchservice.getsourcemap(); querymgr = searchservice.newquerymgr(); IDfQueryBuilder querybuilder = querymgr.newquerybuilder("dm_sysobject"); IDfSearchMetadataManager IDfSearchMetadataManager = querybuilder. getmetadatamgr(); //Getting the source map IDfSearchSourceMap searchsourcemap = searchservice.getsourcemap(); //Getting list of available external sources IDfEnumeration enumsearchsource = searchsourcemap.getavailablesources( EMC Documentum Version 6.5 Search Development Guide 65

66 Customizing Search with DFC } IDfSearchSource.SRC_TYPE_EXTERNAL); while (enumsearchsource.hasmoreelements()) { IDfSearchSource idfsource = ( IDfSearchSource) enumsearchsource.nextelement(); String[] strexternalsource = new String[2]; strexternalsource[0] = idfsource.getname(); System.out.println("External Sources(0):" + strexternalsource[0]); arrexternalsources.add(strexternalsource); //add source to SearchMetadatamanager IDfSearchMetadataManager.addSelectedSource(strExternalSource[0]); //add the source to the query builder querybuilder.addselectedsource(strexternalsource[0]); } IDfExpressionSet rootexp = querybuilder.getrootexpressionset(); //Creating the search query rootexp.addsimpleattrexpression("object_name", IDfValue.DF_STRING, IDfSimpleAttrExpression.SEARCH_OP_CONTAINS, false, false, m_searchstring); querybuilder.addresultattribute("object_name"); idfqueryprocessor = searchservice.newqueryprocessor(querybuilder, true); idfresultobjmgr = searchservice.newresultobjectmanager(querybuilder); idfqueryprocessor.addlistener(this); idfqueryprocessor.search(); System.out.println("ECISearch Method: Query Failed : " + idfqueryprocessor.getquerystatus().getnbrfailed()); Thread.sleep(m_sleepTime); System.out.println("ECISearch Method: Query Status : " + idfqueryprocessor.getquerystatus().getstatus()); IDfResultsSet rs = idfqueryprocessor.getresults(); System.out.println(rs.size() + " result(s)\n"); while (rs.next()) { IDfResultEntry result = rs.getresult(); // Filter the results based on the score attribute if (result.getstring("score").equalsignorecase("1.0") result. getstring("score") == "1.0") { String objectname = result.getstring("object_name"); mmap.put(objectname, result); System.out.println(result); } } addexternalfilestofolder(mmap, idfresultobjmgr); } catch (Exception e) { e.printstacktrace(); } Displaying the ECIS targets at design time //Getting the source map IDfSearchSourceMap searchsourcemap = searchservice.getsourcemap(); 66 EMC Documentum Version 6.5 Search Development Guide

67 Customizing Search with DFC //Getting list of available external sources IDfEnumeration enumsearchsource = searchsourcemap.getavailablesources( IDfSearchSource.SRC_TYPE_EXTERNAL); while (enumsearchsource.hasmoreelements()) { IDfSearchSource idfsource = (IDfSearchSource) enumsearchsource.nextelement(); String[] strexternalsource = new String[2]; strexternalsource[0] = idfsource.getname(); //custom test to check if source belongs to the selection of the user //(design time) if (strexternalsource does not belong to selection at design time) continue; } //add source to SearchMetadatamanager IDfSearchMetadataManager.addSelectedSource(strExternalSource[0]); //add the source to the query builder querybuilder.addselectedsource(strexternalsource[0]); Seting the ECI target at query execution time //Getting the source map IDfSearchSourceMap searchsourcemap = searchservice.getsourcemap(); //Getting list of available external sources IDfEnumeration enumsearchsource = searchsourcemap.getavailablesources( IDfSearchSource.SRC_TYPE_EXTERNAL); while (enumsearchsource.hasmoreelements()) { IDfSearchSource idfsource = (IDfSearchSource) enumsearchsource.nextelement(); String[] strexternalsource = new String[2]; strexternalsource[0] = idfsource.getname(); //custom test to check if source belongs to the selection of the user //(design time) if (strexternalsource does not belong to selection at design time) continue; } //add source to SearchMetadatamanager IDfSearchMetadataManager.addSelectedSource(strExternalSource[0]); //add the source to the query builder querybuilder.addselectedsource(strexternalsource[0]); Sample DFC application In the following example, a login servlet (LoginServlet class) and login.jsp page handle user login. (The login class servlet is not shown in the following code.) The SearchServlet class handles query building and execution. The JSP pages search.jsp and results.jsp display a search form and results. Figure 14, page 68 shows the UI that is displayed in search.jsp. EMC Documentum Version 6.5 Search Development Guide 67

68 Customizing Search with DFC Figure 14. Simple DFC search UI Figure 15, page 69 shows the directory structure for this simple application. 68 EMC Documentum Version 6.5 Search Development Guide

69 Customizing Search with DFC Figure 15. Simple DFC search application structure The SearchServlet class gets the query builder instance to create a search, passing in the parameters from the search JSP page that were entered by the user (ʺft and ʺobject_name ): String fulltextvalue = httpservletrequest.getparameter("ft"); String objectnamevalue = httpservletrequest.getparameter("object_name"); String docbase= httpservletrequest.getparameter("docbase"); IDfSearchService searchservice = client.newsearchservice(smgr, docbase); IDfQueryManager querymanager = searchservice.newquerymgr(); IDfQueryBuilder querybuilder = querymanager.newquerybuilder("dm_document"); In this servlet, IDfSearchService (com.documentum.fc.client.search) is the entry point to search related services: query building, query execution, results manipulation, available sources and query metadata. The following lines in the search servlet set the result attributes to be displayed. The servlet then adds the source repository, which can either be added to the UI or set in the servlet class. These inputs are used to build an expression set: the method addfulltextexpression adds the string that the user entered in the search form, and the method addsimpleattrexpression adds the object name value that was supplied by the user in the search form and sets the search operator to ʺcontains : querybuilder.addresultattribute("object_name"); querybuilder.addresultattribute("summary"); querybuilder.addresultattribute("score"); querybuilder.addselectedsource(docbase); EMC Documentum Version 6.5 Search Development Guide 69

70 Customizing Search with DFC IDfExpressionSet rootexpressionset = querybuilder.getrootexpressionset(); if (fulltextvalue!=null) rootexpressionset.addfulltextexpression(fulltextvalue); if (objectnamevalue!=null) rootexpressionset.addsimpleattrexpression( "object_name", IDfValue.DF_STRING, IDfSimpleAttrExpression. SEARCH_OP_CONTAINS, false, false, objectnamevalue); The following lines execute the query synchronously by using the synchronous call blockingsearch with a timeout of 60 seconds. The query processor handles the query execution. When the query has finished, the control is forwarded to the JSP page to build the results page. IDfQueryProcessor queryprocessor = searchservice.newqueryprocessor( querybuilder, true); queryprocessor.blockingsearch(60000); The results JSP page is generated by the following code. The interface IDfResultEntry is similar to IDfTypedObject but is not modifiable. <%IDfResultsSet results = queryprocessor.getresults(); for (int index = 0; index < results.size(); index++) { IDfResultEntry result = results.getresultat(index); %> <table border="0" cellpadding="0" cellspacing="0" width="100%" style=" margin bottom: 8px"> <tr><td width="5"/><td width="1"> </td> <td> <div class="result title"><%=result.getstring("object_name")%> </div> <div class="result score"> <%= result.getsource() %> <i> <%=(int)(result.getscore() * 100)%>%</i> </div> <br/><font size=" 1"><%=result.getString("summary")%></font> </td> </tr><tr height="1"></tr> </table> <% } %> Using the visitor API You can use the visitor API in DFC to visit nodes in the expression tree. The following example creates a QueryDumper class that visits the expressions in the query. import com.documentum.fc.client.searchdfexpressionvisitor; class QueryDumper extends ExpressionVisitor { private StringBuffer m_expressiondump = new StringBuffer(); public String dump() 70 EMC Documentum Version 6.5 Search Development Guide

71 Customizing Search with DFC { } return m_expressiondump.tostring(); public final void visit(idfexpressionset expr) throws DfException { switch (expr.getlogicaloperator()) { case IDfExpressionSet.LOGICAL_OP_AND:m_expressionDump.append("(and "); break; case IDfExpressionSet.LOGICAL_OP_OR:m_expressionDump.append("(or "); } super.visit(expr); m_expressiondump.append(")"); } public void visit(idfvaluelistattrexpression expr) throws DfException { super.visit(expr); dumpattrandoperator(expr); IDfEnumeration values = expr.getvalues(); while (values.hasmoreelements()) { String value = (String) values.nextelement(); m_expressiondump.append(" ").append(value); } m_expressiondump.append("]"); } public void visit(idffulltextexpression expr) throws DfException { super.visit(expr); m_expressiondump.append("[ft ").append(expr.getvalue()).append("]"); } public void visit(idfsimpleattrexpression expr) throws DfException { super.visit(expr); dumpattrandoperator(expr); if ((expr.getsearchoperationcode()!= IDfSimpleAttrExpression. SEARCH_OP_IS_NULL) && (expr.getsearchoperationcode()!= IDfSimpleAttrExpression.SEARCH_OP_IS_NOT_NULL)) { m_expressiondump.append(" ").append(expr.getvalue()); } m_expressiondump.append("]"); } public void visit(idfrelativedateexpression expr) throws DfException { super.visit(expr); dumpattrandoperator(expr); String timeunitasastring = ReflectionUtil.getConstantName( Calendar.class, expr.gettimeunit()); m_expressiondump.append(" ").append(expr.getrelativetime()).append( " ").append(timeunitasastring).append("]"); } EMC Documentum Version 6.5 Search Development Guide 71

72 Customizing Search with DFC } public void visit(idfvaluerangeattrexpression expr) throws DfException { super.visit(expr); dumpattrandoperator(expr); m_expressiondump.append(" ").append(expr.getfromvalue()).append(" ").append( expr.gettovalue()).append("]"); } private void dumpattrandoperator(idfattrexpression expr) { m_expressiondump.append("[").append(expr.getattrname()).append(" "); String searchopasastring = s_operationmap.get(expr.getsearchoperationcode()); String valuedatatypeasastring = ReflectionUtil.getConstantName( IDfValue.class, expr.getvaluedatatype()); m_expressiondump.append(searchopasastring).append("(").append( valuedatatypeasastring).append(")"); } You can use the expression visitor in a class that accesses the query builder, such as a customized Webtop search class. The following example gets the query expression set that has been submitted by the user: QueryDumper querydumper = new QueryDumper(); rootexpr = querybuilder.getrootexpression(); rootexpr.acceptvisitor(querydumper); System.out.println("query =" + querydumper.dump()); Using a DQL hints file DFC reads a DQL hints file if it is present on the application server. DFC applies the hints to queries based on conditions defined in the file. You can define conditions under which the hints are applied, for example, for certain object types, attributes, or repositories. Refer to Appendix A, DQL Hints File DTD for the hints file DTD. Table 2, page 73 describes the behavior governed by these elements. Hints file location The DQL hints file location is specified in the DFC configuration file dfc.properties on the application server host: dfc.dqlhints.file=c:/documentum/config/dfc dqlhints.xml Alternatively, you can place a DQL hints file in the application server host system classpath or as a system environment variable, for example: Ddfc.dqlhints.file=path_to_hints_file 72 EMC Documentum Version 6.5 Search Development Guide

73 Customizing Search with DFC Use forward slashes for paths in Java properties file (back slash is used for escape). Alternatively, the file can be loaded from classpath or the DFC data home directory on the application server host. Hints file elements The following elements are contained within a root <RuleSet> element to define the hints passed to IDfQueryManager. Table 2. DQL hints file elements Element <Rule> <DisableFullText/> <DisableFTDQL/> <Condition> <Select>, <Where> <SelectOption> <From> <Docbase> Description Can have zero to many <Condition> elements Disables fulltext search on basic search or attributes for the conditions in the rule Disables search for metadata in the FT index. Child elements are ANDed Child <Attribute> elements can be ANDed (condition= all ) or ORed (condition= any ) Adds a permission, for example, FOR READ or FOR BROWSE. For example, FOR DELETE would limit the results of a query that meets the condition to those documents on which the user has delete permission. The following example applies to all Webtop queries: <RuleSet> <Rule> <Condition> <Where> <Attribute operator="like">object_ name</attribute> </Where> </Condition> <SelectOption>FOR DELETE</SelectOption> <DisableFTDQL/> </Rule> </RuleSet> Child <Type> elements can be ANDed (condition= all ) or ORed (condition= any ) The value of this element corresponds to a repository to which the hint applies. The descend attribute is optional, with a default= false. To apply the DQL hint to a folder and all its subfolders,set descend= true. EMC Documentum Version 6.5 Search Development Guide 73

74 Customizing Search with DFC Element <Attribute>, <Type>, <Docbase> <Attribute> <FulltextExpression> <DQLHint> Description Support Java regular expression (java.util.regex. Pattern). For example, <type>custom.*</type> matches all type names beginning with ʺcustomʺ. Operator ʺlikeʺ represents DQL predicates CONTAINS and LIKE. The value ʺis_nullʺ represents DQL predicates NULL, NULLINT, NULLSTRING, and NULLDATE. Child of <condition>. Set the mandatory exists attribute =ʺfalseʺ to add ENABLE(NOFTDQL) to the query when there is no fulltext expression in the search. Contains any valid DQL hint. For the full list of DQL hints, refer to Content Server DQL Reference Manual. Hints file examples Turning off attribute queries in the fulltext index To send all queries on attributes to the database if the query does not contain a fulltext search expression, define the following hint: <RuleSet> <Rule> <Condition> <FulltextExpression exists="false"/> </Condition></Rule></RuleSet> If you disable FTDQL for specific conditions that you have defined within the <rule> element, the attributes portion of the query that meets those conditions will be issued against the database, not the fulltext index. A temp table will be populated with the fulltext result. If the fulltext query is unselective, then the temp table will be large, negatively impacting response time. Turning off FTDQL for specific WHERE clause operators or attributes In the following example, FTDQL is turned off for queries on the object_name attribute that use the ʺlike operator (in the Webtop UI, ʺcontains, ʺbegins with, or ʺends with ). Multiple attributes can be added to the rule. <RuleSet> <Rule> <DQLHint>ENABLE(FT_CONTAIN_WORD)</DQLHint></Rule></RuleSet> Turning off FTDQL for specific types In the following example, attributes for the specified object type are queried in the database, not the fulltext index: <RuleSet> <Rule> 74 EMC Documentum Version 6.5 Search Development Guide

75 Customizing Search with DFC <Condition> <From condition="any"> <Type>km_message</Type> </From> </Condition> <DisableFTDQL/> </Rule> </RuleSet> Adding multiple hints to queries The following example adds two hints to wildcard queries on either of two attributes: <RuleSet> <Rule> <Condition> <Where condition="any"> <Attribute operator="like">subject</attribute> <Attribute operator="like">object_name</attribute> </Where> </Condition> <DQLHint>ENABLE(SQL_DEF_RESULT_SET 100, NOFTDQL)</DQLHint> <DisableFTDQL/> </Rule> </RuleSet> Using multiple rules In the following hints file, one rule applies to queries for one attribute, the second rule applies to a different attribute: <RuleSet> <Rule> <Condition> <Where condition="any"> <Attribute operator="like">subject</attribute> </Where> </Condition> <DQLHint> ENABLE(SQL_DEF_RESULT_SET 100, NOFTDQL) </DQLHint> <DisableFTDQL/> </Rule> <Rule> <Condition> <Where condition="any"> <Attribute operator="like">object_name</attribute> </Where> </Condition> <DQLHint> ENABLE(SQL_DEF_RESULT_SET 10) </DQLHint> <DisableFTDQL/> </Rule> </RuleSet> Make sure that your multiple rules will be mutually exclusive when applied to a single query, or the query will generate a DQL syntax error. The above example would throw an error in Webtop if the user added both attributes to the query. Turning off FTDQL for date ranges Date queries are often queries on a date range, which do not perform well. The following example queries the date range against the database: EMC Documentum Version 6.5 Search Development Guide 75

76 Customizing Search with DFC <RuleSet> <Rule> <Condition> <Where condition="any"> <Attribute>r_creation_date</Attribute> <Attribute>r_modify_date</Attribute> <Attribute>r_access_date</Attribute> </Where> </Condition> <DisableFTDQL/> </Rule> </RuleSet> Turning off FTDQL for a specific repository or for FOLDER(DESCEND) You can turn off FTDQL for attribute queries in a repository, adding conditions as needed, as shown in the following example: <Rule> <Condition> <Docbase> <Name>support</Name> </Docbase> </Condition> <DisableFTDQL/> </Rule> You can turn off FTDQL for FOLDER(DESCEND) queries. In Webtop, this would turn off FTDQL for searches from current location or some other specific location instead of from the repository root. These queries can time out if there are many subfolders. The following example sends the attribute portion of the query to the database instead of the fulltext index for the specific repository. The descend attribute specifies whether to apply the condition and hint to FOLDER(DESCEND) queries: <Rule> <Condition> <Docbase> <Name descend="true">dm_notes</name></docbase> </Condition> <DisableFTDQL/> </Rule> DQL hints and Webtop search components The Webtop search components use the DFC query builder package to construct a query. The DFC query builder adds the DQL hint TRY_FTDQL_FIRST, which handles timeouts and resource exceptions returned from the index server by querying the attributes portion of a query against the repository s database. The query builder also bypasses grammatical normalization using a DQL hint for wildcard and phrase searches. Wildcard attribute searches (ʺcontainsʺ, ʺbegins withʺ, ʺends withʺ) can time out if they have a large number of results. You can configure attributes searches to go directly against the repository metadata, which can be significantly faster than the default behavior of the TRY_FTDQL_FIRST hint. In the following DQL hints file example, FTDQL is turned off for object_name attribute queries: 76 EMC Documentum Version 6.5 Search Development Guide

77 Customizing Search with DFC <Rule> <Condition> <Where> <Attribute operator="like">object_name </Attribute> </Where> </Condition> <DisableFTDQL/> <Rule> The following query is generated when the user searches on the object name and also enters a string into the fulltext box. The string ʺtechnicalʺ is queried in the fulltext index and the query for object_name containing ʺWDKʺ is queried against the database: SELECT r_object_id,text,object_name,... FROM dm_document SEARCH DOCUMENT CONTAINS technical WHERE (UPPER(object_name) LIKE %WDK% ESCAPE \) AND (a_is_hidden = FALSE) ENABLE(NOFTDQL) Note: If your hint contains an object type condition, the hint is applied only for that type and its subtypes, not for the supertype. Example 4 4. Removing the fulltext search option You can also remove the fulltext search option for certain roles or object types. Extend the Webtop search component definition in /webtop/config and scope the search component definition for the role or type. The custom search definition is the following: <scope type='my_custom_type'> <component modifies=' advsearch:webcomponent/config/library/search/searchex/advsearchex_component.xml'> <replace path='pages.start'> <start>/custom/advsearch/advsearchex_mytype.jsp</start> </replace> </component></scope> Create a copy of the search JSP page for your new scope and remove the fulltext search table row: <tr> <td align=right valign=top nowrap> <dmf:label nlsid='msg_containing_words' cssclass='fieldlabel'/> </td> <td align=left valign=top nowrap> <dmfxs:searchfulltext name='fulltextctrl.../></td></tr> Add the DQL hint for the object type. This hint will turn off fulltext search for the type: <Rule> <Condition> <From> <Type>my_custom_type</Type> </From> </Condition> <DisableFullText/> </Rule> EMC Documentum Version 6.5 Search Development Guide 77

78 Customizing Search with DFC Optimizing query batch size You can optimize query performance by setting a smaller batch size. The batch size is the number of results returned at a time by the index server. Set the batch size for an individual query, if you are constructing the query in DFC. Set it for multiple queries using a DQL hints file. Use the ENABLE(OPTIMIZE_TOP XX) hint, in which XX is replaced by your desired batch size. Any value can be used for OPTIMIZE_TOP, but larger values will probably not result in optimization. Note: The RETURN_TOP hint does not optimize query batch size. It has the purpose of limiting the number of rows that are returned. For full details, refer to Content Server DQL Reference Manual. 78 EMC Documentum Version 6.5 Search Development Guide

79 Chapter 5 Troubleshooting Queries The following information can help you troubleshoot problems in performance or querying: Speeding up queries in WDK based applications, page 79 Testing, logging, and debugging queries, page 79 Tracing fulltext query operations, page 80 Speeding up queries in WDK based applications You can check the following for performance: Check bandwidth between the application server and Content server Remove folder(descend) options such as the ability to search from the current location in advanced search or the ability to select folders within a repository in the search preferences component. Consider adding an ENABLE(NOFTDQL) hint in a dql hints file for queries that are likely to time out. For more information, refer to Using a DQL hints file, page 72. Advise users to set the page display to 10 items, not 100. Results for large page sizes are not displayed as they come in; they are accumulated until the page size is filled. Large page sizes also create slower queries because all object attributes are cached in memory for the first page. Testing, logging, and debugging queries You can test queries using DQL (including FTDQL) by calling the dqleditor component with a URL similar to the following: You can enable query debugging in $DOCUMENTUM_HOME/config/log4j.properties: Debug DQL queries by adding the following line: log4j.logger.com.documentum.fc.client.search.impl.generation=debug EMC Documentum Version 6.5 Search Development Guide 79

80 Troubleshooting Queries Debug ECI queries by adding the following line (no line break): log4j.logger.com.documentum.fc.client.search.impl.execution.adapter. remoteeci.ecisadapter=debug After restarting the application, exercise your query and examine the log. The log file name and location is specified in the log4j.properties file. The following excerpt from wdk.log traces a query: 8101 [Search Broker:MSSQL53SP4ECI8:processing started at Thu Jul 05 14:04:34 CEST 2007] DEBUG com.documentum.fc.client.search.impl.generation.docbase.simple.resultentryiterato rwrapper InternalProcess dql=select r_object_id,score,object_name FROM dm_document WH ERE (a_is_hidden = FALSE) ENABLE(TRY_FTDQL_FIRST) Document not returned by search Check the following list of possible causes: Check whether the docbase is indexed Verify that Indexing Agent is running on the Content Server and that the expected document has been properly indexed: 1. Connect to on the host where index server is running. 2. In the query box, enter: dmftkey:id_of_object, substituting the object ID for the object to be searched for. 3. Click Search. The XML data displayed on the screen should have the proper object metadata. If you see object metadata, then the reference object has been indexed. Verify that the DQL query contains ENABLE(FTDQL) Limitations The following limitations may affect your expected search results: Basic search: No wildcard matching Advanced search: Property search is not case sensitive against an indexed repository Tracing fulltext query operations Any DFC application can use log4j to trace queries. For information, refer to Viewing the query smartlist, page 81 and specify the DFC query class to trace. Using Documentum Administrator, you can trace fulltext querying operations. Use the MODIFY_TRACE administration method to turn on tracing. Two tracing levels are available: None: Tracing is turned off. All: Content Server and fulltext messages resulting from queries are logged. You can also trace the index agent operations. Refer to Content Server Fulltext Indexing System Installation and Administration Guide for details. 80 EMC Documentum Version 6.5 Search Development Guide

81 Troubleshooting Queries Viewing the query smartlist You can view the smartlist that is generated by the query builder by control click on Edit in the search results page. This action also displays the list of events that occurred during the search including the DQL being sent, the ECIS query being sent, and the errors from the search sources. Use the smartlist query output o debug the DQL being sent. This view is for troubleshooting purposes only and should not be used to edit queries. The smartlist output is similar to the following: <SmartListDefinition> <QueryBuilder allversions="false" hiddenobjects="false" isdatabasesearchrequested="false" maxresults=" 1" objecttype="dm_document"> <SourceList> <Source description="dm_bugs docbase" name="support" serverversion=" SP3 Win32.Oracle" type="docbase"/> </SourceList> <QueryScopeList> <LocationQueryScope isdescend="false" source="support"> Y5OU+iJ2pAARJg==</LocationQueryScope> </QueryScopeList> <! The fulltext and attribute expressions are ANDed > <ExpressionSet logicaloperator="and"> <FulltextExpression>template</FulltextExpression> <ExpressionSet logicaloperator="and"> <SimpleAttributeExpression attribute="object_name" casesensitive="false" datatype="string" repeated="false" searchoperation="contains">template </SimpleAttributeExpression> </ExpressionSet> </ExpressionSet> <ResultAttributeList> <ResultAttribute name="object_name" />... </ResultAttributeList> </QueryBuilder> <DisplayPreferences> <DisplayAttributeList> <DisplayAttribute name="object_name" />... </DisplayAttributeList> </DisplayPreferences> <PropertyList /> </SmartListDefinition> EMC Documentum Version 6.5 Search Development Guide 81

82 Troubleshooting Queries 82 EMC Documentum Version 6.5 Search Development Guide

83 DQL Hints File DTD Appendix A <!ELEMENT RuleSet (Rule*)> <!ELEMENT Rule (Condition?, DQLHint?, SelectOption?, DisableFullText?, DisableFTDQL?)> <!ELEMENT Condition (Select?, From?, Where?, Docbase?, FulltextExpression?)> <!ELEMENT DQLHint (#PCDATA)> <!ELEMENT SelectOption (#PCDATA)> <!ELEMENT DisableFullText EMPTY> <!ELEMENT DisableFTDQL EMPTY> <!ELEMENT Select (Attribute+)> <!ATTLIST Select condition (all any) "all"> <!ELEMENT From (Type+)> <!ATTLIST From condition (all any) "all"> <!ELEMENT Where (Attribute+)> <!ATTLIST Where condition (all any) "all"> <!ELEMENT Docbase (Name+)> <!ELEMENT FulltextExpression EMPTY> <!ATTLIST FulltextExpression exists (true false) #REQUIRED> <!ELEMENT Attribute (#PCDATA)> <!ATTLIST Attribute operator (equal not_equal greater_than greater_equal less_than less_equal like not_like is_null is_not_null in not_in between) #IMPLIED> <!ELEMENT Type (#PCDATA)> <!ELEMENT Name (#PCDATA)> <!ATTLIST Name descend (true false) #IMPLIED> EMC Documentum Version 6.5 Search Development Guide 83

84 DQL Hints File DTD 84 EMC Documentum Version 6.5 Search Development Guide

85 Appendix B DFC Search Service API Overview Figure 16, page 86 is a UML diagram of the main interfaces in the DFC search service (com.documentum.fc.client.search package). EMC Documentum Version 6.5 Search Development Guide 85

86 DFC Search Service API Overview Figure 16. Data model 86 EMC Documentum Version 6.5 Search Development Guide