EMC Documentum Content Intelligence Services

Transcription

1 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide P/N A01 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright EMC Corporation. All rights reserved. Published December 2007 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners.

3 Table of Contents Preface... 7 Chapter 1 Overview... 9 Content Intelligence Services components Content Intelligence Services architecture Data synchronization Classification information Content Intelligence Services user roles The taxonomy manager The category owner Integrating Content Intelligence Services Organizing your library Workflow and lifecycle processing Web Publisher integration How to make up for 5.3 legacy tools Using Documentum Administrator instead of legacy tools Chapter 2 Processing Documents Selecting documents to process Submitting documents for processing Defining submission schedules Submitting documents on demand Resubmitting documents Understanding information extraction Understanding conceptual analysis How the concept score is computed Understanding auto categorization Configuring the pattern analysis Pattern analysis Limitations Using patterns in rules Defining Patterns Chapter 3 Administering Content Intelligence Services Configuring Content Intelligence Services Administering the CIS server Starting/Stopping CIS server Configuring CIS server As an application As a Windows service Monitoring CIS server processing CIS server log files Common installation errors EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 3

4 Table of Contents Stemming Capability Understanding the dynamics Installing language dictionaries Configuring CIS to work with other languages Chapter 4 Using the Taxonomy Exchange Format Importing taxonomies in Taxonomy Exchange Format Taxonomy Exchange Format Action Files Creating TEF action files Appendix A Extracted properties EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

5 Table of Contents List of Figures Figure 1. CIS architecture overview EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 5

6 Table of Contents List of Tables Table 1. Terminology correspondences Table 2. Actions performed with CIS Administrator Table 3. Actions performed with Domain Map Editor Table 4. Configuration parameters in the cis.properties file Table 5. <category> Element Attributes Table 6. <categoryevidence> Element Attributes Table 7. <categoryevidencedefaults> Element Attributes Table 8. <categorylink> Element Attributes Table 9. <class> Element Attributes Table 10. <definition> Element Attributes Table 11. <details> Element attributes Table 12. <details> Element attributes Table 13. <evidence> Element Attributes Table 14. <evidencepropagation> Element Attributes Table 15. <impliedkeyworddefaults> Element Attributes Table 16. <keyword> Element Attributes Table 17. <keyworddefaults> Element Attributes Table 18. <operation> Element Attributes Table 19. <owner> Element Attributes Table 20. <qualifier> Element Attributes Table 21. <supportedlanguage> Element Attributes Table 22. <taxonomy> Element Attributes Table 23. <add> Element Attributes Table 24. <categoryobject> Element Attributes Table 25. <categoryreference> Element Attributes Table 26. <classobject> Element Attributes Table 27. <classreference> Element Attributes Table 28. <export> Element Attributes Table 29. <taxonomyobject> Element Attributes Table 30. <withinparentreference> Element Attributes Table 31. Extracted properties for MS Office and other documents EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

7 Preface The Content Intelligence Services Administration Guide contains procedures and information for setting up and managing the server-side components of Content Intelligence Services (CIS). This manual assumes you have already installed Content Intelligence Services by following the instructions in the Content Intelligence Services Installation Guide. Intended audience This manual is intended primarily for administrators who are managing Content Intelligence Services applications. CIS server categories documents into taxonomies that you build and maintain using Documentum Administrator. For information about using Documentum Administrator, see the Documentum Administrator User Guide. Revision history The following changes have been made to this document. Revision history Revision Date December 2007 Description Initial publication EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 7

8 Preface 8 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

9 Overview Chapter 1 Content Intelligence Services organizes documents into taxonomies. A taxonomy is a hierarchical set of categories used to organize content in the repository based on a different set of criteria from the cabinet and folder structure. The alternate organization, often based on the subject matter of the content, provides one place for users to look for all content related to common topics of interest. For example, suppose that the folders in repository cabinets organize objects based on which department created the content or on the document type, such as Press Releases in one folder and Product Design Specifications in another folder. A user looking for all available information about a particular product including documents from multiple departments, and both press releases and design specifications needs to look in all folders that could possibly include objects related to that topic. With product-based categories, the user can look in a single category to find all documents related to the product, while the documents themselves remain filed in the original folders. Content Intelligence Services can assign documents to relevant categories based on a semantic analysis of their content. When you define your taxonomy, you identify keywords, phrases, and patterns associated with each category. CIS server uses these words and phrases as evidence terms: when the server processes a document, it assigns the document to these categories based on the evidence terms it finds in the content. For example, the definition of a category called Internet Service Provider might identify the acronym ISP and the names of specific ISP vendors as evidence terms. When CIS server analyzes a document and finds the words ISP and Earthlink in the content, it determines that the document belongs to the category Internet Service Provider. As CIS server processes a document, it determine the document s confidence score for each category in the taxonomy. The confidence score reflects how much evidence CIS server found to indicate that the document belongs to the category. If the document s score for a category exceeds or meets a predefined threshold, CIS server assigns the document to that category. If the confidence score falls just short of the threshold, CIS server can provisionally assign the document to the category as a Pending candidate. The user who owns the category must review pending document candidates before they are fully categorized. When a document is assigned to a category, you can decide to link this document in the folder associated with the category. You can also select to add the category names to an attribute of the document. You can enable or disable these features when you configure CIS. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 9

10 Overview Content Intelligence Services also supports manual categorization, where users (rather than CIS server) manually assign documents to categories. Just as with automatic CIS server processing, category assignments can be used to link documents into a searchable hierarchy of category folders, add the category names to a document attribute, or both. Content Intelligence Services components Content Intelligence Services includes these key components: The Content Intelligence Services client (CIS client), such as Documentum Administrator, Webtop, Web Publisher, or any custom application using the Content Intelligence Application Programming Interface (CI API), can be used for creating and managing the taxonomy used for categorizing documents. You can also use Documentum Administrator to configure CIS. The CI API handles communication between the CIS client, the CIS server, and the Documentum repository. The Content Intelligence Services server (CIS server) performs the automatic categorization of documents based on taxonomy and category definitions. A repository is also required to store CIS data (such as taxonomy definitions and document set definitions) When you create a taxonomy using Documentum Administrator or by importing a prebuilt taxonomy, the objects comprising the taxonomy are saved into the repository containing the documents that CIS will process. However CIS server retrieves all of its configuration data, including the taxonomy definitions, from snapshots which are saved locally on the CIS server host machine. When you are done creating or modifying the taxonomy in Documentum Administrator, you synchronize the new definitions with the snapshots. Content Intelligence Services architecture The Content Intelligence Services are deployed to a WebLogic application server as a dedicated managed server. It is recommended to deploy them on a separate machine from the Content Server machine. CIS server communicates with the Content Server using the Documentum Foundation Classes (DFC). The repository of the Content Server stores the documents to classify or already classified but also the taxonomy definitions and the document set definitions. One CIS server can only point to one repository. CIS must be enabled for a given repository before starting any configuration. Once enabled, the taxonomies and the document sets can be created using Documentum Administrator. It is also possible to import existing taxonomies using the TEF utility. 10 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

11 Overview You can either use one CIS server for both mode (production and test) or two CIS servers: one for each mode. One repository can only use one CIS server for each mode. Using two modes allows the CIS user to modify and test the taxonomies and document sets while the production server is still running. Figure 1. CIS architecture overview Data synchronization The server uses snapshots of the definitions of the taxonomies and document sets. The taxonomy snapshots are XML files representing the taxonomies. They are created or updated when the taxonomies are synchronized by the CIS user. So, if a taxonomy definition has never been synchronized, it can never be used. When the CIS server is set for both Test and Production modes, two versions of the taxonomy snapshots are available: one for the production mode and one for the test mode. The snapshots of the taxonomies are stored locally by the CIS server. When the CIS server restarts, it checks the validity of the snapshots and update them if necessary. Conversely, the document set snapshots are only stored during the CIS server session. On CIS server restart, the document set definitions in the repository are read by the CIS server and a new temporary snapshot is created. Classi cation information The classification information indicate which documents have been processed or are in process. The classification results are first category assignments then, depending on EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 11

12 Overview the CIS configuration, they can also correspond to folder assignments and document metadata updates. Classification information and category assignments are stored in the repository. The classification is done per document tree. Only the CURRENT version of the documents is classified but the entire tree is assigned to a category. When clearing the assignments, only the assignments of the current version of the documents are removed. If you create a new version of a document, remember that it inherits the metadata attributes of the previous version. So, if you are using the Assign as Attributes option, it is recommended to remove the attribute values generated by the classification of the previous version of the document and which might be not relevant any longer. Content Intelligence Services user roles Implementing and working with Content Intelligence Services requires the action of several distinct user roles. A person may combine several roles, for example, the System administrator can also be the Taxonomy manager. The following list describes briefly each role: The System administrator installs CIS and enables CIS in the repository. After that, he/she also monitors the CIS server: ensures the server is up and running, checks the logs for errors, etc. The Taxonomy Manager creates, tests and maintains taxonomies. He/she also sets category owners. If necessary, the Category Owner verifies the correct categorization and reviews pending documents. The General user can submit documents for CIS processing manually, and enjoy the results of the categorization by browsing the categories or using the keywords created by the categorization. All the CIS-related tasks are performed in Documentum Administrator, excepted for the General user which will perform his/her tasks in a CIS client such as Webtop. The taxonomy manager The taxonomy manager is responsible for creating the taxonomy tree: he/she defines the taxonomies, the category classes, and the categories. As part of the category definition, he/she sets the required evidence terms, any necessary qualifying rule, and the category owner. He/she also defines the threshold to automatically assigned documents, to keep them as pending documents or to reject them. Taxonomies and categories can be defined from scratch or imported. 12 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

13 Overview The second job of the taxonomy manager is to define the document sets: which documents will be classified and when, if the document set is meant to be run automatically. Before launching any production process, the taxonomy manager must tests the taxonomies to control if the document sets are correctly classified. If a category receives too many assignments, it means that the evidence terms are too general and must be redefined more precisely. The taxonomy manager has the possibility to clear assignments for a category, an entire document set, or only one selected document. Once the tests are successful, he/she can bring the taxonomies online. Only online taxonomies are used for categorization in production mode. The category owner The category owner is set by the taxonomy manager. He/she is responsible for reviewing pending documents, assign documents manually and clear assignments when necessary. However, none of these responsibilities is compulsory. From the My Categories page in Documentum Administrator, the category owner can view all documents assigned to the categories he/she owns, or he/she can display just the documents assigned to the category with a status of Pending. The category owner must approve or reject pending documents, also called Candidate documents, that did not reach a score high enough to be automatically categorized. Note that if the threshold for automatic categorization is equal to the threshold for Candidate documents, then there are no Candidate documents: documents are either automatically categorized or rejected. Once the documents are categorized, either automatically or after approval, they become viewable by end users. If the category owner rejects a pending document, this document is not viewable by end users. For example, in Webtop, even if the document is viewable in a cabinet folder, it is not viewable under the Categories node if it is not categorized. The category owner also has the possibility to manually assign any document to a category. As the taxonomy manager, the category owner can clear assignments, for example, if he/she mistakenly approved a pending document. Integrating Content Intelligence Services CIS server analyzes documents from a Documentum repository and extracts relevant information about them. It can use the results of the analysis to automatically set values of document attributes (Auto Tagging), link the documents into appropriate repository EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 13

14 Overview folders (Auto Categorization), suggest attribute values to WDK-based application users (Web Publisher integration), or any combination of these tasks. Content Intelligence Services is just one piece of your broader Documentum content management solution. There are three common integration scenarios: Organizing your library, page 14 Workflow and lifecycle processing, page 15 Web Publisher integration, page 15 Once CIS processing is complete, you can use the results of the analysis to: Improve searching By extracting information from the document content and adding it to the document s attributes, you transform unstructured data into searchable structured data. Because CIS adds attributes programmatically, you can be sure that they make consistent use of a standard vocabulary. Organize documents for easy navigation Auto Categorization enables you to automatically link documents into a repository folder structure that makes sense to users. Support personalization Personalization server platforms use document attributes to tailor the content displayed to different users. Using CIS enriches a document s attributes with information based on the content of the document, making the subject matter available as a basis for personalization. Organizing your library A key challenge when implementing a content management system is organizing your repository so that users can easily navigate through the document library and find the documents they need. Content Intelligence Services can help address this challenge by: Adding information about a document s content to its attribute set Providing consistency in attribute assignment Organizing documents into a repository folder structure that makes sense to users CIS server periodically checks the repository for the kinds of documents you want it to process. The server retrieves any new or revised documents, analyzes them, and uses the results to programmatically set attribute values or link the documents into appropriate folders. This option is referred to as batch processing. Adding content-related attributes enables users to search for documents based on their subject matter. It also helps support proper categorization of the document by personalization servers. Personalization server platforms use document attributes to tailor the content displayed to different users. Using CIS makes the subject matter available as a basis for personalization. Because CIS sets attribute values 14 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

15 Overview programmatically, you avoid the issues that result when users vary in how they enter values. Work ow and lifecycle processing Content Intelligence Services can participate in a Documentum workflow. CIS server can analyze documents that have reached a particular stage in their lifecycle, then advance them to the next stage. Workflow and lifecycle processing in Documentum are driven by repository attributes. You select documents for CIS processing based on an attribute value that gets set at a particular stage in a document lifecycle. For example, CIS server could retrieve documents that have a Status attribute set to Approved or Ready for Posting. The Auto Tagging feature enables CIS server to update the value of the Status attribute to indicate that is has completed its work. The new value can serve as an entry criteria to move documents to the next stage in their lifecycles. This option is referred to as workflow processing. Web Publisher integration Web authors using Documentum Web Publisher have to enter document attributes before checking the documents into the repository. When you integrate Content Intelligence Services with Web Publisher, CIS server can suggest attribute values. The Web author can choose to use the CIS-suggested attributes rather than manually entering values. This option not only saves time for authors, but also ensures that all authors use the same vocabulary for the attribute values. Web Publisher integration takes advantage of the Auto Tagging features of Content Intelligence Services. Auto Tagging links resource properties to repository attributes. When a Web Publisher user clicks the See CIS Values button, CIS server analyzes the document and provides suggested values for all attributes that have been mapped to resource properties. This option is referred to as online processing or on-demand processing. When a new folder or cabinet is created under the templates section in Web Publisher, the default value of the CIS node is EMPTY. Any value specified at the folder level overrides the name of the CIS Node that is defined globally at the "Web Publisher Admin Settings" level. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 15

16 Overview How to make up for 5.3 legacy tools In the 5.3 release, several legacy tools were available to administer Content Intelligence Services. These tools either were redundant or had become obsolete and the decision has been made not to support them any more. The legacy tools were: CIS Administrator, for advanced processing options. Domain Map Editor, used to define concepts. CIS Control Panel, the administrative console for monitoring CIS server. Scripts, used to generate reports and perform other administrative functions. Note: The CIS database has also been suppressed and CIS-specific data are now stored in the repository. Almost all the functions you used to perform with the legacy tools can still be performed using Documentum Administrator or the TEF utility. The Content Intelligence Services and Working with Taxonomies and Categories chapters in the Documentum Administrator User Guide provides more information on Documentum Administrator for CIS functions. Chapter 4, Using the Taxonomy Exchange Format provides more information about using the TEF utility. The terminology used for legacy tools was slightly different from the current terminology. You can find below a table with the main correspondences. Table 1. Terminology correspondences Legacy tools terminology resource server resource server node concept type and concept domain map token Current terminology repository (also called a docbase ) document set (also called a docset ) category and sub-category taxonomy simple evidence term Using Documentum Administrator instead of legacy tools To avoid users wonder how they can use Documentum Administrator instead of the legacy tools, we provide tables presenting the actions performed in 5.3 and how they can be performed in EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

17 Overview Note: The table below does not indicate the steps to perform each action but points to the procedure that can be performed using Documentum Administrator. The Documentum Administrator User Guide provides more information on detailed procedures. Table 2. Actions performed with CIS Administrator Actions performed with CIS Administrator Defining resource servers Defining resource server nodes Defining submission schedules Submitting documents on demand Resubmitting documents Defining selection queries Creating taxonomy nodes Mapping taxonomy nodes to repository folders Actions performed with Documentum Administrator Enabling and Configuring CIS in the repository Defining document sets Docset scheduling is part of the docset definition. WDK-based applications users can submit documents on demand using the Submit for categorization function. A document that is part of a document set will automatically be processed with the document set if it is not already assigned. To re-process a document, clear the existing assignment using the Clear assignments function. Selection queries are part of the docset definition. Automatically created when creating a category. The mapping is automatically done when creating a category. The following actions, that were available in CIS Administrator, are no longer available or configurable in the 6.0 release: Defining extracted properties Defining conceptual properties Defining constant properties Mapping resource properties to repository attributes Note: In CIS 5.3, the content of the documents was analyzed in order to reveal undefined concepts, based on the frequency of relevant words in the documents. In CIS 6.0, this feature is no longer available. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 17

18 Overview Table 3. Actions performed with Domain Map Editor Actions performed with Domain Map Editor Creating concept type Creating concept Creating tokens Associating evidence with concepts Setting confidence value threshold To add an association to a concept type Actions performed with Documentum Administrator Defining categories Defining categories Defining simple evidence terms Defining simple evidence terms Threshold values are set during the category definition. A relationship to another category (which is not a parent or a child category) can be created when defining simple evidence terms for the category. The following actions, that were available in Domain Map Editor, are no longer available or configurable in the 6.0 release: Creating synonyms Promoting tokens to concepts Reviewing proposed tokens Suppressing auto-generated tokens Other tools that are no longer available It is no longer possible to configure or administer CIS using XML files (Nodes.xml, Notes.xmlinteger.sc, ResProp.xml, ResTaxonomies.xml) or modifying the INI files (ResourceFormatDefinition.ini, ResourceFormatMap.ini, RisParams.ini, CpsHw.ini, RepotResources.ini). Also SmartQueries and Viewpoints can no longer be used. Batch files such as CpsServer and utilities related to the CIS database such as CpsExport have also been dropped. 18 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

19 Chapter 2 Processing Documents This chapter explains how to submit documents to CIS server for processing and how documents are analyzed. CIS server retrieves documents from specific repositories using a Document Query Language (DQL) query that you create. You use the Documentum Administrator application to configure a repository and define document sets. Selecting documents to process The document set definition tells CIS server which documents to retrieve and provides instructions on how to handle the selected documents. The selection of documents based on their location, their object type or their attributes can also be done creating qualifier rules for a taxonomy or a category. However, defining the documents selection in the document set definition is much more performant because it filters documents before analyzing their content. In addition to determining which documents CIS server processes, the document set has a variety of configuration options that control how CIS server processes the selected documents. For example, the document set definition can include a processing schedule that determines how often CIS server processes documents. The document set definitions are slightly different for each scenario: For batch processing, the DQL query for each document set selects all documents that you want CIS server to process. For example, you might write a query to select all documents checked into a specified folder or folders, or all documents of a particular custom object type. For workflow processing, the DQL query selects documents based on a custom attribute whose value gets set as part of the document s lifecycle. For on-demand processing through Web Publisher, you must enter a DQL query in order to define a valid document set, but the query is not used. Web Publisher users submit documents to CIS server interactively. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 19

20 Processing Documents Submitting documents for processing When CIS server is running, it continuously checks the document set schedule to determine when to retrieve documents for processing. When the scheduled time arrives, CIS server connects to the repository, runs the DQL query for the document set, and checks for documents added or updated since the last time it ran the query. Defining submission schedules, page 20 provides information about defining a schedule for a document set. Defining a schedule for a document set is optional. Typically, you will not define schedules if you are using Content Intelligence Services only for its Web Publisher integration. Even in a more batch-oriented implementation scenario, you have the option to process documents on demand. This option can be especially useful during the development and testing phases of system deployment. Submitting documents on demand, page 20 provides information on methods to submit documents on demand. You can also resubmit documents that CIS server has already processed. For example, you may want CIS server to reanalyze documents after you modify the taxonomy definitions. This situation is common during the development and testing phases of your implementation. Resubmitting documents, page 21 provides information about reanalyzing a document. De ning submission schedules The schedule the CIS server follows to poll for submitted documents is set in the configuration of the document sets. Each document set can have a different schedule. Each time the document set is polled for documents, the start time is reset automatically. The new start time is the sum of the polling interval and the amount of time taken to process the documents. For example, suppose the start time is January 1, 2002 at 2 a.m. and the interval is 1 day (1 0:0:0). Suppose also that processing takes 1 hour. The next start time will be January 2 at 3 a.m. Submitting documents on demand There are several ways to submit documents to CIS server on demand: To submit one document, do either of the following: Click the See CIS values button in WDK-based applications. Select Tools > Submit for categorization in WDK-based applications. 20 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

21 Processing Documents To submit a document set, select Tools > Content Intelligence > Start Processing in Documentum Administrator. Documents submitted on demand are regularly processed by the CIS server using the online taxonomies. The documentation related to WDK-based application provides more information about the See CIS values option. Resubmitting documents Sometimes you want CIS server to reanalyze documents. For example, when you have modified the documents of a document set. You can then submit again the document(s) using one of the procedures described in the above section, Submitting documents on demand, page 20. If a submission schedule has been defined for the document set containing the modified documents, then they will be automatically re-processed the next time the schedule runs. When a schedule process starts, it automatically retrieve the last update of the document set and it checks whether the documents version has been modified since its last run. If the documents have not been modified, CIS server will not start any new process. If you want to process again documents against a new taxonomy, use the Clear assignments function on the document set first. Understanding information extraction Extracted properties are properties whose values CIS server can extract directly from the document itself or from the metadata of the document. An example of an extracted property is Document Size: CIS server can get this information directly from the properties of the physical file without performing any calculations. When a document is processed, its format is automatically recognized by the CIS server. The CIS server can then extract the properties that are expected to be available for the document. Some property values can only be extracted if they have been filled in by the document author. For example, the Title of a PDF document is entered by the author. The Appendix A, Extracted properties provides information about properties automatically extracted depending on the document format. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 21

22 Processing Documents Understanding conceptual analysis When property values have been extracted, the document is analyzed using the taxonomy definition. It identifies the evidence CIS server looks for to determine whether a document includes the category or concept. The confidence values assigned to the various pieces of evidence determine when CIS server signals a hit for the category. When CIS server analyzes a document, it automatically computes a score for each keyword and a score for the whole expression. The expression score is then compared to the category thresholds to define the document as: not assigned if the candidate threshold is not reached. candidate or pending if the candidate threshold is reached but not the on-target threshold. assigned if the on-target threshold is reached. How the concept score is computed Each category definition lists the words and phrases used as evidence that a document belongs to the category. These words and phrases are called evidence terms. We can define three types of evidence terms: simple evidence terms Words or phrases (words in order or not) are simple evidence terms and can also be referred as keywords. compound evidence terms A compound evidence term is an expression, a group of words and phrases that work together to indicate category membership. None of the words or phrases by themselves can be considered as an evidence term. Only their combination can add to the confidence score. patterns A pattern is also known as a regular expression and describes a set of strings. For example, patterns can match credit card numbers, phone numbers or Social Security numbers. First, when defining keywords, remember that keywords are words. They consist of characters, mostly letters; letters are elements of an alphabet system not any character in a character map. However, you can add digits to your keywords or punctuation marks if they are part of the word as, for example, in merry-go-round. If you want to create a keyword that includes only digits, uses a character such as a symbol or a currency sign, or to define an acronym, you must define it as a pattern. Whereas simple and compound evidence terms can be defined in Documentum Administrator, the pattern analysis must be configured manually. Configuring the pattern analysis, page 24 provides information on how to define patterns. 22 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

23 Processing Documents Most of the time, the first definition of a category is a list of simple evidence terms, it is called an evidence set. Each compound term can also be seen as an independent evidence set. When CIS server analyzes a document, it looks for these terms. It then computes a score for each evidence term in the document based on its confidence value. Then CIS server calculates a score for the evidence set. Finally, CIS server compares the evidence set score with the category thresholds to determine whether the document should be assigned to the category or not, or left as a pending candidate. Each evidence term in the category definition has a confidence value assigned to it. The confidence value specifies how certain CIS server can be about scoring a hit for a document when it contains the term. For example, if a document includes the text IBM, CIS server can be nearly certain that the document relates to the category International Business Machines. Therefore, the confidence level for the term IBM is High. Other pieces of evidence may suggest that the category might be appropriate. For example, if a document includes the text Big Blue, CIS server cannot be certain that it refers to International Business Machines. The confidence level is Low, meaning that CIS server should score a hit for the category International Business Machines only if it encounters the text Big Blue and other evidence of the same category in the document. The evidence set made of terms with a low confidence level is called a compound term. You can also assign negative confidence values to evidence terms. Adding evidence with a negative confidence value tells CIS server to reduce the document score. The word or phrase is negative evidence of the concept. For example, suppose you have a category for the company Apple Computers. The term Apple is certainly evidence of the category. However, if the term fruit appears in the same document, you can be fairly sure that Apple refers to the fruit and not the company. To capture this fact, you would add fruit as evidence to the Apple Computers category with a negative confidence value. The confidence values for evidence terms are integers between 0 and 100. When you set confidence values in Documentum Administrator, you can choose a predefined confidence level or enter a number directly. The predefined values are: High Equivalent to the confidence level 75. Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Supporting This confidence level by itself does not cause CIS server to score a hit for a document. However, it increases the confidence level of other evidence found in the same document. Exclude If one of the evidence terms found in a document has this confidence level, then the document will never be assigned to this category. If the resulting score exceeds or meets the category s on-target threshold, CIS server assigns the document to the category. If the score is lower than the on-target threshold but higher than or equal to the candidate threshold, CIS server assigns the document EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 23

24 Processing Documents to the category as a Pending candidate; the category owner must review and approve the document before the assignment is complete. If the score falls below the candidate threshold, CIS server does not assign the document to the category. Understanding auto categorization When CIS server processes a document, it performs a semantic analysis of the document based on the taxonomy you have defined. From the analysis, Content Intelligence Services knows what concepts are covered in the document. With the Auto Categorization feature, Content Intelligence Services uses this information to link the document to appropriate category folders in your repository structure. CIS server uses a taxonomy to analyze the content of documents. The list of concepts in the taxonomy identifies which concepts you want CIS server to look for in each document. The definition of each concept tells the server how to determine whether that concept is covered in a document what evidence to look for. When CIS server completes its analysis of a document, it has a list of the concepts discussed in that document. To use the Auto Categorization feature, the taxonomy structure should mirror a folder structure in the repository. To ensure that the two structures remain parallel, a repository folder is automatically created when you define a category. Auto Categorization occurs automatically when documents are processed. When a document is assigned to a category, it is automatically linked to the repository folder representing that category. Con guring the pattern analysis This section describes the Pattern analyzer feature and includes the following topics: Pattern analysis, page 24 Limitations, page 25 Using patterns in rules, page 25 Defining Patterns, page 25 Pattern analysis Pattern support uses a standard form of regular expressions. A regular expression, often called a pattern, is an expression that describes a set of strings. The simple and compound evidence terms used for content analysis and defined with Documentum 24 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

25 Processing Documents Administrator are words and phrases. As an advanced feature, you can define patterns to be used as evidence terms to describe a category. Patterns can match credit card numbers, phone numbers, Social Security numbers, etc. For information about simple and compound evidence terms, see the Content Intelligence chapter in the Documentum Administrator user guide. Limitations The definition of a pattern is done using a standard regular expression language. However, it is not a natural language. Besides, searching for many patterns can slow down performance significantly. Two or three patterns will not cause noticeable performance degradation, but it will add up. Using patterns in rules Patterns can then be used as keywords when you define the evidence terms for a category. CIS processes patterns by first compiling them and applying them to the document. If the pattern value is matched, a pattern token is inserted. For example, if the Phone pattern above is found in the text, the token $Phone will automatically be inserted. You can treat pattern tokens just like any keywords when defining the CIS taxonomy. In the case of our Phone example, if you set up a Phone category, simply add a simple evidence term $Phone to the list of evidence. It is also possible to create compound evidence terms. In the Phone example, you could specify that it is necessary to find the $Phone and another term (such as $CreditCard) to classify the document to the category. De ning Patterns Patterns are defined in the property file: patterns.properties. The following procedure describes the required steps to define a pattern. To edit the patterns.properties le: 1. Locate the patterns.properties in the directory <Documentum domain in BEA WebLogic directory>\upload\cis\cis.ear\app-inf\classes. 2. In a text editor, open patterns.properties. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 25

26 Processing Documents 3. Since pattern analysis may impact performance of the content analysis, it can be turned off. By default, the feature is enabled. To enable or disable pattern analysis, use the top level setting pattern.processing.enabled. 4. To see additional tracing information in the log file, set the option tracing.enabled to true. This information only applies to pattern loading and processing. 5. Append the required data for defining patterns. For each pattern, you can define the following parameters: Scope reserved for future use, the default value must be (Global). Value value of the actual pattern. Note: Pattern value syntax corresponds to the REGEXP syntax with one exception: each backslash character \ must be escaped by placing another backslash character \ in front of it: \\. For more information about patterns and regular-expression constructs, you may refer to Besides, try not to describe patterns too general that match many words in numerous documents since this would decrease performance. Token value to be added to the document where pattern matches. A guideline to define tokens is to write them with the $ character at the beginning, such as $Phone or $CreditCard. Sample this value can be used to run a special unit test: TCisPatternAnalyzer. This unit test will automatically load and validate all patterns in this file. Note: You may want to create a QA utility based on this unit test. The Sample parameter is ignored by the runtime processing engine. Description pattern creator s notes. For each parameter, you have to indicate the incremented number of the pattern. For example, when defining the third pattern, its parameters should end with 3 such as: pattern.scope.3, pattern.value.3, etc. 6. Save and close patterns.properties. 7. Once you defined patterns, you can use them as evidence terms for a category in Documentum Administrator. 26 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

27 Processing Documents Example 2-1. Pattern description for US Social Security Number pattern.scope=(global) pattern.value=^\\d{3}-\\d{2}-\\d{4} pattern.token=$ssn pattern.sample=text Text pattern.description=u.s. Social Security Number regular In this example, you can see that the pattern is made of three digits: \\d{3}, then two digits: \\d{2}, then four digits: \\d{4}, separated by hyphens: -. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 27

28 Processing Documents 28 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

29 Chapter 3 Administering Content Intelligence Services Con guring Content Intelligence Services The following procedure provides an overview of the steps necessary to get Content Intelligence Services up and running. To con gure Content Intelligence Services 1. Install Content Intelligence Services by following the instructions in the Content Intelligence Services Installation Guide. 2. Start CIS server. See Administering the CIS server, page 30 for details. 3. Enable CIS in the repository. The Documentum Administrator User Guide provides information on this procedure. The properties you specify when you enable CIS (such as the hostname for the test and production servers) can later be modified in the Configure CIS window. 4. Create or import a taxonomy. Content Intelligence Services offers two ways to create the taxonomy that it uses to categorize documents: Import a taxonomy in taxonomy exchange format (TEF) Build a taxonomy using Documentum Administrator Importing a TEF taxonomy is the best approach if you are creating a taxonomy based on an industry-standard thesaurus. You can download several prebuilt TEF taxonomies for different subject areas. The Data Sheet of Content Intelligence Services provides a list of these taxonomies, it is available from the EMC Software site: > Products > Software A to Z > Documentum Content Intelligence Services > Data Sheet. You can import these taxonomies as is, or you can customize the TEF files before you import them. For information about downloading and importing TEF taxonomies, see Importing taxonomies in Taxonomy Exchange Format, page 41. Documentum Administrator is the preferred tool for creating a custom taxonomy. The Documentum Administrator User Guide provides more information on how to create taxonomies with DA. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 29

30 Administering Content Intelligence Services 5. Create the necessary document sets to select the documents that you want to be automatically categorized. 6. Synchronize the taxonomy definitions in the Documentum repository with the taxonomy snapshots that will be used for the classification processing. The CIS objects you import or create with Documentum Administrator are saved in the repository containing the documents, categories, evidence terms that CIS will use in its processing. Before CIS server can use them, you need to synchronize the snapshots of the taxonomies and the document sets. Each time the CIS server process a document set, the CIS server reads the document set definition. However, for scheduled document sets, you need to synchronize them when their schedule has been modified. If the document set is not synchronized, the CIS server does not know that the schedule has been updated. 7. If you want to integrate CIS with Web Publisher, configure Web Publisher so that it can locate CIS server. The Web Publisher documentation provides details on how to integrate CIS with Web Publisher. The main server configuration file is cis.properties. You may need to modify the parameters available in the cis.properties file for example to change the directory where the CIS data are stored. To modify cis.properties, page 31 indicates the steps to modify the cis.properties file. Administering the CIS server This section provides instructions for basic CIS administration activities. It includes the following activities: Starting/Stopping CIS server, page 31 Configuring CIS server, page 31 Monitoring CIS server processing, page 34 CIS Server is deployed in a dedicated managed server in BEA WebLogic Application Server. For WebLogic experts, administration tasks can be performed using the WebLogic administration console. Note that when you stop CIS server, you also need to stop its JVM. For this reason, a dedicated server is recommended. 30 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

31 Administering Content Intelligence Services Starting/Stopping CIS server Since Content Intelligence Services are installed as a Windows Service "Documentum Content Intelligence Sercices", you can go to the Services section of the Computer Management dialog box to start and stop the CIS server. Another possibility to start and stop the CIS server is using the WebLogic server scripts: <Documentum domain in BEA WebLogic directory>\startcis.cmd <Documentum domain in BEA WebLogic directory>\stopcis.cmd The default Documentum domain in BEA WebLogic directory is C:\Documentum\bea9. 2\domains\DctmDomain. Con guring CIS server As for CIS server start and stop, the CIS server can configured in different ways. As an application, page 31 As a Windows service, page 34 As an application A first level of configuration can be set modifying the properties file of the CIS application. Properties files can be found at <Documentum domain in BEA WebLogic directory>\upload\cis\cis.ear\app-inf\classes. They consist of: cis.properties. Modify this file as described in To modify cis.properties, page 31. dfc.properties. The Documentum Foundation Classes documentation provides more information about DFC parameters. patterns.properties. Use this file to define patterns as described in Pattern analysis, page 24. log4j.properties. Use this file to configure Log4j. CIS server log files, page 34 provides more details about log files. To modify cis.properties 1. Stop CIS server. 2. Locate the cis.properties file in <Documentum domain in BEA WebLogic directory>\upload\cis\cis.ear\app-inf\classes. 3. Open the cis.properties file with a text editor. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 31

32 Administering Content Intelligence Services 4. Update the parameter setting as needed. The Table 4, page 32 provides details on the available configuration parameters. 5. Restart CIS server. The following table provides details on the configuration parameters available in the cis.properties file. Table 4. Con guration parameters in the cis.properties le Parameter cis.server.port under the Server port section Description The server port number dedicated to CIS. The port number defined here must match the one defined in DA. In DA, you can specify the port number with the server host name. Default value (for DA): cis.server.port=18460 cis.server.repository under the Docbase settings section cis.server.credentials.dir under the Docbase credentials files directory section The name of the repository that the CIS server will connect to. This repository contains all the taxonomy definitions and all the documents to classify. The path to the directory where files containing login and passwords for repository access are stored. Default value: cis.server.credentials.dir=upload/cis/ cis.ear/cis.war/repodata/authentication cis.server.synchronization.dir under the Synchronized data section The path to the directory where repository data are synchronized locally on the CIS server. This directory must exist. By default: cis.server.synchronization.dir=upload/ CIS/cis.ear/cis.war/repodata/syncdata cis.server.docstatus.dir under the Document processing status section The path to the directory where the document processing statuses (classification information) are kept. This directory must exist. By default: cis.server.docstatus.dir=upload/cis/ cis.ear/cis.war/repodata/docstatus cis.server.patterns.file under the Patterns (regular expressions) section The configuration file that contains pattern definitions. This file is loaded by the CIS server by its path in the classpath. By default: cis.server.patterns.file=patterns. properties 32 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

33 Administering Content Intelligence Services Parameter cis.server.execution.threads cis.server.scheduling.threads under the Thread pools section cis.linguistic.language.default cis.linguistic.stemming.allowed under the Linguistic configuration section cis.vantage.data.dir under Configuration specific to the Vantage linguistic engine section cis.server.limit.max_file_size under Limit settings section Description Respectively, the maximum number of threads allocated to the execution of classification tasks and to the scheduling of document sets. By default, the maximum is set to 5 for both parameters. These parameters set the default language for the stemming feature and whether it is activated or not. By default: cis.linguistic.language.default=english cis.linguistic.stemming.allowed=true Stemming Capability, page 36 provides more information on the linguistic configuration. The path to the directory containing the linguistic resources. By default: cis.vantage.data.dir=upload/cis/cis. ear/app-inf/lib Stemming Capability, page 36 provides more information on the language dictionaries. The maximum size (in bytes) of files processed by CIS server. Files whose size is higher than this limit are not processed and an error is logged in the log file. By default: cis.server.limit.max_file_size= cis.classification.limit. max_content_size under Limit settings section The maximum size (in bytes) of extracted content per file processed. When CIS server processes files whose content s size is higher than this limit, the content is partially extracted up to the limit and a warning is logged in the log file. By default: cis.classification.limit.max_content_ size= Note: Archive files are processed as one file. The limit applies to the archive file s content size not to the content size of the files it contains. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 33

34 Administering Content Intelligence Services As a Windows service The third level of configuration can be set when considering CIS server as a Windows service. Edit the <Documentum domain in BEA WebLogic directory>\serviceconfig\cis\conf\wrapper.conf file and modify the parameters as required, for example, the Wrapper Logging Properties. Monitoring CIS server processing You can monitor CIS server activity using CIS server log files, which contain a record of the actions of CIS server. Additional log files can be configured using the logging utility Log4j or the log files generated by BEA WebLogic application server. CIS server log les There are three server log files: cis.log, which contains information about the operations of the CIS server, can be found at <Documentum domain in BEA WebLogic directory>\servers\<managed server name>\data\cis\logs\cis.log. DmCISService.log, which contains information about dates and time when starting and stopping the server as a Windows service, can be found at <Documentum domain in BEA WebLogic directory>\serviceconfig\cis\dmcisservice.log. DctmServer_CIS_<host name>.log, which contains information about the operations of the WebLogic application server hosting the CIS server, can be found at <Documentum domain in BEA WebLogic directory>\servers\<managed server name>\logs\<managed server name>.log. By default, the <Documentum domain in BEA WebLogic directory> is C:\Documentum\bea9. 2\domains\DctmDomain. The <managed server name> is DctmServer_CIS_<host name> where <host name> is the name of the CIS host computer. Note: It can also be interesting to refer to the Documentum logs for the DFC connection for example. They are located in the logs folder in the Documentum directory. WebLogic also provides logs such as logs for the domain. The BEA WebLogic application server documentation provides details about log files generated by BEA WebLogic application server. 34 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

35 Administering Content Intelligence Services The Log4j setup for CIS server logging is the file <Documentum domain in BEA WebLogic directory>\upload\cis\cis.ear\app-inf\classes\log4j.properties. The Log4j project website ( provides information on configuring log statements. Common installation errors This section provides examples of the most common error messages. The error messages appears in the cis.log file or as traces in the console. Example 3-1. CIS PATH not speci ed (cis.log) ERROR :11:41,156 com.documentum.cis.service.internal. scheduling.scheduler[schedulerexecutor_4] - State machine for docset id 09a98afc reported failure java.lang.unsatisfiedlinkerror: no vantage_linguistic_engine in java.library.path Example 3-2. The DA version and the CIS server version are not compatible. (cis.log) ERROR :49:58,937 com.documentum.cis.service.internal.communication.commandreader [Stream reader (clientid=1)] - IO error when receiving a command from a remote client (clientid=1)java.io.invalidclassexception: com.documentum.cis.service.internal.command.synchronizetaxonomycommand; local class incompatible: stream classdesc serialversionuid = , local class serialversionuid = at java.io.objectstreamclass.initnonproxy(objectstreamclass.java:519) at java.io.objectinputstream.readnonproxydesc(objectinputstream.java:1546) at java.io.objectinputstream.readclassdesc(objectinputstream.java:1460) at java.io.objectinputstream.readordinaryobject(objectinputstream.java:1693) at java.io.objectinputstream.readobject0(objectinputstream.java:1299) at java.io.objectinputstream.readobject(objectinputstream.java:339) at com.documentum.cis.service.internal.communication.commandreader. readcommand (CommandReader.java:86) at com.documentum.cis.service.internal.communication.commandreader.run (CommandReader.java:69) at java.lang.thread.run(thread.java:595) ERROR :49:58,937 com.documentum.cis.service.internal.communication.commandreader [Stream reader (clientid=1)] - Unsupported command java.lang.classcastexception: com.documentum.cis.service.command. CommandExecutionMode at com.documentum.cis.service.internal.communication.commandreader. readcommand (CommandReader.java:86) at com.documentum.cis.service.internal.communication.commandreader.run (CommandReader.java:69) at java.lang.thread.run(thread.java:595) ERROR :49:58,953 com.documentum.cis.service.internal.communication.commandreader [Stream reader (clientid=1)] - Unsupported command java.lang.classcastexception: java.lang.string at com.documentum.cis.service.internal.communication.commandreader.readcommand EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 35

36 Administering Content Intelligence Services (CommandReader.java:86) at com.documentum.cis.service.internal.communication.commandreader.run (CommandReader.java:69) at java.lang.thread.run(thread.java:595) Example 3-3. The client version (DA 6 SP1) and the CIS server version (6) are not compatible. (error in DA) Error in updating content intelligence configurations Response not received for command=version negotiation (maxversion=1) with id=1 Example 3-4. The authentication against the repository failed (console) CIS server starting... Invalid or no repository credentials. Waiting for credentials before CIS server can actually start. CIS server starts listening on port When the CIS server starts, it checks the user credentials against the repository before opening a session. If no credentials are found or if they are invalid, the CIS server starts in a restricted mode that only allows receiving new or updated credentials. The user cannot launch any classification run but he/she can change the credentials in Documentum Administrator. When the CIS server receives the valid credentials, it tries to connect to the repository. If successful, it switches to full mode and the following message appears in the console: Credentials ok, CIS server connected to Repository. Stemming Capability CIS is fully Unicode compliant; this enables the classification of documents written in any language. The stemming capability of Content Intelligence Services allows you to use the stemming for documents in English, French, German, Spanish, Italian, Portuguese, Danish, Dutch, Norwegian or Swedish. The stemming feature is the ability to recognize that fishing, fished, fish, and fisher have the same root word fish. By default, only the English dictionary is installed with CIS. The other language dictionaries have to be downloaded and installed separately. Two dictionary sets are available, one for northern European languages, the other one for southern European languages. Note: CIS server processes documents using the stemming capability only in the configured language. The major sections in this chapter are: 36 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

37 Administering Content Intelligence Services Understanding the dynamics, page 37 Installing language dictionaries, page 37 Configuring CIS to work with other languages, page 38 Understanding the dynamics You need to supply CIS with the initialization data it requires to select the correct combination of language and software provider to use on startup. This initialization data includes the following: The default language whether English, French, German, Spanish, Italian, Portuguese, Danish, Dutch, Norwegian or Swedish. The indication of whether the stemming is activated or not. The directory of the provider for the selected language dictionary. Currently, the only provider that CIS uses is Vantage. This information is supplied to CIS in the cis.properties file. When English is the primary language, as is by default, the content of this file is as shown below. Example 3-5. Extract of the cis.properties le # The default language of the linguistic engine. cis.linguistic.language.default=english # Whether the word stemming feature is allowed globally (true/false). cis.linguistic.stemming.allowed=true # The Vantage data directory containing linguistic resources. # E.g. cis.vantage.data.dir=<cis_root>/web-inf/lib cis.vantage.data.dir=upload\cis\cis.ear\app-inf\lib When CIS starts, it looks under the header Linguistic configuration for the key cis.linguistic.language.default. Then it looks at the cis.linguistic.stemming.allowed key and finds whether stemming is allowed for this language. Finally it looks under the Configuration specific to the Vantage linguistic engine to find the cis.vantage.data.dir key in order to get the directory containing the dictionary resources. At this time, Vantage is the only linguistic dictionary provider for CIS. Installing language dictionaries This section describes how to download and install language dictionaries for CIS. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 37

38 Administering Content Intelligence Services To install language dictionaries: 1. Download the Documentum CIS Northern European Language Dictionary for Danish, English, Dutch, Norwegian or Swedish or the Documentum CIS Western European Language Dictionary for German, English, Spanish, French, Italian or Portuguese from the Documentum download site ( to a temporary directory. If you cannot locate your password, click on Password finder at the bottom of the page. Use your address as your user name. 2. Unzip the language dictionary to a temporary directory. 3. Stop the CIS server. 4. Copy the extracted files from the temporary directory to the directory where CIS is installed. If the dictionary files are already existing in the CIS folder, click Yes to overwrite the existing files. 5. Set the cis.properties file to the right language as described in Configuring CIS to work with other languages, page 38 and restart the machine. 6. Restart the CIS server. Con guring CIS to work with other languages Configuring CIS to run with required languages involves modifying the cis.properties file as shown in the procedures below. (It may be a good idea to keep a copy of the contents before making the changes, in case you want to revert to English.) By default, the cis.properties file is available in the directory <Documentum domain in BEA WebLogic directory>\upload\cis\cis.ear\app-inf\classes. However, it can be located in another folder as long as the Java classpath points to the correct folder. The procedure given below describes how you can change the default language to French. Similarly, you can change the default language to any of the other available languages. 38 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

39 Administering Content Intelligence Services To change the primary language to French 1. Open the cis.properties file. 2. Locate the cis.linguistic.language.default property. 3. Set its value to french or fr. cis.linguistic.language.default=french The language value can either be the full language name in lowercase and in english, such as french, or the two-letter ISO language code such as fr. 4. Restart CI Services. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 39

40 Administering Content Intelligence Services 40 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

41 Chapter 4 Using the Taxonomy Exchange Format This chapter describes the XML elements you can use to write a taxonomy in taxonomy exchange format (TEF). For more information about the objects defined by the TEF elements, see the Documentum Administrator User Guide. A TEF file defines the structure of one or more taxonomies. To import all or part of the structure into the repository for use in CIS processing, you must also create a TEF action file that specifies what actions to take. The TEF processor will use the IDfTypedObject calls (such as getint() and setint()) to access attribute data. Export takes information from the repository and creates a tef xml file. Import takes the tef xml file and creates repository information. The tef xml schema accommodate subtypes and their attributes. See Taxonomy Exchange Format Action Files, page 91. Note: Before importing taxonomies, make sure you enabled CIS functionality in Documentum Administrator. The Enabling Content Intelligence Services section in the Documentum Administrator User Guide provides more information on this procedure. Importing taxonomies in Taxonomy Exchange Format Taxonomy exchange format (TEF) is an XML format that describes a complete taxonomy: its categories, its hierarchical structure, and the evidence used to assign documents to categories during automatic processing. TEF is actually comprised of two related XML schemas: The core TEF schema (defined by tefschema.xsd) defines the structure of one or more taxonomies The TEF action schema (defined by tefactionschema.xsd) defines the actions used to transfer the taxonomy definitions from a TEF file into a Documentum repository and vice versa EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 41

42 Using the Taxonomy Exchange Format To import a TEF taxonomy: 1. Write or obtain a TEF taxonomy. Documentum provides a set of taxonomies defined in TEF. They can be downloaded from the Powerlink site ( You can also write a TEF taxonomy map using an XML or text editor. The TEF format is described in the Taxonomy Exchange Format Action Files, page Create a TEF action file. The TEF action file instructs the TEF utility what actions to take on the definitions in the TEF taxonomy file. The TEF action file format is described in the Taxonomy Exchange Format Action Files, page Import the taxonomy thanks to the TefUtil utility: a. Open the directory that contains the taxonomy XML files. b. Enter this command into a Command Prompt window: java -cp "<dfc.data.dir>\bea9.2\domains\dctmdomain\upload\cis\ cis.ear\app-inf\classes; <dfc.data.dir>\bea9.2\domains\dctmdomain\ upload\cis\cis.ear\app-inf\lib\ci.jar; <dfc.data.dir>\bea9.2\ domains\dctmdomain\upload\cis\cis.ear\app-inf\lib\dfc.jar" com.documentum.ci.tools.tefutil -Docbase:<docbase> -UserName:<login> -Password:<password> -TefActionFile:tef\<TEF action file> where <dfc.data.dir> is the installation directory for the Documentum Foundation Classes (DFC), by default, it is C:\Documentum, <TEF action file> is the name of the TEF action file, <docbase> is the name of the repository into which you want to import the taxonomy, <login> and <password> are the Documentum user name and password for logging into the repository. The TEF action file includes references to TEF taxonomy definition files and to the schema definition file (XSD). TefUtil expects to locate these files in the current working directory; if the filename includes a relative path specification, it must be the relative path from the current working directory. To import any of the taxonomies delivered with CIS, run TefUtil from the directory containing the TEF action file. The type attribute determines what type of object needs to be created in the repository. The default for type is dm_category or dm_taxonomy. The following sections describes the TEF elements. 42 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

43 <category> <category> Purpose Defines a category Diagram Attributes Table 5. <category> Element Attributes Attribute name classname type (for categorysubtype) Description Name of the category, which must be unique among <category> elements of the same category class Name of the category class assigned to this category Subtype of the category The default for type is dm_category or dm_taxonomy. During importing, the type attribute determines what type of object needs to be created in the repository. During EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 43

44 <category> exporting, all attributes are exported from the dm_category/dm_taxonomy subtype to <extended_data>. 44 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

45 <category> Children <details> <definition> <category> <categorylink> Parents <taxonomy> <category> Usage Notes The <category> element represents a category. <category> elements are valid within <taxonomy> elements and within other <category> elements. The nested structure defines the hierarchy of the taxonomy. Every category belongs to a category class. The class determines the default confidence levels used for different types of evidence and the document attribute into which CIS server write the name of the category when it assigns a document. A <category> element is divided into three main parts: <details> provides descriptive information about the category and how it is used and managed. <definition> provides the evidence and qualifying conditions that CIS server uses to determine which documents to assign to the category. <category> and <categorylink> elements define subcategories. Example of <category> <category name="web Content Management Suite"> <details title="web Content Management Suite"> <description>the suite of Documentum products for managing Web content </description> <owners/> <operations> <operation type="user_browse"/> <operation type="manual_assignment"/> </operations> <languageinfo> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 45

46 <category> <supportedlanguage languagecode="es" translatedname="translation 1"/> <supportedlanguage languagecode="jp" translatedname="translation 2"/> </languageinfo> </details> <definition candidatethreshold="60" ontargetthreshold="90"> <evidence evidencepropagation="low" impliedkeyword="33"> <evidenceset> <keyword name="web Content Management" confidence="high" phraseorderexact="true" stem="false"/> <keyword name="wcm" confidence="high"/> </evidenceset> </evidence> </definition> <category name="web Publisher" classname="generic">... [Category definition] </category> <category name="site Deployment Services" classname="generic">... [Category definition] </category>... </category> Example of <CategorySubtype> The Brand Loyalty is a category subtype called dm_category_subtype. This dm_category_subtype has twelve attributes: Attribute for each of the six supported types (boolean, integer, string, id, time and double) repeating and non repeating examples for each type. <category name="brand Loyalty" type= dm_category_subtype > <details title="brand Loyalty"> <extended_attributes> <attribute name="bool single"> <value>true</value> </attribute> <attribute name="int single "> <value>1</value> </attribute> <attribute name="str single "> <value>test string 1</value> </attribute> <attribute name="id single "> <value>98adc6236f527be1</value> </attribute> <attribute name="time single "> <value>thu Feb 01 11:30:01 PST 2001</value> </attribute> <attribute name="double single "> <value>1.0</value> </attribute> <attribute name="bool repeating"> <value>true</value> <value>false</value> <value>true</value> 46 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

47 <category> </attribute> <attribute name="int repeating "> <value>1</value> <value>2</value> <value>3</value> </attribute> <attribute name="str repeating "> <value>test string 1</value> <value>test string 2</value> <value>test string 3</value> </attribute> <attribute name="id repeating "> <value>98adc6236f527be1</value> <value>98adc6236f527be2</value> <value>98adc6236f527be3</value> </attribute> <attribute name="time repeating "> <value>thu Feb 01 11:30:01 PST 2001</value> <value>sat Mar 02 12:30:02 PST 2002</value> <value>thu Apr 03 13:30:03 PST 2003</value> </attribute> <attribute name="double repeating "> <value>1.0</value> <value>2.2</value> <value>3.4</value> </attribute> </extended_attributes> </details> <definition> </definition> </category> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 47

48 <categorydefaults> <categorydefaults> Purpose Provides default values for confidence levels and evidence propagation Diagram Children <impliedkeyworddefaults> <keyworddefaults> <evidencepropagation> <categoryevidencedefaults> Parents <class> Usage Notes Each piece of evidence for a category has three associated attributes that control how CIS server handles it: 48 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

49 <categorydefaults> confidence A confidence level that determines how much CIS server adds to a document s score when the evidence is found stem True or false setting that determines whether CIS server uses stemming to recognize other forms of the word phraseorderexact True or false setting that determines whether the words in a multiple word phrase must appear in exact order or in random order The <categorydefaults> element specifies the default values for these options. Each subelement sets the default values for a particular type of evidence (keyword, implied keyword, propagated evidence, and linked category evidence). The default values are used when the <keyword> or <categoryevidence> element used to declare the evidence does not include a value for the options. Example of <categorydefaults> <class name="class"> <details source="source" targetattribute="target" title="title"> <description>class description</description> </details> <categorydefaults> <impliedkeyworddefaults confidence="100" stem="true" phraseorderexact="false"/> <keyworddefaults confidence="high" stem="true" phraseorderexact="false"/> <evidencepropagation type="@parent" confidence="medium"/> <categoryevidencedefaults confidence="off"/> </categorydefaults> </class> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 49

50 <categoryevidence> <categoryevidence> Purpose Includes another category s evidence as part of the evidence for the parent category Attributes Table 6. <categoryevidence> Element Attributes Attribute name classname confidence Description Name of the category whose evidence you want to use as evidence for the parent category as well Category class assigned to the linked category Level of confidence to assign to evidence from the linked category Children None Parents <evidenceset> Usage Notes Categories can include other categories as evidence: when a document is assigned to one category, CIS server can use that assignment as evidence for a related category. For example, when a document is assigned to the category Documentum Content Intelligence Services, you might want it also assigned to the category Documentum. To accomplish this, you link the category Documentum Content Intelligence Services into an evidence set for the category Documentum. Like all evidence, category link evidence has a confidence value associated with it, telling CIS server how much to add to 50 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

51 <categoryevidence> the document s overall score for the current category when the document is assigned to the linked category. If you do not include values for one or more of the attributes, their values are inherited from the <categoryevidencedefaults> element of the category class. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 51

52 <categoryevidencedefaults> <categoryevidencedefaults> Purpose Provides the defaults for handling evidence from other categories linked into an evidence set Attributes Table 7. <categoryevidencedefaults> Element Attributes Attribute confidence Description Confidence level value to apply to category evidence links by default Children None Parents <categorydefaults> Usage Notes For the confidence attribute, you can enter a predefined confidence level or enter a number directly. The predefined values are: Certain Equivalent to the confidence level 100. High Equivalent to the confidence level 75. Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Support This evidence by itself does not cause CIS server to assign the document to the category. However, it increases the confidence level of other evidence found in the same evidence set. Equivalent to the confidence level EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

53 <categoryevidencedefaults> Exclude This evidence by itself prevents CIS server from assigning the document to the category, regardless of how much other evidence appears. Equivalent to the confidence level 100. Off No weight is added to the document confidence score for this evidence. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 53

54 <categorylink> <categorylink> Purpose Links an existing category into the hierarchy of the taxonomy Attributes Table 8. <categorylink> Element Attributes Attribute name classname Description Name of an existing category Name of the category class assigned to the linked category Children None Parents <taxonomy> <category> Usage Notes The <categorylink> element enables you to include a category in more than one place in the hierarchy. You use the <category> element to define the category and its evidence structure once, then use <categorylink> to link the category into other locations in the taxonomy. Example of <categorylink> <taxonomy name="products" classname="generic" taxonomyversion="version 1"> <details title="products"> <description>products taxonomy</description> 54 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

55 <categorylink> </details> <definition candidatethreshold="50" ontargetthreshold="80"/> <category name="web Content Management Suite" classname="generic">... [Category definition] </category> <categorylink name="enterprise Content Management Suite" classname="generic"/> </taxonomy> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 55

56 <class> <class> Purpose Defines a category class Diagram Attributes Table 9. <class> Element Attributes Attribute name Description Name of the category class, which must be unique among <class> elements Children <details> <categorydefaults> Parents <tef> 56 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

57 <class> Usage Notes The <class> element defines a category class. Each CIS category is assigned to a class, which determines the category s default confidence levels and evidence propagation behavior. The category class also identifies the document attribute to which category assignments are written if the Assign as Attributes option is active. <class> elements appear as children of the <tef> element, outside of the taxonomies and categories. A <class> element has two subelements: <details> provides descriptive information about the class and sets the document attribute into which CIS server writes category assignments when Assign as Attributes is active. <categorydefaults> sets default values for how CIS server handles evidence for categories of this category class. Example of <class> <class name="generic"> <details source="source" targetattribute="keywords" title="generic class"> <description>category class for basic categories</description> </details> <categorydefaults> <impliedkeyworddefaults confidence="100" stem="true" phraseorderexact="false"/> <keyworddefaults confidence="high" stem="true" phraseorderexact="false"/> <evidencepropagation type="@parent" confidence="medium"/> <categoryevidencedefaults confidence="off"/> </categorydefaults> </class> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 57

58 <de nition> <de nition> Purpose Identifies the set of documents belonging to a category Diagram Attributes Table 10. <de nition> Element Attributes Attribute ontargetthreshold candidatethreshold Description Number between 0 and 100 indicating the confidence score that a document must have for CIS server to assign it to the category. If this optional attribute is not included, the category uses the on-target threshold specified for the <taxonomy> it belongs to. Number between 0 and 100 indicating the confidence score that a document must have for CIS server to assign it to the category as a candidate requiring approval. If this optional attribute is not included, the category uses the candidate threshold specified for the <taxonomy> it belongs to. 58 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

59 <de nition> Children <evidence> <qualifiers> Parents <taxonomy> <category> Usage Notes The <definition> element supplies the criteria that CIS server uses to determine which documents to assign to the parent category or taxonomy. The <qualifiers> subelement defines qualifying conditions that a document must meet to be considered for inclusion. The <evidence> subelement provides the evidence and confidence values that CIS server uses to assign a confidence score to the document. If a <taxonomy> element includes any <qualifiers>, the specified qualifying conditions apply to all categories in the taxonomy. If a document submitted for processing does not meet the qualifying conditions for the taxonomy, CIS server does not evaluate it for assignment into any of the categories in the taxonomy. Note: The <definition> under a <taxonomy> element should not include an <evidence> subelement. Documents are not assigned to the root of the taxonomy. Example of <de nition> <category name="web Content Management Suite"> <details title="web Content Management Suite">... [Category details] </details> <definition candidatethreshold="60" ontargetthreshold="90"> <evidence evidencepropagation="low" impliedkeyword="33"> <evidenceset> <keyword name="web Content Management" confidence="high" phraseorderexact="true" stem="false"/> <keyword name="wcm" confidence="high"/> </evidenceset> </evidence> </definition> </category> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 59

60 <description> <description> Purpose Provides a description of the parent element Children Text of the description Parents <details> Usage Notes The <description> element is the only TEF element that includes plain text rather than subelements between its opening and closing tags. See <details>, page 61 for an example. 60 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

61 <details> <details> Purpose Groups together descriptive information about its parent category or taxonomy Diagram Attributes Table 11. <details> Element attributes Attribute title extended_attributes Description Title for the parent category or taxonomy Attributes of the subtype category; Attributes can be multiple; Attributes can be of different types - boolean, integer, string, id, time and double; Attributes can be repeating; <extended_attributes> is used to populate the attributes of the subtype. The type attribute determines what type of object needs to be created in the repository. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 61

62 <details> Children <description> <owners> <operations> <languageinfo> Parents <taxonomy> <category> Note: A different <details> element is a subelement of <class>. See <details>, page 64. Usage Notes The <details> element provides a description of its parent category or taxonomy. The <details> element is composed of four subelements: <description> contains a text description of the parent category or taxonomy. The text appears between the opening tag and the closing tag, not as an attribute as with other TEF elements. <owners> lists the owner of the parent category or taxonomy. The <owners> element groups together any number of <owner> elements, each of which gives the Documentum user name of an owner for the parent category or taxonomy. <operations> lists which user operations are available for the parent category or taxonomy. The <operations> element groups together any number of <operation> elements, each of which identifies a type of operation that is valid. <languageinfo> provides translated names for the parent category or taxonomy. The <languageinfo> element groups together any number of <supportedlanguage> elements, each of which identifies a language and provides a translation of the name into that language. Example of <details> <category name="web Content Management Suite"> <details title="web Content Management Suite"> <description>the suite of Documentum products for managing Web content</description> <owners> 62 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

63 <details> <owner name="dmadmin"/> </owners> <operations> <operation type="user_browse"/> <operation type="manual_assignment"/> </operations> <languageinfo> <supportedlanguage languagecode="es" translatedname="translation 1"/> <supportedlanguage languagecode="jp" translatedname="translation 2"/> </languageinfo> </details>... [The <definition> element and subcategories] </category> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 63

64 <details> <details> Purpose Provides descriptive information about its parent category class Diagram Attributes Table 12. <details> Element attributes Attribute title source targetattribute Description Title for the parent category class Text indicating the source of the parent category class Name of the document attribute into which CIS server writes the names of the categories to which a document is assigned Children <description> Parents <class> Note: A different <details> element is a subelement of <taxonomy> or <category>. See <details>, page EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

65 <details> Usage Notes The <details> element provides a description of its parent category class. It also sets the document attribute into which CIS server writes the names of categories to which a document is assigned. The <description> subelement contains a text description of the parent category class. The text appears between the opening tag and the closing tag, not as an attribute as with other TEF elements. See <class>, page 56 for an example that uses the <details> element. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 65

66 <evidence> <evidence> Purpose Identifies the evidence used to assign documents to the parent category Diagram Attributes Table 13. <evidence> Element Attributes Attribute impliedkeyword evidencepropagation Description Confidence-level value specifying the level of confidence CIS server applies to implied keywords. If this optional attribute is not included, the category uses the confidence level specified in the category class. Confidence-level value specifying the level of confidence CIS server applies to evidence propagated from other categories. If this optional attribute is not included, the category uses the confidence level specified in the category class. Children <evidenceset> 66 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

67 <evidence> Parents <definition> Usage Notes The <evidence> element provides the evidence and confidence values that CIS server uses to assign a confidence score for the parent category to the document. The evidence for a category is organized into evidence sets, each of which defines a collection of evidence keywords that CIS server considers together when calculating a document s score relative to the category. For the impliedkeyword and evidencepropagation attributes, you can enter a predefined confidence level or enter a number directly. The predefined values are: Certain Equivalent to the confidence level 100. High Equivalent to the confidence level 75. Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Auxiliary This evidence by itself does not cause CIS server to assign the document to the category. However, it increases the confidence level of other evidence found in the same evidence set. Equivalent to the confidence level 0. Negative This evidence by itself prevents CIS server from assigning the document to the category, regardless of how much other evidence appears. Equivalent to the confidence level 100. Off No weight is added to the document confidence score for this evidence. See the Documentum Administrator User Guide for more information about confidence levels. Example of <evidence> <evidence> <evidenceset> <keyword name="documentum"/> </evidenceset> <evidenceset> <keyword name="ecm"/> <keyword name="enterprise Content Management" confidence="high" phraseorderexact="true" stem="false"/> <categoryevidence name="@parent" confidence="low"/> </evidenceset> </evidence> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 67

68 <evidencepropagation> <evidencepropagation> Purpose Provides the defaults for propagating evidence Attributes Table 14. <evidencepropagation> Element Attributes Attribute type confidence Description Type of evidence Confidence level to assign to propagated evidence by default Children None Parents <categorydefaults> Usage Notes Evidence of one category or taxonomy can automatically be considered evidence for another category or taxonomy. Sharing evidence across categories and taxonomies is called propagating evidence. Evidence can only be propagated between categories and taxonomies that have a direct parent and child relationship. The propagation direction can be either parent to child or child to parent. 68 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

69 <evidencepropagation> For example, suppose a taxonomy has this hierarchical structure (showing just the top-level elements for the categories and taxonomy): <taxonomy name="united States" classname="country" taxonomyversion="1.0"> <category name="missouri" classname="state"> <category name="st Louis" classname="city"> <category name="branson" classname="city"> </category> </taxonomy> In this structure, if the direction of propagation is child to parent, the categories St. Louis and Branson can propagate their evidence to the Missouri category, because St. Louis and Branson are direct children of the Missouri category. Similarly, Missouri can propagate its evidence to the taxonomy United States, because Missouri is a direct child of United States. If the direction of propagation is parent to child, the taxonomy United States can propagate its evidence to the category Missouri, and Missouri can propagate its evidence to both St. Louis and Branson. However, you can never automatically propagate evidence directly between the taxonomy United States <taxonomy> and the St. Louis or Branson categories, because the categories are indirectly contained within the <taxonomy> element. Additionally, evidence cannot be automatically propagated between sibling categories. In the above example, this means that evidence for St. Louis cannot automatically be propagated to Branson, nor can evidence for Branson be propagated to St. Louis. For the confidence attribute, you can enter a predefined confidence level or enter a number directly. The predefined values are: Certain Equivalent to the confidence level 100. High Equivalent to the confidence level 75. Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Auxiliary This evidence by itself does not cause CIS server to assign the document to the category. However, it increases the confidence level of other evidence found in the same evidence set. Equivalent to the confidence level 0. Negative This evidence by itself prevents CIS server from assigning the document to the category, regardless of how much other evidence appears. Equivalent to the confidence level 100. Off No weight is added to the document confidence score for this evidence. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 69

70 <evidenceset> <evidenceset> Purpose Groups together a set of evidence that CIS server considers together when analyzing documents Diagram Children <keyword> <categoryevidence> Parents <evidence> Usage Notes An evidence set is a collection of keywords that CIS server uses together as evidence of a particular concept. The keywords are identified using the <keyword> and <categoryevidence> subelements. A category can have multiple evidence sets that define separate sets of co-occurring keywords. Confidence levels are not combined across evidence sets. 70 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

71 <impliedkeyworddefaults> <impliedkeyworddefaults> Purpose Provides the defaults for handling implied keywords Attributes Table 15. <impliedkeyworddefaults> Element Attributes Attribute confidence stem phraseorderexact Description Confidence-level value specifying the level of confidence CIS server applies to implied keywords by default true or false, instructing CIS server whether to use stemming for implied keywords by default true or false, instructing CIS server whether to recognize the words in a phrase only if they appear in exact order Children None Parents <categorydefaults> Usage Notes For the confidence attribute, you can enter a predefined confidence level or enter a number directly. The predefined values are: Certain Equivalent to the confidence level 100. High Equivalent to the confidence level 75. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 71

72 <impliedkeyworddefaults> Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Auxiliary This evidence by itself does not cause CIS server to assign the document to the category. However, it increases the confidence level of other evidence found in the same evidence set. Equivalent to the confidence level 0. Negative This evidence by itself prevents CIS server from assigning the document to the category, regardless of how much other evidence appears. Equivalent to the confidence level 100. Off No weight is added to the document confidence score for this evidence. 72 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

73 <keyword> <keyword> Purpose Identifies a string for CIS server to use as evidence for the parent category Attributes Table 16. <keyword> Element Attributes Attribute name confidence stem phraseorderexact Description Text of the keyword, which CIS server looks for in document content Confidence level value added to the document score when CIS server finds the keyword true or false, instructing CIS server whether to use stemming for the keyword true or false, instructing CIS server whether to recognize a multi-word keyword phrase only when the words appear exactly in the specified order Children None Parents <evidenceset> Usage Notes The <keyword> element defines a piece of evidence that CIS server looks for in the content of the documents it processes. The text of the keyword can be one or more EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 73

74 <keyword> words. When the server finds the keyword, it adds the confidence value for the keyword to the document s confidence score for the parent category. If you do not include values for one or more of the attributes, their values are inherited from the <keyworddefaults> element of the category class. 74 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

75 <keyworddefaults> <keyworddefaults> Purpose Provides the defaults for handling evidence keywords Attributes Table 17. <keyworddefaults> Element Attributes Attribute confidence stem phraseorderexact Description Confidence-level value specifying the level of confidence CIS server applies to keywords by default true or false, instructing CIS server whether to use stemming for keywords by default true or false, instructing CIS server whether to recognize the words in a phrase only if they appear in exact order Children None Parents <categorydefaults> Usage Notes For the confidence attribute, you can enter a predefined confidence level or enter a number directly. The predefined values are: Certain Equivalent to the confidence level 100. High Equivalent to the confidence level 75. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 75

76 <keyworddefaults> Medium Equivalent to the confidence level 50. Low Equivalent to the confidence level 15. Auxiliary This evidence by itself does not cause CIS server to assign the document to the category. However, it increases the confidence level of other evidence found in the same evidence set. Equivalent to the confidence level 0. Negative This evidence by itself prevents CIS server from assigning the document to the category, regardless of how much other evidence appears. Equivalent to the confidence level 100. Off No weight is added to the document confidence score for this evidence. 76 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

77 <languageinfo> <languageinfo> Purpose Groups together the translated names of the parent category or taxonomy Diagram Children <supportedlanguage> Parents <details> Usage Notes The subelements of <languageinfo> translate the category or taxonomy name into other languages. Each <supportedlanguage> element identifies a language (using its Documentum language code) and provides a translation of the name into that language. When a user views the category or taxonomy, its name will appear in the same language as the Documentum user interface if a translation is available. See <details>, page 61 for an example that includes the <languageinfo> element. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 77

78 <operation> <operation> Purpose Identifies an operation that is available for the parent category or taxonomy Attributes Table 18. <operation> Element Attributes Attribute type Description Text describing an available operation for categories Children None Parents <operations> Usage Notes The intent of the <operation> element is to identify a user operation that is available for a category or taxonomy. For example, you might include an operation that makes the category available for browsing by users. In this release, the <operation> element does not have any effect on standard CIS processing. Any operations are saved as part of the category definition, but Documentum applications do not refer to them. 78 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

79 <operations> <operations> Purpose Groups together the set of operations available for the parent category or taxonomy Diagram Children <operation> Parents <details> Usage Notes The intent of the <operations> element is to specify what user operations are available for a category or taxonomy. For example, you might not want users to see the documents assigned to certain categories. In this release, the <operations> element does not have any effect on standard CIS processing. The operations are saved as part of the category definition, but Documentum applications do not refer to them. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 79

80 <owner> <owner> Purpose Identifies an owner of a category or taxonomy Diagram Attributes Table 19. <owner> Element Attributes Attribute name Description Documentum user name of an owner of the parent category or taxonomy Children None Parents <owners> Usage Notes The owners of a category are the Documentum users who can review candidate documents and approve or reject their assignment to the category. Candidate documents are documents whose confidence score exceeds the category s candidate threshold but fall short of its on-target threshold, or documents that are assigned to the category manually with the Manual Workflow option active. For categories created using 80 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

81 <owner> Documentum Administrator, the user who created the category is an owner by default. See <details>, page 61 for an example that uses the <owner> element. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 81

82 <owners> <owners> Purpose Groups together the owners assigned to the parent category or taxonomy Diagram Children <owner> Parents <details> Usage Notes The owners of a category are the Documentum users who can review candidate documents and approve or reject their assignment to the category. Candidate documents are documents whose confidence score exceeds the category s candidate threshold but fall short of its on-target threshold, or documents that are assigned to the category manually with the Manual Workflow option active. See <details>, page 61 for an example that uses the <owners> element. 82 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

83 <quali er> <quali er> Purpose Defines a qualifying condition for documents assigned to the parent category or taxonomy Attributes Table 20. <quali er> Element Attributes Attribute tag operation value Description Aspect of a document to use for making the comparison. Valid tag values are: location (to qualify documents based on their location in the repository) type (to qualify documents based on their Documentum object type) Name of a document attribute (to qualify documents based on the value of one of its attributes) Comparison operator used to compare the tag to the value. Valid operators are: equal not_equal include exclude subtype Value to compare the tag to. For tag= location, value is a complete repository path For tag= type, value is the name of a Documentum object type For tag= attribute_name, value is a value for the specified attribute EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 83

84 <quali er> Children None Parents <qualifiers> Usage Notes Before CIS server assigns a document to a category, it verifies that the document meets the qualifying conditions for the category and for the taxonomy. If a document fails to meet any condition, CIS server does not assign the document to the category regardless of the strength of the evidence. See <qualifiers>, page 85 for an example. 84 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

85 <quali ers> <quali ers> Purpose Groups together the qualifying conditions for the parent category Diagram Children <qualifier> Parents <definition> Usage Notes The definition of a category or taxonomy can include qualifying conditions that assigned documents must meet. For CIS server to assign a document to a category, the document must meet all qualifying conditions for the category and all qualifying conditions for the taxonomy to which the category belongs. Example of <quali ers> <definition candidatethreshold="50" ontargetthreshold="80"> <evidence>... [Category evidence] </evidence> <qualifiers> <qualifier tag="location" operation="equal" value="/marketingcabinet"/> <qualifier tag="type" operation="not_equal" value="custom_type"/> </qualifiers> </definition> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 85

86 <supportedlanguage> <supportedlanguage> Purpose Provides a translated category or taxonomy name for a specified language Attributes Table 21. <supportedlanguage> Element Attributes Attribute languagecode translatedname Description Two-letter language code used to identify a language in Documentum Name for the parent category or taxonomy, translated into the language whose code you specify Children None Parents <languageinfo> Usage Notes The subelements of <languageinfo> translate the category or taxonomy name into other languages. Each <supportedlanguage> element identifies a language (using its Documentum language code) and provides a translation of the name into that language. When a user views the category or taxonomy, its name will appear in the same language as the Documentum user interface if a translation is available. See <details>, page 61 for an example that includes the <languageinfo> element. 86 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

87 <taxonomy> <taxonomy> Purpose Defines a taxonomy object Diagram Attributes Table 22. <taxonomy> Element Attributes Attribute name classname taxonomyversion type Description Name of the taxonomy, which must be unique among <taxonomy> elements Name of the category class used as the default for categories in this taxonomy Version label for the taxonomy Subtype for taxonomy EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 87

88 <taxonomy> Children <details> <definition> <category> <categorylink> Parents <tef> Usage Notes The <taxonomy> element represents the root of a hierarchical tree of categories. You can include multiple <taxonomy> elements in a TEF file. Some aspects of the <taxonomy> element establish default values for <category> elements that appear inside of it. A <taxonomy> element is divided into three main parts: <details> provides descriptive information about the taxonomy. <definition> specifies qualifying conditions that documents must meet in order for CIS server to assign them to categories in the taxonomy and provides default threshold values for categories in the taxonomy. <category> and <categorylink> elements define the hierarchical structure of the taxonomy. Example of <taxonomy> <taxonomy name="products" classname="generic" taxonomyversion="version 1"> <details title="products"> <description>products taxonomy</description> </details> <definition candidatethreshold="50" ontargetthreshold="80"> </definition> <category name="web Content Management Suite" classname="generic">... [Category definition] </category> <category name="enterprise Content Management Suite" classname="generic">... [Category definition] </category> </taxonomy> 88 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

89 <tef> <tef> Purpose Root element of a file in taxonomy exchange format Diagram Children <class> <taxonomy> <category> Parents None Usage Notes The <tef> element must be the first element in the file. All other elements must appear inside of it. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 89

90 <tef> Example of <tef> <tef> <class name="generic">... [Class definition] </class> <taxonomy name="products" classname="generic" taxonomy version="version">... [Taxonomy definition] </taxonomy> </tef> 90 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

91 <tef> Taxonomy Exchange Format Action Files A taxonomy exchange format (TEF) action file tells the TEF utility script (TefUtil) what actions to take. Creating TEF action les The available actions that can be carried out by creating a TEF action file are: Add CIS objects from a TEF file into a repository Export CIS objects from a repository into a TEF file Remove CIS objects from a repository Rearrange the hierarchy of categories in a repository A single TEF action file can perform these actions in any combination. To create a TEF action file, you can use the example files delivered with the predefined taxonomies, in the Xsd subfolder: addactionexample.xml, exportactionexample.xml, deleteactionexample.xml, relinkactionexample.xml. Copy any of these files and read the description of the elements below to know how to use them to create your own TEF action files. Example 4-1. Example of a TEF action le to add a CIS object <actions xmlns:xsi=" xsi:nonamespaceschemalocation="tefactionschema.xsd"> <add filename="tefexample.xml"> <classobject xpath="/tef/class"/> <taxonomyobject xpath="/tef/taxonomy[@name='products Example']" branch="true"/> <withinparentreference name="web Content Management Suite" classname="example"> <categoryobject xpath="/tef/category[@name='web Publisher' branch="true"/> </withinparentreference> </add> </actions> For information on how to run a TEF action file, see the Run the TefUtil utility step of the procedure To import a TEF taxonomy:, page 42. The following sections describes the TEF Action files elements. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 91

92 <absoluteparentlist> <absoluteparentlist> Purpose Provides a fixed list of parent categories for a linked category Diagram Children <categoryreference> Parents <categoryreference> Usage Notes <absoluteparentlist> appears inside of a <relink> action, as a subelement of the <categoryreference> that identifies the category being relinked. Its subelements identify the complete list of parent categories for the relinked category. The alternative to <absoluteparentlist> is <addparentlist> and <removeparentlist>. These elements identify which parents to add and remove for the relinked category rather than listing the complete set of parent categories. 92 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

93 <absoluteparentlist> Example of <absoluteparentlist> <actions> <relink> <categoryreference classname="alternate Class" name="alternate Category"> <absoluteparentlist> <categoryreference classname="class" name="maximum Taxonomy"/> </absoluteparentlist> </categoryreference> </relink> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 93

94 <actions> <actions> Purpose Root element of the TEF action file Diagram Children <add> <delete> <relink> <export> Note: The <update> action, which appears in tefactionschema.xsd, is not supported in this release. Parents None 94 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

95 <actions> Usage Notes <actions> is the root element of a TEF action file. Each of its subelements is an action you want to perform on a taxonomy. Example of <actions> <actions> <delete> <categoryreference name="maximum Taxonomy" classname= "Class" branch="true"/> <classreference name="class"/> <classreference name="alternate Class"/> </delete> <add filename="testtef.xml"> <classobject xpath="/tef/class"/> <taxonomyobject xpath="/tef/taxonomy[@name='maximum Taxonomy'] " branch="true"/> <withinparentreference name="maximum Taxonomy" classname="class"> <categoryobject xpath="/tef/category[@name='maximum Category']" branch="true"/> </withinparentreference> </add> <export filename="tefout1.xml" xsdfilename="tefschema.xsd"> <classreference name="class"/> <classreference name="alternate Class"/> </export> <add filename="testtef.xml"> <withinparentreference name="maximum Taxonomy" classname="class"> <categoryobject xpath="/tef/category[@name='maximum Category']" branch="true"/> </withinparentreference> </add> <export filename="tefout2.xml" xsdfilename="tefschema.xsd"> <categoryreference name="maximum Taxonomy" classname="class" branchlevels="all" details="true" definitions="false"/> <categoryreference name="maximum Category" classname="class" branchlevels="all" details="false" definitions="false"/> <categoryreference name="maximum Category" classname="class" branchlevels="all" details="true" definitions="false"/> </export> <relink> <categoryreference classname="alternate Class" name="alternate Category"> <absoluteparentlist> <categoryreference classname="class" name="maximum Taxonomy"/> <categoryreference classname="class" name="maximum Category"/> </absoluteparentlist> </categoryreference> </relink> <export filename="tefout3.xml" xsdfilename="tefschema.xsd"> <categoryreference name="maximum Taxonomy" classname="class" branchlevels="all" details="true" definitions="false"/> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 95

96 <actions> </export> <relink> <categoryreference classname="alternate Class" name="alternate Category2"> <addparentlist> <categoryreference classname="class" name="minimum Category"/> </addparentlist> <removeparentlist> <categoryreference classname="class" name="maximum Category"/> </removeparentlist> </categoryreference> </relink> <export filename="tefout4.xml" xsdfilename="tefschema.xsd"> <categoryreference name="maximum Taxonomy" classname="class" branchlevels="all" details="true" definitions="false"/> </export> <delete> <categoryreference name="maximum Category" classname="class" branch="true"/> </delete> <export filename="tefout5.xml" xsdfilename="tefschema.xsd"> <categoryreference name="maximum Taxonomy" classname="class" branchlevels="all" details="true" definitions="false"/> </export> </actions> 96 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

97 <add> <add> Purpose Adds categories, taxonomies, or category classes to the repository Diagram Attributes Table 23. <add> Element Attributes Attribute filename Description Name of the TEF file containing the definitions of the CIS objects to add Children <classobject> <taxonomyobject> <withinparentreference> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 97

98 <add> Parents <actions> Usage Notes The <add> action adds new CIS objects to the repository based on definitions stored in a TEF file. Each subelement identifies one or more objects to add. The <classobject> and <taxonomyobject> elements identify category classes and taxonomies respectively using an XPath reference to elements in the TEF file. The <withinparentreference> element identifies a position in the hierarchy where the categories referred to inside of it are added. If an object that appears inside of the <add> action already exists, the TEF utility does not add the object again or update it. It ignores the existing object and continues. Example of <add> <actions> <add filename="testtef.xml"> <classobject xpath="/tef/class"/> <taxonomyobject xpath="/tef/taxonomy[@name='maximum Taxonomy']" branch="true"/> <withinparentreference name="maximum Taxonomy" classname="class"> <categoryobject xpath="/tef/category[@name='maximum Category']" branch="true"/> </withinparentreference> </add> </actions> 98 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

99 <addparentlist> <addparentlist> Purpose Provides a list of new parent categories for a linked category in the repository Diagram Children <categoryreference> Parents <categoryreference> Usage Notes <addparentlist> appears inside of a relink action, as a subelement of the <categoryreference> that identifies the category being relinked. Its subelements identify new parent categories for the relinked category. The category is not unlinked from any of its current position(s), only the new parents are added. Use <removeparentlist> to remove existing links for the category. The alternative to <addparentlist> is <absoluteparentlist>. This element identifies the complete set of parent categories for the relinked category rather than identifying only new parents. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 99

100 <addparentlist> Example of <addparentlist> <actions> <relink> <categoryreference classname="alternate Class" name="alternate Category2"> <addparentlist> <categoryreference classname="class" name="minimum Category"/> </addparentlist> <removeparentlist> <categoryreference classname="class" name="maximum Category"/> </removeparentlist> </categoryreference> </relink> </actions> 100 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

101 <categoryobject> <categoryobject> Purpose Identifies a category element in a TEF file Attributes Table 24. <categoryobject> Element Attributes Attribute xpath branch Description XPath reference identifying one or more category elements in a TEF file true or false, specifying whether to add all children of the selected categories or only the selected categories themselves Children None Parents <withinparentreference> Usage Notes The <categoryobject> element appears within an <add> action to identify one or more categories to add from the TEF file to the repository. It identifies categories using an XPath reference. If the XPath reference selects more than one category, the TEF utility adds them all. <categoryobject> appears as a subelement of the <withinparentreference> element, which determines where in the hierarchy the categories are added. EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 101

102 <categoryobject> Example of <categoryobject> <actions> <add filename="testtef.xml"> <withinparentreference name="maximum Taxonomy" classname="class"> <categoryobject Category']" branch="true"/> </withinparentreference> </add> </actions> 102 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

103 <categoryreference> <categoryreference> Purpose Identifies a category in the repository to perform an action on. Diagram Note: This diagram applies only to <categoryreference> elements that appear within a <relink> action. <categoryreference> has no child elements in the context of other actions. Attributes Table 25. <categoryreference> Element Attributes Attribute name classname branch branchlevels Description Name of the category Name of the category class assigned to the category true or false, specifying whether to perform the action on children of the specified category or only on the specified category (Used with the <export> action only) Number of levels of children to export, or all to export all children EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 103

104 <categoryreference> Attribute details definition Description (Used with the <export> action only) true or false, indicating whether to include the <details> element for the exported category (Used with the <export> action only) true or false, indicating whether to include the <definition> element for the exported category Children <absoluteparentlist> <addparentlist> <removeparentlist> Note: These child elements apply only to <categoryreference> elements that appear within a <relink> action. <categoryreference> has no child elements in the context of other actions. Parents <delete> <relink> <export> Usage Notes <categoryreference> identifies an existing category or taxonomy in the repository. Its attributes specify whether the action applies only to the specified category or to the category and its children. 104 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

105 <categoryreference> Example of <categoryreference> <actions> <delete> <categoryreference name="maximum Taxonomy" classname="class" branch="true"/> </delete> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 105

106 <classobject> <classobject> Purpose Identifies a category class in a TEF file to add to a repository Attributes Table 26. <classobject> Element Attributes Attribute xpath Description XPath reference identifying a category class element to add Children None Parents <add> Usage Notes The <classobject> element identifies a category class object from a TEF file using an XPath reference. If the XPath reference selects multiple category classes, the TEF utility adds each class. Example of <classobject> <actions> <add filename="testtef.xml"> <classobject xpath="/tef/class"/> </add> </actions> 106 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

107 <classreference> <classreference> Purpose Identifies a category class element in the repository Attributes Table 27. <classreference> Element Attributes Attribute name Description Name of the category class Children None Parents <delete> <export> Usage Notes The <classreference> element identifies an existing category class using its name. Example of <classreference> <actions> <delete> <classreference name="class"/> </delete> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 107

108 <delete> <delete> Purpose Removes category classes, categories, or taxonomies from the repository Diagram Children <categoryreference> <classreference> Parents <actions> Usage Notes The <delete> action removes CIS objects from the repository. To delete a category or taxonomy, the object referred to by <categoryreference> must not have any children if the branch attribute of <categoryreference> is set to false. As the code sample below shows, you can delete a category and all of its children by setting the branch attribute to true. To delete a category class, no existing categories or taxonomies can use the category class. 108 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

109 <delete> Example of <delete> <actions> <delete> <categoryreference name="maximum Taxonomy" classname="class" branch="true"/> <classreference name="class"/> <classreference name="alternate Class"/> </delete> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 109

110 <export> <export> Purpose Creates a TEF file containing the definitions of specified categories and category classes Diagram Attributes Table 28. <export> Element Attributes Attribute filename xsdfilename Description Name of the TEF file to create Name of the XSD schema file to use for validating the file, including the path Children <classreference> <categoryreference> 110 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

111 <export> Parents <actions> Usage Notes The <export> action creates a TEF file containing elements for the selected classes and categories. The <classreference> and <categoryreference> elements identify existing classes and categories from the repository. The attributes of the <categoryreference> element determines what aspects of the category definition are included in the TEF file; see <categoryreference>, page 103. In most cases, the value of the xsdfilename attribute should be the standard Documentum TEF schema file tefschema.xsd. To create a TEF file without validation, set the xsdfilename attribute to an empty string. The category type or taxonomy type is always exported to the type attribute. Export action automatically picks up all attributes from dm_category/dm_taxonomy subtype and exports these attributes to <extended_data>. Example of <export> <actions> <export filename="tefout1.xml" xsdfilename="tefschema.xsd"> <classreference name="class"/> <classreference name="alternate Class"/> </export> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 111

112 <relink> <relink> Purpose Links existing categories into new hierarchical locations Diagram Children <categoryreference> Parents <actions> Usage Notes The <relink> action links an existing category into an additional location in the hierarchy. The <categoryreference> identifies the existing category and the new locations you want to link it to. The <relink> action can also be used to remove category links; see <removeparentlist>, page 114. However, every category must be linked to at least one parent category, and the TEF utility gives an error if you attempt to remove the final link. 112 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide

113 <relink> Example of <relink> <actions> <relink> <categoryreference classname="alternate Class" name="alternate Category"> <absoluteparentlist> <categoryreference classname="class" name="maximum Taxonomy"/> </absoluteparentlist> </categoryreference> </relink> </actions> EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide 113

114 <removeparentlist> <removeparentlist> Purpose Provides a list of parent categories to remove from a category Diagram Children <categoryreference> Parents <categoryreference> Usage Notes <removeparentlist> appears inside of a relink action, as a subelement of the <categoryreference> that identifies the category being relinked. Its subelements identify current parent categories to remove from the relinked category. Since every category must be linked to at least one parent category, the TEF utility gives an error if you attempt to remove the final link. Example of <removeparentlist> <actions> <relink> <categoryreference classname="alternate Class" name="alternate Category2"> <addparentlist> 114 EMC Documentum Content Intelligence Services Version 6 SP1 Administration Guide