Extended Search Administration

Transcription

1 IBM Lotus Extended Search Extended Search Administration Version 4 Release 0.1 SC

2

3 IBM Lotus Extended Search Extended Search Administration Version 4 Release 0.1 SC

4 Note! Before using this information and the product it supports, read the general information under Notices on page 233. Tenth Edition, July 2003 This edition applies to Version 4 Release 0.1 of the IBM Lotus Extended Search product (5724-E36) and to all subsequent releases and modifications until otherwise indicated by new editions. This edition replaces SC Copyright International Business Machines Corporation 1997, All rights reserved. US Government Users Restricted Rights Use, duplication, or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation.

5 Contents Tasks Starting and stopping Extended Search Start the server on UNIX Shut down the server on UNIX Start the server on Windows Shut down the server on Windows Configuring the search domain Get started Configuration model Configuration roadmap Start the Administration interface Navigate the interface Get help Configure servers Add another search server Add another web server Connect multiple search servers Discover data sources Run discovery as a standalone process Configure data sources Define native fields Define mapped fields Set field usage controls Define data source parameters Define smart result set actions Export and import data sources Configure categories Create a category Categorize a data source Configure applications Define application-specific properties Export and import applications Configure links Define link parameters Define link actions Configure grammars Run configuration wizards Refresh web source definitions Search my own web sites Configure a web crawler Refresh the search domain Refresh a single server Use search templates to test changes Log and view events Enable event logging View event logs Enable a servlet-level logging plug-in Enable a broker-level logging plug-in Configure security options Enable impersonation Enable a servlet-level security plug-in Enable a broker-level security plug-in Enable an agent-level security plug-in Change the database administrator password 51 Configure scalability options Separate brokers from the web server Configure remote agents Partition agents and data sources Configure multiple brokers Configure performance options Configure copies of server tasks Localize Notes data sources and agents Manage server memory Manage disk cache space Manage data source connections Running search applications Configure the browser to launch Lotus Notes.. 60 Run a search application Invoke the search servlet Search guidelines Saving and scheduling searches Save and schedule queries Work with saved queries Access saved queries Saved query actions Customize results for Retrieve and store result documents How documents are stored Retrieve and store guidelines Monitoring server activity Run the Monitor in stand-alone mode View server task details View a server task summary Display data source statistics Log server activity Run the Monitor in the Administration interface 79 Refresh the server Backing up and restoring data Backing up the configuration database Restoring the configuration database Concepts Configuration Actions Agents Applications Brokers Categories Data sources Adding data sources Configuring data sources Date formats Discovery Predefined discoverers Adding DB2 Information Integrator for Content sources Adding file system sources Adding LDAP sources Adding Lotus Discovery Server sources Adding Lotus Domino Domain Index sources 102 Adding Lotus Domino.Doc sources iii

6 Adding Lotus Notes sources Searching attachments to Notes documents Searching date fields in Notes documents 104 Full text vs. formula searches Generating abstract fields Indexing Notes databases Adding Lotus QuickPlace sources Adding Lotus Sametime sources Adding Microsoft Access sources Adding Microsoft Exchange Server sources 108 Adding Microsoft Index Server sources Adding Microsoft SharePoint Portal Server sources Adding Microsoft Site Server sources Adding Microsoft SQL Server sources Adding ODBC sources Adding WebSphere Portal Search sources Event logging Fields Reserved field names Native fields Mapped fields Grammars Predefined grammars Links Predefined links National language support UTF-8 codeset NLS configuration file content Specifying locale-specific information Code page support Parameters Link parameters Data source parameters Application parameters Predefined parameters Result set URL classes Web crawlers Web crawler configuration file Remote web crawlers Web crawler XML template Web sources Downloading web source definitions Adding support for your web sites Security Security plug-ins User authentication Security enforced through the web Security enforced by the broker Access control Impersonation Data source filtering Link-level security Connecting to data sources Security enforced by an agent Message encryption Notes database controls Database-level access controls Document- and field-level access controls 157 Scalability Assessing scalability Creating multiple applications Separating brokers from the web server Separating agents from brokers Configuring remote links Assigning unique agent names Partitioning agents on a single machine Setting up multiple brokers Advantages of multiple brokers Partitioning the user population Broker network configurations Fully connected network Hub network Multi-hub network Performance Assessing performance Optimizing end-user performance Setting timeout values Sorting search results Optimizing server performance Server tasks Determining the optimum number of tasks 172 Broker caching Localizing agents with data sources Reference Administration dialogs Application properties Application-specific properties Assign mapped field to data source Category properties Data source properties single data source Data source properties multiple data sources 184 Discover data sources DB2 Information Integrator for Content discoverer File System discoverer LDAP discoverer Lotus Discovery Server discoverer Lotus Domino Domain Index discoverer Lotus Domino.Doc discoverer Lotus Notes discoverers Lotus QuickPlace discoverer Lotus Sametime discoverer Microsoft Access discoverer Microsoft Exchange Server discoverer Microsoft Index Server discoverer Microsoft SharePoint Portal Server discoverer 193 Microsoft Site Server discoverer Microsoft SQL Server discoverer ODBC discoverer Sample C++ Link discoverer Sample Java Link discoverer Web discoverer WebSphere Portal Search discoverer Grammar properties Link properties Mapped field properties Query folder properties Saved query properties Extended Search server properties iv Extended Search Administration

7 Server position properties System properties Web server properties Agent requirements Errors, trace codes, and log files Java servlet client error codes Server side error codes Trace codes Log files Keyboard access Notices Trademarks Index Contents v

8 vi Extended Search Administration

9 Tasks This collection of topics represents the major activities behind setting up, administering, extending, and maintaining an IBM Lotus Extended Search system. Starting and Stopping the System Learn how to start the Extended Search server components and how to gracefully shut down the system. v Start the server on UNIX on page 3 v Shut down the server on UNIX on page 4 v Start the server on Windows on page 4 v Shut down the server on Windows on page 5 Configuring the Search Domain Learn how to use the Administration interface to discover data sources, configure options for how users can search and retrieve information, and set advanced options to enhance, secure, and troubleshoot your Extended Search environment. Understanding Configuration: v Get started on page 7 v Configuration model on page 7 v Configuration roadmap on page 9 v Start the Administration interface on page 10 v Navigate the interface on page 11 v Get help on page 14 Basic Configuration Tasks: v Discover data sources on page 18 v Configure data sources on page 20 v Configure categories on page 28 v Configure applications on page 30 v Refresh web source definitions on page 39 v Refresh the search domain on page 42 v Use search templates to test changes on page 43 v Log and view events on page 45 Advanced Configuration Tasks: v Configure servers on page 14 v Configure links on page 33 v Configure grammars on page 36 v Run configuration wizards on page 38 v Search my own web sites on page 39 v Configure a web crawler on page 41 v Configure security options on page 48 v Configure scalability options on page 52 v Configure performance options on page 55 Setting Up Search Applications Learn about the Demo application that is provided with Extended Search and how to set up and run your own search applications through a web browser. v Configure the browser to launch Lotus Notes on page 60 v Run a search application on page 61 v Invoke the search servlet on page 61 v Search guidelines on page 62 1

10 Saving and Scheduling Searches Learn how to use the Extended Search Saved Queries applet to store queries in the configuration database and run them at a scheduled time. This feature also allows you to save result sets for sharing with other users, or store the search result documents in a file system. v Save and schedule queries on page 64 v Work with saved queries on page 66 v Retrieve and store result documents on page 70 Monitoring Server Activity Learn how to use the Extended Search Monitor to keep track of processes running on remote servers. You can run the Monitor as a stand-alone program or run it from within the Administration interface. This feature allows you to make real-time adjustments to the system configuration. v Run the Monitor in stand-alone mode on page 74 v View server task details on page 75 v View a server task summary on page 76 v Log server activity on page 78 v Run the Monitor in the Administration interface on page 79 v Refresh the server on page 81 Backing Up and Restoring Configuration Data Learn how to use the backup and restore programs provided by Extended Search to protect your configuration data. v Backing up the configuration database on page 83 v Restoring the configuration database on page 84 2 Extended Search Administration

11 Starting and stopping Extended Search If you installed IBM Lotus Extended Search on the IBM AIX, Sun Solaris, or Red Hat Linux for Intel operating system, use the following procedures to start and stop an Extended Search system: v Start the server on UNIX on page 3 v Shut down the server on UNIX on page 4 If you installed Extended Search on the Microsoft Windows NT or Windows 2000 operating system, use the following procedures to start and stop an Extended Search system: v Start the server on Windows on page 4 v Shut down the server on Windows on page 5 Start the server on UNIX Use the following procedure to start an Extended Search system. 1. On the machine that hosts the Extended Search configuration database, make sure that the database server has been started and is running. 2. Start the Extended Search RMI server: a. Log in to the machine that hosts the Extended Search configuration database. b. Change to the Extended Search /bin directory. The default path is /opt/extendedsearch4.0/bin. (If you use Domino as the web application server, the recommended path is /home/server/extendedsearch4.0/bin.) c. Enter the following command:./desrmistart If you prefer, you can enter the following command to start the RMI server in the background. This feature allows you to exit out of the telnet session without shutting down the server../desrmistart -bg 3. On each machine where you installed the Extended Search Web Server component, start the web server and web application server. If you need assistance, consult your Domino or WebSphere product documentation. 4. Start the Extended Search servers: a. On each machine that hosts an Extended Search server, log in with a user ID that has write access to the Extended Search /bin directory. If you are running Extended Search with a Domino web server, start both the Extended Search server and Domino server under the same user ID (typically server). This user ID must be a Domino administrator user ID. b. Change to the Extended Search /bin directory. The default path is /opt/extendedsearch4.0/bin. (If you use Domino as the web application server, the recommended path is /home/server/extendedsearch4.0/bin.) c. Enter the following command:./desstart Tasks 3

12 If you prefer, you can enter the following command to start Extended Search in the background. This feature allows you to exit out of the telnet session without shutting down the server../desstart -bg Shut down the server on UNIX How you shut down the Extended Search system depends on whether you are currently running the Monitor to track server activity. v If you started the Monitor for this server, select Server Shutdown in the Monitor. If you started the stand-alone version of the Monitor for this server, you will be prompted to specify the Extended Search administrator ID and password to complete this task. v If you are not running the Monitor for this server, take the following steps: 1. Change to the Extended Search /bin directory. The default path is /opt/extendedsearch4.0/bin. (If you use Domino as the web application server, the recommended path is /home/server/extendedsearch4.0/bin.) 2. Enter the following command to shut down the Extended Search server:./desstop 3. To shut down the RMI server, enter the following command:./desrmistop Start the server on Windows Use the following procedure to start an Extended Search system. Keep in mind that proxy settings are different for each Windows user. If you plan to search web sources, you must ensure that the user ID that you use to start the Extended Search server is able to access the Internet. If necessary, review the installation instructions for information on how to set up the proxy server for use with the Extended Search server. 1. On the machine that hosts the Extended Search configuration database, start the Windows Administrative Tools and select Services. Ensure that the database service has been started. If you use DB2, and it is running on a machine that is a member of a domain, double-click the DB2 Security Server service, select Autostart as the Startup Type, and click OK. 2. On each machine where you installed the Extended Search Web Server component, start the web server and web application server. If you need assistance, consult your Domino or WebSphere product documentation. 3. On each machine that hosts an Extended Search server, take one of the following actions to start the Extended Search server: v Use the Windows Start menu to select Programs IBM Extended Search Extended Search Server. v Start the Windows Administrative Tools and select Services. Right-click the IBM Extended Search Server service and select Start. Note: To autostart the Extended Search Server service, you must ensure that the RMI server always gets started first. The installation program installs the RMI server on the machine that hosts the Extended Search configuration database and configures it to automatically start. 4 Extended Search Administration

13 Shut down the server on Windows It is recommended that use the Extended Search Monitor to shut down an Extended Search server. To do so: 1. Select Server Shutdown in the Monitor. You can shut down the server while running the Monitor as a stand-alone program or while you are monitoring server activity from within the Administration interface. If you are running the stand-alone version, you will be prompted to specify the Extended Search administrator ID and password to complete this task. If you started Extended Search by using the Windows Start Programs menu, you can also use the following procedure to shut it down: 1. Open the console window on the Extended Search server. 2. Enter one of the following commands: Quit Exit To shut down the Extended Search RMI server, take the following steps: 1. On the machine where you installed the configuration database, start the Windows Administrative Tools and select Services. 2. Right-click the IBM Extended Search RMI Server service and select Stop. desabort utility You should never stop Extended Search from the Windows Task Manager. If you stop Extended Search in this manner, the program stops, but the executables that are launched during startup do not. When this occurs, use the desabort utility to remove the unwanted tasks. Enter the following command at the console command prompt only as a last resort, when the state of the system prevents you from using normal system shutdown procedures: desabort If the Extended Search server does not restart after you run the desabort utility, recycle the server. If any related processes were still open, it might prevent the Extended Search server from successfully restarting. Tasks 5

14 Configuring the search domain The Extended Search server obtains information about the resources it is to manage from the Extended Search configuration database, a database that is stored in IBM DB2 or Oracle. To update the configuration database, you run the Administration interface and specify the following types of information: v The locations of the data sources that you want to include in the Extended Search domain v Which data source fields users can search, view in a result set, or retrieve from a data source v Which field names should be mapped to permit field-based searching across dissimilar data sources v Which data sources should be grouped into a common category v Which categories should be accessible to users through a given search application v Which brokers should process requests for a given data source v Which agent, and how many agents, should service transactions between brokers and data sources v Which link should handle search and retrieval operations within the data source v Which grammar should be used to translate queries from generalized query language (GQL) into the native search language of the data source When you first install Extended Search, you can use it to search the following types of data sources: v File system directories and files v IBM DB2 Information Integrator for Content v Lightweight Directory Access Protocol (LDAP) v Lotus Discovery Server v Lotus Domino.Doc v Lotus Domino Domain Index v Lotus Notes v Lotus QuickPlace v Lotus Sametime v Microsoft Access v Microsoft Exchange Server v Microsoft Index Server v Microsoft Site Server v Microsoft SQL Server v Open Database Connectivity (ODBC) databases v Relational database management systems (RDBMS) v Web search sites (the product includes predefined connections to many sites) v IBM WebSphere Portal Search IBM Lotus business partners offer support for other types of data sources. Furthermore, your organization can create custom links to support additional proprietary data sources. See Predefined links on page 123 for information about which sources are supported on various operating systems. 6 Extended Search Administration

15 Get started Before you configure your Extended Search domain, you should be familiar with the system components and how they interact. If you have not already done so, read the Technical Overview in Extended Search General Information before proceeding. To help you prepare for the configuration tasks, you should also review the following topics: v Configuration model v Configuration roadmap on page 9 v Adding data sources on page 92 v Configuring data sources on page 92 v Discovery on page 96 As discussed in Tasks on page 1, there are certain basic configuration activities that everyone should perform. Other more advanced activities depend on the size and complexity of your system. Configuration model The Extended Search configuration data model is illustrated in Figure 1. It depicts the system as a set of inter-related boxes that are connected by directed arrows. Each box represents a configurable element. Extended Search Search Domain Logical View Physical View Links Data Sources Mapped and Native Fields Brokers Grammars Categories Agents Applications Servers Figure 1. Configuration data model The servers depicted in this diagram represent the physical composition of an Extended Search system. For interprocess communication to take place, the locations of the servers, including the broker and agent components they host, must be known to your network s Domain Name System (DNS) server. The remaining components in this diagram represent the logical view that you present to users. The data sources that you plan to search, and the ways that you plan to make data sources recognizable to and accessible by users, must be defined in the configuration database. By using the Administration interface, you can configure or reconfigure the following types of information. Servers When you configure a search server, you identify its physical location, event logging preferences, and information about the brokers and agents that run on that server. You also discover and add data sources to your search domain, and can run the Extended Search Monitor to track activity Tasks 7

16 on the server. To enhance scalability and performance, you should install multiple Extended Search servers. You can easily graph the relationships between servers by drawing connector lines between them. All network addressing activity required to reflect these communication links is handled in the background for you. When you configure a web server, you identify its physical location and event logging preferences. You can also configure plug-ins that instruct Extended Search to use your preferred routines for handling security and logging at the web server level, and specify options for scheduling queries and web crawlers. Data Sources After you discover a data source, you need to identify the link, grammar, and agent processes that should connect to the data source and perform search and retrieval operations. You also define which parameters, if any, the link should use when it connects to or searches the data source. Although you can make changes, this data is generally provided for you by the discoverer. Native Fields and Mapped Fields When you configure a data source, you also define which fields should be accessible to end users. It is possible for two native fields in separate data sources to contain the same type of information but be known by different names. To perform a single fielded search across these sources, each source must have a common field that enables it to be searched by a common query. You achieve this by configuring mapped fields. For example, if one data source has a field named Author and another data source has a field named Creator, you might create a mapped field named Owner and link both native fields to it. You can then submit a query on the Owner field and get appropriate results from the two data sources. Links If you have a data source that is not supported by the set of predefined links, you can create a new link for searching it. See Extended Search Programming for information about developing custom links to support search and retrieval operations in a new type of data source. Grammars If you have a data source that is not supported by the set of predefined grammars, you can create a new translator for searching it. See Extended Search Programming for information about developing custom functions to translate a query into the native search language of a data source. Categories Each data source that you want to search must be associated with at least one category. Categories make it easy to search and administer potentially large groups of related data sources. When submitting a search request, users can select a category name and search all the data sources that belong to it. Applications To ensure that users access only those data sources for which they have a need, you must associate each category with at least one search application. You then assign application-specific controls, such as which fields the users of that application can search, view in a result set, or retrieve from the data source. You can also specify the order in which fields should be presented in a result set, and the order in which documents should be retrieved. 8 Extended Search Administration

17 Configuration roadmap Configuration of an Extended Search system is a multi-step process that assumes a successful installation of the product. Much of the configuration data gets generated for you through installation and data source discovery. There are additional steps that you should take to meet the needs of your organization. Step 1. Add the data sources that you want to search. The term data source refers to a logical entity that represents a collection of structured or unstructured data that can be searched. This collection of data sources constitutes your Extended Search search domain. Step 2. Decide how you want to configure fields in the data source. You should identify which fields contain information of interest to your users, such as which fields users will want to search by name, view in result sets, or retrieve from data sources. Options you specify here apply to the data source globally. Later, you can define additional options on an application-by-application basis. Step 3. Decide how you want to categorize groups of data sources. A category provides an easy way for you to group similar types of data sources so that a common set of rules applies to them all. To be searchable, each data source must belong to at least one category. The category, in turn, must be associated with at least one application. Step 4. Configure your search applications. The configuration of each application includes tasks such as associating the categories of data sources to be searched by users of this application, selecting the fields to be searched, returned, or retrieved, and specifying the order in which fields should be presented in a result set or retrieved from the data source. Step 5. Perform advanced configuration tasks as necessary. These tasks depend on the security, scalability, and performance requirements of your enterprise. If you install any server components on remote machines, such as additional brokers or agents, you need to configure information about these remote components. You might also need to configure additional links or grammars to support the connection requirements and native search languages of some sources. Step 6. Optionally enable event logging. Choose the types of events that you want to log, from informational to critical. Then use the event log viewer to review the event messages and the suggested corrective actions. Step 7. Test your configuration changes. Extended Search provides sample search templates and a Demo application that enable you to immediately test your ability to search newly configured data sources. Tasks 9

18 Start the Administration interface To make changes to the configuration database, you use a Java application, es_admin. On startup, the application prompts you for the Extended Search Administrator s user ID and password. To implement additional security, you must do so through custom programs that you write, by securing the server within your firewall, or by physically securing the machines that host the Administration application and the configuration database. Use the following procedure to start the Administration application: 1. If you are starting the application on Windows, select Start Programs IBM Extended Search Extended Search Administration. 2. If you are starting the application on a UNIX system, enter the following command:./es_admin 3. Enter the Extended Search Administrator user ID and password. Initially, these values match the values that were specified for the database administrator during the installation of Extended Search. They might have been changed as part of configuring System Properties. The application starts and the main Search Domain view is displayed. Figure 2. Administration interface Search Domain view 10 Extended Search Administration

19 Using the Administration applet For backward compatibility, the Administration interface is still available as an applet. Because running the Administration interface from a web browser is not secure, it is strongly recommended that you run the Administration application. Note that the application is the supported database administration tool; the applet will be deprecated in a future release. See Security enforced through the web on page 151 for important information about how web authentication works with the Administration applet. If you want to run the Administration applet, follow these steps: 1. If you did not previously install the Java 2 Runtime Environment (J2RE) plug-in on this machine, do so before you start the applet (see Extended Search Installation for system requirements). 2. Access the case-sensitive URL for the Administration applet. For example: Where: v hostname is the fully qualified host name of the web server. v :port is the web server port (required only if the port is not 80). v servlet is the base directory on the web server where Extended Search servlet files are installed. On a Domino server, this value is servlet. On a WebSphere server, this value is the Extended Search web application name (the default value is lotuskms). For example: You might be prompted to log in. This occurs only if the security setup of your web environment requires user authentication. The applet loads and starts. Wait until the applet has completely loaded and the Search Domain view has appeared (Figure 2 on page 10). Navigate the interface The Navigator on the left side of the Administration interface lists the configurable elements in the Extended Search search domain. When the program first starts, the view pane on the right side contains icons that represent each of the configurable elements. To select the element you want to work with, click its name in the Navigator. Alternatively, you can click the element s name with your second mouse button and select Explore. (The second button is typically on the right side of the mouse. Throughout this discussion, this directive is referred to as right-click.) After you select an element, the view pane presents the configurable items that are currently defined for that element. There are a couple of ways to work with these items: v Double-click the item name or the icon in the view pane. This action opens the Properties window for that item, from which you can define or change configuration data. Tasks 11

20 v Right-click the item name or the icon to open a menu from which you can choose the actions that are available for that item, such as Copy or Delete. The Properties action opens the Properties window for that item, from which you can define or change configuration data. The following table describes the configurable elements and their icons. Icon Title Description Search Domain Select Search Domain to see an overview of the Administration interface. You can also review tips on how to perform common administration tasks or how to request technical support. Servers Lists all known search servers and web servers. Use this option to identify information about the servers and how they interact to process search requests. Also use this option to discover data sources that you want to add to the search domain or to monitor server activity. Data Sources by Link Data Sources by Agent Categories Lists all known data sources organized by link type. Use this option to associate a link, grammar, and agent with a data source, and to define field usage controls, processing parameters, and result set actions for the data source. Lists all known data sources organized by agent name. Use this option to associate a link, grammar, and agent with a data source, and to define field usage controls, processing parameters, and result set actions for the data source. Lists all data sources organized by the categories to which they belong. Use this option to define new categories and to associate data sources with categories. Applications Search Templates: - Demo - Other Samples Mapped Fields Lists all applications organized by application name. Use this option to define new applications and to associate categories with applications. You can also assign application-specific controls, such as which fields users of the application can search, view in a result set, or retrieve from a data source, and the order in which fields should be returned or retrieved. Lists search templates that you can use to run a test query after making configuration changes. The Demo template is the most comprehensive example. The other sample templates demonstrate ways to submit a simple query, multiple queries, and advanced GQL queries. Lists all known mapped fields. Use this option to define a mapped field and make it available for use with individual data sources. Links Extended Search is preloaded with link definitions that enable many common data sources to be searched. Use this option to define new links or to define link parameters and result actions for existing links. 12 Extended Search Administration

21 Icon Title Description Grammars Extended Search is preloaded with grammar definitions that enable many common data sources to be searched. Use this option to define new grammars; you should not change the preloaded definitions. Event Log Viewer Opens the log viewer, which enables you to view the text of logged messages, print them, save them to a file, or delete them from the event log database. Saved Queries Lists saved queries and search results. As an administrator, you can view and work with all saved queries and search results. Wizards Documentation Lists wizards that make it easy for you to add data sources to the search domain; configure mapped fields, data source parameters, and result sets; export the configuration database; or export and import data sources and applications. An icon for selecting the wizard that you want to run is also conveniently located above the Navigator. Lists the Extended Search product documents. Use this option to learn about the product and how to perform installation, administrative, and programming tasks. An icon for selecting the document that you want to view is also conveniently located above the Navigator. Note: On Windows systems, a document that you select will be displayed automatically in a new web browser window. On UNIX systems, you must first open a web browser before selecting the action to display a document. The icons and buttons in and above the view pane can help you view the contents of the configuration database and perform configuration tasks. Icon Title Description Icon View Displays configurable items as icons. List View Displays configurable items in a list. Detail View Lists the configurable items and provides details about each item. Tree View Shows the relationship between configurable items. Tasks 13

22 Icon Title Description Graph View Sort ascending Shows the connections between items in the current view. For example, if you installed two or more servers, you would use this view to draw the layout of the physical system. Sorts items in the current view in ascending order. Sort descending Sorts items in the current view in descending order. No sort Does not sort the items in the current view. Refresh Refreshes the contents in the current view of the configuration database. Find Searches the contents of the current view to locate a term that you enter in the search box to the left of the icon. Expand All button Collapse All button Apply Connections button Expands the list of items in the current List view. Collapses the list of items in the current List view. Applicable only in the Servers view, allows you to save the arrangement of and connections between multiple servers. Get help There are a couple of ways that you can obtain help while using the Administration interface: v In most of the views, you can right-click in an open area of the view pane and select Help. This action displays task-oriented information that can help you make selections and define options for the configurable items that are displayed in the current view. v After you open an item to define properties for it, you can click the Help button in any Properties window. This action tells you about the information that you must type or select to configure the selected item. The system displays help in a new browser window, which allows you to review the information while you complete the task. To obtain additional help, click in the top frame of any help window, such as on the book icon. This action opens this document, Extended Search Administration. Configure servers When you install an Extended Search server, information about its network location is written to the configuration database. The database also stores default information about the broker, agent, Internet protocols, and web server that are used to process search requests. Typically, you can use this initial configuration data to run Extended Search out-of-the-box, without making any changes to the server definition in the configuration database. 14 Extended Search Administration

23 Most of the actions that you take to configure the server occur as the result of completing another task. For example, to enhance security or performance, you might need to install another server, alter the number of broker tasks that run on a particular server, or associate a plug-in to perform user authentication. The following table lists options that you can configure for an Extended Search server along with pointers to where you can read about how to make changes. To learn about this task... Define additional Extended Search servers. Define additional web servers. Graphically connect two or more search servers to allow them to share request processing. Read this procedure Add another search server on page 16 Add another web server on page 16 Connect multiple search servers on page 17 Run discoverers to add data sources to your search domain. Discover data sources on page 18 Enable event logging, and then use the event log viewer to review and administer logged messages. Instruct the web server to use a logging plug-in. Instruct the broker to use a logging plug-in. Instruct the web server to use a security plug-in. Instruct the broker to use a security plug-in. Instruct one or more agents to use a security plug-in. Increase or decrease the number of copies of broker and agent tasks that run on this server. Configure the amount of shared virtual memory the server should use for caching search results. Configure the amount of disk space the server should use for caching search results. Separate the broker from the web server to balance resources. Configure agents and data sources on remote machines but allow them to interact with a central broker. Define multiple agents on one machine. Configure multiple brokers to balance the processing load across multiple machines. Run the Monitor to keep track of activity on this server. Log and view events on page 45 Enable a servlet-level logging plug-in on page 46 Enable a broker-level logging plug-in on page 47 Enable a servlet-level security plug-in on page 49 Enable a broker-level security plug-in on page 50 Enable an agent-level security plug-in on page 50 Configure copies of server tasks on page 55 Manage server memory on page 57 Manage disk cache space on page 57 Separate brokers from the web server on page 52 Configure remote agents on page 52 Partition agents and data sources on page 53 Configure multiple brokers on page 54 Run the Monitor in the Administration interface on page 79 Tasks 15

24 Add another search server To run more than one search server in your Extended Search domain, you need to install the server software on another machine. You also need to configure options for the server, and decide how you want to connect the new search server to one or more previously configured servers. When you add a server definition, you can use default configuration data or use data that you copy from an existing server. When you copy a server, you copy its entire definition. This approach makes defining a new server easy. For example, the only options that you must specify for the new server are a unique server name and a unique broker name. 1. Follow procedures in Extended Search Installation to install the server software. (Select the Extended Search Server option from the list of components to install.) 2. Start the Administration interface, and use one of the following approaches to configure options for this new server. v To define a new search server based on the default options, either right-click Servers in the Navigator and select New Search server, or right-click in an open area of the Servers view and select New Search server. v To define a new search server based on the options defined for an existing server, right-click an existing server in the view pane and select Copy, and then right-click in an open area of the view pane and select Paste. 3. When the Extended Search Server Properties window is displayed, specify configuration options for this server (page 200 discusses these options). After you click OK to save your settings, an icon for the new server is displayed in the view pane. 4. Connect the new server to an existing server. (See Connect multiple search servers on page 17 for instructions.) 5. If you need to change a server configuration, right-click the server icon in the view pane and select Properties. When the Extended Search Server Properties window is displayed, make your changes and then click OK. 6. If you need to delete a server, right-click the server icon in the view pane and select Delete. Deleting a server also deletes all of the configuration data that was stored for that server. 7. When you have completed your configuration changes, refresh the search domain to make the new server available to users. Add another web server To run more than one web server in your Extended Search domain, you need to install the Extended Search Web Server component on your web server machine and configure options for how the web server should interact with Extended Search search servers. When you add a server, you can use default configuration data or use data that you copy from an existing server. When you copy a server, you copy its entire definition. This approach makes defining a new server easy. For example, the only option that you must specify for the new web server is a unique server name. 1. Follow procedures in Extended Search Installation to install server software. (Select the Web Server option from the list of components to install.) 2. Start the Administration interface, and use one of the following approaches to configure options for this new server. 16 Extended Search Administration

25 v To define a new web server based on the default options, either right-click Servers in the Navigator and select New Web server, or right-click in an open area of the Servers view and select New Web server. v To define a new web server based on the options defined for an existing server, right-click an existing web server in the view pane and select Copy, and then right-click in an open area of the view pane and select Paste. 3. When the Web Server Properties window is displayed, specify configuration options for this server (page 207 discusses these options). After you click OK to save your settings, an icon for the new server is displayed in the view pane. 4. If you need to change a web server configuration, right-click the web server icon in the view pane and select Properties. When the Web Server Properties window is displayed, make your changes and then click OK. 5. If you need to delete a web server, right-click the web server icon in the view pane and select Delete. Deleting a server also deletes all of the configuration data that was stored for that server. 6. When you have completed your configuration changes, refresh the search domain to make the new web server available to users. Connect multiple search servers If you have more than one Extended Search search server in your Extended Search domain, you must specify the relationships between them. The Administration interface provides an easy way for you to do this. First, graphically position the server icons in a manner that best represents the physical layout of your network. You can then draw connector lines to define the communication links between the search servers. When you connect one search server to another, the sources become available to the web server that services those search servers. For example, when you configure a remote agent, you need to connect the server that hosts the remote agent with the server that hosts the broker who provides service for that agent. Note that you do not need to draw connections between web servers or between search servers and web servers. There are two ways to connect a new Extended Search search server with your existing network of servers. Use the first procedure below if you prefer to use the mouse and work in a visually graphic mode. Use the second procedure if you prefer to type values and make selections with the keyboard. Drag and draw connections in the view pane: 1. While in the Servers view, click the Graph View icon. 2. Click the new search server icon, and drag it to a location in the view pane that represents how it fits in with your existing server topology. 3. Right-click the new server and select Connect to. 4. Click the existing server to which this server should connect. The system draws a line between the two servers and, in the background, sets up the appropriate network addresses. 5. If you need to delete a connection between two servers, click the connector line and then press the Delete key. 6. Click Apply Connections to save your changes. The system recalculates the network addresses to reflect the correct locations of your data sources. Note: While you are working in Graph View mode, you can reposition the servers as needed. Any time that you connect two servers, or delete a line to disconnect them, be sure to click Apply Connections. This is necessary to Tasks 17

26 commit your changes and recalculate the network addresses. Do not click the Refresh icon until you have applied the connections or you will lose your changes. Type values for the server position and connections: 1. While in the Servers view, right-click the new search server icon and select Server position. 2. In Horizontal position, type a number to specify the horizontal alignment of the top-left corner of the server icon (the X axis). 3. In Vertical position, type a number to specify the vertical alignment of the top-left corner of the server icon (the Y axis). 4. In Connect to, select the check box for each search server that you want this server to be connected to. The Host name and Port columns identify the servers that you can connect to in this Extended Search domain. (If no other servers are defined, the Connect to options are not displayed.) 5. If you need to delete a connection, clear the check box for the server that should no longer be connected. 6. Click OK. The system recalculates the network addresses to reflect the correct locations of your data sources. When the Servers view pane is displayed, it shows the layout that you specified. Connecting lines appear between search servers that are able to communicate requests with each other. Discover data sources Extended Search provides an automated data source discovery feature that makes it easy for you to add data sources to your search domain. To discover certain types of data sources, the Extended Search Server component, including an agent, might need to reside on the same machine with the data source that you are discovering. See Agent requirements on page 210 for information about local and remote data source support. You must also ensure that the user ID by which you log in to the system has the necessary permissions to access any remote network drives or servers that you are discovering. If the user ID does not have access permissions, you will see an access denied message instead of a list of data sources that are available for discovery Add data sources wizard Extended Search provides a wizard that makes it easy for you to add data sources to your search domain. This wizard discovers data sources that you select, puts them in one or more categories, updates all applications that include those categories, and refreshes the search domain. For information about how to use this wizard, select Search Domain and then, in the view pane, select the Add data sources link. The following procedure shows you how to use the Administration interface to discover data sources. If you want to run discovery outside of this interface, see Run discovery as a standalone process on page Start the Administration interface. 18 Extended Search Administration

27 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the server that you want to work with and select Discover Data Sources. You can also discover data sources by opening the Data Sources by Link or Data Sources by Agent view, right-clicking in an open area of the view pane, and selecting Discover Data Sources. If you have only one Extended Search server, the Discover Data Sources window will be displayed. If you have multiple servers, you will first be prompted to select the server that you want to discover. 4. On the Discover Data Sources window, select the discoverer that matches the type of data source that you want to add to the search domain. 5. The system requests information about the selected data source type. For example, if you are adding a file system source, it asks you to specify the system path to the source. For other data source types, it might ask you for a database name or for a user ID and password that enables the discovery process to access the data source. Select or type the requested information and click Start Discovery. If you need assistance when you provide this discoverer-specific information, see Discover data sources on page The bottom half of the window lists all data sources of the selected type that were found on the target server. Select one or more data sources from the list and click Add to ES. 7. Repeat step 4 through step 6 to add data sources of a different type. 8. When you have finished discovering data sources on this server, click Close. 9. After you add a data source, there are several steps that you must take to make it available to users: a. Configure the field definitions. For example, you can add fields, delete fields, select fields that you want users to search or retrieve, and so on. b. Categorize the data source. The discoverer automatically associates each data source with a category that has the same name as the link that is used to connect to the source. If you prefer, you can put the source into different categories, including a new category that you create. c. Associate the category with an application. You can use the Demo application to search newly added sources. If you want to use a different search application, you must associate the data source with a category that belongs to that application. You can also define application-specific properties to govern how users of the application can search the data source. 10. When you are ready to use the new data source, refresh the search domain to disseminate your changes to the user community. Run discovery as a standalone process If you prefer, you can run discovery outside the Administration interface. This feature might be useful if you want to share the workload among multiple administrators, or if you want to allow certain users to discover data sources but not allow them to perform all the Extended Search administrative tasks. To run discovery as a standalone process, start a web browser and pass dbdiscover.txt to the JKMSearchController servlet. Note that the Java 2 Runtime Environment (J2RE) plug-in must be installed on any machine from which the discovery process is launched. Tasks 19

28 For readability, the following example has been entered on two lines. You should type the entire string on a single line. Note, too, that the servlet name and its arguments are case-sensitive. &AppID=appname&desClientLocale=locale Where: v hostname is the fully qualified host name of the web server. v servlet is the base directory on the web server where Extended Search servlet files are installed. On a Domino server, this value is servlet. On a WebSphere server, this value is the Extended Search WebSphere application name (the installation default is lotuskms). v appname is the name (application ID) of the search application that you want to run, such as the Demo application provided with Extended Search. v locale is the language and country code that identifies the locale of the data sources that this application searches, such as enus for U.S. English. For information about specifying a valid language and country code, review National language support on page 124. For example: &AppID=Demo&desClientLocale=enUS When you run data source discovery in this manner, the dbdiscover applet automatically stores data in the configuration database and writes it to an XML file. (You can later edit and import this file into Extended Search.) If you prefer, you can alter this default behavior. To do so: 1. Edit the dbdiscovermainpage.txt file. This file is located in the /templates/enus/jkm directory, where enus is the subdirectory for your locale (in this example, U.S. English). 2. If you do not want to store data in the configuration database, locate the following lines, and change the value "Yes" to "No". For example: <PARAM NAME="WriteDB" VALUE="No"> WriteDB = "No" 3. If you do not want to write data to an XML file, locate the following lines and change the value "/temp/discover.xml" to blank (""). For example: <PARAM NAME="WriteXMLFile" VALUE=""> WriteXMLFile = "" Configure data sources Data source discovery automatically loads the configuration database with information about the data source and the fields that are defined in the data source. The following procedure shows you how to set general options for the data source. Refer to the following topics for information about configuring search and retrieval options for the data source and associating the data source with one or more categories: v How do I define native fields? v How do I define mapped fields? v How do I define field usage controls? v How do I categorize the data source? v How do I define parameters for accessing or searching the data source? v How do I specify actions for handling results returned from the data source? v How do I export and import data source definitions? 20 Extended Search Administration

29 Use the following procedure to configure an individual data source. 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, review and optionally change information on the General tab. 5. In Data source name, type a short, unique, and descriptive name for the data source. This name is used throughout the Extended Search domain to identify the source as a distinct repository. It cannot match the name of any other data source, including differences in case. For example, if you have a source named MySource, you cannot add a data source named mysource. If you replicate a Notes database, be sure to assign a unique name to the replicated database. 6. Select the Disable data source check box only if you want to hide the data source from end users. For example, if you developed a custom link for this data source, you might want to temporarily disable the data source until you have finished testing your changes. 7. The Link, Grammar, and Agent values are all set as part of discovery and, typically, you do not need to change them. If you want to make a change, select the value that you want to use from the appropriate list. For example, your enterprise might have developed a custom link or set up a separate agent to service this data source 8. The Language used value, which identifies the locale of the data source content, is specified at the time of discovery and, typically, you do not need to change it. 9. The Code page used value, which identifies the code page that will be used when communicating with the data source, is specified at the time of discovery. Typically, you do not need to change it. 10. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. You can specify general options for more than one data source at a time. For example, if you configure a new link, grammar, or agent, you can associate a group of data sources with the new link, grammar, or agent without having to update each data source individually. Use the following procedure to configure multiple data sources. 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Select two or more data sources that you want to configure, right-click, and select Properties. (To select multiple sources, press and hold the Ctrl key or Shift key as you click.) 4. On the Data Source Properties window, select the Disable data source check box only if you want to disable all the data sources you selected. 5. Select the new Link, Grammar, and Agent values as appropriate for this group of data sources. 6. Click OK to save your changes and exit. Tasks 21

30 22 Extended Search Administration Define native fields When you define native fields, you specify which fields should be accessible to end users through a search application. For example, you might want to remove certain fields that you do not want users to search or retrieve. In addition, you might want to add fields that were not defined during data source discovery. This is particularly true with Lotus Notes and ODBC-compliant databases which, typically, contain numerous fields that you opted not to spend time loading during the discovery process. When you add a field, make sure that the field name is a valid name that actually exists in the data source. Use the following procedure to define native fields: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, select the Native Fields tab. 5. To add a field to the list of fields that are accessible to search applications, click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. a. In Native field name, type the name of the field as it is defined in the data source. You cannot use the following characters in field names: Forward Slash (/), Backslash (\), Comma (,), Semi-colon(;), Colon (:), Left Bracket([), and Right Bracket (]). b. In Type, click to display a list of available data types, and select the data type of this field. Note: If you add a field after the data source has been assigned to an application, the addition is reflected automatically in any application that includes the data source. However, you must update the application-specific properties to specify whether or not users are permitted to search the new field, view it in a result set, or retrieve it from the data source. 6. To remove a field from the data source, select the field from the Native field name list and click Delete. This action does not delete the field from the physical data source. It simply hides the field from the user s view of the data source and prevents users from searching it. To prevent users from performing certain operations on fields, as opposed to completely obscuring the field s presence in the data source, you can configure field usage controls (page 24 discusses this procedure). Note: If you delete a field after the data source has been assigned to an application, the deletion is reflected automatically in any application that includes the data source. 7. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. Define mapped fields The process of defining mapped fields is a two-fold. First you must define the mapped field at a global level where it can be known to all data sources and applications in your Extended Search domain. You must then identify the native fields to which the mapped field should link. See Mapped fields on page 119 for more information about how to take advantage of this feature.

31 Step 1: Define a Mapped Field. 1. Start the Administration interface. 2. Click Mapped Fields in the Navigator. The view pane lists all mapped fields that are defined in the database. For descriptions about the mapped fields that are included in the default installation of the product, see Mapped fields on page To create a new mapped field: a. Either right-click Mapped Fields in the Navigator and select New, or right-click in an open area of the Mapped Fields view pane and select New. b. Type a descriptive name for the field. For example, you might create a mapped field named Author to encompass all native fields that identify a document s author, such as Creator, Owner, Editor, and so on. The name you assign must be unique, including case. For example, a mapped field named EmployeeNumber is the same as a mapped field named employeenumber. c. Click OK. 4. To change a mapped field, right-click the mapped field s name or icon in the view pane and select Properties. Type a new name and then click OK. 5. To delete a mapped field, right-click the mapped field s name or icon in the view pane and select Delete. Note: If you delete a field after the data source has been assigned to applications, the deletion is reflected automatically in any application that includes the data source. Step 2: Assign Native Fields to the Mapped Field. Note: After you define a mapped field, you can use an Extended Search wizard to map native fields to it instead of using the steps below. For details, see Run configuration wizards on page Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, select the Mapped Fields tab. 5. Click Assign. 6. On the Assign Mapped Field to Data Source window, select the field that you want to map within this data source and then click OK. 7. In the Mapped Fields with their Native Fields area, first select the mapped field. Then, from the Native field name list, select one or more native fields to which the mapped field should link, and click the forward arrow button. For example, a mapped field named Author might need to link to three native fields to adequately reflect the author s first name, middle name, and last name. 8. To specify how the data should be concatenated when the mapped field is returned in a result set, expand the contents of the mapped field, and select a field that you want to reposition. Click the up arrow button to move it higher in the list, or click the down arrow button to move it lower in the list. For example, you might want to position a field named First Name above a field named Last Name to ensure that the first name precedes the last name when search results are displayed. Tasks 23

32 9. To unmap a native field, expand the mapped field to which it is linked. Then select the native field that you want to disassociate, and click the backward arrow button. 10. To remove the definition of a mapped field from this data source, select the mapped field and click Unassign. When asked to confirm that you want to remove the mapped field, click Yes. 11. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. Note: If you configure a mapped field for a data source after the data source has been assigned to an application, the addition is reflected automatically in any application that includes that data source. Because mapped fields inherit their usage rules from the native fields to which they link, you do not need to update application-specific properties for this change. Set field usage controls When you define field usage controls, you specify which fields should be accessible to end users for field-based search and retrieval operations. Usage controls that you set at the data source level apply to the data source on a global level throughout the Extended Search domain. When you configure an application, you can modify these settings for the specific needs of users of that application. Restrictions that you set at the data source level can only be made more restrictive at the application level. If you define a field as searchable, you can configure the field to be not searchable in a given application. However, if you define a field as not searchable at the data source level, an application cannot later make that field searchable. You cannot select usage options for mapped fields. A mapped field derives its accessibility from the native fields it references. If all the native fields that are linked to a mapped field are searchable, the mapped field is searchable. If one or more of the native fields is not searchable, then the mapped field is not searchable. Note: Extended Search provides a wizard that makes it easy for you to configure the fields that you want to include in a result set. For details, see Run configuration wizards on page 38. Use the following procedure to define field usage controls: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, select the Field Usage tab. 5. For each listed native field, specify whether the field can be searched or returned: v If the field can be searched with a field-based query (one that refers to the field by name), click to put a checkmark in the Searchable column. Alternatively, you can select one or more fields and click the Searchable button to toggle the checkmarks for the selected fields. If you mark a field searchable, and the user does not submit a field-based query, some backend data source engines (such as Notes full text) might search all the fields in the data source. For this situation, you can control 24 Extended Search Administration

33 which fields are searched by configuring the Default field. This mapped field identifies the native fields that will be searched when no fields are named in the query. See Configuring data sources on page 92 for more information. v If the field can be returned whether for displaying in a result set, storing on disk, or retrieving from the data source click to put a checkmark in the Returnable column. Alternatively, you can select one or more fields and click the Returnable button to toggle the checkmarks for the selected fields. 6. If you change a field s usage definition after the data source has been categorized and assigned to applications, you must take the following steps to ensure that the change is reflected in any application-specific properties that are defined for the data source. a. If a field previously could not be searched or returned, and you make the field searchable or returnable at the data source level, you must decide, on an application-by-application basis, whether you want users to be able access this field when searching the data source, viewing a result set, or retrieving documents from the data source. For each application that refers to the data source, update the application-specific properties as appropriate. b. If a field previously could be searched, returned, or retrieved, and you make the field not searchable or not returnable at the data source level, you must update the application. When you view the application-specific properties for the data source, the previously available field appears grayed out and is unavailable. You must click OK or Apply for the change to take effect. 7. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. Define data source parameters When you define data source parameters, you specify options that are passed to the agent at the time that the data source is accessed. For example, you might want to instruct the agent to search with a specific date format. You can define parameters at the data source level and link level. If you define the same parameter at both levels, the link parameter value serves as the default for the data source parameter. You can override the default value to invoke different rules for this data source. Note: Extended Search provides a wizard that makes it easy for you to configure data source parameters and associate them with specific data sources or with all data sources of a given link type. For details, see Run configuration wizards on page 38. You can also define application parameters. These parameters provide search application designers with the means to provide information to users, such as information about the data sources that can be searched by the application. During data source discovery, many parameters are automatically configured for you. Predefined parameters on page 134 describes the parameters and provides information about which ones you can specify at the link or data source level. Use the following procedure to configure parameters for a data source: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. Tasks 25

34 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, select the Parameters tab. 5. To add a new parameter, click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. a. In Parameter name, type the parameter name. b. In Value, type a valid value for the parameter. c. In Usage, select Used by data source if the parameter is to be used exclusively by the link that connects to the data source. Select Used by application if the parameter will be used by your search applications. 6. To modify an existing data source parameter, select it from the list and type a new name, value, or description as necessary. 7. To remove a previously defined parameter, select the parameter from the list and click Delete. If this parameter was also defined as a link parameter for data sources of this type, the value defined at the link level still applies to this data source. 8. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. Define smart result set actions If you define actions for a data source, you enable Extended Search to return a smart result set, one that enables users to do more than just browse and retrieve documents in a result set. For example, you might want to connect users to a specific web site to further process a result item. For information about how Extended Search supports smart result sets, see Actions on page 89. You can define actions at the data source level and link level. If you define the same action at both levels, the link action value serves as the default for the data source action. You can override the default value to specify different actions for the results returned from this data source. Use the following procedure to configure result set actions for a data source: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to configure and select Properties. 4. On the Data Source Properties window, select the Actions tab. 5. To add a new action, click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. a. In Action name, type a name for this action. This name will either be displayed in the search results page or, depending on how your organization designed the results page, serve as the key to another label that is displayed in the search results. b. In Action URL, type the URL for the page that should be called when a user selects this action from the search results page. This page must contain logic to handle the requested action. 6. To modify an existing action, select it from the list and type a new name or URL. 7. To remove a previously defined action, select the action from the list and click Delete. If this action was also defined as a link action for data sources of this type, the value defined at the link level still applies to this data source. 26 Extended Search Administration

35 8. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this data source. Export and import data sources You can export data source definitions and import them to another Extended Search domain. This feature, for example, can help you move between test and production machines. To facilitate system testing, you can also import the exported data source data into the same domain. Exporting enables you to quickly deploy data sources without having to go through data source discovery again. Note: Extended Search provides a wizard that makes it easy for you to export and import multiple data sources. For details, see Run configuration wizards on page 38. When you export a data source, the system creates an XML file that includes all the native fields, mapped fields, field usage controls, parameters, actions, and categories that are configured for the data source (that is, it exports all the information defined in the Data Source Properties window). Note that the export process does not include any application-specific information that exists for the data source, such as application-specific field usage controls. If you include this data source in applications that exist in the target Extended Search domain, you must update the applications to specify your preferred application-specific properties. Warning You can modify the XML file that gets created before you import it. If you make changes, do so carefully. If you are unsure of what an XML entry is or what impact a change to its value will have, you are strongly cautioned to not change it; doing so can cause irreparable harm to the system. Use the following procedure to export a data source: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Select one or more data sources that you want to export, right-click, and select Export. 4. In the Export Selected Items window, type the fully qualified path for a file on the web server where you want to export the data source data, and then click OK. Use the following procedure when you are ready to import a data source: 1. Ensure that a server definition in the target system matches the server from which you exported the data source. 2. Ensure that mapped fields and the link, grammar, and category definitions in the target system match the values that you configured for the data source in the original system. 3. Click Data Sources by Link or Data Sources by Agent in the Navigator. 4. Right-click in an open area of the Data Source view and select Import. Tasks 27

36 5. In the Import Data window, type the fully qualified path for the file on the web server that contains the data source data that you want to import, and then click OK. 6. If the name of the agent that is associated with this data source is not the default (agent), update the Data Source Properties and select the correct agent name. Configure categories Categories enable users to search data sources as a group and simplifies their view of the system. When you add a data source to your search domain, Extended Search automatically associates it with a category that has the same name as the link that is used to connect to data sources of that type. To organize data according to how you intend to search it, you can put the data source in one or more different categories or create a new category for it. 28 Extended Search Administration You can create hierarchies of categories, a feature that enables you to organize data sources into categories and subcategories much the way Windows Explorer organizes files into folders and subfolders. This arrangement allows you to fine-tune the way users search data sources. For example, a search that queries a subcategory searches only the data sources associated with that subcategory. A search that queries a top-level category searches the data sources associated with it and the data sources associated with each of its subcategories. To configure a category, first create the category and then assign data sources to it. Create a category Use the following procedure to create a category: 1. Start the Administration interface. 2. Either right-click Categories in the Navigator and select New, or right-click in an open area of the Categories view and select New. 3. On the Category Properties window, type a unique and descriptive name for this new category. Category names and data source names must be unique. You cannot give the category the same name that you give to a data source or another category. 4. Select the locale for this category. This defines the language in which the category name will be displayed in the interface. 5. Click OK. There are a couple of ways to create and work with subcategories. v Follow the preceding steps to create a category but, when specifying the category name, type the name of an existing category that you want to use as the top-level category. Then type a backward slash (\) and the name of the new category. You can nest multiple levels of categories as necessary. For example: CategoryA\SubcategoryB CategoryA\SubcategoryB\SubcategoryC v In the Categories view, right-click the category that you want to make into a subcategory and select Copy. Then right-click the category that you want to use as the top-level category and select Paste. The category that you copied, along with any data sources it contains, will be nested below the target category. Categorize a data source There are two ways that you can assign a data source to a category. v If you are working in the Categories view, you can select one or more data sources and assign them to a category.

37 v If you are working in the Data Sources view, you can select an individual data source and, while defining other data source properties, assign the data source to a category. Note: When installed, Extended Search provides the following preconfigured categories. These groupings provide a convenient way for you to locate and select the data sources that you want to associate with a given category. v The [Not Categorized Data Sources] category contains only those data sources that have not been put into any category except the system-defined category. v The [System Defined Categories] category contains all the predefined categories, including: [All Data Sources], which contains all the data sources in the search domain, regardless of whether or not they have been categorized. When you discover a new data source, the system automatically adds it to the [All Data Sources] category. Link-specific categories. For each link type, Extended Search defines a category of the same name. When you discover a data source, the system automatically adds it to a category that has a name that matches the link name. Categories View: Use the following procedure to categorize a data source from within the Categories view: 1. Start the Administration interface. 2. Click Categories in the Navigator. The view pane lists all the categories in the search domain. The names of the data sources that belong to a category appear beneath the category name. 3. In the view pane, select one or more data sources that you want to put into a category, right-click, and select Copy. 4. Right-click the target category and select Paste. The data sources that you copied will be listed beneath the category s name. 5. Follow the procedures in Configure applications on page 30 to associate the category with one or more applications. 6. When you are ready to make your configuration changes available to Extended Search users, refresh the search domain. The following actions are also available for working with categories that are listed in the view pane. Note that you cannot perform these actions if you select [System Defined Categories] or [Not Categorized Data Sources]. v To remove a data source from a category, right-click the data source name and select Remove from this category. v To delete an entire category, right-click the category name and select Delete. v To rename a category, right-click the category name and select Properties. Type the new name and click OK. Data Sources View: Use the following procedure to categorize a data source from within the Data Sources view: 1. Start the Administration interface. 2. Click Data Sources by Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain, and organizes them by link name or agent name. 3. Right-click the data source that you want to categorize and select Properties. Tasks 29

38 4. Select the Categories tab. 5. To see only the names of available categories, click Collapse All. To see the names of the data sources that belong to each category, click Expand All. 6. In the Available Categories list, select the category to which you want to assign this data source and click Assign. 7. To remove the data source from a previously assigned category, select the category from the Assigned Categories list and click Unassign. 8. Click OK. 9. Follow the procedures in Configure applications to associate the category with one or more applications. 10. When you are ready to make your configuration changes available to Extended Search users, refresh the search domain. Configure applications When you configure a search application, you configure the particular fields that users can access through the application. You also specify the order in which fields are to be presented in the result set or retrieved from the data source. Think of applications as a way of grouping users for purposes of access control and search capability. 30 Extended Search Administration For example, members of an accounting department might be interested in retrieving financial data from a data source that also contains personnel data about employees. By setting up separate Accounting and Human Resources applications, you can ensure that each user group accesses only those fields that are relevant to their tasks. You can also present results in a manner meaningful to the intended use of the information. To enable users to search a new data source, the data source must belong to at least one category and the category must belong to at least one search application. When you add a data source to your search domain, Extended Search automatically updates any application that includes the category into which the new data source was added. To organize data according to how you intend to search it, you can put categories into different applications or create a new application. Follow the steps below to define an application. For each application, you must define application-specific properties. If you are moving between systems, you might also want to review the procedures for exporting and importing applications. 1. Start the Administration interface. 2. Click Applications in the Navigator. The view pane lists all the applications in the search domain. Listed beneath each application are the data sources that are assigned to the application, organized by category name. Extended Search provides two system-defined applications: v The Demo application is provided for example purposes. If you add a data source to a category that is included in the Demo application, you can use the Demo search template with the Demo application to search the new data source. For information about how to use this sample application, see Running search applications on page 60. v The [All Categories] application includes all the categories in the search domain. This list makes it easy for you to select the categories that you want to associate with a given application. The [All Categories] application is not available for searching.

39 3. To create an application, either right-click Applications in the Navigator and select New, or right-click in an open area of the Applications view and select New. The Application Properties window is displayed. a. In Application name, type a unique, descriptive name for this new application, one that will enable you to readily recognize its intended purpose. The application name can contain ASCII characters only, regardless of the language that is being used to run Extended Search. b. In Application description, type a description of the application. c. Select the Disable application check box only if you want to prevent users from accessing this application. For example, you might need to prevent access while you are working in test or debug mode. d. From the Entry broker list, select the check box for each broker that you want to use as an entry broker with this application. An entry broker is the first broker in a potential chain of brokers that receives queries and aggregates results. You must select at least one entry broker. Note: If you disable or rename a broker that is associated with this application, you must update the broker information here. Each application must identify the name of a valid, enabled broker. e. Click OK. 4. In the view pane, right-click a category that contains the data sources that you want to make available in this application and select Copy. Note: You must select a top-level category name. You cannot select a subcategory name. If a data source in a category that you are copying also belongs to a category that was previously associated with the application, the system preserves the original application-specific properties. That is, it uses the application-specific properties that were defined for the data source in the category that was first associated with the application. 5. Right-click the target application and select Paste. The following actions are also available for working with applications that are listed in the view pane. Note that you cannot perform these actions if you select the [All Categories] application. v To remove a category from an application, right-click the category name or icon and select Delete. You cannot remove individual data sources from an application. v To delete an application, right-click the application name and select Delete. Define application-specific properties For each data source in an application, take the following steps to specify how users of that application can access fields in the data source. Note: Extended Search provides a wizard that makes it easy for you to configure the fields that you want to include in a result set. For details, see Run configuration wizards on page In the Applications view pane, right-click a data source that you want to configure for this application and select Properties. Tasks 31

40 2. On the Application-Specific Properties window, select the Field Usage tab. By setting the appropriate field usage controls, you can ensure that the intended users of this application access only those fields that contain data relevant to their needs. If a field is marked Searchable, Returnable, or Retrievable, and you do not want users of this application to be able to search that field, view it in a result set, or retrieve it from the data source, click to remove the checkmark. Alternatively, you can select one or more fields, and then click the Searchable, Returnable, or Retrievable buttons to toggle the checkmarks for the selected fields. Note: If a field is not searchable or returnable in the base data source properties, it appears grayed out and is unavailable. You cannot make a field searchable, returnable, or retrievable at the application level unless you first make it searchable or returnable at the data source level. 3. Select the Result Set Field Order tab. This tab lists the fields that can be returned in a result set. To specify the order in which the fields should be presented in the result set, select a field that you want to reorder. Next, click the up arrow button or down arrow button to position it. By setting the appropriate order, you can ensure that search results are presented to users in a format that enables them to easily identify the item they want to view. 4. Select the Retrieve Field Order tab. This tab lists the fields that users can retrieve from the data source. To specify the order in which the fields should be retrieved, select a field that you want to reorder. Next, click the up arrow button and down arrow button to position it. To retrieve documents of a particular type, click File extension, and type or select the file extension for the type of document that users can retrieve. 5. When you have finished defining application-specific controls for the fields in this data source, click OK. 6. When you are ready to make your configuration changes available to Extended Search users, refresh the search domain. Export and import applications You can export application definitions and import them to another Extended Search domain. This feature, for example, can help you move between test and production machines. To facilitate system testing, you can also import the exported application data into the same domain. Note: Extended Search provides a wizard that makes it easy for you to export and import multiple applications. For details, see Run configuration wizards on page 38. When you export an application, the system creates an XML file that includes all the data source, category, and application-specific data configured for the application (that is, it exports all the information defined in the Application Properties and Application-Specific Properties windows). Warning You can modify the XML file that gets created before you import it. If you make changes, do so carefully. If you are unsure of what an XML entry is or what impact a change to its value will have, you are strongly cautioned to not change it; doing so can cause irreparable harm to the system. 32 Extended Search Administration

41 Use the following procedure to export application data: 1. Start the Administration interface. 2. Click Applications in the Navigator. The view pane lists all the applications in the search domain. 3. Select one or more applications that you want to export, right-click, and select Export. 4. In the Export Selected Items window, type the fully qualified path for a file on the web server where you want to export the application data, and then click OK. 5. After the export process successfully completes, a message indicates how many applications were exported. Click OK. Use the following procedure when you are ready to import application data: 1. Ensure that a server definition in the target system matches the server from which you exported the application. 2. Ensure that the application name does not already exist in the target system. 3. Ensure that any mapped fields that were defined for data sources in the exported application exist in the target system. 4. Click Applications in the Navigator. 5. Right-click in an open area of the Applications view and select Import. 6. In the Import Data window, type the fully qualified path for the file on the web server that contains the application data that you want to import, and then click OK. Usage Guidelines The import process properly handles multi-level categories and data sources that occur in multiple categories. It also automatically assigns applications to new broker names that were generated when server definitions on the target system were created. However, certain limitations exist: v If any like-named data sources already exist in the target system, the import process appends an identifier, DB_ID, to the name of the imported data source. v If any like-named categories already exist in the target system, the import process will not import the category. Configure links When you add a data source to Extended Search, the data source discovery process automatically associates it with an appropriate link type. The link definition identifies a C++ shared library or Java class that enables Extended Search to access data sources of that type. Extended Search comes with many predefined link types, which you should be able to use as provided. However, you can create a new link or copy an existing link and modify it as necessary. For more information about how Extended Search uses links, see Links on page For a description of the preconfigured link types, see Predefined links on page 123. Take the following steps to configure a new link: Tasks 33

42 1. Start the Administration interface. 2. Either right-click Links in the Navigator and select New or, if you are working in the Icon View, right-click in an open area of the Links view and select New. 3. On the Link Properties window, type a descriptive name for the link, one that indicates the type of data source that is supported by this link. 4. Optionally type a description of the link. 5. In Result set URL exit, type the name of the Java class that you want to use to control the format of URLs in the result set. If the class is in a package, type the full package name. Do not type the.class extension. For a description of the classes that the system-defined links use, see Result set URL classes on page 140. For information about developing your own custom class, see the discussion about the CustomHitReturn Interface in Extended Search Programming. 6. Click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. 7. In Discoverer library name, type the name of the C++ shared library or Java class that contains logic to discover data sources of this type. Be sure to specify the correct extension (.dll,.a,.so, or.class). This discoverer will be called when you add data sources of this link type. You can use an existing discoverer or develop your own. For information about developing discoverers, see Extended Search Programming. 8. In Link library name, type the name of the C++ shared library or Java class that contains logic to perform search and retrieval operations in data sources that are associated with this link type. Be sure to specify the correct extension (.dll,.a,.so, or.class). You can use an existing link or develop your own. For information about developing links, see Extended Search Programming. Note: Links typically work with an agent to retrieve documents that are returned in a result set from the target data source. However, a result set from a web source contains URLs that allow the browser to retrieve the document instead of the link and agent. 9. In Platform, select the operating system where this discoverer and link will run. 10. If you want to associate another discoverer or link library with this link (for example, to use on a different platform), repeat steps 6 through If you want to delete a library that was previously associated with this link, select its name in the list and click Delete. 12. If you want to define or modify link parameters for this link, see Define link parameters on page If you want to define or modify result set actions for this link, see Define link actions on page Click OK to save your changes and exit. Take the following steps to modify a link or disseminate new and changed links to your user community: 1. To change an existing link, right-click the link s name or icon in the view pane and select Properties. You cannot change the name of a link. You can change the description as needed. You can change the names of the libraries that are associated with the link. You can also add or change link parameters and result set actions. After making your changes, click OK. 34 Extended Search Administration

43 2. To copy a link, click the Icon View icon, right-click the link s icon in the view pane and select Copy. Then right-click in an open area of the view pane and select Paste. You must specify a new name for this link. Make other changes as necessary and then click OK. When you copy a link, you copy its complete definition, including any link parameters and result set actions that are defined for it. 3. To delete a link, right-click the link name or icon in the view pane and select Delete. 4. To associate the link with one or more data sources: a. Click Data Sources By Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain. b. Select one or more data sources that you want to associate with this link, right-click, and select Properties. c. At Link, select the name of your new link. d. Either click OK or refer to Data source properties single data source on page 180 for information about specifying other information about the data source. 5. When you are ready to make your configuration changes available to Extended Search users, refresh the search domain. If you associated a custom shared library with this link, you must shut down and restart Extended Search to activate it. Define link parameters Link parameters provide a way for you to specify options for all the data sources that are associated with a given link type. If you define the same parameter at the data source level, the link parameter s value serves as the default value for the data source parameter. The value of the data source parameter overrides the link-level value for that particular data source. During data source discovery, many parameters are automatically configured for you. Predefined parameters on page 134 describes the parameters and provides information about which ones you can specify at the link level. Note: Extended Search provides a wizard that makes it easy for you to configure data source parameters and associate them with specific data sources or all data sources of a given link type. For details, see Run configuration wizards on page 38. Use the following procedure to configure a link parameter: 1. Start the Administration interface. 2. Click Links in the Navigator. The view pane displays all the links in the search domain 3. Right-click the link for which you want to configure parameters and select Properties. 4. On the Link Properties window, select the Parameters tab. 5. Take the following steps to define a new link parameter: a. Click Add Row. A blank line is added beneath the currently selected parameter or, if none is selected, to the end of the list. b. In Link parameter name, type the name of the parameter that you want to set for this link. Tasks 35

44 c. In Value, type the parameter value. When the link gets initialized, the parameter will be set to this value and it will apply to all operations within data sources of this link type. d. In Description, briefly describe the purpose of this parameter. 6. To modify an existing link parameter, select it from the list and type a new name, value, or description as necessary. 7. To remove a previously defined parameter, select the parameter from the list and click Delete. If this parameter was also defined as a data source parameter, the value defined at the data source level still applies. 8. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this link. Define link actions If you define actions for a link, you enable Extended Search to return a smart result set, one that enables users to do more than just browse and retrieve documents in a result set. For example, you might want to connect users to a specific web site to further process a result item. For information about how Extended Search supports smart result sets, see Actions on page 89. You can define actions at the data source level and link level. An action that you define at the link level applies to all data sources of that link type. If you define the same action at both levels, the link action value serves as the default for the data source action. The value of the data source action can override the link-level value for that particular data source. Use the following procedure to configure result set actions for a link: 1. Start the Administration interface. 2. Click Links in the Navigator. The view pane lists all the links in the search domain. 3. Right-click the link that you want to configure and select Properties. 4. On the Link Properties window, select the Actions tab. 5. To add a new action, click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. a. In Action name, type a name for this action. This name will either be displayed in the search results page or, depending on how your enterprise designed the results page, serve as the key to another label that is displayed in the search results. b. In Action URL, type the URL for the page that should be called when a user selects this action from the search results page. This page must contain logic to handle the requested action. 6. To modify an existing action, select it from the list and type a new name or URL. 7. To remove a previously defined action, select the action from the list and click Delete. If this action was also defined as a data source action, the value defined at the data source level still applies. 8. Either click OK to save your changes and exit, or click Apply and select another tab to continue defining options for this link. Configure grammars When you add a data source to Extended Search, the discovery process automatically associates it with an appropriate grammar. The grammar definition 36 Extended Search Administration

45 identifies a C++ shared library or Java class that is responsible for translating a GQL statement into the native search grammar of the data source. Extended Search comes with many predefined grammars, which you should be able to use as provided. However, you can create a new grammar or copy an existing grammar and modify it as necessary. For more information about how Extended Search uses grammars, see Grammars on page 121. For a description of the preconfigured grammars, see Predefined grammars on page 121. Take the following steps to configure a new grammar: 1. Start the Administration interface. 2. Either right-click Grammars in the Navigator and select New, or if you are working in the Icon View, right-click in an open area of the Grammars view and select New. 3. On the Grammar Properties window, type a descriptive name for the grammar, one that indicates the type of data source or link that is targeted by this grammar. 4. Optionally type a description of the grammar. 5. Click Add Row. A blank line is added beneath the currently selected row or, if none is selected, to the end of the list. 6. In Library name, type the name of the C++ shared library or Java class that should be called when a user submits a query against data sources that are associated with this grammar. Be sure to specify the correct extension (.dll,.a,.so, or.class). You can use an existing grammar or develop your own. For more information about developing an Extended Search translator, see Extended Search Programming. 7. In Platform, select the operating system where this grammar will run. 8. If you want to associate another library with this grammar (for example, to use on a different platform), repeat steps 5 through If you want to delete a library that was previously associated with this grammar, select its name in the list and click Delete. 10. Click OK. Take the following steps to modify a grammar or disseminate new and changed grammars to your user community: 1. To change an existing grammar, right-click the grammar s name or icon in the view pane and select Properties. You cannot change the name of a grammar. You can change the description as needed. You can change the name of the library that is associated with the grammar. After you make your changes, click OK. 2. To copy a grammar, right-click the grammar s name or icon in the view pane and select Copy. Then right-click in an open area of the view pane and select Paste. You must specify a new name for this grammar. Make other changes as necessary and then click OK. 3. To delete a grammar, right-click the grammar s name or icon in the view pane and select Delete. 4. To associate a new grammar with one or more data sources: a. Click Data Sources By Link or Data Sources by Agent in the Navigator. The view pane lists all the data sources in the search domain. Tasks 37

46 b. Select one or more data sources that you want to associate with this grammar, right-click, and select Properties. c. At Grammar, select the name of your new grammar. d. Click OK. 5. When you are ready to make your configuration changes available to Extended Search users, refresh the search domain. If you associated a custom shared library with this grammar, you must you must shut down and restart Extended Search to activate it. Run configuration wizards The Administration interface provides several wizards that can make it easier for you to perform certain configuration tasks. Before using most of the wizards, you should be familiar with the basic task that is being facilitated and understand any requirements that apply when the task is performed manually. Wizard Add data sources Description Use this wizard to add data sources to your search domain. This wizard discovers data sources that you select, puts them in one or more categories, updates all applications that include those categories, and refreshes the search domain. This wizard is helpful to both new and experienced Extended Search administrators. For information about how to use it, select Search Domain and then select the Add data sources link. Configure mapped fields Use this wizard to select a mapped field to which you want to map one or more native fields. Selection criteria enables you to easily map fields that belong to similarly named data sources or data sources of a specific link type, fields that have similar names or are of a specific data type, or any combination of these criteria. Review the following topics before you use this wizard: v Mapped fields on page 119 v Define mapped fields on page 22 Configure data source parameters Use this wizard to define a data source parameter and apply it to a specific data source, data sources with similar names, or data sources of a specific link type. Review the following topics before you use this wizard: v Parameters on page 132 v Define data source parameters on page 25 Configure result set fields Use this wizard to design a result set and specify whether it applies to all data sources in a search application, or to data sources within the application that have similar names or that are of a specific link type. Review the following topics before you use this wizard: v Set field usage controls on page 24 v Define application-specific properties on page Extended Search Administration

47 Wizard Export and import data Description Use this wizard to export data sources, applications, or the configuration database, and to import data sources and applications that you previously exported. Review the following topics before you use this wizard: v Export and import data sources on page 27 v Export and import applications on page 32 v Backing up and restoring data on page 83 Take the following steps to run a configuration wizard: 1. Start the Administration interface. 2. Click the Wizards icon above the Navigator and select the wizard name. Alternatively, click Wizards in the Navigator. When the view pane presents the list of wizards, right-click the wizard that you want to run and select Display. 3. After the wizard starts, follow the online instructions to complete the task. Refresh web source definitions As discussed in Web sources on page 146, Extended Search comes preconfigured with data source definitions for many popular web search sites. Each web data source is preconfigured with a standard set of field and parameter definitions. A single file contains the interface descriptions for all of the supported web sites. If you install an Extended Search server on a Windows platform, you can configure the server to read this file at startup and at a scheduled frequency. Use the following procedure to specify how often Extended Search should refresh these predefined web source definitions. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search server that you want to work with and select Properties. 4. On the Server Properties window, select the Extended Search tab. Page 204 describes these options. 5. In the Internet area, review the proxy settings for the web server. Typically you do not need to change these settings. 6. Ensure that the Disable download check box is clear. 7. In Download frequency, specify how often, in hours, you want the server to refresh the definitions of the web sources. 8. Click OK. Search my own web sites As discussed in Web sources on page 146, Extended Search is preconfigured with data source definitions for many popular web search sites. If there are other web sources that you want to make available for searching, you can easily add them to your Extended Search search domain. The procedure below describes one approach for adding support for your own web sites. For guidelines on using other approaches, see Adding support for your web sites on page 148. Tasks 39

48 The first step is to contact Intelligent Algorithms Enterprises, Ltd. (infogist) to obtain the IBM Lotus Extended Search SBB Authoring Product, a tool that enables you to build your own web source definition file (.sbb file) for Extended Search. For more information, and to view a sample.sbb file, access the following URL: Note that the Intelligent Algorithms code is not fully enabled for national language support. Through the Extended Search Web link and an.sbb file, you can search for languages that use the ISO code page (such as English, French, German, Portuguese, Swedish, and so on). You cannot search double byte character set (DBCS) languages such as Korean, bi-directional languages such as Hebrew, or other non-iso languages. After you create an.sbb file that defines your web sources, use the following procedure to configure support for your new web sources in Extended Search: 1. Ensure that your custom.sbb file is stored in the base Extended Search installation directory. 2. Start the Administration interface. 3. Discover the new web sources: a. Click Servers in the Navigator. The view pane lists all the servers in the search domain. b. Right-click the server where the new web sources are available and select Discover Data Sources. c. In Type of source to discover, select Web. d. In ESWebConfig link parameter value, type the name of your custom.sbb file. e. In Language used, select the locale of the web sources. f. In Code page used, select the code page that will be used when communicating with the data source. See Code page support on page 130 for information about the default code page support in Extended Search. g. Click Start Discovery. h. After the web sources that match those defined in your.sbb file are displayed, select the ones that you want to add to your search domain, and then click Add to ES. i. Click Close to close the Discover Data Sources window. 4. Configure the Web link: a. Click Links in the Navigator. The view pane displays all the links in the search domain. b. Right-click the Web link in the view pane and select Properties. c. Click the Parameters tag, and then select the ESWebConfig Value field. d. Append a question mark (?) and the name of your.sbb file to the existing ESWebConfig value. For example: deswebdef.sbb?custom.sbb. e. Click OK. 5. Group the new web data sources into a category, and associate the category with at least one application. 6. To update the system with your configuration changes, refresh the search domain. 40 Extended Search Administration

49 Configure a web crawler As discussed in Web crawlers on page 141, you can configure an Extended Search server to automatically traverse web sites and download data to a local repository. You can then use an indexing program to index the data, include it in your Extended Search domain, and make it available to users through a search application. To use web crawling, you must enable the Extended Search server to use it, and then specify options to control when the crawlers should run and which web sites they should crawl. Each crawler that you enable uses a unique configuration file that identifies the specific web site to be crawled. Web crawler configuration file on page 142 describes escrawler.xml, a sample configuration file that is included with the product. Use the following procedure to configure a web crawler. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the web server on which you want to enable web crawling and select Properties. 4. Select the Web Server tab, and then select Scheduled Crawling to expand the options. 5. Ensure that the Enable crawling check box is selected. 6. In Check for scheduled crawling, specify how often (in minutes) the crawler process on the web server should check for the existence of newly scheduled crawler requests. 7. Click OK. Once enabled, when a crawler process detects a crawler request that is ready to run, it sends the request to the Extended Search server for processing. 8. Right-click the Extended Search server that you want to use for crawling and select Properties. 9. Select the Extended Search tab, then select Crawler to expand the options. 10. Click Add Crawler. A blank line is added beneath the currently selected crawler or, if none is selected, to the end of the list. 11. Select the Enable check box to enable the newly added web crawler process on this server. 12. In Crawler, select the type of web crawler you want to use. In this release, you must select the escrawler crawler. 13. In Configuration File, type the fully qualified path for the XML configuration file that is used by this web crawler. 14. In Start Day, type the day of the week that this web crawler should begin crawling the web. The default start day is Sunday. 15. In Start Time, type the time of day, using a 24 hour clock, that this web crawler should begin crawling the web. The default start time is midnight (24:00). 16. In Frequency, type a value that indicates how often web crawling should be repeated. (This value works in conjunction with Interval.) The default frequency is once (1), such as once every week. Tasks 41

50 17. In Interval, select a value that indicates how often web crawling should be repeated. (This value works in conjunction with Frequency.) You can configure the web crawler to run on a monthly, weekly, daily, or hourly basis. The default interval is weekly. 18. Click OK. 19. To update the system with your changes, refresh the search domain. You can also take the following actions when configuring a web crawler: v If you want to temporarily suspend a particular crawler, clear the Enable check box by its name to disable it. v If you want to remove a previously defined crawler from this server, select it from the list and click Delete. Refresh the search domain Any time that you update the configuration database, you need to refresh the search domain to disseminate the changes to your user community. Note: If you change configuration values on the Extended Search tab of the Server Properties window, you must also shut down and restart the Extended Search server for the changes to take effect. These tasks include modifying the server heap size, Broker settings, Agent settings, and Internet settings. If you change event logging options for discoverers and web crawlers, you must also shut down and restart the Extended Search server for the changes to take effect. Take the following steps to refresh the domain and enable users to access new or changed configuration data. 1. Start the Administration interface. 2. Right-click Search Domain in the Navigator and select Refresh domain. In a large production environment, this action can take some time to complete. 3. Instruct your users to close and then re-open their web browsers. Users will not experience the changes until they start a new browser session. Refresh a single server For certain configuration changes, you can refresh the configuration of only the server, or servers, that are affected by the change. For example: v You can update event logging or tracing options for an individual server without having to refresh the entire search domain. v If you update properties for the web server, you can refresh just the web server to have the changes take effect. v For data source parameters that are not cached by the web server, such as impersonation, you need to refresh the configuration of only the server that processes requests for that data source. Note: In most cases, you must refresh the search domain to disseminate changes throughout the system and, in some cases, you must shut down and restart Extended Search. For example, if you update data sources, categories, or applications, you must refresh the domain to make the changes known to all of the servers or, at a minimum, refresh each of the servers that are impacted by the change (such as the web server and entry broker server). 42 Extended Search Administration

51 Take the following steps to refresh the configuration of an individual server: 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the server for which you updated configuration data select Refresh configuration. 4. Instruct your users to close and then re-open their web browsers. Users will not experience the changes until they start a new browser session. Use search templates to test changes Extended Search provides several search templates that enable you to verify changes that you make to the configuration database. Note: The templates demonstrate many of the Extended Search features that you can use when you develop your own search applications. These templates have been partially tested, and are provided to illustrate functionality. They might not look and function the same in all environments. Use the following procedure to test your configuration changes: 1. Refresh the search domain. 2. Under Search Templates in the Navigator, select one of the following categories: v The Demo template is the most comprehensive and tested example. Use this template to test configuration changes. To quickly search just the Extended Search documentation and web sources with default search options, use the Simple Sample template. v The Other Samples demonstrate ways that you might provide support for multiple queries and advanced GQL queries. 3. When the view pane lists the available search templates, right-click the template that you want to use and select Display. Note: On Windows systems, a template that you select will be displayed automatically in a new web browser window. On UNIX systems, you must first open a web browser before you select the action to display a search template. The search templates have been implemented in different ways to illustrate different application development approaches. v JKM templates. These search templates illustrate how to incorporate search functionality in HTML pages by embedding Extended Search proprietary Java Knowledge Management (JKM) tags and beans. v JSP bean templates. These search templates illustrate how to incorporate search functionality in JSP pages by embedding Java code and Extended Search beans. The templates conform to the Sun JavaServer Pages standard. v JSP tag templates. These search templates illustrate how to incorporate search functionality in JSP pages by embedding an Extended Search tag library. The templates, which conform to the Sun JavaServer Pages tag standard, have names that begin with tag. If you use a Domino server, you must select a JKM template. If you use a WebSphere server, you can select either a JKM template or a JSP template. Tasks 43

52 Display Name Template Name Description Demo Simple Sample Other Samples Multiple Query Sample Advanced GQL Sample AllOptions.txt (JKM) AllOptions.jsp (JSP beans) tagalloptions.jsp (JSP tags) simplesample.txt (JKM) simplesampe.jsp (JSP beans) tagsimplesample.jsp (JSP tags) tagmultiqueryloadpage.jsp (JSP tags) gqladv.txt (JKM) gqladv.jsp (JSP beans) This template, which uses a tabbed page format and is the most comprehensive example, includes most of the options supported by Extended Search. It allows you to easily construct a complex GQL query, schedule the query, save the query and search results, specify constraints, and specify a message logging preferences. The result set includes options for viewing detailed or summary information and for selecting result set actions. This template presents a simple query interface that uses default search options to search sources that are preselected by the search application (in this case, the Extended Search documentation and web sources). This template demonstrates how you can submit multiple queries at the same time from a single interface. Each query can search different categories and data sources. The same search options apply to each query, and the result set combines the results from all queries. This template demonstrates the power of searching with GQL. It uses a tabbed format and includes features that enable you to build a complex GQL query by selecting fields, operators, and boolean constructs. 4. After the template has been displayed, specify search terms, select the categories or data sources that you want to search, and optionally set search and retrieval options. When ready, select the action to submit your search request. v If you are testing the initial installation of Extended Search on UNIX, query the ES Documentation category. v If you are testing the initial installation of Extended Search on Windows, query the ES Documentation or Web category. Many web search sites are preconfigured in Extended Search, which enables you to submit queries without configuring your own data sources. If you experience problems when you try to connect to the web, consult the FAQ on the Extended Search web site. 5. After the search results are returned, click various entries to verify your ability to view documents that are listed in the result set. 44 Extended Search Administration

53 Usage Notes v When you launch one of these search templates, the Demo application name is hard-coded in the link from the Administration interface. To test your own search application, change the application name in the URL that is displayed in the browser. For example, change &AppID=Demo to &AppID=Finance. v If you open a search template from within the same browser session, you need to refresh the browser to effect the change. Press the Shift key while you click the browser s Refresh button. v The Extended Search templates set a limit on the number of results that can be returned from each data source. Certain web search engines impose their own limitations on the number of results that can be returned. This number might be less than the default Extended Search limitation. Log and view events Extended Search provides features that enable you to track server activity and resolve problems that arise. Event management is a two-step process: v You must first enable event logging on individual Extended Search servers and web servers. v You can then use the event log viewer to review the messages and determine the appropriate corrective action. v You can optionally configure plug-ins to perform custom logging, both at the servlet and broker levels. Enable event logging Use the following procedure to log error messages according to the severity of the event that occurs. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers defined in the search domain. 3. Right-click the Extended Search server or web server on which you want to enable event logging and select Properties. You can enable event logging only at the server level. You cannot enable logging on a product component basis. 4. On the General tab of the Server Properties window, click Event logging to expand the entries beneath it. 5. For each type of event (critical, non-critical, warning, or informational), select the Write to log check box if you want errors of that type to be written to the event log. Clear the check box if you do not want to log error messages for a particular event type. 6. If your Extended Search support representative instructs you to trace a component s processing, click Tracing to expand the entries beneath it. a. Select the Enable tracing check box. b. In Trace levels, type the trace level (or range of levels) appropriate to the process you need to diagnose (see page 227 for a description of valid trace codes). Separate trace codes by a comma (,). Join a range of trace codes with a dash ( ). For example: , 20 40, Tasks 45

54 c. In Trace log file, type the fully qualified path for a location where the trace log file should be created. 7. Click OK. 8. To have your event logging choices take effect, refresh the search domain. Note that the Extended Search discoverer and web crawler responders do not get refreshed on a Refresh domain request. To have your event logging choices take effect for discovery and web crawling, you must stop and restart the Extended Search server. View event logs If you enabled event logging for particular types of events, error messages will be logged to the Extended Search configuration database. You can then use the event log viewer to view and administer the messages. 1. Start the Administration interface. 2. Click Event Log Viewer in the Navigator. The view pane displays the contents of the event log organized by the names of the Extended Search servers that logged events. For each event, the viewer displays the name of the server that logged the event, the event type or severity level of the event, the time and date the event was logged, and the error message that describes the event. 3. By default, events are listed in descending order. You can toggle the sort order on a column-by-column basis. To sort entries in ascending order, click the column you want to reorder. Click the column again to return to descending order. 4. To get details about a particular event, either double-click it, or right-click it and select Properties. The Event Properties window shows the name of the file that generated the event, the method that defines the event, and the line number in the code where the event originates. 5. If you want to print the event log, click Print Preview. A printable version of the event log is displayed, enabling you to use your browser s print function to print it. This feature is provided as a template, msglistprintpreview.txt, which you can customize for your reporting purposes. 6. If you want to save the contents of the event log to a file on the server, click Save As. When prompted, type the fully qualified path for a file in which to save the event log messages. This feature is provided as a template, msglistsave.txt, which you can customize for your reporting purposes. 7. If you want to clear the contents of the event log, click Clear All. To conserve disk space and to keep the size of the event log to a manageable length, you should clear the event log on a regular basis. If the log reaches a certain size, the system will automatically append its contents to the esevents.log file in the log subdirectory of the Extended Search installation directory and clear the data from this view. A warning message tells you when this has action has occurred. Enable a servlet-level logging plug-in You can instruct the Extended Search servlets to keep a log of their activities and to record message traffic between search applications and the web server. To implement logging at the servlet level, your application developer must create a Java class to handle it. For instructions, see the discussion of Plug-ins in Extended Search Programming. You must then use the following procedure to configure the server to call this plug-in. 46 Extended Search Administration

55 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the web server that you want to configure and select Properties. 4. On the Server Properties window, select the Web Server tab and then click Logging to expand the entries beneath it. 5. In Logging plug-in name, type the name of the Java class that you want the servlet to use. If the class is in a package, type the full package name (such as mypackage.myclass). Do not specify the.class extension. 6. In Log file name, type the fully qualified pathname of a file to which logged output should be written. 7. Click OK. 8. To begin using the plug-in with future search requests, refresh the search domain. Tip The Extended Search installation comes with a sample Java class, CustomLogger.java, that you can use as a guideline for encoding your logging preferences. For more information about this sample code, see the discussion of Plug-ins in Extended Search Programming. Enable a broker-level logging plug-in You can instruct the broker to keep a log of its activities and to record message traffic between search applications and the target data sources. To implement broker-level logging, your application developer must create a plug-in to handle it. For instructions, see the discussion of Plug-ins in Extended Search Programming. You must then use the following procedure to configure the broker to use the plug-in: 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search server that runs the broker that you want to configure and select Properties. 4. On the Server Properties window, select the Extended Search tab and then click Broker to expand the entries beneath it. 5. In Logging plug-in name, type the file name and extension of the C++ shared library that you want the broker to use. If the library is not stored in the Extended Search installation directory, you must type its fully qualified pathname. 6. Click OK. 7. To begin using the plug-in with future requests that this broker receives, refresh the search domain. Tasks 47

56 Tip The Extended Search installation comes with sample shared libraries for plugging in logging rules at the broker level: SampleBrokerLoggingPlugin.dll (Windows), libsamplebrokerloggingplugin.a (AIX) and libsamplebrokerloggingplugin.so (Solaris and Linux). To test broker-level logging, enter one of these file names in step 5 on page 47. For more information about these samples, see the discussion of Plug-ins in Extended Search Programming. Configure security options To enhance the security of your Extended Search system, you can validate users against a data source s access control list, and enable authentication at the servlet, broker, and agent levels. Security-related configuration tasks include the following: v Using impersonation to verify user access permissions v Enabling servlets to perform web authentication v Enabling the broker to authenticate users v Enabling the agent to authenticate users v Changing the database administrator password Before performing any of these tasks, be sure to review the discussion of Security (beginning on page 149) to learn about the various ways that security has been implemented in Extended Search. Enable impersonation As discussed in Impersonation on page 153, when a user submits a query, Extended Search does not automatically check the permissions granted to that user. You can enhance security by setting a parameter, Impersonate, that instructs Extended Search to determine whether the user ID passed in on the request has permission to access the data source. You can enable impersonation for the following types of data sources: Notes 4.6.4, Notes 5.0, Discovery Server, Domino.Doc, Microsoft Access, ODBC, QuickPlace, SQL Server, Web, and WebSphere Portal Search. If you enable impersonation, Extended Search will attempt to establish a connection to the target data source by using the user ID and password passed in on the request. If you plan to configure the link for a persistent connection, see the description of pre-defined parameters on page 134 for information about how impersonation works in conjunction with the DatabaseRemainOpen parameter. If you set Impersonate as a link parameter, the system verifies the user s authority to access all sources that are supported by that link. If you set Impersonate as a data source parameter, the system verifies the user s authority to access just that data source. Use the following procedure to enable impersonation at the link level: 1. Start the Administration interface. 2. Click Links in the Navigator. The view pane lists all the links in the search domain. 48 Extended Search Administration

57 3. Right-click the link for which you want to enable impersonation and select Properties. 4. On the Link Properties window, select the Parameters tab and then click Add Row. A blank line is added beneath the currently selected parameter or, if none is selected, to the end of the list. 5. In Link parameter name, type Impersonate. 6. In Value, type Yes. 7. In Description, briefly describe the purpose of this parameter. 8. Click OK. Use the following procedure to enable impersonation at the data source level: 1. Start the Administration interface. 2. Click Data Sources By Link or Data Sources By Agent in the Navigator. The view pane The view pane lists all the data sources in the search domain. 3. Right-click the source for which you want to enable impersonation and select Properties. 4. On the Data Source Properties window, select the Parameters tab and then click Add Row. A blank line is added beneath the currently selected parameter or, if none is selected, to the end of the list. 5. In Parameter name, type Impersonate. 6. In Value, type Yes. 7. In Usage, select Used by data source. 8. Click OK. Enable a servlet-level security plug-in To control which users are able to access Extended Search data sources, you can set options for servlet-level security and plug in your own rules for user authentication, access control, or user ID mapping. To set up this level of support, your application developer must create a Java class to handle it. For instructions, see the discussion of Plug-ins in Extended Search Programming. You must then use the following procedure to configure the server to call this plug-in. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the web server that you want to configure and select Properties. 4. On the Server Properties window, select the Web Server tab and then click Security to expand the entries beneath it. 5. In Security plug-in name, type the name of the Java class that you want the servlet to use. If the class is in a package, type the full package name (such as mypackage.myclass). Do not type the.class extension. 6. Click OK. 7. To begin using the plug-in with future search requests, refresh the search domain. Tasks 49

58 Tip The Extended Search installation comes with two sample Java classes, CustomAuthenticator.java and CustomAuthenticator2.java, that you can use as a guideline for encoding your security preferences. For more information about this sample code, see the discussion of Plug-ins in Extended Search Programming. Enable a broker-level security plug-in Upon receipt of a search request, the broker can be instructed to determine whether or not the user is authorized to access all the target data sources, access a data source with the parameters specified in the request, issue a search request against a particular data source, or retrieve content from a data source. To implement this level of access control, your application developer must create a plug-in to handle it. For instructions, see the discussion of Plug-ins in Extended Search Programming. You must then use the following procedure to configure the broker to call this plug-in. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search server that runs the broker that you want to configure and select Properties. 4. On the Server Properties window, select the Extended Search tab and then click Broker to expand the entries beneath it. 5. In Security plug-in name, type the file name and extension of the C++ shared library that you want the broker to use. If the library is not stored in the Extended Search installation directory, you must type its fully qualified pathname. 6. Click OK. 7. To begin using the plug-in with future requests that this broker receives, refresh the search domain. Tip The Extended Search installation comes with sample shared libraries for plugging in security rules at the broker level: SampleBrokerSecurityPlugin.dll (Windows), libsamplebrokersecurityplugin.a (AIX), and libsamplebrokersecurityplugin.so (Solaris and Linux). To test broker-level security, enter one of these file names in step 5. For more information about these samples, see the discussion of Plug-ins in Extended Search Programming. Enable an agent-level security plug-in Prior to sending a query to a search engine, an agent can be instructed to determine whether or not the requesting user has permission to access the target data source. After search results are returned, the agent can assess the responses one by one. It can determine whether the user is allowed to view particular references in a result set or retrieve referenced documents. 50 Extended Search Administration

59 To implement this level of access control, your application developer must create a plug-in to handle it. For instructions, see the discussion of Plug-ins in Extended Search Programming. You must then use the following procedure to configure the agent to call this plug-in. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search server that runs the agent that you want to configure and select Properties. 4. On the Server Properties window, select the Extended Search tab and then click Agents to expand the entries beneath it. 5. Select the agent that you want to configure from the list of agents. 6. In Agent plug-in, type the name of the plug-in that you want the agent to use. v For a C++ shared library, type the file name and extension. If the library is not stored in the Extended Search installation directory, you must type its fully qualified pathname. v For a Java class, type the class name or, if the class is in a package, type the full package name. Be sure to include the.class extension. 7. Click OK. 8. To begin using the exit with future tasks that this agent handles, refresh the search domain. Tip The Extended Search installation comes with sample code for plugging in security rules at the agent level: SampleAgentSecurityPlugin.dll (Windows), libsampleagentsecurityplugin.a (AIX), libsampleagentsecurityplugin.so (Solaris and Linux), and SampleESAgentUserExit.java (all platforms). To test agent-level security, enter one of these file names in step 6. For more information about these samples, see the discussion of Plug-ins in Extended Search Programming. Change the database administrator password During installation, Extended Search encrypts the password for the DB2 or Oracle configuration database administrator. Because it is encrypted, you cannot change the password simply by updating the value for the es_database_password entry in the admin.properties file. Instead, use the es_password command. To run es_password.bat on Windows, change to the Extended Search installation directory and enter the following command, where current_password is your current database administrator password and new_password is the password that you want to change it to: es_password current_password new_password To run es_password.sh on UNIX, change to the bin subdirectory of the Extended Search installation directory and enter the following command, where current_password is your current database administrator password and new_password is the password that you want to change it to:./es_password current_password new_password Tasks 51

60 If the current password that you specify matches the currently stored database administrator password, your new password will be encrypted and stored in the admin.properties file. Configure scalability options To enhance the scalability of your Extended Search system, you can perform such tasks as configuring multiple brokers and separating brokers from the web server. You can also configure remote agents and partition multiple agents on a single machine. Scalability-related configuration tasks include the following: 52 Extended Search Administration v Separating brokers from the web server v Configuring remote agents v Partitioning agents and data sources on a single machine v Configuring multiple brokers Before performing any of these tasks, be sure to review the discussion of Scalability (beginning on page 158) to learn about the various ways that you can extend your Extended Search system. Separate brokers from the web server During installation, you have the option of installing Extended Search Server component separate from the Web Server component. To accommodate future growth in the number of users and data sources that constitute your Extended Search domain, you should install these components on separate machines. This approach allows the resources on each machine to be dedicated to a given task. The broker handles all search request processing and aggregates the result set. The web server handles all the web processing that is required by Extended Search. If you already installed Extended Search on the same machine with the web server, it is still possible to separate the two processes. The following procedure assumes that you want to retain the original machine to use as your Extended Search server and that you will set up the web server on the new machine. 1. Follow procedures in Extended Search Installation to install only the Web Server component on a machine where you previously installed a WebSphere or Domino web server. 2. Start the Administration interface, and click Servers in the Navigator. 3. In the view pane, right-click the web server icon and select Properties. 4. On the General tab of the Server Properties window, type information about the machine where you installed the web server (the server name, host name, IP address, port, and platform). 5. Click OK. Configure remote agents Agents are server tasks and thus need an Extended Search server to start them up, handle their communications, and provide other client-server architectural needs. Accordingly, there needs to be configuration data for the Extended Search server and its agent tasks on each server. A remote server, however, does not have to host the broker. You might want to set up remote agents to co-reside with their respective data sources, yet allow a central broker to distribute incoming search requests to the appropriate agents and aggregate results into a single result set. Take the following steps to set up such a configuration.

61 1. Follow procedures in Extended Search Installation to install only the Extended Search Server component on the machine that hosts the sources you want to search. 2. Start the Administration interface. 3. Follow the procedure in Add another search server on page 16 to add an icon for the new server. When you define Server Properties for this new server, select the Extended Search tab, click Broker to expand broker options, and then select the Disable broker check box. This action prevents a broker process from running on the new server. 4. In the view pane, click the Graph View icon. 5. Right-click the new server s icon and select Connect to. 6. Click the search server that hosts the broker (or a broker in the broker chain) that you want to use for processing search requests for the new server. A line shows that the two servers are connected. 7. Click Apply Connections. 8. Run data source discovery on the new server to add data sources. Configure data source fields, group the data sources into categories, and associate the categories with applications as appropriate. 9. To disseminate your changes to the user community, refresh the search domain. When a search request that targets the data sources on the new server is received, the broker automatically forwards it to the agent on that machine. The agent processes the request and returns search results to the broker. The broker sorts the results, aggregates them with results received from other agents (if applicable), and returns a single result set to the requesting user. Partition agents and data sources A single agent can access many different types of data sources. However, to support this flexibility, the agent must load shared libraries to support each type of data source that it can service. To reduce the number of libraries that get loaded, you might want to set up multiple agents and dedicate them to particular data sources. Before you begin this task, be sure to read about how partitioning agents on a single machine can enhance your system s scalability and security. This discussion begins on page 162. To distribute resources, you need to define multiple agents, determine the optimum number of server tasks that should be started for each agent, and associate each agent with the data sources that it is to service. Multiple agents can exist on servers that host enabled brokers and on servers that host only remote agents. The following example walks you through the process of partitioning agents on the same server. Note that this task is different than configuring multiple copies of tasks for a single agent. Consider a server named Paris that hosts a set of Notes databases and a number of DB2 databases. You decide to dedicate an agent to handle requests for each type of data source. To better utilize the system s resources, you also decide to grant the agent for the Notes sources more copies of server tasks. The following settings in the configuration database establish an agent named NotesAgent with 60 copies, and an agent named DB2Agent with 10 copies. 1. Start the Administration interface. Tasks 53

62 2. Click Servers in the Navigator. 3. In the view pane, right-click the Extended Search server named Paris and select Properties. 4. On the Server Properties window, select the Extended Search tab and then click Agents. 5. Click Add Row. A blank line is added beneath the currently selected agent or, if none is selected, to the end of the list. 6. In Agent name type NotesAgent and, in Number of copies, type Click Add Row again. This time, type DB2Agent as the agent s name and 10 for the copies of agent tasks that should be started. 8. Click OK. 54 Extended Search Administration 9. Click Data Sources by Link or Data Sources by agent in the Navigator. 10. In the view pane, select all the data sources that should be serviced by the NotesAgent agent, right-click, and select Properties. 11. On the Data Source Properties window, select NotesAgent from the list of agents and click OK. 12. In the view pane, select all the data sources that should be serviced by the DB2Agent agent, right-click, and select Properties. 13. On the Data Source Properties window, select DB2Agent from the list of agents and click OK. 14. To disseminate your changes to the user community, refresh the search domain. Configure multiple brokers One of the most extendable features of Extended Search is the ability to install and configure multiple brokers to achieve load balancing. In this scenario, data sources and processing resources are spread across multiple machines. A request coming in to one broker is distributed to the other brokers in the chain. Each broker then communicates the request to the responsible agents, and handles the aggregation and sorting of returned search results. Be sure to review Setting up multiple brokers on page 163 for broker scalability guidelines. Whenever you add a broker or rename an existing broker, update the properties for any applications that use that broker as an Entry broker. Select or clear the check box as appropriate for the brokers that you want to use with a given application. If you omit this step, applications will continue to use only the previously configured brokers as their entry brokers. Consequently, your processing load will not be distributed in the manner that you hoped to achieve. Use the following procedure to install and configure another broker. 1. Follow procedures in Extended Search Installation to install a new Extended Search server. (Select only the Extended Search Server option from the list of components to install.) 2. Start the Administration interface. 3. Follow the procedure in Add another search server on page 16 to add an icon for the new server and configure the new Server Properties. Be sure to assign a unique broker name for the new broker. 4. In the view pane, click the Graph View icon. 5. Right-click the new server and select Connect to. Then click an existing server where a broker that should be part of this broker chain is enabled. A line is drawn to indicate that the two servers are connected. 6. Click Apply Connections.

63 7. You can now use this new broker to be an entry broker with some of your applications. Take the following steps to implement this support: a. Click Applications in the Navigator. b. In the view pane, right-click an application that you want this broker to service and select Properties. c. In Entry broker, select the check box to enable this new broker and then click OK. 8. You can also take the following steps to configure the server where this new broker resides: a. Run data source discovery on the new server to add data sources to the Extended Search search domain. b. For each new data source, configure fields, including field-level access controls. c. Group the data sources into categories. d. Associate the categories with applications. 9. To disseminate your changes to the user community, refresh the search domain. Configure performance options To enhance the performance of your Extended Search system, you can configure multiple copies of server tasks and set up data sources on the same machine as the agent that services them. You can also specify options to manage memory and disk cache space on the Extended Search server and control how sessions are opened when an agent connects to a data source. Performance-related configuration tasks include the following: v Configuring copies of server tasks v Localizing data sources with their responsible agents v Managing memory usage on the server v Managing disk cache space on the server v Managing how Extended Search connects to data sources Before performing any of these tasks, be sure to review the discussion of Performance (beginning on page 168) to learn about the various ways that you can improve performance in an Extended Search environment. Configure copies of server tasks Each broker and agent task can have one or more copies started by the Extended Search server. Increasing the number of copies increases the number of requests for service that the system can process at the same time. As the Extended Search server initializes, it starts all copies of each type of server task, and then listens for requests. The installation process sets the number of copies for each task to a default value. For a higher powered machine, it is almost certain that you will need to adjust these initial numbers. Take care when increasing the number of server tasks because each copy is a separate component that consumes its own set of resources. Setting an optimum number of copies offers a way to optimize search performance in accordance with the frequency of access. Tasks 55

64 Use the following procedure to specify how many copies of server tasks the Extended Search server should start. Before making any changes, be sure to review the guidelines on how to determine the optimum number of server tasks (see page 172). 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane displays all the servers in the search domain. 3. Right-click the Extended Search server that you want to work with and select Properties. 4. On the Server Properties window, select the Extended Search tab. 5. Click to expand the entries under Broker. 6. In Number of copies, type the number of broker tasks that you want the server to start and click Apply. 7. Click to expand the entries under Agents. 8. In Number of copies, type the number of agent tasks that you want the server to start and click OK. 9. To begin using the new copies of server tasks, refresh the search domain. Localize Notes data sources and agents If you added remote Notes data sources through data source discovery, you might want to consider replicating the Notes database onto a local machine. When you use Extended Search to search Notes data sources, you can achieve optimum performance by setting up the agent on the same machine with its target database. For information about why you might want to replicate databases and co-locate them with agents, see Localizing agents with data sources on page 176. An agent is told how to access a Notes source by the LNServer data source parameter. For a remote data source, the discovery process automatically sets LNServer to the name of the remote host. The agent accesses the Notes database remotely through this specified server. After replicating a database to make it local to the agent s machine, you should set LNServer to blank. This causes the agent to assume local access and to read the database directly. Use the following procedure to localize an agent with its data source: 1. Follow the procedures in your Notes documentation to replicate the database onto the server where the agent is installed. 2. Start the Administration interface. 3. Click Data Sources By Link or Data Sources By Agent in the Navigator. The view pane lists all the data sources in the search domain. 4. Right-click the data source that you replicated and select Properties. 5. On the Data Source Properties window, select the Parameters tab. 6. Examine the entry for the LNServer parameter. 7. If the LNServer Value field contains text, delete the text. 8. In Usage, select Used by data source. 9. Click OK. 10. Repeat steps 4 through 9 for each Notes data source that you replicated or localized on the agent s machine. 11. To have these changes take effect, refresh the search domain. 56 Extended Search Administration

65 Manage server memory The Extended Search server manages a segment of shared memory for interprocess communication between the server tasks that run under its control. The size of this shared memory segment is obtained from the value set for the heap usage field in the Server Properties. The heap size value is defined in kilobytes, and it is set initially to 20 megabytes. If search results are returned when available, as opposed to being sorted, the broker reserves one third (initially 6.6 megabytes) of the allocated heap size for caching result set pages in memory. The remainder of the allocated space is used for internal messages between server tasks. To determine whether an appropriate heap size has been allocated, run the Monitor and observe the Heap Use value. If usage is generally at 80% or higher, you should increase the heap size. Likewise, if you observe error messages concerning memory allocation or the inability to create messages in the Extended Search event log viewer, it generally indicates an insufficient amount of heap size. Increasing the heap size can also help ensure that large documents retrieved from a result set will be displayed correctly. Take the following steps to increase the heap size (and the size of the memory cache) configured for an Extended Search server: 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search server that you want to reconfigure and select Properties. 4. Select the Extended Search tab. 5. In Heap size, type a new value for the amount of space that you want to allocate. For example, you might start by increasing the value from 20 MB (20000 kilobytes) to a value between 70 to 100 MB. 6. Click OK. 7. For this change to take effect, you must shut down and restart the Extended Search server. If the Monitor indicates that heap usage is still around 80%, or if memory warnings still appear in the Event Viewer, repeat this procedure to increase the heap size. If heap usage seems to remain high, and you occasionally receive message creation or server not responding errors, you might want to enable or increase the server s disk cache space. With a disk cache, the system can automatically cycle result sets to disk when it runs out of memory cache space. See Manage disk cache space for information about how you can extend the overall cache space available to the broker and improve response times for users as they page through large search results. Manage disk cache space As discussed in Broker caching on page 173, you can improve performance by setting optimum memory and disk cache sizes. To increase the size of the memory cache, you need to increase the heap size configured for the Extended Search server. See Manage server memory if you need assistance when setting an appropriate heap size. Tasks 57

66 To better handle very large result sets, and to improve the user s ability to page through result sets quickly, you should also configure disk cache space. The disk cache works in tandem with the memory cache to provide fast response times when rendering search results. Take the following steps to enable or increase disk cache space for an Extended Search server. Keep in mind that if you configured a pool of brokers to process requests, only the entry broker who receives the request and aggregates the search results will use the disk cache. 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane lists all the servers in the search domain. 3. Right-click the Extended Search search server that you want to reconfigure and select Properties. 4. Select the Extended Search tab. 5. Click Brokers to expand the Entry Broker Disk Cache options. 6. To allocate disk space for caching search results, clear the Disable disk cache check box. By default, disk caching is disabled. 7. In Disk cache directory path, type a fully qualified or relative system path where the cache should be created. This directory cannot be on a remote or networked machine and the path cannot exceed 254 characters in length. If the path does not exist, it will be created when the Extended Search server is started. If you specify a relative path (such as the default value,.\cache) the directory will be created relative to the Extended Search server s installation directory. 8. In Maximum size (in Kbytes), type the amount of cache space that you want to allocate, in kilobytes. The default value is kilobytes (10 megabytes). 9. Click OK. 10. To have these changes take effect, refresh the search domain. Manage data source connections In a production environment where many users perform searches with Extended Search, you could have many agents and a large number of sources that run simultaneously. Some of these sources might include ODBC and Notes database tables or LDAP directories. By default, an agent opens a connection to each ODBC table, Notes table, or LDAP directory when a request that targets the source is received. Thus, if you are running 50 instances of an agent, one table could prompt Extended Search to open 50 connections to the database an undesirable scenario for most organizations. To limit the number of open connections, you can configure the DatabaseRemainOpen parameter to service specific ODBC tables (such as tables for DB2 or Oracle), Notes databases, and LDAP directories. v If you set the DatabaseRemainOpen parameter value to No, the agent opens a connection at query initiation and closes the connection at query completion. This option can have performance implications because session handles get opened and closed on each query. v If you set the DatabaseRemainOpen parameter value to Yes, the agent maintains a persistent connection between queries. By using a combination of uniquely named agents and setting the DatabaseRemainOpen parameter to an appropriate value, you can better control system resources. 58 Extended Search Administration

67 You can activate this parameter at either the link level or data source level. If you set it at the link level, it applies to all data sources of the link type. If you set it at the data source level, it overrides the link-level setting and applies only to the data source for which it is configured. Use the following procedure to set DatabaseRemainOpen as a link parameter: 1. Start the Administration interface. 2. Click Links in the Navigator. The view pane lists all the links in the search domain. 3. Right-click the link that you want to configure. 4. On the Link Properties window, select the Parameters tab and then click Add Row. A blank line is added beneath the currently selected parameter or, if none is selected, to the end of the list. 5. In Link parameter name, type DatabaseRemainOpen. 6. In Value, type Yes or No, as appropriate for this type of data source. 7. In Description, briefly describe the purpose of this parameter. 8. Click OK. 9. To have these changes take effect, refresh the search domain. Use the following procedure to set DatabaseRemainOpen as a source parameter: 1. Start the Administration interface. 2. Click Data Sources By Link or Data Sources By Agent in the Navigator. The view pane lists all the links in the search domain. 3. Right-click the data source that you want to configure. 4. On the Data Source Properties window, select the Parameters tab and then click Add Row. A blank line is added beneath the currently selected parameter or, if none is selected, to the end of the list. 5. In Parameter name, type DatabaseRemainOpen. 6. In Value, type Yes or No, as appropriate for this data source. 7. In Usage, select Used by data source. 8. Click OK. 9. To have these changes take effect, refresh the search domain. Tasks 59

68 Running search applications The installation process automates many of the steps required to set up search applications. For example: v The installation program automatically configures the Domino Servlet Manager and WebSphere Application Server. v If you use Domino as your web application server, the installation program updates the servlets.properties file to include the names of the Extended Search servlets. When you start the application server, the servlets will be started. Typically, you do not need to make any changes to the servlets.properties file. v If you use WebSphere Application Server, the installation program adds the Extended Search servlet definitions to WebSphere. After you install Extended Search, use the Extended Search verification program to test your ability to search data sources. See Extended Search Installation for information about this program. You can then take the following steps to use your own search applications: v Optionally configure your web browser to launch Lotus Notes. v Run a search application. Configure the browser to launch Lotus Notes If you use Microsoft Internet Explorer to access an Extended Search application, you can configure it to recognize the Lotus Notes protocol. If the result set includes results from Notes data sources, the browser will be able to launch Lotus Notes, enabling you to access the referenced document. 60 Extended Search Administration To set up Internet Explorer to handle links to Notes documents, you must make the following changes in the Windows Registry. 1. Open the Registry Editor. Select Start Run, type regedit, and click OK. 2. Right-click [HKEY_CLASSES_ROOT] and select New Key. 3. Rename the new key to notes. 4. Right-click the (Default) entry for the notes key and select Modify. 5. At Value Data, type URL:Notes Protocol and click OK. 6. Right-click the notes key and select New String Value. 7. Rename the new value URL Protocol. 8. You can leave the URL Protocol value data blank, or right-click the value, select Modify, and specify a protocol-specific string. Contact your web server administrator if you are unsure of this information. 9. Right-click the notes key and select New Key. 10. Rename the new key to DefaultIcon. 11. Right-click the (Default) entry for the Default Icon key and select Modify. 12. At Value Data, type the file name to use as an icon for the Notes URL protocol and click OK. For example, if Notes exists on your C: drive, you could specify c:\notes\notes.exe. 13. Right-click the notes key and select New Key. 14. Rename the new key to shell. 15. Right-click the notes key and select New Key. 16. Rename the new key to open. 17. Right-click the notes key and select New Key. 18. Rename the new key to command.

69 19. Right-click the (Default) entry for the command key and select Modify. 20. At Value Data, type the command that should be used to call the Notes application and click OK. For example, if Notes exists on your C: drive, you could specify c:\notes\notes.exe % Confirm your changes. The Registry should have an entry similar to the following structure: [HKEY_CLASSES_ROOT] [notes] (Default) = "URL:Notes Protocol" URL Protocol = "" [command] (Default) = "c:\notes\notes.exe %1" [DefaultIcon] (Default) = "c:\notes\notes.exe" [open] (Default) = (value not set) [shell] (Default) = (value not set) 22. Select Registry Exit to exit the Registry Editor. Run a search application To start a search application, an HTTP request must explicitly invoke the JKMSearchController servlet and specify information about the application and the sources to be searched. After results have been returned, users can retrieve a result document simply by clicking a URL in the result set. To make it easy for users to submit a query, and to ensure that the query targets the appropriate data sources in the appropriate language, you will probably want to create a front-end web page that provides links to the various applications that are configured in your Extended Search search domain. By encoding URLs in this page, you can hide the complexity of the HTTP request from your users. Users can make simple selections on this page, and the servlet will be invoked with the appropriate parameters. The installation process installs several sample search templates that demonstrate Extended Search query and retrieval operations. You can change the look and feel of the applications, or create your own search applications, by customizing these templates or creating new templates. For information about how to do so, see the discussion about developing search applications in Extended Search Programming. Demo application Extended Search provides a sample application that has the ID Demo. This application is configured to search all of the sources in the search domain. On Windows machines, the default installation enables you to use the Demo application to search the predefined web sources and the Extended Search product documentation. On UNIX machines, the default installation enables you to search only the product documentation. (To search web sources, you must install Extended Search on a Windows machine.) Invoke the search servlet To invoke the search servlet, your custom search page needs to call a URL similar to the following example for each application and language that your enterprise supports. If you do not provide a front-end web page, users must type this URL into the location toolbar of a web browser. For readability, the following examples are split across two lines. You should type the entire string on a single line. The servlet name and its arguments are case-sensitive. Tasks 61

70 &AppID=appname&desClientLocale=locale Where: v hostname is the fully qualified host name of the web server. v servlet is the base directory on the web server where Extended Search servlet files are installed. On a Domino server, this value is servlet. On a WebSphere server, this value is the Extended Search web application name (typically lotuskms). v template must be destemplatefile if you are invoking a search page that embeds Extended Search JKM tags, or desjspfile if you are invoking a search page that embeds Extended Search JSP beans or tags. v filename is the name of the file that you want to use for searching the Extended Search domain, such as the AllOptions.txt, AllOptions.jsp, or tagalloptions.jsp templates that are provided with Extended Search. v appname is the name (application ID) of the search application that you want to run, such as the Demo application. v locale is the language and country code that identifies the locale of the data sources that this application searches, such as enus for U.S. English. For information about specifying a valid language and country code, review National language support on page 124. Examples: &AppID=Demo&desClientLocale=enUS &AppID=Finance&desClientLocale=frFR Search guidelines If you are testing your ability to submit a search request after the initial installation of the product, query the preconfigured Web and ES Documentation categories (if you installed Extended Search on a Windows machine), or the preconfigured ES Documentation category (if you installed Extended Search on a UNIX machine). To test another type of data source, you must first run the Administration interface to discover the data source, assign it to a category, and associate the category with at least one application. After search results have been returned, click on various entries to ensure that you are able to view or retrieve documents that are listed in the result set. By default, the Extended Search search templates allow up to 100 result items to be returned. Certain web search engines impose their own limitations on the number of results that can be returned, which might be less than the default limitation. To switch applications, change the URL in the browser to identify the application ID of the application that you want to run, and press Enter. If you use a Netscape browser with the JKM search templates, performance can be slow. For example, if you click the Sources tab in the Demo (JKM) search template, it might take awhile for the available sources to be displayed. Selecting the Schedule or Options tabs can also be slow. To resolve this problem, either use the JSP search templates or use an Internet Explorer browser. 62 Extended Search Administration

71 Saving and scheduling searches The Saved Queries applet enables users to save and schedule queries and search results. Settings in the Administration interface must be enabled to make these features available. The ability to save and schedule queries means that users can process search requests at a convenient time. For example, you can schedule a search that requires connections to many data sources, and that returns numerous results, to run after hours when other users and resources are not contending for processor time. Saved queries enable multiple users to collaborate and are useful for specifying syntactically complex queries that are difficult to duplicate. For example, if one user submits a complex search that is of interest to all the members of a work group, all users can view the results and retrieve relevant documents. If you enable the option to search results, Extended Search will send result sets from scheduled queries to recipients who are named in the search request. A key feature is the ability to write search result documents to a file system data source. Extended Search can operate as an intelligent crawler and automatically retrieve content in accordance with query criteria. Once a document has been stored, you can use other products to render, index, or analyze the content. For example, you might embed returned URLs in a spreadsheet, or index the search results and feed them into a workflow or knowledge management application. Retrieve and store result documents on page 70 describes this feature. The Extended Search security model enables you to control the accessibility of saved queries and search results. v If the web server does not challenge users to specify a user ID and password when they set up a search request, Extended Search saves the queries and result sets in the public domain. All users can view the query and search results. v If the web server is configured to use basic authentication or single sign-on security, users will be prompted to specify a user ID and password when they access the search application. If a user saves the query, Extended Search saves the query and result set under the specified user ID, and only that user and Extended Search administrators can view the saved data. To allow multiple users to share queries and result sets, create a group ID and assign individual users to it. Users can then specify this group ID when they access the search application and the Saved Queries applet. Using Impersonation Extended Search stores only the user ID of the user who set up the search, not the user s password. Except for Notes databases, you cannot re-run saved queries that target sources for which you have enabled impersonation. When impersonation is enabled, the user must provide both a user ID and password to be able to access and search the data source. The saved query provides only the user ID. Because a Notes server requires only the user ID when it verifies a user with a Notes ACL, you can use impersonation with saved queries that target Notes data sources. Tasks 63

72 To learn how to use the saved query features of Extended Search, review the following topics: v How to save and schedule queries v How to work with saved queries and search results v How to retrieve search results and store them into a file system Save and schedule queries How you save and schedule a query depends on whether you are an administrator or end user. v The Demo search template includes options that enable an administrator to schedule and save queries while testing the configuration of the Extended Search search domain. v The search application that an end user accesses must include options for scheduling and saving the query. For complete information about creating search templates for your custom search applications, see Extended Search Programming. Use the following procedures to set up Extended Search to save and schedule queries, set up and schedule a search request, and save the query and results. Set up the query scheduler: 1. Start the Administration interface, select Servers in the Navigator, right-click the web server that you want to configure, and select Properties. 2. Select the Web Server tab, and then select Scheduled Query to expand the options. 3. Select the Enable query scheduling check box to allow users to save queries. When you enable this option, an Extended Search servlet will start checking for scheduled queries and run them at the scheduled times. You can clear this check box at any time if you need to prevent scheduled queries from being run. For example, you might need to disable the scheduler to troubleshoot problems, such as a rogue query that you suspect is consuming too many resources. 4. In Check for scheduled queries, specify how often, in minutes, the server should check for queries that are due to be run. 5. In Concurrent retrieve and store processes, type the number of retrieve and store processes that can run concurrently. At a minimum, this number should equal the number of brokers that are configured for the Extended Search server that is being used to process the scheduled queries. You should also ensure that the number of available agents is at least one greater than the number of brokers. See Retrieve and store result documents on page 70 for configuration guidelines. 6. In Wait for a process to time out, specify how long, in seconds, the broker should wait for a response from an agent before cancelling the request to retrieve and store a document. The default value is 60 seconds. 7. To allow users to the search results from a scheduled query, specify the following options: a. Select the Enable ing of results from scheduled queries check box. If this check box is clear, users will not be able to search results. 64 Extended Search Administration

73 b. In server host name, type the fully qualified host name or IP address of a valid Simple Mail Transfer Protocol (SMTP) server. c. In server port, type the port number for the SMTP server. The default value is 25, the Internet standard for SMTP ports. 8. Click OK. 9. Refresh the search domain to have your changes take effect. Access the search template: v If you are an administrator, start the Administration interface, select Search Templates in the Navigator, and open a Demo search template. v If you are an end user, follow instructions from your local administrator to access the URL for a search template that includes options to save and schedule searches in Extended Search. Set up the search request: Described below are options that are typically available for scheduling and saving queries. These options are available in the Demo search template. The search template that you access might or might not include these options. 1. Assign a name to the query. Type a descriptive name, one that will enable you and other users to recognize its objective. 2. Type a brief description of the query. 3. Assign the query to a query folder. If you previously created query folders, you can select one from the list of existing folders and add this query to that group. Alternatively, you can create a new folder for this query by typing a name for it. Note that if you type the name of an existing folder, the query will be saved in that folder. 4. Your search template might allow you to save the query without running it or scheduling it. If you want to save the query for later refinement or processing, click the option to save this query now. 5. Schedule the query: a. Select the option that indicates that you want to schedule the query as well as save it. For example, a Schedule this query check box. b. Select an interval for how often the query should be run (1 time per day, 1 time per week, 2 times per month, and so on). c. Select the day of the week that the query should be run the first time (this sets the start date for when the scheduled frequency should be repeated). d. Type the time of day that the scheduled query should be run. Specify the time by using the 24 hour clock (09:30, 12:45, 23:00, and so on). e. Optionally write the search result documents to a file system data source. Extended Search lists the names of the file system data sources that are available to the search application. Select the one that you want to use for the result documents. See Retrieve and store result documents on page 70 for more information about this feature. f. Optionally process the search results. You can save the results for viewing at a later time. You can also choose whether you want to retain all the result sets for this query, or overwrite the previous result set each time this query is run. Tasks 65

74 g. To the search results, specify the address of the person who should receive the results and the person who should be shown as the sender of the note. To send the search results to more than one person, use a comma to separate the addresses (for example, abc@yahoo.com,xyz@hotmail.com). The address that you use for sending the (the From address) will receive notifications about any delivery failures. h. Click the option to save this query. The next time the Extended Search servlet checks for queries that are due to be run, this newly saved query will be added to the queue of scheduled requests. The query will be processed the next time the servlet runs after its scheduled start time occurs. See Work with saved queries for information about how to work with saved queries and search results. Work with saved queries Extended Search presents saved queries and search results in a familiar tree structure, much the way Windows Explorer organizes files into folders and subfolders. v At the top level is the query folder, which you can create in the Administration interface. Depending on options in the search form, you can also create a folder as part of creating a search request. v If you expand a query folder, you see a list of the queries that belong to it (each query must be associated with a folder before it can be saved). Icons help you identify the saved state of a query: Indicates that this saved query is also scheduled. Indicates that this saved query was previously scheduled, but is now disabled (currently, it is not scheduled to be run again). Indicates that this saved query was never scheduled. To schedule it, you must modify the query, specify schedule information, and save the modified query. 66 Extended Search Administration v If you expand a saved query after the query has been run, and you chose to save the results for later viewing, you will see one or more result sets. Multiple result sets appear if the query specified that previous results were to be retained. The names of the result sets match the dates and times that the queries were run. How you access saved queries depends on whether you are an administrator or end user. The actions you can take also vary according to your role and whether the saved query is public, owned by you, or owned by another user. v Administrators can view and work with all saved queries and result sets. v End users can view and work with only those saved queries and result sets that were created under the user ID that they specified when they accessed the applet, or that were created as public queries (that is, queries that do not require a user ID for authentication). Access saved queries If you are an administrator, there are two ways for you to access saved queries. First, start the Administration interface. Then use one of the following methods:

75 v Click Saved Queries in the Navigator. The view pane lists all the queries that were saved by you or by end users, and organizes them by query folder name. v Click Search Templates Demo in the Navigator, right-click one of the Demo templates, and select Display. Click the Schedule tab in the Demo template, and click the option to View Saved Searches. The Saved Queries view lists all the queries that were saved by you and end users, and organizes them by query folder name. If you are an end user, there are also two ways for you to access saved queries. Note that the Java 2 Runtime Environment (J2RE) plug-in must be installed on any machine on which you want to work with saved queries (see Extended Search Installation for system requirements). After the Saved Queries applet starts, you see a list of all public queries and the queries to which you have access, organized by query folder name. v Depending on options in the search template, you might be able to access saved queries by clicking a link that automatically invokes the Saved Queries applet. The Schedule tab in the Demo search template demonstrates this action. v Start a web browser and pass savedqueries.txt to the JKMSearchController servlet. For readability, the following example is split across two lines. Type the entire string on a single line. The servlet name and its arguments are case-sensitive. &AppID=appname&desClientLocale=locale Where: hostname is the fully qualified host name of the web server. servlet is the base directory on the web server where Extended Search servlet files are installed. On a Domino server, this value is servlet. On a WebSphere server, this value is the Extended Search web application name (the default value is lotuskms). appname is the name (application ID) of the search application that you want to run, such as the Demo application. locale is the language and country code that identifies the locale of the data sources that this application searches, such as enus for U.S. English. For information about specifying an ISO-compliant language and country code, review National language support on page 124. For example: &AppID=Demo&desClientLocale=enUS When the list of available saved queries is displayed, you can take any number of actions to work with them. Saved query actions After you access saved queries, take any of the following actions to view or work with the saved queries and result sets: Create or modify a query folder: 1. To create a query folder, either click Saved Queries in the Navigator and select New folder, or right-click in an open area of the Saved Queries view pane and select New Folder. 2. On the Query Folder Properties window, type a name for this folder, type a brief description of its purpose, and click OK. Tasks 67

76 This window also displays the user ID of the user who created the query (the user ID is blank if the query was created by an administrator). This field is provided for informational purposes only; you cannot modify it. 3. To change the name or description of an existing query folder, right-click the folder that you want to change and select Properties. Make your changes, and click OK. Delete a query folder, query, or result set: 1. Select one or more query folders, queries, or result sets that you want to delete, right-click, and select Delete. When you delete a query folder, all queries and result sets within it are also deleted. When you delete a saved query, all result sets that were stored for it are also deleted. To avoid cluttering the database, periodically review saved queries and their results, and delete those that no longer need to be retained. Temporarily prevent a query from being run: 1. Select one or more queries that you want to suspend, right-click, and select Properties. 2. On the Saved Query Properties window, select the Disable query check box and click OK. The next time that the scheduler servlet checks for scheduled queries, this request will be removed from the set of scheduled queries. When you are ready to process the search again (for example, after you debug or modify the query), clear the Disable query check box and click OK. Modify information about a query: 1. To change the name or description of a query, right-click the query that you want to change and select Properties. 2. On the Saved Query Properties window, type a new name and description and click OK. This window also displays information about when the query was created. If the query is scheduled, it also shows when it was last run and when it is scheduled to run again. These fields are provided for informational purposes only; you cannot change them. Modify the content of a query: 1. To change the search terms or search options, or to change options for saving and scheduling a query, right-click the query and select Modify Query. 2. When the search form for the query is displayed, make your changes and select the option to save the query. If the query is scheduled, the next time that the servlet runs the query, the changed query will be processed. Note: If you attempt to change the name of the query folder or the name of the query from within the search form, the following actions take place: v If you are an administrator working with saved queries in the Administration interface, the requested changes will be saved in the configuration database. v If you are an end user working with the Saved Queries applet, either a new folder or query will be created, or an existing folder or query of that same name will be overwritten. To rename a query folder or 68 Extended Search Administration

77 query, update the Query Folder Properties or Saved Query Properties instead of changing this information in the search form. Retrieve search results: 1. To display search results, right-click the result set that you want to view and select Display Query Results. 2. After the result set is displayed, click the document link for a document that you want to retrieve. Depending on the design of the result page, you might be able to modify the query at this point and submit the search request again. Depending on features that are included in your search page, you might be able to take the following actions when you work with search results. v Retrieve documents by selecting them from a scrolling ticker tape display. The esticker applet that is provided with Extended Search can display information, such as result documents, while you are working with a query. v Toggle between the detailed and summary views of the result set. The detailed view shows all the returnable fields that were defined for the data source in the configuration database. The summary view shows the result item name, the name of the data source that returned the result, and the name of the category that the data source belongs to. v Select an action for processing result set items. When you select a smart action, the system invokes the URL that was defined for that action in the configuration database. The sample smart actions provided in the Demo template enable you to link to the Extended Search home web page (ES Home), the page for downloading software upgrades (Downloads), and the page for getting troubleshooting information (FAQ). As discussed in Define smart result set actions on page 26, it us up to your local web designer and administrator to define web pages that support the intended actions. Usage Guidelines v A maximum of 254 characters can be stored in the configuration database per search option. This limitation is typically not a concern unless your application developer chose to use the DESSources variable in a JKM-based search template and you selected numerous individual data sources to search. If you receive an error message indicating that you exceeded the 254 character limitation, either select fewer data sources to search or select categories instead of data source names. v Your web browser might limit your ability to re-display a saved query. Even though Extended Search is able to store your query in the configuration database, certain browsers impose limitations on the amount of data they can retrieve (such as 2048K characters). If the total amount of data in the saved query exceeds the browser limitations, the search page will appear blank. Customize results for For users to be able to search results, options must be defined in the configuration database as part of setting up scheduled queries, as described on page 64. Tasks 69

78 If a user elects to results returned from a scheduled query, Extended Search creates the result set in XML format. It then uses an XSL style sheet, ESMailResults.xsl, to transform the XML text to readable HTML. It sends this HTML output to each of the recipients specified in the search request. If the user specified a query description as part of saving and scheduling the query, the description serves as the subject line of the . If there is no query description, the subject line specifies the query name. You can customize the look and feel of the ed search results by updating the ESMailResults.xsl file. This file is stored in the JSP subdirectory of the Extended Search installation directory. To render the search results in your preferred format, you can modify this file any way that you like. However, you cannot rename the file or store it in a different location. Before making any changes to this XSL file, be sure to make a backup copy of it. Retrieve and store result documents In addition to saving queries and result sets in the Extended Search configuration database, you can save result documents to a file system data source. Because the saved documents consume disk space, you might want to restrict this option to users who have administrative responsibilities. With this feature, Extended Search is able to operate as an intelligent, query-driven crawler. It can automatically retrieve content of different formats on a scheduled basis, and store the retrieved data in a file system data source. Once stored, a document can be rendered in its native form or be analyzed and indexed by another product, such as Lotus Discovery Server. This feature provides the following benefits: v The ability to directly search the result documents whenever you submit a query against the file system data source. v The ability to direct result data to an applet. The Extended Search esticker applet demonstrates how to stream the contents of result documents across a screen in a ticker tape fashion. v The ability to add file extensions to the stored data, thus enabling a document to be rendered by its native viewer or by a web browser. For example, adding a.doc extension enables Microsoft Word to open the document. Adding the extension.html enables a browser to display the document regardless of whether the native application is available. v The ability to import the stored data into another application. This feature provides flexibility with regard to how you intend to use the data: You can crawl and index the data, thus making it meaningful for data mining and knowledge management purposes. Users can rapidly access relevant information by using keywords that were associated during the indexing process. You can make the data available to users without first indexing it. When storing a document, Extended Search can create an XML wrapper that identifies the original document and the location of the document as stored in the file system. The document identifier, which typically takes the form of a URL, enables users to access the original source of the document, not just the saved version in the file system data source. Review the following topics before you store search results to a file system: 70 Extended Search Administration

79 v How documents are stored describes how Extended Search stores the data it retrieves. To index a result document for use with other applications, read about how to access the stored data. v Retrieve and store guidelines on page 72 discusses performance considerations that you need to be aware of when you configure retrieve and store features in the configuration database. How documents are stored When Extended Search retrieves documents as the result of a scheduled search, you control how you want to store the data: v You can store the native content of the document into the file system as is. v You can have the native content enveloped in XML. v You can store both the native content and its XML wrapper. Your choice depends on the presence and content of an XML template named desschedule.xml, which is located in the Extended Search installation directory. v If desschedule.xml does not exist in the installation directory, then the native content of the document will be stored into the file system. This is usually an HTML document, but it could be a document of any type, such as a document produced from typical office software products. When stored, the native documents are named consecutively with a unique alphanumeric suffix that increments with each document and field that is returned. The file name consists of the characters hit followed by a unique number for each document, a unique alphabetic character for each field, and the file extension.des. For example: hit0a.des, hit0b.des, hit0c.des (and so on) hit1a.des, hit1b.des (and so on) hit2a.des, hit2b.des, hit2c.des, hit2d.des (and so on) v If desschedule.xml exists in the installation directory, Extended Search will use it to format and store retrieved documents. For each retrieved document, Extended Search creates an XML file whose content is determined by the desschedule.xml template. When stored, the XML files are named consecutively with a unique alphanumeric suffix that increments with each document and field that is returned. The file name consists of the characters hit followed by a unique number for each document, a unique alphabetic character for each field, and the file extension.xml. If you store both the native content and a companion XML file, the filenames correlate. For example: hit0a.xml, hit0b.xml, hit0c.xml (and so on) hit1a.xml, hit1b.xml (and so on) hit2a.xml, hit2b.xml, hit2c.xml, hit2d.xml (and so on) Tasks 71

80 The Extended Search product ships with the following desschedule.xml template file: <?xml version="1.0"?> <!DOCTYPE rdf:rdf SYSTEM " <rdf:rdf xmlns:rdf=" xmlns:dc =" <rdf:description about="$es:storepath:es$"> <dc:identifier> $ES:DOCID:ES$ </dc:identifier> <dc:body> $ES:DOCCONTENT:ES$ </dc:body> </rdf:description> </rdf:rdf> You can modify or replace this file with your own XML as long as your file includes at least one of the following Extended Search replacement variables. $ES:STOREPATH:ES$ This variable, if present, tells Extended Search to store the native content of the document into the file system (with a.des extension) and to replace this variable with the file system path of where it was stored. You might choose this option if the content of the document is binary in nature (such as a native spreadsheet file) and thus cannot be wrapped in XML. $ES:DOCID:ES$ This variable, if present, will be replaced with the identifier of the document. Normally, the identifier is a URL, but it can be anything depending on the source repository of the document. $ES:DOCCONTENT:ES$ This variable, if present, will be replaced with the actual content of the document. Note that the content of the document should be all text and not contain any nondisplayable characters. You can combine these variables as needed. For example, if the documents that you are retrieving contain a mixture of text and binary content, you might use the $ES:STOREPATH:ES$ and $ES:DOCID:ES$ variables only. This would result in.des and.xml files being produced (the default in Extended Search). Alternatively, you could choose to omit the $ES:STOREPATH:ES$ variable and generate a single XML file that contains both the document identifier and content. Note that the maintenance of this data is beyond the control of Extended Search. You must ensure that there is sufficient free space available in the file system to store the saved data. You should also periodically review the data and delete information that is no longer needed. Retrieve and store guidelines The configuration of retrieve and store features in the configuration database is critical to your system s performance and to your ability to process retrieve and store requests concurrently. v The default installation of Extended Search defines three brokers and five agents. This configuration is acceptable for small production and test environments. For smooth performance in a typical or large production environment, additional brokers and agents will be needed. 72 Extended Search Administration

81 v When you configure the number of retrieve and store processes that can run concurrently, specify a number that is at least as great as the number of available brokers. This ensures that retrieve and store requests can be processed in parallel. v If the agents and brokers that are doing the retrieving and storing are all located on the same machine, the number of agents should be double the number of brokers. To avoid deadlock, you must ensure that the number of agents is always at least one greater than the number of brokers. v If the agents that retrieve data are located on machines that are separate from the machines that are doing the storing, the number of agents should equal the number of brokers. This configuration permits the maximum number of concurrent stores. For optimum performance, you should install multiple Extended Search servers and configure multiple brokers and multiple agents on each machine. Ideally, you have a separate machine with extensive disk space available. The file systems on this machine should be configured as data sources within your Extended Search search domain, which enables them to be available for storing result data from scheduled queries. To achieve the greatest throughput, schedule queries to run after hours or during non-peak hours of the work day. Doing so can avoid resource contention with users who submit queries for immediate processing. This recommendation is especially true if your scheduled query targets numerous data sources or is expected to return very high numbers of search results. If you use a Netscape browser, version 4.73 or older, to view or modify saved queries that were created with JKM-based search templates, the query data will not be restored properly by the browser and will appear to be lost. To resolve this problem, either upgrade to version 6.0 of Netscape or use a Microsoft Internet Explorer browser. This performance problem is not an issue with JSP-based search templates. Tasks 73

82 Monitoring server activity The Extended Search Monitor provides a graphical interface to Extended Search servers. It enables you to observe server activity and, when necessary, make adjustments to improve processing. Because it runs independently of the Extended Search server, you can start it and stop it without affecting server processing. You can run the Monitor in two modes: v As a stand-alone C++ program. The stand-alone version of the Monitor runs on Windows systems only. However, you can start the Monitor on a Windows machine and then use it to monitor an Extended Search server that you installed on a UNIX machine. You should run the Monitor on a machine other than the machine where the Extended Search server is running. This approach enables you to check the status of the server without having to be at the host console. Furthermore, because the Monitor uses a graphical interface, it consumes machine resources that might otherwise be dedicated to Extended Search server activity. v While running the Administration interface. This feature enables you monitor activity, make real-time changes to the system, and refresh the Extended Search domain with those changes from a single point of control. You can refresh Extended Search servers while running the Monitor. Refreshing a server s configuration causes each server task to reload its configuration data from the configuration database. This feature enables an Extended Search server to remain online. You do not have to shut down and restart the server each time you make a change to its operations. The Monitor is a primary tool for helping you assess system performance and make adjustments. Although the following topics show you how to use the Monitor s graphical interface, you should also review the discussion of Performance on page 168. It can help you understand how to interpret reported statistics and make changes that are needed to optimize server performance. Run the Monitor in stand-alone mode Perform the following steps to start the Monitor on a Windows machine: 1. Select Start Programs IBM Extended Search Extended Search Monitor. 2. Type or select the host name of the Extended Search server that you want to monitor, type its port number (typically 6001), and click OK. v If the host name is correct, and the Monitor is able to connect to the server, a detailed view of the server s run-time statistics is displayed. v If the Monitor is unable to connect, you see a message that asks whether you want to cancel or retry the request. If you click No, an empty Monitor is displayed. Select Monitor Open, and re-select or retype the server s host name. For example, you might have typed a wrong value the first time, or the server might not be online or available. Selecting Monitor Open causes statistics to be displayed in the current Monitor window. If you want to display statistics for this server or for a different server in a new Monitor window, select Monitor New. After the Monitor has been started, select View Always on top. This feature ensures that the Monitor window is always visible, which enables you to instantly 74 Extended Search Administration

83 monitor the results of all server activity that is currently occurring. To move the window to the background, toggle this selection. Shut Down the Monitor To shut down the Monitor, either close the Monitor window or select Monitor Exit. If you select Server Shutdown, you shut down the Extended Search server, not just the Monitor. To shut down the Extended Search server, you must specify the Extended Search Administrator user ID and password. View server task details The detailed view of server tasks provides detailed information about each copy of each server task that was started on the server. This mode is the only view available within the Administration interface. For an explanation of server tasks, and for guidelines on configuring copies of tasks, see Server tasks on page 171 and Determining the optimum number of tasks on page 172. To access the detailed view at any time, either select Server tasks (Detail) in the Monitor area on the left side of the interface, or select View Server task detail. Figure 3. Extended Search Monitor detail view Server area The Server area displays the host name and port number of the server that you are monitoring. It also shows the date and time when the server was last started. Message counter A counter in the lower left corner of the window displays the total number of messages that have been received by the server since it was last started. Memory heap usage A scale along the bottom of the window reflects the percentage of memory resources that have been used, based on the configured heap size for this server. Each Extended Search server allocates a segment of shared memory Tasks 75

84 for interprocess communication between the server tasks that run under its control. To manage server memory, you might need to adjust the server s heap size (see page 57). Server tasks area The Server tasks area provides information about each copy of a server task. Each line represents a unique copy. Scroll through the list to examine information about each task: v The icons in the Status column change color and text as the processing state of a given task changes: The gray Idle icon indicates that the task is active but no activity is occurring. The green Busy icon indicates that the task is processing a message. The yellow Wait icon indicates that the task is waiting for another task to process a request before it can continue. The blue Conf icon indicates that the task is getting information from the Configurator. You see this icon only when the server is started or refreshed. The red Down icon indicates that the task is not active. This generally indicates an abnormal condition. v The Title column displays the name of the task. v The PID column displays the process identifier. A PID is a code that uniquely identifies each copy of each task. The PID also enables you to distinguish among the server tasks when you use a system analysis tool such as the Windows NT Task Manager. v The Count column specifies the number of messages that have been completely processed by a given task. v The Time column displays the average length of time that it takes to process a message through a given server task. View a server task summary The summary view of server tasks is available only if you are running the Monitor as a stand-alone program. It is not accessible from within the Administration interface. To access the Summary view at any time, either select Server tasks (Summary) in the Monitor area on the left side of the interface, or select View Server task summary to switch to this view. 76 Extended Search Administration

85 Figure 4. Extended Search Monitor summary view The Server, message counter, and memory heap usage areas of the Summary view mirror the same areas in the detailed view. The Server tasks area, however, differs in that it lists each type of server task. It does not list the individual process IDs for each copy of a server task. However, a status icon is displayed for each copy of a server task. The color and content of the status icons are identical to those that are displayed in the detailed view. Display data source statistics The data source statistics view provides information about the target data sources to which this server is connected. You can scroll through the list and review the number of requests that have been directed to each data source since the server was last started. You can also assess the average elapsed time that it took for the agent associated with a given data source to process the requests. Note that this average elapsed time reflects the time that elapsed during an agent s processing of the request, not the overall response time that users experience. To access this view at any time, either select Data source statistics in the Monitor area on the left side of the interface, or select View Data source statistics to switch to this view. Tasks 77

86 Figure 5. Extended Search Monitor data source statistics view Log server activity You can keep track of a server s activity by writing statistics to a log file. This feature enables you to run the Monitor in an unattended mode and review a log of the server s activity at a later time. To log server activity, you must specify how often the log file needs to be refreshed, activate logging, and specify a location for the log file. Logging is available only if you are running the Monitor as a stand-alone program, not if you are monitoring server activity from within the Administration interface. Use the following procedure to record server activity in a log file: 1. Select Monitor Refresh every, and then select a value to specify how often messages should be written to the log file. a. Select Set Interval to specify a refresh rate in milliseconds: 1) On the Set monitor refresh interval window, specify how often messages should be written to the log file. The default value is 200 milliseconds. You must specify a value that is greater than one second. 2) Click Set. v Do not select.05 or 1.0. Logging will occur only if the refresh rate is greater than one second. v Do not select Continuously to constantly refresh the log file. This selection can cause heavy resource consumption. 2. Select Monitor Log activity. The Log server activity to a file window is displayed. 78 Extended Search Administration

87 Figure 6. Extended Search Monitor log server activity 3. To enable logging, select the Activate logging to file check box. 4. Type the fully qualified path and file name for a file where log messages should be written. 5. If you want to append messages to an existing file, select Add to file. If you want to replace an existing log file and start a new one, select Replace file. 6. If you want to log messages about server task activity, select the Include message statistics check box. 7. If you want to log messages about data source search activity, select the Include database statistics check box. 8. Click Set. Run the Monitor in the Administration interface The following procedure shows you how to use and navigate the Monitor while you are using the Administration interface. Before using this method, review the following topics to learn how the Monitor works in Extended Search: v Run the Monitor in stand-alone mode on page 74. If you encounter problems with resource contention while running the Administration interface and the Monitor, run the Monitor in stand-alone mode. v Performance on page 168. This discussion might help you interpret reported statistics and make appropriate changes to optimize server performance. It also explains server tasks, and provides guidelines for determining how many copies of tasks are best for your environment. Tasks 79

88 Use the following procedure to monitor server activity from within the Administration interface: 1. Start the Administration interface. 2. Click Servers in the Navigator. The view pane displays all the servers in the search domain. 3. Right-click the Extended Search server that you want to monitor, and select Monitor. (On UNIX, select the server, press Shift+F10, and then select Monitor.) The Monitor starts and provides detailed information about each copy of each server task that was started on the server. Figure 7. Extended Search Monitor Administration interface view 4. Use the following guidelines to interpret the run-time statistics that are displayed by the Monitor: Server area Displays the host name and port number of the server that you are monitoring. It also shows the date and time when the server was last started, and the total number of messages that were received by the server since it was last started. The Heap Use field reflects the percentage of memory resources that have been used, as determined by the configured heap size for this server. Search Statistics area Provides information about the target data sources to which this server is connected. You can scroll through the list of data sources and review 80 Extended Search Administration