Full Text Search Agent Throughput

Transcription

1 Full Text Search Agent Throughput Best Practices Guide Perceptive Content Version: 7.0.x Written by: Product Knowledge, R&D Date: December 2014

2 2014 Perceptive Software. All rights reserved Perceptive Software is a trademark of Lexmark International Technology S.A., registered in the U.S. and other countries. All other brands and product names mentioned in this document are trademarks or registered trademarks of their respective owners. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or any other media embodiments now known or hereafter to become known without the prior written permission of Lexmark.

3 Table of Contents Introduction to Full Text Search Agent Throughput... 4 About Full Text Search Agent... 4 Full Text Search Agent architecture... 5 Full Text Search Agent performance model... 6 Interaction with MQ Agent... 6 Interaction with OSM Agent... 6 Full Text Agent internals... 7 Scaling Full Text Search Agent throughput and performance... 8 General tuning... 8 Worker thread tuning... 8 External performance influences and effects... 9 Recognition Agent... 9 System impact of tuning Full Text Agent... 9 Content Index Throughput Testing Results... 9 Constants... 9 Index performance Full Text Search Performance Testing Results Constants Search performance of text documents Search performance of Microsoft Word documents Search performance of PDF documents Search performance varying Full Text Agent worker threads Full Text Performance with Large-Scale Content Collections System specs Scalability recommendations Appendix A: INI Configuration File Settings inserverft.ini settings inow.ini settings inservermq.ini settings Page 3 of 19

4 Introduction to Full Text Search Agent Throughput For customers who process high numbers of pages, Perceptive Software recommends tuning systems for high throughput and efficiency to maximize performance. This document provides information to help you determine your throughput performance needs for Full Text Search Agent based on the volume of documents handled by your organization using Perceptive Content. About Full Text Search Agent Full Text Search Agent collects text data and creates searchable indexes based on the content of each document page. This allows you to identify documents containing specific keywords or phrases via content searches executed inside Perceptive Content. Content searches allow you to perform various types of full text searches, including fuzzy, phonetic, stemming and synonym, proximity, and relevance ranking searches. Your search can also contain wildcards to represent one or more characters. The server displays the document pages that contain matches for the search and highlights the matching keywords within the documents. The main service and executable of Full Text Search Agent is the Full Text Agent. The Full Text Agent service (or inserverft binary on UNIX platforms) is used throughout this guide synonymously with the Full Text Search Agent product. Page 4 of 19

5 Full Text Search Agent architecture The following diagram illustrates the relationship between the Full Text Agent and other Perceptive Content components, as explained after the diagram. MQ Agent. All documents submitted to Full Text Search Agent start at Message Queueing (MQ) Agent. This agent manages internal queues that are responsible for allowing agents to determine what action to take. Full Text Agent periodically sends requests to MQ Agent asking for the next document indexing job. OSM Agent. This agent is responsible for providing the Full Text Agent the physical files to index. When Full Text Agent receives a request to index a document it will send a request to the OSM Agent to retrieve the document for indexing. Content Collections. This database resides on the same server node as Full Text Agent and is required to store document indexing information. Perceptive Content Database. Full Text Agent communicates with the system database to update the document content status. Perceptive Content Database also stores information necessary to reference Content index information with the documents stored in your system. Recognition Agent. This agent handles text recognition for raster-based (graphical) documents. Recognition Agent can strongly affect throughput performance. For more information about optimizing Recognition Agent Throughput, refer to the Recognition Agent Throughput Best Practices Guide. Page 5 of 19

6 Full Text Search Agent performance model Overall Full Text Search Agent throughput performance is rooted in balancing the interactions among Full Text Agent, MQ Agent, Recognition Agent, and OSM Agent. This balance is achieved through understanding how each agent s individual throughput affects the throughput performance of Full Text Search Agent. Interaction with MQ Agent MQ Agent s first function is to identify document indexing jobs to delegate to Full Text Agent. It does this by searching at regular intervals for document indexing jobs created by other agents, the Perceptive Content Client, or other business solutions. When MQ Agent has identified a specified number of jobs, it caches them into a queue for delivery to Full Text Agent. Full Text Agent can request outstanding document indexing jobs by contacting MQ Agent. MQ Agent keeps a pool of worker threads to respond to incoming requests from all system agents. This means that the number of worker threads must be balanced against the overall volume of requests coming from Full Text and other system agents. If the number of incoming requests to MQ Agent outpace the capacity of the worker threads, then incoming requests will be blocked or serialized and negatively impact performance. Interaction with OSM Agent Since Full Text Agent has no direct access to the Perceptive Content OSM it relies on the OSM Agent to deliver the physical files needed for indexing. Much like MQ Agent, OSM Agent maintains a pool of worker threads to handle these incoming requests for OSM objects. Also like MQ Agent, other system agents may contact the OSM Agent and the number of incoming requests must be balanced against the number of worker threads. OSM Agent s throughput is thus entirely based on the availability of its worker threads and the speed of the OSM storage. Page 6 of 19

7 Full Text Agent internals When Full Text Agent receives indexing jobs from MQ Agent it does not immediately start processing them, instead it waits for one of two conditions to be met. The first condition is an internal timer and when the duration expires, the Full Text Agent starts indexing all documents it has received. The other condition is a high water mark measured in document pages. When this high water mark is met the Full Text Agent starts indexing the pages. The following diagram helps illustrate the logical flow that Full Text Agent follows in order to index documents. Start Request content job from MQ Job available? No Yes Retrieve job document(s) from OSM server Update Document indexing status in ImageNow Database Cache job content for indexing batch No Batching time limit exceeded? No Job size at max pages? Index cached content Yes Yes Page 7 of 19

8 Scaling Full Text Search Agent throughput and performance The following sections how you can scale Full Text Search Agent throughput and performance through general tuning and worker thread tuning. General tuning After the initial installation of both the Perceptive Content Server and Full Text Search Agent, the most beneficial first step is to increase the queue size and search frequency of MQ Agent. Changing queue size and search frequency improves both the overall throughput of content jobs and the response time to complete content jobs. To configure the MQ Agent for a maximum throughput of 500 jobs per minute, complete the following steps. Note If you require more throughput, increase the max.message.per.queue setting or decrease the message.polling.interval. 1. On the Perceptive Content Server computer, navigate to the [drive:] \inserver\etc\ folder and then open the inservermq.ini file in a text editor. 2. In the [General] group, change the following settings: Increase max.message.per.queue to 500 for this example. Leave message.polling.interval at Save and close the file. To further improve response time for content jobs, tune Full Text Agent to start jobs at a lower high water mark or a shorter timer delay. Changing these two settings reduces the time that Full Text Agent waits to start indexing content jobs. To configure the Full Text Agent to wait for a maximum one minute or to begin indexing when it receives 100 pages, complete the following steps. 1. On the Perceptive Content Server computer, navigate to the [drive:] \inserver\etc\ folder and then open the inow.ini file in a text editor. 2. In the [Full Text] group, change the following settings: Decrease content.submission.delay seconds to 60, for this example. Decrease content.submission.threshold to 100, for this example. 3. Save and close the file. Worker thread tuning You should only increasing the number of worker threads for MQ Agent and OSM Agent in cases where the existing worker thread availability is low. Page 8 of 19

9 External performance influences and effects The following sections describe the external influences and effects of the Recognition Agent and Full Text Agent. Recognition Agent One of the most critical performance considerations to a Full Text Search Agent solution is the Recognition Agent. Extracting text via OCR from images is a resource intensive operation. For more information about evaluating and tuning Recognition Agent performance, refer to the Recognition Agent Throughput Best Practices Guide. System impact of tuning Full Text Agent Much of the performance tuning for Full Text Search Agent requires adjustments outside of the Full Text Agent configuration. Before making any changes made to the configuration of MQ Agent, OSM Agent, and Recognition Agent, you should evaluate the changes for any performance impact on other business processes and the Perceptive Content system as a whole. All Full Text Search Agent installations function as remote installations. There is no performance or throughput benefit (other than one based on network performance) to an installation local to the Perceptive Content Server. Currently, Perceptive Software only supports Full Text Search Agent installations on 32-bit based operating systems. Content Index Throughput Testing Results The data in the following test runs was imported using Import Agent. System documents consisted of one to six random pages of text. For non-legacy runs, Import Agent was configured to automatically submit new content for indexing. Legacy runs were configured to index only content added for this test. Constants The average text page size was approximately 3 KB with each page representing a single document page. Due to the nature of how Import Agent submits new pages to content for indexing, the results for this test are lower than would normally be expected. As part of its processing, Import Agent submits each newly added page to Full Text Search Agent as if it were a unique document. This causes each document to be reviewed for indexing once per page processed by Import Agent. Each PDF document imported into the system was a single PDF file averaging 96 KB in size. Each MS Word document imported into the system was a single word document average 32 KB in size. Full Text Agent was configured to submit legacy documents to Full Text Search Agent as fast as possible to keep the agent saturated with requests. Configuration for legacy document submission was the same as in Test Run 4. Unlike Test Run 1, each document was submitted a single time, which resulted in less duplicate requests, eliminating additional processing by Full Text Agent. Page 9 of 19

10 Index performance 1400 Pages Indexed per Minute Pages Indexed per Minute Text PDF Doc Legacy PDF Legacy Text Test Run Data Type Pages Indexed Run Time Pages Indexed per Minute 1 Text :21: PDF :14: Doc :42: Legacy PDF :14: Legacy Text :40: Page 10 of 19

11 Full Text Search Performance Testing Results The data in the following test runs was imported using Import Agent. System documents consisted of one to six random pages of text. Constants During the test, the following constants were used: 64-bit Perceptive Content Server running on Windows Server 2008 R2 Enterprise, 64-bit Operating System, 8 CPU cores, 48GB RAM. 32-bit Full Text Agent running on Windows Server 2008, 32-bit Operating System, 8 CPU cores, 48GB RAM. Search performance of text documents This scale up test varied the number of user connections with each test run. Each user performed a random full text search and opened five random documents from the search results. Simulated user wait times between search and opening of the documents were between 10 and 120 seconds. The database contained approximately 13,500 text files previously indexed by Full Text Agent. Test Run Connected Users Full Text Agent CPU Total Searches Avg Search Time (sec) % 1, % 2, % 4, % 5, % 7, % 8, Page 11 of 19

12 Search performance of Microsoft Word documents This scale up test varied the number of user connections with each test run. Each user performed a random full text search and opened five random documents from the search results. Simulated user wait times between search and opening of the documents were between 10 and 120 seconds. The database contained approximately 13,500 Microsoft Word documents previously indexed by Full Text Agent. Test Run Connected Users Full Text Agent CPU Total Searches Avg Search Time (sec) % 1, % 2, % 4, % 5, % 7, % 8, Page 12 of 19

13 Search performance of PDF documents This scale up test varied the number of user connections with each test run. Each user performed a random full text search and opened five random documents from the search results. Simulated user wait times between search and opening of the documents were between 10 and 120 seconds. The database contained approximately 13,500 PDF documents previously indexed by Full Text Agent. Test Run Connected Users Full Text Agent CPU Total Searches Avg Search Time (sec) % 1, % 2, % 4, % 5, % 7, % 8, Page 13 of 19

14 Search performance varying Full Text Agent worker threads This thread scale up varied the number of Full Text Agent worker threads. We collected data for test runs with 4, 8, 12, 16, 20, and 24 worker threads. Each test run was configured with 600 users, and each of these users performed random full text searches and opened one random document from the search results. Simulated user wait time between searches was between 10 to 30 seconds. We ran this scale up against a database with about 13,500 PDF documents previously indexed by Full Text Agent. Search Performance - PDF Time (sec) Num Threads Test Run FT Threads Full Text Agent CPU Total Searches Avg Search Time (sec) % 16, % 15, % 17, % 16, % 16, % 16, Page 14 of 19

15 80.00 CPU Utilization - PDF CPU (%) inserver inserverft imagenow db Number of Worker Threads Page 15 of 19

16 Full Text Performance with Large-Scale Content Collections This scale-up test investigated search and indexing performance of large-scale content collections. Testing consisted of capturing search and indexing metrics at different milestones, during which Full Text Search Agent indexed over 4 million documents. The tests resulted in these outcomes: Search and indexing functionality remained stable through the steady increase in content collection size. Search performance and indexing performance both scaled successfully as content collection size increased. When collection size was increased by more than 300%, indexing throughput decreased only by 15%. Average search times gained approximately one additional second throughout the entire test. Resource consumption for inserverft increased with content collection size. See the following graph: Resource utilization varies by hardware and resource utilization. System specs Intel Xeon X GB RAM iscsi Equallogic Storage Scalability recommendations For environments expecting content search volume to exceed 50 searches per minute, Perceptive Software recommends that you make the following configuration change to the inserverft.ini file: num.connection.workers=5 With this setting (five incoming search request workers), the server can process over 100 searches per minute. Page 16 of 19

17 Appendix A: INI Configuration File Settings The following tables provide definitions and sample data for the settings in the INI configuration files that have an impact on throughput efficiency for Full Text Agent. inserverft.ini settings The following table provides definitions and sample data for the settings in the inserverft.ini configuration file that have an impact on throughput efficiency for Full Text Agent only. For a complete list of settings in the inserverft.ini configuration file, refer to the Full Text Search Agent Installation Guide. Group Setting Options Description Network num.connection.workers Any positive integer Specifies the number of worker threads in the connection pool. The default is 2. inow.ini settings The following table provides definitions and sample data for the settings in the inow.ini configuration file that have an impact on throughput efficiency for Full Text Agent only. For a complete list of settings in the inow.ini configuration file, refer to the Perceptive Content Server and Client Installation and Setup Guide. Group Setting Options Description Full Text full.text.integration TRUE FALSE Specifies whether Full Text Search Agent submissions, indexing, and searching are enabled. TRUE = Enabled. FALSE = Disabled. The default is FALSE. submissions.dir Any valid directory Specifies where internal bulk files are stored while waiting to be processed. The default is $(IMAGENOWDIR6)/fulltext/submissions. cleanup.bulk.files TRUE FALSE Specifies whether internal bulk files are deleted after they are used. This setting may be temporarily turned off in order to help debug the Full Text Search Agent. TRUE = Internal bulk files are deleted after use. FALSE = Internal bulk files are not deleted. The default is TRUE. Page 17 of 19

18 Group Setting Options Description search.last.n. collections collection.high.water. mark Any positive integer up to the total number of collections Any positive integer Specifies how many collections to search when a query is submitted to Full Text Search Agent. 0 = Search all collections. The default is 0. Specifies the maximum number of records allowed per Full Text Search Agent collection. The default is Full Text (continued) collection.group<n>.dir Any valid directory Specifies the directory to which new collections are dynamically allocated. Full Text Search Agent collections can be distributed among group directories, each of which can house multiple collections. You can define up to nine different group directories by creating new, subsequently numbered collection.group<n>.dir settings. content.submission. delay.seconds content.submission. threshold index.new.content.in. batch Any positive integer Any positive integer TRUE FALSE Specifies how long in seconds ImageNow waits to submit the batch to Verify. The longer ImageNow waits to submit the batch, the more documents are included. The default is 300. Specifies the maximum number of documents ImageNow allows in a batch. If you reach the maximum, ImageNow submits the batch to Verify regardless of the content.submission.delay.seconds setting. The default is 500. Specifies whether ImageNow indexes documents in batches. TRUE = ImageNow indexes documents in batches. FALSE = ImageNow does not index documents in batches. The default is TRUE. Page 18 of 19

19 inservermq.ini settings The following table provides definitions and sample data for the settings in the inservermq.ini configuration file that have an impact on throughput efficiency for Message Queuing Agent only. For a complete list of settings in the inservermq.ini configuration file, refer to the Perceptive Content Server and Client Installation and Setup Guide. Group Setting Options Description General num.workers Any positive integer Specifies the number of worker threads Message Queuing Agent uses for processing messages. The default is 5. message.polling.interval max.messages.per. queue sleep.interval.max sleep.interval.between. send Any positive integer Any positive integer Any positive integer Any positive integer Specifies, in seconds, how often Message Queuing Agent searches for message in the directory. The default is 60. Specifies the maximum number of messages Message Queuing Agent retrieves from the database at one time. The default is 100. Specifies, in milliseconds, how long worker threads wait to check for messages when messages are not available from the database. The default is Specifies, in milliseconds, how long worker threads pause after successfully sending a message before starting the next process. The default is 100. Page 19 of 19