Performance Metrics and Recommendations

Transcription

1 Performance Metrics and Recommendations January 29, Version For the most recent version of this document, visit our documentation website.

2 Table of Contents 1 Performance baselines and recommendations Environment details Data set details 6 2 Assisted Review Sample set creation results Saving categorization results Performance recommendations 7 3 Concurrency Image results summary Native review (with Imaging) results summary Processing results summary Native Review (with Processing) results summary 9 4 dtsearch Results summary dtsearch index performance recommendations 9 5 Export Results summary Export fields 11 6 Imaging Results summary 12 7 Import Results summary Import fields 13 8 Legal Hold Active Directory import 13 Relativity Performance Metrics and Recommendations - 2

3 9 Load testing Native review Image review Text review Document viewer OCR Results summary Processing Data set details Inventory performance Processing performance End to end performance (4 Publish Threads) Relativity 8.2 and 9.0 comparison (EDRM data set) General performance recommendations Processing environment and system requirements Productions and branding Results summary Relativity Analytics Results summary Relativity Collection Scout performance Collect performance Structured Analytics Results summary 26 Relativity Performance Metrics and Recommendations - 3

4 1 Performance baselines and recommendations This section provides baselines for certain Relativity functions. All results are for the EDRM Enron V2 data set, which consists of 1.1 million documents, with the exception of Processing, which was tested on the Real World data set, which also consists of 1.1 million documents. Note: These performance results have a +/- 5% margin of error. The following table provides a summary of the performance of several common operations in Relativity. For details, click the link used for each piece of functionality. Functionality Operation Rate Assisted Review Round start - statistical sampling 16,238 Documents/s Round start - stratified sampling 1,067 Documents/s Concurrency Imaging w/doc to Doc Review 2.7 hours Processing w/doc to Doc Review 2.5 hours Doc to Doc Review w/imaging (user wait time) 7.4 seconds Doc to Doc Review w/processing(user wait time) 1.1 seconds dtsearch Index builds (1 worker agent) 1,124,575 Documents/hr Search Terms Reports (4 search threads) 5,541,575 Documents/hr Export Natives + metadata 57,498 MB/hr Images + metadata 380,184 MB/hr Import Natives + metadata (no copy to repository) 50,267 MB/hr Natives + metadata (copy to repository) 37,653 MB/hr Image append (no copy to repository) 18,040 MB/hr Imaging Basic imaging - TIFF 20, images/hr Basic imaging - JPEG 21, images/hr Native imaging (2 worker servers) 47, images/hr Legal Hold Active Directory Import (1 Agent) 47, custodians/hr Active Directory Import (8 Agents) 214,285.7 custodians/hr Load testing Native review cached user wait time seconds Image review cached user wait time seconds Text review user wait time seconds OCR Low accuracy (8 worker agents) 25,343 Images/hr Medium accuracy (8 worker agents) 12,887 Images/hr High accuracy (8 worker agents) 10,600 Images/hr Processing Inventory (4 workers) GB/hr Discovery (8 workers) 53.1 GB/hr Publish (4 threads) 52.1 GB/hr Total expanded rate (8 workers) 47.1 GB/hr Relativity Performance Metrics and Recommendations - 4

5 Functionality Operation Rate Productions and branding 8 worker agents 142,199 Images/hr Relativity Analytics Indexing 1,761,720.7 documents/hr Categorization (5,000 examples) 1,181,105.7 documents/hr Clustering 1,107,759.4 documents/hr Relativity Collection Collect (34 GB) 00:10:02 minutes/seconds Scout (200 GB) 00:05:03 minutes/seconds Structured Analytics threading 1,184,623 records/hr Near duplicate identification 1,132,678 records/hr Repeated content identification 1,813,110 records/hr Language identification 1,481,551 records/hr All operations 804,279 records/hr 1.1 Environment details The following table provides Relativity environment server specifications: Server Memory (GB) Cores Host details SQL * Dell PowerEdge R720 2 x Intel E Ghz 8 Core Processor 8 x 16GB (128GB) 1600MHz RDIMM Memory 2 x Intel i350 Quad Port Gb Network Adapter PERC H710P RAID Controller Web 8 8 Dell PowerEdge R820 4 x Intel E Ghz 8 Core Agent 4 4 Processor File x 16GB (128GB) 1600MHz RDIMM Memory Search/CA 64 4 Intel i350 Quad Port Gb Network Adapter Loading 4 4 Brocade 825 Dual Port 8Gb Fibre Invariant SQL 32 8 Channel Host Bus Adapter PERC H710P RAID Controller Workers * 16 8 Dell PowerEdge R420 Relativity Performance Metrics and Recommendations - 5

6 Server Memory (GB) Cores physical 16 logical Host details 2 x Intel E Ghz 6C Processor 2 x 8GB (16GB) 1600Mhz RDIMM Intel i350 Quad Port Gb Network Adapter * The SQL Server and Workers are physical machines. All others are VM's. 1.2 Data set details The following are the details of the data set on which the performance tests were run: Data set name File count Compressed Size (GB) Expanded Size (GB) EDRM Enron V2 1,104, Real World 1,040, Note: To obtain the folders containing the EDRM and Real World test data sets, support@kcura.com. 2 Assisted Review All sample sets were created in a workspace with no prior Assisted Review projects or rounds using the same 1,104,190 document set with no prior coding. Statistical samples were created with 95% confidence level and 2.5% margin of error. Categorization counts were artificially inflated in the first round by editing the categorization set associated with the round. All tests were performed with eight Assisted Review worker agents and times were measured through Assisted Review audits. Note: These performance results have a +/- 5% margin of error. 2.1 Sample set creation results The following table provides a breakdown of sample set creation performance: Test case name Sample size Time (mm:ss) Docs/second (universe) Sample set creation - statistical 1,535 01:08 16, Sample set creation - fixed size 50,000 02:40 6, Sample set creation - stratified sample 1,454 17:15 1, Docs/second (sample set) Relativity Performance Metrics and Recommendations - 6

7 Note: Stratified sampling is data dependent. Two data sets of the same size could perform drastically different depending on the composition of the data. 2.2 Saving categorization results The following table provides a breakdown of categorization save performance: Test case name Saving categorization results Categorized document count Save duration (hh:mm:ss) 1,039,739 0:36:00 1,732,898 Categorization results saved/hr 2.3 Performance recommendations Assisted Review uses Analytics Categorization in ways that other workflows do not. Up to two categorization sets, one each for issues and designation, can be active at the same time. You can generate far more examples than many other use cases for Categorization. Both of these factors produces additional strain on the CA server, and requires an additional Categorization Manager agent. Assisted review takes advantage of this functionality by default. If you wish to only categorize one categorization set at a time, verify that only one Categorization Manager agent is enabled in the resource pool. Additional examples cause a significant increase in memory utilization by the Analytics process. It is highly recommended that sufficient memory be allocated to the search server prior to starting any Assisted Review projects, in excess of what searching would need. For a workspace of approximately 3 million documents, no less than 64 GB should be reserved for the index and categorization in later rounds. 3 Concurrency In Relativity 9.0, document review, processing, and imaging make use of the same workers. As a result, careful consideration should be taken when scheduling large imaging or processing jobs. This page provides performance baselines for processes using the same workers at the same time. Note: These performance results have a +/- 5% margin of error. 3.1 Image results summary Native imaging was running with 100,00 documents while 500 document reviewers were concurrently viewing and coding documents. Relativity Performance Metrics and Recommendations - 7

8 Test case # of workers Imaging job (total execution time in hours) Imaging ONLY 4 (4 dedicated to Imaging only) 2.45 Imaging w/ Doc to Doc Review Imaging w/doc to Doc Review 8 (4 Imaging dedicated, 4 Conversion Dedicated) 8 (openly assignable to either Imaging or Conversion) Native review (with Imaging) results summary 500 document reviewers were concurrently viewing and coding documents while native imaging was running with 100,00 documents. Test case # of workers Download document time Doc to Doc Review Only 4 (4 dedicated to conversion only) Doc to Doc 8 (4 Imaging dedicated, 4 Review w/imaging Conversion Dedicated) Doc to Doc 8 (openly assignable to Review w/imaging either Imaging or Conversion) 2.7 Code document, Save and Next time Load persistent highlights time Imaging takes approximately 40% longer when run concurrently with doc to doc native review. Doc to doc review response times double when concurrently running an imaging job. When running imaging or processing concurrently with doc to doc review, imaging/processing job level errors may become more frequent. 3.3 Processing results summary Processing was discovering 1.1 million documents while 500 document reviewers were concurrently viewing and coding documents. Test Case Number of Workers Discovery Job(Total Execution Time) Processing ONLY 4 (4 dedicated to Processing only) 2.61 hrs Processing w/ Doc to Doc Review Processing w/doc to Doc Review 8 (4 Processing dedicated, 4 Conversion Dedicated) 8 (openly assignable to either Processing or Conversion) 3 hrs 2.5 hrs Relativity Performance Metrics and Recommendations - 8

9 3.4 Native Review (with Processing) results summary 500 document reviewers were concurrently viewing and coding documents while Processing was discovering 1.1 million documents. All times are listed in seconds to complete. Test Case Number of Workers Download Document Doc to Doc Review ONLY Doc to Doc Review w/processing Doc to Doc Review w/processing 4 dtsearch Code Document, Save and Next 4 (4 dedicated to conversion only) (4 Processing dedicated, 4 Conversion Dedicated) 8 (openly assignable to either Processing or Conversion) Load Persistent Highlights The index was built with one dtsearch index worker agent using the All Documents in Workspace searchable set. Included in this 1,104,990 document set were three fields: Extracted Text, Document Identifier, and Group identifier. A 10,000 term list was selected based on terms that occurred in at least 1,000 of the searchable documents. Each term was a simple keyword, no advanced searching functionality was present (proximity search, Boolean conditions, or wildcards). The test was performed with one Search Terms Report manager agent and one dtsearch Search manager agent (hosted on a unique server). Note: These performance results have a +/- 5% margin of error Results summary The following table provides a breakdown of dtsearch performance: Test case name Execution time (h:mm:ss) Documents/hr Terms/hr dtsearch Index Build 0:50:35 1,124,529 N/A Search Terms Report - 10,000 terms 0:11:58 5,541,575 50, dtsearch index performance recommendations Refer to the information below when deciding how large to make a dtsearch sub-index. Workspace document count Recommended sub-index size Less than 10 million 250,000 documents 10 million to 25 million 500,000 documents Greater than 25 million 1,000,000 documents Relativity Performance Metrics and Recommendations - 9

10 Agent count - often the easiest and most intuitive way of making dtsearch indexes build faster is to add more dtsearch index worker agents to the job. Only one agent can work on one sub-index. Adding more agents than the number of sub-indexes currently building will result in excess agents sitting idle. Agent temporary directory - you can now configure agent servers to use specific UNC paths for dtsearch builds, which allows administrators to segregate storage by agent server (reducing contention) or to use a NAS with faster write speeds (for building) without impacting the NAS which is optimized for reads (for searching). o o Use caution when modifying this setting. In a virtual environment with shared storage, changing from the default may result in diminished performance due to additional data copy activities performed through the index build. Relativity recommends that you test any changes on your hardware before committing to any system-wide configuration changes. If shared storage is involved or the local storage is not sufficiently isolated and fast, then you should use the default index share. Agent server configuration - for some other processes in Relativity, it is better to have many small agent servers, as opposed to a few large agent servers, or vice versa. Subindex size - this number represents the maximum size of each subindex and the number of subindexes. While less intuitive, this is perhaps the most critical setting to get right when building the index, as it impacts searching later. There are pros and cons to smaller indexes, such as the ability to make parallel at the cost of higher I/O cost. Finding a good default is a high priority. Agent count - as with other processes in Relativity, adding more agents tends to make things faster. Unlike other agents, however, the dtsearch search agent will utilize all resources on the server it is running on. Because of this, there should only ever be one search agent running on any given agent server. Agent configuration - adding more CPU cores to an existing search server will make searches faster without any additional configuration. 5 Export All exports were completed in direct mode, copying files directly from the repository. Export was the only action executing on the machine. Images were exported as single-page tiffs. Of the 1,104,990 documents tested, 100,000 had images. Extracted Text was exported into separate files. All metrics are calculated using the total number of documents. The average file size of all documents exported was 0.11 MB. Note: These performance results have a +/- 5% margin of error Results summary The following tables provide a breakdown of export performance: Relativity Performance Metrics and Recommendations - 10

11 Test case name Natives + metadata Images + metadata Images.dat file size (MB).opt size (MB) File size, natives + images (MB) 0 1,050 N/A 133,351 2:07:12 353, ,328 0:55:46 Export time (hh:mm:ss) Test case Records/hr Images/hr MB/hr name Natives + 472,723 N/A 57,498 metadata Average file transfer rate (MB/sec) Metadata transfer rate (MB/sec) Images + 1,188, , , metadata Export fields Natives + metadata o 46 fields were exported: 11 long text fields (including Extracted Text) 18 fixed-length text fields 2 whole number fields 7 date fields 1 yes-no field 2 single-choice fields 2 decimal fields 1 multi-choice field 1 single-object field Images + metadata o 11 fields were exported: 5 fixed-length text fields 2 long text fields 3 date fields 1 single choice field 6 Imaging Starting in Relativity 9, basic imaging uses Invariant workers to complete both basic and native imaging. Basic imaging tests were set up to utilize one Processing worker. Each set output original sized images with 300 DPI. Native imaging tests used eight imaging worker agents per imaging worker Relativity Performance Metrics and Recommendations - 11

12 server. The default imaging profile was used and images were generated in tiff format. The document pool for all tests was 100,000 documents. All imaging tests were submitted through an imaging set and times recorded from audits on the imaging set. No other actions were taking place in the environment during the imaging scenarios, and memory on all machines was cleared immediately prior to eliminate effects of data caching. Note: These performance results have a +/- 5% margin of error Results summary The following tables provides a breakdown of imaging performance: Test case name Images created Imaging time (hh:mm:ss) Images/hr Basic imaging TIFF 118,676 0:5:50:17 20, Basic imaging JPEG 136,442 06:28:42 21, Native imaging 1 worker server 124,586 04:47:37 25, Native imaging 2 worker servers 124,586 02:38:51 47, Import All imports were completed in direct mode, copying files directly from the repository. Import was the only action executing on the machine. Imported images were exported as single-page tiffs. Of the 1,104,990 documents tested, 100,000 had images. Performance baselines are calculated using the total number of documents. Appending Natives imported Extracted Text stored in separate files. There was no Extracted Text imported when appending images. Note: These performance results have a +/- 5% margin of error. 7.1 Results summary The following tables provide a breakdown of import performance: Import type Images.dat file size (MB).opt size (MB) File size, natives + images (MB) Import time (hh:mm:ss) Native append N/A N/A 136,537 2:44:13 Native append - copy to repository N/A N/A 136,537 3:39:14 Image append 353,326 N/A ,739 1:02:26 Relativity Performance Metrics and Recommendations - 12

13 Import type Records/hr Images/hr MB/hr Metadata transfer rate (MB/sec) Native 403,731 N/A 50,267 append Native append - copy to repository Image append 7.2 Import fields File transfer rate (MB/Sec) N/A ,991 N/A 37, N/A 339,555 18, N/A fields were imported: o o o o o o o o o 8 Legal Hold 12 long text fields (including Extracted Text) 18 fixed-length text fields 2 whole number fields 7 date fields 1 yes-no field 2 single-choice fields 2 decimal fields 1 multi-choice field 1 single-object field SQL process rate (docs/sec) The following table breaks down the performance of the active directory import component of Relativity Legal Hold. For all tests, 400,000 active directory users/custodians were imported in batches of 2,000. Note: These performance results have a +/- 5% margin of error Active Directory import Agents Total time (hh:mm:ss) Custodians per hour 1 08:28:00 47, :15:00 123, :52:00 214, Relativity Performance Metrics and Recommendations - 13

14 9 Load testing This section provides data on web server performance and response times for certain operations. In Relativity 8.2 and earlier, the ActiveX viewer was responsible for all activities around viewing a file, including downloading it and converting it to a format usable by the browser. The new viewer in Relativity 9 has separated these concerns. Invariant now converts natives and images to a browserfriendly format, and the browser downloads and renders the file. Once a document has been converted, Relativity stores the converted files on the cache server, and in the ConvertedCacheFile table in the workspace. Viewing the document a second time skips the conversion workflow. You'll encounter the following terms throughout this section: Conversion duration - the time that the user waits on conversion. Note that this time is reduced by preconvert. This time does not include downloading the converted document. Cache hit duration - the time that the user waits on the system to respond with a cache ID, in the event that the system did not need to convert a document. This happens when viewing a document after the first time, or when preconvert has finished converting the document while the user was on a prior document. Download duration - the time that the user waits on the relevant portion of the document to download. This includes the time to download persistent highlight set and markup set information. User wait time - the time taken by the download, render, and conversion or cache hit time. This measures the total amount of time the user is waiting on Relativity. Cache hit rate - the percentage of the time when the user did not need to wait on conversion, because the document was already in the cache. Note: These performance results have a +/- 5% margin of error. 9.1 Native review The Relativity 9 native viewer no longer downloads a single document, instead the converted document consists of many smaller HTML files. The following table breaks down native review times. User Review type Conversion count duration 500 Native (empty cache) 500 Native (documents in cache) Cache hit duration Download duration Persistent highlight set retrieval time User wait time Cache hit rate (%) N/A Relativity Performance Metrics and Recommendations - 14

15 9.2 Image review The Relativity 9 image viewer no longer downloads and views one page at a time, nor are thumbnails generated on the fly. Relativity streams down the visible part of a collection of images on demand. While a user is viewing page 1, later pages stream in the background; this allows the page to render before being completely downloaded, while being responsive to paging. There was an average of 3.54 images per document. The following table breaks down image review times. User Review type count 500 Images (empty cache) 500 Images (documents in cache) Conversion duration Cache hit duration Download duration Redaction retrieval time User wait time Cache hit rate (%) N/A Text review The following table breaks down text review times. User count Review type 500 Text- SQL Download duration Persistent highlight set retrieval time User wait time Failure rate (%) Document viewer The following table measures the time (in seconds) for native files of various sizes to render in the document viewer. Viewer Document extension File size Legacy ActiveX Not cached Not cached Windows/Chrome Windows/IE 10 HTML Cached Cached Windows/Chrome Native.DOCX 32KB Native.DOCX 88KB Native.DOCX 221KB Native.XLSX 100KB Windows/ IE10 HTML Relativity Performance Metrics and Recommendations - 15

16 Viewer Document extension File size Legacy ActiveX Not cached Not cached Windows/Chrome Windows/IE 10 HTML Cached Cached Windows/Chrome Native.XLSX 250KB Native.XLSX 500KB Windows/ IE10 HTML The following table measures the time (in seconds) for image files of various sizes to render in the document viewer. Viewer Document extension # of pages Legacy ActiveX Not cached Not cached Windows/Chrome Windows/IE 10 HTML Cached Cached Windows/Chrome Image.JPG Image.JPG Image.JPG Image.JPG Image.JPG Image.JPG Image.JPG Image.TIF Image.TIF Image.TIF Image.TIF Image.TIF Image.TIF Image.TIF Windows/ IE10 HTML 10 OCR All OCR tests were performed with eight OCR Worker agents using the default OCR set settings. The document pool contained 100,000 documents consisted of 353,326 TIFF images. All times were taken from the Audits for the OCR Set. OCR Time is the measurement of how much time is spent reading image files from disk, running the OCR process for each page, and saving the text for a page to the database. Compilation time is the measurement of how much time is spent combining page-level OCR results into document-level results, and saving these to the Document table. Relativity Performance Metrics and Recommendations - 16

17 Note: These performance results have a +/- 5% margin of error Results summary The following table provides a breakdown of OCR performance: Test case name Documents to OCR Images OCR time (hh:mm:ss) Compilation time (hh:mm:ss) Total time (hh:mm:ss) Images/hr OCR Low Accuracy 100, ,326 13:36:10 0:18:32 13:56:31 25,343 OCR Medium Accuracy 100, ,326 26:58:43 0:24:30 27:25:00 12,887 OCR High Accur-100,00acy 353,326 32:45:43 0:32:25 33:19:58 10,600 Relativity Performance Metrics and Recommendations - 17

18 11 Processing The following tables break down performance baselines for Relativity Processing, as run on the Real World data set, with all workers dedicated to processing only. Note: These performance results have a +/- 5% margin of error Data set details The Real World data set includes the following specifications: Source size - 60 GB Source file count - 100K assorted files Expanded size GB Note: Exporting all natives in a workspace, including duplicates, will give the expanded size of a data set. Published documents - 1,104,949 Fileshare size GB Note the following file breakdown: 70% s - 42 GB of various PSTs and loose s 15% Office files - 10 GB of Excels, Word, PPTs, 10% PDF - 6GB PDFs 5% other (CAD files, Text files) - ~3GB Note: To obtain the folders containing the EDRM and Real World test data sets, support@kcura.com. Publish was performed using 4 threads. Note: There is almost no contention on the file share during the publish phase, but there is some SQL contention as data is inserted into the workspace database. SQL processor utilization will increase during the publishing phase Inventory performance Test case name Inventory time (mm:ss) Finalize time (mm:ss) Total time (mm:ss) GB/hr Expanded GB/hr 1 Worker 00:21:23 00:00:08 00:21: Workers 00:05:27 00:00:05 00:05: Workers 00:03:39 00:00:05 00:03: Relativity Performance Metrics and Recommendations - 18

19 Test case name Inventory time (mm:ss) Finalize time (mm:ss) Total time (mm:ss) GB/hr Expanded GB/hr 8 Workers 00:03:39 00:00:05 00:03: Processing performance All time entries in the following table appear in the (hh:mm) format. Test case name 1 Worker 2 Workers 4 Workers 8 Workers 4 Threads Ingestion time Text extraction time Discovery time GB/hr Expanded GB/hr Publish time GB/hr Expanded GB/hr 02:51 06:33 09: N/A N/A N/A 00:53 01:49 02: N/A N/A N/A 00:42 00:57 01: N/A N/A N/A 00:33 00:34 01: N/A N/A N/A N/A N/A N/A N/A N/A 01: Relativity Performance Metrics and Recommendations - 19

20 * Publish uses one worker thread per workspace End to end performance (4 Publish Threads) Test case name Total time (hh:mm) Total GB/hr Total Expanded GB/hr 1 Worker 10: Workers 03: Workers 02: Worker 02: Relativity 8.2 and 9.0 comparison (EDRM data set) The following table and graphic provide a comparison of processing rates between Relativity 8.2 and 9.0. The following tests were performed with only two publish threads, so as to provide an accurate comparison to 8.2 tests. Test case name 8.2 time to complete 9.0 time to complete 8.2 Expanded GB/hr 9.0 Expanded GB/hr 1 Worker 10:27 11: Relativity Performance Metrics and Recommendations - 20

21 Test case name 8.2 time to complete 9.0 time to complete 8.2 Expanded GB/hr 9.0 Expanded GB/hr 2 Workers 06:39 04: Workers 05:23 03: General performance recommendations Worker Threads - by default and if enough memory is available (750mb/thread), Processing will run two worker threads per CPU core on each worker server up to 16 worker threads per worker machine. Further improvements could be realized by scaling horizontally with multiple worker machines. Worker Machines - when scaling Relativity Processing Worker machines horizontally, it is equally important to scale the file server if you want to continue to see linear improvements in performance. It will do no good to put 20+ worker machines online if your file server does not have the available IOPS to support them. Invariant Queue - during discovery, the Invariant Queue Manager, workers, and default file share for the workspace are heavily utilized. CPU, network, and disk resources should be closely monitored, to detect and prevent any issues which may impact performance for other processes in the system, particularly review and imaging Processing environment and system requirements It's important to determine your infrastructure needs based on your desired throughput. We recommend that you refer to the applicable tier to locate the necessary hardware components to complete your processing infrastructure. Relativity Performance Metrics and Recommendations - 21

22 The below table specifies the environment these tests were conducted on. This environment falls into the Tier 2 specifications as outlined in the system requirements page. Note: The environment used to test Processing in 9.0, although similar, has some significant differences than the environment used for testing in prior versions of Relativity. Server Memory (GB) Cores Host details SQL * Dell PowerEdge R720xd 2 x Intel E Ghz 8 Core Processor 16 x 16GB (256GB) 1866MHz RDIMM Memory Broadcom x10Gb BT / 2x1Gb BT Network Adapter PERC H710P RAID Controller Web 8 8 Split between two of the following: Agent 8 8 File 4 4 Search/CA/Loading 64 8 Invariant SQL 32 8 Worker * 16 8 physical 16 logical Dell PowerEdge R820 4 x Intel E v2 3.3Ghz 8 Core Processor 16 x 8GB (128GB) 1866MHz RDIMM Memory Broadcom x10Gb BT / 2x1Gb BT Network Adapter PERC H710P RAID Controller Dell PowerEdge R420 2 x Intel E v2 2.2Ghz 6C Processor 4 x 4GB (16GB) 1600Mhz RDIMM Onboard Dual Gigabit and Broadcom x10Gb BT Network Adapter File Server * 4 4 Dell PowerEdge R820xd 2 x Intel E v2 2.1Ghz 6C Processors 16GB 1600Mhz RDIMM (4 x 4GB) 26 x 1.2TB 10K SAS Hard Drives Broadcom x10Gb BT / 2x1Gb BT Network Adapter PERC H710P RAID Controller * Machines are physical machines. All others are virtual. You can find this information in the Processing System Requirements. Relativity Performance Metrics and Recommendations - 22

23 12 Productions and branding Eight Branding Managers were used in all tests, with four agents per agent server. Each production consisted of images only, no natives. No redactions were present in the markup set used for the scenarios. The document pool contained 100,000 documents and consisted of 353,326 TIFF images. Page-level numbering was used in all scenarios with two stamps used: Production Bates Number in the left header and Control Number in the right footer. Scenarios with Relational Attachment used the relational field Group Identifier. Families ranged in size between 1 and 327 documents per family. The Median family size was 1 document per family, mean family size was documents per family. No other actions were taking place in the environment during the production or branding steps, and memory on all machines was cleared immediately prior to eliminate effects of data caching. All times were measured using the Windows Event Log on the agent servers. Branding time was measured as the difference between when the first agent started work on the first document, until the last production image had been created. Note: These performance results have a +/- 5% margin of error Results summary The following table provides a breakdown of production/branding performance: Test case name Documents Images Average family size Production time (hh:mm:ss) Branding time (hh:mm:ss) Images/hr New Bates 100, ,326 N/A 0:11:41 2:17:24 142, New Bates with Relational Attachment 100, , :11:05 2:18:00 142, Original Bates 100, ,326 N/A 0:11:38 2:18:25 141, Original Bates with Relational Attachment 100, , :10:54 2:20:45 139, Relativity Performance Metrics and Recommendations - 23

24 13 Relativity Analytics The Analytics index build was performed using the default searchable and training sets. Start time was measured as the time the first document was sent to the Analytics server, and end time was measured as when the index of 1,104,990 documents was active and searchable. Indexes were built to not automatically remove English signatures and footers. Analytics clustering was performed on all documents in the workspace, in a new cluster. Start time was measured as the time the mass operation was initiated, and end time was measured as the time the job was removed from the relevant queue in the database. Each Analytics categorization was run using a new categorization set consisting of all 1,104,990 documents. Samples were taken by coding the first 1,000 or 5,000 documents with extracted text, sorted by ArtifactID ascending, and coding them to one of the categories. Times were recorded from audits of the categorization set. Note: These performance results have a +/- 5% margin of error Results summary The following tables provide a breakdown of Analytics performance: Test case name Operation time (h:mm:ss) Documents/hr Analytics Index Build 00:37:38 1,761, Analytics Clustering 00:59:31 1,107, The following table provides a breakdown of categorization performance. Test case name Examples Categories Operation time (h:mm:ss) Documents/hr Categorization 1 1, :21:59 3,015, Categorization 2 5, :56:08 1,181, Categorization 3 5, :56:57 1,164, Relativity Collection This section provides performance baselines for the following components of Relativity Collection: Scout Collect Note: These performance results have a +/- 5% margin of error. Relativity Performance Metrics and Recommendations - 24

25 14.1 Scout performance The following table provides a comparison of execution times between Collection 1.1 and 1.2. Functionality Collection 1.1 execution time (mm:ss) Scout (75,000 files, 35 GB) 1:52 1:08 Scout (150,000 files, 35 GB) 3:46 4:47 Scout (700,000 files, 200 GB) 9:02 5:03 Exchange Scout (38,000 files) N/A 11:16 Collection 1.2 execution time (mm:ss) 14.2 Collect performance The following table provides a comparison of execution times between Collection 1.1 and 1.2. Functionality Collection 1.1 execution time (hh:mm:ss) Collect (8,500 files, 25 GB) 19:02 6:52 Collect (26,000 files, 30 GB) 38:19 11:13 Collect (60,000 files, 34 GB) 2:21:45 10:02 Collect (423,301 files, 28 GB) N/A 48:32 Collection 1.2 execution time (hh:mm:ss) Collect (615,051 files, 80 GB) N/A 1:13:02 Exchange Collect (38,000 files) N/A 6:24:53 15 Structured Analytics All 1,104,990 documents in the workspace were used in each job. Each job was executed with eight Structured Analytics Worker agents and times were measured by polling the Structured Analytics job queue. Each Structured Analytics operation consists of three distinct phases: 1. Export - relevant document metadata and text is exported from Relativity to the Analytics engine. 2. Structured Analytics operations - the Analytics engine processes all documents, and determines the relevant information to be sent back to Relativity. 3. Importing results - the data that the Analytics engine extracted from the documents is sent back to Relativity through the Import API. This phase also includes job finalization and cleanup. Relativity Performance Metrics and Recommendations - 25

26 Note: These performance results have a +/- 5% margin of error Results summary The following table provides a breakdown of Structured Analytics performance. All durations in the following table are in (h:mm:ss) format. Test case name Export dur-structureatioations Analytics oper- duration Import results duration Total duration Documents/hr Language Identification 0:16:58 0:08:14 0:12:49 0:44:45 1,148,551 Textual Near Duplicate identification Repeated Content Identification 0:16:31 0:22:40 0:12:28 0:58:32 1,132,678 0:16:24 0:14:19 0:00:15 0:36:34 1,813,110 Threading 0:17:41 0:17:46 0:13:57 0:55:58 1,184,623 All operations 0:17:45 0:40:35 0:17:18 1:22:26 804,279 Relativity Performance Metrics and Recommendations - 26

27 Proprietary Rights This documentation ( Documentation ) and the software to which it relates ( Software ) belongs to kcura Corporation and/or kcura s third party software vendors. kcura grants written license agreements which contain restrictions. All parties accessing the Documentation or Software must: respect proprietary rights of kcura and third parties; comply with your organization s license agreement, including but not limited to license restrictions on use, copying, modifications, reverse engineering, and derivative products; and refrain from any misuse or misappropriation of this Documentation or Software in whole or in part. The Software and Documentation is protected by the Copyright Act of 1976, as amended, and the Software code is protected by the Illinois Trade Secrets Act. Violations can involve substantial civil liabilities, exemplary damages, and criminal penalties, including fines and possible imprisonment kcura Corporation. All rights reserved. Relativity and kcura are registered trademarks of kcura Corporation. Relativity Performance Metrics and Recommendations - 27