Tuesday, April 19, 2016

Size: px

Start display at page:

Download "Tuesday, April 19, 2016"

Judith McCormick
5 years ago
Views:

1 Relativity Users Group Richmond Assisted Review Best Practices & Tips Tuesday, April 19, 2016

2 Agenda Planning Stage Checklists Index Optimization Reviewer Training Project Stage Random vs. Judgmental Sampling Meta Rounds Adding Documents End Game Stage Handling Uncategorized/Not Responsive Documents

3 Computer-assisted Review Workflow Total Number of Documents 1,000,000 Documents Seed Set for Human Review 1,000 Training Round Results from Categorization Responsive 298,200 Non-Responsive 695,800 5,000 Uncategorized QC of 1st Round (Statistical Sample) 1,533 1,534 QC Round Overturn Report Validation Criteria Not Met 2nd Round of Categorization Responsive 317,589 Non-Responsive 674,877 QC of 2nd Round (Statistical Sample) 1,534 Overturn Report Validation Criteria Met 9,068 documents manually reviewed 990,932 documents categorized by Relativity 317,098 responsive documents prioritized for 2 nd pass review Potential Savings: More than 16,000 hours of review time

4 PLANNING - Assisted Review Checklist RAR Case Checklist Minimum of 50,000 records with text Concept-rich files, e.g. s, word documents, limited excel files, or graphic images Issue or privilege coding in separate field not part of Assisted Review workflow Is this previously reviewed data? If so, how were families reviewed? Expected timelines given Expected process given Number of reviewers Level of reviewers Level of accuracy Meets Criteria

5 PROJECT - Computer-assisted Review Workflow Sample Review Categorize Report/Verify Sample Review

6 PLANNING - Documents to Index Searchable Set Index does not learn any new concepts from searchable documents Includes any documents that you would want to perform analytics functions on: Query, cluster, categorize, etc. Can include more junk data than training Training Set Teaches the index the language of your documents Defines concepts and how documents are related Includes documents that teach your index valuable information Uses representative data from different groups (departments, custodians, etc.) Excludes as much junk as possible

7 PLANNING - Documents to Exclude Searchable Set Documents with more than 30 MB of extracted text Database files or system files (.exe,.rar,.zip) Excel files that contain mostly numbers Documents with poor-quality OCR Extremely short documents (a few words) Training Set Documents with more than 2 MB of extracted text Database files or system files (.exe,.rar,.zip) Excel files that contain mostly numbers Documents with poor-quality OCR Extremely short documents (a few words)

8 PLANNING Analytics Index Filters Header Repeated Content Regular Expression OCR and Go Words

PLANNING - Email Filtering To, From, CC, BCC removed from emails throughout

9 PLANNING - Filtering To, From, CC, BCC removed from s throughout entire thread Subject line remains intact Metadata does not contain concepts

10 PLANNING - Reviewer Training Selecting good example documents Quality of the example is independent of responsiveness it s a second dimension Some documents may be highly responsive, yet undesirable as example documents for a computer-assisted review project Four Corners Doctrine Applying the Use as Example field Using the Excerpt Text feature

11 Headers and Repeated Content Workbook Page 35

12 Images Workbook Page 35

13 Spreadsheet Bad Example Workbook Page 36

14 Spreadsheet Good Example Workbook Page 37

15 The Four Corners Test Workbook Page 38

16 The Four Corners Test The following scenarios violate the Four corners test and will not yield good example documents: The document is conceptually empty, but is a family member of another document which is substantively Responsive. The document comes from a Custodian whose documents are presumptively Responsive. The document was created within a date range which is presumptively Responsive. The document comes from a location or repository where documents are typically Responsive. Workbook Page 37

17 False Negatives Workbook Page 38

18 PROJECT Judgmental Sampling Options Hot Documents Important Custodian Keyword Search Hits Previously Coded Documents Clustering

19 PROJECT Meta Rounds: Overturned Documents

20 Round Summary Report

21 Rank Distribution Report

22 PROJECT - Meta Rounds: Document Comparison

23 PROJECT: Adding More Documents Best Case: Finish coding current round Load new documents Fix Index Saved Searches Rebuild index Categorize Start next round REMEMBER: Never categorize on un-indexed examples

24 END GAME: Uncategorized/Non Responsive Documents Options Save until last (still reviewing all) One last sample Keyword searches for hot documents

25 Additional Online Resources Blog Educational Posts Workflow Recipes Articles Industry Interviews Relativity Assisted Review Page White Papers Webinars Monthly Newsletter Industry Interviews Articles

26 Questions? Available Resources RAR Guide Documentation page Workflow Advice Technical Support Weekly Webinars

Los Angeles User Group

Los Angeles User Group Good to Great: Optimizing Review with 5 Simple Workflow Enhancements How does a user group work You, the community, drive the user group s topics and conversation kcura is here to