Building Blocks of a Modern Search Engine

Transcription

1

2 Building Blocks of a Modern Search Engine 2018 National Laboratory Information Technology Summit (NLIT) Nicole M Shaw, Los Alamos (nshaw@lanl.gov) Clayton J Pryor, Sandia (cjpryor@sandia.gov) Amy Rein, Sandia (abethel@sandia.gov) May 22,2018 LA-UR

3 Introductions Nicole M Shaw Systems Programmer Started at LANL in 2015 Bachelor s in Computer Engineering from University of New Mexico. Primary web system administrator for institutional web server infrastructure. Deployed Fusion to production in late 2017 Clay J Pryor Principal Member of Technical Staff Over 30 years of experience at Sandia Mostly software development ranging from desktop systems to distributed enterprise systems Been supporting enterprise search since 2015 (post initial Solr deployment) Ran a three month Fusion Proof Of Concept for our environment in late 2016 to early 2017 Deployed Fusion in our environment from approximately mid 2017 to early 2018 Amy Rein Information Management Professional at Sandia National Laboratories Master s degree in Library and Information Science Currently works in the Knowledge Systems department on search optimization, information analysis, quality testing, and customer outreach. 6/13/2018 3

4 A History of Search at LANL and Sandia 6/13/2018 4

5 History of LANL Search mid 2000 s 2017: LANL uses Google Search Appliances for web search. Pros: All in one search solution front end, back end, analytics all bundled together Simplicity only needed to set up some options and plug and play Self-hosted Cons: Yellow black box Google appliance was proprietary. Dependency on key matches manually matched search terms to the correct search result. Documentation very sparse reliance on outside support for assistance Other than the GSA, Apache Solr has also been used, primarily in the Research Library 2017 Lucidworks Fusion launched in production All in one, self-hosted, search solution built on top of open source Apache Solr Customization options for different data sources (i.e. Web, SharePoint) Lots of Apache Solr documentation Cons For web, heavy reliance on metadata Updating can be complicated. 6/13/2018 5

6 History of Sandia Search Before my time Earlier COTS solutions Custom J2EE Middle Tier (SPN) Custom Aggregate Behavior Model (ABM) Custom batch jobs to crawl content from internal web pages When I arrived all of the above plus Single Solr node replaced earlier COTS solutions Difficult to enhance and maintain Unable to index all content - we could not honor ACLs My influence Lucidworks Fusion Proof of Concept in our environment Deployed Fusion to replace SPN and ABM To Be Continued replace custom batch jobs with Fusion connectors Investigate Machine Learning and other Fusion features 6/13/2018 6

7 What is your current search solution? Are you planning to upgrade/change soon? 6/13/2018 7

8 What are the most important considerations when implementing a new search solution or updating the current infrastructure? 6/13/2018 8

9 What are the biggest obstacles in your search engine infrastructure? What are the challenges you anticipate facing? 6/13/2018 9

10 To Cloud or Not to Cloud? 6/13/

11 In the end.. Whatever search solution you pursue, always look at your data first! Understanding your data will effectively help you tailor your search solutions to your needs Plan into your project timeline If deploying a brand new solution, make sure to have plenty of time to learn it. Collaboration! Reaching out to other laboratories, and exchanging ideas can be critical to the success of your search solution. Expect the unexpected! As we all know, projects never go as planned. Make sure to take time for any delays or unforeseeable circumstances. 6/13/

12 Q & A 6/13/