Language Support, Linguistics, and Text Analytics in Solr

Transcription

1 Boston Apache Lucene and Solr Meetup Language Support, Linguistics, and Text Analytics in Solr Carl Steve W. Kearns Hoffman Product Manager Basis Technology Founder & CEO

2 Agenda About Basis Technology Language Identification Linguistics for Search Entity Extraction in Solr Demonstration Application

3 About Basis Technology This is Headquarters, offices in: Tokyo, San Francisco, Washington DC Specialists in natural language processing for Web/enterprise search Document/OSINT/media exploitation E-discovery Digital forensics Developer of a mature and widely used platform for multilingual text analytics and information retrieval Solutions for commercial enterprises and government agencies

4 How well will it work for me? Define! Definition of success varies widely Log File Search: Return only things that match exactly exception Product Search: Return similar results organized by category. Measure! Create examples, track performance.

5 Language Identification Detect dominant language Find language regions

6 Language Identification Why? Faceting Language-specific indexing Entity extraction

7 Language Identification in Solr Preprocessor to Solr Custom OpenPipeline Solr UpdateRequestProcessor Chain of URP s may be defined in SolrConfig.xml Has full access to the document Add, Edit, Remove fields UpdateRequestProcessor Chain Field Analysis Solr Doc Language Identification URP Add Language field Rename Text: text_<lang> Solr Field Language text_swedish Other Custom URP text_english SolrConfig.xml Schema.xml

8 Language Identification Challenges Identifying query language is hard How do you query multiple fields at the same time? Use the Dismax parser: /solr/select?qt=dismax&qf=text_english%20text_swedish%20detext&q=hello%20world The QT specifies the query type as dismax The QF specifies the fields to search

9 Linguistics for Search Why? Improve recall! Every language has a unique set of challenges: Tokenization Chinese, Japanese, Korean, Thai Morphological Analysis vs. N-Gram Stemming vs. Lemmatization All European and Middle Eastern languages Compound words Swedish, Danish, Norwegian, Dutch, German, Korean, Japanese

10 Morphological Analysis vs. N-Gram Search Term: 東京 N-Gram: 東京ルパン上映時間 Morphological:

11 Stemming vs. Lemmatization Stemming: Set of language-specific rules for removing leading and trailing characters from words Intended to increase recall at the expense of precision Example EN rule: Remove trailing ing Lemmatization: Complex set of language-specific approaches for producing the dictionary form of a given word Intended to increase recall without hurting precision. Uses context to disambiguate when multiple dictionary forms exist

12 Stemming vs. Lemmatization English: I have spoken at several conferences Stemming: Lemmatization:

13 Stemming vs. Lemmatization French: Je n étais pas là Stemming: Lemmatization:

14 Stemming vs. Lemmatization + Decompounding German: Am Samstagmorgen fliege ich zurueck nach Boston. Stemming: Lemmatization (and decompounding!)

15 Stemming vs. Lemmatization Swedish: En person skadades lindrigt i en trafikolycka i Pernå Stemming: Lemmatization:

16 Linguistics in Solr Easy to customize as Analyzer/Tokenizer/TokenFilter UpdateRequestProcessor Chain Field Analysis Solr Doc Language Identification URP Add Language field Rename Text: text_<lang> Solr Field Language text_swedish Other Custom URP text_english SolrConfig.xml Schema.xml

17 Related Challenges Can I index text from many languages into the same field? Yes, but it s not always a good idea, because query language ID is not accurate. You need a custom Query Analyzer that does stemming/lemmatization in many languages for the same query. How do I query text in multiple fields? Dismax parser!

18 Text Analytics in Solr: Entity Extraction Process of identifying people, places, organizations, dates, times, etc. in unstructured text. Methods: Lists Rules Statistical Define your goals upfront! Some extraction methods work better for certain entity types Rules work well for dates, addresses, and URL s, but not people Lists work well for titles, but not locations Statistical extractors work well for ambiguous entities: people, locations, organizations

19 Entity Extraction

20 Entity Extraction in Solr Pre-processor to Solr Custom OpenPipeline UpdateRequestProcessor Store entities in new fields per entity type <field name="person" type="string" indexed="true" multivalued="true" stored="false" /> UpdateRequestProcessor Chain Field Analysis Solr Doc Language Identification URP Add Language field Rename Text: text_<lang> Solr Field Language text_swedish Entity Extraction URP text_english SolrConfig.xml Schema.xml

21 Entity Extraction Challenges How do you use extracted entities as facets? For retrieving counts: &facet=true&facet.field=person&facet.field=location For filtering results: &facet.query=person:steve Kearns&facet.query=LOCATION:Stockholm How else can Entities be used? Improve relevance by searching the entity fields with a boost Entity-specific search phonetic matching and other name-specific search appoaches Measure accuracy! F-Score is a measurement that combines precision and recall Vendors should provide this, but evaluate on your own data!

22 Demo: Odyssey Information Navigator Example search application built on Solr I personally built this in < 2 months using Solr and products from Basis Technology I spent more time on the UI than integration of text analytics components I would be happy to show you the Solr config and let you try it out