Best Practices for World-Class Search

Size: px

Start display at page:

Download "Best Practices for World-Class Search"

Vincent McBride
5 years ago
Views:

1 Best Practices for World-Class Search MARY HOLSTEGE Distinguished Engineer, 4 June 2018 MARKLOGIC CORPORATION

2 SLIDE: 2 4 June 2018 MARKLOGIC CORPORATION

3 Search Application: Search for a Purpose To answer questions about the world To discover useful information To find relevant information so it can be acted upon To bring important information to someone's attention To partition data into useful chunks for analysis SLIDE: 3 4 June 2018 MARKLOGIC CORPORATION

4 Search Application: Search for a Purpose To answer questions about the world To discover useful information To find relevant information so it can be acted upon To bring important information to someone's attention To partition data into useful chunks for analysis SLIDE: 4 4 June 2018 MARKLOGIC CORPORATION

5 Oh, the humanities! Oh, the humanities! Great search needs great understanding of humans Linguistics Psychology Anthropology SLIDE: 5 4 June 2018 MARKLOGIC CORPORATION

CC at: rs3794604 Blocks some melanin. Often gives light colored eyes. GG at: rs7174027 Blocks some melanin. Often gives light colored eyes. CC at: rs4778241 Low Melanin.

6 CC at: rs Blocks some melanin. Often gives light colored eyes. GG at: rs Blocks some melanin. Often gives light colored eyes. CC at: rs Low Melanin. Basis for Gray, Blue, Green, or Yellow Eyes if no other pigmentation is present. CC at: rs Blocks some melanin. Often gives light colored eyes. CT at: rs Blue. CT at: rs Blue. CT at: rs Contrasting sphincter around pupil. AA at: rs Med Brown on Sphincter AA at: rs Weak Amber Gradient TT at: rs Penetrance modifier. Blue. GG at: rs Gray ring around outer edge TT at: rs Starburst (Collarette) SLIDE: 6 4 June 2018 MARKLOGIC CORPORATION

7 Search Application: Search for a Purpose To answer questions about the world To discover useful information To find relevant information so it can be acted upon To bring important information to someone's attention To partition data into useful chunks for analysis SLIDE: 7 4 June 2018 MARKLOGIC CORPORATION

8 Improve Search Application: Fitter for Purpose Improve answers to questions Improve discoverability of information Improve usefulness of information Improve relevance of information Improve ability to act on information Improve visibility of important information Improve ability to partition information SLIDE: 8 4 June 2018 MARKLOGIC CORPORATION

9 SLIDE: 9 4 June 2018 MARKLOGIC CORPORATION

10 The Search Tripod Data Query Response SLIDE: 10 4 June 2018 MARKLOGIC CORPORATION

11 Battleplan Understanding Analysis Continual improvement Pertinence Feedback Augmentation Interaction Use SLIDE: 11 4 June 2018 MARKLOGIC CORPORATION

12 Search for a Purpose: Improve answers to questions Understand question, understand answer Make answer pertinent to question Make answer useful and usable SLIDE: 12 4 June 2018 MARKLOGIC CORPORATION

13 Data

14 Data Analysis Augmentation Interaction Feedback Keywords Classification Data modeling Entity recognition Link relationships Quality rankings Metrics Metadata Collections Semantic markup Entity markup Quality Range dimensions Linked data Reference data "SEO" Document proxies Visualizations Clustering Linking Popularity Reviews/ratings Folk taxonomies Annotations Linking SLIDE: 14 4 June 2018 MARKLOGIC CORPORATION

15 SLIDE: 15 4 June 2018 MARKLOGIC CORPORATION

16 Semantic Markup With Linkage to Facts Augment data with RDF triples, ontology Create entity dictionary from SKOS ontology Analyze and augment data with entity markup SLIDE: 16 4 June 2018 MARKLOGIC CORPORATION

17 geo:cayman_islands a skos:concept; skos:inscheme geo:area; skos:preflabel "Cayman Islands"^^xsd:string; geo:agriculturalareanotes "Manual Estimation"^^xsd:string; geo:agriculturalareatotal 2.7; geo:agriculturalareaunit "1000 Ha"^^xsd:string; geo:agriculturalareayear "2009"^^xsd:int; geo:populationtotal 56.0; geo:populationunit "1000"^^xsd:string; geo:populationyear "2010"^^xsd:int;; geo:codedbpediaid "Cayman_Islands"^^xsd:string; geo:countryareatotal 26.4; geo:countryareaunit "1000 Ha"^^xsd:string; SLIDE: 17 4 June 2018 MARKLOGIC CORPORATION

18 <para> <ex:business id=" Partners</ex:business> has a controlling interest in several companies with accounts in the <geo:area id=" Islands</geo:area>, regulators have learned. Other offshore accounts in <geo:area id=" and <geo:area id=" have been tied to the chairman, Mr. Q. A spokesperson for Mr. Q declined to comment. </para> <para> Investigations continue. </para> SLIDE: 18 4 June 2018 MARKLOGIC CORPORATION

19 SLIDE: 19 4 June 2018 MARKLOGIC CORPORATION

20 SLIDE: 20 4 June 2018 MARKLOGIC CORPORATION

21 SLIDE: 21 4 June 2018 MARKLOGIC CORPORATION

22 SLIDE: 22 4 June 2018 MARKLOGIC CORPORATION

23 SLIDE: 23 4 June 2018 MARKLOGIC CORPORATION

24 SLIDE: 24 4 June 2018 MARKLOGIC CORPORATION

25 SLIDE: 25 4 June 2018 MARKLOGIC CORPORATION

26 Query

27 Query Analysis Augmentation Interaction Feedback Canned queries FAQs Query patterns Entity recognition Natural(istic) language Metrics User interests Synonyms Related terms Entity queries Disambiguation Semantic queries Parsed string Facets Sliders Timelines Maps Breadcrumbs Scatter/gather User behaviors User interests Common queries SLIDE: 27 4 June 2018 MARKLOGIC CORPORATION

28 SLIDE: 28 4 June 2018 MARKLOGIC CORPORATION

29 Analyze and Transform Query Create special dictionaries for query analysis Normalize query words, strip stopwords Replace with tagged forms Match tagged query to query patterns Parse tagged query with appropriate bindings SLIDE: 29 4 June 2018 MARKLOGIC CORPORATION

30 // Input query "Does XYZW Partners have business in the Cayman Islands?" // Normalized "XYZW Partners has business Cayman Islands" // Analyzed and tagged query string (business:221932) (control:hold OR get:receive OR suffer:have) (person:clientele OR organization:enterprise OR event:commerce) (location:cayman_islands) SLIDE: 30 4 June 2018 MARKLOGIC CORPORATION

31 // Matched to query pattern "business:* control:hold organization:*" to disambiguate "business: control:hold organization:enterprise location:cayman_islands" // Parsed with appropriate bindings cts:and-query(( cts:field-value-query("id", " cts:or-query(("control","hold","govern","run","have"),"synonym"), cts:or-query(("organization","business","enterprise","insitution"),"synonym"), cts:or-query(( cts:field-value-query("id"," "George Town", "West Bay", "Bodden Town", "East End", "North Side", "West End" )) )) SLIDE: 31 4 June 2018 MARKLOGIC CORPORATION

32 SLIDE: 32 4 June 2018 MARKLOGIC CORPORATION

34 SLIDE: 34 4 June 2018 MARKLOGIC CORPORATION

35 SLIDE: 35 4 June 2018 MARKLOGIC CORPORATION

36 SLIDE: 36 4 June 2018 MARKLOGIC CORPORATION

37 SLIDE: 37 4 June 2018 MARKLOGIC CORPORATION

38 SLIDE: 38 4 June 2018 MARKLOGIC CORPORATION

39 Response

40 Response Analysis Augmentation Interaction Feedback Clusters Links Disambiguation Re-ranking Best bets Metrics Document proxies Result clusters Facts Info boxes Related queries Relevance tuning Navigational cues Annotated TOCs Scatter/gather Linked data views Facets Sliders Timelines Maps User behavior SLIDE: 40 4 June 2018 MARKLOGIC CORPORATION

41 SLIDE: 41 4 June 2018 MARKLOGIC CORPORATION

42 Infobox Extract concept IDs from query string Select best entity for results Query for facts related to ID SLIDE: 42 4 June 2018 MARKLOGIC CORPORATION

43 // Input query "Does XYZW Partners have business in the Cayman Islands?" // Extracted entity IDs " " SLIDE: 43 4 June 2018 MARKLOGIC CORPORATION

44 // Facts for chosen ID (predicate + object) "Manual Estimation"^^xsd:string "1000 Ha"^^xsd:string "2009"^^xsd:int "1000"^^xsd:string "2010"^^xsd:int "Cayman_Islands"^^xsd:string "1000 Ha"^^xsd:string SLIDE: 44 4 June 2018 MARKLOGIC CORPORATION

45 SLIDE: 45 4 June 2018 MARKLOGIC CORPORATION

46 SLIDE: 46 4 June 2018 MARKLOGIC CORPORATION

47 SLIDE: 47 4 June 2018 MARKLOGIC CORPORATION

48 SLIDE: 48 4 June 2018 MARKLOGIC CORPORATION

49 SLIDE: 49 4 June 2018 MARKLOGIC CORPORATION

50 SUMMARY Building Great Search Applications Humans Understand user purposes Answers Discovery Analysis Action Humanities Search tripod Strengthen all legs Data Query Response Battleplan Analysis Understanding Augmentation Pertinence Interaction Use Feedback Continual improvement SLIDE: 50 4 June 2018 MARKLOGIC CORPORATION

51 Questions?

52 Appendix

53 SLIDE: 53 4 June 2018 MARKLOGIC CORPORATION

54 SLIDE: 54 4 June 2018 MARKLOGIC CORPORATION

55 SLIDE: 55 4 June 2018 MARKLOGIC CORPORATION

56 SLIDE: 56 4 June 2018 MARKLOGIC CORPORATION

57 SLIDE: 57 4 June 2018 MARKLOGIC CORPORATION

58 SLIDE: 58 4 June 2018 MARKLOGIC CORPORATION

59 SLIDE: 59 4 June 2018 MARKLOGIC CORPORATION

61 SLIDE: 61 4 June 2018 MARKLOGIC CORPORATION

62 SLIDE: 62 4 June 2018 MARKLOGIC CORPORATION

63 Data

64 Goals Improve searchability - Precision: specific simplified scoping - Recall: related information explicit - Ranking: explicit context, quality Improve usability - Facts from text - Summarization, TOCs, other proxies Improve discoverability - Facets, classifications SLIDE: 64 4 June 2018 MARKLOGIC CORPORATION

65 Analysis Keywords and classification Modeling Entity recognition Link analysis Quality rankings SLIDE: 65 4 June 2018 MARKLOGIC CORPORATION

66 Augmentation Metadata Collections Semantic markup Entity markup Quality Range dimensions Linked data Contextual information Reference data SLIDE: 66 4 June 2018 MARKLOGIC CORPORATION

67 Interaction Document proxies - KWIC snippets - Summaries - TileBar - Color lines Relationships - Clustering - Link visualizations SLIDE: 67 4 June 2018 MARKLOGIC CORPORATION

68 Feedback Popularity, reviews, ratings => quality adjustments, range adjustments Folk taxonomies => classification Annotations => related information Additional linked data => context SLIDE: 68 4 June 2018 MARKLOGIC CORPORATION

69 Selected Techniques Expert indexers (keywords, classifications) - Primed/augmented with classification engines Semantic markup - Authoring toolchain - Harmonization/projection Entity markup - NER integrations - Ontology-driven or query-based entity extraction Linked data, reference data SLIDE: 69 4 June 2018 MARKLOGIC CORPORATION

70 Queries

71 Goals Improve expressibility - Simpler expression => better query Improve search effectiveness - Precision, recall, ranking - Focus in, focus out SLIDE: 71 4 June 2018 MARKLOGIC CORPORATION

72 Analysis Canned queries, query patterns, FAQs Identify known entities Natural(istic) language interpretation SLIDE: 72 4 June 2018 MARKLOGIC CORPORATION

73 Augmentation Boost queries - User interests, related terms Synonym, thesaurus expansion Canned query recognition Entity recognition Query expansion Contextual augmentation Relevance tweaks SLIDE: 73 4 June 2018 MARKLOGIC CORPORATION

74 Interaction Natural(istic) language input Query builders: facets, timelines, maps, links, forms More like this, less like that Result zooming/slicing Breadcrumbs SLIDE: 74 4 June 2018 MARKLOGIC CORPORATION

75 Feedback User behavior => disambiguation User interests => query augmentation Common queries => canned queries and results SLIDE: 75 4 June 2018 MARKLOGIC CORPORATION

76 Selected Techniques Query string parsing/pre-processing - NLP - Regex analysis - NER, ontology-driven entity extraction Augmentation with profiles, thesauri, etc. Reverse query against FAQs Range/geospatial indexes SLIDE: 76 4 June 2018 MARKLOGIC CORPORATION

77 Response

78 Goals Improve usefulness - Immediate answers - Navigation - Context Improve conversation - What happens next? SLIDE: 78 4 June 2018 MARKLOGIC CORPORATION

79 Analysis Clustering Disambiguation Re-ranking Best bets SLIDE: 79 4 June 2018 MARKLOGIC CORPORATION

80 Augmentation Document proxies and context Results clustering Specific facts Info boxes Related queries Relevance tuning Navigational cues (within/across documents) Exploration interfaces SLIDE: 80 4 June 2018 MARKLOGIC CORPORATION

81 Interaction Results cluster navigation Linked data views Facet/timeline/map/sliders Annotated TOCs Counts/coloring Immediate action affordances SLIDE: 81 4 June 2018 MARKLOGIC CORPORATION

82 Feedback User actions => adjust rankings, adjust interests Hide/expose => streamlined personal interface SLIDE: 82 4 June 2018 MARKLOGIC CORPORATION

83 Selected Techniques Snippeting, summarization Clustering Re-ranking, LTR Query entities => result entities => related information Range/geospatial indexes => display widgets Linked data presentations SLIDE: 83 4 June 2018 MARKLOGIC CORPORATION

AutoFocus, an Open Source Facet-Driven Enterprise Search Solution

AutoFocus, an Open Source Facet-Driven Enterprise Search Solution ISKO UK Event, November 5, 2007 RANGANATHAN REVISITED: FACETS FOR THE FUTURE presentation by Jeroen Wester, CTO Aduna key facts Open source