HOW TO BUILD AN AWESOME SEARCH APP

Transcription

1 HOW TO BUILD AN AWESOME SEARCH APP Stu McLean, Ph.D., Principal Consultant, Public Sector - Civilian, MarkLogic Ganesh Vaideeswaran, Senior Director, Development, MarkLogic

2 Agenda Components of a search application Deep dive SLIDE: 2

3 SLIDE: 3

4 SLIDE: 5

5 SLIDE: 6

6 SLIDE: 7

7 SLIDE: 8

8 SLIDE: 9

9 SLIDE: 10

10 SLIDE: 11

11 SLIDE: 12

12 SLIDE: 13

13 SLIDE: 14

14 SLIDE: 15

15 SLIDE: 16

16 360 VIEW WHAT MARKLOGIC PROVIDES Awesome Search for the Enterprise SLIDE: 17

17 NBCUniversal SNL40 App Smart Content Enriched and targeted content Constantly tuned recommendations > 5,550 sketches SLIDE: 19

18 NBCUniversal SNL40 App Era Includes Part of Segment The Lawrence Welk Show Acted in Played Talent Kristen Wiig Acted in Episode 4 Anne Hathaway and Killers Part of Aired on Date 10/4/08 Character Maharelle Sister Has Season 34 Characteristic Tiny hands Smart Content: Enriched and targeted content Constantly tuned recommendations SLIDE: 20

19 Fairfax County Mapping Crime Data Data Variety 500,000 GIS data points Service Calls Source Variety 800 GIS layers Structured & Unstructured SLIDE: 21

20 Mitchell1 Streamlined Auto Repair Information REPAIR INFO EXPERT ADVICE MITCHELL1 PRODEMAND Structured & Unstructured Data Sophisticated Search Multi-device Delivery REPAIR ORDERS SLIDE: 23

21 Needs for an Awesome Search App DATA QUERY PLATFORM Structured & Unstructured Multi-Model approach Relevance Snippets Highlight Alerts Transactions Security Context Auto Complete Real-time Presentation Spell Suggest Languages Synonyms Configurability Development Platform SLIDE: 25

22 Aboutness The Key to Successful Search What is this document about? What is this submitted query about? What is the user s activity about? What are the responses about? SLIDE: 26

23 Implementing Aboutness in MarkLogic Data What is this document about? SLIDE: 27

24 Implementing Aboutness in MarkLogic Data SLIDE: 28

25 Implementing Aboutness in MarkLogic Data Query What is this query about? SLIDE: 29

26 Implementing Aboutness in MarkLogic Data Query SLIDE: 30

27 Implementing Aboutness in MarkLogic Data Query Context What is the user s activity about? SLIDE: 31

28 Implementing Aboutness in MarkLogic Data Query Context SLIDE: 32

29 Implementing Aboutness in MarkLogic Data Query Context Results What are the responses about? SLIDE: 33

30 Implementing Aboutness in MarkLogic Data Query Context Results SLIDE: 34

31 Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure Query Query as a Dialogue Context Building an Information Resource Results Relevant Response SLIDE: 35

32 Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure Query Query as a Dialogue Context Building an Information Resource Results Relevant Response SLIDE: 36

33 Domain Knowledge Tools for Establishing Aboutness Taxonomies vocabulary of the organization Ontologies Tie your documents to relevant data and documents Thesaurus Organizational units, people, products Public vocabularies Geographic, DBPedia SLIDE: 37

34 Taxonomies Terms to look for in your documents SLIDE: 38

35 Categorization Finding Aboutness Taxonomies and Reverse Queries Public taxonomies Getty art and architecture World Bank humanitarian projects Library of Congress subject headings National Library of Medicine medical headings Domain taxonomies Best Buy Google SLIDE: 39

36 Categorizing Your Documents Where does Aboutness reside? title: abstract: glossary: Body Expert searchers compose carefully crafted searches with domain knowledge The rest of us hope for the best SLIDE: 40

37 Implementing Categorization Turn a Taxonomy Entry into a Query <cts:or-query xmlns:cts=" <cts:near-query distance="5"> <cts:near-query distance="5"> <cts:element-word-query weight="10"> <cts:element xmlns:_1=" :title</cts:element> <cts:text xml:lang="en">non-government</cts:text> <cts:option>case-insensitive</cts:option> <cts:option>punctuation-insensitive</cts:option> <cts:option>stemmed</cts:option> </cts:element-word-query> <cts:element-word-query weight="10"> <cts:element xmlns:_1=" :title</cts:element> <cts:text xml:lang="en">bond</cts:text> <cts:option>case-insensitive</cts:option> <cts:option>punctuation-insensitive</cts:option> <cts:option>stemmed</cts:option> </cts:element-word-query> </cts:near-query> SLIDE: 41

38 Reverse Query Categorization Reverse Queries SLIDE: 42

39 Reverse Query Categorization Reverse Queries Ingest mydoc.xml 1 MB document Save as /tmp/mydoc.xml in collection tmp SLIDE: 43

40 Reverse Query Categorization Reverse Queries Ingest mydoc.xml 1 MB document Save as /tmp/mydoc.xml in collection tmp In-memory copy of abridged document (10% = 100K) SLIDE: 44

41 Reverse Query Categorization Reverse Queries Ingest mydoc.xml 1 MB document Save as /tmp/mydoc.xml in collection tmp In-memory copy of abridged document (10% = 100K) SLIDE: 45

42 Reverse Query Categorization Reverse Queries cts:search(fn:doc(), cts:and-query(( cts:collection-query("topicqueries"), cts:reverse-query($reduceddoc) )), "unfiltered" ) Ingest mydoc.xml 1 MB document Save as /tmp/mydoc.xml in collection tmp In-memory copy of abridged document (10% = 100K) SLIDE: 46

43 Reverse Query Categorization Scoring Reverse Queries /tmp/mydoc.xml SLIDE: 47

44 Reverse Query Categorization Scoring Reverse Queries /tmp/mydoc.xml for $q in $reversesearchresults let $score := cts:fitness( cts:search( fn:doc(), cts:and-query(( cts:document-query($myuri), cts:query($q/query) )), ("score-logtf","unfiltered") ) SLIDE: 48

45 Envelope Pattern Know Your Data Homogenized Variant Data Types Remain Schema-Agnostic, Load As-Is Annotate Your Data SLIDE: 49

46 Building the Envelope Metadata Ingested Content Category Annotation SLIDE: 50

47 Envelope Know Your Data Standardize ingested documents of multiple types Common names may be different (e.g. Title, Heading, Subject) Normalize your searchable elements into a common format Multiple versions of the same document (e.g. Translations) Preserve artifacts Indexed search on metadata (facets) Full-text search on content SLIDE: 51

48 Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure Query Query as a Dialogue Context Building an Information Resource Results Relevant Response SLIDE: 52

49 Establish Aboutness Through Dialogue What is the Query About? Type-ahead Match Lexicons Match Search History (user, global) Background Search Phrase Identification Object Matching Boost Queries SLIDE: 53

50 Establishing Aboutness Through Dialogue (Cont d) What is the Query About? Correction and Confirmation Spell Check Thesaurus Expansion Semantic Expansion Drill Down Follow Up Search Box Facets Constraints on the Fly SLIDE: 54

51 Extending Suggest REST API extension Search multiple lexicons for each request Type-ahead must be fast Resist temptation to put multiple suggest calls in the app declare function get( $context as map:map, $params as map:map ) as document-node()? { let $suggestlist := ("person", "country", "organization", "project", "region", "topic") let $partial := xdmp:url-decode(map:get($params, "partial-q")) let $suggestions := map:new(( for $s in $suggestlist let $term := fn:concat($partial,"*") return map:entry($s,search:suggest( $term, $options, if ($s eq "person") then 20 else 3 ))) return document {xdmp:to-json($suggestions)} };

52 Query Expansion Background Search Finding phrases when there are no quotes Approach 1 SPARQL - Match phrases in metadata, taxonomies Put the matched text into a weighted custom constraint (e.g. phrase: ) OR the phrase: constraint with the full query Approach 2 Build NEAR queries using pairs of nearby terms OR the phrase and boost the weight on the NEAR query if matched SLIDE: 56

53 Example Constraints on the Fly Selected documents share common elements Elements not known in advance but can be used as in-app facets Set as constraints in the submitted options SLIDE: 57

54 Example Selecting from Feature Fields SLIDE: 58

55 Example Selecting from Feature Fields SLIDE: 59

56 Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure Query Query as a Dialogue Context Building an Information Resource Results Relevant Response SLIDE: 60

57 Aboutness The Context Behind The Query Document LexisNexis WestLaw Medline USPTO SLIDE: 61

58 Aboutness The Context Behind The Query Document Facts LexisNexis WestLaw Medline USPTO WikiPedia CIA World Factbook SLIDE: 62

59 Aboutness The Context Behind The Query Document Facts Information LexisNexis WestLaw Medline USPTO WikiPedia CIA World Factbook Google Bing SLIDE: 63

60 Reference Data Link or Load External Data Real Time Read SLIDE: 64

61 Reference Data Link or Load External Data Ingest Retrieval Embedded Data SLIDE: 65

62 Reference Data Link or Load External Data Copied Reference Data Embedded Links additional-query SPARQL SLIDE: 66

63 Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure Query Query as a Dialogue Context Building an Information Resource Results Relevant Response SLIDE: 67

64 Planning Your Data For Results When to index metadata Facets Sortable Lexicon dropdowns Document joins When not to index Wildcarding Search:search constraints SLIDE: 68

65 Results Focus On Relevance Boost the Document: Quality Boost the Element: Term Weighting Boost the Term: Query Weighting SLIDE: 69

66 Boost the Document Set quality xdmp:document-insert xdmp:document-load xdmp:document-insert( "/example.xml", <a>aaa</a>, xdmp:default-permissions(), xdmp:default-collections(), 10 ) xdmp:document-load( " <options xmlns="xdmp:document-load"> <uri>/documents/myfile.xml</uri> <repair>none</repair> <permissions>{xdmp:default-permissions()}</permissions> < <format>xml</format> <quality>10</quality> </options>)

67 Boost the Element In the database configuration Select Word Query from the menu Under the includes tab add your ranked element with its weight SLIDE: 71

68 Boost the Term With no Query Weighting the MarkLogic Companies document appears first SLIDE: 72

69 The Boost-Query A conditional secondary query that increases the weight of results matching both criteria The boost-query pushes the MarkLogic World document to the top SLIDE: 73

70 Aboutness The Key to Successful Search What is this document about? What is this query about? What is the user s activity about? What are the responses about? SLIDE: 74

71 Rendering Awesome Apps SLIDE: 75

72 Q&A SLIDE: 76