Full-Text Search. Explained. Philipp

Size: px

Start display at page:

Download "Full-Text Search. Explained. Philipp"

Gerard Bates
5 years ago
Views:

1 Full-Text Search Explained Philipp

5 Infrastructure Developer Advocate

6 ViennaDB Papers We Love Vienna

7 Who uses Databases?

8 Who uses Search?

9 Database vs Full-Text Search

11 But I can do... SELECT * FROM my_table WHERE my_text LIKE %my_term%

13 1. Performance B-Tree

17 2. Features Fuzziness, synonyms, scoring,...

18 Store

19 Indexing Remove formating

20 Indexing Tokenize

21 Indexing Stop words

22 Indexing Stemming

23 Indexing Synonyms

25 Apache Lucene Elasticsearch

29 --- version: '2' services: kibana: image: docker.elastic.co/kibana/kibana:5.2.1 links: - elasticsearch ports: :5601 elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:5.2.1 cap_add: - IPC_LOCK volumes: - esdata1:/usr/share/elasticsearch/data ports: :9200 volumes: esdata1: driver: local

31 Example These are not the droids you are looking for.

32 html_strip Char Filter These are not the droids you are looking for.

33 standard Tokenizer These are not the droids you are looking for

34 lowercase Token Filter these are not the droids you are looking for

35 stop Token Filter droids you looking

36 snowball Token Filter droid you look

37 GET /_analyze "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "snowball" ], "text": "These are not the droids you are looking for."

38 "tokens": [ "token": "droid", "start_offset": 27, "end_offset": 33, "type": "<ALPHANUM>", "position": 4, "token": "you", "start_offset": 34, "end_offset": 37, "type": "<ALPHANUM>", "position": 5,... ]

39 Stop Words a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will, with

40 Russian Это не те дроиды, которых ты ищешь

41 Russian эт те дроид котор ищеш

42 GET /_analyze "analyzer": "russian", "text": "Это не те дроиды, которых ты ищешь"

43 "tokens": [ "token": "эт", "start_offset": 0, "end_offset": 3, "type": "<ALPHANUM>", "position": 0, "token": "те", "start_offset": 7, "end_offset": 9, "type": "<ALPHANUM>", "position": 2,... ]

44 German, please! Das sind nicht die Droiden, nach denen du suchst.

45 German, please! droid den such

46 GET /_analyze "analyzer": "german", "text": "Das sind nicht die Droiden, nach denen du suchst."

47 German with the English Analyzer das sind nicht die droiden nach denen du suchst

48 German Stop Words analysis/common/src/resources/org/apache/lucene/analysis/ snowball/german_stop.txt

49 Detecting Languages elasticsearch-ingest-langdetect

50 Another Example Obi-Wan hat mir nie erzählt, was mit meinem Vater geschehen ist.

51 Another Example obi wan nie erzahlt vat gescheh

52 Another Example Nein. Ich bin dein Vater

53 Another Example nein vat

54 Languages in 5.0 arabic, armenian, basque, brazilian, bulgarian, catalan, cjk, czech, danish, dutch, english, finnish, french, galician, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, thai

55 Ukrainian Lucene 6.2 Elasticsearch 5.1

56 Another Example Obi-Wan never told you what happened to your father.

57 Another Example obi wan never told you what happen your father

58 Another Example no. I am your father.

59 Another Example i am your father

60 Language Rules English: Philipp's philipp French: l'église eglis German: äußerst ausserst

61 phonetic Token Filter Plugin

62 GET /_analyze "tokenizer": "standard", "filter": [ "type": "phonetic", "encoder": "metaphone", "replace": false ], "text": "Obi-Wan"

63 "tokens": [ "token": "OB", "start_offset": 0, "end_offset": 3, "type": "<ALPHANUM>", "position": 0, "token": "Obi", "start_offset": 0, "end_offset": 3, "type": "<ALPHANUM>", "position": 0, "token": "WN", "start_offset": 4, "end_offset": 7, "type": "<ALPHANUM>", "position": 1, "token": "Wan", "start_offset": 4, "end_offset": 7, "type": "<ALPHANUM>", "position": 1 ]

64 Elasticsearch Index, typ, mapping

65 Multi-Language Support One language per index One language per type One language per field

66 PUT /starwars "settings": "analysis": "filter": "my_synonym_filter": "type": "synonym", "synonyms": [ "droid,machine", "father,dad" ],

67 , "analyzer": "my_analyzer": "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "snowball", "my_synonym_filter" ]

68 "mappings": "quotes": "properties": "quote": "type": "text", "analyzer": "my_analyzer"

69 GET /starwars/_mapping GET /starwars/_settings

70 PUT /starwars/quotes/1 "quote": "These are not the droids you are looking for." PUT /starwars/quotes/2 "quote": "Obi-Wan never told you what happened to your father." PUT /starwars/quotes/3 "quote": "no. I am your father."

71 GET /starwars/quotes/1 GET /starwars/quotes/1/_source

72 ID 1 ID 2 ID 3 am 0 0 1[2] droid 1[4] 0 0 father 0 1[9] 1[4] happen 0 1[6] 0 i 0 0 1[1] look 1[7] 0 0 never 0 1[2] 0 obi 0 1[0] 0 told 0 1[3] 0 wan 0 1[1] 0 what 0 1[5] 0 you 1[5] 1[4] 0 your 0 1[8] 1[3] Inverted Index

73 Search

74 POST /starwars/_search "query": "match_all":

75 GET vs POST

76 "took": 1, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 3, "max_score": 1, "hits": [ "_index": "starwars", "_type": "my_type", "_id": "2", "_score": 1, "_source": "quote": "Obi-Wan never told you what happened to your father.",...

77 POST /starwars/_search "query": "match": "quote": "droid"

78 "took": 2, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "1", "_score": , "_source": "quote": "These are not the droids you are looking for." ]

79 POST /starwars/_search "query": "match": "quote": "dad"

80 ... "hits": "total": 2, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father.", "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father." ]

81 POST /starwars/_search "query": "match_phrase": "quote": "I am your father"

82 "took": 3, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father." ]

83 POST /starwars/_search "query": "match_phrase": "quote": "I am not your father"

84 "took": 15, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 0, "max_score": null, "hits": []

85 POST /starwars/_search "query": "match_phrase": "quote": "query": "I am not your father", "slop": 1

86 "took": 5, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father." ]

87 POST /starwars/_search "query": "match": "quote": "query": "van", "fuzziness": "AUTO"

88 "took": 14, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father." ]

89 POST /starwars/_search "query": "match": "quote": "query": "ovi-van", "fuzziness": 1

90 "took": 109, "timed_out": false, "_shards": "total": 5, "successful": 5, "failed": 0, "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father." ]

91 FuzzyQuery History Before: Brute force Now: Levenshtein Automaton

92 SELECT * FROM starwars WHERE quote LIKE "?an" OR quote LIKE "V?n" OR quote LIKE "Va?"

93 Scoring

94 Term Frequency / Inverse Document Frequency (TF/IDF) Search one term

95 BM25 Default in Elasticsearch 5.0

96 Term Frequency

98 Inverse Document Frequency

100 Field-Length Norm

101 Putting it Together score(q,d) = querynorm(q) coord(q,d) ( tf(t in d) idf(t)² t.getboost() norm(t,d) ) (t in q)

102 POST /starwars/_search?explain "query": "match": "quote": "father"

103 ... "_explanation": "value": , "description": "weight(synonym(quote:dad quote:father) in 0) [PerFieldSimilarity], result of:", "details": [ "value": , "description": "score(doc=0,freq=2.0 = termfreq=2.0\n), product of:", "details": [ "value": , "description": "idf(docfreq=1, doccount=1)", "details": [], "value": , "description": "tfnorm, computed from:", "details": [ "value": 2, "description": "termfreq=2.0", "details": [],...

104 Score : i am your father : obi wan never told you what happen your father

105 Vector Space Model Search multiple terms

106 Score each term Vectorize Calculate angle

107 Search your father

108

109 Function Score Script, weight, random, field value, decay (geo or date)

110 POST /starwars/_search "query": "function_score": "query": "match": "quote": "father", "random_score":

111 More Features

112 POST /starwars/_search "query": "match": "quote": "father", "highlight": "pre_tags": [ "<tag>" ], "post_tags": [ "</tag>" ], "fields": "quote":

113 ... "hits": [ "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father.", "highlight": "quote": [ "no. I am your <tag>father</tag>." ],...

114 Boolean Queries must must_not should filter

115 POST /starwars/_search "query": "bool": "must": "match": "quote": "query": "father", "should": [ "match": "quote": "query": "your", "match": "quote": "query": "obi" ]

116 ... "hits": "total": 2, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father.", "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father." ]

117 POST /starwars/_search "query": "bool": "filter": "match": "quote": "query": "father", "should": [ "match": "quote": "query": "your", "match": "quote": "query": "obi" ]

118 ... "hits": "total": 2, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father.", "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father." ]

119 POST /starwars/_search "query": "bool": "must": "match": "quote": "query": "father", "should": [ "match": "quote": "query": "your", "match": "quote": "query": "obi", "match": "quote": "query": "droid" ], "minimum_number_should_match": 2

120 ... "hits": "total": 1, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father." ]

121 Boosting Default 1 greater or smaller

122 POST /starwars/_search "query": "bool": "must": "match": "quote": "query": "father", "should": [ "match": "quote": "query": "your", "match": "quote": "query": "obi", "boost": 3 ]

123 ... "hits": "total": 2, "max_score": , "hits": [ "_index": "starwars", "_type": "quotes", "_id": "2", "_score": , "_source": "quote": "Obi-Wan never told you what happened to your father.", "_index": "starwars", "_type": "quotes", "_id": "3", "_score": , "_source": "quote": "no. I am your father." ]

124 Suggestion Suggest a similar text _search end point _suggest deprecated

125 POST /starwars/_search "query": "match": "quote": "fath", "suggest": "my_suggestion" : "text" : "drui", "term" : "field" : "quote"

126 ... "hits": "total": 0, "max_score": null, "hits": [], "suggest": "my_suggestion": [ "text": "drui", "offset": 0, "length": 4, "options": [ "text": "droid", "score": 0.5, "freq": 1 ] ]

127 NGram Partial matches Trigram Edge Gram

128 GET /_analyze "char_filter": [ "html_strip" ], "tokenizer": "type": "ngram", "min_gram": "3", "max_gram": "3", "token_chars": [ "letter" ], "filter": [ "lowercase" ], "text": "These are not the droids you are looking for."

129 "tokens": [ "token": "the", "start_offset": 0, "end_offset": 3, "type": "word", "position": 0, "token": "hes", "start_offset": 1, "end_offset": 4, "type": "word", "position": 1, "token": "ese", "start_offset": 2, "end_offset": 5, "type": "word", "position": 2, "token": "are", "start_offset": 6, "end_offset": 9, "type": "word", "position": 3,...

130 GET /_analyze "char_filter": [ "html_strip" ], "tokenizer": "type": "edge_ngram", "min_gram": "1", "max_gram": "3", "token_chars": [ "letter" ], "filter": [ "lowercase" ], "text": "These are not the droids you are looking for."

131 "tokens": [ "token": "t", "start_offset": 0, "end_offset": 1, "type": "word", "position": 0, "token": "th", "start_offset": 0, "end_offset": 2, "type": "word", "position": 1, "token": "the", "start_offset": 0, "end_offset": 3, "type": "word", "position": 2, "token": "a", "start_offset": 6, "end_offset": 7, "type": "word", "position": 3, "token": "ar", "start_offset": 6, "end_offset": 8, "type": "word", "position": 4,...

132 Conclusion

133 Indexing Formatting Tokenize Lowercase, Stop Words, Stemming Synonyms

134 Scoring Term Frequency Inverse Document Frequency Field-Length Norm Vector Space Model

135 Querying Match: Phrase, Slop, Fuzziness Boolean Suggestion NGram

136 Danke! Questions? Philipp PS: Stickers

137 Image Credits Schnitzel Architecture Conchita

2. bizhub Remote Access Function Support List

2. bizhub Remote Access Function Support List MFP Function Support List for bizhub Remote Access Basic s MFP model Firmware v C754/ C654/ C754e/ C654e 754/ 654 C554/ C454/ C364/ C284/ C224 (*5) A1610Y