Combining Solr and Elasticsearch to Improve Autosuggestion on Mobile Local Search. Toan Vinh Luu, PhD Senior Search Engineer local.

Transcription

1 Combining Solr and Elasticsearch to Improve Autosuggestion on Mobile Local Search Toan Vinh Luu, PhD Senior Search Engineer local.ch AG

2 In this talk Autosuggestion feature Autosuggestion architecture Evaluation

3 local.ch Local search engine in Switzerland (web, mobile) Each month: > 4 millions unique users > 8 millions queries on mobile (ios, android, ) Users search for: Services (e.g restaurant zurich ) Resident information (e.g peter meier ) Phone number (e.g ) Addresses, point of interest...

4 Why autosuggestion is important? User taps on the phone 8 times instead of 34 times to get to the result list when searching for Electric installation Wallisellen

5 What should we suggest to user?

6 Popular data suggestion

7 Popular queries suggestion mc donalds has less entries than muller but is queried >10x >2000 queries/month for cablecom which have only 1 entry

8 Query history suggestion 9% mobile queries are historical queries. 38% users search by a query in the past

9 Spellchecker suggestion > mistakes per month on mobile (9%)

10 Detail entry suggestion

11 Special information suggestion

12 Autosuggestion Architecture Autosuggest API/Search API SuggestData component Spellchecker component Popular query component Query history component Index Index Index Index Local.ch Database Popular query processor Index Query log

13 How do we process popular queries Popular is just not high frequency! User s language 4 languages are used in Switzerland. Fail if we suggest bäckerei for a French speaking user Location Fail if we suggest a hospital in Zurich for an user in Geneva Misspell Fail if we suggest zürich and züruch Unique users Fail if we suggest toan just because I searched my name thousands of times Blacklist Fail if we suggest f**k, pe**is

14 Popular query processor Preprocessing query log: Text normalization, stopword, blacklist, keep only queries return results A query log item in elasticsearch index { "q": "restaurant", "language": "de", "lon": , "lat": , "datetime": " :10:07, "user": eeaad0c09abc41676c1c

15 Find candidate popular queries for each language { "query" : {, "query_string" : { aggs" : { "q" : { "query" : "language:%s AND date:[%s TO %s] AND -q.untouched:/[0].*/ % (language, fromdate, todate) "terms" : { "field" : "q.untouched", "size" : TOP_POPULAR

16 Find number of unique users given a query { "query" : { "query_string" : { "query" : "q.untouched:%s AND date:[%s TO %s] % (query, fromdate, todate), "aggs": { "num_users": { "cardinality": { "field": "user"

17 Bounding box to limit popular queries given location % Popular query: Chuv (Centre Hospitalier Universitaire Vaudois)

18 Histogram of query chuv based on freq, longitude and latitude

19 46.52, , ,6.64

20 Percentiles aggregation to find min, max value of querying location "query" : { "match" : {"q" : {"query" : chuv, "aggs" : { "lat_outlier" : { "percentiles" : { "field" : "lat", "percents" : [5, 95], "lon_outlier" : { "percentiles" : { "field" : "lon", "percents" : [5, 95]

21 Popular query stored in Solr index { "q": "chuv", "lang": ["de,"fr, "en ], "users": 7435, "min_lat": , "max_lon": , "max_lat": , "min_lon": , "freq": 9524

22 Solr request to suggest popular query q:ch* lang:en users: [100 TO *] min_lat:[* TO " + user_lat + "] min_lon:[* TO " + user_lon + "] max_lat:[" + user_lat + " TO *] max_lon:[" + user_lon + " TO *] & sort=freq desc

23 Evaluation Several metrics are used to evaluate autosuggestion feature Number of typed characters to get to result list Average length of input: 10.0 chars Average length of clicked suggestion: 15.4 chars Number of clicks on suggested items Average rank of clicked item

24 Number of clicks on suggested items since new feature release Release date

25 2.5 Average rank of clicked item Release new query suggestion 0

26 Conclusion We can combine 2 search frameworks to bring better search experience to user: Solr is efficient for querying, faceting and caching Elasticsearch is efficient for big data aggregation and query log storing

27 Contact information Search team at local.ch We are hiring a search engineer! Contact: yannick.suter@localsearch.ch