Is Elasticsearch the Answer?

Transcription

1

2 High-Performance Big-Data Computation Solution Is Elasticsearch the Answer? Yoav Melamed

3 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

4 The need

5 The need

6 The problem Run ad hoc calculations based on large amount of data Count unique users related to a specific campaign in conjunction with geographic/demographic data limited by date range

7 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

8 Relational DBMS Hadoop key-value store Document store Search engine Optional Solutions

9 DB Classification

10 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

11 What is Elasticsearch Open source search engine based on Lucene Real time data and real time analytics Document oriented Inverted index Full text search Distributed High availability

12 What is Elasticsearch

13 Elasticsearch Caches Elasticsearch tries to keep as much as possible in memory: Page cache - term dictionaries, posting lists etc. Field cache - field values that are used for aggregation, sorting or scripting Filter cache - filters can be cached as bitmaps

14 Elasticsearch Document { "_index": "sample", "_type": "user", "_id": "0c31644ad41e32c819be29ba16e14300", "_version": 4, "_score": 1, "_source": { "events": [ { "event_time": " ", "segments": [ { "segment": "female" },{ "segment": "US_AZ" } ] } ] } }

15 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

16 Not Out Of The Box

17 Search

18 OutOfMemory Crash Upload large fields into memory Upload too many fields into memory Waiting too long before flushing New documents Indexes growth Elasticsearch does not look before it leaps It assumes you have provided it with enough memory

19 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

20 Shard Limitations and Capabilities

21 Index Limitations and Capabilities

22 Query Limitations and Capabilities

23 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

24 Index Structure

25 Index Structure

26 Routing

27 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

28 Nested { "_index": "sample", "_type": "user", "_id": "0c31644ad41e32c819be29ba16e14300", "_version": 4, "_score": 1, "_source": { "events": [ { "event_time": " ", "segments": [ { "segment": "female" },{ "segment": "US_AZ" } ] } ] } }

29 Parent-Child { "_index": "sample", "_type": "user", "_id": "0c31644ad41e32c819be29ba16e14300", "_version": 3, "_score": 1, } { "_index": "sample", "_type": "event", "_id": "0c31644ad41e32c819cc29ba16abd901", _parent : "0c31644ad41e32c819be29ba16e14300", "_version": 4, "_score": 1, "_source": { "event_time": " ", "segments": [ { "segment": "female" },{ "segment": "US_AZ" } ] } }

30 Nested Vs. Parent-Child Nested Parent-Child Store Same Lucene block Routed to the same shard Search Index Fast read/query performance Updating a single field forces ES to reindex the entire nested document Slightly less performance on read/query than nested Updating a child does not affect the parent or any other children

31 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

32 Indexing Method

33 Navigation The need Optional solutions What is Elasticsearch Not out of the box Shard limitations and capabilities Index structure Document structure Indexing More to consider

34 More Improvement Actions Query method: Filtering Field cache Aggregation Doc values String mapping / Global ordinals

35 Is Elasticsearch the Answer?