Elasticsearch: Past, Present, & Future. Adrien Grand Software Engineer - Elasticsearch

Transcription

1 Elasticsearch: Past, Present, & Future Adrien Grand Software Engineer - Elasticsearch

2 Elasticsearch October 2016

3 Elasticsearch 5.0 Better at Numbers 3 Safe Simple Things Should Be Simple

4 Elasticsearch 5.x: Feature-rich 5.6 Released September 11, 2017

5 Elasticsearch 5.x Still ^ Keyword normalization Unified highlighter Field collapse Multi-word synonyms+proximity Cancellable searches Parallel scroll & reindex 5

6 Elasticsearch 5.x Still ^ ^ and for logs and for numbers and for geo and... Numeric & date range fields Automatic optimizations for range searches Massive aggregations with partitioning Faster geo-distance sorting Faster geo-ip lookups 6

7 Ever-increasing Scale More clusters / resource isolation Easier to manage Easier to upgrade Reduce potential outages Need to query across clusters 7

8 Tribe Node Cluster UK Tribe Node Cluster US Merged Cluster State t1 Node Client t2 Node Client Master Nodes Master Nodes Data Node Data Node Data Node Data Node Data Node Kibana Data Node 8

9 Cross-Cluster Search Cluster UK Optional dedicated cross-cluster search cluster Cluster US Master/Data Node Master/Data Node Master Nodes Master Nodes Data Node Data Node Data Node Data Node Data Node Kibana Data Node 9

10 Elasticsearch 6.0+ Some insights into the future

11 What are the major pain points? Disk usage Resiliency & recovery Major version upgrades 11 Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5

12 What are the major pain points? Disk usage Resiliency & Recovery Major version upgrades 12 Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5

13 Doc Values - Sparse Data (5.x) Segment 1 ID fname lname 1 Simon Willnauer 2 Shay Banon 3 Tanya Bragin Segment 2 ID fname lname mi state city 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Bill Swerski Null Null Chicago Merged Segment 3 Docs fname lname mi state city 1 Simon Willnauer Null Null Null 2 Shay Banon Null Null Null 3 Tanya Bragin Null Null Null 4 Steve Kearns Null Null Boston 5 George Burdell P GA Null 6 Bill Swerski Null Null Baz 13

14 Disk Space Improvements Sparse value support coming in 6.0 _all field off by default with special query handler More efficient configurations in ingest products Future: Rollups 14 Tony Weman / CC-BY 2.5

15 Sample Metricbeat Dataset Apples-to-apples index size improvements 100 OOTB _all disabled x

16 What are the major pain points? Disk usage Resiliency & recovery Major version upgrades 16 Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5

17 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 17 Primary Replica

18 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 2 Segment 3 Segment 3 18 Offline Primary

19 Recovery (5.x) Segment 1 Segment 1 Segment 2 Segment 4 Segment 3 19 Offline Primary

20 Recovery (5.x) Segment 1 Segment 1 Segment 2 File copy recovery Segment 4 Segment 3 20 Repica Primary

21 Recovery (6.x) Transaction-log recovery Repica Primary

22 What are the major pain points? Disk usage Resiliency & recovery Major version upgrades 22 Marie-Lan Nguyen Wikimedia Commons / CC-BY 2.5

23 Major version upgrades pain Full cluster restart == downtime Client/server version compatibility (Java) Data retention/lifecycle Too many hard-to-resolve breaking changes Little/no warnings of deprecations & breaking changes 23 Famartin Wikimedia Commons / CC-BY 2.5

24 Major version upgrades pain Full cluster restart == downtime Client/server version compatibility (Java) Data retention/lifecycle Too many hard-to-resolve breaking changes Little/no warnings of deprecations & breaking changes 24 Famartin Wikimedia Commons / CC-BY 2.5

25 Rolling upgrades Upgrade from 5.latest to 6.x without full cluster restart Some caveats to be aware of: There are still breaking changes! However many have options backported. Read & test! All nodes must be 5.latest before Security+TLS 25

26 Major version upgrades pain Full cluster restart == downtime Client/server version compatibility (Java) Data retention/lifecycle Too many hard-to-resolve breaking changes Little/no warnings of deprecations & breaking changes 26 Famartin Wikimedia Commons / CC-BY 2.5

27 Java client All other languages use REST interface Transport client tied to Elasticsearch major version Complicates firewalls & security 27

28 Java Low level REST client Released in 5.0 JSON strings only Resilient, but not user friendly Low level like other language clients Can sniff nodes and Round-Robin requests 28

29 Java high level REST client Released in 5.6 Works across major version upgrade IDE friendly Similar API to Transport Client Based on low-level REST client Supports CRUD & Search 29

30 Major version upgrades pain Full cluster restart == downtime Client/server version compatibility (Java) Data retention/lifecycle Too many hard-to-resolve breaking changes Little/no warnings of deprecations & breaking changes 30 Famartin Wikimedia Commons / CC-BY 2.5

31 Data Retention / Lifecycle: What are Your Options? Delete old data Reindex Run v1.2.0 and hope your security team doesn t notice Keeps cluster & servers tidy :) Good where there s a practical retention period :) Still a period that you can t upgrade :( No in-between / rollups :( Get the latest & greatest features :) Can be time consuming depending on data volumes :( No easy management :( Path of least resistance? Eventually, we stop backporting security fixes :( You never get the latest & greatest :( 31

32 Cross Major Version Search v5.6.0 v6.0.0 Master Nodes Master Nodes Data Node Your App Data Node Data Node v5.latest Cross Cluster Client 32

33 The latest and greatest in 6.1 and 6.2 Shard Splitting Adaptive Replica Selection Paging Aggregations Java 9 Support (6.2) SSO / SAML (6.2) Plugin Extensibility via SPI (6.2) 33

34 34 QUESTIONS?

35 Thank You