Elasticsearch. Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec

Transcription

1 Elasticsearch Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec

2 What about Elasticsearch the Company?! Support 100s of Companies in Production environments Training Developers and Ops around the world on ELK Drive the ELK Projects forward, great things to come! Commercial products: Marvel to monitor and manage ELK! Backed by the best: Benchmark, Index Ventures, NEA Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

3 Little known facts Elasticsearch now makes over 70% of the Lucene commits Elasticsearch now employs over 100 people in the US, UK, Netherlands, Germany, Poland, Romania, Czech R., Italy, Japan, Canada, France, Spain,? Elasticsearch powers Wikipedia.en search and the over 80 million sites on wordpress.com

4 The Elasticsearch ELK Stack Elasticsearch Kibana Data From Any Source Instantly Analyze Actionable Insights Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

5 Marvel Monitor and Manage your Elasticsearch Cluster Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

6 Who s using Elasticsearch? Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

7 Search as Navigation Update searches in real time based on user content Roll out new features early and often automatically Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

8 Start small ES curl XGET { } "status" : 200, "name" : "Turner Century", "version" : { "number" : "1.3.1", "build_hash" : "2de6dc5268c32fb49b205233c138d93aaf772015", "build_timestamp" : " T14:45:15Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search"

9 Add your app REST over http ES

10 Double in size ES ES What about load balancing? Cluster

11 Clients java PHP ES Perl.NET Ruby Python ES JS Cluster

12 Scale your app java PHP ES Perl.NET Ruby Python JS ES Cluster

13 Scale the cluster java PHP ES ES Perl.NET Ruby ES ES Python JS Cluster

14 Roles java PHP Perl Data Data.NET Ruby Python JS Master Cluster Data/ Master

15 Variations java PHP Perl Client Data Data.NET Ruby Python JS Client Master Cluster Data/ Master Servers 1,2 Servers 3,4,5,6

16 Log Analytics Index Billions of events per day Offer Log Analytics as a service to over 3500 customers Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

17 What about Logging Analytics? Logs,! Packets,! Events,! Transactions,!! Timestamp Data ES Kibana

18 Scale up a bit Forwarder Forwarder Data Data Kibana Shippers Shippers Master Data/ Master Cluster

19 Scale up some more Kibana Forwarder Forwarder Shippers Shippers Forwarder Forwarder Brokers!! Redis!! RabbitMQ!! Kafka Data Master Cluster Data Data/ Master Shippers Shippers

20 Process as a Stream Kibana Log Ship Log Ship Brokers Storm!! Spark!! Samza Brokers Cluster

21 Sentiment Analysis Index and search over millions of social interactions every day Measure sentiment changes as they happen Copyright Elasticsearch Copying, publishing and/or distributing without written permission is strictly prohibited

22 Where does Hadoop fit in? Kibana Log Ship Log Ship Brokers Storm!! Spark!! Samza Brokers Cluster Flume HDFS es-hadoop

23 Scaling Which hardware do I buy? Physical vs Virtual Sizing a shard - Distributed systems knowledge applies Dedicated Masters, Minimum Masters Replica - Pros and Cons

24 Common Pitfalls Not fencing network, locking down Elasticsearch Messing with defaults blindly Shoehorn relational model into NoSQL The ins and outs of Upserts Distributed systems knowledge and architecture a must

25 Q&A