Elasticsearch Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec
What about Elasticsearch the Company?! Support 100s of Companies in Production environments Training Developers and Ops around the world on ELK Drive the ELK Projects forward, great things to come! Commercial products: Marvel to monitor and manage ELK! Backed by the best: Benchmark, Index Ventures, NEA Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Little known facts Elasticsearch now makes over 70% of the Lucene commits Elasticsearch now employs over 100 people in the US, UK, Netherlands, Germany, Poland, Romania, Czech R., Italy, Japan, Canada, France, Spain,? Elasticsearch powers Wikipedia.en search and the over 80 million sites on wordpress.com
The Elasticsearch ELK Stack Elasticsearch Kibana Data From Any Source Instantly Analyze Actionable Insights Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Marvel Monitor and Manage your Elasticsearch Cluster Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Who s using Elasticsearch? Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Search as Navigation Update searches in real time based on user content Roll out new features early and often automatically Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Start small ES curl XGET http://localhost:9200/' { } "status" : 200, "name" : "Turner Century", "version" : { "number" : "1.3.1", "build_hash" : "2de6dc5268c32fb49b205233c138d93aaf772015", "build_timestamp" : "2014-07-28T14:45:15Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search"
Add your app REST over http ES
Double in size ES ES What about load balancing? Cluster
Clients java PHP ES Perl.NET Ruby Python ES JS Cluster
Scale your app java PHP ES Perl.NET Ruby Python JS ES Cluster
Scale the cluster java PHP ES ES Perl.NET Ruby ES ES Python JS Cluster
Roles java PHP Perl Data Data.NET Ruby Python JS Master Cluster Data/ Master
Variations java PHP Perl Client Data Data.NET Ruby Python JS Client Master Cluster Data/ Master Servers 1,2 Servers 3,4,5,6
Log Analytics Index Billions of events per day Offer Log Analytics as a service to over 3500 customers Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
What about Logging Analytics? Logs,! Packets,! Events,! Transactions,!! Timestamp Data ES Kibana
Scale up a bit Forwarder Forwarder Data Data Kibana Shippers Shippers Master Data/ Master Cluster
Scale up some more Kibana Forwarder Forwarder Shippers Shippers Forwarder Forwarder Brokers!! Redis!! RabbitMQ!! Kafka Data Master Cluster Data Data/ Master Shippers Shippers
Process as a Stream Kibana Log Ship Log Ship Brokers Storm!! Spark!! Samza Brokers Cluster
Sentiment Analysis Index and search over millions of social interactions every day Measure sentiment changes as they happen Copyright Elasticsearch 2013. 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Where does Hadoop fit in? Kibana Log Ship Log Ship Brokers Storm!! Spark!! Samza Brokers Cluster Flume HDFS es-hadoop
Scaling Which hardware do I buy? Physical vs Virtual Sizing a shard - Distributed systems knowledge applies Dedicated Masters, Minimum Masters Replica - Pros and Cons
Common Pitfalls Not fencing network, locking down Elasticsearch Messing with defaults blindly Shoehorn relational model into NoSQL The ins and outs of Upserts Distributed systems knowledge and architecture a must
Q&A