ELK. Elasticsearch Logstash - Kibana

Transcription

1 ELK Elasticsearch Logstash - Kibana

2 Welcome to Infomart Infomart is a media monitoring app which monitors both Social and Traditional Media. Social media includes Twitter, Facebook, Youtube, Wordpress, Tumblr, Blogs, Forums and Web news. Traditional Media includes all the major newspapers, magazines, broadcasts, radio etc from Canada, UK, US and International. We are using Elasticsearch for storing over 500 million documents.

3 Meetup Goals Install Elasticsearch: (For storing logs) Install Kibana 4: (Dashboard for discover and visualizing logs) Install and configure Nginx: (For kibana) Install Logstash Server: (Processing logs before storing) Install Logstash Forwarder: (Forwards server s log on which it is installed to Logstash Server) Create Dashboards in Kibana to visualize these logs.

4 Meetup Goals Logstash Forwarder Logstash Forwarder Logstash Server Elastic Search Kibana Logstash Forwarder

5 Install Elasticsearch sudo add-apt-repository -y ppa:webupd8team/java sudo apt-get update sudo apt-get -y install oracle-java8-installer wget -O - sudo apt-key add - echo 'deb stable main' sudo tee /etc/apt/sources.list.d/elasticsearch.list sudo apt-get update

6 Install Elasticsearch(Cont ) sudo apt-get -y install elasticsearch=1.4.4 sudo service elasticsearch restart sudo update-rc.d elasticsearch defaults cd /usr/share/elasticsearch./bin/plugin -i elasticsearch/marvel/latest

7 Install Elasticsearch(Cont ) Success: IPADDRESS:9200 should return you similar response

8 Install Kibana 4 cd /opt wget linux-x64.tar.gz tar xvf kibana linux-x64.tar.gz rm kibana linux-x64.tar.gz mv kibana linux-x64/ kibana vim kibana/config/kibana.yml host: "localhost"

9 Install Kibana 4(Cont ) cd /etc/init.d sudo wget 8b15ac909aed214ad04a/raw/ bce61d85643c2dcdfbc2728c55a41dab444dca20/kibana4 sudo chmod +x /etc/init.d/kibana4 sudo update-rc.d kibana4 defaults 96 9 sudo service kibana4 start

10 Install and configure nginx(kibana) sudo apt-get install nginx apache2-utils sudo htpasswd -c /etc/nginx/htpasswd.users USERNAME sudo vi /etc/nginx/sites-available/default server { listen 80; server_name localhost; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }

11 Install Kibana 4(Cont ) Success: IPADDRESS:80 should return you similar response

12 Install Logstash Server echo 'deb stable main' sudo tee /etc/apt/sources.list.d/logstash.list sudo apt-get update sudo apt-get install logstash vim /etc/ssl/openssl.cnf Insert after v3_ca subjectaltname = IP:IPADDRESS

13 Install Logstash Server(Cont ) cd /etc/ssl/ sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days batch -nodes -newkey rsa: keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt vim /etc/logstash/conf.d/01-lumberjack-input.conf input { lumberjack { port => 5000 type => "logs" ssl_certificate => "/etc/ssl/certs/logstash-forwarder.crt" ssl_key => "/etc/ssl/private/logstash-forwarder.key" } } vim /etc/logstash/conf.d/30-lumberjack-output.conf output { elasticsearch { host => localhost } stdout { codec => rubydebug } }

14 Install Logstash Server(Cont ) cd /etc/logstash sudo curl -O " gunzip GeoLiteCity.dat.gz vim /etc/logstash/conf.d/12-apache.conf filter { if [type] == "apache-access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/mmm/yyyy:hh:mm:ss Z" ] } geoip { source => "clientip" target => "geoip" database => "/etc/logstash/geolitecity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] } } } sudo service logstash restart

15 Install Logstash Forwarder Copy the cert from logstash server to logstash forwarder client server. ("/etc/ssl/ certs/logstash-forwarder.crt ) echo 'deb stable main' sudo tee /etc/apt/sources.list.d/logstashforwarder.list wget -O - sudo aptkey add - sudo apt-get update sudo apt-get install logstash-forwarder

16 Install Logstash Forwarder(Cont ) sudo vi /etc/logstash-forwarder.conf "servers": [ "IPADDRESS OF LOGSTAH SERVER:5000" ], "timeout": 15, "ssl ca": "/etc/ssl/certs/logstash-forwarder.crt { "paths": [ "/var/log/apache2/access*.log" ], "fields": { "type": "apache-access" } } sudo service logstash-forwarder restart Success: IPADDRESS:9200/_plugin/marvel should show indexed data

17 Kibana 4 Settings Discover Visualize Dashboard

18 Kibana 4 Kibana 4 Demo

19

20

21

22

23

24 Special Thanks To Warren Gedge Julia Andrews Amit Sheldon Sawchuk Katrina Zivanovich Adam Hutchinson Jose Bento Neil

25 Thank You