Ingest David Pilato, Developer Evangelist Paris, 31 Janvier 2017
Data Ingestion The process of collecting and importing data for immediate use in a datastore 2
? Simple things should be simple. Shay Banon Elastic{ON} 17 3
Ingest Technologies Elasticsearch Beats Logstash ES-Hadoop APIs Ingest Node Lightweight Data Shippers Centralized Data Collection Engine Hadoop Ecosystem Connector 4
Elastic Ingestion Technologies Elasticsearch API ingest node Transform data node Store 5
Elastic Ingestion Technologies Elasticsearch ingest node Transform data node Store es-hadoop CUSTOM CONNECTORS 6
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform Logs Metrics data node Store servers, containers 7
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices DB data Flows JDBC Logstash 8
Ingestion Architecture Scalable and robust centralized ETL Java event rewrite Multiple pipelines 9
Ingestion Architecture Scalable and robust centralized ETL Persistent queues Dead letter queues 10
Cooperative Ingestion DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers data node Store 11
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices Logstash 12
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats CENTRALIZED COLLECTION ingest node Transform servers, containers Logstash data node Store network devices 13
Easy migration between ingest technologies Ingest Node to Logstash conversion tool Elasticsearch ingest node Logstash 14
Use Cases & Data Sources
Use Cases & Data Sources Logging Metrics Security Common Log Formats System Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation 16
Modules: Data sources made easy Collect specific type of data Parse and enrich it Default dashboards, alerts, ML jobs./filebeat -e -modules=system -setup 17
Metricbeat modules (introduced in 5.0) Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vsphere Windows ZooKeeper 18
Filebeat modules (introduced in 5.3) Apache2 Auditd Icinga Kafka MySQL Nginx PostgreSQL Redis System 19
Logstash modules (introduced in 5.6) ArcSight Netflow 20
21 ArcSight Module (Introduced in 5.6)
22 DEMO
Logging Data Sources FILEBEAT WINLOGBEAT Infrastructure Applications System Linux / MacOS Windows Events Containers Docker (6.0) Kubernetes (6.0) Databases MySQL PostgreSQL (6.1) Queues Kafka (6.1) Redis (6.0) Web servers Apache Nginx Other HAProxy* Zookeeper* * Near-term roadmap 23
Metrics & Event Data METRICBEAT PACKETBEAT LOGSTASH Infrastructure System Linux MacOS Windows Perfmon (6.0) Containers Docker Kubernetes (6.0) Virtualization vsphere (6.0) Cloud AWS GCP Azure* DigitalOcean Network Netflow (5.6) Packets Storage Ceph WMI*. 24 * Near-term roadmap
Metrics & Event Data METRICBEAT HEARTBEAT LOGSTASH Applications Datastores Queues Uptime Web servers MySQL Kafka Heartbeat Apache PostgreSQL Redis Custom apps Nginx MongoDB RabbitMQ (6.0) JMX/Jolokia Other Couchbase Caches PHP-FPM HAProxy Aerospike (6.0) Memcached (6.0) Golang (6.0) Zookeeper Graphite (6.1) Dropwizard (6.0) Prometheus * Near-term roadmap 25
Security Data Sources FILEBEAT METRICBEAT PACKETBEAT LOGSTASH Security SIEM Augmentation ArcSight (5.6) more* Audit Auditd Auditbeat (6.0) Systems Access SSH Applications Connections Users Activity Network IPs / GeoIP DNS Packets Netflow (5.6) Firewalls* IDS/IPS* * Near-term roadmap 26
Business Analytics LOGSTASH Structured Databases JDBC input JDBC filter SaaS services Salesforce Heroku Github Azure* Activity Social media Twitter * Near-term roadmap 27
Administration
Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) * Near-term roadmap 29
30 Logstash Monitoring
Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) * Near-term roadmap 31
Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) Beats (Roadmap) Centralized monitoring Centralized management 32
Next steps Familiarize yourself with latest integrations (including in X-Pack) Watch UI roadmap for additional add-data workflows Come talk to us at the AMA booth 33
Thank You Find me at AMA booth or email david@elastic.co