Ingest Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017
Data Ingestion The process of collecting and importing data for immediate use 2
? Simple things should be simple. Shay Banon Elastic{ON} 17 3
Ingest Technologies Elaticsearch Beats Logstash ES-Hadoop APIs Ingest Node Lightweight Data Shippers Centralized Data Collection Engine Hadoop Ecosystem Connector 4
Elastic Ingestion Technologies Elasticsearch API ingest node Transform data node Store 5
Elastic Ingestion Technologies Elasticsearch ingest node Transform data node Store es-hadoop CUSTOM CONNECTORS 6
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform Logs Metrics data node Store servers, containers 7
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices DB data Flows JDBC Logstash 8
Ingestion Architecture Scalable and robust centralized ETL Persistent queues Dead letter queues 9
Ingestion Architecture Scalable and robust centralized ETL Java event rewrite Multiple pipelines 10
Ingestion Architecture Scalable and robust centralized ETL Java event rewrite Multiple pipelines 11 Logstash 5.x
Ingestion Architecture Scalable and robust centralized ETL Java event rewrite Multiple pipelines 12 Logstash 6.0
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices DB data Flows JDBC Logstash 13
Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices DB data Flows JDBC Logstash 14
Easy migration between ingest technologies Ingest Node to Logstash conversion tool Elasticsearch ingest node Logstash 15
Data Sources
Use Cases & Data Sources Logging Metrics Security Common Log Formats System Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation 17
Modules: The Data to Dashboard Experience Collect specific type of data Parse and enrich it Default dashboards, alerts, ML jobs./filebeat -e -modules=system -setup 18
19 Packetbeat (It all started with Beats 1.0)
20 Metricbeat Modules (Introduced in 5.0) Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vsphere Windows ZooKeeper
Filebeat Modules (Introduced in 5.3) Apache2 Auditd Icinga Kafka MySQL Nginx PostgreSQL Redis System 21
Logstash Modules (Introduced in 5.6) ArcSight Netflow 22
23 ArcSight Module (Introduced in 5.6)
Modules Demo NGINX Netflow ArcSight 24
Logging Data Sources FILEBEAT WINLOGBEAT Infrastructure Applications System Linux / MacOS Windows Events Containers Docker (6.0) Kubernetes (6.0) Databases MySQL PostgreSQL (6.1) Queues Kafka (6.1) Redis (6.0) Web servers Apache Nginx Other HAProxy* Zookeeper* * Near-term roadmap 25
Metrics & Event Data METRICBEAT PACKETBEAT LOGSTASH Infrastructure System Linux MacOS Windows Perfmon (6.0) Containers Docker Kubernetes (6.0) Virtualization vsphere (6.0) Cloud AWS GCP Azure* DigitalOcean Network Netflow (5.6) Packets Storage Ceph WMI*. * Near-term roadmap 26
Metrics & Event Data METRICBEAT HEARTBEAT LOGSTASH Applications Datastores Queues Uptime Web servers MySQL Kafka Heartbeat Apache PostgreSQL Redis Custom apps Nginx MongoDB RabbitMQ (6.0) JMX/Jolokia Other Couchbase Caches PHP-FPM HAProxy Aerospike (6.0) Memcached (6.0) Golang (6.0) Zookeeper Graphite (6.1) Dropwizard (6.0) Prometheus * Near-term roadmap 27
Security Data Sources FILEBEAT METRICBEAT PACKETBEAT LOGSTASH Security SIEM Augmentation ArcSight (5.6) more* Audit Auditd Auditbeat (6.0) Systems Access SSH Applications Connections Users Activity Network IPs / GeoIP DNS Packets Netflow (5.6) Firewalls* IDS/IPS* * Near-term roadmap 28
Business Analytics LOGSTASH Structured Databases JDBC input JDBC filter SaaS services Salesforce Heroku Github Azure* Activity Social media Twitter * Near-term roadmap 29
Administration
Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) 31
Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) Beats (Roadmap) Centralized monitoring Centralized management 32
Calls to Action Familiarize yourself with latest integrations (including in X-Pack) Watch UI roadmap for additional add-data workflows Take the Data Sources Survey: http://go.es.io/2geboln Come talk to us at the AMA booth 33
Thank You Find me at AMA booth or email untergeek@elastic.co