Ingest. David Pilato, Developer Evangelist Paris, 31 Janvier 2017

Transcription

1 Ingest David Pilato, Developer Evangelist Paris, 31 Janvier 2017

2 Data Ingestion The process of collecting and importing data for immediate use in a datastore 2

3 ? Simple things should be simple. Shay Banon Elastic{ON} 17 3

4 Ingest Technologies Elasticsearch Beats Logstash ES-Hadoop APIs Ingest Node Lightweight Data Shippers Centralized Data Collection Engine Hadoop Ecosystem Connector 4

5 Elastic Ingestion Technologies Elasticsearch API ingest node Transform data node Store 5

6 Elastic Ingestion Technologies Elasticsearch ingest node Transform data node Store es-hadoop CUSTOM CONNECTORS 6

7 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform Logs Metrics data node Store servers, containers 7

8 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices DB data Flows JDBC Logstash 8

9 Ingestion Architecture Scalable and robust centralized ETL Java event rewrite Multiple pipelines 9

10 Ingestion Architecture Scalable and robust centralized ETL Persistent queues Dead letter queues 10

11 Cooperative Ingestion DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers data node Store 11

12 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform servers, containers CENTRALIZED COLLECTION data node Store network devices Logstash 12

13 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats CENTRALIZED COLLECTION ingest node Transform servers, containers Logstash data node Store network devices 13

14 Easy migration between ingest technologies Ingest Node to Logstash conversion tool Elasticsearch ingest node Logstash 14

15 Use Cases & Data Sources

16 Use Cases & Data Sources Logging Metrics Security Common Log Formats System Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation 16

17 Modules: Data sources made easy Collect specific type of data Parse and enrich it Default dashboards, alerts, ML jobs./filebeat -e -modules=system -setup 17

18 Metricbeat modules (introduced in 5.0) Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vsphere Windows ZooKeeper 18

19 Filebeat modules (introduced in 5.3) Apache2 Auditd Icinga Kafka MySQL Nginx PostgreSQL Redis System 19

20 Logstash modules (introduced in 5.6) ArcSight Netflow 20

21 21 ArcSight Module (Introduced in 5.6)

22 22 DEMO

23 Logging Data Sources FILEBEAT WINLOGBEAT Infrastructure Applications System Linux / MacOS Windows Events Containers Docker (6.0) Kubernetes (6.0) Databases MySQL PostgreSQL (6.1) Queues Kafka (6.1) Redis (6.0) Web servers Apache Nginx Other HAProxy* Zookeeper* * Near-term roadmap 23

24 Metrics & Event Data METRICBEAT PACKETBEAT LOGSTASH Infrastructure System Linux MacOS Windows Perfmon (6.0) Containers Docker Kubernetes (6.0) Virtualization vsphere (6.0) Cloud AWS GCP Azure* DigitalOcean Network Netflow (5.6) Packets Storage Ceph WMI*. 24 * Near-term roadmap

25 Metrics & Event Data METRICBEAT HEARTBEAT LOGSTASH Applications Datastores Queues Uptime Web servers MySQL Kafka Heartbeat Apache PostgreSQL Redis Custom apps Nginx MongoDB RabbitMQ (6.0) JMX/Jolokia Other Couchbase Caches PHP-FPM HAProxy Aerospike (6.0) Memcached (6.0) Golang (6.0) Zookeeper Graphite (6.1) Dropwizard (6.0) Prometheus * Near-term roadmap 25

26 Security Data Sources FILEBEAT METRICBEAT PACKETBEAT LOGSTASH Security SIEM Augmentation ArcSight (5.6) more* Audit Auditd Auditbeat (6.0) Systems Access SSH Applications Connections Users Activity Network IPs / GeoIP DNS Packets Netflow (5.6) Firewalls* IDS/IPS* * Near-term roadmap 26

27 Business Analytics LOGSTASH Structured Databases JDBC input JDBC filter SaaS services Salesforce Heroku Github Azure* Activity Social media Twitter * Near-term roadmap 27

28 Administration

29 Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) * Near-term roadmap 29

30 30 Logstash Monitoring

31 Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) * Near-term roadmap 31

32 Monitoring & Management Logstash Centralized monitoring (5.3) Centralized management (6.0) Beats (Roadmap) Centralized monitoring Centralized management 32

33 Next steps Familiarize yourself with latest integrations (including in X-Pack) Watch UI roadmap for additional add-data workflows Come talk to us at the AMA booth 33

34 Thank You Find me at AMA booth or