Ingest Node: (re)indexing and enriching documents within

Transcription

1 Ingest Node: (re)indexing and enriching documents within #

2 Agenda 1 Why ingest node? 2 How does it work? 3 Where can it be used? 2

3 # Why ingest node?

4 # I just want to tail a file.

5 Logstash: collect, enrich & transport input grok date mutate output The file Filters Elasticsearch 5

6 Logstash common setup [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET /cgi-bin/try/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET /cgi-bin/try/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" message 6

7 Ingest node setup [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET /cgi-bin/try/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET /cgi-bin/try/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET /robots.txt HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" [19/Apr/2016:12:00: ] "GET / HTTP/1.1"

8 Filebeat: collect and ship [19/Apr/2016:12:00: ] "GET / HTTP/1.1" [19/Apr/2016:12:00: ] "GET /not_found/ HTTP/1.1" [19/Apr/2016:12:00: ] "GET /favicon.ico HTTP/1.1" { "message" : " [19/Apr/2016:12:00: ] \"GET / HTTP/1.1\" " { "message" : " [19/Apr/2016:12:00: ] \"GET /not_found/ HTTP/1.1\" " { "message" : " [19/Apr/2016:12:00: ] \"GET /favicon.ico HTTP/1.1\" " 8

9 Elasticsearch: enrich and index { "message" : " [19/Apr/2016:12:00: ] \"GET / HTTP/1.1\" " { "request" : "/", "auth" : "-", "ident" : "-", "verb" : "GET", "@timestamp" : " T10:00:04.000Z", "response" : "200", "bytes" : "24", "clientip" : " ", "httpversion" : "1.1", "rawrequest" : null, "timestamp" : "19/Apr/2016:12:00: " 9

10 How does ingest node work? #

11 Ingest pipeline document grok date remove enriched document Pipeline: a set of processors 11

12 Define a pipeline PUT /_ingest/pipeline/apache-log { "processors" : [ { "grok" : { "field": "message", "pattern": "%{COMMONAPACHELOG", { "date" : { "match_field" : "timestamp", "match_formats" : ["dd/mmm/yyyy:hh:mm:ss Z"], { "remove" : { "field" : "message" ] 12

13 Index a document Provide the id of the pipeline to execute PUT //apache/1?pipeline=apache-log { "message" : " [19/Apr/2016:12:00: ] \"GET / HTTP/1.1\" " 13

14 What has actually been indexed GET //apache/1 { "request" : "/", "auth" : "-", "ident" : "-", "verb" : "GET", "@timestamp" : " T10:00:04.000Z", "response" : "200", "bytes" : "24", "clientip" : " ", "httpversion" : "1.1", "rawrequest" : null, "timestamp" : "19/Apr/2016:12:00: " 14

15 Pipeline management Create, Read, Update & Delete PUT /_ingest/pipeline/apache-log { GET /_ingest/pipeline/apache-log GET /_ingest/pipeline/* DELETE /_ingest/pipeline/apache-log 15

16 grok uppercase split rename attachment set geoip append remove fail gsub foreach trim convert lowercase join date 16

17 { "grok": { Grok processor "field": "message", "pattern": "%{DATE:date" Extracts structured fields out of a single text field 17

18 { "remove": { Mutate processors "field": "message" set, remove, rename, convert, gsub, split, join, lowercase, uppercase, trim, append 18

19 { "date": { Date processor "field": "timestamp", "match_formats": ["YYYY"] Parses a date from a string 19

20 { Geoip processor "geoip": { "field": "ip" Adds information about the geographical location of IP addresses 20

21 { "foreach": { "field" : "values", "processors" : [ { Foreach processor "uppercase" : { "field" : "_value" Do something for every element of an array ] 21

22 { Plugins "your_plugin": { Introducing new processors is as easy as writing a plugin 22

23 # Ingest node internals

24 Default scenario Cluster State All nodes are equal: - node.data: true cluster - node.master: true - node.ingest: true node1 node2 2P 3R 3P 1R node3 index: 3 primary Client 1P 2R shards, 1 replica each 24

25 Default scenario All nodes are equal: - node.data: true cluster - node.master: true - node.ingest: true node1 node2 2P 3R 3P 1R index request for shard 3 node3 Pre-processing on the Client 1P 2R coordinating node 25

26 Default scenario All nodes are equal: - node.data: true cluster Indexing on the primary shard - node.master: true - node.ingest: true node1 node2 2P 3R 3P 1R index request for shard 3 node3 Client 1P 2R 26

27 Default scenario All nodes are equal: - node.data: true cluster - node.master: true - node.ingest: true node1 node2 Indexing on the replica shard 2P 3R 3P 1R index request for shard 3 node3 Client 1P 2R 27

28 Ingest dedicated nodes node.data: true node.master: true node.ingest: false cluster node1 node2 2P 3R 3P 1R node3 node4 node5 Client 1P 2R node.data: false node.master: false node.ingest: true 28

29 Ingest dedicated nodes cluster node1 node2 Forward request to an ingest node 2P 3R 3P 1R index request for shard 3 node3 node4 node5 Client 1P 2R 29

30 Ingest dedicated nodes cluster node1 node2 2P 3R 3P 1R Pre-processing on the ingest node index request for shard 3 node3 node4 node5 Client 1P 2R 30

31 Ingest dedicated nodes cluster Indexing on the primary shard node1 node2 2P 3R 3P 1R index request for shard 3 node3 node4 node5 Client 1P 2R 31

32 Ingest dedicated nodes cluster Indexing on the replica shard node1 node2 2P 3R 3P 1R index request for shard 3 node3 node4 node5 Client 1P 2R 32

33 Where can ingest pipelines be used? #

34 PUT //apache/1?pipeline=apache-log Index api { "message" : " " 34

35 PUT //_bulk { "index": { "_type": "apache", "_id": Bulk api "1", "pipeline": "apache-log" \n { "message" : " " \n { "index": {"_type": "mysql", "_id": "1", "pipeline": "mysql-log" \n { "message" : " " \n 35

36 POST /_reindex { "source": { "index": "", Reindex api, "type": "apache" "dest": { "index": "apache-", Scan/scroll & bulk indexing made easy "pipeline" : "apache-log" 36

37 Go get Elasticsearch alpha3! #

38 # Thank you