Inputs: File, Network, Script, and More! Splunkd: Pipelines & Processors & Queues, Oh my!

Transcription

1 Copyright 2014 Splunk Inc. Inputs: File, Network, Script, and More! Splunkd: Pipelines & Processors & Queues, Oh my! Amrit Bath Jag Kerai

2 Disclaimer During the course of this presentanon, we may make forward looking statements regarding future events or the expected performance of the company. We caunon you that such statements reflect our current expectanons and esnmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentanon are being made as of the Nme and date of its live presentanon. If reviewed ayer its live presentanon, this presentanon may not contain current or accurate informanon. We do not assume any obliganon to update any forward looking statements we may make. In addinon, any informanon about our roadmap outlines our general product direcnon and is subject to change at any Nme without nonce. It is for informanonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obliganon either to develop the features or funcnonality described or to include any such feature or funcnonality in a future release. 2

3 Agenda Splunkd building blocks: Pipelines, processors, queues Not here: Indexing/clustering, searching. Where data goes & how Where data comes from Debugging/opNmizing (What Would Octavio Do?)

4 About Us Jag Kerai (2008) Eng & Manager Forwarding/receiving, Splunkd framework, HA/ Clustering, Deployment server, SAML, Previously: Xsigo Systems, Brocade CommunicaNons, SAP Lab Web apps in 1995! Is a smart dude Amrit Bath (2005) Eng & Hungover CLI, Deploy Serv, Tailing, REST API, Universal Forwarder, Indexed ExtracNons, Cloud Previously: Unemployment, College Tries to enjoy working on cars 4

5 Splunk Architecture Splunk CLI Interface Splunk > Engine Splunk Web Interface REST API Other Interfaces Scheduling/AlerNng ReporNng Knowledge Distributed Search Search Distributed Search Deployment Server Index Forward Cluster Users / ACL Parse / Extract / Manipulate Monitor Files Detect File Changes Network input Script / Stdout 5

6 It s All a Pipeline C A B D E 6

7 Really C 7200 RPM/ 10k/15k? A B Forwarder 100Mb? D E 7

8 Internally, too! C Queue Processors Queue A B D Monitor Input E Index 8

9 And Across MulNple Instances! (This is the biggun ) Universal Forwarder Processors TCP input Indexer Queue Queue 100 Mb Queue Monitor Input TCP Output Index 9

10 Data Structures & RouNng

11 _conf Host Index www2, access_log, /var/log/hupd/access_log www2 prod_servers [21/Jul/2011:20:34: ] "GET / results/bonnie- solns_vm_nick.html HTTP/1.1" UTF- 8 Line Broken 11

12 Pipelines Parsing Pipeline Input UTF- 8 Converter Line Breaker Header ExtracNon Output Pipeline: thread Data flows through linearly, hits mulnple pipelines before indexing Naturally allow parallelism, modularity Config files: $SPLUNK_HOME/etc/modules/ Merged: var/run/splunk/composite.xml 12

13 Processor Pipeline Input Processor 1 Processor 2 Output Processor: performs small but logical unit of work Contained within a pipeline Executed by pipeline thread Example: LineBreaker, Aggregator, TcpInput, Index 13

14 Pipelines/Processors (Debugging) UF Parsing Pipeline Merging Pipeline Typing Pipeline Index Pipeline TCP/UDP pipeline Tailing uw8 regex replacement tcp out UF + IDX_E FIFO pipeline FSChange Parsing Queue linebreaker Agg Queue aggregator Typing Queue annotator Index Queue syslog out Exec pipeline header indexer 14

15 Pipelines/Processors (Debugging) Parsing Pipeline Merging Pipeline Typing Pipeline Index Pipeline TCP/UDP pipeline Tailing uw8 regex replacement tcp out FIFO pipeline FSChange Parsing Queue linebreaker Agg Queue aggregator Typing Queue annotator Index Queue syslog out Exec pipeline header indexer 15

16 Pipelines/Processors (Debugging) Parsing Pipeline Merging Pipeline Typing Pipeline Index Pipeline TCP/UDP pipeline uw8 Tailing FIFO pipeline Parsing Queue linebreaker Agg Queue aggregator FSChange Exec pipeline tcp out regex replacement Typing Queue Index Queue syslog out annotator header indexer 16

17 Queue Process Thread Queue Thread Insert pdata pdata pdata pdata Remove Process Queue size bounded by memory Variable size pipeline data 17

18 Typing Pipeline Queue Index Queue (Full) Tcp Output Processors Process Insert (Blocks) pdata pdata pdata pdata Remove Write to Network (Fails) Network Down Queue (Full) Pipeline Thread Insert Blocked Queue (Full) Pipeline Thread Insert Blocked Queue (Full) 1. Remove 1. Remove pdata 2. Process 2. Process pdata 3. Insert (Blocked) 3. Insert (Blocked) pdata 18

19 Persistent Queue Regular Queues pdata pdata Writer blocks if Q is full RAM Persistent Queues pdata pdata If memory used up, use file system Writer does not block if memory is used up Think virtual memory RAM File System 19

20 Persistent Queue Host Network Internal Queues Full Network Splunk Input Q Persistent Q pdata pdata Tcpout Q Full Much Bigger Queue 20

21 Processors: Input & Parsing

22 Input Pipelines Network File Monitor FIFO FS Change Script Registry Monitor Host Index www2 prod_servers Jul 30 00:21:19 amritdesktop sshd[30416]: Accepted publickey for amrit from port ssh2 \n Jul 30 00:21:26 amritdesktop sshd[30418]: Received disconnect from : 11: disconnected by user \n Parsing Queue 22

23 Monitor Input (aka Tailing Processor) Two synchronous components: File Update NoNficaNon (FUN!) Tailer: reads files Files are read: 1) One at a Nme 2) In 64KB chunks, 3) UnNl EOF. Can read large files & archives in parallel. Send <=64 KB chunks to output queue FUN! Tailer Batch Archive Parsing Queue 23

24 Parsing: UTF- 8 Processor Host Charset www2 ShiY- JIS ƒ ƒ ƒeƒbƒaƒbƒneƒgƒ ƒ ƒuƒ UTF- 8 Processor Host Charset www2 UTF- 8 24

25 Parsing: Line Breaker Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: UpdateController: Message tracing { "interval_since_last_invocanon" = 23000; "power_source" = ac; "power_state" = wake; "start_date" = " :10: "; } Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: Asserted BackgroundTask power assernon (returned 0) Line Breaker Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: UpdateController: Message tracing { "interval_since_last_invocanon" = 23000; "power_source" = ac;. 25

26 **OR** Parsing: CSV/JSON Line Breaker (6.0+) From file containing: Subject,DescripNon,Presenter How Splunkd works, Learn about how Splunk ingests and parses data,amrit/jag How search works,learn how Splunk s search language works,dr. Z sourcetype How Splunkd works, Learn about how Splunk ingests and parses data,amrit/jag How search works,learn how Splunk s search language works,dr. Z csv CSV/JSON Line Breaker See also: sourcetype=_json INDEXED_EXTRACTIONS=(csv json ) sourcetype _meta sourcetype How Splunkd works, Learn about how Splunk ingests and parses data,amrit/jag csv Subject:: How Splunkd works DescripNon:: Learn about how Splunk ingests and\nparses data Presenter::Amrit/Jag How search works,learn how Splunk s search language works,dr. Z csv Subject:: How search works DescripNon:: Learn how Splunk s search _meta language works Presenter:: Dr. Z 26

27 Parsing: Header Processor Host www2 ***SPLUNK*** host=database_1 Host www2 Unknown database error Header Processor Host database_1 Unknown database error 27

28 Merging: Line Merge (aka Aggregator) Data Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: UpdateController: Message tracing { "interval_since_last_invocanon" = 23000; "power_source" = ac;. Line Merge Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: UpdateController: Message tracing { "interval_since_last_invocanon" = 23000; "power_source" = ac; "power_state" = wake; "start_date" = " :10: "; } Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: Asserted BackgroundTask power assernon (returned 0) 28

29 Typing: Regex Replacement Host Sourcetype logbox syslog Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Regex Replacement Host Sourcetype abath- mba13.no.cox.net syslog Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan 29

30 Typing: Annotator Host abath- mba13.no.cox.net Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan Annotator Host Punct abath- mba13.no.cox.net ::_- _[]_<>: Pipeline Pipeline Data Data Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan 30

31 Indexer Pipeline: TCP/Syslog Out, Indexer Host Index abath- mba13.no.cox.net main Sep 12 06:11:58 abath- mba13.no.cox.net storeagent[49597] <CriNcal>: StarNng update scan TCP Output To remote server Syslog Output To remote server Indexer To disk.. Or cluster! 31

32 Scripted Input (aka Exec Processor) mysql_fetch.sh web_ping.bat data args data args data Exec Processor tracert.exe args Parsing Q 32

33 TCP/UDP Input Splunk TCP/UDP Pipeline Parsing Pipeline TCP Network Read Parsing Q Processors Port Insert Remov e Insert to Queue UDP Host Wait 33

34 TCP Output Qs Splunk Typing Pipeline TCPOut Q Processors Insert (Blocks) Index Q pdata Clone or Route pdata Load Balance Group TCPOut Q pdata Network Down Splunk Load Balance Group Splunk Network Splunk 34

35 Nerdier Stuff 35

36 Resource Management Raw Storage :15:46,674 DEBUG [com.splunk.doom] com.sun.ebank.ejb.customer. \n java.lang.numberformatexcepnon: For input string: "fish!" \n at java.lang.numberformatexcepnon.forinputstring(numberformatexcepnon.java:48) \n at java.lang.integer.parseint(integer.java:447) \n :15:46,674 DEBUG [com.splu... at java.lang.numberformatexcepnon.f java.lang.numberformatexcepnon: For in at java.lang.integer.parseint(integer.jav 36

37 Debugging! Metrics! S.O.S App (and DMC )! 37

38 metrics.log: Queues via S.O.S 38

39 metrics.log: Queues via Search Search: index=_internal group=queue eval pc=(current_size_kb*100)/max_size_kb Nmechart perc90(pc) by name 39

40 metrics.log: Processor Time via S.O.S 40

41 metrics.log: Processor Time via Search Search: index = _internal group = pipeline processor!= sendout Nmechart perc90(cpu_seconds) by processor 41

42 metrics.log: Indexing Rate via S.O.S 42

43 metrics.log: Indexing Rate via Search Search: index=_internal source=*metrics.log* group=thruput Nmechart per_second(kb) 43

44 metrics.log: Scenarios Indexing instance: Index Queue at 100% Forwarding disabled: Indexing rate? Slow disk? Full disk? Forwarding enabled: Indexing rate? Slow disk on remote indexer? Full remote disk? TCPOut rate? Low network bandwidth? High network latency? Local indexing rate? Slow local disk? Full local disk? Universal Forwarder: Parsing Queue at 100% Indexing rate? Slow disk on remote indexer? Full remote disk? TCPOut rate? Low bandwidth? High latency? (No local indexing here) Start from end, work backwards 44

45 metrics.log: Universal Forwarder No indexing/searching capability Can forward metrics to indexer May not get there! ê Configure S.O.S: hup://splunk- base.splunk.com/answers/48874/how- can- i- monitor- the- resource- usage- of- my- forwarder- using- the- sos- app#50315 ê Fix forwarding: hup://splunk- base.splunk.com/answers/38091/best- pracnces- to- deploy- the- splunk- on- splunk- app- in- a- distributed- search- environment Fallback to raw file (grep!) $ grep group=queue metrics.log grep - - color 'max_size.*current_size_kb[^,]*, Metrics - group=queue, name=typingqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1821, largest_size=1821, smallest_size=0 45

46 Search: index=_internal source=*metrics.log* Groups pipeline queue per_source_thruput per_sourcetype_thruput per_index_thruput per_host_thruput Metrics Log 46

47 Recap Splunk instance consists of linear pipelines Splunk topology emulates pipelines Downstream slowdown results in upstream blockage metrics.log across the topology reveals the whole picture Queue sizes Indexing thruput Forwarding thruput CPU usage per PipelineData Processor This is how you should debug the same way we do! See also: PipelineSets Talk(Abhinav, Sourav, Tameem)

48 QuesNons? 48

49 THANK YOU