Splunk for Ad Hoc Explora2on of Twi6er (and more) Stephen Sorkin VP Engineering, Splunk

Size: px

Start display at page:

Download "Splunk for Ad Hoc Explora2on of Twi6er (and more) Stephen Sorkin VP Engineering, Splunk"

Julia Bradley
5 years ago
Views:

1 Splunk for Ad Hoc Explora2on of Twi6er (and more) Stephen Sorkin VP Engineering, Splunk

2 Who am I Berkeley PhD dropout. LeH to work at HP Labs. At Splunk since VP Engineering since Run the core product team.

3 Agenda Inspira2on for Splunk Architecture: Collec2on Indexing Search Real- 2me Search Demo

4 Sources What Does Machine Data Look Like? Order Processing Middleware Error Care IVR Twi,er 4

5 Sources Machine Data Contains Cri2cal Insights Customer ID Order ID Product ID Order Processing Order ID Customer ID Middleware Error Time Wai2ng On Hold Care IVR Customer ID Twi6er ID Customer s Tweet Twi,er Company s Twi6er ID 5

Splunk Enterprise with Hadoop GPS, RFID,

Clickstreams, Mobile, Telephony, IVR, Data

Servers, Security devices, Desktops, CDRs,

6 Splunk Enterprise with Hadoop GPS, RFID, Hypervisor, Web Servers, , Messaging, Clickstreams, Mobile, Telephony, IVR, Data Databases, Sensors, Telema2cs, Storage, Servers, Security devices, Desktops, CDRs, Applica2ons Ad hoc search Add knowledg e Splunk storage Monitor and alert Custom dashboard s Report and analyze Other Data Stores 6

Ge[ng Data into Splunk Agent and Agent- less

File Monitoring log filesconfig files dumps and

$devices Mounted File Systems \\hostname\mount WMI$ Scripted Inputs shell scripts custom parsers

performance counters registry monitoring Ac@ve

Unix, Linux and Windows hosts Windows hosts

7 Ge[ng Data into Splunk Agent and Agent- less Approach for Flexibility syslog TCP/UDP Local File Monitoring log filesconfig files dumps and trace files syslog compagble hosts and network devices Mounted File Systems \\hostname\mount WMI Event Logs Performance AcGve Director yy code Scripted Inputs shell scripts custom parsers batch loading Windows Inputs Event Logs performance counters registry monitoring Directory monitoring shell perf virtual host Unix, Linux and Windows hosts Windows hosts Custom apps and scripted API connecgons Windows hosts Agent- less Data Input Splunk Forwarder 7

8 Pipelines/Processors Parsing Pipeline Merging Pipeline Typing Pipeline Index Pipeline Network Inputs u_8 Regex replacement tcp out File System Inputs Parsing Queue linebreaker Agg Queue aggregator Typing Queue Index Queue syslog out annotator Scripted Inputs header indexer

9 Index Processor IDX 2 IDX 3 IDX 1 Source/Sourcetype/Host Metadata 1 source : : /my/log 100 et lt it Home Path 2 source: : /blah 150 et lt it hot_v1_100 *.data *.tsidx rawdata TSIDX cream apple beer coke ice java LEXICON hot_v1_101 apple POSTING db_lt_et_101 beer Cold Path db_lt_et_80 Rawdata apple pie and ice cream is delicious Thawed Path db_lt_et_70 an apple a day keeps doctor away 9

10 Events Bucket Lifecycle [Hot Bucket is Full] [Too Many Warms] [Out of Space or Bucket is Old] $ Home Path $ Cold Path [Cheaper Storage] [Explicit User Ac2on] $ Thawed Path $ Frozen Path or Deleted 10

11 Scales to TBs/day and Thousands of Users Automa2c load balancing linearly scales indexing " Distributed search and MapReduce linearly scales search and repor2ng 11

12 Search Model Splunk Database as a table Columns = fields, rows = events No fixed schema Unlimited number of rows, can be very sparse Special fields: _raw, _time, host, source, sourcetype search: series of commands with arguments implicit search command usually first Input/output of every command is a table 12

13 Search Model Example 13

14 Search Command Expand Search: lookups, tags, savedsearch, even6ypes, etc LISPY Expression (per index) DB Lookup s Calculated fields (5.0+) Field aliasing Field extracgons sourcetype renaming Filter Apply even,ype s Apply tags 14

15 Inside Universal Indexing Automa2c event boundary iden2fica2on Automa2c 2mestamp normaliza2on...enable accurate searching and trending by 2me across all data: 15

16 Inside Search- 2me Knowledge Extrac2on Automa2cally discovered fields And user- defined fields... enable sta2s2cs and precise search on specific fields: 16

17 Inside Search- 2me Knowledge Searches saved as event types Extrac2on Plus tagging of event types, hosts and other fields... enable normalized repor2ng, knowledge sharing and granular access control. 17

18 Integrate External Data Extend analysis with lookups to external data sources LDAP, AD Watch Lists CMDB CRM/ ERP Correlate IP addresses with loca2ons, accounts with regions 18

1. POST to /services/ search/jobs on search head 2.

Search UI Search Head REST Splunkd DB Search Head Search Process Indexer 1 Indexer 2 3.

Each search peer spawns another search process to run remote search 5.

19 1. POST to /services/ search/jobs on search head 2. Search head spawns search in a separate process Distributed Searching Distributed Search UI Search Head REST Splunkd DB Search Head Search Process Indexer 1 Indexer 2 3. Send remote version of search to each search peers via /services/streams/ search 4. Each search peer spawns another search process to run remote search 5. Read data from indexes 5b. For real2me, connect back to splunkd REST REST Search Peers Splunkd Search Process Splunkd Search Process DB DB 19

Real- 2me Search Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normaliza2on Line

20 Real- 2me Search Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normaliza2on Line breaking Timestamp iden2fica2on Regex transforms Index Queue Real- 2me Buffer Indexing Pipeline Raw data Index Files Real- 2me Search Process Index 20

Real- 2me Aler2ng source= /var/log/secure.

Timestamp iden2fica2on Regex transforms Index Queue

21 Real- 2me Aler2ng source= /var/log/secure.log BAD SU Data Monitor Input TCP/UDP Input Scripted Input Parsing Queue Parsing Pipeline Source, event typing Character set normaliza2on Line breaking Timestamp iden2fica2on Regex transforms Index Queue Real- 2me Buffer Indexing Pipeline Raw data Index Files Real- 2me Search Process Index 21

22 Demo " h6p://socialsplunk.com/ " h6p://socialsplunk.com:8081/map " h6ps://splunk4good- rtv.s3.amazonaws.com/ rtv.png

23 The 2012 Elec2on source="twitter_httpstream" romney OR obama eval text=lower(body) fields text rex field=text max_match=1000 za- Z]{5,})" eval token=mvfilter(not match(token, `clean_tweets` eval candidate=if(searchmatch("*obama* AND *romney*"), "obama:romney", if(searchmatch("*romney*"), "romney", if(searchmatch("*obama*"), "obama", null))) where NOT isnull(candidate) makemv delim=":" candidate top token by candidate limit=50

Understanding Splunk AcceleraGon Technologies David Marquardt

Copyright 2013 Splunk Inc. Understanding Splunk AcceleraGon Technologies David Marquardt Senior So?ware Engineer #splunkconf Legal NoGces During the course of this presentagon, we may make forward- looking