Copyright 2014 Splunk Inc. Data On- Boarding. Andrew Duca Sr. Professional Services Consultant, Splunk

Transcription

1 Copyright 2014 Splunk Inc. Data On- Boarding Andrew Duca Sr. Professional Services Consultant, Splunk

2 Disclaimer During the course of this presentagon, we may make forward- looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentagon are being made as of the Gme and date of its live presentagon. If reviewed aqer its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward- looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. 2

3 About Me! Senior Professional Services Consultant based in Boston, MA! 14+ Years of world- wide Professional Services ConsulGng with the last two at Splunk! Involved in 20+ deployments from 1GB to 5TB 3

4 Agenda! Data! Splunk Components! Index Data! Proper Parsing! Challenging Data! Advanced Inputs 4

5 Are You in The Right Room?! You have used Splunk at least once, or at least read about it! You are interested in Splunk best pracgces! You like to use Splunk s default parsing rules! You just took over a Splunk deployment and you re not sure what to do! This is not an educagon class; it s best pracgce 5

6 Data Splunk is the engine for machine data! Machine data is more than just logs - it's configuragon data, data from APIs and message queues, change events, the output of diagnosgc commands and more! Log types: ApplicaGon, Web Access and Proxy, Call Detail Records (CDR), Clickstream, Message Queues, Packet, Database audit and tables, File audit, Syslog, WMI, PerfMon! Manual: Gecng Data In hdp://docs.splunk.com/documentagon/splunk/latest/data/ WhatSplunkcanmonitor 6

7 Splunk Apps! Look to Splunk Apps first and uglize Technical Add- On (TA)! Applies the Common InformaGon Model (CIM)! CIM details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data! Example TAs: Windows Unix Exchange AcGve Directory VMware Vcenter WebSphere 7

8 Splunk Distributed Components Search Head Deployment Server Indexer Forwarder 8

9 Test Environment! Every Splunk deployment should have a test environment! It can be a laptop, virtual machine or spare server! Should have the same version of Splunk running in producgon! Accessible to other Splunk developers and administrators 9

10 One Shot! Easiest way to get data into your test environment! Components of the oneshot:./splunk add oneshot user_conf.txt index indexname sourcetype sourcetype name! Where to find more informagon: hdp://docs.splunk.com/documentagon/splunk/latest/data/ MonitorfilesanddirectoriesusingtheCLI 10

11 Data - Broken 11

12 Props! Always set these six parameters # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

13 Props! Defaults to empty # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

14 Props! strpgme Style format # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

15 Props! By default MAX_TIMESTAMP_LOOKAHEAD = 150 characters # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

16 Props! By default set to True # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

17 Props! By default set to ([\r\n]+); change to posigve lookahead # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

18 Props! By default set to bytes; set to 0 to never truncate # USER CONFERENCE [user_conf_2014] TIME_PREFIX = ^ TIME_FORMAT = %Y- %m- %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = False LINE_BREAKER = ([\n\r]+)\d{4}- \d{2}- \d{2}\s\d{2}:\d{2}:\d{2} TRUNCATE =

19 Data - Fixed 19

20 6.2 Splunk Web Data On- Boarding

21 Why to Use Splunk Web to On- board? Quick and easy way to! Easily visualize the data into events rather then lines of text! Quickly get the data properly broken into events! Accurately get the Gme stamp extracted All in a wicked cool GUI Once everything is good you take your PROPS secngs and deploy 21

22 Splunk Web Data On- Boarding! Locate the source file on the Splunk Server s file system 22

23 Splunk Web Data On- Boarding! Validate event breaking and Gmestamp recognigon 23

24 Splunk Web Data On- Boarding! Resolve event breaking 24

25 Splunk Web Data On- Boarding! Set Gmestamp format even if Splunk figures it out automagcally 25

26 Splunk Web Data On- Boarding! Copy the props.conf secngs and deploy in a custom app 26

27 Challenging Data

28 Limit Indexed Data! Anonymize data: [source::.../accounts.log] SEDCMD- accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}- ){3}(\d{4})/cc=xxxx- xxxx- xxxx- \2/g! Rewrite raw data: [source::.../sql.log] SEDCMD- sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g! Discard events: props [source::/var/log/user_conf.txt] TRANSFORMS- null= setnull transforms [setnull] REGEX = (?i)debug DEST_KEY = queue FORMAT = nullqueue 28

29 Limit Indexed Data! Anonymize data: [source::.../accounts.log] SEDCMD- accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}- ){3}(\d{4})/cc=xxxx- xxxx- xxxx- \2/g! Rewrite raw data: [source::.../sql.log] SEDCMD- sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g! Discard events: props [source::/var/log/user_conf.txt] TRANSFORMS- null= setnull transforms [setnull] REGEX = (?i)debug DEST_KEY = queue FORMAT = nullqueue 29

30 Limit Indexed Data! Anonymize data: [source::.../accounts.log] SEDCMD- accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}- ){3}(\d{4})/cc=xxxx- xxxx- xxxx- \2/g! Rewrite raw data: [source::.../sql.log] SEDCMD- sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g! Discard events: props [source::/var/log/user_conf.txt] TRANSFORMS- null= setnull transforms [setnull] REGEX = (?i)debug DEST_KEY = queue FORMAT = nullqueue 30

31 Limit Indexed Data 6.X or later Windows forwarders! Whitelist events or blacklist specific events! Inputs.conf ConfiguraGon 31

32 Index ExtracGons! Provides reliable and consistent indexing of data with headers.! Address issue on forwarder: INDEX_EXTRACTIONS = {CSV W3C TSV PSV JSON}! Supports custom header parsing and easy mode for common formats.! Extract IIS fields using Props.conf on Windows forwarder: [iis] INDEX_EXTRACTIONS = w3c 32

33 MulGple Timestamps 12- Sep- 2014,09:01:00,12- Sep- 2014,09:02:00,- 4 INFO Gtle="User Conference" msg="splunk hosted user conference in Las Vegas." 12- Sep- 2014,19:01:00,12- Sep- 2014,19:02:00,- 5 DEBUG Gtle="User Conference" msg="gecng Data In, Correctly is a solid session." datepme.xml <datetime> <define name= two_tz" extract="day, litmonth, year, hour, minute, second, zone"> <text><![cdata[^(\d+)- (\w+)- (\d+),(\d+):(\d+):(\d+),(?:[^,]*,){2}([\w\- ]*)]]></text> </define> <timepatterns> <use name= two_tz"> </timepatterns> <datepatterns> <use name= two_tz"> </datepatterns> </datetime> props.conf # USER CONF [user_conf] DATETIME_CONFIG = /etc/apps/splk_ps_user_conf_props/local/datetime.xml * Do not set TIME_FORMAT 33

34 Database Connect

35 Database Connect! Allows for indexing data from database sources directly! Allows for adding meta data to events from database sources using lookups Caveats! Java required on Splunk server! Search head pooling requires custom configuragon to share the DB connecgon passwords. Not meant for data input sources 35

36 Database Connect Best PracGces! Normalize Gmestamps nagvely inside the SQL Query! Filter results down in SQL Query to reduce garbage in Splunk index! Repeated DBLookups should be converted to stagc lookup! Search head pooling requires encrypted password replicagon 36

37 Modular and Scripted Inputs

38 Modular and Scripted Inputs Benefits! Almost any program that can output text can be used to index! Modular inputs allow for configuragon files and configuragon secngs inside Splunk Differences! Scripted inputs require configuragon to be done in the script! Modular inputs can be configured via deployed.conf files and accessed via REST API! Scripted inputs need are specific to the OS deployed on where modular inputs can support mulgple Examples vmstat, iostat, Checkpoint Opsec, Twider, Stream, Amazon S3 Online storage and more 38

39 Scripted Inputs Example! Shell script saved in /opt/splunk/bin/scripts/ OR in a specific app! Allows you to execute any program on Splunk forwarder and index STDOUT data.! UGlizing key value pairs makes for easier searching. Sample output from custom script /Applica3ons/Splunk/bin/scripts/FantasyFootball.sh 39

40 Scripted Inputs Example Shell script calls local system binary programs and can provide configuragon opgons. Use Inputs.conf to define INDEX, SOURCETYPE, and INTERVAL for the scripted input 40

41 ProducGon Deployment

42 ProducGon Environment! Complexity managing configuragons across tens, hundreds, or thousands of forwarders! Not all indexers and search heads receive the same configuragons! Should think about version control for deployment apps, e.g., GitHub SHP 42

43 Deployment Server Terminology! Deployment Server - A Splunk instance that acts as a centralized configuragon manager, grouping together and collecgvely managing any number of Splunk instances. Any Splunk instance can act as a deployment server, even one that is indexing data locally. Splunk instances that are remotely configured by deployment servers are called deployment clients.! Deployment Client - A Splunk instance that is remotely configured by a deployment server.! Server Class - Represents a configuragon of Splunk deployment clients. Server classes enable the management of a group of deployment clients as a single unit. A server class can be used to group deployment clients together by applicagon, OS, data type to be indexed, or any other feature of your Splunk deployment. 43

44 Deployment App! A deployment app (configuragon bundle) is a set of deployment content (including configuragon files) deployed as a unit to clients of a server class! Located in $SPLUNK_HOME/etc/deployment- apps and pushed to deployment client s $SPLUNK_HOME/etc/apps folder! DO NOT store configuragons in $SPLUNK_HOME/etc/system/local! Use deployment apps regardless of your deployment tool 44

45 Deployment App - Naming ConvenGon org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer Base splk ps user_conf inputs 45

46 Deployment App - Naming ConvenGon org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer base splk ps user_conf inputs 46

47 Deployment App - Naming ConvenGon org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer base splk ps user_conf inputs 47

48 Deployment App - Naming ConvenGon org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer base splk ps user_conf inputs 48

49 Deployment App - Naming ConvenGon org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer base splk ps user_conf inputs 49

50 Deployment App - Naming ConvenGon splk_ps_user_conf_inputs org group applicagon configuragon acme finance apache inputs acme markegng iis props splk all indexer base splk ps user_conf inputs 50

51 Deployment Apps mba13:apps $ ls - la! SplunkForwarder! SplunkLightForwarder! Splunk_for_AcGveDirectory! Splunk_for_Exchange! splk_all_deploymentclient! splk_all_forwarder_outputs! splk_all_indexer_base! splk_all_search_base! splk_ps_user_conf_inputs! splk_ps_user_conf_props! splk_ps_user_conf_web! splunk_app_was user- prefs 51

52 CollecGng Syslog! Send device, e.g., routers, firewalls to a syslog collector! Write files to this directory structure: /sourcetype/host/log.txt! Monitor the sourcetype level cisco_asa my.firewall.name # CISCO ASA [monitor:///data/cisco_asa/ /] sourcetype = cisco_asa host_segment = 3 index = firewall 52

53 Summary! Test in a non- producgon environment! Always use key props parameters: TIME_PREFIX TIME_FORMAT MAX_TIMESTAMP_LOOKAHEAD SHOULD_LINEMERGE LINE_BREAKER TRUNCATE! Deploy apps to /etc/apps; not /etc/system/local! Clear predictable naming convengon! When you re stuck, use Answers and Re- Use apps from Apps.Splunk.com 53

54 Resources! Get educated: hdp:// CAAAAH9! Download Splunk applicagons: hdp://apps.splunk.com/! Hire Splunk Professional Services: hdp:// services/sp- CAAABH9! Watch some videos: hdp:// 54

55 THANK YOU