Onboard Data into Splunk, Correctly

Transcription

1 Copyright 2013 Splunk Inc. Onboard Data into Splunk, Correctly Ma:hew Se=pane Professional Services Manager, Splunk #splunkconf

2 Legal NoJces During the course of this presentajon, we may make forward- looking statements regarding future events or the expected performance of the company. We caujon you that such statements reflect our current expectajons and esjmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in this presentajon are being made as of the Jme and date of its live presentajon. If reviewed auer its live presentajon, this presentajon may not contain current or accurate informajon. We do not assume any obligajon to update any forward- looking statements we may make. In addijon, any informajon about our roadmap outlines our general product direcjon and is subject to change at any Jme without nojce. It is for informajonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligajon either to develop the features or funcjonality described or to include any such feature or funcjonality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respeccve owners Splunk Inc. All rights reserved. 2

3 About Me! Professional Services Manager! More than three years of Splunk experience! Involved in more than 300 deployments from 1GB to 10TB 3

4 Agenda! Data! Splunk Components! Index Data! Proper Parsing! Deploying in ProducJon! Deployment Apps and Naming ConvenJons! Challenging Data 4

5 Are You in The Right Room! You have used Splunk at least once, or at least read about it! You are interested in Splunk best pracjces! You like to use Splunk s default parsing rules! You just took over a Splunk deployment and you re not sure what to do! This is not an educajon class; it s best pracjce 5

6 Data Splunk is the engine for machine data! Machine data is more than just logs it's configurajon data, data from APIs and message queues, change events, the output of diagnosjc commands and more! Log types: ApplicaJon, Web Access and Proxy, Call Detail Records (CDR), Clickstream, Message Queues, Packet, Database audit and tables, File audit, Syslog, WMI, PerfMon! Manual: Ge=ng Data In: h:p://docs.splunk.com/documentajon/splunk/latest/data/ WhatSplunkcanmonitor 6

7 Splunk Distributed Components Search Head Deployment Server Indexer Forwarder 7

8 Test Environment! Every Splunk deployment should have a test environment! It can be a laptop, virtual machine or spare server! Should have the same version of Splunk running in producjon! Accessible to other Splunk developers and administrators 8

9 One Shot! Easiest way to get data into your test environment! Components of the oneshot:./splunk add oneshot user_conf.txt index indexname soucetype sourcetype name!! Where to find more informajon: h:p://docs.splunk.com/documentajon/splunk/latest/data/ MonitorfilesanddirectoriesusingtheCLI 9

10 Data Broken 10

11 Splunk Apps! Look to Splunk Apps first and ujlize Technical Add- On (TA)! Applies the Common InformaJon Model (CIM)! CIM details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data! Example TAs: Windows Unix Exchange AcJve Directory VMware Vcenter WebSphere 11

12 Props! Always set these six parameters # USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 12

13 Props! Defaults to empty # USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 13

14 Props! strpjme style format!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 14

15 Props! By default MAX_TIMESTAMP_LOOKAHEAD = 150 characters!!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 15

16 Props! By default set to True!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 16

17 Props! By default set to ([\r\n]+); change to posijve lookahead!!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 17

18 Props! By default set to bytes; set to 0 to never truncate!!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER = ([\n\r]+)(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!truncate = ! 18

19 Props! By default set to bytes; set to 0 to never truncate!!# USER CONFERENCE!!![user_conf_2012]!!!TIME_PREFIX = ^!!!TIME_FORMAT = %Y-%m-%d %H:%M:%S!!!MAX_TIMESTAMP_LOOKAHEAD = 19!!!SHOULD_LINEMERGE = False!!!LINE_BREAKER=([\n\r]+)(?=\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2})!!!TRUNCATE = ! 19

20 Data Fixed 20

21 ProducJon Deployment

22 ProducJon Environment! Complexity managing configurajons across tens, hundreds, or thousands of forwarders SHP! Not all indexers and search heads receive the same configurajons! Should think about version control for deployment apps, e.g., GitHub 22

23 Deployment Server Terminology! Deployment Server A Splunk instance that acts as a centralized configurajon manager, grouping together and collecjvely managing any number of Splunk instances. Any Splunk instance can act as a deployment server, even one that is indexing data locally. Splunk instances that are remotely configured by deployment servers are called deployment clients! Deployment Client A Splunk instance that is remotely configured by a deployment server! Server Class Represents a configurajon of Splunk deployment clients. Server classes enable the management of a group of deployment clients as a single unit. A server class can be used to group deployment clients together by applicajon, OS, data type to be indexed, or any other feature of your Splunk deployment 23

24 Deployment App! A deployment app (configurajon bundle) is a set of deployment content (including configurajon files) deployed as a unit to clients of a server class! Located in $SPLUNK_HOME/etc/deployment- apps and pushed to deployment client s $SPLUNK_HOME/etc/apps folder! DO NOT store configurajons in $SPLUNK_HOME/etc/system/local! Use deployment apps regardless of your deployment tool 24

25 Deployment App Naming ConvenJon org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer Base splk ps user_conf inputs 25

26 Deployment App Naming ConvenJon org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer base splk ps user_conf inputs 26

27 Deployment App Naming ConvenJon org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer base splk ps user_conf inputs 27

28 Deployment App Naming ConvenJon org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer base splk ps user_conf inputs 28

29 Deployment App Naming ConvenJon org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer base splk ps user_conf inputs 29

30 Deployment App Naming ConvenJon splk_ps_user_conf_inputs org group applicajon configurajon acme finance apache inputs acme markejng iis props splk all indexer base splk ps user_conf inputs 30

31 Deployment Apps msettipane-mba13:apps msettipane$ ls -la!! SplunkForwarder! SplunkLightForwarder! Splunk_for_AcJveDirectory! Splunk_for_Exchange! splk_all_deploymentclient! splk_all_forwarder_outputs! splk_all_indexer_base! splk_all_search_base! splk_ps_user_conf_inputs! splk_ps_user_conf_props! splk_ps_user_conf_web! splunk_app_was! user- prefs 31

32 Challenging Data

33 Limit Indexed Data! Anonymize data: [source::.../accounts.log]!!sedcmd-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!! Rewrite raw data: [source::.../sql.log]!!sedcmd-sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g!! Discard events:!props!![source::/var/log/user_conf.txt]!!transforms-null= setnull! transforms! [setnull]! REGEX =!(?i)debug! DEST_KEY =!queue! FORMAT =!nullqueue! 33

34 Limit Indexed Data! Anonymize data:![source::.../accounts.log]!!sedcmd-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!! Rewrite raw data:![source::.../sql.log]!!sedcmd-sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g!! Discard events:!props!![source::/var/log/user_conf.txt]!!transforms-null= setnull! transforms! [setnull]! REGEX =!(?i)debug! DEST_KEY =!queue! FORMAT =!nullqueue! 34

35 Limit Indexed Data! Anonymize data:![source::.../accounts.log]!!sedcmd-accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-){3}(\d{4})/cc=xxxx-xxxx-xxxx-\2/g!! Rewrite raw data:![source::.../sql.log]!!sedcmd-sqllog = s/(.*?)command:execute[.\d\d\w\w]*/\1/g!! Discard events:!props!![source::/var/log/user_conf.txt]!!transforms-null= setnull! transforms! [setnull]! REGEX =!(?i)debug! DEST_KEY =!queue! FORMAT =!nullqueue! 35

36 CollecJng Syslog! Send device, e.g., routers, firewalls to a syslog collector! Write files to this directory structure: /sourcetype/host/log.txt! Monitor the sourcetype level cisco_asa my.firewall.name # CISCO ASA! [monitor:///data/cisco_asa/ /]! sourcetype = cisco_asa! host_segment = 3! index = firewall!! 36

37 Check for Header! Steps to fixing sourcetype- 2, 3, 4 problems (e.g., iis- 2, iis- 3)! Address issue on forwarder: CHECK_FOR_HEADER = False! Extract fields using delimiter: [sourcetype]! DELIM =,! FIELDS = one, two, three!! On search head rename already indexed events: rename = iis 37

38 MulJple Timestamps 12- Sep- 2012,09:01:00,12- Sep- 2012,09:02:00,- 4 INFO Jtle="User Conference" msg="splunk hosted user conference in Las Vegas." 12- Sep- 2012,19:01:00,12- Sep- 2012,19:02:00,- 5 DEBUG Jtle="User Conference" msg="ge=ng Data In, Correctly is a solid session." datelme.xml <datetime>! <define name= two_tz" extract="day, litmonth, year, hour, minute, second, zone">! <text><![cdata[^(\d+)-(\w+)-(\d+),(\d+):(\d+):(\d+),(?:[^,]*,){2}([\w\-]*)]]></text>! </define>! <timepatterns>! <use name= two_tz">! </timepatterns>! <datepatterns>! <use name= two_tz">! </datepatterns>! </datetime>! props.conf # USER CONF! [user_conf]! DATETIME_CONFIG = /etc/apps/splk_ps_user_conf_props/local/datetime.xml! * Do not set TIME_FORMAT 38

39 Summary! Test in a non- producjon environment! Always use key props parameters: TIME_PREFIX TIME_FORMAT MAX_TIMESTAMP_LOOKAHEAD SHOULD_LINEMERGE LINE_BREAKER TRUNCATE! Deploy apps to /etc/apps; not /etc/system/local! Clear predictable naming convenjon! When you re stuck, use Splunk Answers 39

40 Resources! Get educated: h:p:// CAAAAH9! Download Splunk applicajons: h:p://splunk- base.splunk.com/apps/! Hire Splunk Professional Services: h:p:// services/sp- CAAABH9! Watch some videos: h:p:// 40

41 Next Steps 1 2 Download the.conf2013 Mobile App If not iphone, ipad or Android, use the Web App Take the survey & WIN A PASS FOR.CONF2014 Or one of these bags! 3 Go to the Search Party! Marquee Nightclub at The Cosmopolitan Today, 7:30-10:30pm 41

42 THANK YOU