Copyright 2014 Splunk Inc. Taming Your Data. Mark Runals Sr Security Engineer The Ohio State University

Size: px

Start display at page:

Debra Goodwin
5 years ago
Views:

2 Disclaimer During the course of this presentafon, we may make forward- looking statements regarding future events or the expected performance of the company. We caufon you that such statements reflect our current expectafons and esfmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentafon are being made as of the Fme and date of its live presentafon. If reviewed arer its live presentafon, this presentafon may not contain current or accurate informafon. We do not assume any obligafon to update any forward- looking statements we may make. In addifon, any informafon about our roadmap outlines our general product direcfon and is subject to change at any Fme without nofce. It is for informafonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligafon either to develop the features or funcfonality described or to include any such feature or funcfonality in a future release. 2

3 Disclaimer During the course of this presentafon, we may make forward looking statements regarding future events or the expected performance of the company. We caufon you that such statements reflect our current expectafons and esfmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentafon are being made as of the Fme and date of its live presentafon. If reviewed arer its live presentafon, this presentafon may not contain current or accurate informafon. We do not assume any obligafon to update any forward looking statements we may make. In addifon, any informafon about our roadmap outlines our general product direcfon and is subject to change at any Fme without nofce. It is for informafonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligafon either to develop the features or funcfonality described or to include any such feature or funcfonality in a future release. 3

4 Agenda! OSU Splunk deployment environmental background! Props/field extracfon score methodology! Look at data curator app FYI - Splunk Admin Focused PresentaFon 4

+ Desired lightweight onboarding process For

5 Some Background & Program Drivers OSU Environment 135 Distributed IT units around OSU Each group is autonomous No standardizafon Huge variety of technologies Splunk use not mandatory + Desired lightweight onboarding process For units & for Splunk team = Incredible roll- on/ adopfon rate 5

6 Fast Forward a Year or 2 +/-! 2TB Of data! 1,800+ Splunk agents! 10k Devices! 12 Types of firewalls! MulFple OS! 90+ Teams with data in Splunk! 700+ Sourcetypes many learned! 350+ People 6

7 Fast Forward a Year or 2 +/-! 2TB Of data! 1,800+ Splunk agents! 10k Devices! 12 Types of firewalls! MulFple OS! 90+ Teams with data in Splunk! 700+ Sourcetypes many learned! 350+ People Is data being ingested correctly? What fields have been defined? Where? What types of data are in Splunk? What s not configured correctly? 7

8 Issue Overview Out of the box and without specific data definifon Splunk will generally ingest data correctly Host names Sourcetypes Timestamp Line breaking Auto key- value fields At best though, this isn t efficient. At worst, it can strain your deployment and may drop/lose events Factors in play Hardware RaFo of indexers to total log volume Sourcetype velocity Data distribufon (forwarders pre will favor first indexer listed in autolb outputs.conf) Weird date/fme informafon in your logs Etc 8

Data Import/DefiniFon Pipeline (Mark s View) Get Data

Index Time Processing Sourcetyping Line breaking

Base level field extracfon Normalized field names Field

9 Data Import/DefiniFon Pipeline (Mark s View) Get Data to Splunk Data Management Knowledge Management DM = Index Time Processing Sourcetyping Line breaking Timestamp Host field etc KM = Search Time Processing Base level field extracfon Normalized field names Field name alignment within Common InformaFon Model (CIM) Knowledge objects 9

10 The Plan Data Management Score based on Gepng Data in Correctly.conf 2012 preso Data Curator App Knowledge Management Score based on length of fields relafve to _raw length (conversafon with Kevin Meeks) IdenFfy Common Issues Munge through internal logs Data Taxonomy Create way to classify sourcetypes 10

11 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = LINE_BREAKER = TRUNCATE = TZ = 11

12 Data Management Props Score [mah_data_stanza] TIME_PREFIX = +1 MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = +1 SHOULD_LINEMERGE = LINE_BREAKER = TRUNCATE = TZ = +1 OR DATETIME_CONFIG = +3 12

13 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = False +1 LINE_BREAKER = TRUNCATE = TZ =.but what if my data should be merged? 13

14 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = AND SHOULD_LINEMERGE = True LINE_BREAKER = TRUNCATE = TZ = +1 One of these is populated BREAK_ONLY_BEFORE MUST_BREAK_AFTER MUST_NOT_BREAK_BEFORE MUST_NOT_BREAK_AFTER 14

15 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = LINE_BREAKER = +1 TRUNCATE = TZ = Default is ([\r\n\]+) Don t want to line break? ((?!)) or ((*FAIL)) are a couple opfons* *hyp://answers.splunk.com/answers/106075/each- file- as- one- single- splunk- event 15

16 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = LINE_BREAKER = TRUNCATE = +1 TZ = Default is Game your score! Ø Set this to anything other than the default i.e or

17 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = LINE_BREAKER = TRUNCATE = TZ = +1 If sepng this across your environment isn t possible/pracfcal reduce the max score macro in the app. It s used as a variable. Macro: props_score_upper_bounds = \

18 Data Management Props Score [mah_data_stanza] TIME_PREFIX = MAX_TIMESTAMP_LOOKAHEAD = TIME_FORMAT = SHOULD_LINEMERGE = LINE_BREAKER = TRUNCATE = TZ = Max Score = 7 (st_score * `props_score_scale`) / `props_score_upper_bounds` 10 18

19 Props Score Caveats There are a lot of addifonal props sepngs that could be applicable for your data/environment. This method/app doesn t address host fields that are incorrect syslog Default host field? Splunk UF 19

20 Props Score Caveats There are a lot of addifonal props sepngs that could be applicable for your data/environment. This method/app doesn t address host fields that are incorrect syslog Default host field? Splunk UF 20

21 Field ExtracFon Score Methodology [20/Aug/2014:13:44: ] "POST /services/broker/phonehome/ connecfon_ _8089_ _test- TS_68D CC1D CA- 6E24F9FE6538 HTTP/1.0" ms Length of Fields _raw length 1. Account for any autokv field names 2. Do convoluted search to get length of fields 3. Account for Fmestamp in log 4. Get total length 1. Remove spaces 2. Remove newline characters 3. Get _raw length = % of Event has Fields Defined 21

22 Field ExtracFon Score Methodology [20/Aug/2014:13:44: ] "POST /services/broker/phonehome/ connecfon_ _8089_ _test- TS_68D CC1D CA- 6E24F9FE6538 HTTP/1.0" ms Length of Fields _raw length 1. Account for any autokv field names 2. Do convoluted search to get length of fields 3. Account for Fmestamp in log 4. Get total length 1. Remove spaces 2. Remove newline characters 3. Get _raw length = % of Event has Fields Defined 22

23 Field ExtracFon Score Methodology [20/Aug/2014:13:44: ] "POST /services/broker/phonehome/ connecfon_ _8089_ _test- TS_68D CC1D CA- 6E24F9FE6538 HTTP/1.0" ms Length of Fields _raw length 1. Account for any autokv field names 2. Do convoluted search to get length of fields 3. Account for Fmestamp in log 4. Get total length 1. Remove spaces 2. Remove newline characters 3. Get _raw length = % of Event has Fields Defined * Not a great example Splunk forwarder phonehome logs actually have +100% field length compared to _raw 23

24 Field ExtracFon Score Methodology Caveats/ConsideraFons Doesn t account for field alias (will arfficially inflate score) If field extracfon % is over 100 the score is set to 100 DirecFonally correct is about the best this will get Ø Fields extracted!= field value 24

25 Version 1 deprecated out of the box Data Taxonomy Designed to answer What type of data is in Splunk? Created a 2 nd field classificafon csv for several hundred sourcetypes Data family Data subtype Very useful but too many one- to- many relafonships based on data use netstat ConfiguraFon? Networking? Server Monitoring Server InformaFon Server ConfiguraFon Server Performance Too many server * 25

26 Data Taxonomy InteracFve Host Dashboard Host A 26

27 Data Taxonomy InteracFve Host Dashboard Host B 27

28 Data Curator App Goals Flexible scoring scale Generate aggregate, system maturity scores Generate ~accurate individual maturity score Show what app/package contained props sepngs Show current props sepngs Highlight issues related to/solvable by props sepngs Line breaking Timestamp Transforms issues Take Note! Will NOT tell you what the sepngs should be Requires Splunk 6 search head Only able to work through issues I saw in my environment - you may have others. I can troubleshoot my app not your deployment =) 28

29 Deployment At A Glance 29

30 Props Score Breakdown Holy Crap!! Lots of Work.but before you slit your wrists 30

31 Props Score Breakdown 31

32 Learned Sourcetypes (- too_small OR - #) Beware of diminishing returns on working the long tail 32

33 Sourcetype Deep Dive Dashboard Avamar Logs 33

34 Sourcetype Deep Dive Dashboard Avamar Logs Not all items factor into score 34

35 Sourcetype Deep Dive Dashboard Avamar Logs Loaded score based on volume of events per punct. Score created on the fly 35

36 Sourcetype Deep Dive Dashboard Avamar Logs Based on volume of events per punct. Quick way to see how unique logs in a parfcular sourcetype are. Had 75 unique punct 36

37 Sourcetype Deep Dive Dashboard ABDCB (learned) 37

38 Sourcetype Deep Dive Dashboard Argus 38

39 IdenFfying Date/Time Issues 39

40 IdenFfying Date/Time Issues These events don t have Fmestamps! 40

41 IdenFfying Date/Time Issues These events don t have Fmestamps! What if Splunk thinks the last known good Fmestamp was 6 years ago? 41

42 IdenFfying Date/Time Issues These events don t have Fmestamps! What if Splunk thinks the last known good Fmestamp was 6 years ago? 42

43 Date/Time Workspace Dashboard Pre- populated with sourcetypes having issues (No Fme informafon set) AddiFonal Dashboard Elements Clustered internal logs giving you a level of visibility 100 most recent events (DATETIME_CONFIG added to view arer screenshot) 43

44 Line Breaking/Truncate Workspace Dashboard 44

45 Line Breaking/Truncate Workspace Dashboard 45

46 Line Breaking Sanity Check Dashboard Sourcetypes have line breaking set but have mulfple line counts in recent events 46

47 Line Breaking Sanity Check Sourcetypes have line breaking set but have mulfple line counts in recent events Set in mulfple apps; potenfal problem down the road? 47

48 Query TroubleshooFng Two main scheduled searches that are somewhat computafonally expensive. Dashboard allows admin to compare run length & frequency to coverage Sourcetype field length percentage query 48

49 Extract/Report/Transforms Issues Example Internal Warning Logs :55: WARN SearchOperator:kv - IndexOutOfBounds invalid The FORMAT capturing group id: id=7, transform_name='message' :59: WARN SearchOperator:kv - Invalid key- value parser, ignoring it, transform_name='extract_cmd_change' :59: WARN SearchOperator:kv - Invalid key- value parser, ignoring it, transform_name='(?i)^(?:[^\ ]*\ ){3}(?P<dest_domain>[^\ ]+)' wut? Which app? In props or transforms? SoluFon: grep - r through 520+ packages in deployment- apps directory for Message? 49

50 Extract/Report/Transforms Issues 50

51 Extract/Report/Transforms Issues Only 5 tokens 51

52 Extract/Report/Transforms Issues Anyone know what the issue is? 52

53 Extract/Report/Transforms Issues Should be an EXTRACT 53

54 KM Sourcetype Fields Comparison Boyom of explanatory text. There is a freeform text search box at top of dashboard 54

55 App Roadmap Now Props maturity scores Field extracfon scores Issues workspaces Data taxonomy RelaFvely non- scaling Next Dashboard opfmizafon (ie searchtemplate) Tag based data taxonomy Any inifal app bug fixes ARer Next Tie in data model fields Field value? Expand issue troubleshoofng Based on community feedback 55

56 ?.conf 14 updated Ge8ng Data in Correctly presentafon Andrew Duca Blog: runals.blogspot.com Check out the Forwarder Health app in Splunkbase 56

57 THANK YOU

Search Language Intermediate Lincoln Bowser

Copyright 2013 Splunk Inc. Search Language Intermediate Lincoln Bowser Sr. Technical Instructor, Splunk #splunkconf Legal NoFces During the course of this presentafon, we may make forward- looking statements