Copyright 2014 Splunk Inc. Search in 500 easy steps. Julian Harty. SE, Splunk>

Transcription

1 Copyright 2014 Splunk Inc. Search in 500 easy steps Julian Harty SE, Splunk>

2 Disclaimer During the course of this we may make forward looking statements regarding future events or the expected performance of the company. We you that such statements reflect our current and based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this are being made as of and date of its live If reviewed aser its live this may not contain current or accurate We do not assume any to update any forward looking statements we may make. In any about our roadmap outlines our general product and is subject to change at without It is for purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no either to develop the features or described or to include any such feature or in a future release. 2

3 Am I in the right Session and Who is this guy? Goal of Presenta:on: Search Op:miza:on How the hell do I speed this search up? Background of your Presenter: Julian Harty Splunker for 2+ Years - Variety of installa@ons from 10GB to 100TB s+ Ex- Oracle/MySQL DBA (Recovering) Contact info julian@splunk.com 3

4 Background Great to Not So Great Growth without = performance - > our goal: gejng great performance at scale Op@miza@on Steps More Data More Users New Searches Even More Data Even More Users Even More Searches 4

5 Challenge Why so slow? The maturity of a Splunk deployment Question? Has your deployment been architected correctly? Solution: Architecting And Designing Your Splunk Deployment - Simeon Yep Question? Is your environment tuned correctly? Solution: Jiffy Lube Quick Tune Up For Your Splunk Environment Sean Delaney 5 Question? Are your searches optimized? Solution: Welcome to this session!!!

6 Agenda: of this Session The Basics: Common pinalls - Best prac@ces and what not to do Take away: Basic steps to a beoer search Beyond the Basics: Search Architecture and Workflow Detailed Search review using Job inspector search examples Take away: Job Inspector Cheat- Sheet Q&A 6

7 Poorly Performing Searches

8 End User Enquiries 8

9 SOS Expensive Searches Search Usage Paoerns - > SOS > Search - > Search Detail Ac@vity - > Expensive Searches 9

10 For Splunk 6.2 Users Index 10

11 Search Tuning The Basics

12 The Basics: Common Search Behavior Bad Behavior Good Behavior Performance Comment Improvement > AND be=specific All Time Searches - 24h@h 365x 30x Limit Time Range >* index=xyz source=www range 10-50% > foo bar Index and default fields > foo search bar > foo bar 30% Combine Searches > host=web sourcetype=access* Verbose Mode Fast/Smart 20-50% Fast Mode Use Fast/Smart Mode where Possible A NOT B A AND C AND D AND E 5-50% Avoid NOTS Searches over large datasets Searches over long periods Data Models and Report 1000% Use Intelligently Accelera@on Use Summary Indexing Use Report Accel or Summary Indexing Summary Indexing 1000% Use Sparingly 12

13 The Basics: Common Mistakes Summary indexing is Awesome! - Summarize EVERYTHING!!! ê Summarizing too much data negates the point Report Accelerate = Turbo buoon Ini@al reac@on - Report Accelerate EVERYTHING!!! ê Too many searches = skipped search issues Data Models are the answer! Ini@al reac@on everything can be included! ê Convoluted data models can increase workload 13

14 OK, But How can you enforce these

15 How do you enforce Best Architect Perspec:ve: User Best for Users Admin Perspec:ve: User Controls: Pulling in the reins Role Limit index Limit search terms Limit range Power user role Restrict Number of RT+ Concurrent Searches 15

16 How do you enforce Best Admin Perspec:ve: Time range defaults (ui- prefs.conf) Time range Web dropdown (Times.conf) 16

17 OK Now More advanced Lets start with - the skinny on How Search Works

18 How Search Works Physical history db_ _ _1 _internal main db_lt_et_4 db_lt_et_1 db_lt_et_3.tsidx Sources.data.gz.gz.gz.gz.gz.gz.gz.gz SourceTypes.data db_lt_et_2 Hosts.data 18

19 How Search Works - Logically Search Query Structure Parse, Fetch, Summarize, Display Index=mydata eval loc=long+lat+alt stats count retrieve events filter/transform/map 19

20 Splunk Distributed Search 4 Steps to a Splunk Search: Parse, Fetch, Summarize, Display " StreamingCommand: Applies a transforma@on to search results as they travel through the processing pipeline. Eval rex where " Repor:ngCommand: Processes search results and generates a repor@ng data structure. Examples: stats, top, 20

21 Types of Searches Dense Low cardinality Example: sourcetype=access method=get Sparse High cardinality Example: sourcetype=access method=get Super Sparse (or Needle in a Haystack) Very high cardinality Example: sourcetype=cisco:asa ac@on=denied src= Rare Use Case: user behavior tracking Example: sourcetype=magicsource rare Dense Sparse Super Sparse 21

22 Dense Searches (>10% matching results) (scancount vs eventcount in Job Inspector) > sourcetype=access_combined method=get Challenge: CPU and I/O- bound spike in CPU due to decompression of raw events. Retrieval rate: 50K events per second per server Solu:on: Divide and conquer Distribute search to an indexing cluster Parallel compute and merge results Report or use of Summaries divide and Conquer Report on summarized data vs. raw data 22

23 Sparse Searches > sourcetype=access_combined status=404 Challenge: CPU- bound Dominant cost is uncompressing *.gz raw data files need to read far into a file to retrieve a few events Solu:on: Avoid cherry picking Be selec@ve about exclusions (avoid NOT foo or field!=value ) Leverage indexed fields Filter using whole terms Instead of > sourcetype=access_combined clientip= *! Use > sourcetype=access_combined clientip=term( )! 23

24 Super Sparse Searches > sourcetype=access_combined status= Needle in Haystack Very I/O intensive May take up to 2 Seconds to parse each bucket 24

25 Rare Term Searches > sourcetype=access_combined sessionid=1234 Bloom Filters* Bloom filters stored in each bucket 50- buckets processed per second I/Os reduced as buckets are excluded from to just a few x faster than Super Sparse searches on conven@onal storage, >1000x faster on SSD (Due to random reads) * A Bloom filter is a data structure designed to tell you whether or not an element is present in a set 25

26 How can I determine if my search is Dense or Sparse? Use Job Inspector Component Descrip:on scancount eventcount The number of events that are scanned or read off disk. Number of events that are returned to base search For dense searches scancount ~= eventcount. For sparse searches, scancount >> eventcount. > sourcetype=access_combined status=

27 Job Inspector Review

28 Job Inspector Measuring Search Using the Splunk Search Inspector Key Metrics: Time Number of Events Scanned Search SID * Timings from the search command Timings from distributed peers 28

29 Job Inspector Walkthrough Search Command Rawdata: Improving I/O and CPU load KV: Are field efficient Lookups: Used appropriately Autolookups causing issues Typer: Inefficient Evenoypes Alias: Cascading alias 29

30 Reading Job Inspector - Search.Index Search.index = Time to parse and read the tsidx files to determine where to read in rawdata How do you op:mize this? Improving I/O 30

31 Reading Job Inspector - search.rawdata Search.rawdata = Time to read actual events from rawdata files How do you op:mize this? Filtering as much as possible Add Peers Alloca@ng more CPU, improving I/O 31

32 Reading Job Inspector - search.kv Search.KV= Time taken to apply field extrac@ons to events How do you op:mize this? Regex op@miza@ons Avoid greedy operators.*? Use of Anchors ^ $ Non Capturing groups for repeats 32

33 Reading Job Inspector - search.lookups Search.lookups = Time to apply lookups to search How do you op:mize this? Use Appropriately (at end of search) Autolookups maybe causing issues 33

34 Reading Job Inspector - search.typer and tags Search.typer = Time to apply event types to the search How do you op:mize this? Use Appropriately Removed unused tags and evenoypes 34

35 Job Inspector Walkthrough Distributed Search Dispatch.createProviderQueue Time to establish with peers Dispatch.fetch Time spent to fetch events Dispatch.evaluate spent parsing the search and sejng up the data structures needed to run the search. How do you op:mize this? Improving Peer Improve Bundle Faster storage 35

36 Job Inspector Walkthrough Distributed Search Dispatch.stream.remote Time to retrieve events from each remove peer Issue: 1. Unequal Indexer performance Either Hardware mismatch Uneven of indexes 2. AutoLB issues 36

37 Component Job Inspector Conclusions: Search Command Summary Descrip:on index rawdata kv filter alias lookups typer tags look in tsidx files for where to read in rawdata read actual events from rawdata files apply fields to the events filter out events that don t match (e.g., fields, phrases) rename fields according to props.conf create new fields based on exis@ng field values assign evenoypes to events assign tags to events 37

38 Job Inspector Conclusion: Distributed Search Summary Metric Descrip:on Area to review createprovider Queue fetch stream.remote evaluate to connect to all search peers. spent for or fetching events from search peers. spent the remote search in a distributed search environment, aggregated across all peers. spent parsing the search and sejng up the data structures needed to run the search. Peer conduc@vity Faster Storage Possible bundle issues 38

39 Key Logfiles related to search Search log: " Stored in $SPLUNK_HOME/var/run/splunk/dispatch/ " Detailed analysis of every step taken by the search " Search stack trace 39

40 What is the best search command to use?

41 Stats vs Search Goal: compute on the of web session (JSESSIONID=unique Not so Great: Much BeUer: > sourcetype=access_combined JSESSIONID chart count by span=log2 > stats as by JSESSIONID chart count by span=log2 41

42 Dedup vs Latest Note: dedup can't be used with report Search Goal: Return latest cart for each web site customer Not so Great: > sourcetype=access* dedup sortby - _@me table clien@p, ac@on Much BeUer: > sourcetype=access* stats latest(clien@p) by ac@on 42

43 Joins and Subsearches Search Goal: Return latest JESSIONID across two sourcetypes Not so Great: > sourcetype="access_combined" join type="inner" JSESSIONID [search sourcetype="applogs" dedup JSESSIONID table JSESSIONID, othervalue] Much BeUer: > (sourcetype="access_combined") OR (sourcetype="applogs") stats latest(*) as * by JSESSIONID 43

44 Wrap- up

45 In Closing 1. Architecture best for performance at scale With search behavior in mind 2. User Onboarding Best Basic steps 3. Periodic Performance Review Applying technologies where appropriate Removing unused searches 4. Review sides for Search flow detail Splunk Web 45

46 And By the way Other Sessions to look out for: How to Actually Use Splunk Data Models - David Clawson Presented on Tuesday Check out the session notes Jiffy Lube Tune- Up for your Splunk Deployment - Sean Delaney Presented on Tuesday Check out the session notes ArchitecCng and Sizing your Splunk Environment - Simeon Yep 2:15-3:15 Today Splunk Search AcceleraCon Technologies Gerald Kanapathy 10:30-11:30 Tomorrow My Contact informa:on: 46

47 THANK YOU

48 Take Away: Basic Steps to a beoer search Avoid use of * where ever possible. Avoid the use of All Time. Avoid subsearches searches. Incorporate the use default fields (source, sourcetype, host) as well as specific indexes to every search (where possible). Use Fast or Smart mode where possible avoid Verbose mode. Use Report Accelera@on Sparingly (and Strategically) on reports on large datasets. Use Summary Indexing when building reports spans beyond target index reten@on. Use Job Inspector and Search inspector to get more info (hold on for more details!!!) 48

49 A few notes on how to Splunk Web fields Collapse Timeline Use Fast Mode Change 49

50 Search flow Local and Distributed Key Flow: 1. Find which Bundle to use 2. Find Buckets to use range) 3. LISPY TSIDX search 4. Process + Summarizes Events Key Files: Info Status Results Preview hop://wiki.splunk.com/community:howdistsearchworks 50