Understanding Splunk AcceleraGon Technologies David Marquardt

Transcription

1 Copyright 2013 Splunk Inc. Understanding Splunk AcceleraGon Technologies David Marquardt Senior So?ware Engineer #splunkconf

2 Legal NoGces During the course of this presentagon, we may make forward- looking statements regarding future events or the expected performance of the company. We caugon you that such statements reflect our current expectagons and esgmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in this presentagon are being made as of the Gme and date of its live presentagon. If reviewed a?er its live presentagon, this presentagon may not contain current or accurate informagon. We do not assume any obligagon to update any forward- looking statements we may make. In addigon, any informagon about our roadmap outlines our general product direcgon and is subject to change at any Gme without nogce. It is for informagonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligagon either to develop the features or funcgonality described or to include any such feature or funcgonality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respeccve owners Splunk Inc. All rights reserved. 2

3 About Me! Been coding core splunkd for over 5 years! Worked on various components: eval/where commands MulG- index search AuthenGcaGon/authorizaGon Rawdata Now high performance analygcs store 3

4 Agenda! Overview Current Index Structure! Review How ReporGng is Currently Done! How We Can Do Be`er! Demo 4

5 Splunk Enterprise Index Structure IDX 3 IDX 2 IDX 1 Source/Sourcetype/Host Metadata Home Path 1 source : : /my/log 2 source: : /blah et et lt lt it it hot_v1_100 *.data *.tsidx rawdata TSIDX cream apple beer coke ice java LEXICON hot_v1_101 apple POSTING db_lt_et_101 beer Cold Path db_lt_et_80 Thawed Path db_lt_et_70 Rawdata an apple a day keeps doctor away apple pie and ice cream is delicious 5

6 TSIDX? What? Time series index! Inverted index opgmized for Gme! Two basic components Lexicon Arrays of informagon about events Why: Given a Gme range and query, where s my matching data? 6

7 Lexicon Raw Events Deep likes Bud light Amrit likes Makers Ledion likes cognac Dave likes Jack Daniels Zhang likes vodka Deep likes Makers Dave likes Makers Term PosCngs List Amrit 1 Bud 0 Daniels 3 Dave 3,6 Deep 0,5 Jack 3 Ledion 2 Makers 1,5,6 Zhang 4 cognac 2 likes 0,1,2,3,4,5,6 light 0 vodka 4 7

8 Values Arrays PosCng value Seek address _Cme host source sourcetype Raw events Deep likes Bud light Amrit likes Makers Ledion likes cognac Dave likes Jack Daniels Zhang likes vodka Deep likes Makers Dave likes Makers 8

9 Okay, How Do I Search? Query: likes (vodka OR cognac) STEP 1: Consult the lex, combining posgngs lists! Doing an OR? Use a union! Doing an AND? Use an intersecgon vodka OR cognac = (4) U (2) = (2, 4) likes (vodka OR cognac) = (0,1,2,3,4,5,6) int. (2, 4) = (2, 4) We now have the right posgng values! Term PosCngs List Amrit 1 Bud 0 Daniels 3 Dave 3,6 Deep 0,5 Jack 3 Ledion 2 Makers 1,5,6 Zhang 4 cognac 2 likes 0,1,2,3,4,5,6 light 0 vodka 4 9

10 ConverGng PosGng Values to Events PosCng value Seek addr _Cme host source sourcetype evenjype STEP 2: Use the values array to look up _Gme, seek address, host, source, sourcetype for (2, 4) STEP 3: Use the seek addresses to read rawdata at offsets (120, 170) Ledion likes cognac Zhang likes vodka STEP 4: Back to search land; field extracgons, lookups, etc. 10

11 Reading Compressed Rawdata journal.gz Example: Reading offsets (120, 170) 1. Group offsets into residing chunks 120 falls into range (78, 148) 170 falls into range (148, 236) 2. Read data off disk and decompress EXPENSIVE! 11

12 How Expensive? Example bucket: 521,629 events Limited to ~175,000 events per second 12

13 What Are We Doing in TSIDX Land? In SQL terms: SELECT _time, seekaddr, host, source, sourcetype!!where <some query>! And then we re off to rawdata and search land How can we do more here?! OR even: SELECT foo, bar WHERE <some query> SELECT avg(baz), stdev(baz) WHERE <some query> GROUPBY foo, bar! 13

14 Indexed Fields Term PosCngs list bar::ab 1,3,7,39,98 bar::cez 0,6,9,12 bar::xyz 3,4,5,6 baz::1 3,6,85 baz::2567 0,5 baz::462 3,24,45 baz::98 2,3,5,8,9 baz:: ,5,6,76,99 foo::afdjsi 4,567,2345 foo::aghdafo 2,234,6667 foo::bazcxuid 0,1,623,7777 foo::cef 0,1,2,3,4,43 foo::zaz 4 Big idea: Use the lexicon as a field value store! By simply separagng fields and values with :: we can store sufficient informagon to run more interesgng queries 14

15 How Does it Work? Term PosCngs list bar::ab 1,3,7,39,98 bar::cez 0,6,9,12 bar::xyz 3,4,5,6 baz::1 3,6,85 baz::2567 0,5 baz::462 3,24,45 baz::98 2,3,5,8,9 baz:: ,5,6,76,99 foo::afdjsi 4,567,2345 foo::aghdafo 2,234,6667 foo::bazcxuid 0,1,623,7777 foo::cef 0,1,2,3,4,43 foo::zaz 4 SELECT sum(baz) WHERE bar=xyz!! Evaluate query: 3,4,5,6! Iterate over baz, updagng sum for matching events baz::1 ê Sum += 2 * 1 baz::2567 ê Sum += 1 * 2567 baz::462 ê Sum += 1 * 462 baz::98 ê Sum += 2 * 98 baz::99023 ê Sum += 2 *

16 How Can You Use This in Splunk Enterprise 5.x? tscollect! Creates TSIDX files in the indexed fields format! index=main fields a, b, c tscollect namespace=demo! Only admins can run this indexes_edit capability tstats! Runs stats over the TSIDX files in the created namespace! tstats avg(a) from demo groupby b, c 16

17 Drawbacks to the Splunk Enterprise 5.x Approach! Only on the search head! No retengon policy or limits! Manual process How to schedule collect? Timing problems Fault tolerance? Data lag Search head $SPLUNK_DB/tsidxstats Indexer 1 Indexer 2 Indexer N 17

18 Splunk Enterprise 6: Making it Easy What data do we want to accelerate? 18

19 Create a Data Model 19

20 Splunk Enterprise 6: Making it Easy How do we accelerate that data? 20

21 Click The Checkbox! 21

22 Introducing the High Performance AnalyGcs Store! AutomaGcally collected Handles Gming issues, backfill! AutomaGcally maintained Search head Uses acceleragon window! Stored on the indexers Peer to the buckets! Fault tolerant collecgon Indexer 1 Indexer 2 Indexer N 22

23 Completely Transparent!! No administragon overhead! Missing collecgon data filled in by search No data lag!! AnalyGc queries just get faster! Results come from HPAS first! Checking acceleragon status Data models management page Job inspector 23

24 Great, How Do I Use it?! In pivot: AutomaGcally used when acceleragon is on!! Manually: tstats from datamodel=<name> 24

25 What About Report AcceleraGon? Report AcceleraGon High Performance AnalyGcs Store! Accelerates a pargcular search! Stores results of map step! Pre- computed aggregate! Doesn t help for high- cardinality! Typically lower storage costs But requires storage per- search! Accelerates an engre dataset! Stores field value informagon! Nothing pre- computed! Works well for high- cardinality! Higher storage costs (~25%) Storage shared by all searches on datamodel Varies by collecgon: # events, fields, values 25

26 Splunk Enterprise 6: Making it Easy What if I already have indexed fields? 26

27 Bonus!! You can query exisgng indexed fields directly! Just omit the FROM clause in tstats You can specify indexes in WHERE clause Supports search filters Search head! Don t forget the default indexed fields! host, source, sourcetype _indexgme, linecount, punct date_second, date_minute, etc. 27

28 More InformaGon! Data models h`p://docs.splunk.com/documentagon/splunk/6.0/knowledge/ Managedatamodels! AcceleraGon h`p://docs.splunk.com/documentagon/splunk/6.0/knowledge/ Acceleratedatamodels! tstats command h`p://docs.splunk.com/documentagon/splunk/6.0/searchreference/tstats 28

29 Demo

30 Key Takeaway Build a datamodel and try it yourself! 30

31 Next Steps 1 2 Download the.conf2013 Mobile App If not iphone, ipad or Android, use the Web App Take the survey & WIN A PASS FOR.CONF2014 Or one of these bags! 3 Go to the Search Party! Marquee Nightclub at The Cosmopolitan Today, 7:30-10:30pm 31

32 QuesGons?

33 THANK YOU