Data Models for Developers

Transcription

1 Copyright 2013 Splunk Inc. Data Models for Developers Alice Neels So<ware Engineer, Splunk Brian Bingham So<ware Engineer, Splunk Content #splunkconf

2 Legal NoKces During the course of this presentakon, we may make forward- looking statements regarding future events or the expected performance of the company. We caukon you that such statements reflect our current expectakons and eskmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in this presentakon are being made as of the Kme and date of its live presentakon. If reviewed a<er its live presentakon, this presentakon may not contain current or accurate informakon. We do not assume any obligakon to update any forward- looking statements we may make. In addikon, any informakon about our roadmap outlines our general product direckon and is subject to change at any Kme without nokce. It is for informakonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligakon either to develop the features or funckonality described or to include any such feature or funckonality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respeccve owners Splunk Inc. All rights reserved. 2

3 Alice: About Us! At Splunk since 2011! Before this, UW CSE, then worked on ios at Apple! On the core search team! Backend architect for Data Model Brian:! Splunk since 2012! Past Life - 15 years as an SA! Lead Dev for Infra- Ops Content Team! Major Apps: VMware, ES, and several others 3

4 Agenda! What is data model, and why do I care?! Data models 101! Building a data model! AcceleraKon and management! Using data models! Q&A 4

5 What is a Data Model?

6 search and filter munge report clean- up sourcetype=access_combined source = "/home/ssorkin/banner_access.log gz" eval unique=(uid + useragent) stats dc(unique) by os_name rename dc(unique) as "Unique Visitors" os_name as "OperaKng System" 6

7 What is a Data Model? A data model is a search- Cme mapping of data onto a hierarchical structure Encapsulate the knowledge needed to build a search Pivot reports are build on top of Data Models Data- independent Screenshot here 7

8 Who is it for?! Admins/power users build data models! Business users use data models via Pivot UI! Data models can be used in apps to organize and generate searches 8

9 Why do I Care?! Search is hard! Admins and power users know how their data works! Non- technical users (usually) don t! A data model makes it easy to share and organize that knowledge 9

10 But Really! This search ( sourcetype="access_*" OR sourcetype="iis*" ) ( uri="*" ) uri=* uri_path=* status=* clientip=* referer=* useragent=* ( status=2* ) ( uri_path!=*.php OR uri_path!=*.html OR uri_path!=*.shtml OR uri_path!=*.rhtml OR uri_path! =*.asp ) ( uri_path=*.avi OR uri_path=*.swf ) ( uri_path=*.itpc OR uri_path=*.xml ) litsearch ( sourcetype=access_* OR sourcetype=iis* ) ( uri="*" ) uri=* uri_path=* status=* clientip=* referer=* useragent=* ( status=2* ) ( uri_path!=*.php OR uri_path!=*.html OR uri_path!=*.shtml OR uri_path!=*.rhtml OR uri_path!=*.asp ) ( uri_path=*.avi OR uri_path=*.swf ) ( uri_path=*.itpc OR uri_path=*.xml ) eval newx = " " eval "useragent ::: status"='useragent'+" ::: "+'status' addinfo type=count label=prereport_events fields keepcolorder=t "newx" "prestats_reserved_*" "psrsvd_*" "useragent ::: status" fillnull value=null "useragent ::: status" prestats count by newx "useragent ::: status"!! Becomes this search pivot WebIntelligence PodcastDownload count(podcastdownload) AS "Count of PodcastDownload" SPLITCOL useragent SPLITCOL status FILTER uri isnotnull NUMCOLS 100! And that s cool! 10

11 How Can I Use It? Three ways 1. Use the pivot UI to build dashboards with simple XML 2. Use the search commands (data model, pivot, tstats) to simplify building searches 3. Use data model rest endpoints 11

12 Data Models 101

13 A Data Model is a CollecKon of Objects Screenshot here 13

14 Objects Have Constraints and A/ributes Screenshot here 14

15 Child Objects Inherit Constraints and Asributes Screenshot here 15

16 Child Objects Inherit Constraints and Asributes 16

17 Pivot UI Subhead events Count of http_success events, split by useragent fields 17

18 More Info! See slides from other data models talks: AnalyKcs with Splunk Enterprise 1 & 2 18

19 Building Data Models

20 Three Root Object Types! Event maps to Splunk events requires constraints and asributes Search maps to arbitrary Splunk search (may include generakng, transforming and reporkng search commands) requires search string and asributes TransacKon maps to groups of Splunk events or groups of Splunk search results requires objects to group, fields/ condikons to group by, and asributes

21 Three Root Object Types! Event maps to Splunk events requires constraints and asributes! Search maps to arbitrary Splunk search (may include generakng, transforming and reporkng search commands) requires search string and asributes TransacKon maps to groups of Splunk events or groups of Splunk search results requires objects to group, fields/ condikons to group by, and asributes

22 Three Root Object Types! Event maps to Splunk events requires constraints and asributes! Search maps to arbitrary Splunk search (may include generakng, transforming and reporkng search commands) requires search string and asributes! TransacKon maps to groups of Splunk events or groups of Splunk search results requires objects to group, fields/ condikons to group by, and asributes

23 Child Object Facts! A child object is a type of its parent object: e.g. An HTTP_Success object is a type of HTTP_Access! Adding a child object is essenkally a way of adding a filter on the parents! A parent- child relakonship makes it easy to do queries like What percentage of my HTTP_Access events are HTTP_Success events? 23

24 Object Asributes! Auto- Extracted default and pre- defined fields! Eval Expression a new field based on an expression that you define! Lookup leverage an exiskng lookup table! Regular Expression extract a new field based on regex! Geo IP add geographical fields such as lat / lon, country, etc.

25 Object Asributes! Set field types! Configure various flags Note: Child object configurakon can differ from parent

26 Display Names! Models have a modelname and a displayname! Objects have an objectname and a displayname! They re usually the same! modelname and objectname are used internally and in search, must be unique, and can only contain! displayname is what s displayed to label charts and graphs 26

27 Other Stuff! Search and transackon objects can have children too it works the same way (they re filters)! Only event- based objects are accelerated (more on that later)! Performance degrades with the depth of the hierarchy 27

28 Other Splunk Knowledge Objects

29 How it Works with! Various Splunk knowledge objects can help make your data model more powerful! If you already have event types/tags etc., you can absolutely use these in your object constraints and asributes! When starkng from scratch, consider using your data model to do the same thing 29

30 Event Types: Background! Event types in Splunk Enterprise are a way of separakng out a single source file, into separate classificakons. Example: Apache access logs: 400 s and 200 s are in the same file, these can be broken out and used in search E.g. sourcetype=apache_access hsp_code=* NOT hsp_code=4* becomes: evensype=apache_success Can be used for tags 30

31 Event Types With Data Model! Before: set up event types via Manager E.g. Two event types: apache_success and apache_failure! With data models: in most cases, use a data model object E.g. One data model apache, with 2 objects, success and failure.! Objects provide extra power and flexibility! 31

32 Tags! A tag is associated with a parkcular field/value combinakon, or with whether a field is present on an event! In apps, the best use case for tags, is to link up mulkple data sources together based on a common goal! Most o<en based on event types. Example: tag=authenkcakon tag=success could be used to Ke together evensypes from LDAP, MySQL, Unix, MS AcKve Directory or VPN where a user successfully logged in, like evensype = ldap_auth_success or vpn_auth_success! Add through Manager 32

33 Tags with Data Model! Data model may be used with tags in several ways: To create fields that make it easier to define objects ê Usually event type tags used to make several different sources fit a common model An asribute of an object: ê Frequently a way of finding out field tagging differences in events. Great for finding out missing fields or extrackons for creakng objects that are based on a field value ê Can also use calculated fields for this 33

34 Macros, Saved Searches! Data model does not replace the common use cases for macros or saved searches, and it s important to understand when to use what! For dashboards, saving a pivot report may be preferable to saving a search (easier to modify and maintain and share)! Macros happen at a lower level of the search than data model best prackce is to NOT mix and match 34

35 Knowledge Objects Summary! There are a lot of Splunk Enterprise features that can be used with data models! By combining these features, we get lots of power and flexibility! Data model helps you manage complex data! Which features you should use will depend on your needs and data, but keep it simple! 35

36 AcceleraKon and Management

37 What is a Data Model Really?! Data models are stored as JSON files on disk (spec in docs)! They live in <myapp>/local/data/models (or <myapp>/default/ data/models for pre- installed models)! They also have associated conf stanzas and metadata

38 It Looks Like this

39 EdiKng JSON At your own risk!! EdiKng data models by hand: NOT SUPPORTED! Very easy to shoot yourself in the foot, hard to recover! When you edit models via the UI, we validate them! ExcepKon: Installing a model by adding the file to <myapp>/<local OR default>/data/models is probably okay 39

40 DeleKng a Model! Use the UI! This will do appropriate cleanup for you! If you go delekng files on disk, it s easy to break things

41 Permissions! Data models have permissions just like other splunk knowledge objects! Permissions are per model, NOT per object! Edit permissions through the UI

42 Permissions cont.! Data models exist in a parkcular app! Different user roles may or may not have read or write access! If your model relies on lookups, etc., they need to also be available in that app

43 Data Model AcceleraKon Admin or Power User Turn on acceleration via UI Setting written to conf file Backend Magic Polling: is are there new accelerated models? Kick off collection accelerakon Run search using on- disk accelerakon Non- technical User Run a pivot report no accelerakon Kick off ad- hoc accelerakon and run search 43

44 AcceleraKon Facts! Works with search- head pooling we collect on indexers! Only the first event- based object and its children are accelerated! No accelerakon for search and transackon- based objects! No edikng accelerated models

45 Using Data Models

46 Pivot Interface! Build a data model! Build a report with Pivot! Embed report in a dashboard

47 Demo 47

48 Search Commands! You can use data models in the search language! They re basically just macros

49 datamodel! Look at models datamodel Returns JSON model descripkons as separate events! Look at just one model datamodel mymodel Returns JSON model descripkon for just one model! Look at an object datamodel mymodel myobject Returns JSON object descripkon! Run the search for an object datamodel mymodel myobject search Runs the search

50 datamodel cont.! Model name and object name must be the internal names, not the display names! No accelerakon

51 pivot! Any table you can build in the pivot UI can be expressed with the pivot command! Syntax is to complex to fully cover here see docs! Open in search from pivot UI uses the pivot command! The pivot command will take advantage of accelerakon where available! Example: pivot WebIntelligence HTTP_Request count(is_http_success) AS "Count of is_http_success" count(is_http_error) AS "Count of is_http_error" count(is_http_redirect) AS "Count of is_http_redirect" FILTER status!= 404!

52 Demo 52

53 REST Endpoints! See docs for details! Two main endpoints: servicesns/<user>/<app>/datamodel/model examine models and change se~ngs servicesns/<user>/<app>/datamodel/pivot get the search for a pivot report

54 Demo 54

55 Next Steps 1 2 Download the.conf2013 Mobile App If not iphone, ipad or Android, use the Web App Take the survey & WIN A PASS FOR.CONF2014 Or one of these bags! 55

56 Q&A

57 THANK YOU