Tightly Integrated: Mike Cormier Bill Thackrey. Achieving Fast Time to Value with Splunk. Managing Directors Splunk Architects Concanon LLC

Transcription

1 Copyright 2014 Splunk Inc. Tightly Integrated: Achieving Fast Time to Value with Splunk Mike Cormier Bill Thackrey Managing Directors Splunk Architects Concanon LLC

2 Disclaimer During the course of this we may make forward- looking statements regarding future events or the expected performance of the company. We you that such statements reflect our current and based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this are being made as of and date of its live If reviewed aser its live this may not contain current or accurate We do not assume any to update any forward- looking statements we may make. In any about our roadmap outlines our general product and is subject to change at without It is for purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no either to develop the features or described or to include any such feature or in a future release. 2

3 Agenda Tightly Integrated: Achieving Fast Time to Value with Splunk! Some Background! With Splunk! Tight The Common Model Demo: Correlsense SharePath Leading Edge Performance Tight Extending SPL Demo: First Class Extended SPL Scianta Fraud! Next- Gen Demo: Next Fraud Scianta Fraud! and Answers

4 Some Background

5 Mike Cormier Founder, Managing Director, Concanon LLC Splunk Architect Splunk 2014 Professional Services Engineer of the Year Bill Thackrey Founder, Managing Director, Concanon LLC Splunk Architect NOT Splunk 2014 Professional Services Engineer of the Year 5

6 Splunk Professional Services Partner Provider Reseller and Development Arm of Scianta Analy/cs Developer of First Class Splunk SPL Extension Based Apps Nine Splunk Architects Amazon Web Services PS Partner Fraud Preven/on Enterprise Security Cloud IT Dev Ops Behavior Modeling 6

7 Focus Today on Three Tightly Integrated SharePath 7

8 SharePath 8

9 SharePath! Next Performance Efficiently isolate and diagnose performance problems in Track every throughout its lifecycle! Provides both a bird s- eye and a detailed view of how transac@ons perform across Problem Isola@on: Time- to- resolu@on measured in minutes, not days 9

10 10

11 11

12 12

13 13

14 Built upon the Scianta Analy/cs Cogni/ve Correla/on Engine! Reasoning / Conceptual Search Conceptual Object Classifica@on! Cogni@ve Knowledge Discovery & Cogni@ve Modeling Conceptual Behavior Classifica@on & Clustering 14

15 Built upon the Scianta Analy/cs Cogni/ve Correla/on Engine! Reasoning/Conceptual Search Conceptual Object Knowledge Discovery & Modeling Conceptual Behavior & Clustering 15

16 INTEGRATING with Splunk

17 How to Integrate with Splunk REST Splunk s RESTful API Exposes Most Resources as Endpoints Common Informa@on Model Map Your External App s Data to Splunk s CIM SPL Extend Splunk s Capabili@es with Custom SPL 17

18 How to Integrate with Splunk Full access to Splunk resources Simple CLI with CURL, WGET Basis of all Splunk SDKs Well documented We re not going to concentrate on REST today REST Splunk s RESTful API Exposes Most Resources as Endpoints 18

19 How to Integrate TIGHTLY with Splunk Common Model Map your external app s data to Splunk s CIM SPL Extend Splunk s capabili@es with custom SPL 19

20 Tight Common Model

21 The Splunk Common Model! Loosely based on open standard: DMTF/WBEM! Splunk s approach: more narrowly focused! Underpinnings of Splunk App for Enterprise Security! Underpinnings of en@re emerging Splunk ecosphere! Available at apps.splunk.com/

22 The Splunk Common Model! Purpose: Normalize Data Allows consistent way to view data from disparate sources sourcetypea user - > sourcetypeb userid - > sourcetypec customerid! Now implemented in Splunk as a set of Data Models 22

23 Example: SharePath! Powerful third party suite! Independent of Splunk! Tightly integrated to SplunkThrough CIM! Modular input from SharePath REST! Enables Splunk as single pane of glass SharePath 23

24 The Splunk Common Model 24

25 The Splunk Common Model 25

26 Mapping to the CIM: SharePath 26

27 Mapping to the CIM: SharePath 27

28 Mapping to the CIM: SharePath Standard 28

29 Mapping to the CIM: SharePath Via Common Model 29

30 Mapping to the CIM: Correlsense PROBLEM: Some data available in Correlsense SharePath does not map to the current Splunk CIM What do you do? Extend the CIM!! Develop a Common Informa/on Model for your Enterprise Refer to DMTF/WBEM Standard Add Data Models as Necessary 30

31 Mapping to the CIM: SharePath SharePath 31

32 DEMO Performance Correlsense SharePath

33 Tight Extending SPL

34 Extending SPL! Currently splunkd calls Python! Extraordinarily powerful But Presents Scaling Challenges! Beware data marshaling in Python! Concanon solves this by calling C through Python! Minimal performance hit! First Class of SPL! Commands can run in parallel on search peers 34

35 SPL Command Classes STREAMING Run in Parallel on Search Peers GENERATING Generate Event Records PREOP Perform a pre- opera@on to generate data for a command 35

36 STREAMING Command 36

37 Extending SPL Streaming Command Examples 37

38 GENERATING Command 38

39 Extending SPL Command Examples 39

40 PREOP Command 40

41 Extending SPL PreOP Command Examples 41

42 DEMO First Class Extended SPL Cogni*ve An*- Fraud Scianta

43 Next

44 The of Insight 44

45 Why Has the ability to incorporate:! Ambiguity! Imprecision! Evidence! Concepts! Vagueness! Measures of Self- Adaptability! Feedback 45

46 Reasoning Conceptual Search What if Splunk Users Could converse with their data in the same way they converse with their co- workers? using rich, expression rather than arcane parameter- based formulas? 46

47 Reasoning Conceptual Search What if Splunk Users Could converse with their data using the same rich seman/c terms the same concepts they re thinking about. 47

48 Reasoning Conceptual Search What if, instead of this Show me all fiber networks where the latency is more than 2 standard devia@ons above 12ms AND all MPLS networks where the latency is more than 2 standard devia@ons above 143ms AND all fast_ethernet networks where latency is more than 2 standard devia@ons above 47ms and all wireless networks where the latency is more than 2 standard devia@ons above 220ms 48

49 Reasoning Conceptual Search A Splunk user could simply ask this: Show me all of the network segments, by network type, where the latency is long. 49

50 Conceptual Search Example High Number of Hosts Not Malware Signatures as search daydiff>7 stats count,values(dest) as dest search count>10 eval const_dedup_id="endpoint - High Number of Hosts Not Upda@ng Malware Signatures - Rule `malware_opera@ons_tracker` as _@me `daydiff(_@me)` stats count,values(dest) as dest xswhere count in infected_hosts_count is above medium eval const_dedup_id="endpoint - High Number of Hosts Not Upda@ng Malware Signatures - Rule" 50

51 Reasoning Conceptual Search! Pillar of Scianta Used by InfraRed Fusion! Used by Scianta Fraud! Used by more upcoming for Splunk 51

52 DEMO Next- Gen Fraud Cogni*ve An*- Fraud Scianta

53 Wrap Up

54 Geyng Fast Time to Value Through Tight with Splunk! Avoid Using Splunk as an External Datastore ETL is Expensive, Brizle! Capitalize on the Power of Splunk REST SPL Common Model / Data Models Pivot Fast, Powerful Visualiza@on 54

55

56 For More at.conf: Find us in the Partner Pavilion Meet with a Concanon Architect or CorrelSense Rep in the Concanon Hospitality Suite ext 99 Answers Karma Points mcormier_splunk, wethackrey 56

57 Appendix! Extending SPL hzp://dev.splunk.com/view/python- sdk/sp- CAAAEU2! Streaming Commands hzp://docs.splunk.com/splexicon:streamingcommand! Data Model hzp://docs.splunk.com/splexicon:datamodel! Common Informa/on Model App hzp://apps.splunk.com/1621! CIM CIM Schema: schema.html! Cogni/ve Compu/ng Correlsense SharePath hzp:// InfraRed FUSION hzp://infrareddigital.com/wordpress/product- overview/! Concanon.conf2014 Pages hzp:// 57

58 THANK YOU