Analyze Data in MongoDB with the Hunk App Asya Kamsky

Transcription

1 Copyright 2014 Splunk Inc. Analyze Data in MongoDB with the Hunk App Asya Kamsky Principle Developer Advocate, MongoDB

2 Disclaimer During the course of this presentaion, we may make forward- looking statements regarding future events or the expected performance of the company. We cauion you that such statements reflect our current expectaions and esimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentaion are being made as of the Ime and date of its live presentaion. If reviewed ater its live presentaion, this presentaion may not contain current or accurate informaion. We do not assume any obligaion to update any forward- looking statements we may make. In addiion, any informaion about our roadmap outlines our general product direcion and is subject to change at any Ime without noice. It is for informaional purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaion either to develop the features or funcionality described or to include any such feature or funcionality in a future release. 2

3 Speaker Bio! Principal Community Advocate Helping users get the most out of their MongoDB deployments! Over 20 years of industry experience ranging from big companies to start- ups! Her career has spanned work in database technologies, security, sotware tesing, networking, and the web 3

4 MongoDB Fastest Growing Database of 2013* Business Agility: Dynamic Data Model Scale Fast: Operational Flexibility Low TCO: Commodity Hardware *DB-Engines

5 MongoDB High Performance & Strong Consistency Application! Highly Available - Replica Sets Agile Flexible { author: roger, date: new Date(), text: Spirited Away, tags: [ Tezuka, Manga ]} Horizontally Scalable - Sharding

6 RelaIonal Data Models: Hard to Change New Table New Table New Column Name Age Phone New Column

7 RelaAonal Documents are Easier MongoDB {! givenname: Paul,! surname: Miller! city: London,! location: [45.123,47.232],! cars: [! { model: Bentley,! year: 1973,! value: , },! { model: Rolls Royce,! year: 1965,! value: , }! ]! }!

8 No SQL But SIll Flexible Querying Rich Queries GeospaAal Text Search AggregaAon Map Reduce Find all Mark s policies Find everybody who purchased a policy last month Find all customers that live within 10 miles of NYC Find all tweets that menaon the company within the last 2 days What s the total value of Mark s policies How many customers that bought an auto policy got a home policy within three months {!customer_id : 1,!!first_name : "Mark",!!last_name : "Smith",!!city : "San Francisco",!!policies: [!{!!!policy_number : 13,!!!type : auto,!!!deductible: 500!! },!! {!policy_number : 14,!!!type : life,!!!beneficiaries: [ ]!! }! ]! }!

9 Copyright 2014 Splunk Inc. Clint Sharp Director of Product MarkeIng, Splunk

10 The AcceleraIng Pace of Data Volume Velocity Variety Variability Machine data is the fastest growing, most GPS, complex, most valuable area of big data RFID, Hypervisor, Web Servers, , Messaging, Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, TelemaIcs, Storage, Servers, Security Devices, Desktops

11 Industry Leading Big Data Product Porjolio Real- Ime indexing Real- Ime search App Dev & App Mgmt. IT Ops. Security & Compliance Digital Intelli- gence Business AnalyIcs Ad hoc analyics of historical data in Hadoop Product and Service AnalyIcs Customer View Complete Security AnalyIcs Developers building big data apps on top of Hadoop Splunk Apps Vibrant and passionate developer community Splunk Hadoop Connect ODBC DB Connect

12 Powerful Developer Plajorm with Familiar Tools Add New UI components Integrate into ExisIng Systems With Known Languages and Frameworks JavaScript Java Python PHP C# Ruby REST API

13 Components of Hunk Server Explore Analyze Visualize Dashboards Share splunkweb Web and ApplicaIon server Python, AJAX, CSS, XSLT, XML REST API COMMAND LINE splunkd ODBC Search Head Virtual Indexes C++, Web Services Hadoop Interface Hadoop Client Libraries JAVA 64- bit Linux OS Streaming Resource Libraries NoSQL & Other Stores

14 Where is Hunk Used? FINANCIAL SERVICES Detect / prevent fraud Model and manage risk Personalize banking & insurance products RETAIL Behavior analysis Cross- selling, recommendaion engine OpImize pricing, placement, design OpImize inventory and distribuion MANUFACTURING SimulaIon, analysis and design Improve service via product sensor data Digital factory for lean manufacturing WEB / SOCIAL / MOBILE Product and customer analyics SenIment analysis, personalizaion Web log, image and video analysis GOVERNMENT Detect and prevent fraud Security and intelligence Support open data iniiaives HEALTHCARE Drug pedigree and supply chain PaIent monitoring Compliance, archival, text search Data- driven research

15 Virtual Indexes Enables seamless use of almost the enire Splunk stack on data AutomaIcally handles query execuion to Mongo, Hadoop, etc. Technology is patent pending

16 Examples of Virtual Indexes External System 1 index = syslog (/home/syslog/ ) Hunk Search Head > External System 2 index = apache_logs index = sensor_data External System 3 index = twiyer

17 Hunk Applies Schema on the Fly Structure applied at search Ime No briyle schema to work around AutomaIcally find payerns and trends Hunk applies schema for all fields including transacions at search Ime

18 IntegraIon

19 Hunk Search Architecture Hunk Search Head > Splunk Distributed Search MongoDB 1. Search Processor Query per Index/ Virtual Index 2. MongoDB Streaming Resource Library 3. JSON Config MongoDBProvider MongoDB 4. Results ReducIon Hadoop External Results Provider MongoDB

20 1. 2. Install via GUI 3. 20

21 Install via Command Line! Go to <apps.splunk.com URL>! Download MongoDBProvider.spl! Either: Copy MongoDBProvider.spl to $SPLUNK_HOME/etc/apps tar zxvf MongoDBProvider.spl 21

22 Configure Indexes.conf Overview! Indexes.conf defines indexes, physical and virtual! Need to two configuraion items, a provider and a virtual index Provider should be 1:1 to your MongoDB Server There can be muliple virtual indexes per Provider! Indexes.conf can be in any Splunk App, probably easiest to put it in MongoDBProvider folder 22

23 Provider [provider:local-mongodb]! vix.family = mongodb_erp_family! vix.splunk.search.debug = 0! vix.mongodb.host = localhost: 27017! Virtual Index 1 Configure Indexes.conf [mongodb_vix]! vix.provider = local-mongodb! vix.mongodb.db = hunk! vix.mongodb.collection = test! vix.mongodb.field.time = _id! vix.mongodb.field.time.format = ObjectId Provider Name (referenced in Virtual Indexes)! Family! Disable Debugging! Hostname:Port! [wocorders] Name vix.provider of the = local- mongodb Virtual Index (used by users)! vix.mongodb.db = demo Provider vix.mongodb.collecion Name (matches = wocorders earlier stanza)! MongoDB vix.mongodb.field.ime DB Name! = Imestamp MongoDB vix.mongodb.field.ime.format Collection Name! = date Field to extract time from! Format of the Field to Extract Time From (Valid Options are ObjectID, Date, or Epoch)! 23

24 Virtual Index 2 Configure Indexes.conf [wocorders]! vix.provider = local-mongodb! vix.mongodb.db = demo! vix.mongodb.collection = wocorders! vix.mongodb.field.time = timestamp! vix.mongodb.field.time.format = date Name of the Virtual Index (used by users)! Provider Name (matches earlier stanza)! MongoDB DB Name! MongoDB Collection Name! Field to extract time from! Format of the Field to Extract Time From (Valid Options are ObjectID, Date, or Epoch)! 24

25 How to Query Mongo index=mongodb (foo=xyz OR other=val) fields foo, bar, baz Query your MongoDB Virtual Index Match any fields by specifying the field name and matching parameters Minimize results returned by projecing down only the fields you want returned 25

26 Mongo Specific IntegraIon Highlights index=mongodb foo=xyz Imechart avg(bar) by baz Predicate Pushdown ProjecAons Filtering terms are processed on the MongoDB side, so only results where the field foo matches xyz are returned We only return back fields which are menioned in the paricular search, in this case _Ime, bar and baz 26

27 ! Full text search engine! BSON support Roadmap for the Future 27

28 Get The Bits!! Hunk hyp://splunk.com/download! MongoDB App hyp://apps.splunk.com/app/1810/ Or search for MongoDB on apps.splunk.com 28

29 THANK YOU