Running a Virtualized Splunk Enterprise Infrastructure Ted Knudsen

Copyright 2013 Splunk Inc. Running a Virtualized Splunk Enterprise Infrastructure Ted Knudsen Co- Founder and Engineering Manager, Message Bus #splunkconf

Agenda! Message Bus PlaKorm! Running a plakorm in the cloud! Every day is Splunk day! Splunk architecture! How it all works together! Future plans! Q & A 2

Message Bus PlaKorm Provides email delivery at scale, done right 3

Message Bus PlaKorm! API driven! Cost effectve! No MTA required to send email! Scalable (> 1000 mps)! 100% cloud natve! DMARC compliant security 4

Message Bus PlaKorm SDK s available in 6 languages! PHP! Ruby! nodejs! Python! C#! Java Screenshot here 5

Message Bus PlaKorm Technologies! OS: CentOS 6.2! Languages: Scala (JVM 1.6 & 1.7), nodejs, Ruby! Database: Mysql (Amazon RDS), Google Big Query! Caching: Redis (2.4, 2.6)! Queuing: HornetQ! Deployment: Chef (hosted), ArTfactory! Monitoring: Splunk, Nagios, collectd, PagerDuty! Source Control: Github 6

Running PlaKorm in the Cloud Cloud NaTve Message Bus started with cloud only infrastructure; Caveat is you have to make some assumptons when running a plakorm in the cloud! You can t assume reliable performance! Build failure into everything! Server problems solved by building a new one! Size of the server can help ensure beier performance 7

Running PlaKorm in the Cloud Which Cloud Provider? Choose the provider that best works for you! Do they provide a specific soluton for your needs?! Do they have the capacity you need?! Does their plakorm have a robust API for automaton?! What kind of pricing discounts do they offer? Don t be afraid to use more than one provider 8

Running PlaKorm in the Cloud Cloud Providers! Joyent API, message sending, api.messagebus.com! Google Cloud Pla7orm Big Query, Cloud Storage! Amazon Federated services, reportng, global account informaton, RDS! Rackspace TesTng, Monitoring, ConTnuous IntegraTon 9

Running PlaKorm in the Cloud Cloud Providers Strengths! Joyent ipv4 address block (/20) Vyaia for SNAT Custom networking! Amazon Industry leader Service available for almost any need! Google Cloud Pla7orm Big Query scaling is amazing Cloud Storage is cheap and easy! Rackspace Customer Service 10

Running PlaKorm in the Cloud Where does Splunk fit in? Message Bus plakorm currently requires 80-90 servers per cluster Analyzing log data and monitoring that many servers can only be done effectvely with Splunk Started with Splunk from the very beginning; Engineering team formats logging with Splunk in mind 11

Every Day is Splunk Day How Message Bus uses Splunk Daily operatons done with Splunk! Message volumes validaton! ProducTon trouble shootng! Data validaton! Monitoring validaton! Customer support! Development/tesTng 12

Every Day is Splunk Day ProducTon monitoring Engineering team deploys update and new features at least once per week Splunk used to monitor and analyze components before and aqer deployments 13

Splunk Architecture Version and Layout Version: Splunk Enterprise 5.0.2! Fowarders on every server! Indexers! Search Heads! Receivers 14

Splunk Architecture Receivers IniTal deployment just had forwarders and indexers; As cluster size grew we found that this had some draw backs If an indexer goes offline, all forwarders need to be updated with chef; This can take a while depending on the number of servers and the level of automaton Implemented the receivers and this way the cluster components are unaware of the state of indexers or the number of indexers currently running 15

Splunk Architecture Volume and Server SpecificaTons Volume and Events! 10 million emails/day! 400 million log lines! 200 GB/day! 2 Data Centers Splunk Servers! 8 CPU! 8 GB Memory! 250GB data volumes! 6 indexers per data center! 12 indexes total, 2 search heads 16

Splunk Infrastructure Splunk Receivers Search Head Indexer Indexer Indexer Indexer (Offline) 17

How it All Works Together Message Bus plakorm runs with three cloud providers in multple data centers; A single message touches all three providers:! Joyent (east and west)! Amazon! Google 18

How it All Works Together API Clients Google Cloud Pla7orm Messaging Clusters Joyent West DNS Load Balanced Joyent East AWS MTA 19

Future Plans Splunk summary indexes for key data points from each cluster and cloud provider Forward key performance data to global Splunk instance which will allow high level analysis by cloud provider; If further detail is required then go to that cluster for detailed analysis using the local Splunk 20

Future Plans MulT- cloud ImplementaTon Messaging Clusters Joyent West DNS Load Balanced Joyent East Forward KPI summary data and key errors to master Splunk for global reportng Splunk Splunk AWS MTA Global Splunk 21

Summary Running in the Cloud build components and services to handle the unique qualites of cloud servers; Think cloud natve Cloud Providers each one has strengths and weaknesses. Choose the one that best suits your needs; Don t be afraid to use multple vendors Splunk in the Cloud it works; Plan around the uncertaintes of the cloud and you will be successful 22

QuesTons Ted Knudsen ted@messagebus.com www.messagebus.com github.com/messagebus (SDK s)! Co- founder of Message Bus in Oct 2010! Enterprise Soqware since 1998! Splunk user since Jan 2011! Presented at.conf2012 Using Splunk for just about everything 23

THANK YOU