Inside Secrets From Support- How to Solve the Top 10 Support Issues

Transcription

1 Copyright 2014 Splunk Inc. Inside Secrets From Support- How to Solve the Top 10 Support Issues Barak Reeves Sales Engineer, Splunk Todd Gow Sales Engineer, Splunk

2 Disclaimer During the course of this presentajon, we may make forward- looking statements regarding future events or the expected performance of the company. We caujon you that such statements reflect our current expectajons and esjmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentajon are being made as of the Jme and date of its live presentajon. If reviewed aqer its live presentajon, this presentajon may not contain current or accurate informajon. We do not assume any obligajon to update any forward- looking statements we may make. In addijon, any informajon about our roadmap outlines our general product direcjon and is subject to change at any Jme without nojce. It is for informajonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligajon either to develop the features or funcjonality described or to include any such feature or funcjonality in a future release. 2

3 Agenda! IntroducJons (<2mins)! Top 6 Issues (40mins)! Q & A (5mins) 3

4 Before Delving Into the Issues

5 But Support, my Splunk instance was working great yesterday! 5

6 TroubleshooJng Tips 1. What changed? Was it a firewall change? Storage migrajon? More data inputs? Communicate well with the teams that support and use Splunk; MAKE NO ASSUMPTIONS! Diags don t tell Support what changed 2. Use SoS tool of choice for Support Many gems hidden in plain sight 3. Consult Answers forum Support does You are (hopefully!) not alone 4. Stay calm, resolujon orientated (AKA be proacjve) Confusion is fine, avoid frustrajon 6

7 Splunk on Splunk (SoS) App 7

8 Support Is More Than Willing to Help! Michael Wiggy Wegener 8

9 Top 10 Support Issues

10 Issue #1: Input MisconfiguraJon! Typical Issues Files not showing up RotaJng log files Whitelist/blacklist not working Subdirectories screwing up wildcarding! Most Common Causes File permissions vs user running Splunk differ Bad regex! Finding the error in SoS Data Inputs Overview 10

11 Issue #2: Poor Search Performance! Typical Issue Searches slower than normal! Most Common Causes Too much going on inside Splunk instance Newer inputs! Finding the error in SoS Resource Usage view Indexing Performance Queue CPU/Memory view ê See if search process taking up resources ê What is the search doing? ê How many events are being returned? 11

12 Issue #3: High CPU, Memory ConsumpJon! Typical Issues Varies sorry! Breaking the problem down Is it splunkd? ê High number of monitored files ê High forwarder to indexer rajo Is it splunkweb? ê Search taking a lot of memory? Wildcard search?! Finding the error in SoS CPU/Memory view ê See if search process taking up resources ê What is the search doing? ê How many events are being returned? 12

13 Issue #4: Why is My App Broken?! Typical Issue MisconfiguraJon Permissions! Most Common Causes Unable to located README ê Typical locajons $SPLUNK_HOME/etc/apps/<your app> $SPLUNK_HOME/etc/apps/<your app>/bin $SPLUNK_HOME/etc/apps/<your app>/ appserver ê High forwarder to indexer rajo App pieces not installed correctly ê TAs on forwarder & indexer, SAs on Search Head! Finding the error Sourcetype ê Is the data coming in with the correct sourcetype? 13

14 Splunk App Directory Structure 14

15 Issue #5: Alerts Not Working! Most Common Causes High alert to CPU core rajo Too many real Jme alerts! Finding the error Use Search - > Scheduler ê See if alerjng searches are being skipped and why! MiJgaJon How many real Jme searches do you need? What problem(s) are you trying to solve? ê Use Splunk as a why engine vs what engine Consider a search head dedicated to scheduled searches 15

16 Issue #6: I Keep Exceeding My License! Typical Issues Chauy sourcetypes Event storms! Finding the talker in LURV Licensing Report Past 30 days ê Breakdown by: Sourcetype Source Indexer 16

17 SoS License Usage Report View (LURV) 17

18 Issue #7: Distributed Search! Typical Issues Search Head with ES + Deployment Server + License Master Search Head Pooling with Slow NAS! Finding the error Analyze architecture In SoS, Search - > Dispatch Directory Inspector ê See if search process taking up resources ê What is the search doing? ê How many events are being returned (events per second)? Can be found in job inspector ê Search peers not set up correctly Cannot communicate with peer (firewall, wrong credenjals) 18

19 Issue #8: Splunk is not reading my config! Typical Issue Could be (almost) anything! Finding the error In SoS, Deployment Status - > ConfiguraJon File Viewer ê Same results as using: $SPLUNK_HOME/bin splunk cmd btool list <config file name> - - debug Check config file precedence: default vs local App permissions ê Example Field ExtracJons in 1 app context Saved search (report) on same data in another context Global permissions not set 19

20 Issue #9: Splunk Instance MiscommunicaJon! Typical Issues Forwarder <- > Indexer communicajons misconfigured Indexer <- > Search Head communicajon misconfigured! Finding the error Forwarder <- > Indexer (TCP or UDP) ê NO: Seyngs - > Data Inputs - > TCP (or UDP) ê YES: Seyngs - > Forwarding & Receiving - > TCP (or UDP) Search Head <- > Search Peers (Indexers) ê Verify correct Management ports TCP or domain name 20

21 Issue #10: I Can t Seem to Log a Support Case! Managing Your Case! Call Splunk SPLUNKS = ! Have SoS installed and do your due diligence Splunk Support is for break/fix Review Know Issues (this is a living document) ê hup://docs.splunk.com/documentajon/ Splunk/latest/ReleaseNotes/Knownissues We have bugs in our soqware, eliminate human error and configurajon issues first 21

22 You Asking for Diags??!?? 22

23 ! Snapshot into your environment What Diags Provide! Supports imports the diags into a SoS- like tool 23

24 Special Thanks Barak and Todd would like to send a special Thank You! to Michael Wiggy Wegener in Support who helped create the outline for this presentajon 24

25 Q & A 25

26 THANK YOU