KV Store: Hammer Time

Transcription

1 Copyright 2016 Splunk Inc. KV Store: Hammer Time Nadine Miller Technical Support Engineer, Splunk aka 'vraptor' on IRC and Slack

2 Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 2

3 Who am I? Senior Technical Support Engineer Search Head Cluster & Index Cluster SME Keeper of Support KV Store Trouble-Shooting Docs Splunk Trust Member Senior UNIX Systems Administrator in a Previous Life 3

4 What are we talking about? Focused on KV Store within the context of SHC Discussion of backup/restore important to Standalone Where to disable KV store Merging! No discussion of development/custom use 4

5 What is this beast? KV Store is: a database Mongodb Stores user-created data in Splunk that can be linked to events via searches Another method for lookups Scheduler 5

6 Why is KV Store Important? ES, ITSI, other premium apps SHC user-created data stored in KV Store keeps track of completed jobs to prevent duplicate jobs if SHC member goes offline Stand-alone SH tracking of completed jobs in case of downtime 6

7 KV Store 101 Mongodb - project has good docs, do not hesitate to refer to them Logs: $SPLUNK_HOME/var/log/splunk/mongodb.log $SPLUNK_HOME/var/log/introspection/kvstore.log KV Store wholly unnecessary on IDX & HWF by default unless you explicitly use KV Store or a SH is enabled on the instance (may not be necessary, though, depends on use) Note: setting "replication = true" creates CSVs that replicate to IDX, not mongodb replication to IDX 7

8 KV Store 101 Searches to get KV Store status SHC status commands DMC/MC info Collection counts, size, etc. REST curl -ku admin kvstore/status Others in REST endpoint docs 8

9 #1 Failure

10 No Backups! Folks don't realize this data is not in an index If you don't take regular backups, easy to lose all data in the KV store Pro tip: RAID is not a backup 10

11 Backup Methods OS level SHC - first make sure all SHs are in sync 6.5 or later: splunk show kvstore-status before 6.5: curl -s -k server/info grep kvstorestatus Check DMC/MC for collection counts Take SH offline Take a filesystem snapshot or copy entire KV store directory Bring SH back on-line 11

12 Backup Methods ITSI Use backup/restore in WebUI Use kvstore_to_json.py (repurpose?) Starcher's backup script Outputlookup large kv store could be a problem - high memory consumption multi-kv could also be a problem - flattens 12

13 Restoring Ze Whole Enchilada Pro tip: TEST in a dev environment first! OS Level stand alone SH Offline Splunk Copy kvstore directory back in correct location Restart Splunk 13

14 Restoring in SHC SHC version 6.3 or later: Shut down SHC members splunk clean kvstore --cluster Copy kvstore directory into place on one member Verify KV store is good Restart other SHC members SHC prior to 6.3, complicated Re-bootstrap cluster from scratch after temporarily setting replication_factor=1 Advise support case 14

15 SHC: When Everything is a Nail...

16 KV Store Complications KV store is a separate cluster within the SHC KV store captain probably different from SHC captain Monitor KV store: Staleness Collection counts: Out of sync = problem Monitor your SHs, if one is offline for any significant time, likely to lose sync 16

17 Use the Monitoring Console, Luke Especially in 6.5 and later 17

18 KV Store Basic Status MC -> Search -> KV Store: Deployment Screenshot here 18

19 KV Instance Collection Metrics MC -> Search -> KV Store: Instance 19

20 KV Store Replication Latency MC -> Search -> KV Store: Deployment 20

21 OpLog Mongodb uses a circular operations log, aka "oplog" If a KV store member cannot keep up with the transactions in this log, eventually it will lose sync Examples: SHC member off-line for a period of time exceeding oplog size Large KV store changes are performed in a non-transactional way (e.g. create temp table => replace existing table) SHC is so busy that oplog head is overwritten faster than the KV store can replicate across all members 21

22 OotB Oplog size = 1GB KV Store Basic Status 22

23 But why? Mongodb by default sets epilog size to 50GB on large partitions 1GB generally fine for a normal SHC that's just keeping track of scheduler jobs 1GB even works fine for a while with premium apps until it doesn't 23

24 How big should "oplogsize" be? "It depends" Look at size of "kvstore" directory on disk or in MC Has there been a recent resync (compact only occurs on resync)? Latency? Any oddities in Window size? Doing anything usual (temp tables/replace)? Generally something between 10-20gb should work unless you're doing something unusual 24

25 KV OpLog Window MC -> Search -> KV Store: Deployment 25

26 Other OpLog Impacts How big is the SHC? Slow disk I/O Network latency w/in SHC If you are having problems replicating config changes inside SHC, can impact kv store May need tuning in SHC (hit me in Slack/IRC) 26

27 Merging KV Stores Thanks to Splunk, it is possible to merge KV Stores unlike raw Mongodb Caveats: Must be done on a collection by collection basis Have to understand key fields in collection Multi-kv flattened Example: inputlookup incident_review_lookup table _key,time,rule_id,owner,urgency,status,comment,user,rul e_name outputlookup new_data.csv 27

28 General Method Back up the KV Store you want to merge into "good" kvstore Perform the previous search on the incomplete "bad" kvstore to get a CSV file Copy both to a test instance Restore "good" kvstore into test instance Perform whatever surgery is necessary on the CSV file to remove unwanted records Merge using another search: inputlookup new_data.csv outputlookup append=true incident_review_lookup 28

29 Other Suggestions Duane and George's.conf 2016 talk: Shop Smart at the KV Store: Best Value Tricks from the Splunk KV Store and REST API George's TA - TA-SyncKVStore Gemini KV Store Tools for Splunk Looks interesting, haven't tested 29

30 Errata On a standalone ootb SH, KV store is only used for users search history; no scheduled job status is tracked using KV store 30

31 Addendum #1 You can do a round robin resync to increase oplogsize Slow and tedious if you have a large cluster 31

32 Addendum #2 Find out if KV Store is in use on your IDX or HWF: curl -k -u admin servicesns/nobody/search/storage/collections/ config egrep -i "\<title\>" Should only return: <title>collections-conf</title> <title>savedsearchhistory</title> Caveats: Only admins can log into IDX/HWF; no apps are installed; you used a clean instance as an IDX 32

33 Questions?

34 THANK YOU