Splunk & Git. The joys and pitfalls of managing your Splunk deployment with Git. Copyright 2018

Transcription

1 Splunk & Git The joys and pitfalls of managing your Splunk deployment with Git

2 About me: Lowell Alleman Working with Splunk since 2008 Splunk consulting since 2013 Background Software development Systems integration Linux administration Live in a small town in central Pennsylvania with my wife and 3 kids (ages 5 and under)

3 What is Git? Open source version control system Scalable, fast, and efficient storage Workflow flexibility (distributed, client/server, local only - KISS) But, a steep learning curve :-(

4 Why use Git with Splunk? Splunk All configurations are text files Synchronize apps across servers in a distributed deployment Selling points: Audit trail Recovery from human error Facilitate changes from multiple orgs (people, dept, external consulting group) Deployment mechanisms: Deployment server Cluster Master -> Peer nodes Deployer -> SHC What s missing: Can t deploy to apps on deployment server or peer nodes. Corner cases; but sometimes requires an out-of-band mechanism

5 Naïve Git Usage: Good Git Gone Wrong! Whoohoo! Let s use git! cd $SPLUNK_HOME/etc/apps git init git add. git commit -m "We're done!" What s wrong with this? What about deployment-apps, master-apps,? Do we want all apps versioned? (search, splunk_archiver) Local.conf file updated by Splunk are re-ordered by last-modified. (Why are my diffs so ugly?) Do we want local and default?

6 Admin-centric Git use Typical developer-centric advice: Feature branching Semantic Versioning Tagging Atomic commits My advice for Admins: K.I.S.S. Sync often (pull/push) Always pull first! Avoid branching Good commit messages Pick a convention and stick to it. Ensure author is set correctly. Learn Git basics; but don t get sucked into the abyss! Find a knowledgeable Git buddy

7 Why is this so difficult? What causes changes to an installed app? A new version of the app is posted on Splunkbase Users create/update shared knowledge objects Scheduled searches update lookups Admins change resource limits Fixes upstream bugs How do you preserve all of this?

8 Managing Splunk Apps in Git Lowell s design suggestions*: Use a single repo Store local and default Store apps in a single flat apps directory Deploy using symlinks (Yes, this works on Windows too!) Use tools to make this easier! * These suggestions have changed over time. NOT applicable everywhere.

9 KSCONF - Kintyre s Splunk CONFig tool Why I wrote this Tired of resolving the same problem over and over again. Need a way to simplify storing Splunk apps in Git Command line tool Modular subcommand design Designed to be scriptable Python Installable as a Python package using pip Can be installed with Splunk s embedded Python 2.7 environment Available as a standalone executable (zipapp)

10 KSCONF - Basic commands ksconf diff a.conf b.conf Compare the content, not the text Ignores sort order Shows multiline diffs (savedsearches or macros) Can be setup as a Git external diff tool (See appendix slide) ksconf check [pre-commit] Simple syntax validation ksconf sort [pre-commit] Sorts stanzas and keys. Consistent textual representation of conf files

11 KSCONF - Merge / Promote ksconf merge --target=tags.conf default/tags.conf local/tags.conf Layers multiple.conf files into one file Useful for previewing default + local Primarily targets Ad-hoc use Somewhat similar to btool list ksconf promote local/savedsearches.conf default/savedsearches.conf Propagate.conf settings applied in one file to another Supports interactive and batch modes Move local changes made via the UI and push them into a default file By default, changes are moved, not copied

12 KSCONF - Combine ksconf combine --target=myapp/default myapp/default.d/* Merge.conf settings from multiple source directories into a combined target directory. Example use cases: Unix System V concept (etc.d) Server migration of the user folder. Works in both one-shot and recurring mode (Intelligent file updates) Supports --dry-run mode to preview changes.

13 KSCONF - Other commands ksconf unarchive Simplifies extraction and importing of.spl files Supports app upgrade Various levels of Git sanity checking and interaction Custom default folder: (e.g., default.d/10-upstream) for use with combine mode ksconf minimize Remove duplicate (default) entries from a.conf file Example: Copy the default inputs.conf to local. Minimize will remove the unchanged content. Better upgrades

14 Practical Git tips Always pull first! (Pattern: pull, commit, push) pull --ff-only pull --rebase (when the above fails) Ensure author is set correctly. git commit --author joe

15 Git Hub

16 Resources Ksconf: GitHub: Bugs and pull requests welcome Docs: Read online or download PDF/Epub Install it now: pip install kintyre-splunk-conf Contact me: Splunk Answers:

17 Thank you.

18 Not for everyone Simple or throwaway deployments Adds additional overhead to otherwise simple tasks. Ideal for: Complex environments Mandatory Auditing

19 Landscape - Why another tool? Tool Purpose Differentiator Validation Confs from Available from btool Config file validation and troubleshooting Precedence & Validation Compares against README/*.spec files File system Splunk Enterprise (builtin) Splunk AppInspect Reports issues of quality, compliance and robustness Compliance Complex rules; Cloud checks, manual Bundle (.spl) Splunk DEV Splunk Add-on Builder Simplify the creation of TAs Rapid development; Has a UI Overlaps with App Inspect; has ad Combined Splunk Base Splunk Packaging Toolkit (slim) Refactor and manage app dependencies Partitioning apps None File system Built in & Splunk DEV KSCONF Adhoc management \w version control Version control Generic / Syntax only File system GitHub / PyPI REST EAI (API & SPL) Internally used mechanism; protection via ACLs Cross server visibility & ACL support Custom via code Splunkd (in memory); can be out-of-sync Built-in: API, SPL, EAI

20 Pre-commit example Example.pre-commit-config.yaml: repos: - repo: sha: v1.3.0 hooks: - id: trailing-whitespace exclude: README.md - id: end-of-file-fixer exclude: README.md$ - id: check-json - id: check-xml - id: check-ast - id: check-added-large-files args: [ '--maxkb=50' ] - id: check-merge-conflict - id: detect-private-key - id: mixed-line-ending args: [ '--fix=lf' ] - repo: sha: v0.4.9 hooks: - id: ksconf-check - id: ksconf-sort Create and commit your.pre-commit-config.yaml (example shown here) to the root of your repository. Run for each clone of your repo: # Install pre-commit sudo pip install pre-commit # Activate pre-commit hook git pull --ff-only pre-commit install --install-hooks

21 Git difftool integration Setup ksconf as an external difftool provider vim ~/.gitconfig: [difftool "ksconf"] cmd = "ksconf --force-color diff \"$LOCAL\" \"$REMOTE\" less -R" [difftool] prompt = false [alias] ksdiff = "difftool --tool=ksconf" Now you can run: git ksdiff savedsearches.conf

22 Make Git Natively Recognize [section] Context in diff Random text shown = NOT helpful Add to ~/.gitconfig [diff "conf"] xfuncname = "^(\\[.*\\])$" Add these attributes (global in ~/.config/git/attributes or per-repo in.gitattributes) *.conf *.meta diff=conf diff=conf Test attributes setup git check-attr -a -- *.conf Output should looks like this: indexes.conf: diff: conf props.conf: diff: conf savedsearches.conf: diff: conf Example showing stanza context (search name)