Splunk & Git. Managing Splunk deployments with Git and KSCONF. Copyright 2018

Size: px

Start display at page:

Gerald Singleton
5 years ago
Views:

1 Splunk & Git Managing Splunk deployments with Git and KSCONF

2 About me: Lowell Alleman Working with Splunk since 2008 Splunk consulting since 2013 Background Software development Systems integration Linux administration Live in a small town in central Pennsylvania with my wife and 3 kids (ages 5 and under)

3 What is Git? Open source version control system Scalable, fast, and efficient storage Workflow flexibility (distributed, client/server, local only - KISS) But, a steep learning curve :-(

4 Why use Git with Splunk? Splunk All configurations are text files Synchronize apps across servers in a distributed deployment Selling points: Audit trail Recovery from human error Facilitate changes from multiple orgs (people, dept, external consulting group) Deployment mechanisms: Deployment server Cluster Master -> Peer nodes Deployer -> SHC What s missing: Can t deploy to apps on deployment server or peer nodes. Corner cases; but sometimes requires an out-of-band mechanism

5 Naïve Git Usage: Best Intentions Whoohoo! Let s use git! cd $SPLUNK_HOME/etc/apps git init git add. git commit -m "We're done!" What s wrong with this? What about deployment-apps, master-apps,? Do we want all apps versioned? (search, splunk_archiver) Local.conf file updated by Splunk are re-ordered by last-modified. (Why are my diffs so ugly?) Do we want local and default?

6 Why is this so difficult? Changes made post-install New Splunkbase release Users add/make changes to knowledge objects Admin fix bugs, tweak limits Live system Files are updated in real time: race conditions, locks, Scheduled searches update lookups Multiple Server types Same app deployed to different locations Config changes can vary per use

Design decisions How many Splunk environments? (Prod, stage, dev,...) Are multiple variations of same app required? How many repos? What version control experience does my team have?

7 Design decisions How many Splunk environments? (Prod, stage, dev,...) Are multiple variations of same app required? How many repos? What version control experience does my team have? Are local uncommitted changes normal? Devs working locally or shared-instance? What network restrictions exist? (Air gap?) Should all apps, or just a few of them? Include base-apps? (How to protect passwords) What s in place now? How to plan/execute consolidation or migration effort?

8 Admin-centric Git use Typical developer-centric advice: Feature branching Semantic Versioning Tagging Atomic commits

9 Admin-centric Git use Typical developer-centric advice: Feature branching Semantic Versioning Tagging Atomic commits My advice for Admins: K.I.S.S. Sync often (pull/push) Always pull first! Avoid branching Good commit messages Pick a convention and stick to it. Ensure author is set correctly. Learn Git basics; but don t get sucked into the abyss! Find a knowledgeable Git buddy

Deploy using symlinks (Yes, this works on Windows too!

10 Managing Splunk Apps in Git Lowell s design suggestions*: Use a single repo Store local and default Store apps in a single flat apps directory Deploy using symlinks (Yes, this works on Windows too!) Use tools to make this easier! * These suggestions have changed over time. NOT applicable everywhere.

KSCONF - Kintyre s Splunk CONFig tool Why I wrote it Tired of

Need a way to simplify storing Splunk apps in Git Command line tool

Installable as a Python package using pip; Python 2.7 and 3.

11 KSCONF - Kintyre s Splunk CONFig tool Why I wrote it Tired of resolving the same problem over and over again. Need a way to simplify storing Splunk apps in Git Command line tool Modular subcommand design Designed to be easily scriptable Python Installable as a Python package using pip; Python 2.7 and 3.4+ Can be installed with Splunk s embedded Python 2.7 environment Available as a standalone executable (zipapp)

12 KSCONF - Basic commands - Diff ksconf diff a.conf b.conf Compares content, not text Ignores sort order Shows multiline diffs (e.g., savedsearches or macros) Setup as a Git external diff tool Example in appendix slides

13 KSCONF - Basic commands - Check ksconf check props.conf Syntax validation Sometimes pickier than Splunk s parser (helps find bugs) Available as pre-commit hook

14 KSCONF - Basic commands - Sort ksconf sort props.conf Sorts stanzas and keys Consistent textual representation Normalization Available as pre-commit hook

15 KSCONF - Merge ksconf merge --target=tags.conf default/tags.conf local/tags.conf Merges.conf files into one file Preview default + local Like btool list (with valid output)

16 KSCONF - Promote ksconf promote local/savedsearches.conf default/savedsearches.conf Propagate.conf settings between layers Interactive and batch modes Move local changes made via the UI and push them into a default file NOTE: By default, changes are moved, not copied

KSCONF - Combine ksconf combine --target=myapp/default myapp/default.d/* Merge.conf settings from multiple source directories into a combined target directory.

17 KSCONF - Combine ksconf combine --target=myapp/default myapp/default.d/* Merge.conf settings from multiple source directories into a combined target directory. Example use cases: Unix System V concept (etc.d) Server migration of the user folder. One-shot and recurring (intelligent) file updates Use --dry-run to preview changes.

18 KSCONF - Combine ksconf combine --target=myapp/default myapp/default.d/*

sanity checking and interaction Works with combine mode: Set custom default

19 KSCONF - Unarchive ksconf unarchive myapp-101.tgz Extraction/importing of.spl files (Not limited to etc/apps ) Supports app upgrade Configurable Git sanity checking and interaction Works with combine mode: Set custom default folder: (e.g., default.d/10-upstream) Simplifies upgrades by providing clean layers

20 KSCONF - Minimize ksconf minimize --target=local/inputs.conf default/inputs.conf Reduce duplicate/default entries Example: Copy the default inputs.conf to local. Minimize will remove the unchanged content. Better upgrades

21 Practical Git tips Always pull first! (Pattern: pull, commit, push) pull --ff-only pull --rebase (when the above fails) Ensure author is set correctly. git commit --author joe

22 Git Hub

23 Resources Ksconf: GitHub: Bugs and pull requests welcome Docs: Read online or download PDF/Epub Install it now: pip install kintyre-splunk-conf Contact me: Splunk Answers:

24 Thank you.

25 Tips & Tricks

26 Pre-commit example Example.pre-commit-config.yaml: repos: - repo: sha: v1.3.0 hooks: - id: trailing-whitespace exclude: README.md - id: end-of-file-fixer exclude: README.md$ - id: check-json - id: check-xml - id: check-ast - id: check-added-large-files args: [ '--maxkb=50' ] - id: check-merge-conflict - id: detect-private-key - id: mixed-line-ending args: [ '--fix=lf' ] - repo: sha: v0.5.2 hooks: - id: ksconf-check - id: ksconf-sort Create and commit your.pre-commit-config.yaml (example shown here) to the root of your repository. Run for each clone of your repo: # Install pre-commit sudo pip install pre-commit # Activate pre-commit hook git pull --ff-only pre-commit install --install-hooks

$gitconfig: [difftool "ksconf"] cmd = "ksconf --force-color diff \"$LOCAL\" \"$REMOTE\" less -R" [difftool] prompt = false [alias] ksdiff =$

27 Git difftool integration Setup ksconf as an external difftool provider vim ~/.gitconfig: [difftool "ksconf"] cmd = "ksconf --force-color diff \"$LOCAL\" \"$REMOTE\" less -R" [difftool] prompt = false [alias] ksdiff = "difftool --tool=ksconf" Now you can run: git ksdiff savedsearches.conf

$helpful Add to ~/.gitconfig [diff "conf"] xfuncname = "^(\\[.$ $*\\])$" Add these attributes (global in ~/.$ meta diff=conf diff=conf Test attributes setup git check-attr -a -- *.

28 Make Git Natively Recognize [section] Context in diff Random text shown = NOT helpful Add to ~/.gitconfig [diff "conf"] xfuncname = "^(\\[.*\\])$" Add these attributes (global in ~/.config/git/attributes or per-repo in.gitattributes) *.conf *.meta diff=conf diff=conf Test attributes setup git check-attr -a -- *.conf Output should looks like this: indexes.conf: diff: conf props.conf: diff: conf savedsearches.conf: diff: conf Example showing stanza context (search name)

29 Landscape - Why another tool? Tool Purpose Differentiator Validation Confs from Available from btool Config file validation and troubleshooting Precedence & Validation Compares against README/*.spec files File system Splunk Enterprise (builtin) Splunk AppInspect Reports issues of quality, compliance and robustness Compliance Complex rules; Cloud checks, manual Bundle (.spl) Splunk DEV Splunk Add-on Builder Simplify the creation of TAs Rapid development; Has a UI Overlaps with App Inspect; additional tests Combined Splunk Base Splunk Packaging Toolkit (slim) Refactor and manage app dependencies Partitioning apps None File system Built in & Splunk DEV KSCONF Adhoc management \w version control Version control Generic / Syntax only File system GitHub / PyPI REST EAI (API & SPL) Internally used mechanism; protection via ACLs Cross server visibility & ACL support Custom via code and eval field validators Splunkd (in memory); can be out-of-sync Built-in: API, SPL, EAI

Splunk & Git. The joys and pitfalls of managing your Splunk deployment with Git. Copyright 2018

Splunk & Git. The joys and pitfalls of managing your Splunk deployment with Git. Copyright 2018 Splunk & Git The joys and pitfalls of managing your Splunk deployment with Git About me: Lowell Alleman Working with Splunk since 2008 Splunk consulting since 2013 Background Software development Systems