Best Practices and Better

Practices for Admins

…while you get settled…

▶ Latest Slides:
• https://splunk.box.com/v/blueprints-practices-admin

▶ Collaborate: #bestpractices
• Sign Up @ http://splk.it/slack

▶ Load Feedback --------------------------------------->

Best Practices and Better
Practices for Admins
Presented by Splunk Blueprints

Burch | Senior Best Practices Engineer

.conf2017 | Version 0.0

 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change at
any time without notice. It is for informational purposes only and shall not be incorporated into any contract or
other commitment. Splunk undertakes no obligation either to develop the features or functionality described or
to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 © 2017 SPLUNK INC.

“Scale customer success

through the automation of
adoption services and
best practices”

Blueprint’s Mission
 What’s a “Burch”?
Senior Best Practices Engineer

▶ Was a Senior Sales Engineer

▶ Before that, Splunk Customer

▶ Before that, Middleware Eng

▶ Before that, Computer Science

▶ Before that, an idea of my parents

 © 2017 SPLUNK INC.

1. User Management

2. Data Onboarding
3. Splunk Health

4. Config Management
Agenda
Are you in the right 5. App & TA Creation
place?
6. Architecture

7. Search Tier
8. Indexing Tier

9. Securing Splunk
User Management
 User Education & Enablement

▶ Creating Content:
• Teaching + Videos + Wikis

▶ Is that your core competency?

▶ Outsource it to us!
• Capture unique things
 © 2017 SPLUNK INC.

Search Tutorial
Free Search Tutorial -> docs.splunk.com -> Search Tutorial

▶ Download & Installs

Splunk

▶ Local sandbox

▶ Add tutorial data

Splunk! The Book
www.splunk.com/goto/book
 Community Q&A
answers.splunk.com

▶ E-mail notifications

▶ Fast answers

▶ Larger distribution
 App as Workspace

▶ Default App with Default Dashboard

• Welcome page

▶ Dashboard for new users

• not search box

▶ Drive their eyes/focus

• Hide other apps – even Search!
• show_in_nav = false
 Welcome Page Creator
https://splunkbase.splunk.com/app/2991
Incentive Driven User
Onboarding
“I can’t believe those users did those
things I let them do!”

▶ Don’t be a data butler

▶ Identify & coach & promote to power

▶ Work with you to implement and

learn best practices
Blueprints for Onboarding Teams
Thursday @ 11:35am
 Banner Notifications
docs.splunk.com “Splunk Web messages”

▶ Examples:
• Scheduled restart
• Ongoing issues
• Cool KO to check out

▶ Specific audiences
• Role
• Capability
BAU Account ▶ Use non-admin account

Dog Food! • Prevents accidents

• Live with limitations

• Appreciate user experience

▶ Admin on MC
Data Onboarding
 Log Management
Solicit Constructive Discussion

▶ “If you log it, then you should Splunk

it”
• App/System performance to write logs
• Disk to store logs

▶ cronjobs/scheduled tasks to Splunk

• Scripted Inputs
• standard output/error captured
• Example: Log Rotation crontab
 Onboarding != Ingestion
A David Paper Joint!

Onboarding Phases Ingestion

1. Initial Request ▶ Event Breaks

2. Definition ▶ Time Stamps
3. Implementation à ▶ Source
4. Value ▶ Sourcetype
5. Validation ▶ Index
6. Announcement ▶ Host

▶ Why does this matter?

 Logging
Search: dev.splunk.com “logging best practices”
 Hidden Fields: Time
Search: docs.splunk.com “search time modifiers”
Event Time Index Time
▶ What does a big difference mean?
_time _indextime

earliest _index_earliest
▶ Search over last 5min every 5min but
latest _index_latest there’s a 10min delay in _indextime

▶ When is this ok vs needs attention?

Splunk Sandboxing
Splunk Health
 Support Tickets
docs.splunk.com “How to file a great Support case”

▶ Open Cases
• break/fix only
• Details, details, details
• Diags everywhere!
• Remote
• Upload to case

▶ Schedule webex
• Delay and much lost in email
 Monitoring Console Setup
docs.splunk.com -> Splunk Enterprise -> Administer -> Monitoring Splunk Enterprise
 Point & Purpose
Renamed from “Distributed
Management Console (DMC)”

▶ Buddy with License Server

▶ Standalone instance

▶ Conceptually “Admin Console”

• No user stuff
• Only MC apps/jobs
Health Check
Add your own!
Find Impacting Searches

▶ Search Activity:
• Top 20 Memory-Consuming Searches

▶ Search Usage Statistics

• Long-Running Searches

▶ Great for
• Clean up
• Identifying users to mature
Config Management
 To btool, or not to btool

▶ btool <configuration> list <stanza|> <--debug|>

▶ Add to your env path! (source a profile file from an app)

• Linux: export LD_LIBRARY_PATH=$SPLUNK_HOME/lib
• Mac: export DYLD_LIBRARY_PATH=$SPLUNK_HOME/lib

▶ No “.conf”

▶ Use --debug with | grep –v “system/default”

▶ Not current runtime

 Indent Config

Example: Benefit

[general] ▶ Easily see system vs hand edits

pass4SymmKey = $1$ShiC+P0X
serverName = elBurcho ▶ Detect hand config updated by
sessionTimeout = 30m system
 Simple Version Control

▶ Good: Scripted Input ▶ Targets

• Specific Diag (or just etc dir) • Utilities
• Clean old copies • SHC Working Folder

▶ Better: Scripted Input ▶ Source Control != High Availability

• Check in to git • VMotion type stuffs

▶ Best: Custom Built Solution

• Source Control
 Keep It Clean: Naming Conventions
Handout at Customer Success Studio
▶ Template: <summary|>_<company>_<function>_<environment>

▶ <company>
• Yours or from a 3rd party/splunk app

▶ <function>
• Nothing that changes (i.e. organization/teams)

▶ <environment>
• PROD, DR, QA, TEST, DEV, etc…

▶ <summary|>
• Exists as a modifying of corresponding index
 App Management
What practices do you notice?
Burch_configbackup_ta Burch_license_client_ta Burch_splunkUpgrade_ta
Burch_CustomerOverview Burch_license_server_ta Burch_splunk_admin
Burch_datacollection_ta Burch_master_ta Burch_splunk_default
Burch_deployer_ta Burch_multisite_site1_ta Burch_splunk_developer
Burch_deploymentserver_ta Burch_multisite_site2_ta Burch_splunk_power
Burch_dmc_ta Burch_sandbox_ta Burch_splunk_user
Burch_dreamhost_ta Burch_searchheadcluster_ta Burch_stopdeploymentclient_ta
Burch_es_ta Burch_searchhead_distributed_ta Burch_utility_ta
Burch_forwarder_ta Burch_searchhead_ta Burch_zglobal_ta
Burch_heavyforwarder_ta Burch_searchtimeko_ta
Burch_indexer_ta Burch_splunkAdmin_nix_ta
 Bootstrap
Minimal system/local

1. Install Splunk Enterprise

2. Bootstrap
• Point to DS/Master/Deployer
• system/local overwritten by apps
• Centralized control
• Global App < Function Apps

3. Download app with scripted input

• Non config changes
• Risky!
App & TA Creation
 App Development

▶ No index please! ▶ Macros & Tags

• Provide recommendation • easy modification
• Volumes vs Retentions vs RBAC etc.. • imagine rewriting every
search/dashboard
▶ Inputs disabled • Candidates: index, sourcetype, source
• Don’t touch my license & storage!
▶ Prebuilt Panels vs Dashboards
▶ Remove files
• .DS_Store ▶ ./splunk package app <app>
• .pyc .pyo • Tar non-compatibilities
• local.meta
Certification for Practices
Add-on Builder includes App Inspect
Architecture
 Configuration If you expect to grow big…
Distribution Recap
”Deployment Server is not the Deployer?!”
▶ Separate Installs:
• Easier scalability
Deployment Deployer Master Node
Server • Avoid reload deploy-server on restart
Forwarders Search Head Index Cluster • Cheap VMs
Cluster

▶ Keep Utility apps in sync

• DS -> Master -> IDXC
• DS -> Deployer -> SHC
• Not for faint of heart…
 Data Management
“Compare QA & PROD…D’oh!”

▶ Non PROD data -> PROD SPLUNK!

• “If a single team depends on it, then it’s
production” – Terry Martin
• Or Search Head traverses

▶ Logical Separation:
• Role Based Access Control
• Separate indexes per env
• Use eventtypes/tags

▶ forwardedindex.filter.disable
Data Distribution Quirks

▶ Consolidated data == serial search

▶ Forwarders:Indexers Ratio

▶ autoLBVolume + autoLBFrequency
 Data Collection Tier
Practices whether push or pull data

▶ Easier to scale
• Vertical (VM specs)
• Horizontal (cheaper than indexers)
• Load balancer (not hardcoded)

▶ Minimize IDX/SH Restarts

Search Tier
 Help me?!

▶ n00b ▶ Ninja
 Ninja: Debug This
Where’s Waldo eval max_runtime?!
 n00b: Debug This
Keyboard Command: Ctrl + \ or Command + \
 Search Interface Improvements
user-prefs.conf with export = system

Default Suggestion
 SHC Need 2 Knows
“So…I can’t just treat it like a Deployment Server?!”

Benefits Caveats

▶ Deployer not critical path ▶ Min 3+ SHs

• Odd number for consensus
▶ Config -> default
▶ Same specs
▶ More effective hardware utiliz.
▶ No manual conf edits on SHs
▶ Eliminates dedicated alerting SHs • Split Brain
• A.K.A. Job Servers
 Search Head limits.conf

Example: Benefit

[scheduler] ▶ Defaults to 50%

max_searches_perc ▶ Ad Hod takes precedent regardless
auto_summary_perc ▶ Additional controls for scheduling

shc_role_quota_enforcement ▶ Quota cluster wide

shc_syswide_quota_enforcement • Default is instance specific
Indexing Tier
Trivia: What does an indexer do?
 Cluster of One
“We lost that data even though we had replication”

Benefits Challenges

▶ “Retroactive” data replication ▶ ONLY IF YOU PLAN TO NEED

REPLICATION
• Multisite
▶ No additional disk
• Long Retention Times
• If factors are still 1

▶ summary_replication ▶ Administratively difficult

• Higher chance of errors
• Conceptually abstract
 Indexer Discovery
Search docs.splunk.com for “indexerdiscovery”

Pros Cons

▶ Dynamic indexer listings ▶ Requires network traffic to master

node
• Forwarder silence if master down @ start
▶ indexerWeightByDiskCapacity
• Indexers with different volume sizes
▶ Total Disk != Free Space

▶ Lead to uneven data distribution

 Data Rebalance
Search docs.splunk.com for “Rebalance the indexer cluster”
 Index Definitions Let’s talk about…

[volume:home]
▶ volume:
path = $SPLUNK_DB
maxVolumeDataSizeMB =
▶ maxVolumeDataSizeMB
[volume:cold]
path = $SPLUNK_DB • Indexes compete for storage
maxVolumeDataSizeMB =

[default]
▶ [default]
homePath = volume:home/$_index_name/db
coldPath = volume:cold/$_index_name/colddb ▶ [newindex]
thawedPath = $SPLUNK_DB/$_index_name/thaweddb

[newindex] ▶ $_index_name
Buckets, and TSIDX, and Data Roll…
Hadoop Data Roll vs MiniTSIDX
Securing Splunk
 Security Through Obscurity
docs.splunk.com “Securing Splunk Enterprise”

▶ Security Through Obscurity

• Change default ports
• Change default system account ($SPLUNK_HOME/etc/default/user-seed.conf)

▶ Auditable Logins
• Empty $SPLUNK_HOME/etc/passwd and $SPLUNK_HOME/etc/.ui_login
• Distribute authentication.conf

▶ “Best practices for Splunk Enterprise security” in docs.splunk.com

 © 2017 SPLUNK INC.

1. User Management

2. Data Onboarding
3. Splunk Health

4. Config Management
Wrap Up
5. App & TA Creation

6. Architecture

7. Search Tier
8. Indexing Tier

9. Securing Splunk
 © 2017 SPLUNK INC.

1. Rate this! (be honest)

2. Collaborate: #bestpractices
• Sign Up @ http://splk.it/slack

3. Customer Success Studio

What Now?
Related breakout 4. More talks, search for
sessions and • Blueprints
activities… • Burch
• Champagne
• Delaney
• Optimization
• Best Practices
• Veuve
 © 2017 SPLUNK INC.

Questions & Discussion?

Don't forget to rate this session in the
.conf2017 mobile app