SlideShare a Scribd company logo

Worst Splunk practices...and how to fix them

Download as PPTX, PDF
Splunk
Splunk

This document provides a summary of best practices and common pitfalls when using Splunk for data collection, management, and resiliency. It discusses best practices for collecting syslog data over UDP, direct TCP/UDP collection, using forwarders, and data onboarding processes like sourcetype recognition, timestamps, and event parsing. Common mistakes like over-engineering syslog collection, sending data directly to indexers, creating "data funnels" through intermediate forwarders, and letting Splunk automatically determine sourcetypes and timestamps are also summarized.

1 of 61
Downloaded 125 times
© 2019 SPLUNK INC.© 2019 SPLUNK INC. Worst Practices… and How to Fix Them
© 2019 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved. Forward-Looking Statements
© 2019 SPLUNK INC. Target Audience: Splunk Admin or Knowledge Manager • You should be familiar with general Splunk architectures − N00bs, you’ll learn a lot...but some topics won’t be explained in-depth ► Questions you may have… • What is the best way to collect my syslog data? • Why are my searches running slowly? • How can I speed up indexing? • Are there limitations to clustering? • What are the best practices for HA/DR? Am I in the right place? You’ll find this session helpful if you…
© 2019 SPLUNK INC. Data Collection Data Management Data Resiliency What will I learn? Agenda
© 2019 SPLUNK INC. Lossless Syslog/UDP
© 2019 SPLUNK INC. Busting The Myth… “I want to collect 100% of my UDP syslog data” Lossless data transmission over UDP does not exist ► UDP lacks error control AND flow control • Delivery cannot be guaranteed • Packets may be lost − They never arrived due to network issues − They were dropped due to a busy destination • Retransmits can result in duplicates ► You can engineer for redundancy • Loss can still happen • Avoid over-engineering
© 2019 SPLUNK INC. Worst Practice Over-Engineering Don’t engineer a solution for syslog that is more complex than Splunk itself! • Loss of data is still possible − UDP does not guarantee delivery…make peace with it ► Design for redundancy while maintaining minimal complexity
© 2019 SPLUNK INC. Best Practice Simplified syslog collection Goal: Minimize loss • K.I.S.S. – Keep it Simple…Silly Incorporate redundancy without making it overly complex ► Utilize a syslog server • Purpose built solution • Gives more flexibility − Host extraction, log rolling/retention ► Minimize # of network hops between source and syslog server Syslog Server + UF Load Balancer/vIP UDP Source Indexers
© 2019 SPLUNK INC. Direct TCP/UDP Data Collection
© 2019 SPLUNK INC. Worst Practice Sending TCP/UDP straight to Indexers ► TCP/UDP stream sent to Indexers − Directly or via Load Balancer Event distribution on Indexers is CRITICAL • Distribute your search workload as much as possible across Indexers • Load Balancers • Typically only DNS load balancing − Large streams can get stuck on an Indexer • Don’t switch Indexers often enough Load Balancer/vIP TCP/UDP Sources Indexers
© 2019 SPLUNK INC. Best Practice Use Splunk Auto Load Balancing This looks familiar… − It should, it’s the same as the recommended UDP/Syslog configuration Splunk AutoLB • Handles distributing events across Indexers automatically • [forceTimebasedAutoLB] or [event_breaker] − Can be used for large files or streams ► Utilize a syslog server − For all the same reasons we discussed before Syslog Server + UF - or - Splunk HEC Load Balancer/vIP TCP/UDP Source Indexers
© 2019 SPLUNK INC. Forwarder Load Balancing
© 2019 SPLUNK INC. What is it? • Distributes events across Indexers ► Why is it important? • Distributed Processing − Distributes workload − Parallel processing ► When does it break? • Large files • Continuous data streams Load Balancing A Primer… How does it break? • Forwarder keeps sending to the same Indexer until: − Regardless of autoLB settings ► Why does that happen? • UF doesn’t see event boundaries • We don’t want to truncate events autoLB = true autoLBFrequency = 30 autoLBVolume = <bytes> outputs.conf [monitor://<path>] time_before_close = 3 * Secs to wait after EoF [tcp://<remote server>:<port>] rawTcpDoneTimeout = 10 inputs.conf
© 2019 SPLUNK INC. Worst Practices Sticky forwarders Using the UF to monitor… • Very large files • Frequently updated files • Continuous data streams …Without modifying default autoLB behavior • Forwarder can become “locked” onto an Indexer − We have settings that can help Forwarder Indexers Past 30sec LB time BigFile.log
© 2019 SPLUNK INC. Best Practices Un-stick your forwarders ► If you’re running 6.5+ UFs… • Use UF event breaking • Applied per sourcetype − Default behavior is followed if not configured ► If you’re running a pre-6.5 UF... • Use [forceTimebasedAutoLB] Events may be truncated if an individual event exceeds size limit • Know the limits − File Inputs: 64KB − TCP/UDP Inputs: 8KB − Mod Inputs: 65.5KB (Linux Pipe Size) autoLB = true autoLBFrequency = 30 forceTimeBasedautoLB = true outputs.conf [<sourcetype>] EVENT_BREAKER_ENABLE = true EVENT_BREAKER = <regex> props.conf
© 2019 SPLUNK INC. forceTimebasedAutoLB How does it work? UF IDX 1 Chunk 1 EVT1 EVT2 EVT3 EVT4 EVT5 Control Key Chunk 2 EVT1 EVT2 EVT3 EVT1 EVT2 EVT3 EVT4 EVT5 IDX 2 EVT1 EVT2 EVT3 EVT4Problem Event: Chunk 1 Chunk 2 Chunk 3
© 2019 SPLUNK INC. UF Event Breaking A better way to get un-stuck ► Available in Splunk 6.5+ − Only available on the Universal Forwarder (UF) What does it do? • Provides lightweight event breaking on the UF • AutoLB processor now sees event boundaries − Prevents locking onto an Indexer − [forceTimeBasedautoLB] not needed for trained Sourcetypes How does it work? • Props.conf on UF • Event breaking happens for specified Sourcetypes • Sourcetypes without an event breaker are not processed − Regular AutoLB rules apply [<sourcetype>] EVENT_BREAKER_ENABLE = true EVENT_BREAKER = <regex> props.conf
© 2019 SPLUNK INC. Intermediate Forwarders Gone Wrong
© 2019 SPLUNK INC. Intermediate forwarder noun : A Splunk Forwarder, either Heavy or Universal, that sits between a Forwarder and an Indexer.
© 2019 SPLUNK INC. Worst Practice Using Heavy Forwarders vs. Universal Forwarders Only use Heavy Forwarders (HWF) if there is a specific need ► You need Python ► Required by an App/Feature − HEC, DBX, Checkpoint, etc… ► Advanced Routing/Transformation − Routing individual events − Masking/SED ► Need a UI on the Forwarder What’s Wrong with my HWFs? • Additional administrative burden − More conf files needed on HWFs − Increases difficulty in troubleshooting • Cooked Data vs. Seared Cooked: ~20% larger over the network • UFs can usually do the same thing − Intermediate Forwarding − Routing based on data stream
© 2019 SPLUNK INC. Worst Practice Creating data funnels ► Intermediate Forwarders • No data is being sent directly to indexers • HWFs are used when UFs will do Avoid data funnels • Forwarders sending data to a handful of intermediate forwarders • Causes indexer starvation − Indexers aren’t receiving events for periods of time − Results in data imbalance and poor search performance Cooked Data Seared Data Indexers Heavy Forwarders (Intermediate) Universal Forwarders TCP/UDP Data Sources Universal Forwarders Heavy Forwarders (Intermediate)
© 2019 SPLUNK INC. The Funnel Effect -vs-
© 2019 SPLUNK INC. The Funnel Effect -vs-
© 2019 SPLUNK INC. The Funnel Effect -vs-
© 2019 SPLUNK INC. Best Practice Reduce funnels ► Intermediate Forwarders • Limit their use − Most helpful when crossing network boundaries Utilize forwarder parallelization − Avoid the “funnel effect” ► UFs  Indexers • Aim for 2:1 ratio − Parallelization or Instances • More UFs avoids Indexer starvation ► UF vs. HWF − Seared data vs. cooked − Less management required for conf files Uncooked Data Indexers Universal Forwarders (Endpoint) Universal Forwarders (Intermediate) Universal Forwarders (Endpoint)
© 2019 SPLUNK INC. Data Onboarding Get it tight, get it right
© 2019 SPLUNK INC. Sourcetype Recognition Who is your daddy and what does he do? Avoid automatic sourcetype recognition where possible ► Specify the sourcetype in inputs.conf ► Don’t let Splunk guess for you • Requires additional processing due to RegEx matching − “too small” sourcetypes may get created [monitor:///var/log] sourcetype = mylog Inputs.conf
© 2019 SPLUNK INC. Timestamps What did this happen? Don’t let Splunk guess • Are you sensing a theme? ► Side Effects − Incorrect Timestamp/TZ extraction − Missing/Missed Events • Bucket Explosion ► These parameters are your friends [mySourcetype] TIME_PREFIX = TIME_FORMAT = MAX_TIMESTAMP_LOOKAHEAD = What comes before the timestamp? What does the timestamp look like? How far into the event should Splunk look to find the timestamp? Props.conf
© 2019 SPLUNK INC. Event Parsing Break it down ► Line Breaking Avoid Line Merging − SHOULD_LINEMERGE = true − BREAK_ONLY_BEFORE_DATE, BREAK_ONLY_BEFORE, MUST_BREAK_AFTER, MUST_NOT_BREAK_AFTER, etc… LINE_BREAKER is much more efficient • Uses RegEx to determine when the raw text should be broken into individual events [mySourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = <regex> Props.conf
© 2019 SPLUNK INC. Indexed Extractions and Accelerations Speeding things up
© 2019 SPLUNK INC. Splunk stores the Key-Value pair inside the TSIDX • Created at index-time • Lose Schema-on-the-fly flexibility • Can improve search performance − Can also negatively impact performance ► Example • KV Pair: Trooper=TK421 • Stored in TSIDX as: Trooper::TK421 What is an Indexed Extraction?
© 2019 SPLUNK INC. ► Indexing all ”important” fields • Unique KV pairs are stored in the TSIDX • KV Pairs with high cardinality increase the size of the TSIDX − Numerical values, especially those with high precision Large TSIDX = slow searches ► Statistical queries vs. filtering events • Indexed extractions are helpful when filtering raw events • Accelerated Data Models are a better choice for statistical queries − A subset of fields/events are accelerated − Accelerations are stored in a different file from the main TSIDX Worst Practice Indexed Extractions Gone Wild
© 2019 SPLUNK INC. Best Practice When should I use Indexed Extractions? ► The format is fixed or unlikely to change − You loose schema on the fly with indexed extractions ► Values appear outside of the key more often than not ► Almost always filter using a specific key (field) − Categorical values (low cardinality) − Don’t index KV pairs with high cardinality ► Frequently searching a large event set for rare data − KV pair that appears in a very small % of events − foo!=bar or NOT foo=bar and the field foo nearly always has the value of bar index=myIndex Category=X1 2016-11-12 1:02:01 PM INFO Category=X1 ip=192.168.1.65 access=granted message=Access granted to X1 system 2016-11-15 12:54:12 AM INFO Category=F2 ip=10.0.0.66 message=passing to X1 for validation
© 2019 SPLUNK INC. Restricted Search Terms Lock it down
© 2019 SPLUNK INC. What are Restricted Search Terms? Nothing to see here… ► Filtering conditions • Added to every search for members of the role as AND conditions − All of their searches MUST meet the criteria you specify • Terms from multiple roles are OR’d together ► Where do I find this? − Access Controls > Roles > [Role Name] > Restrict search terms ► Not secure unless filtering against Indexed Extractions • Users can override the filters using custom Knowledge Objects • Indexed Extractions use a special syntax • key::value Ex: sourcetype::bluecoat
© 2019 SPLUNK INC. Worst Practice All the hosts! ► Inserting 100s or 1,000s of filtering conditions • Hosts, App IDs ► “Just-In-Time” Restricted Terms • Built dynamically on the fly − Custom search commands/Macros • Can be complex/delay search setup host=Gandalf OR host=frodo OR host=Samwise OR host=Aragorn OR host=Peregrin OR host=Legolas OR host=Gimli OR host=Boromir OR host=Sauron OR host=Gollum OR host=Bilbo OR host=Elrond OR host=Treebeard OR host=Arwen OR host=Galadriel OR host=Isildur
© 2019 SPLUNK INC. Best Practice When should I filter? ► Filter based on categorical fields that are Indexed • Remember…low cardinality • Indexed extractions are secure, Search-time extractions are not − Use key::value format Less is more • Reduce the # of KV-Pairs you’re inserting into the TSIDX − Larger TSIDX = slower searches • Limit the # of filters you’re inserting via Restricted Search Terms − Find ways to reduce the # of roles a user belongs to • Don’t create specific filters for data that doesn’t need to be secured − Use an ”All” or “Unsecured” category 37
© 2019 SPLUNK INC. Multi-Site Search Head Clusters
© 2019 SPLUNK INC. Search Head Clustering A Primer… ► SHC members elect a captain from their membership ► Minimum of 3 nodes required − Captain election vs. static assignment ► Odd # of SHC members is preferred ► Captain Manages − Knowledge object replication − Replication of scheduled search artifacts − Job scheduling − Bundle replication Multi-Site SHC does not exist • What?! • SHC is not site-aware − You’re creating a stretched-SHC
© 2019 SPLUNK INC. Worst Practice A ship without a captain ► Captain Election not possible with site or link failure • No site has node majority − Original SHC size: 4 Nodes − Node Majority: 3 Nodes • Odd # of SHC members is preferred ► WAN Latency is too high • We’ve tested up to 200ms Site A Site B 300ms latency
© 2019 SPLUNK INC. Best Practices Two Sites: Semi-Automatic Recovery ► Site A has node majority − Captain can be elected in Site A if Site B fails − Captain must be statically assigned in Site B if Site A fails ► WAN latency is <200ms Three Sites: Fully Automatic Recovery ► Node majority can be maintained with a single site failure • Keep Indexers in 2 sites − Simplifies index replication • Limit workload on SH in 3rd site Designing a better Search Head Cluster Site A <200ms latency Site A Site B Site C Site B <200ms latency [shclustering] adhoc_searchhead = true preferred_captain = false no_artifact_replication = true server.conf
© 2019 SPLUNK INC. Ask me anything (well, not anything) Questions?
© 2019 SPLUNK INC.© 2019 SPLUNK INC. Thank You.
© 2019 SPLUNK INC. Appendix
© 2019 SPLUNK INC. Index Management Where should you put your data?
© 2019 SPLUNK INC. Search Goals How do I make my searches fast? ► Find what we're looking for quickly in the Index (TSIDX) − Lower cardinality in the dataset = fewer terms in the lexicon to search through ► Decompress as few bucket slices as possible to fulfill the search − More matching events in each slice = fewer slices we need to decompress ► Match as many events as possible − Unique search terms = less filtering after schema is applied • Scan Count vs. Event Count TSIDX Bucket Slices
© 2019 SPLUNK INC. Worst Practice When should I create Indexes? Goldilocks for Your Splunk Deployment Dedicated Indexes for SourcetypesMix of data in a handful of Indexes This deployment has too few Indexes… This deployment has too many Indexes…
© 2019 SPLUNK INC. Too Few Indexes …and the problems it creates ► What do we write to the Index (TSIDX)? • Unique terms • Unique KV Pairs (Indexed Extractions) ► Higher data mix can mean higher cardinality • More unique terms = Larger TSIDX − Larger TSIDX files take longer to search ► More raw data to deal with • Potentially uncompressing more bucket slices • Searches can become less dense • Lots of raw data gets filtered out after we apply schema
© 2019 SPLUNK INC. Too Many Indexes If small indexes are faster, why not just create a lot of them? ► Complex to manage ► Index Clustering has limitations • Cluster Master can only manage so many buckets − Total buckets = original and replicas ► What if I’m not using Index Clustering? • Create as many indexes as you want! Version Unique Buckets Total Buckets 6.3 & 6.4 1M 3M 6.5 1.5M 4.5M 6.6, 7.0, 7.1 5M 15M 7.2 9.5M 28.5M
© 2019 SPLUNK INC. Best Practice When to Create Indexes ► Retention • Data retention is controlled per index ► Security Requirements • Indexes are the best and easiest way to secure data in Splunk ► Keep “like” data together in the same Index • Service-level Indexes − Sourcetypes that are commonly searched together − Match more events per bucket slice • Sourcetype-Level Indexes − Data that has the same format − Lower cardinality = smaller TSIDX
© 2019 SPLUNK INC. ► Don’t.  • More indexes = more buckets = bad for your Index Cluster Look for ways to reduce the complexity of your security model • Organize by Service − Collection of apps/infrastructure • Organize by groups − Org, Team, Cluster, Functional Group ► Consider Indexed Extractions & Restricted Search Terms What if I need thousands of Indexes to secure my data?
© 2019 SPLUNK INC. Index Replication Give me 10 of everything!
© 2019 SPLUNK INC. Worst Practice Replicate all the things! ► Lots of Replicas & Sites − 8 replicas in this example − 4 sites Index Replication is Synchronous • Bucket slices are streamed to targets − Excess replication can slow down the Indexing pipeline ► Replication failures cause buckets to roll from hot to warm prematurely − Creates lots of small buckets Site A Origin: RF:2 SF:1 Total: RF:8 SF:4 Site B Origin: RF:2 SF:1 Total: RF:8 SF:4 Site C Origin: RF:2 SF:1 Total: RF:8 SF:4 Site D Origin: RF:2 SF:1 Total: RF:8 SF:4
© 2019 SPLUNK INC. Best Practice K.I.S.S. ► Reduce the number of replicas − 2 local copies and 1 remote is common ► Reduce the number of remote sites • Disk space is easier to manage with 2 sites ► WAN Latency • Recommended: <75ms − Max: 100ms ► Keep an eye on replication errors − Avoid small buckets Site A Local: RF:2 SF:1 Total: RF:3 SF:2 Site B Local: RF:2 SF:1 Total: RF:3 SF:2 Site C Local: RF:2 SF:1 Total: RF:3 SF:2 Site D Local: RF:2 SF:1 Total: RF:3 SF:2
© 2019 SPLUNK INC. High Availability MacGyver Style
© 2019 SPLUNK INC.
© 2019 SPLUNK INC. Some Worst Practices ► Cloned Data Streams • Data is sent to each site • Inconsistency is likely − If a site is down, it will miss data • Difficult to re-sync sites ► Index and Forward • RAID1-style HA − Failover to backup Indexer • Forwarders must be redirected manually • Complex recovery Quick ‘n Dirty HA 57 Load-Balanced Stream 1 Load-Balanced Stream 2 Load-Balanced Index and Forward Primary Site Backup Site Backup Site
© 2019 SPLUNK INC. Another Worst Practice Job servers are so 2006 Rsync & Dedicated Job Servers • Wasted ”standby” capacity in DR • Inefficient use of resources between Ad-Hoc and Job Servers • Conflict management is tricky if running active-active • Search artifacts are not proxied or replicated − Jobs must be re-run at backup site 58 Primary Site Ad-Hoc Job Server Backup Site Ad-Hoc rsync Job Server
© 2019 SPLUNK INC. Some Best Practices Splunk HA ► Index Clustering − Indexes are replicated − Failure recovery is automatic ► Search Head Clustering − Relevant Knowledge Objects are replicated − Search artifacts are either proxied or replicated • Managed Job scheduling − No dedicated job servers − Failure recovery is automatic ► Forwarder Load Balancing − Data is spread across all sites − Replicas are managed by IDX Clustering − DNS can be used to ”failover” forwarders between sites or sets of Indexers 59 Primary Site Backup Site
© 2019 SPLUNK INC. Ask me anything (well, not anything) Questions?
© 2019 SPLUNK INC.© 2019 SPLUNK INC. Thank You.
Ad

Recommended

Best Practices for Forwarder Hierarchies
Best Practices for Forwarder HierarchiesBest Practices for Forwarder Hierarchies
Best Practices for Forwarder Hierarchies
Splunk
 
Best Practices for Splunk Deployments
Best Practices for Splunk DeploymentsBest Practices for Splunk Deployments
Best Practices for Splunk Deployments
Splunk
 
Worst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix them Worst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix them
Splunk
 
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk
 
Splunk Enterprise Security
Splunk Enterprise SecuritySplunk Enterprise Security
Splunk Enterprise Security
Splunk
 
Splunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy ForwardersSplunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy Forwarders
Harry McLaren
 
Splunk Search Optimization
Splunk Search OptimizationSplunk Search Optimization
Splunk Search Optimization
Splunk
 
Splunk 101
Splunk 101Splunk 101
Splunk 101
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Splunk-Presentation
Splunk-Presentation Splunk-Presentation
Splunk-Presentation
PrasadThorat23
 
SplunkLive! Data Models 101
SplunkLive! Data Models 101SplunkLive! Data Models 101
SplunkLive! Data Models 101
Splunk
 
The Power of SPL
The Power of SPLThe Power of SPL
The Power of SPL
Splunk
 
SplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners SessionSplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners Session
Splunk
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
Splunk Distributed Management Console
Splunk Distributed Management Console Splunk Distributed Management Console
Splunk Distributed Management Console
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt
Splunk
 
Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
Splunk
 
Splunk Cloud
Splunk CloudSplunk Cloud
Splunk Cloud
Splunk
 
Splunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfSplunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdf
nitinscribd
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise SecurityExploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
Altinity Ltd
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
Splunk
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
Kishore Chaganti
 
Presto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix IcebergPresto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix Iceberg
kbajda
 
Observability & Datadog
Observability & DatadogObservability & Datadog
Observability & Datadog
JamesAnderson599331
 
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
DevOps.com
 
Turning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk PlatformTurning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk Platform
Splunk
 
FNC2751.pdf
FNC2751.pdfFNC2751.pdf
FNC2751.pdf
CristhianEspinosa6
 
Ad

More Related Content

What's hot (20)

Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Splunk-Presentation
Splunk-Presentation Splunk-Presentation
Splunk-Presentation
PrasadThorat23
 
SplunkLive! Data Models 101
SplunkLive! Data Models 101SplunkLive! Data Models 101
SplunkLive! Data Models 101
Splunk
 
The Power of SPL
The Power of SPLThe Power of SPL
The Power of SPL
Splunk
 
SplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners SessionSplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners Session
Splunk
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
Splunk Distributed Management Console
Splunk Distributed Management Console Splunk Distributed Management Console
Splunk Distributed Management Console
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt
Splunk
 
Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
Splunk
 
Splunk Cloud
Splunk CloudSplunk Cloud
Splunk Cloud
Splunk
 
Splunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfSplunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdf
nitinscribd
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise SecurityExploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
Altinity Ltd
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
Splunk
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
Kishore Chaganti
 
Presto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix IcebergPresto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix Iceberg
kbajda
 
Observability & Datadog
Observability & DatadogObservability & Datadog
Observability & Datadog
JamesAnderson599331
 
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
DevOps.com
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Splunk-Presentation
Splunk-Presentation Splunk-Presentation
Splunk-Presentation
PrasadThorat23
 
SplunkLive! Data Models 101
SplunkLive! Data Models 101SplunkLive! Data Models 101
SplunkLive! Data Models 101
Splunk
 
The Power of SPL
The Power of SPLThe Power of SPL
The Power of SPL
Splunk
 
SplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners SessionSplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners Session
Splunk
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
Splunk Distributed Management Console
Splunk Distributed Management Console Splunk Distributed Management Console
Splunk Distributed Management Console
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt"Splunk Worst Practices"... und wie man diese behebt
"Splunk Worst Practices"... und wie man diese behebt
Splunk
 
Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
Splunk
 
Splunk Cloud
Splunk CloudSplunk Cloud
Splunk Cloud
Splunk
 
Splunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfSplunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdf
nitinscribd
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise SecurityExploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
ClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and howClickHouse Monitoring 101: What to monitor and how
ClickHouse Monitoring 101: What to monitor and how
Altinity Ltd
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
Splunk
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
Kishore Chaganti
 
Presto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix IcebergPresto Summit 2018 - 09 - Netflix Iceberg
Presto Summit 2018 - 09 - Netflix Iceberg
kbajda
 
Observability & Datadog
Observability & DatadogObservability & Datadog
Observability & Datadog
JamesAnderson599331
 
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
More Than Monitoring: How Observability Takes You From Firefighting to Fire P...
DevOps.com
 

Similar to Worst Splunk practices...and how to fix them (20)

Turning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk PlatformTurning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk Platform
Splunk
 
FNC2751.pdf
FNC2751.pdfFNC2751.pdf
FNC2751.pdf
CristhianEspinosa6
 
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward
 
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Harry McLaren
 
Splunk in Staples: IT Operations
Splunk in Staples: IT OperationsSplunk in Staples: IT Operations
Splunk in Staples: IT Operations
Timur Bagirov
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout Session
Splunk
 
Taking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - ArchitectureTaking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - Architecture
Splunk
 
Splunk Data Stream Processor (DSP)
Splunk Data Stream Processor (DSP)Splunk Data Stream Processor (DSP)
Splunk Data Stream Processor (DSP)
Anthony Reinke
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-Cloud
Splunk
 
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
Splunk
 
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Databricks
 
Splunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech DaySplunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech Day
Zivaro Inc
 
Apache Pulsar @Splunk
Apache Pulsar @SplunkApache Pulsar @Splunk
Apache Pulsar @Splunk
Karthik Ramasamy
 
Machine Learning in Action
Machine Learning in ActionMachine Learning in Action
Machine Learning in Action
Splunk
 
Why Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik RamasamyWhy Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik Ramasamy
StreamNative
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-final
Karthik Ramasamy
 
SplunkLive! Paris 2018: Legacy SIEM to Splunk
SplunkLive! Paris 2018: Legacy SIEM to SplunkSplunkLive! Paris 2018: Legacy SIEM to Splunk
SplunkLive! Paris 2018: Legacy SIEM to Splunk
Splunk
 
Splunk and Multicloud
Splunk and MulticloudSplunk and Multicloud
Splunk and Multicloud
Splunk
 
Turning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk PlatformTurning Data Into Business Outcomes with the Splunk Platform
Turning Data Into Business Outcomes with the Splunk Platform
Splunk
 
FNC2751.pdf
FNC2751.pdfFNC2751.pdf
FNC2751.pdf
CristhianEspinosa6
 
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward San Francisco 2019: Using Flink to inspect live data as it flow...
Flink Forward
 
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Splunk Phantom, the Endpoint Data Model & Splunk Security Essentials App!
Harry McLaren
 
Splunk in Staples: IT Operations
Splunk in Staples: IT OperationsSplunk in Staples: IT Operations
Splunk in Staples: IT Operations
Timur Bagirov
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout Session
Splunk
 
Taking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - ArchitectureTaking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - Architecture
Splunk
 
Splunk Data Stream Processor (DSP)
Splunk Data Stream Processor (DSP)Splunk Data Stream Processor (DSP)
Splunk Data Stream Processor (DSP)
Anthony Reinke
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk Discovery: Warsaw 2018 - Legacy SIEM to Splunk, How to Conquer Migrati...
Splunk
 
Splunk und Multi-Cloud
Splunk und Multi-CloudSplunk und Multi-Cloud
Splunk und Multi-Cloud
Splunk
 
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
SplunkLive! Munich 2018: Legacy SIEM to Splunk, How to Conquer Migration and ...
Splunk
 
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Databricks
 
Splunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech DaySplunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech Day
Zivaro Inc
 
Apache Pulsar @Splunk
Apache Pulsar @SplunkApache Pulsar @Splunk
Apache Pulsar @Splunk
Karthik Ramasamy
 
Machine Learning in Action
Machine Learning in ActionMachine Learning in Action
Machine Learning in Action
Splunk
 
Why Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik RamasamyWhy Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik Ramasamy
StreamNative
 
Pulsar summit-keynote-final
Pulsar summit-keynote-finalPulsar summit-keynote-final
Pulsar summit-keynote-final
Karthik Ramasamy
 
SplunkLive! Paris 2018: Legacy SIEM to Splunk
SplunkLive! Paris 2018: Legacy SIEM to SplunkSplunkLive! Paris 2018: Legacy SIEM to Splunk
SplunkLive! Paris 2018: Legacy SIEM to Splunk
Splunk
 
Splunk and Multicloud
Splunk and MulticloudSplunk and Multicloud
Splunk and Multicloud
Splunk
 
Ad

More from Splunk (20)

Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
Building Resilience with Energy Management for the Public Sector
Building Resilience with Energy Management for the Public SectorBuilding Resilience with Energy Management for the Public Sector
Building Resilience with Energy Management for the Public Sector
Splunk
 
IT-Lagebild: Observability for Resilience (SVA)
IT-Lagebild: Observability for Resilience (SVA)IT-Lagebild: Observability for Resilience (SVA)
IT-Lagebild: Observability for Resilience (SVA)
Splunk
 
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Splunk
 
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Splunk
 
Praktische Erfahrungen mit dem Attack Analyser (gematik)
Praktische Erfahrungen mit dem Attack Analyser (gematik)Praktische Erfahrungen mit dem Attack Analyser (gematik)
Praktische Erfahrungen mit dem Attack Analyser (gematik)
Splunk
 
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Splunk
 
Security - Mit Sicherheit zum Erfolg (Telekom)
Security - Mit Sicherheit zum Erfolg (Telekom)Security - Mit Sicherheit zum Erfolg (Telekom)
Security - Mit Sicherheit zum Erfolg (Telekom)
Splunk
 
One Cisco - Splunk Public Sector Summit Germany April 2025
One Cisco - Splunk Public Sector Summit Germany April 2025One Cisco - Splunk Public Sector Summit Germany April 2025
One Cisco - Splunk Public Sector Summit Germany April 2025
Splunk
 
.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
Splunk
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
Splunk
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
Splunk
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
Splunk
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
Splunk
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
Splunk
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
Splunk
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
Splunk
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
Splunk
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
Splunk
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
Building Resilience with Energy Management for the Public Sector
Building Resilience with Energy Management for the Public SectorBuilding Resilience with Energy Management for the Public Sector
Building Resilience with Energy Management for the Public Sector
Splunk
 
IT-Lagebild: Observability for Resilience (SVA)
IT-Lagebild: Observability for Resilience (SVA)IT-Lagebild: Observability for Resilience (SVA)
IT-Lagebild: Observability for Resilience (SVA)
Splunk
 
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Nach dem SOC-Aufbau ist vor der Automatisierung (OFD Baden-Württemberg)
Splunk
 
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Monitoring einer Sicheren Inter-Netzwerk Architektur (SINA)
Splunk
 
Praktische Erfahrungen mit dem Attack Analyser (gematik)
Praktische Erfahrungen mit dem Attack Analyser (gematik)Praktische Erfahrungen mit dem Attack Analyser (gematik)
Praktische Erfahrungen mit dem Attack Analyser (gematik)
Splunk
 
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Cisco XDR & Splunk SIEM - stronger together (DATAGROUP Cyber Security)
Splunk
 
Security - Mit Sicherheit zum Erfolg (Telekom)
Security - Mit Sicherheit zum Erfolg (Telekom)Security - Mit Sicherheit zum Erfolg (Telekom)
Security - Mit Sicherheit zum Erfolg (Telekom)
Splunk
 
One Cisco - Splunk Public Sector Summit Germany April 2025
One Cisco - Splunk Public Sector Summit Germany April 2025One Cisco - Splunk Public Sector Summit Germany April 2025
One Cisco - Splunk Public Sector Summit Germany April 2025
Splunk
 
.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
Splunk
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
Splunk
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
Splunk
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
Splunk
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
Splunk
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
Splunk
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
Splunk
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
Splunk
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
Splunk
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
Splunk
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk
 
Ad

Recently uploaded (20)

2025-05-Q4-2024-Investor-Presentation.pptx
2025-05-Q4-2024-Investor-Presentation.pptx2025-05-Q4-2024-Investor-Presentation.pptx
2025-05-Q4-2024-Investor-Presentation.pptx
Samuele Fogagnolo
 
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Impelsys Inc.
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath MaestroDev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
UiPathCommunity
 
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-UmgebungenHCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
panagenda
 
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveDesigning Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
ScyllaDB
 
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
Procurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptxProcurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptx
Jon Hansen
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
2025-05-Q4-2024-Investor-Presentation.pptx
2025-05-Q4-2024-Investor-Presentation.pptx2025-05-Q4-2024-Investor-Presentation.pptx
2025-05-Q4-2024-Investor-Presentation.pptx
Samuele Fogagnolo
 
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Impelsys Inc.
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...
Noah Loul
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?How Can I use the AI Hype in my Business Context?
How Can I use the AI Hype in my Business Context?
Daniel Lehner
 
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath MaestroDev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
UiPathCommunity
 
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-UmgebungenHCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
panagenda
 
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveDesigning Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive
ScyllaDB
 
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell: Transforming Business Strategy Through Data-Driven Insights
Andrew Marnell
 
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxIncreasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx
Anoop Ashok
 
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond SparkRusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Spark
carlyakerly1
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
Procurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptxProcurement Insights Cost To Value Guide.pptx
Procurement Insights Cost To Value Guide.pptx
Jon Hansen
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In FranceManifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
chb3
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 

Worst Splunk practices...and how to fix them

  • 1. © 2019 SPLUNK INC.© 2019 SPLUNK INC. Worst Practices… and How to Fix Them
  • 2. © 2019 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved. Forward-Looking Statements
  • 3. © 2019 SPLUNK INC. Target Audience: Splunk Admin or Knowledge Manager • You should be familiar with general Splunk architectures − N00bs, you’ll learn a lot...but some topics won’t be explained in-depth ► Questions you may have… • What is the best way to collect my syslog data? • Why are my searches running slowly? • How can I speed up indexing? • Are there limitations to clustering? • What are the best practices for HA/DR? Am I in the right place? You’ll find this session helpful if you…
  • 4. © 2019 SPLUNK INC. Data Collection Data Management Data Resiliency What will I learn? Agenda
  • 5. © 2019 SPLUNK INC. Lossless Syslog/UDP
  • 6. © 2019 SPLUNK INC. Busting The Myth… “I want to collect 100% of my UDP syslog data” Lossless data transmission over UDP does not exist ► UDP lacks error control AND flow control • Delivery cannot be guaranteed • Packets may be lost − They never arrived due to network issues − They were dropped due to a busy destination • Retransmits can result in duplicates ► You can engineer for redundancy • Loss can still happen • Avoid over-engineering
  • 7. © 2019 SPLUNK INC. Worst Practice Over-Engineering Don’t engineer a solution for syslog that is more complex than Splunk itself! • Loss of data is still possible − UDP does not guarantee delivery…make peace with it ► Design for redundancy while maintaining minimal complexity
  • 8. © 2019 SPLUNK INC. Best Practice Simplified syslog collection Goal: Minimize loss • K.I.S.S. – Keep it Simple…Silly Incorporate redundancy without making it overly complex ► Utilize a syslog server • Purpose built solution • Gives more flexibility − Host extraction, log rolling/retention ► Minimize # of network hops between source and syslog server Syslog Server + UF Load Balancer/vIP UDP Source Indexers
  • 9. © 2019 SPLUNK INC. Direct TCP/UDP Data Collection
  • 10. © 2019 SPLUNK INC. Worst Practice Sending TCP/UDP straight to Indexers ► TCP/UDP stream sent to Indexers − Directly or via Load Balancer Event distribution on Indexers is CRITICAL • Distribute your search workload as much as possible across Indexers • Load Balancers • Typically only DNS load balancing − Large streams can get stuck on an Indexer • Don’t switch Indexers often enough Load Balancer/vIP TCP/UDP Sources Indexers
  • 11. © 2019 SPLUNK INC. Best Practice Use Splunk Auto Load Balancing This looks familiar… − It should, it’s the same as the recommended UDP/Syslog configuration Splunk AutoLB • Handles distributing events across Indexers automatically • [forceTimebasedAutoLB] or [event_breaker] − Can be used for large files or streams ► Utilize a syslog server − For all the same reasons we discussed before Syslog Server + UF - or - Splunk HEC Load Balancer/vIP TCP/UDP Source Indexers
  • 12. © 2019 SPLUNK INC. Forwarder Load Balancing
  • 13. © 2019 SPLUNK INC. What is it? • Distributes events across Indexers ► Why is it important? • Distributed Processing − Distributes workload − Parallel processing ► When does it break? • Large files • Continuous data streams Load Balancing A Primer… How does it break? • Forwarder keeps sending to the same Indexer until: − Regardless of autoLB settings ► Why does that happen? • UF doesn’t see event boundaries • We don’t want to truncate events autoLB = true autoLBFrequency = 30 autoLBVolume = <bytes> outputs.conf [monitor://<path>] time_before_close = 3 * Secs to wait after EoF [tcp://<remote server>:<port>] rawTcpDoneTimeout = 10 inputs.conf
  • 14. © 2019 SPLUNK INC. Worst Practices Sticky forwarders Using the UF to monitor… • Very large files • Frequently updated files • Continuous data streams …Without modifying default autoLB behavior • Forwarder can become “locked” onto an Indexer − We have settings that can help Forwarder Indexers Past 30sec LB time BigFile.log
  • 15. © 2019 SPLUNK INC. Best Practices Un-stick your forwarders ► If you’re running 6.5+ UFs… • Use UF event breaking • Applied per sourcetype − Default behavior is followed if not configured ► If you’re running a pre-6.5 UF... • Use [forceTimebasedAutoLB] Events may be truncated if an individual event exceeds size limit • Know the limits − File Inputs: 64KB − TCP/UDP Inputs: 8KB − Mod Inputs: 65.5KB (Linux Pipe Size) autoLB = true autoLBFrequency = 30 forceTimeBasedautoLB = true outputs.conf [<sourcetype>] EVENT_BREAKER_ENABLE = true EVENT_BREAKER = <regex> props.conf
  • 16. © 2019 SPLUNK INC. forceTimebasedAutoLB How does it work? UF IDX 1 Chunk 1 EVT1 EVT2 EVT3 EVT4 EVT5 Control Key Chunk 2 EVT1 EVT2 EVT3 EVT1 EVT2 EVT3 EVT4 EVT5 IDX 2 EVT1 EVT2 EVT3 EVT4Problem Event: Chunk 1 Chunk 2 Chunk 3
  • 17. © 2019 SPLUNK INC. UF Event Breaking A better way to get un-stuck ► Available in Splunk 6.5+ − Only available on the Universal Forwarder (UF) What does it do? • Provides lightweight event breaking on the UF • AutoLB processor now sees event boundaries − Prevents locking onto an Indexer − [forceTimeBasedautoLB] not needed for trained Sourcetypes How does it work? • Props.conf on UF • Event breaking happens for specified Sourcetypes • Sourcetypes without an event breaker are not processed − Regular AutoLB rules apply [<sourcetype>] EVENT_BREAKER_ENABLE = true EVENT_BREAKER = <regex> props.conf
  • 18. © 2019 SPLUNK INC. Intermediate Forwarders Gone Wrong
  • 19. © 2019 SPLUNK INC. Intermediate forwarder noun : A Splunk Forwarder, either Heavy or Universal, that sits between a Forwarder and an Indexer.
  • 20. © 2019 SPLUNK INC. Worst Practice Using Heavy Forwarders vs. Universal Forwarders Only use Heavy Forwarders (HWF) if there is a specific need ► You need Python ► Required by an App/Feature − HEC, DBX, Checkpoint, etc… ► Advanced Routing/Transformation − Routing individual events − Masking/SED ► Need a UI on the Forwarder What’s Wrong with my HWFs? • Additional administrative burden − More conf files needed on HWFs − Increases difficulty in troubleshooting • Cooked Data vs. Seared Cooked: ~20% larger over the network • UFs can usually do the same thing − Intermediate Forwarding − Routing based on data stream
  • 21. © 2019 SPLUNK INC. Worst Practice Creating data funnels ► Intermediate Forwarders • No data is being sent directly to indexers • HWFs are used when UFs will do Avoid data funnels • Forwarders sending data to a handful of intermediate forwarders • Causes indexer starvation − Indexers aren’t receiving events for periods of time − Results in data imbalance and poor search performance Cooked Data Seared Data Indexers Heavy Forwarders (Intermediate) Universal Forwarders TCP/UDP Data Sources Universal Forwarders Heavy Forwarders (Intermediate)
  • 22. © 2019 SPLUNK INC. The Funnel Effect -vs-
  • 23. © 2019 SPLUNK INC. The Funnel Effect -vs-
  • 24. © 2019 SPLUNK INC. The Funnel Effect -vs-
  • 25. © 2019 SPLUNK INC. Best Practice Reduce funnels ► Intermediate Forwarders • Limit their use − Most helpful when crossing network boundaries Utilize forwarder parallelization − Avoid the “funnel effect” ► UFs  Indexers • Aim for 2:1 ratio − Parallelization or Instances • More UFs avoids Indexer starvation ► UF vs. HWF − Seared data vs. cooked − Less management required for conf files Uncooked Data Indexers Universal Forwarders (Endpoint) Universal Forwarders (Intermediate) Universal Forwarders (Endpoint)
  • 26. © 2019 SPLUNK INC. Data Onboarding Get it tight, get it right
  • 27. © 2019 SPLUNK INC. Sourcetype Recognition Who is your daddy and what does he do? Avoid automatic sourcetype recognition where possible ► Specify the sourcetype in inputs.conf ► Don’t let Splunk guess for you • Requires additional processing due to RegEx matching − “too small” sourcetypes may get created [monitor:///var/log] sourcetype = mylog Inputs.conf
  • 28. © 2019 SPLUNK INC. Timestamps What did this happen? Don’t let Splunk guess • Are you sensing a theme? ► Side Effects − Incorrect Timestamp/TZ extraction − Missing/Missed Events • Bucket Explosion ► These parameters are your friends [mySourcetype] TIME_PREFIX = TIME_FORMAT = MAX_TIMESTAMP_LOOKAHEAD = What comes before the timestamp? What does the timestamp look like? How far into the event should Splunk look to find the timestamp? Props.conf
  • 29. © 2019 SPLUNK INC. Event Parsing Break it down ► Line Breaking Avoid Line Merging − SHOULD_LINEMERGE = true − BREAK_ONLY_BEFORE_DATE, BREAK_ONLY_BEFORE, MUST_BREAK_AFTER, MUST_NOT_BREAK_AFTER, etc… LINE_BREAKER is much more efficient • Uses RegEx to determine when the raw text should be broken into individual events [mySourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = <regex> Props.conf
  • 30. © 2019 SPLUNK INC. Indexed Extractions and Accelerations Speeding things up
  • 31. © 2019 SPLUNK INC. Splunk stores the Key-Value pair inside the TSIDX • Created at index-time • Lose Schema-on-the-fly flexibility • Can improve search performance − Can also negatively impact performance ► Example • KV Pair: Trooper=TK421 • Stored in TSIDX as: Trooper::TK421 What is an Indexed Extraction?
  • 32. © 2019 SPLUNK INC. ► Indexing all ”important” fields • Unique KV pairs are stored in the TSIDX • KV Pairs with high cardinality increase the size of the TSIDX − Numerical values, especially those with high precision Large TSIDX = slow searches ► Statistical queries vs. filtering events • Indexed extractions are helpful when filtering raw events • Accelerated Data Models are a better choice for statistical queries − A subset of fields/events are accelerated − Accelerations are stored in a different file from the main TSIDX Worst Practice Indexed Extractions Gone Wild
  • 33. © 2019 SPLUNK INC. Best Practice When should I use Indexed Extractions? ► The format is fixed or unlikely to change − You loose schema on the fly with indexed extractions ► Values appear outside of the key more often than not ► Almost always filter using a specific key (field) − Categorical values (low cardinality) − Don’t index KV pairs with high cardinality ► Frequently searching a large event set for rare data − KV pair that appears in a very small % of events − foo!=bar or NOT foo=bar and the field foo nearly always has the value of bar index=myIndex Category=X1 2016-11-12 1:02:01 PM INFO Category=X1 ip=192.168.1.65 access=granted message=Access granted to X1 system 2016-11-15 12:54:12 AM INFO Category=F2 ip=10.0.0.66 message=passing to X1 for validation
  • 34. © 2019 SPLUNK INC. Restricted Search Terms Lock it down
  • 35. © 2019 SPLUNK INC. What are Restricted Search Terms? Nothing to see here… ► Filtering conditions • Added to every search for members of the role as AND conditions − All of their searches MUST meet the criteria you specify • Terms from multiple roles are OR’d together ► Where do I find this? − Access Controls > Roles > [Role Name] > Restrict search terms ► Not secure unless filtering against Indexed Extractions • Users can override the filters using custom Knowledge Objects • Indexed Extractions use a special syntax • key::value Ex: sourcetype::bluecoat
  • 36. © 2019 SPLUNK INC. Worst Practice All the hosts! ► Inserting 100s or 1,000s of filtering conditions • Hosts, App IDs ► “Just-In-Time” Restricted Terms • Built dynamically on the fly − Custom search commands/Macros • Can be complex/delay search setup host=Gandalf OR host=frodo OR host=Samwise OR host=Aragorn OR host=Peregrin OR host=Legolas OR host=Gimli OR host=Boromir OR host=Sauron OR host=Gollum OR host=Bilbo OR host=Elrond OR host=Treebeard OR host=Arwen OR host=Galadriel OR host=Isildur
  • 37. © 2019 SPLUNK INC. Best Practice When should I filter? ► Filter based on categorical fields that are Indexed • Remember…low cardinality • Indexed extractions are secure, Search-time extractions are not − Use key::value format Less is more • Reduce the # of KV-Pairs you’re inserting into the TSIDX − Larger TSIDX = slower searches • Limit the # of filters you’re inserting via Restricted Search Terms − Find ways to reduce the # of roles a user belongs to • Don’t create specific filters for data that doesn’t need to be secured − Use an ”All” or “Unsecured” category 37
  • 38. © 2019 SPLUNK INC. Multi-Site Search Head Clusters
  • 39. © 2019 SPLUNK INC. Search Head Clustering A Primer… ► SHC members elect a captain from their membership ► Minimum of 3 nodes required − Captain election vs. static assignment ► Odd # of SHC members is preferred ► Captain Manages − Knowledge object replication − Replication of scheduled search artifacts − Job scheduling − Bundle replication Multi-Site SHC does not exist • What?! • SHC is not site-aware − You’re creating a stretched-SHC
  • 40. © 2019 SPLUNK INC. Worst Practice A ship without a captain ► Captain Election not possible with site or link failure • No site has node majority − Original SHC size: 4 Nodes − Node Majority: 3 Nodes • Odd # of SHC members is preferred ► WAN Latency is too high • We’ve tested up to 200ms Site A Site B 300ms latency
  • 41. © 2019 SPLUNK INC. Best Practices Two Sites: Semi-Automatic Recovery ► Site A has node majority − Captain can be elected in Site A if Site B fails − Captain must be statically assigned in Site B if Site A fails ► WAN latency is <200ms Three Sites: Fully Automatic Recovery ► Node majority can be maintained with a single site failure • Keep Indexers in 2 sites − Simplifies index replication • Limit workload on SH in 3rd site Designing a better Search Head Cluster Site A <200ms latency Site A Site B Site C Site B <200ms latency [shclustering] adhoc_searchhead = true preferred_captain = false no_artifact_replication = true server.conf
  • 42. © 2019 SPLUNK INC. Ask me anything (well, not anything) Questions?
  • 43. © 2019 SPLUNK INC.© 2019 SPLUNK INC. Thank You.
  • 44. © 2019 SPLUNK INC. Appendix
  • 45. © 2019 SPLUNK INC. Index Management Where should you put your data?
  • 46. © 2019 SPLUNK INC. Search Goals How do I make my searches fast? ► Find what we're looking for quickly in the Index (TSIDX) − Lower cardinality in the dataset = fewer terms in the lexicon to search through ► Decompress as few bucket slices as possible to fulfill the search − More matching events in each slice = fewer slices we need to decompress ► Match as many events as possible − Unique search terms = less filtering after schema is applied • Scan Count vs. Event Count TSIDX Bucket Slices
  • 47. © 2019 SPLUNK INC. Worst Practice When should I create Indexes? Goldilocks for Your Splunk Deployment Dedicated Indexes for SourcetypesMix of data in a handful of Indexes This deployment has too few Indexes… This deployment has too many Indexes…
  • 48. © 2019 SPLUNK INC. Too Few Indexes …and the problems it creates ► What do we write to the Index (TSIDX)? • Unique terms • Unique KV Pairs (Indexed Extractions) ► Higher data mix can mean higher cardinality • More unique terms = Larger TSIDX − Larger TSIDX files take longer to search ► More raw data to deal with • Potentially uncompressing more bucket slices • Searches can become less dense • Lots of raw data gets filtered out after we apply schema
  • 49. © 2019 SPLUNK INC. Too Many Indexes If small indexes are faster, why not just create a lot of them? ► Complex to manage ► Index Clustering has limitations • Cluster Master can only manage so many buckets − Total buckets = original and replicas ► What if I’m not using Index Clustering? • Create as many indexes as you want! Version Unique Buckets Total Buckets 6.3 & 6.4 1M 3M 6.5 1.5M 4.5M 6.6, 7.0, 7.1 5M 15M 7.2 9.5M 28.5M
  • 50. © 2019 SPLUNK INC. Best Practice When to Create Indexes ► Retention • Data retention is controlled per index ► Security Requirements • Indexes are the best and easiest way to secure data in Splunk ► Keep “like” data together in the same Index • Service-level Indexes − Sourcetypes that are commonly searched together − Match more events per bucket slice • Sourcetype-Level Indexes − Data that has the same format − Lower cardinality = smaller TSIDX
  • 51. © 2019 SPLUNK INC. ► Don’t.  • More indexes = more buckets = bad for your Index Cluster Look for ways to reduce the complexity of your security model • Organize by Service − Collection of apps/infrastructure • Organize by groups − Org, Team, Cluster, Functional Group ► Consider Indexed Extractions & Restricted Search Terms What if I need thousands of Indexes to secure my data?
  • 52. © 2019 SPLUNK INC. Index Replication Give me 10 of everything!
  • 53. © 2019 SPLUNK INC. Worst Practice Replicate all the things! ► Lots of Replicas & Sites − 8 replicas in this example − 4 sites Index Replication is Synchronous • Bucket slices are streamed to targets − Excess replication can slow down the Indexing pipeline ► Replication failures cause buckets to roll from hot to warm prematurely − Creates lots of small buckets Site A Origin: RF:2 SF:1 Total: RF:8 SF:4 Site B Origin: RF:2 SF:1 Total: RF:8 SF:4 Site C Origin: RF:2 SF:1 Total: RF:8 SF:4 Site D Origin: RF:2 SF:1 Total: RF:8 SF:4
  • 54. © 2019 SPLUNK INC. Best Practice K.I.S.S. ► Reduce the number of replicas − 2 local copies and 1 remote is common ► Reduce the number of remote sites • Disk space is easier to manage with 2 sites ► WAN Latency • Recommended: <75ms − Max: 100ms ► Keep an eye on replication errors − Avoid small buckets Site A Local: RF:2 SF:1 Total: RF:3 SF:2 Site B Local: RF:2 SF:1 Total: RF:3 SF:2 Site C Local: RF:2 SF:1 Total: RF:3 SF:2 Site D Local: RF:2 SF:1 Total: RF:3 SF:2
  • 55. © 2019 SPLUNK INC. High Availability MacGyver Style
  • 57. © 2019 SPLUNK INC. Some Worst Practices ► Cloned Data Streams • Data is sent to each site • Inconsistency is likely − If a site is down, it will miss data • Difficult to re-sync sites ► Index and Forward • RAID1-style HA − Failover to backup Indexer • Forwarders must be redirected manually • Complex recovery Quick ‘n Dirty HA 57 Load-Balanced Stream 1 Load-Balanced Stream 2 Load-Balanced Index and Forward Primary Site Backup Site Backup Site
  • 58. © 2019 SPLUNK INC. Another Worst Practice Job servers are so 2006 Rsync & Dedicated Job Servers • Wasted ”standby” capacity in DR • Inefficient use of resources between Ad-Hoc and Job Servers • Conflict management is tricky if running active-active • Search artifacts are not proxied or replicated − Jobs must be re-run at backup site 58 Primary Site Ad-Hoc Job Server Backup Site Ad-Hoc rsync Job Server
  • 59. © 2019 SPLUNK INC. Some Best Practices Splunk HA ► Index Clustering − Indexes are replicated − Failure recovery is automatic ► Search Head Clustering − Relevant Knowledge Objects are replicated − Search artifacts are either proxied or replicated • Managed Job scheduling − No dedicated job servers − Failure recovery is automatic ► Forwarder Load Balancing − Data is spread across all sites − Replicas are managed by IDX Clustering − DNS can be used to ”failover” forwarders between sites or sets of Indexers 59 Primary Site Backup Site
  • 60. © 2019 SPLUNK INC. Ask me anything (well, not anything) Questions?
  • 61. © 2019 SPLUNK INC.© 2019 SPLUNK INC. Thank You.

Editor's Notes

  • #7: I'd tell you the joke about UDP, but you might not get it.
  • #17: How it works… Events are sent to indexers in chunks When [autoLBFrequency] is reached, a control key is set The chunk of data preceding the control key is sent to both indexers IDX 1 discards partial events seen after the control key IDX 2 discards complete events in the first chunk seen before the control key IDX 2 joins partial event from the first chunk with partial event from the second chunk