SlideShare a Scribd company logo

Splunk .conf2011: Real Time Alerting and Monitoring

Download as PPT, PDF
Erin Sweeney
Erin Sweeney

The document discusses Splunk's capabilities for monitoring and alerting. It describes how Splunk can help various IT roles like service desk, developers, and DBAs respond faster to issues compared to not using Splunk. It provides an overview of real-time searching, alerting, and the alert manager in Splunk. It also demonstrates how to create simple and advanced alerts, enable throttling, and check the alert manager.

1 of 21
Downloaded 127 times
Monitoring and Alerting Ledion Bitincka, Search and Alerting Team
Search and Reporting Team @ Splunk for 4+ years - since 3.0 Things I’ve worked on: Key-value extractions Transactions, Eventtyping, Typeahead, Summary Indexing Monitoring and alerting framework Other random @#$% Intro … Ledion Bitincka (aka Splunk Albanian)
Why use Splunk for monitoring and alerting? Basic alerting Advanced alerts and config options Real-time alerting and throttling (new in 4.2) Alert Manager (new in 4.2) Sneak peek into new features … Feel free to interrupt when you don’t follow!!! Agenda
Life Without Splunk Service Desk Application Support Systems Administrator Application Developer Application Developer Database Administrator Log call. The console says everything is green. App monitoring tools don’t show anything either. Call the developer. Stop working on new code to troubleshoot. Need production logs! Stop what they’re doing to identify and gather production logs for developer. Manual investigation establishes not application problem. DBA analyzes the logs which points to corrupted database files. Escalate. Escalate. Escalate. Respond. Escalate. Now what?
Life With Splunk Service Desk Trouble Ticket Search on IP address shows related Web session and User ID “ 192.168.169.100” Last 60 minutes 192.168.169.100 Search at same time reveals database error due to corrupted files Search for failure or error across entire IT Last 2 minutes failure OR error Search on corruption in the db logs shows that an index file has been corrupted Search for corruption in db logs Last 1 minute host=db.domain.com source=*db.log corrupt* Setup monitoring and alerting for db file corruption Set up Monitoring and Alerting Last hour host=db.domain.com source=*db.log corrupt*
One Splunk. Many Uses.
Monitor and Alert in Real Time 2. Evaluate alerting condition 1. Get data Scheduled search Real-time search Alert Condition 3. Execute actions RSS Email SNMP Script Yes Noop No
Create simple alert using wizard … Available alert actions … Configure email settings (MTA, link hostname)... Demo… (5-10 min)
Advanced Alerting Options Specify an advanced schedule using cron notation Use custom alert conditions Invoke scripts to perform custom actions Integrate with other tools file trouble ticket Other custom processing restart a faulty service update a firewall rule temporarily disable a user account etc …
Real-time Search Primer Searches forward in time Never completes (unless stopped) Constantly updating result set Only generates results preview All search commands supported
Splunkd/ Scheduler Search Process time Search Start historical search Suppress? Logging Scheduled Search Alerts audit.log search.log Y N Notify splunkd splunkd_access.log audit.log Search done Execute actions Update artifact TTL Suppression update Alert manager N Y Done scheduler.log Condition Results
Real-time Alerts Splunkd/ Scheduler Search Process time RT Search Start RT search Suppress? Logging … .. audit.log search.log Y N Notify splunkd splunkd_access.log Execute actions Update artifact TTL Suppression update Alert manager N Y Condition ResPrev Done scheduler.log Condition ResPrev N Y Results Snapshot
Real-time Alerts Reduce response time Continuously monitor a condition Scheduler ensures real-time search is always running Throttling is almost always necessary Compatible with all alert actions Visible through Alerts Manager
Alert Throttling Natively support alert action throttling Useful in: Alert when database server is down, but don’t alert me about this condition for one hour Available for both standard and real-time alerts
Alerts Manager System-wide view of all triggered alerts Basic alert management features Ability to drill down and view why the alert was triggered Real-time alert results are snapshots in time when triggered
Demo… (5-10 min) Show custom alert conditions, when to use them Demo real-time alerts: Throttling Alert manager
Managing Search Load System wide concurrent searches limited to Total: 4 + 4 x number of cores Limit used for ad-hoc and scheduled searches Scheduler queues over limit searches Scheduler allocation is configurable in limits.conf [scheduler] max_searches_perc = 25 // percentage of system wide concurrent searches to use … Use the Scheduler Activity dashboards Search App >> Status >> Scheduler Activity >> Overview Search allocation
savedsearches.conf Search string, schedule, alert condition, actions etc… alert_actions.conf Alert action options such as: email server, format, subject line, ttl etc… limits.conf Scheduler’s concurrent search limit Action execution related limits scheduler.log Look in $SPLUNK_HOME/etc/system/README/<filename>.conf.spec for more detailed info .conf & .log File Summary
Per result alerting and throttling More alert actions to enable more complex alerting conditions Once five failed login attempts occur enable a monitoring search that alerts on suspicious user activity Sneak Peek Into New Features
How scheduled and real-time alerts work Create simple and advanced real-time alerts Enable alert throttling and check for throttled alerts Check fired alerts using Alerts Manager Change scheduler limit defaults Be an IT hero  Now You Should Know …
August 15, 2011 Questions? Ledion Bitincka, Search and Alerting Team
Ad

Recommended

Using Sumo Logic - Apr 2018
Using Sumo Logic - Apr 2018Using Sumo Logic - Apr 2018
Using Sumo Logic - Apr 2018
Sumo Logic
 
Sumo Logic Cert Jam - Metrics Mastery
Sumo Logic Cert Jam - Metrics MasterySumo Logic Cert Jam - Metrics Mastery
Sumo Logic Cert Jam - Metrics Mastery
Sumo Logic
 
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with KubernetesSumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic
 
Sumo Logic Cert Jam - Metrics Mastery
Sumo Logic Cert Jam - Metrics MasterySumo Logic Cert Jam - Metrics Mastery
Sumo Logic Cert Jam - Metrics Mastery
Sumo Logic
 
Dealing with delayed events in Splunk
Dealing with delayed events in SplunkDealing with delayed events in Splunk
Dealing with delayed events in Splunk
Daniel E. Hernández R.
 
Sumo Logic QuickStart Webinar - Get Certified
Sumo Logic QuickStart Webinar - Get CertifiedSumo Logic QuickStart Webinar - Get Certified
Sumo Logic QuickStart Webinar - Get Certified
Sumo Logic
 
Security Certification: Security Analytics using Sumo Logic - Oct 2018
Security Certification: Security Analytics using Sumo Logic - Oct 2018Security Certification: Security Analytics using Sumo Logic - Oct 2018
Security Certification: Security Analytics using Sumo Logic - Oct 2018
Sumo Logic
 
Level 2 Certification: Using Sumo Logic - Oct 2018
Level 2 Certification: Using Sumo Logic - Oct 2018Level 2 Certification: Using Sumo Logic - Oct 2018
Level 2 Certification: Using Sumo Logic - Oct 2018
Sumo Logic
 
Welcome Webinar Slides
Welcome Webinar SlidesWelcome Webinar Slides
Welcome Webinar Slides
Sumo Logic
 
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic
 
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with KubernetesSumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic
 
Sumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security AnalyticsSumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security Analytics
Sumo Logic
 
Sumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - FundamentalsSumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - Fundamentals
Sumo Logic
 
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the CloudAn Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
Edison Lascano
 
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic
 
Sumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - AdministrationSumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - Administration
Sumo Logic
 
Vector clock algorithm
Vector clock algorithmVector clock algorithm
Vector clock algorithm
S. Anbu
 
Sumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search MasterySumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search Mastery
Sumo Logic
 
.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit
Erin Sweeney
 
Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101
allanpyetro
 
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech DaySplunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Zivaro Inc
 
Scalable Monitoring & Alerting
Scalable Monitoring & AlertingScalable Monitoring & Alerting
Scalable Monitoring & Alerting
Franklin Angulo
 
Splunk | Reporting Use Cases
Splunk | Reporting Use CasesSplunk | Reporting Use Cases
Splunk | Reporting Use Cases
Beth Goldman
 
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
Jesse Andrew
 
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
asalters
 
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk ScoringSplunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk
 
SplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud Detection
Splunk
 
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Phil Legg
 
Threat Hunting
Threat HuntingThreat Hunting
Threat Hunting
Tripwire
 
Rapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDealRapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDeal
Splunk
 
Ad

More Related Content

What's hot (10)

Welcome Webinar Slides
Welcome Webinar SlidesWelcome Webinar Slides
Welcome Webinar Slides
Sumo Logic
 
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic
 
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with KubernetesSumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic
 
Sumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security AnalyticsSumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security Analytics
Sumo Logic
 
Sumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - FundamentalsSumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - Fundamentals
Sumo Logic
 
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the CloudAn Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
Edison Lascano
 
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic
 
Sumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - AdministrationSumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - Administration
Sumo Logic
 
Vector clock algorithm
Vector clock algorithmVector clock algorithm
Vector clock algorithm
S. Anbu
 
Sumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search MasterySumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search Mastery
Sumo Logic
 
Welcome Webinar Slides
Welcome Webinar SlidesWelcome Webinar Slides
Welcome Webinar Slides
Sumo Logic
 
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic Certification - Level 2 (Using Sumo)
Sumo Logic
 
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with KubernetesSumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic Cert Jam - Advanced Metrics with Kubernetes
Sumo Logic
 
Sumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security AnalyticsSumo Logic Cert Jam - Security Analytics
Sumo Logic Cert Jam - Security Analytics
Sumo Logic
 
Sumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - FundamentalsSumo Logic Cert Jam - Fundamentals
Sumo Logic Cert Jam - Fundamentals
Sumo Logic
 
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the CloudAn Infectious Disease Surveillance Simulation (IDSS) in the Cloud
An Infectious Disease Surveillance Simulation (IDSS) in the Cloud
Edison Lascano
 
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic
 
Sumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - AdministrationSumo Logic Cert Jam - Administration
Sumo Logic Cert Jam - Administration
Sumo Logic
 
Vector clock algorithm
Vector clock algorithmVector clock algorithm
Vector clock algorithm
S. Anbu
 
Sumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search MasterySumo Logic Cert Jam - Search Mastery
Sumo Logic Cert Jam - Search Mastery
Sumo Logic
 

Viewers also liked (20)

.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit
Erin Sweeney
 
Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101
allanpyetro
 
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech DaySplunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Zivaro Inc
 
Scalable Monitoring & Alerting
Scalable Monitoring & AlertingScalable Monitoring & Alerting
Scalable Monitoring & Alerting
Franklin Angulo
 
Splunk | Reporting Use Cases
Splunk | Reporting Use CasesSplunk | Reporting Use Cases
Splunk | Reporting Use Cases
Beth Goldman
 
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
Jesse Andrew
 
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
asalters
 
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk ScoringSplunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk
 
SplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud Detection
Splunk
 
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Phil Legg
 
Threat Hunting
Threat HuntingThreat Hunting
Threat Hunting
Tripwire
 
Rapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDealRapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDeal
Splunk
 
Tintricity on the Road: Unified Critical Communications
Tintricity on the Road: Unified Critical CommunicationsTintricity on the Road: Unified Critical Communications
Tintricity on the Road: Unified Critical Communications
Tintri
 
Data Mining with Splunk
Data Mining with SplunkData Mining with Splunk
Data Mining with Splunk
David Carasso
 
Insider Threat Kill Chain: Detecting Human Indicators of Compromise
Insider Threat Kill Chain: Detecting Human Indicators of CompromiseInsider Threat Kill Chain: Detecting Human Indicators of Compromise
Insider Threat Kill Chain: Detecting Human Indicators of Compromise
Tripwire
 
Insider threat event presentation
Insider threat event presentationInsider threat event presentation
Insider threat event presentation
IISPEastMids
 
ThousandEyes Alerting Essentials for Your Network
ThousandEyes Alerting Essentials for Your NetworkThousandEyes Alerting Essentials for Your Network
ThousandEyes Alerting Essentials for Your Network
ThousandEyes
 
Delivering business value from operational insights at ING Bank
Delivering business value from operational insights at ING BankDelivering business value from operational insights at ING Bank
Delivering business value from operational insights at ING Bank
Splunk
 
Procurement Transformation with S/4 HANA Sourcing and Procurement
Procurement Transformation with S/4 HANA Sourcing and ProcurementProcurement Transformation with S/4 HANA Sourcing and Procurement
Procurement Transformation with S/4 HANA Sourcing and Procurement
SAP Ariba
 
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
SAP Ariba
 
.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit.conf2011: Web Analytics Throwdown: with NPR and Intuit
.conf2011: Web Analytics Throwdown: with NPR and Intuit
Erin Sweeney
 
Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101Jornal O Metropolitano - Edição 101
Jornal O Metropolitano - Edição 101
allanpyetro
 
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech DaySplunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day
Zivaro Inc
 
Scalable Monitoring & Alerting
Scalable Monitoring & AlertingScalable Monitoring & Alerting
Scalable Monitoring & Alerting
Franklin Angulo
 
Splunk | Reporting Use Cases
Splunk | Reporting Use CasesSplunk | Reporting Use Cases
Splunk | Reporting Use Cases
Beth Goldman
 
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
IT Incident Communication Buyer's Guide: 10 Questions to ask an IT Alerting V...
Jesse Andrew
 
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
Everbridge Webinar: Top 10 Emergency Notification Predictions for 2011
asalters
 
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk ScoringSplunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk conf2014 - Detecting Fraud and Suspicious Events Using Risk Scoring
Splunk
 
SplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud Detection
Splunk
 
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Visualizing the Insider Threat: Challenges and tools for identifying maliciou...
Phil Legg
 
Threat Hunting
Threat HuntingThreat Hunting
Threat Hunting
Tripwire
 
Rapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDealRapidly Improving Security Posture - CanDeal
Rapidly Improving Security Posture - CanDeal
Splunk
 
Tintricity on the Road: Unified Critical Communications
Tintricity on the Road: Unified Critical CommunicationsTintricity on the Road: Unified Critical Communications
Tintricity on the Road: Unified Critical Communications
Tintri
 
Data Mining with Splunk
Data Mining with SplunkData Mining with Splunk
Data Mining with Splunk
David Carasso
 
Insider Threat Kill Chain: Detecting Human Indicators of Compromise
Insider Threat Kill Chain: Detecting Human Indicators of CompromiseInsider Threat Kill Chain: Detecting Human Indicators of Compromise
Insider Threat Kill Chain: Detecting Human Indicators of Compromise
Tripwire
 
Insider threat event presentation
Insider threat event presentationInsider threat event presentation
Insider threat event presentation
IISPEastMids
 
ThousandEyes Alerting Essentials for Your Network
ThousandEyes Alerting Essentials for Your NetworkThousandEyes Alerting Essentials for Your Network
ThousandEyes Alerting Essentials for Your Network
ThousandEyes
 
Delivering business value from operational insights at ING Bank
Delivering business value from operational insights at ING BankDelivering business value from operational insights at ING Bank
Delivering business value from operational insights at ING Bank
Splunk
 
Procurement Transformation with S/4 HANA Sourcing and Procurement
Procurement Transformation with S/4 HANA Sourcing and ProcurementProcurement Transformation with S/4 HANA Sourcing and Procurement
Procurement Transformation with S/4 HANA Sourcing and Procurement
SAP Ariba
 
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
Going Digital with SAP S/4HANA Sourcing and Procurement and SAP Ariba Solutio...
SAP Ariba
 
Ad

Similar to Splunk .conf2011: Real Time Alerting and Monitoring (20)

Splunk .conf2011: Search Language: Intermediate
Splunk .conf2011: Search Language: IntermediateSplunk .conf2011: Search Language: Intermediate
Splunk .conf2011: Search Language: Intermediate
Erin Sweeney
 
Top 10 Tricks and Tools of an Oracle EPM Administrator
Top 10 Tricks and Tools of an Oracle EPM AdministratorTop 10 Tricks and Tools of an Oracle EPM Administrator
Top 10 Tricks and Tools of an Oracle EPM Administrator
nking821
 
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
BIOVIA
 
Responding to extended events in near real time
Responding to extended events in near real timeResponding to extended events in near real time
Responding to extended events in near real time
Gianluca Sartori
 
SplunkLive Oslo/Stockholm Beginner Workshop
SplunkLive Oslo/Stockholm Beginner WorkshopSplunkLive Oslo/Stockholm Beginner Workshop
SplunkLive Oslo/Stockholm Beginner Workshop
jenny_splunk
 
Setting Up Sumo Logic - Apr 2017
Setting Up Sumo Logic - Apr 2017Setting Up Sumo Logic - Apr 2017
Setting Up Sumo Logic - Apr 2017
Sumo Logic
 
Zentral presentation MacAdmins meetup Univ. Utah
Zentral presentation MacAdmins meetup Univ. Utah Zentral presentation MacAdmins meetup Univ. Utah
Zentral presentation MacAdmins meetup Univ. Utah
Henry Stamerjohann
 
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
NoNameCon
 
Zentral combine power of osquery_santa
Zentral combine power of osquery_santaZentral combine power of osquery_santa
Zentral combine power of osquery_santa
Henry Stamerjohann
 
EnCase Enterprise Basic File Collection
EnCase Enterprise Basic File Collection EnCase Enterprise Basic File Collection
EnCase Enterprise Basic File Collection
Damir Delija
 
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk DevelopmentTSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
Harry McLaren
 
File access auditing
File access auditingFile access auditing
File access auditing
BUSINESS SOFTWARES & SOLUTIONS
 
Regain Control Thanks To Prometheus
Regain Control Thanks To PrometheusRegain Control Thanks To Prometheus
Regain Control Thanks To Prometheus
Etienne Coutaud
 
splunkquickstartsplunkquickstartsplunkquickstart
splunkquickstartsplunkquickstartsplunkquickstartsplunkquickstartsplunkquickstartsplunkquickstart
splunkquickstartsplunkquickstartsplunkquickstart
mitsubishiturbo
 
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike MullerFaster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
PyData
 
Storage, Virtual, and Server Profiler Training
Storage, Virtual, and Server Profiler TrainingStorage, Virtual, and Server Profiler Training
Storage, Virtual, and Server Profiler Training
SolarWinds
 
PCD - Process control daemon
PCD - Process control daemonPCD - Process control daemon
PCD - Process control daemon
haish
 
Continuous Profiling in Production: What, Why and How
Continuous Profiling in Production: What, Why and HowContinuous Profiling in Production: What, Why and How
Continuous Profiling in Production: What, Why and How
Sadiq Jaffer
 
HP Quick Test Professional
HP Quick Test ProfessionalHP Quick Test Professional
HP Quick Test Professional
Vitaliy Ganzha
 
PCD - Process control daemon - Presentation
PCD - Process control daemon - PresentationPCD - Process control daemon - Presentation
PCD - Process control daemon - Presentation
haish
 
Splunk .conf2011: Search Language: Intermediate
Splunk .conf2011: Search Language: IntermediateSplunk .conf2011: Search Language: Intermediate
Splunk .conf2011: Search Language: Intermediate
Erin Sweeney
 
Top 10 Tricks and Tools of an Oracle EPM Administrator
Top 10 Tricks and Tools of an Oracle EPM AdministratorTop 10 Tricks and Tools of an Oracle EPM Administrator
Top 10 Tricks and Tools of an Oracle EPM Administrator
nking821
 
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
(ATS3-PLAT07) Pipeline Pilot Protocol Tips, Tricks, and Challenges
BIOVIA
 
Responding to extended events in near real time
Responding to extended events in near real timeResponding to extended events in near real time
Responding to extended events in near real time
Gianluca Sartori
 
SplunkLive Oslo/Stockholm Beginner Workshop
SplunkLive Oslo/Stockholm Beginner WorkshopSplunkLive Oslo/Stockholm Beginner Workshop
SplunkLive Oslo/Stockholm Beginner Workshop
jenny_splunk
 
Setting Up Sumo Logic - Apr 2017
Setting Up Sumo Logic - Apr 2017Setting Up Sumo Logic - Apr 2017
Setting Up Sumo Logic - Apr 2017
Sumo Logic
 
Zentral presentation MacAdmins meetup Univ. Utah
Zentral presentation MacAdmins meetup Univ. Utah Zentral presentation MacAdmins meetup Univ. Utah
Zentral presentation MacAdmins meetup Univ. Utah
Henry Stamerjohann
 
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
Nazar Tymoshyk et al - Night in Defense Workshop: Hunting for a needle in a h...
NoNameCon
 
Zentral combine power of osquery_santa
Zentral combine power of osquery_santaZentral combine power of osquery_santa
Zentral combine power of osquery_santa
Henry Stamerjohann
 
EnCase Enterprise Basic File Collection
EnCase Enterprise Basic File Collection EnCase Enterprise Basic File Collection
EnCase Enterprise Basic File Collection
Damir Delija
 
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk DevelopmentTSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
TSTAS, the Life of a Splunk Trainer and using DevOps in Splunk Development
Harry McLaren
 
File access auditing
File access auditingFile access auditing
File access auditing
BUSINESS SOFTWARES & SOLUTIONS
 
Regain Control Thanks To Prometheus
Regain Control Thanks To PrometheusRegain Control Thanks To Prometheus
Regain Control Thanks To Prometheus
Etienne Coutaud
 
splunkquickstartsplunkquickstartsplunkquickstart
splunkquickstartsplunkquickstartsplunkquickstartsplunkquickstartsplunkquickstartsplunkquickstart
splunkquickstartsplunkquickstartsplunkquickstart
mitsubishiturbo
 
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike MullerFaster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
PyData
 
Storage, Virtual, and Server Profiler Training
Storage, Virtual, and Server Profiler TrainingStorage, Virtual, and Server Profiler Training
Storage, Virtual, and Server Profiler Training
SolarWinds
 
PCD - Process control daemon
PCD - Process control daemonPCD - Process control daemon
PCD - Process control daemon
haish
 
Continuous Profiling in Production: What, Why and How
Continuous Profiling in Production: What, Why and HowContinuous Profiling in Production: What, Why and How
Continuous Profiling in Production: What, Why and How
Sadiq Jaffer
 
HP Quick Test Professional
HP Quick Test ProfessionalHP Quick Test Professional
HP Quick Test Professional
Vitaliy Ganzha
 
PCD - Process control daemon - Presentation
PCD - Process control daemon - PresentationPCD - Process control daemon - Presentation
PCD - Process control daemon - Presentation
haish
 
Ad

Recently uploaded (20)

How analogue intelligence complements AI
How analogue intelligence complements AIHow analogue intelligence complements AI
How analogue intelligence complements AI
Paul Rowe
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Aqusag Technologies
 
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessGenerative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in Business
Dr. Tathagat Varma
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Impelsys Inc.
 
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
organizerofv
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-UmgebungenHCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
panagenda
 
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfComplete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Software Company
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 
How analogue intelligence complements AI
How analogue intelligence complements AIHow analogue intelligence complements AI
How analogue intelligence complements AI
Paul Rowe
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBuild Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For Devs
Brian McKeiver
 
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...
Aqusag Technologies
 
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessGenerative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in Business
Dr. Tathagat Varma
 
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfThe Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf
Abi john
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...
Impelsys Inc.
 
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...
organizerofv
 
Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025Splunk Security Update | Public Sector Summit Germany 2025
Splunk Security Update | Public Sector Summit Germany 2025
Splunk
 
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc
 
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-UmgebungenHCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
panagenda
 
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfComplete Guide to Advanced Logistics Management Software in Riyadh.pdf
Complete Guide to Advanced Logistics Management Software in Riyadh.pdf
Software Company
 
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025
BookNet Canada
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)
Ortus Solutions, Corp
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...
BookNet Canada
 
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsAI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global Trends
InData Labs
 
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionDrupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy Consumption
Exove
 

Splunk .conf2011: Real Time Alerting and Monitoring

  • 1. Monitoring and Alerting Ledion Bitincka, Search and Alerting Team
  • 2. Search and Reporting Team @ Splunk for 4+ years - since 3.0 Things I’ve worked on: Key-value extractions Transactions, Eventtyping, Typeahead, Summary Indexing Monitoring and alerting framework Other random @#$% Intro … Ledion Bitincka (aka Splunk Albanian)
  • 3. Why use Splunk for monitoring and alerting? Basic alerting Advanced alerts and config options Real-time alerting and throttling (new in 4.2) Alert Manager (new in 4.2) Sneak peek into new features … Feel free to interrupt when you don’t follow!!! Agenda
  • 4. Life Without Splunk Service Desk Application Support Systems Administrator Application Developer Application Developer Database Administrator Log call. The console says everything is green. App monitoring tools don’t show anything either. Call the developer. Stop working on new code to troubleshoot. Need production logs! Stop what they’re doing to identify and gather production logs for developer. Manual investigation establishes not application problem. DBA analyzes the logs which points to corrupted database files. Escalate. Escalate. Escalate. Respond. Escalate. Now what?
  • 5. Life With Splunk Service Desk Trouble Ticket Search on IP address shows related Web session and User ID “ 192.168.169.100” Last 60 minutes 192.168.169.100 Search at same time reveals database error due to corrupted files Search for failure or error across entire IT Last 2 minutes failure OR error Search on corruption in the db logs shows that an index file has been corrupted Search for corruption in db logs Last 1 minute host=db.domain.com source=*db.log corrupt* Setup monitoring and alerting for db file corruption Set up Monitoring and Alerting Last hour host=db.domain.com source=*db.log corrupt*
  • 7. Monitor and Alert in Real Time 2. Evaluate alerting condition 1. Get data Scheduled search Real-time search Alert Condition 3. Execute actions RSS Email SNMP Script Yes Noop No
  • 8. Create simple alert using wizard … Available alert actions … Configure email settings (MTA, link hostname)... Demo… (5-10 min)
  • 9. Advanced Alerting Options Specify an advanced schedule using cron notation Use custom alert conditions Invoke scripts to perform custom actions Integrate with other tools file trouble ticket Other custom processing restart a faulty service update a firewall rule temporarily disable a user account etc …
  • 10. Real-time Search Primer Searches forward in time Never completes (unless stopped) Constantly updating result set Only generates results preview All search commands supported
  • 11. Splunkd/ Scheduler Search Process time Search Start historical search Suppress? Logging Scheduled Search Alerts audit.log search.log Y N Notify splunkd splunkd_access.log audit.log Search done Execute actions Update artifact TTL Suppression update Alert manager N Y Done scheduler.log Condition Results
  • 12. Real-time Alerts Splunkd/ Scheduler Search Process time RT Search Start RT search Suppress? Logging … .. audit.log search.log Y N Notify splunkd splunkd_access.log Execute actions Update artifact TTL Suppression update Alert manager N Y Condition ResPrev Done scheduler.log Condition ResPrev N Y Results Snapshot
  • 13. Real-time Alerts Reduce response time Continuously monitor a condition Scheduler ensures real-time search is always running Throttling is almost always necessary Compatible with all alert actions Visible through Alerts Manager
  • 14. Alert Throttling Natively support alert action throttling Useful in: Alert when database server is down, but don’t alert me about this condition for one hour Available for both standard and real-time alerts
  • 15. Alerts Manager System-wide view of all triggered alerts Basic alert management features Ability to drill down and view why the alert was triggered Real-time alert results are snapshots in time when triggered
  • 16. Demo… (5-10 min) Show custom alert conditions, when to use them Demo real-time alerts: Throttling Alert manager
  • 17. Managing Search Load System wide concurrent searches limited to Total: 4 + 4 x number of cores Limit used for ad-hoc and scheduled searches Scheduler queues over limit searches Scheduler allocation is configurable in limits.conf [scheduler] max_searches_perc = 25 // percentage of system wide concurrent searches to use … Use the Scheduler Activity dashboards Search App >> Status >> Scheduler Activity >> Overview Search allocation
  • 18. savedsearches.conf Search string, schedule, alert condition, actions etc… alert_actions.conf Alert action options such as: email server, format, subject line, ttl etc… limits.conf Scheduler’s concurrent search limit Action execution related limits scheduler.log Look in $SPLUNK_HOME/etc/system/README/<filename>.conf.spec for more detailed info .conf & .log File Summary
  • 19. Per result alerting and throttling More alert actions to enable more complex alerting conditions Once five failed login attempts occur enable a monitoring search that alerts on suspicious user activity Sneak Peek Into New Features
  • 20. How scheduled and real-time alerts work Create simple and advanced real-time alerts Enable alert throttling and check for throttled alerts Check fired alerts using Alerts Manager Change scheduler limit defaults Be an IT hero  Now You Should Know …
  • 21. August 15, 2011 Questions? Ledion Bitincka, Search and Alerting Team