SplunkLive! Splunk for Security

Editor's Notes

• #5: The number of threats is increasing and also becoming more advanced. Today’s advanced threats are stealthy and sophisticated and evade detection from traditional, point security products that look for specific threat signatures. Above are 3 types of advanced threats. They are good at stealing confidential data, wherther it be credit cards or IP, and many of their victims unfortunately end up in the headlines.FYI these advanced threats are also commonly called APTs, or Advanced Persistent Threats.
• #6: APT are hard to detect because they are not signature-based and hide behind legitimate credentialed activity to evade detection from traditional, point security products. Every year companies like Mandiant produce reports that describe the trends identified based on the breach investigation work that they do as part of their consulting practices. There are a couple metrics that I found interesting reading their recent reports.100% is often via stealing password hashes or using keyloggers. Often they steal admin-level credentials so they can access many other systems and not be detected.The 40 implies that even if you see malware in one place, you need to look much further as there are likely multiple infected machines and backdoors243 days shows how they can evade detection for months at a time. They move slow and low and do not set off alarms from point, signature-based security products like anti-malware solutions.63% of victims were notified by an external entity. Notification usually starts with customer complaints like bank account drained or credit card maxed out. Often FBI informs them.
• #7: Key part of IT security is protecting confidential data. Which means detecting advanced threats, like cybercriminals or malicious insiders, before they can steal your data. To detect or investigatethem, you need non-security and security data because advanced threats avoid detection from signature-based security products; the fingerprints of an advanced threat often are in the “non-security” data. Most traditional SIEMs just focus on gathering signature-based threats which do *not* have the fingerprints of advanced threats.Also the above scenario is worse if there is no SIEM. Instead point UIs and grep are used and aggregating data is very manual and time consuming.
• #8: Here is why SIEMs are not catching the advanced threats and not generating value in general. (A later slide has more detail on Splunk points of differentiation. )For point 1, traditional SIEMs suffer from:Collectors or backend database require data reduction or normalization – hampers security use casesLittle support for custom data sourcesBrittle collectors break when data format changesClock icon indicates slow searches and long time to deploy. Dollar sign is the expensive costs behind the long deployment (lots of prof serv) and multiple products.
• #9: Splunk can ingest any type of machine data, from any source in real time. These are listed here on the left and are flowing into Splunk for indexing. Once indexed, users can perform the use cases on the top right on the data. They can search through the data, monitor the data and be alerted in real-time if scheduled search parameters are met. The raw data can be aggregated in seconds for custom reports and dashboards. Also Splunk is a platform that developers can build on. It uses a well documented Rest API and several SDKs so developers and external; applications can directly access and act on the data within Splunk. Lastly, besides indexing raw data into its flat file data store, Splunk can also retreive and index data that resides in other data stores such as a SQL database or Hadoop.
• #10: Over 2500 security/compliance customers worldwide. Customers cover all sizes and verticals, and are all over the world. While not listed here, hundreds of SMBs and individuals also use for security/compliance.
• #11: SC magazine award was determined by the readers of SC Mag who are IT Security professionals. We beat out:HP for ArcSight ExpressIBM Software Group for QRadar SIEMLogRhythm for LogRhythmNetIQ for NetIQ Sentinel 7SolarWinds for SolarWinds Log & Event Manager (LEM)
• #12: SC magazine award was determined by the readers of SC Mag who are IT Security professionals. We beat out:HP for ArcSight ExpressIBM Software Group for QRadar SIEMLogRhythm for LogRhythmNetIQ for NetIQ Sentinel 7SolarWinds for SolarWinds Log & Event Manager (LEM)
• #13: make sure to stress SIP and we can meet their needs w/4 use cases. We are more than a SIEM in that we are much more flexible and also can be used for use cases outside of security
• #14: Use case 1. Alert from a point product UI or traditional SIEM. Pin board image on right indicates “cold case/CSI” sort of investigation that Splunk can enable. (FYI - papers on the pin board image do not tell a “real” investigation story so do not try to read all the images on the pin board). From a forensics perspective, things like endpoint OS logs or packet captures can be put into Splunk at the time of the investigation to get deeper into the details.With exiting SIEMs they struggle with incident investigations because they cannot:Retain all the original unmodified data (because the normalize/reduce it)Often it is hard to pivot among the data b/c it is in different data stores (logger/SIEM/Hadoop/etc) with no common UIQuickly return back search results (because their DB causes scale/speed issues)Limited flexibility/ability to do external lookups
• #15: Use case 2 (a slide with building images)(*) Ad-hoc reports for security investigation, executives, auditors. Show me all internal machines connecting back to a known, bad external IP over the last 30 days.(*) Daily list of new security events that IT Security reviews each morning. They use this to decide what incidents to look into that day. Malware discovered, data loss events, etc.(*) Reports that count up top 10 sort of lists showing top malware infections, IDS attacks, failed logins by user, etc. (*) Security or Operations Network Center might have world map overlayed with security incidents, or maybe yellow/red/green dials showing the overall threat levels.Executives may want to see trendlines showing threats over time to see that risk is decreasing.This is a sample dashboard built by a customer
• #16: Use case 3. It is about about taking thousands of security events that are low severity in isolation and connecting the dots in an automated, policy-driven manner to see when a combination of seemingly low severity events, when correlated, is actually a high-severity incident that needs immediate attention.There are hundreds of possible cross-product correlations. One is above and tells the story of a data loss event being detected by signature-based security productsFor a specific internal IP address running Windows, someone logs into it using a default administrative user name “Administrator” which is not good. All users should have a unique user name (not root or Administrator) so you know exactly who is doing what in the IT environment. The OS logs see this log in.Endpoint-based anti-malware sees known, bad malware running on that machine. Malware means “malicious software” and is a red flag because it may lead to data being stolen by a hackerA data loss prevention tool (in this case the Snort Intrusion Detection Prevention product) sees unencrypted credit card numbers leaving the organization from the above machine. This data loss of credit cards is a major red flag.Why these 3 events are bad: These 3 events happening on the same machine in a short time period indicates a hacker inappropriately logged into the machine, probably using stolen credentials, then put malware on the machine, perhaps a backdoor to remotely connect back to the machine later, then exfiltrated stolen credit cards from the machine. The credit cards may have then been used for illegal purposes which ultimately may have resulted in the costs of re-issuing credit cards, bad publicity, unhappy customers taking their business elsewhere, customer lawsuits, fines for PCI non-compliance, etc.Splunk can correlate on all these 3 events happening on the same machine and within a short time period. It has connected the dots to find the proverbial needle in the haystack. Splunk can detect and/or alert on these sorts of correlations in real-time or on a scheduled bases.Two other sample correlations:Firewall on an internal PC indicates the PC is being port scanned from an internal IP addressNetwork-based firewall indicates it is being port scanned from the same internal IP addressImportant settings have been changed on the suspicious internal machineWhy: The machine associated with the IP address may have been compromised by a threat which is doing internal reconnaissanceVulnerability scanner shows that an internal server has an unpatched OSIntrusion Detection System sees an external attack on that specific server that exploits the vulnerability in the OS Why: The server is likely to be successfully compromised
• #17: Use case 4. Like prior slide so see notes at the top. But in this case the events being correlated on all are from “non-security” data sources. This is because the threats are “unknown” to traditional security products because no signature exists for them. Each of these events in isolation would raise no alarms. Only when combined can see you see that they are risky because they represent outliers/anomolies that could be advanced threats like a sophisticated cybercriminal or a nation-state.There are hundreds of possible cross-product correlations. One is above and tells the story of a spearphising attack done in order to obtain and steal confidential data. More sample correlations are on the next slide. In this scenario above Splunk is keeping track of all the external email domains that are sending emails into the company, all external web sites being visited by internal employees, and all the services and executables running on internal machines. It can automatically count up things like # of emails received by each external domain, # of times employees visit external web domains, etc, to see the rarely seen items that are outliers. In the above scenario:Splunk sees an email reach an internal employee from an external email domain that has never/rarely been seen beforeThat same employee then visits a web site that is never/rarely visited by internal employeesA service starts up on the employees machine that is never/rarely seen in the organizationWhy these 3 events are bad: These 3 events happening on the same machine in a short time period indicates a hacker has performed a spearphising attack. They sent a realistic looking email to an internal employee that compelled the employee to click on a link or open an executable. This resulted in the web site dropping malware on the machine.Splunk can correlate on all these 3 events happening on the same machine and within a short time period. It has connected the dots to find the proverbial needle in the haystack. Splunk can detect and/or alert on these sorts of correlations in real-time or on a scheduled bases.
• #18: More on use case 4. Nothing on this slide and next would generate an alert in a traditional security tool like anti-malware or firewall. All of these “what to look for” can be automated, real-time searches. These are just a few examples of how to detect what may be an advanced threat.#2 Haversine algorithm used to calculate distance
• #19: Customers start by using Splunk Enterprise to address one specific solution area. Then they leverage it and their machine data to solve other pressing problems over time.Consequently, Splunk Enterprise has many critical uses across IT and the business: Application Management: provide end-to-end visibility across distributed infrastructures; troubleshoot across application environments; monitor for performance degradation; trace transactions across distributed systems and infrastructure.Development: accelerate development and test cycles; support advanced development methodologies like agile, continuous; integrate enterprise applications with SDKs and a robust API; build enterprise applications that leverage Splunk software.Infrastructure and Operations Management: proactively monitor across IT silos to ensure uptime; rapidly pinpoint and resolve problems; report on SLAs/track SLAs of service providers.Security and Compliance: provide rapid incident response, real-time correlation and in-depth monitoring across data sources; statistical analysis for advance pattern detection and threat defense.Web and Business Analytics: gain visibility and intelligence on customers, services and transactions; identify trends and patterns in real time; fully understand the impact of new product features on back-end services.
• #21: 1 solution for Splunk for Security, but 3 offerings. At bottom is Splunk Enterprise, our core product. Every Splunk deployment includes this as this is where the core indexing and searching resides. Many customers build their own searches/reports/dashboards on top of it. On top of it, optional Apps can be installed. Apps are basically a collection of reports, dashboards, and searches purpose-built for a specific use case or product. Can be built by Splunk, customer, partners and all but a few are free on Splunkbase. Apps are great for customers who want out-of-the-box content and do want to have to build it themselves, and want to extend point solutions. One key App is the Splunk-built Enterprise Security app with the arrow pointing at it. It is basically an out-of-the-box SIEM with reports, dashboards, correlation rules, and workflow for security use cases. (It does have a cost though) Besides this app there are over 80 security-centric free Apps on Splunkbase. These are offering 3.The majority of Splunk security customers do Splunk Enterprise and the free apps. Also customers do leverage the API and SDKs that come with Splunk to further extend the platform.
• #22: You need to look for abnormal events in normal user activity data. SIEM not built for this and the architecture doesn’t support it. Data reduction modelTypical funnel representation of:Limited data collectionData subset used for securitySubset may fire one of 200+ rulesLinear approach – no going back in time1. Traditional SIEM / log management sources – almost always not all your sources are supported. Custom application data almost always left out2. Data sent to a log managed solution for scale and filtering only to ‘security related” events. Nice to have information level data from security devices and data sent by applications left out.3. Security data sent to the SIEM – only that which fires a rule responded to.4.Reinforces silos between applications, operations and security. This means investigations limited to only what you expect to see – you never know what you don’t know! Splunk – a data inclusion model – ALL data is security relevantIf a correlation search produces a positive result – investigations aren’t artificially limited. Investigations can include seeing the affect of a security event all the way to the application – up to the second. Pattern and threshold based rather than rules based – watch for a pattern of activity get an alert before and after it hits a thresholdMassive scale can offer broader view of attack vectors and surfacesOperations and security data seen in the same system – breaks down silos/barriers