Splunk for Security - Hands-On
0 likes•754 views
The document outlines a presentation by Splunk Inc. focusing on using their software to disrupt the cyber kill chain in security investigations. It emphasizes the importance of collecting and analyzing machine data to identify and mitigate security threats, with a practical example illustrating the investigation of a phishing attack involving malware embedded in a PDF file. The content highlights the effectiveness of Splunk in enhancing security intelligence beyond traditional SIEM solutions, enabling real-time monitoring and forensic capabilities.
Splunk for Security - Hands-On
- 1. Copyright © 2015 Splunk Inc. Hands-On Security Disrupting the Cyber Kill Chain using Splunk
- 2. 2 Safe Harbor Statement During the course of this presentation,we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described orto includeany suchfeatureor functionalityina futurerelease.
- 3. 3 Agenda Splunk & Security – Unknown threats – Connect the dots across all data Kill Chain* Disruption – Overview Exercise/Demo – Security investigation example
- 6. Machine Data contains a definitive record of all Human <-> Machine & Machine <-> Machine Interaction. Splunk is a very effective platform to collect, store, and analyze all of that data.
- 7. Servers Storage DesktopsEmail Web Transaction Records Network Flows DHCP/ DNS Hypervisor Custom Apps Physical Access Badges Threat Intelligence Mobile CMDB Intrusion Detection Firewall Data Loss Prevention Anti- Malware Vulnerability Scans Traditional Authentication Connect the Dots acrossAll Data
- 8. 8 Splunk software complements, replaces and goes beyond traditional SIEMs. Moving Past SIEM to Security Intelligence Small Data. Big Data. Huge Data. SECURITY & COMPLIANCE REPORTING REAL-TIME MONITORING OF KNOWN THREATS MONITORING OF UNKNOWN THREATS INCIDENT INVESTIGATIONS & FORENSICS FRAUD DETECTION INSIDER THREAT
- 9. 9 Splunk software complements, replaces and goes beyond traditional SIEMs. Moving Past SIEM to Security Intelligence Small Data. Big Data. Huge Data. SECURITY & COMPLIANCE REPORTING REAL-TIME MONITORING OF KNOWN THREATS MONITORING OF UNKNOWN THREATS INCIDENT INVESTIGATIONS & FORENSICS FRAUD DETECTION INSIDER THREAT
- 10. Kill Chain*Disruption Your system is compromised and the adversary begins its work Exploitation The adversary works to understand your organization looking for opportunities Reconnaissance The attacker steals data, disrupts your operations or causes damage Act on Intent *mostly….
- 11. • Q. How can the security analysts at Buttercup Games, Inc. discover that their systems have been compromised by way of a stolen document from their web portal? • A. They would want to discover and disrupt the kill chain: • Where did the adversary start? (Recon) • How did they get a foothold? (Exploitation) • What was their motive and what did they take? (Actions on Intent) Security InvestigationExample bu tercup games Let’s get hands-on!
- 12. 13 Reconnaissance Weaponization Delivery Exploitation Installation Command & Control Act on Objectives Web Kill Chain Demo Data Source - Activity Email and Endpoint Endpoint Endpoint, DNS, Proxy Endpoint, DNS, Proxy A brute force attack takes place on the customer web site, access is gained, and a sensitive pdf file is downloaded and weaponized with malware. A convincing phishing email is crafted and sent to an internal target The pdf document is opened then exploits the vulnerable pdf reader app creating a dropper which installs the malware. Command/Control activity is highlighted by it’s association with Threat Intelligence Demo Story line Threat Intelligence Integration
- 13. 14 APT Transaction Flow Across Data Sources http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web Data Sources .pdf .pdf executes & unpacks malware overwriting and running “allowed” programs Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment Our Investigation begins by detecting high risk communications through the proxy, at the endpoint, and even a DNS call.
- 14. System Monitor (SYSMON) is the application programming interface (API) that you use to configure the Microsoft System Monitor ActiveX control. The System Monitor control lets you view real-time and previously logged performance counter data such as memory, disk, and processor counter data. SYSMON is available starting with Microsoft Windows 2000. Provides a data input and CIM-compliant field extractions for Microsoft Sysmon data input. The Microsoft SYSMON utility provides data on process creation (including parent process ID) and network connections. MicrosoftSystem Monitor (SYSMON)
- 15. 16 Demo
- 16. 17 To begin our investigation, we will start with a quick search to familiarize ourselves with the data sources. In this demo environment, we have a variety of security relevant data including… Web DNS Proxy Firewall Endpoint Email
- 17. 18 Take a look at the endpoint data source. We are using the Microsoft Sysmon TA. We have endpoint visibility into all network communication and can map each connection back to a process. } We also have detailed info on each process and can map it back to the user and parent process.} Lets get our day started by looking using threat intel to prioritize our efforts and focus on communication with known high risk entities.
- 18. 19 We have multiple source IPs communicating to high risk entities identified by these 2 threat sources. We are seeing high risk communication from multiple data sources. We see multiple threat intel related events across multiple source types associated with the IP Address of Chris Gilbert. Let’s take closer look at the IP Address. We can now see the owner of the system (Chris Gilbert) and that it isn’t a PII or PCI related asset, so there are no immediate business implications that would require informing agencies or external customers within a certain timeframe. This dashboard is based on event data that contains a threat intel based indicator match( IP Address, domain, etc.). The data is further enriched with CMDB based Asset/identity information.
- 19. 20 We are now looking at only threat intel related activity for the IP Address associated with Chris Gilbert and see activity spanning endpoint, proxy, and DNS data sources. These trend lines tell a very interesting visual story. It appears that the asset makes a DNS query involving a threat intel related domain or IP Address. ScrollDown Scroll down the dashboard to examine these threat intel events associated with the IP Address. We then see threat intel related endpoint and proxy events occurring periodically and likely communicating with a known Zeus botnet based on the threat intel source (zeus_c2s).
- 20. 21 It’s worth mentioning that at this point you could create a ticket to have someone re-image the machine to prevent further damage as we continue our investigation within Splunk. Within the same dashboard, we have access to very high fidelity endpoint data that allows an analyst to continue the investigation in a very efficient manner. It is important to note that near real-time access to this type of endpoint data is not not common within the traditional SOC. The initial goal of the investigation is to determine whether this communication is malicious or a potential false positive. Expand the endpoint event to continue the investigation. Proxy related threat intel matches are important for helping us to prioritize our efforts toward initiating an investigation. Further investigation into the endpoint is often very time consuming and often involves multiple internal hand-offs to other teams or needing to access additional systems. This encrypted proxy traffic is concerning because of the large amount of data (~1.5MB) being transferred which is common when data is being exfiltrated.
- 21. 22 Exfiltration of data is a serious concern and outbound communication to external entity that has a known threat intel indicator, especially when it is encrypted as in this case. Lets continue the investigation. Another clue. We also see that svchost.exe should be located in a Windows system directory but this is being run in the user space. Not good. We immediately see the outbound communication with 115.29.46.99 via https is associated with the svchost.exe process on the windows endpoint. The process id is 4768. There is a great deal more information from the endpoint as you scroll down such as the user ID that started the process and the associated CMDB enrichment information.
- 22. 23 We have a workflow action that will link us to a Process Explorer dashboard and populate it with the process id extracted from the event (4768).
- 23. 24 This is a standard Windows app, but not in its usual directory, telling us that the malware has again spoofed a common file name. We also can see that the parent process that created this suspicuous svchost.exe process is called calc.exe. This has brought us to the Process Explorer dashboard which lets us view Windows Sysmon endpoint data. Suspected Malware Lets continue the investigation by examining the parent process as this is almost certainly a genuine threat and we are now working toward a root cause. This is very consistent with Zeus behavior. The initial exploitation generally creates a downloader or dropper that will then download the Zeus malware. It seems like calc.exe may be that downloader/dropper. Suspected Downloader/Dropper This process calls itself “svchost.exe,” a common Windows process, but the path is not the normal path for svchost.exe. …which is a common trait of malware attempting to evade detection. We also see it making a DNS query (port 53) then communicating via port 443.
- 24. 25 The Parent Process of our suspected downloader/dropper is the legitimate PDF Reader program. This will likely turn out to be the vulnerable app that was exploited in this attack. Suspected Downloader/Dropper Suspected Vulnerable AppWe have very quickly moved from threat intel related network and endpoint activity to the likely exploitation of a vulnerable app. Click on the parent process to keep investigating.
- 25. 26 We can see that the PDF Reader process has no identified parent and is the root of the infection. ScrollDown Scroll down the dashboard to examine activity related to the PDF reader process.
- 26. 27 Chris opened 2nd_qtr_2014_report.pdf which was an attachment to an email! We have our root cause! Chris opened a weaponized .pdf file which contained the Zeus malware. It appears to have been delivered via email and we have access to our email logs as one of our important data sources. Lets copy the filename 2nd_qtr_2014_report.pdf and search a bit further to determine the scope of this compromise.
- 27. 28 Lets search though multiple data sources to quickly get a sense for who else may have have been exposed to this file. We will come back to the web activity that contains reference to the pdf file but lets first look at the email event to determine the scope of this apparent phishing attack.
- 28. 29 We have access to the email body and can see why this was such a convincing attack. The sender apparently had access to sensitive insider knowledge and hinted at quarterly results. There is our attachment. Hold On! That’s not our Domain Name! The spelling is close but it’s missing a “t”. The attacker likely registered a domain name that is very close to the company domain hoping Chris would not notice. This looks to be a very targeted spear phishing attack as it was sent to only one employee (Chris).
- 29. 30 Root Cause Recap 3 Data Sources .pdf executes & unpacks malware overwriting and running “allowed” programs http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web .pdf Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment We utilized threat intel to detect communication with known high risk indicators and kick off our investigation then worked backward through the kill chain toward a root cause. Key to this investigative process is the ability to associate network communications with endpoint process data. This high value and very relevant ability to work a malware related investigation through to root cause translates into a very streamlined investigative process compared to the legacy SIEM based approach.
- 30. 31 Lets revisit the search for additional information on the 2nd_qtr_2014- _report.pdf file. We understand that the file was delivered via email and opened at the endpoint. Why do we see a reference to the file in the access_combined (web server) logs? Select the access_combined sourcetype to investigate further.
- 31. 32 The results show 54.211.114.134 has accessed this file from the web portal of buttergames.com. There is also a known threat intel association with the source IP Address downloading (HTTP GET) the file.
- 32. 33 Select the IP Address, left-click, then select “New search”. We would like to understand what else this IP Address has accessed in the environment.
- 33. 34 That’s an abnormally large number of requests sourced from a single IP Address in a ~90 minute window. This looks like a scripted action given the constant high rate of requests over the below window. ScrollDown Scroll down the dashboard to examine other interesting fields to further investigate. Notice the Googlebot useragent string which is another attempt to avoid raising attention..
- 34. 35 The requests from 52.211.114.134 are dominated by requests to the login page (wp-login.php). It’s clearly not possible to attempt a login this many times in a short period of time – this is clearly a scripted brute force attack. After successfully gaining access to our website, the attacker downloaded the pdf file, weaponized it with the zeus malware, then delivered it to Chris Gilbert as a phishing email. The attacker is also accessing admin pages which may be an attempt to establish persistence via a backdoor into the web site.
- 35. 36 Kill Chain Analysis Across Data Sources http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web Data Sources .pdf .pdf executes & unpacks malware overwriting and running “allowed” programs Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment We continued the investigation by pivoting into the endpoint data source and used a workflow action to determine which process on the endpoint was responsible for the outbound communication. We Began by reviewing threat intel related events for a particular IP address and observed DNS, Proxy, and Endpoint events for a user in Sales. Investigation complete! Lets get this turned over to Incident Reponse team. We traced the svchost.exe Zeus malware back to it’s parent process ID which was the calc.exe downloader/dropper. Once our root cause analysis was complete, we shifted out focus into the web logs to determine that the sensitive pdf file was obtained via a brute force attack against the company website. We were able to see which file was opened by the vulnerable app and determined that the malicious file was delivered to the user via email. A quick search into the mail logs revealed the details behind the phishing attack and revealed that the scope of the compromise was limited to just the one user. We traced calc.exe back to the vulnerable application PDF Reader.
- 36. Questions?
- 37. 38 Prizes in Exchange for Your Survey Feedback! Text Splunk to 878787 OR Scan this QR Code Then stop by our registration desk for a free gift and a chance to win a $100 AMEX gift cards
- 38. Thank You
Editor's Notes
- #3: Splunk safe harbor statement.
- #5: We are going to get hands on, and I want to make sure we have enough time to go through the exercises. But I need to frame up why this is important first, so bear with me.
- #6: TinyURL link to student doc
- #7: If you could have a record of every machine to machine interaction, and every human to machine interaction, in your computing environment, what would that mean to you in terms of security? It would mean you would have a great platform to quickly find the known and unknown threats that are working against your business. As we all know there are plenty of adversaries out there: organized cyber criminal rings, nation states, insider threats, and we need to try and keep tabs on all of them. Mandiant and Verizon reports – updated yearly – all breaches use some sort of valid credentials. A small number of systems are involved – maybe an average of ~40 – that’s a really small footprint. And there’s usually 229 days that go by before someone outside your organization (secret service, FBI, etc) tells you that you have been breached. We need to react quickly to indicators of compromise to shut down attackers before they can do harm. So how do we do that? We collect a lot of data….
- #8: So to find all of the “known” threats – that’s pretty simple. We’re going to grab all of the data from all of the things in your environment that are regularly updated with knowledge about known threats. All the traditional security stuff. - IDS/IPS. Anti-malware defenses. DLP, Vulnerability scans. SIEM technology. And of course we will collect firewall data and auth data. But what about unknown threats? How do we find those? Well we need to look at a much bigger set of data, and then find the unusual patterns in that data. So we want to look at things like threat intel, email, web, desktops – the first four items on the top line are what we’ll focus on in this hands-on exercise. But note that Splunk can collect a whole lot more data that we believe is extremely security relevant – especially when it comes to detecting those unknown threats.
- #9: These are the main areas we see Splunk being used for security. We can provide demos and examples and case studies for all of these, but today’s hands-on will focus on…
- #10: …incident investigation and forensics. This is particularly interesting because even if you have a SIEM today, and you like it, Splunk can complement that SIEM by being a highly powerful and scalable investigative engine for your incident investigations group. We often see customers that are not quite ready to get rid of their traditional SIEM, using us for this function, because we are so flexible and scalable and fast.
- #12: The phrase, Cyber Kill Chain, was coined by researchers from Lockheed Martin. For a complete list of Phases please see: http://www.lockheedmartin.com/content/dam/lockheed/data/corporate/documents/LM-White-Paper-Intel-Driven-Defense.pdf This is how a common adversary – i.e. someone that wants to steal your data, or your intellectual property, or cause downtime, or embarrass your org, sees you and plans an attack. They could be cyber criminals, they could be nation states, they could be coming from the inside. Recon: Discover/identify a target. Maybe not even any direct communications with the target. They might be googling things to see if you have web portals that they can attack. Look at public records or proposals to see who is working on what, or what new products you are going to offer, or see who you have recently hired. Basically looking for something to exploit… …which then happens via a multitude of ways. This is where the actual compromise takes place. Commonly, some artifact that is trusted is weaponized and delivered. This is also where social engineering may come in to steal credentials (100% of breaches involve stolen, valid credentials). Then act on intent – what do they want to do to you? Steal your data or IP or affect your operations or cause physical harm, like we saw with Stuxnet a few years ago with the centerfuges. The ones we are glossing over here are weaponization, delivery, installation, command and control. We will see those in the exercise.
- #13: Splunk is the only security analytics platform that allows analysts and incident investigators to leverage these disparate data sources to disrupt the adversary kill chain. This demo shows a real world investigation scenario for the Zeus attack. We begin the investigation by searching for events for new threat intelligence and investigate the infection and identify the complete adversary kill chain. This hands-on exercise shows a real world investigation scenario for the Zeus malware. Q. Why zeus? A. Malware that reports into a botnet, been around since 2009 Disrupts services, acquires financial data, installs ransomware to lock up machines. Very effective, very elusive – every time we think we have a handle on it it comes back because it keeps morphing. Attempts to shut it down largely unsuccessful. This is NOT to show you how we can perform an investigation surrounding Zeus – we are only using that because it’s a commonly known bit of malware. These techniques can be used for any kind of security investigation within Splunk.
- #14: We map the Kill Chain to Data Sources then explain what we are going to show them in the demo. We are integrating Threat Intel and adding context to events when matching indicators are found to help prioritize the investigation activities within the SOC.
- #16: The Endpoint data we’re using in this exercise comes from Microsoft System Monitor. a.k.a Sysmon. There are, of course, other sources of endpoint data, such as Bit9+Carbon Black, but sysmon is relatively easy to get into Splunk and doesn’t require 3rd party software.
- #17: Demo GDI and AFE Example and info here: https://splunk.box.com/s/zg6964cc15nj9kcldd9w
- #20: Cymru is pronounced (“cum-ree”)