Hands-On Security Breakout Session- Disrupting the Kill Chain
2 likes708 views
The document summarizes a security investigation conducted using Splunk software. The investigation began by detecting threat intelligence related network activity from an employee's system. Further investigation across endpoint, email, web, and DNS data sources traced the activity back to a targeted phishing email containing a weaponized PDF file. The file exploited a vulnerable PDF reader and installed Zeus malware. The root cause was determined to be a brute force attack on the company's website that stole the weaponized file. The investigation disrupted the cyber kill chain from reconnaissance to actions on objectives.
![[CB20] Operation Chimera - APT Operation Targets Semiconductor Vendors by CK ...](https://cdn.slidesharecdn.com/ss_thumbnails/ckcheninndylinshang-dejiangcb20-210112134635-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (18)
Similar to Hands-On Security Breakout Session- Disrupting the Kill Chain (20)
Ad
More from Splunk (20)
Ad
Recently uploaded (20)
Hands-On Security Breakout Session- Disrupting the Kill Chain
- 1. Copyright © 2015 Splunk Inc. Hands-On Security Disrupting the Kill Chain using Splunk Salt Lake City, June 2015
- 2. Copyright © 2014 Splunk Inc. Name: Hilton Meeting Access Code: SPLUNK2015
- 3. 3 Safe Harbor Statement During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described orto includeany suchfeatureor functionalityina futurerelease.
- 4. 4 Agenda Splunk & Security – Unknown Threats – Connect the Dots across All Data Kill Chain* Disruption – Overview Exercise/Demo – Security Investigation Example
- 5. URL #1: https://54.234.52.9 URL #2: https://54.81.91.128 URL #3: https://54.161.169.143
- 6. 6 Want a hard copy? Link to walkthrough: https://splunk.box.com/slc-splunklive-security Servers: URL #1: https://54.234.52.9 URL #2: https://54.81.91.128 URL #3: https://54.161.169.143
- 7. 42
- 8. 42 The answer to life, the universe, and everything?
- 9. 42% Of customers buy Splunk with Security as primary use case.
- 10. RapidAscent in the Gartner MQ for SIEM 10 2012 2013+2011
- 11. Machine Data contains a definitive record of all Human <-> Machine & Machine <-> Machine Interaction Splunk is a very effective platform to collect, store, and analyze all of that data.
- 12. Servers Storage DesktopsEmail Web Transaction Records Network Flows DHCP/ DNS Hypervisor Custom Apps Physical Access Badges Threat Intelligence Mobile CMDB Intrusion Detection Firewall Data Loss Prevention Anti- Malware Vulnerability Scans Traditional Authentication Connect the Dots acrossAll Data 12
- 13. 13 Splunk software complements, replaces and goes beyond traditional SIEMs. Moving Past SIEM to Security Intelligence Small Data. Big Data. Huge Data. SECURITY & COMPLIANCE REPORTING REAL-TIME MONITORING OF KNOWN THREATS MONITORING OF UNKNOWN THREATS INCIDENT INVESTIGATIONS & FORENSICS FRAUD DETECTION INSIDER THREAT
- 14. Hands-OnSession: Kill Chain*Disruption 14 Your system is compromised and the adversary begins its work Exploitation The adversary works to understand your organization looking for opportunities Reconnaissance The attacker steals data, disrupts your operations or causes damage Act on Intent *mostly….
- 15. • How can the security analysts at Buttercup Games, Inc. discover that their systems have been compromised by way of a stolen document from their web portal? • They would want to discover and disrupt the kill chain: • Where did the adversary start? (Recon) • How did they get a foothold? (Exploitation) • What was their motive and what did they take? (Actions on Intent) Security InvestigationExample 15 bu tercup games Let’s get hands-on!
- 16. 16 1 Reconnaissance Weaponization Delivery Exploitation Installation Command & Control Act on Objectives Web Kill Chain Demo Data Source - Activity Email and Endpoint Endpoint Endpoint, DNS, Proxy Endpoint, DNS, Proxy A brute force attack takes place on the customer web site, access is gained, and a sensitive pdf file is downloaded and weaponized with malware. A convincing phishing email is crafted and sent to an internal target The pdf document is opened then exploits the vulnerable pdf reader app creating a dropper which installs the malware. Command/Control activity is highlighted by it’s association with Threat Intelligence Demo Story line Threat Intelligence Integration
- 17. 17 APT Transaction Flow Across Data Sources 1 http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web Data Sources .pdf .pdf executes & unpacks malware overwriting and running “allowed” programs Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment Our Investigation begins by detecting high risk communications through the proxy, at the endpoint, and even a DNS call.
- 19. 19 To begin our investigation, we will start with a quick search to familiarize ourselves with the data sources. In this demo environment, we have a variety of security relevant data including… Web DNS Proxy Firewall Endpoint Email
- 20. 20 Take a look at the endpoint data source. We are using the Microsoft Sysmon TA. We have endpoint visibility into all network communication and can map each connection back to a process. } We also have detailed info on each process and can map it back to the user and parent process.} Lets get our day started by looking using threat intel to prioritize our efforts and focus on communication with known high risk entities.
- 21. 21 We have multiple source IPs communicating to high risk entities identified by these 2 threat sources. We are seeing high risk communication from multiple data sources. We see multiple threat intel related events across multiple source types associated with the IP Address of Chris Gilbert. Let’s take closer look at the IP Address. We can now see the owner of the system (Chris Gilbert) and that it isn’t a PII or PCI related asset, so there are no immediate business implications that would require informing agencies or external customers within a certain timeframe. This dashboard is based on event data that contains a threat intel based indicator match( IP Address, domain, etc.). The data is further enriched with CMDB based Asset/identity information.
- 22. 22 We are now looking at only threat intel related activity for the IP Address associated with Chris Gilbert and see activity spanning endpoint, proxy, and DNS data sources. These trend lines tell a very interesting visual story. It appears that the asset makes a DNS query involving a threat intel related domain or IP Address. ScrollDown Scroll down the dashboard to examine these threat intel events associated with the IP Address. We then see threat intel related endpoint and proxy events occurring periodically and likely communicating with a known Zeus botnet based on the threat intel source (zeus_c2s).
- 23. 23 It’s worth mentioning that at this point you could create a ticket to have someone re-image the machine to prevent further damage as we continue our investigation within Splunk. Within the same dashboard, we have access to very high fidelity endpoint data that allows an analyst to continue the investigation in a very efficient manner. It is important to note that near real-time access to this type of endpoint data is not common within the traditional SOC. The initial goal of the investigation is to determine whether this communication is malicious or a potential false positive. Expand the endpoint event to continue the investigation. Proxy related threat intel matches are important for helping us to prioritize our efforts toward initiating an investigation. Further investigation into the endpoint is often very time consuming and often involves multiple internal hand-offs to other teams or needing to access additional systems. This encrypted proxy traffic is concerning because of the large amount of data (~1.5MB) being transferred which is common when data is being exfiltrated.
- 24. 24 Exfiltration of data is a serious concern and outbound communication to external entity that has a known threat intel indicator, especially when it is encrypted as in this case. Lets continue the investigation. Another clue. We also see that svchost.exe should be located in a Windows system directory but this is being run in the user space. Not good. We immediately see the outbound communication with 115.29.46.99 via https is associated with the svchost.exe process on the windows endpoint. The process id is 4768. There is a great deal more information from the endpoint as you scroll down such as the user ID that started the process and the associated CMDB enrichment information.
- 25. 25 We have a workflow action that will link us to a Process Explorer dashboard and populate it with the process id extracted from the event (4768).
- 26. 26 This is a standard Windows app, but not in its usual directory, telling us that the malware has again spoofed a common file name. We also can see that the parent process that created this suspicuous svchost.exe process is called calc.exe. This has brought us to the Process Explorer dashboard which lets us view Windows Sysmon endpoint data. Suspected Malware Lets continue the investigation by examining the parent process as this is almost certainly a genuine threat and we are now working toward a root cause. This is very consistent with Zeus behavior. The initial exploitation generally creates a downloader or dropper that will then download the Zeus malware. It seems like calc.exe may be that downloader/dropper. Suspected Downloader/Dropper This process calls itself “svchost.exe,” a common Windows process, but the path is not the normal path for svchost.exe. …which is a common trait of malware attempting to evade detection. We also see it making a DNS query (port 53) then communicating via port 443.
- 27. 27 The Parent Process of our suspected downloader/dropper is the legitimate PDF Reader program. This will likely turn out to be the vulnerable app that was exploited in this attack. Suspected Downloader/Dropper Suspected Vulnerable AppWe have very quickly moved from threat intel related network and endpoint activity to the likely exploitation of a vulnerable app. Click on the parent process to keep investigating.
- 28. 28 We can see that the PDF Reader process has no identified parent and is the root of the infection. ScrollDown Scroll down the dashboard to examine activity related to the PDF reader process.
- 29. 29 Chris opened 2nd_qtr_2014_report.pdf which was an attachment to an email! We have our root cause! Chris opened a weaponized .pdf file which contained the Zeus malware. It appears to have been delivered via email and we have access to our email logs as one of our important data sources. Lets copy the filename 2nd_qtr_2014_report.pdf and search a bit further to determine the scope of this compromise.
- 30. 30 Lets search though multiple data sources to quickly get a sense for who else may have have been exposed to this file. We will come back to the web activity that contains reference to the pdf file but lets first look at the email event to determine the scope of this apparent phishing attack.
- 31. 31 We have access to the email body and can see why this was such a convincing attack. The sender apparently had access to sensitive insider knowledge and hinted at quarterly results. There is our attachment. Hold On! That’s not our Domain Name! The spelling is close but it’s missing a “t”. The attacker likely registered a domain name that is very close to the company domain hoping Chris would not notice. This looks to be a very targeted spear phishing attack as it was sent to only one employee (Chris).
- 32. 32 Root Cause Recap 3 Data Sources .pdf executes & unpacks malware overwriting and running “allowed” programs http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web .pdf Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment We utilized threat intel to detect communication with known high risk indicators and kick off our investigation then worked backward through the kill chain toward a root cause. Key to this investigative process is the ability to associate network communications with endpoint process data. This high value and very relevant ability to work a malware related investigation through to root cause translates into a very streamlined investigative process compared to the legacy SIEM based approach.
- 33. 33 3 Lets revisit the search for additional information on the 2nd_qtr_2014- _report.pdf file. We understand that the file was delivered via email and opened at the endpoint. Why do we see a reference to the file in the access_combined (web server) logs? Select the access_combined sourcetype to investigate further.
- 34. 34 3 The results show 54.211.114.134 has accessed this file from the web portal of buttergames.com. There is also a known threat intel association with the source IP Address downloading (HTTP GET) the file.
- 35. 35 3 Select the IP Address, left-click, then select “New search”. We would like to understand what else this IP Address has accessed in the environment.
- 36. 36 3 That’s an abnormally large number of requests sourced from a single IP Address in a ~90 minute window. This looks like a scripted action given the constant high rate of requests over the below window. ScrollDown Scroll down the dashboard to examine other interesting fields to further investigate. Notice the Googlebot useragent string which is another attempt to avoid raising attention..
- 37. 37 3 The requests from 52.211.114.134 are dominated by requests to the login page (wp-login.php). It’s clearly not possible to attempt a login this many times in a short period of time – this is clearly a scripted brute force attack. After successfully gaining access to our website, the attacker downloaded the pdf file, weaponized it with the zeus malware, then delivered it to Chris Gilbert as a phishing email. The attacker is also accessing admin pages which may be an attempt to establish persistence via a backdoor into the web site.
- 38. 38 Kill Chain Analysis Across Data Sources 3 http (proxy) session to command & control server Remote control Steal data Persist in company Rent as botnet Proxy Conduct Business Create additional environment Gain Access to systemTransaction Threat Intelligence Endpoint Network Email, Proxy, DNS, and Web Data Sources .pdf .pdf executes & unpacks malware overwriting and running “allowed” programs Svchost.exe (malware) Calc.exe (dropper) Attacker hacks website Steals .pdf files Web Portal.pdf Attacker creates malware, embed in .pdf, emails to the target MAIL Read email, open attachment We continued the investigation by pivoting into the endpoint data source and used a workflow action to determine which process on the endpoint was responsible for the outbound communication. We Began by reviewing threat intel related events for a particular IP address and observed DNS, Proxy, and Endpoint events for a user in Sales. Investigation complete! Lets get this turned over to Incident Response team. We traced the svchost.exe Zeus malware back to it’s parent process ID which was the calc.exe downloader/dropper. Once our root cause analysis was complete, we shifted out focus into the web logs to determine that the sensitive pdf file was obtained via a brute force attack against the company website. We were able to see which file was opened by the vulnerable app and determined that the malicious file was delivered to the user via email. A quick search into the mail logs revealed the details behind the phishing attack and revealed that the scope of the compromise was limited to just the one user. We traced calc.exe back to the vulnerable application PDF Reader.
- 39. 39 www.splunk.com/apptitude July 20th, 2015 Submission deadline
- 40. The 6th Annual Splunk Worldwide Users’ Conference September 21-24, 2015 The MGM Grand Hotel, Las Vegas • 50+ Customer Speakers • 50+ Splunk Speakers • 35+ Apps in Splunk Apps Showcase • 65 Technology Partners • 4,000+ IT & Business Professionals • 2 Keynote Sessions • 3 days of technical content (150+ Sessions) • 3 days of Splunk University – Get Splunk Certified – Get CPE credits for CISSP, CAP, SSCP, etc. – Save thousands on Splunk education! 40 Register at: conf.splunk.com
- 41. 41 We Want to Hear your Feedback! After the Breakout Sessions conclude Text Splunk SLC to 878787 And be entered for a chance to win a $100 AMEX gift card!
- 42. Questions?
- 43. Thank You
Editor's Notes
- #8: Does anyone know what this number is?
- #9: Probably dating myself here. If you’ve ever read The Hitchhikers guide to the galaxy, you’d know this is the answer to life, the universe, and everything. Google will prove that to you if you type “the answer to life, the universe, and everything” in. What does that have to do with Splunk?
- #10: Our security biz is on fire - Between 35 and 42% of splunk customers buy splunk with security as the primary use case. Used to be that the security department would purchase splunk and keep it for themselves Now we see security spearheading Splunk and then driving the adoption across the org We’ve adjusted our field organization to complement this. Pre-Sales: SMEs on every sales team that focus on security. Specialists within every region that ONLY do security. Post-Sales: Security Practice that builds new content based on field experiences with customers. SPLICE as an example. Security Services to do SOC building, breach response, tabletop exercises, etc. Development and Partners: Entire development organization built around our Enterprise App for Security Partner program to get the very best integrations built with key security technologies.
- #11: Another interesting thing happened. I’ve been at Splunk now for two years. In that time I’ve seen our customers shift. First complement traditional SIEM – the ones listed up here: Q1, Arcsight. Nitro. But wait, 90%... Now, they’re actively sunsetting these and using pure Splunk in place of a traditional SIEM. The reasons are always the same: -Faster development -Better context -Easier onboarding -Staff can hunt instead of maintain
- #12: Splunk excels at creating a data fabric Machine data: Anything with a timestamp, regardless of incoming format. Throw it all in there! Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting. DETECTION NOT PREVENTION! ASSUME BREACH! So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the “weird.” So if you had a place to see “everything” that happened… ….what would that mean for your SOC and IR teams?
- #13: It’s not just about security data. We collect all of that data – sure. You have to. We don’t want you to throw out any good sources of data. In a lot of cases, Splunk is blind, and reliant on third parties to send it data That’s one of the reasons our partners out in the hall are so important In some cases our own software gathers a lot of native useful data – we will see that later In many cases – you’re sending data from existing sources into splunk There’s a lot of non traditional data though that is useful for security, and we’ll see much of it here in the demo Examples: -Analyze email senders, recievers, subject lines, and perhaps full email text to understand phishing activity. -Get badge reader data to correlate physical location of individual with login and file access behavior. -Correlate data against CMDB to provide risk scoring and other context
- #14: Splunk is used across the security spectrum these days Within security, 6 major areas. Each could have its own presentation in itself. Starting from the right: Insider threat. A significant number of breaches are attributed to insiders. But more importantly, an outsider often looks like an insider due to use of valid, often privileged, credentials. We want to capture the east-west trafffic – the behavior “inside the perimeter” and alert you to behavior that isn’t normal. Fraud. Because we can capture the details of every online interaction, we can again look for patterns that shouldn’t be there. There’s a few great published use cases here – but if you think about looking for unusual use of coupons, strange price changes, credits to subscribers, access to accounts from unusual IP addresses or devices, you can start to get your head around what we do for fraud. Unknown threats. These are the zero days – and here we are again looking for things that are not normal. Strange ports. Process or service names. Unusual login activity. Data transfer patterns. All of these can be IOCs. Known threats, in real time. Here’s where we’ll consume data from those traditional sources. IDS, endpoint, malware, firewalls, antivirus – you name it. Correlate it against threat intelligence, provide risk scoring, and alert you to the threat. Security and compliance reporting. Very simple – we collect all of the data relevant to just about any compliance standard you can think of, and we make it very easy to create reports to satisfy your external or internal auditors. Some mappings are more concrete than others – PCI, HIPAA, SANS 20. Some are more abstract but we can provide guidance. And finally, the one we’ll concentrate on in this session: SOC incident investigations and then escalation to incident response and forensics. Again – we store everything, we make it quickly searchable, we never throw anything out, and we provide instant context to the reviewer. In IR, speed is your best ally – for every minute you waste waiting for analytics in a real incident, more damage can be done…
- #15: The phrase, Cyber Kill Chain, was coined by researchers from Lockheed Martin. For a complete list of Phases please see: http://www.lockheedmartin.com/content/dam/lockheed/data/corporate/documents/LM-White-Paper-Intel-Driven-Defense.pdf This is how a common adversary – i.e. someone that wants to steal your data, or your intellectual property, or cause downtime, or embarrass your org, sees you and plans an attack. They could be cyber criminals, they could be nation states, they could be coming from the inside. Recon: Discover/identify target. Maybe not even any direct communications with the target. They might be googling things to see if you have web portals that they can attack. Look at public records or proposals to see who is working on what, or what new products you are going to offer, or see who you have recently hired. Basically looking for something to exploit… …which then happens via a multitude of ways. This is where the actual compromise takes place. Commonly, some artifact that is trusted is weaponized and delivered. This is also where social engineering may come in to steal credentials (100% of breaches involve stolen, valid credentials). Then act on intent – what do they want to do to you? Steal your data or IP or affect your operations or cause physical harm, like we saw with Stuxnet a few years ago with the centerfuges. The ones we are glossing over here are weaponization, delivery, installation, command and control. We will see those in the exercise.
- #16: Splunk is the only security analytics platform that allows analysts and incident investigators to leverage these disparate data sources to disrupt the adversary kill chain. This demo shows a real world investigation scenario for the Zeus attack. We begin the investigation by searching for events for new threat intelligence and investigate the infection and identify the complete adversary kill chain. This hands-on exercise shows a real world investigation scenario for the Zeus malware. Why zeus? Malware that reports into a botnet, been around since 2009, Disrupts services, acquires financial data, installs ransomware to lock up machines. Very effective, very elusive – every time we think we have a handle on it it comes back because it keeps morphing. Attempts to shut it down largely unsuccessful. Most important: the data that we will be working with is real data, from an actual Splunk customer. We have sanitized it of course and taken a small portion of it to make it more manageable. The goal here is NOT to show you how Splunk can help you combat Zeus, specifically. The goal here is to show you how Splunk integrates threat intelligence, and helps you jump from event data across the entire security stack, to get a full picture of what’s going on, and help you disrupt an adversary. Incident investigation can be intimidating! What do you click on first? How do you follow the path of an attacker? It’s helpful to work backwards through the cyber kill chain, because that will make us better defenders in the future. The earlier we can disrupt the kill chain, the better. This use of Splunk can be intimidating too! We’re going to get away from the pretty dashboards we saw in the morning session and delve deep into log files here. Security practictioners are in general a really smart group – but Splunk is a great tool to make smart people smarter….
- #17: Just to explain what we’re going to do here and map it to the kill chain phases…plus to give you a preview of the kinds of data we will see in the exercise…
- #19: It’s available on any modern Windows platform It’s tunable now – you can tell it what to capture which it reports in XML-style logs It’s free And there’s a free Splunk app to ingest the data. At RSA back in April, the guy that wrote Sysmon – SysInternals author Mark Russinovich, during his presentation, said that Sysmon should be enabled on every one of your windows endpoints, and you should collect the data it generates in an analysis tool like Splunk. Now – you might have other endpoint threat detection (and remediation) technologies in place on your endpoints – this could be Tripwire, Carbon Black, Ziften – those are all excellent, all partners of ours, and all have feature sets above what Sysmon can give you. But for free, it turns out, this is pretty useful…
- #21: Scroll down on sysmon and talk about field extraction!
- #22: This is a splunk dashboard – these are very easy to create from your existing searches. You can make them interactive, like this one is This one maps IP addresses found in our data to known bad IP addresses. Let’s talk about lookups – data does not have to live in Splunk to be useful to Splunk -static lookups -databases -hadoop -web services -dynamic lookups (like bing or virustotal) -STIX/TAXII CMDB – we want to gain context. All about speed. Threat Intel – we want to match artifacts in our data against IOCs from threat feeds. IP address Domain URL Now with STIX: just about anything. Registry values, filenames, hashes, process names, email addresses, etc Cymru is pronounced (“cum-ree”)
- #23: Same dash but now scoped just to Chris Gilbert’s IP.
- #26: Talk about workflow actions here
- #33: So we know what happened here – targeted malware was sent to Chris Gilbert in a weaponized PDF file and he opened it up. Question is – how did the attackers get a copy of the confidential quarterly report?
- #41: And finally, I would like to encourage all of you to attend our user conference in September. The energy level and passion that our customers bring to this event is simply electrifying. Combined with inspirational keynotes and 150+ breakout session across all areas of operational intelligence, It is simply the best forum to bring our Splunk community together, to learn about new and advanced Splunk offerings, and most of all to learn from one another.