Splunk for Security-Hands On

Editor's Notes

• #3: We don’t have a ton of time and ES is quite a feature-rich product. It would take many hours to go through everything the app can do. So we’ll spend only a few minutes on some intro slides, and then the great bulk of this session will be hands-on.
• #4: Now unfortunately, you do need a modern laptop with a modern browser to participate. You can probably get away with a Surface or something like that, but iPads, old browsers, and especially IBM PCjr’s will not work. (don’t laugh – I actually had one of those.)
• #5: Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though. Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
• #6: Splunk excels at creating a data fabric Machine data: Anything with a timestamp, regardless of incoming format. Throw it all in there! Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting. DETECTION NOT PREVENTION! ASSUME BREACH! So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the “weird.” So if you had a place to see “everything” that happened… ….what would that mean for your SOC and IR teams?
• #7: The Splunk platform consists of multiple products and deployment models to fit your needs. Splunk Enterprise – for on-premise deployment Splunk Cloud – Fully managed service with 100% SLA and all the capabilities of Splunk Enterprise…in the Cloud Splunk Light – log search and analytics for small IT environments Hunk – for analytics on data in Hadoop The products can pull in data from virtually any source to support multiple use cases. Splunk Apps extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types.
• #8: Our rapid ascent reflects the customer traction we have and value we deliver to customers – with thousands of security customers and 40% year-over-year growth, we are the fastest growing SIEM vendor in the market. 2011 was our first time in the MQ; In 2 short years we raced up to the top quadrant in the MQ.
• #10: 3.3. 3.0 was the first release done against Splunk 6 and that was a huge step forward – mainly because of the use of CIM and accelerated data models. Unlike other competitive solutions ES is constantly evolving – on average twice a year. Upgrades are pretty seamless. Where does content come from? All of the typical places but most importantly it comes from YOU. We take the best ideas that you give us, and we productize them and make them scalable and supportable. Splunk is more than a product – it is a wide open platform that inspires. None of this is lost in ES – splunk with ES is just as flexible and customizable. And it leverages technology in the core product like mapreduce and data models. You need ES to scale to the security intelligence needs of a huge enterprise? No problem. ES has its own dev team and roadmap, dedicated support individuals, a services practice schooled in it and other complementary infosec. Also lots of training is available.
• #12: This should look familiar to you. What we’re doing here is giving a starting point for any Security Analyst to understand at a high level what’s going on in the environment. A single pane of glass, if you will, for all security data.   Everything we are seeing here is customizable – the panels, the indicators, via standard Splunk functionality.   Most of the data on this dashboard is centered on Notable Events. Notable Events are a concept unique to Splunk with ES – there’s an entire Notable Event framework that allows us to perform simple or complex correlations, and then create events by analyzing disparate events from disparate sources.   Notable Events in ES are categorized into various high-level security domains: access, audit, identity, network, and threat. We’ll see those categories throughout the app.   You can see Splunk Sparklines here – these little green lines. These are great for detecting quick trends in the security events – a continuous line means something constant, which could be a heartbeat or a scripted attack. A spike could be a single attack or maybe just someone fat-fingering their password a few times.   We’ll drill into some of these incidents in a few minutes, but let’s continue on with our tour. How does all this data get into Splunk?
• #18: This should look familiar to you. What we’re doing here is giving a starting point for any Security Analyst to understand at a high level what’s going on in the environment. A single pane of glass, if you will, for all security data.   Everything we are seeing here is customizable – the panels, the indicators, via standard Splunk functionality.   Most of the data on this dashboard is centered on Notable Events. Notable Events are a concept unique to Splunk with ES – there’s an entire Notable Event framework that allows us to perform simple or complex correlations, and then create events by analyzing disparate events from disparate sources.   Notable Events in ES are categorized into various high-level security domains: access, audit, identity, network, and threat. We’ll see those categories throughout the app.   You can see Splunk Sparklines here – these little green lines. These are great for detecting quick trends in the security events – a continuous line means something constant, which could be a heartbeat or a scripted attack. A spike could be a single attack or maybe just someone fat-fingering their password a few times.   We’ll drill into some of these incidents in a few minutes, but let’s continue on with our tour. How does all this data get into Splunk?
• #23: So what does the data look like once it’s onboarded into Splunk in a CIM-compatible format?   Let’s look at one example in ES: Malware Center.
• #26: Let’s do a quick pivot to show what we can do with these fields. First we’ll load up the Malware Attacks data model and change the time to last 60 minutes. Then we’ll go to an area chart which by default shows us this time period stretched out on an X axis…
• #27: So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
• #28: So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
• #29: So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
• #30: Here we have a simple dashboard showing us all sorts of detail about recent malware activity in the environment. Like Security Posture, this is high level information, but more granular about a certain security domain (Malware, which is under Endpoint). We have these “centers” throughout ES for things like Access, Traffic, Intrusions, Updates, Vulnerabilities, and many other security-relevant areas, and you can investigate them later.   For now, let’s drill into two of the “top infections” to see CIM at work. Looking at this dashboard we can’t tell that we actually have at least two different endpoint protection systems feeding data into Splunk: Sophos, Trend Micro, and Symantec Endpoint Protection. Splunk normalizes the data on search time, according to CIM, to create this (and the other) dashboards.   Click on Mal/Packer, and you’ll see that this infection was detected by Sophos. The raw logs are literally a click away:  
• #31: The more data you have flowing into Splunk and into ES, the more useful it becomes. And ES is self auditing to tell you which data sources you are missing:
• #35: In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.   Let’s bring up the Risk Analysis page associated with Advanced Threat:
• #36: The main reason why this risk framework is important is that it gets you away from writing specific rules for specific threats or assets. You don’t need 1,000 correlation rules anymore – you simply can elevate risk scores on whatever object you want, based on the behavior you’re seeing in the environment. So the idea here is, a correlation rule fires, and then a risk modifier takes effect and changes the risk score based on cumulative scoring of whatever else has happened to that user, or system, or other object.   On the dashboard, we can define filters to find a particular system or user or timeframe.   Note the natural language descriptions (in the screenshot they are medium and low). We track how your overall risk scoring is doing over time, and constantly re-calculate the baseline. Got a lot of activity going on that isn’t “normal” for that timeframe and you might see things going from “increasing minimally” to “extremely increasing” – all based on what the historical norm is.   We can of course see which objects have the highest risk and which correlation rules are contributing the most to the highest risk.
• #42: In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.   Let’s bring up the Risk Analysis page associated with Advanced Threat:
• #43: On the dashboard we can see that we’re using the power of Splunk search to match artifacts in our incoming data against IoC’s we find in our threat feeds. Splunk de-duplicates the threat feeds so that if an artifact shows up in multiple feeds you don’t get duplicate notifications.   We can filter the display by threat_group, which is essentially the source of the IoCs. This could be something commercial like ThreatStream or ThreatConnect or Norse, something open-source like Sans or iblocklist, or something from your ISAC that is delivered over a TAXII feed in STIX format.   The threat collection shows that we can use various IoCs to match up against artifacts in our data – IP addresses, domain names, URLs, filenames, certificate common names and organizations, email addresses, registry keys – as long as it can be defined in your incoming feed or locally, you can use it as an IoC.   You can see the most active threat sources, and if you scroll down, you can see the most recent matches against your threat feeds.   How are these configured? Let’s go to the configuration, and see.
• #46: In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.   Let’s bring up the Risk Analysis page associated with Advanced Threat:
• #47: Rounding out the Threat Intelligence capabilities are the Threat Artifacts browser, which allows us to search through all of the artifacts stored in ES:
• #48: We don’t have time to go through each and every one of the advanced threat capabilities in the ES app. However, let’s just see that up here under Advanced Threat we have some very interesting capabilities: Some of the most useful ones are the Protocol Intelligence that leverages wire data from things like Splunk Stream, Netflow, and Bro. Also the Access Anomalies and User Activity, which are very useful to detect possible insider threat. And the New Domain Analysis, which analyzes traffic patterns and DNS queries to domains, and then tells you if you have devices communicating with recently registered garbage domains (that are often associated with DGA). Again – this is something you can go through on your own time.
• #57: We will see all of the details of the event, including our most recent comments and ownership activity.
• #58: So we know from the title of the event that we have a device on our network communicating out to a known bad IP address that’s a Zeus C2 address. But Splunk has enriched this event with some very useful info. We can see here that this particular machine is a laptop, and that it is owned by someone in Sales named Chris Gilbert. We see the IP addresses associated with the communication. We see the locations that this person Chris Gilbert works from. This correlation happens automatically against our ES Asset and Identity frameworks – we get the information an incident responder needs right up front.   Everything we see here is pivotable. We can go to places within ES, within Core Splunk and outside of Splunk too, and use that field as an argument. As an example, let’s drill into the arrow next to “Destination” and see what Domain Dossier has to say about this external IP address:  
• #59: We can see that this netblock is assigned to an organization in China.   While there are a lot of these “workflow actions” associated with Notable Events configured already in the product, you can feel free to create custom ones.   Next, let’s understand what else has been going on with this laptop.
• #60: One thing that we assume is that traffic from laptops outbound to C2 servers occurs via web proxy, at least when the laptops are on our corporate network. So we can look in our proxy logs to verify.
• #61: Note that we have only one source machine (Chris Gilbert’s laptop at 192.168.56.102) communicating with this known bad IP. That’s good at least – this doesn’t appear to be a widespread infection.   Some other interesting things about this data – notice a fairly large transaction in terms of bytes. Notice also that the connection is “tcp” over port 443 not “https” which would be considered normal.
• #62: Go back to the notable event and let’s look at Asset Investigator to get a more detailed view of this possibly-infected asset:
• #63: Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset: We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).  
• #64: Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset: We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).  
• #66: These are all Microsoft Sysmon events. Sysmon is a great, free utility from Microsoft that is lightweight and runs on all modern Windows variants. We’re simply collecting this data from Sysmon into Splunk, in real time, from our workstations. It gives us granular process data that includes parent/child relationships, hash data, and network connections, among other things.  
• #67: Note that the second event is that strange calc.exe event. Let’s click the small arrow to the far left of the event:
• #68: Note that the second event is that strange calc.exe event. Let’s click the small arrow to the far left of the event:
• #75: Finally, let’s see some of the auditing that ES does of the activity carried out against Notable Events.  
• #76: The recent activity that you have carried out should appear in the panels. Clicking on a reviewer’s name will bring you detail about that reviewer’s activity.
• #83: This is a search that’s been created that returns any IP address where we see, over the timeframe selected, a lot of login attempts (greater than 10) and then loading of the admin pages of the portal from that same IP. If any IP address returns from the search, we can consider this an alertable event.
• #84: Note that you could turn this into a simple Splunk alert by just doing a “save as” alert and running it regularly. But we want to see how to turn this into a Notable Event in ES.   Using your mouse, select the entire text of the search and copy it to the clipboard.
• #95: So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
• #96: The main reason why this risk framework is important is that it gets you away from writing specific rules for specific threats or assets. You don’t need 1,000 correlation rules anymore – you simply can elevate risk scores on whatever object you want, based on the behavior you’re seeing in the environment. So the idea here is, a correlation rule fires, and then a risk modifier takes effect and changes the risk score based on cumulative scoring of whatever else has happened to that user, or system, or other object.   On the dashboard, we can define filters to find a particular system or user or timeframe.   Note the natural language descriptions (in the screenshot they are medium and low). We track how your overall risk scoring is doing over time, and constantly re-calculate the baseline. Got a lot of activity going on that isn’t “normal” for that timeframe and you might see things going from “increasing minimally” to “extremely increasing” – all based on what the historical norm is.   We can of course see which objects have the highest risk and which correlation rules are contributing the most to the highest risk.
• #97: On the dashboard we can see that we’re using the power of Splunk search to match artifacts in our incoming data against IoC’s we find in our threat feeds. Splunk de-duplicates the threat feeds so that if an artifact shows up in multiple feeds you don’t get duplicate notifications.   We can filter the display by threat_group, which is essentially the source of the IoCs. This could be something commercial like ThreatStream or ThreatConnect or Norse, something open-source like Sans or iblocklist, or something from your ISAC that is delivered over a TAXII feed in STIX format.   The threat collection shows that we can use various IoCs to match up against artifacts in our data – IP addresses, domain names, URLs, filenames, certificate common names and organizations, email addresses, registry keys – as long as it can be defined in your incoming feed or locally, you can use it as an IoC.   You can see the most active threat sources, and if you scroll down, you can see the most recent matches against your threat feeds.   How are these configured? Let’s go to the configuration, and see.
• #100: Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset: We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).  
• #101: The recent activity that you have carried out should appear in the panels. Clicking on a reviewer’s name will bring you detail about that reviewer’s activity.