![STACKING
sourcetype="stream:http"
| bin span=1d _time
| stats count as curr_count by _time
| appendcols [search index=botsv1 sourcetype="stream:http" | stats count as
total_count]
| eval avg_count = round(total_count/30,0)
| stats list(avg_count) as "Average Count", list(total_count) as "Total Count",
values(curr_count) as curr_count
Copyright © - ECS 2018](https://image.slidesharecdn.com/sigint-bigdataforthreatdetectionresponse-180329111252/85/Big-Data-For-Threat-Detection-Response-29-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Big Data For Threat Detection & Response (20)
Ad
More from Harry McLaren (20)
Ad
Recently uploaded (20)
Big Data For Threat Detection & Response
- 1. BIG DATA FOR THREAT DETECTION & RESPONSE Harry McLaren – Managing Consultant at ECS Sam Farmer – Security Operations Specialist
- 2. WHO AM I? HARRY MCLAREN •Alumnus of Edinburgh Napier (Now a Mentor) •Managing Security Consultant at ECS • Big Data Consultancy (Splunk) • Building SOC Technology (SIEM) Copyright © - ECS 2018
- 3. •Building/Running Security Operations Centres •Fastest Growing Practice in UK •Supports 80% of Top UK Banks •FTSE 100 Client Base Copyright © - ECS 2018
- 4. AGENDA • Introduction & Agenda • Security Operations Overview • Challenge: Monitoring, Detection & Hunting • Solution 1: Big Data, Splunk & Heterogeneous Data • Example: Example of Advanced Threat Activity • Solution 2: SIEM, Platform Evolution & Frameworks • Successful SIEM Deployments & Operation • Splunk User Group & Questions Copyright © - ECS 2018
- 5. Copyright © - ECS 2018
- 6. ADVANCED THREATS ARE HARD TO FIND • Human directed • Goal-oriented • Dynamic (adjust to changes) • Coordinated • Multiple tools & activities • New evasion techniques • Fusion of people, process, & technology • Contextual and behavioral • Rapid learning and response • Share info & collaborate • Analyze all data for relevance • Leverage IOC & Threat Intel Threat Attack Approach Security Approach Technology People Process Copyright © - ECS & Splunk 2018
- 7. ADVANCED THREATS ARE HARD TO FIND • Human directed • Goal-oriented • Dynamic (adjust to changes) • Coordinated • Multiple tools & activities • New evasion techniques Threat Attack Approach Security Approach Technology People Process Analytics-driven Security Connecting Data and People Risk-Based Context and Intelligence Copyright © - ECS & Splunk 2018
- 8. ADVANCED THREATS ARE HARD TO FIND ▶ Continuously Protect the business against: • Data Breaches • Malware • Fraud • IP Theft ▶ Comply with audit requirements ▶ Provide enterprise Visibility ▶ 70% to 90% improvement with detection and research of events ▶ 70% to 95% reduction in security incident investigation ▶ 10% to 30% reduction in risks associated with data breaches, fraud and IP theft ▶ 70% to 90% reduction in compliance labor Top Goals Top Splunk Benefits Copyright © - ECS & Splunk 2018
- 9. ADVANCED THREATS ARE HARD TO FIND Servers Storage DesktopsEmail Web Transaction Records Network Flows DHCP/ DNS Hypervisor Custom Apps Physical Access Badges Threat Intelligence Mobile CMDB Intrusion Detection Firewall Data Loss Prevention Anti-Malware Vulnerability Scans Traditional Authentication Copyright © - ECS & Splunk 2018
- 10. SOLUTION: SPLUNK, THE ENGINE FOR MACHINE DATA Custom Dashboards Report & Analyze Monitor & Alert Developer Platform Ad-hoc Search References – Coded fields, mappings, aliases Dynamic information – Stored in non-traditional formats Environmental context – Human maintained files, documents System/application – Available only using application request Intelligence/analytics – Indicators, anomaly, research, white/blacklist Real-Time Machine Data On-Premises Private Cloud Public Cloud Storage Online Shopping Cart Telecoms Desktops Security Web Services Networks Containers Web Clickstreams RFID Smartphones and Devices Servers Messaging GPS Location Packaged Applications Custom Applications Online Services DatabasesCall Detail Records Energy Meters Firewall Intrusion Prevention Copyright © - ECS & Splunk 2018
- 11. EXAMPLE OF ADVANCED THREAT ACTIVITIES .pdf executes & unpacks malware overwriting and running “allowed” programs Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Transaction Gain Access to System Create Additional Environment Conduct Business Svchost.exeCalc.exe Attacker hacks website. Steals .pdf files Web Portal Attacker creates malware, embed in .pdf Read email, open attachment Emails to the target EMAIL HTTP (web) session to command & control server Remote control, Steal data, Persist in company, Rent as botnet WEB Copyright © - ECS & Splunk 2018
- 12. EXAMPLE OF ADVANCED THREAT ACTIVITIES .pdf executes & unpacks malware overwriting and running “allowed” programs Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Transaction Gain Access to System Create Additional Environment Conduct Business Svchost.exeCalc.exe Attacker hacks website. Steals .pdf files Web Portal Attacker creates malware, embed in .pdf Read email, open attachment Emails to the target EMAIL HTTP (web) session to command & control server Remote control, Steal data, Persist in company, Rent as botnet WEB Intrusion Detection Credit card transmitted Endpoint Security Hacker tool found Windows Authentication Admin account used Copyright © - ECS & Splunk 2018
- 13. CONNECT THE “DATA-DOTS” TO SEE THE WHOLE STORY Persist, Repeat Attacker, know relay/C2 sites, infected sites, IOC, attack/campaign intent and attribution Where they went to, who talked to whom, attack transmitted, abnormal traffic, malware download What process is running (malicious, abnormal, etc.) Process owner, registry mods, attack/malware artifacts, patching level, attack susceptibility Access level, privileged users, likelihood of infection, where they might be in kill chain • Third-party Threat Intel • Open source blacklist • Internal threat intelligence • Firewall • IDS / IPS • Vulnerability scanners • Web Proxy • NetFlow • Network • Endpoint (AV/IPS/FW) • Malware detection • PCLM • DHCP • OS logs • Patching • Active Directory • LDAP • CMDB • Operating System • Database • VPN, AAA, SSO Delivery, Exploit Installation Gain Trusted Access Upgrade (escalate) Lateral Movement Data Gathering Exfiltration Persist, Repeat Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Copyright © - ECS & Splunk 2018
- 14. CONNECT THE “DATA-DOTS” TO SEE THE WHOLE STORY phishing Download from infected site 1 2 5 6 7 8 3 4 Threat Intelligence Data Host or ETDR Data Web or Firewall Data Threat Intelligence Data Identity Data Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Delivery Exploitation & Installation Command & Control Accomplish Mission EMAIL WEB EMAIL WEB Copyright © - ECS & Splunk 2018
- 15. Security Information & Event Management (SIEM) Software products and services combine security information management (SIM) and security event management (SEM). They provide real-time analysis of security alerts generated by network hardware and applications. Source: Wikipedia & Gartner Copyright © - ECS 2018
- 16. SIEM USE CASES Security & Compliance Reporting Real-time Monitoring of Known Threats Detecting Unknown Threats Fraud Detection Insider Threat Incident Investigations & Forensics Copyright © - ECS & Splunk 2018
- 17. SIEM EVOLUTION Term Initially Coined in 2005 by Gartner v1.0 Ticketing & Workflow Integrations v1.5 Risk Based Analysis & “Intelligence” v2.0 “Next-Gen SIEM”v3.0 Initial Rule Sets & Event Queues Environment Awareness & Correlation Searches Risk Management & Threat Data Intelligence Machine Learning & Orchestration Copyright © - ECS 2018
- 18. SO WHAT'S THE PROBLEM? Copyright © - ECS 2018
- 19. SIEM COMPONENT PARTS RULES Correlation Searches, Thresholds & Grouping CONTEXT Organisational Awareness & Impact Assessment FRAMEWORKS Scalable Functionality & User Empowerment INTEGRATION Data Compatibility, Extensibility & Workflow Management Copyright © - ECS 2018
- 20. Source: Splunk Developer PortalCopyright © - ECS & Splunk 2018
- 21. A B C D INTEGRATION Maximize cross-silo visibility by on-boarding ALL data sources. Automate repetitive tasks and setup orchestration for the rest. PREPARATION Understand your project’s input and output requirements. Champion the project and identify project dependencies. SUCCESS CRITERIA Identify the problem(s) you’re trying to solve. Document the risks/threats and the controls/mitigations. EMBEDDING Position SIEM project as part of transformative change. Enable and engage SecOps to own and evolve platform. SUCCESSFUL SIEM Copyright © - ECS 2018
- 22. QUESTIONS?
- 23. WHO AM I? SAM FARMER •Alumnus of Edinburgh Napier •Security Operations Specialist at ECS • Security Operations SME • Security Monitoring (SOC) • SIEM Implementation • Threat Hunter Copyright © - ECS 2018
- 25. DIAMOND MODEL
- 27. BASIC SEARCHING sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational |eval length=len(CommandLine) | where length>1000 | table host CommandLine length | sort - length Copyright © - ECS 2018
- 28. GROUPING sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational | bin span=10m _time | search (process=svchost.exe OR process=lsass.exe OR process=dns.exe OR process=explorer.exe) | stats earliest(_time) as earliest, latest(_time) as latest, values(process) as recon_process, dc(process) as processes, by host | where processes>2 | eval duration=(latest-earliest) Copyright © - ECS 2018
- 29. STACKING sourcetype="stream:http" | bin span=1d _time | stats count as curr_count by _time | appendcols [search index=botsv1 sourcetype="stream:http" | stats count as total_count] | eval avg_count = round(total_count/30,0) | stats list(avg_count) as "Average Count", list(total_count) as "Total Count", values(curr_count) as curr_count Copyright © - ECS 2018
- 30. STANDARD DEVIATION | bin span=3m _time | stats count as curr_count by _time | streamstats window=1 current=false avg(curr_count) as prev_count | eval growth=curr_count-prev_count | stats avg(curr_count) as average stdev(curr_count) as std_dev latest(curr_count) as latest_vol latest(_time) as lt count(eval(curr_count>150)) as qualifying count as tots | eval conf_int=average+(3.69*(std_dev/sqrt(tots))) | where ((latest_vol>150 AND qualifying=1 AND relative_time(now(), "-4m")<lt) OR (latest_vol>conf_int AND qualifying>=8)) | rename average as "Average" std_dev as "Standard Deviation" latest_vol as "Latest Volume" lt as "Latest Time" qualifying as Qualifying tots as Total conf_int as "Confidence Interval" | convert ctime("Latest Time") timeformat="%H:%M:%S %d/%m/%y" Copyright © - ECS 2018
- 31. SPLUNK USER GROUP - EDINBURGH • When: • TBA (Register for Invite) • Where: • Edinburgh Napier University, 10 Colinton Road, Edinburgh, EH10 5DT • Register: https://usergroups.splunk.com/group/spl unk-user-group-edinburgh.html Copyright © - ECS 2018
Editor's Notes
- #3: Short Bio: Harry McLaren is a Senior Consultant at ECS and is responsible for service delivery, technical leadership and people development in the rapidly growing Splunk consulting practice and is responsible for growing our team of talented Splunk Consultants. ECS, a specialist in enterprise IT services, has an award-winning IT security capability which is focused on Cybersecurity Operations Centres and IT security consulting. 1min
- #17: Few Security based use cases you have leverage big data platforms for, but how? 1mins
- #18: SIEM evolution and the (often fallacy) that is ‘next-gen’ SIEM. “Next-gen” shouldn’t even be a term as your security operational capability to grow organically and the tools should be able to keep up. How a platform which can grow as your security maturity and technical ability also grows (not limited to only “out-of-the-box features”). 2mins
- #19: Building full featured SIEMs is hard. Many try, many fail. Big data platforms only provide access to (hopefully) easy to search data. Most end up as very basic rule engines similar in function to a distributed IDS (NIDS or HIDS). 2mins
- #20: Rules Threshold Based Anomaly/Behaviour Based Boolean Based Context Asset & Identity Awareness Risk Profiling/Analytics Approved Types of Activity vs Not Frameworks Scalability (Volume, Complexity) User Empowerment (without being a platform expert) Expansion and development of custom use cases. Integration Data Source Compatibility (Schema vs Write one, read multiple ways). Workflow Integration & Centralised Investigation Orchestration 3mins
- #21: Example high-level architecture of a SIEM platform. Lots of components working together. Inputs, procedures and outputs are covered. Five frameworks mentioned covered in more detail. Not going to talk all the way through each one, purpose is to show the types of frameworks required and illustrate the contents of them. 2mins
- #22: Understand the reasons for the project, use cases, motivations and what constraints might apply. Prepare, prepare, prepare. Ensure you have scoped all required inputs, outputs and the level of dependencies between them. Integrate everything! Not just the data sources, but workflow, automation and orchestration. SIEM can be very powerful tools, however if the team which is going to own it/use it doesn’t know how, it’ll go to waste. SecOps teams should be a the forefront of exploring the data, hunting and defining their own use cases. 2mins
- #25: Image: https://www.techiexpert.com/difference-data-science-machine-learning/
- #26: Image: ThreatConnect https://www.threatconnect.com/blog/threatconnect-announces-context-enriched-intelligence/
- #27: Image: https://sqrrl.com/cyber-threat-hunting-1-intro/
- #32: Registration: https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html LinkedIn Group: https://www.linkedin.com/groups/12013212 1min