Scaling Security Threat Detection with Apache Spark and Databricks
2 likes699 views
Apple must detect a wide variety of security threats, and rises to the challenge using Apache Spark across a diverse pool of telemetry. This talk covers some of the home-grown solutions we’ve built to address complications of scale
![Detection and Alert Abstraction
▪ Logic is described
in form of Spark
DataFrame
▪ Supports additional
post-processing
transformation
▪ Basic interface for
consumption by
other code
Detection
val alerts: Map[String, Alert] =
Alert
val modules: ArrayBuffer[Transformer] =
def PostProcessor(input: DataFrame): DataFrame = ???
def df: DataFrame = /* alert logic here */
val config: DetectionConfig
Input and other runtime configs
Test generation](https://image.slidesharecdn.com/367joshgillner-200627154217/85/Scaling-Security-Threat-Detection-with-Apache-Spark-and-Databricks-15-320.jpg)
![Automated Suggestions
alert_hash 112233445
serial = C12345678ABC
src_ip = 88.88.88.123
mime_type = [“text/html"]
dt = 2020-03-15
C12345678ABC
88.88.88.123
Entities
C12345678ABC
88.88.88.123
00:de:ad:00:be:ef
01:8b:ad:00:f0:0d
joshsaccount
joshshostname
Enriched Entities
{
Emails
Tickets
Alerts
Notebooks
Detections
Wikis
Document Search
Suggestion](https://image.slidesharecdn.com/367joshgillner-200627154217/85/Scaling-Security-Threat-Detection-with-Apache-Spark-and-Databricks-40-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Scaling Security Threat Detection with Apache Spark and Databricks (20)
![[CB16] COFI break – Breaking exploits with Processor trace and Practical cont...](https://cdn.slidesharecdn.com/ss_thumbnails/cb16shinaen-161108081131-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Databricks (20)
Ad
Recently uploaded (20)
Scaling Security Threat Detection with Apache Spark and Databricks
- 2. Scaling Security Threat Detection with Spark and Databricks Josh Gillner Apple Detection Engineering
- 3. ▪ Protecting Apple’s Systems ▪ Finding & responding to security threats using log data ▪ Threat research and hunting ^^^ Looking for this guy Who are we? - Apple Detection Engineering
- 5. What is a detection?
- 6. Detection === Code That Finds Bad Stuff ▪ Get an input dataset(s) ▪ Apply logic ▪ Output ▪ 1 notebook -> 1 job -> 1 detection
- 7. What Happens Next? Analyst Review Suggestion, Enrichment & Automated Containment Alert Orchestration System Contain Issue+ This needs to be fast
- 9. Problem #1 — Development Overhead ▪ Average time to write, test, and deploy a basic detection === 1 week ▪ New ideas/week > deployed jobs/week (unsustainable) ▪ Writing scalatests, preserving test samples…testing is too cumbersome ▪ > 60% of new code is boilerplate (!!)
- 10. Problem #2 — Mo’ Detections, Mo’ Problems Want to add a cool new feature to all detections? Refactor many different notebooks Config all over the place in disparate notebooks Want to configure multiple detections at once? Ongoing tuning and maintenance? One-off tuning doesn’t scale to hundreds of detections
- 11. Problem #3 — No Support for Common Patterns ▪ Common enrichments or exclusions ▪ Creating and using statistical baselines ▪ Write detection test using scalatest Things People Often Do (but must write code for) …everyone implements in a different way …fixes/updates must be applied in 10 places
- 12. DetectionKit Auto-Tuning Alerts Modular Postprocessing Automated Enrichments Complex Exclusions Notebook-Based CI Centralized Configuration Test Generation Alert Standardization Future-Looking Abstraction Multi-Stage Alerting Modular Investigation Templates Signal-Based Detections Rate Limit Failsafes Preprocessing Transformations Automated Tagging Statistics Tables Entity-Based Deduplication Asset Attribution
- 13. Components ▪ Input ▪ Detection and Alert abstractions ▪ Emitters ▪ Configuration ▪ Tuning ▪ Modular Pre/Postprocessing ▪ Functional Testing ▪ Complex Exclusions ▪ Templatized Investigations
- 14. Input ▪ All detection begins with input loading ▪ Pass in inputs through config object ▪ External control through config ▪ decide spark.read vs .readStream ▪ path, schema, format ▪ no hardcoding -> dynamic input behavior ▪ Abstracts away details of getting data ^^^ This should not change if someDataset is a production table or test sample file
- 15. Detection and Alert Abstraction ▪ Logic is described in form of Spark DataFrame ▪ Supports additional post-processing transformation ▪ Basic interface for consumption by other code Detection val alerts: Map[String, Alert] = Alert val modules: ArrayBuffer[Transformer] = def PostProcessor(input: DataFrame): DataFrame = ??? def df: DataFrame = /* alert logic here */ val config: DetectionConfig Input and other runtime configs Test generation
- 16. Emitter ▪ Takes output from Alert and send them elsewhere ▪ Also schedules the job in Spark cluster Alert MemoryEmitter FileEmitter KinesisEmitter DBFS on AWS S3 In-memory Table AWS Kinesis
- 17. Config Inference ▪ If things can (and should) be changed, move it outside of code ▪ eg. detection name, description, input dataset, emitter ▪ Where possible, supply a sane default or infer them val checkpointLocation: String = "dbfs:/mnt/defaultbucket/chk/detection/ / / .chk/" name = "CodeRed: Something Has Happened" alertName = "JoshsCoolDetection" version = "1" DetectionConfigInfer
- 18. Config Inheritance ▪ Fine-grained configurability ▪ Could be multiple Alerts in same Detection ▪ Individually configurable, otherwise inherit parent config Detection Alert val config: DetectionConfig Alert Alert
- 19. Modular Pre/PostProcessing ▪ DataFrame -> DataFrame transform applied to input dataset ▪ Supplied in config ▪ Useful for things like date filtering without changing detection Preprocessing Postprocessing ▪ Mutable Seq of transform functions inside Detection ▪ Applied sequentially to output foreachBatch Transformers ▪ Some operations not stream-safe ▪ Where the crazy stuff happens
- 20. Manual Tuning Lifecycle ▪ Tuning overhead scales with number of detections ▪ Feedback loop can take days while analysts suffer :( ▪ This need to be faster… ideally automated and self- service The data/ environment changes DE tweaks detection False positive alerts Analyst requests tuning pain
- 21. Self-Tuning Alerts Detection Analyst Review Alert Labels (FP, TP, etc.) Analyst Consensus! Modify Behavior Alert Orchestration System
- 22. Complex Exclusions ▪ Arbitrary SQL expressions applied on all results in forEachBatch ▪ Stored in rev-controlled TSV ▪ Integrated into Detection Test CI…malformed or over-selective items will fail tests ▪ Preservation of excluded alerts in a separate table Eventually, detections look like this >>> So….
- 23. Repetitive Investigations…What Happens? • Analysts run queries in notebooks to investigate • Most of these queries look the same, just different filter Analyst Review Alert Orchestration System
- 24. Automated Investigation Templates ▪ Find corresponding template notebook ▪ Fill it out ▪ Attach to cluster ▪ Execute Alert Orchestration System Workspace API
- 25. This lets us automate useful things like… Interactive Process Trees in D3 Baselines of Typical Activity
- 26. Automated Containment Machines can find, investigate, and contain issues without humans Automated Investigation Alert Orchestration System ODBC API • Run substantiating queries via ODBC • Render verdict Contain Issue
- 27. Detection Testing Why is it so painful? ▪ Preserving/exporting JSON samples ▪ Local SparkSession isn’t a real cluster ▪ Development happens in notebooks, testing happens in IDE ▪ Brittle to even small changes to schema, etc
- 28. Detection Functional Tests ▪ 85% reduction in test LoC ▪ write and run tests in notebooks! ▪ use Delta sample files in dbfs, no more exporting JSON ▪ scalatest generation using config and convention Trait: DetectionTest ^^ this is a complete test ^^
- 29. Detection Test CI Git PR CI System Test Notebooks Workspace API /Alerts/Test/PRs/<Git PR number>_<Git commit hash> Jobs API Build Scripts pass/fail “Testing has never been this fun!!” — detection engineers, probably
- 30. Jobs CI — Why? ▪ Managing hundreds of jobs in Databricks UI ▪ Each job has associated notebook, config, dbfs files ▪ No inventory of which jobs should be running, where ▪ We need job linting >>>
- 32. Deploy/Reconfigure Jobs with Single PR CI System Config Linter Stacks CLI Jobs Helper Deploy Job/ Notebooks/Files Kickstart/Restart Set Permissions
- 33. Cool Things with Jobs CI! ▪ Deploy or reconfigure many jobs concurrently ▪ Auto job restarts on notebook/ config change ▪ Standardization of retries, timeout, permissions ▪ Automate alarm creation for new jobs ^^^ No one likes manually crafting Stacks JSON — so we generate it
- 34. Saving Time with Automated Historical Insight
- 35. Problem #1 — Cyclical Investigations ▪ Alert comes in, analysts spend hours looking into it ▪ But the same thing happened 3 months ago and was determined to be benign ▪ Lots of wasted cycles on duplicative investigations
- 36. Problem #2 — Disparate Context ▪ Want to find historical incident data? ▪ look in many different places ▪ many search UIs, syntaxes ▪ Manual, slow & painful ▪ New analysts won’t have historical knowledge
- 37. Problem #3 — Finding Patterns Which incidents relate to other incidents? Do we see common infrastructure, actors? How much work is repeated? Case #55557 Case #44447 Case #33337 }(Some IP Address)
- 38. Solution: Document Recommendations ▪ Collect all incident-related tickets, correspondence, and investigations ▪ Normalize them into a Delta table ▪ Automate suggestion of related knowledge using our own corpus of documents Emails Tickets Alerts Notebooks Detection Code Wikis
- 39. “Has This Happened Before?” -> Automated Includes analyst comments and verdicts displayHTML suggestions, clickable links to original document
- 40. Automated Suggestions alert_hash 112233445 serial = C12345678ABC src_ip = 88.88.88.123 mime_type = [“text/html"] dt = 2020-03-15 C12345678ABC 88.88.88.123 Entities C12345678ABC 88.88.88.123 00:de:ad:00:be:ef 01:8b:ad:00:f0:0d joshsaccount joshshostname Enriched Entities { Emails Tickets Alerts Notebooks Detections Wikis Document Search Suggestion
- 41. Anatomy of an Alert These are not valuable for search! (too common) These are good indicators of document relevance
- 42. Entity Tokenization and Enrichment IP Address Regex Domain Hashes Accounts Serials UDIDs File Path Emails MAC Addresses Alert Payload VPN Sessions Enrichments DHCP Sessions Asset Data Account Data
- 43. Suggestion Algorithm ▪ Gather match statistics for each entity: ▪ historical rarity ▪ document count rarity ▪ doc type distribution ▪ Compute entity weight based on average ranked percentiles of those features ▪ More common terms == less valuable ▪ Return the best n hits by confidence ▪ Not That Expensive™
- 44. Q & A ?
- 45. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.