From Python Scikit-learn to Scala Apache Spark—The Road to Uncovering Botnets with Avi Aminov

Jun 14, 20174 likes1,236 views

The landscape of security threats an enterprise faces is vast. It is imperative for an organization to know when one of the machines within the network has been compromised. One layer of detection can take advantage of the DNS requests made by machines within the network. A request to a Command & Control (CNC) domain can act as an indication of compromise. It is thus advisable to find these domains before they come into play. The team at Akamai aims to do just that. In this session, Aminov will share Akamai’s experience in porting their PoC detection algorithms, written in Python, to a reliable production-level implementation using Scala and Apache Spark. He will specifically cover their experience regarding an algorithm they developed to detect botnet domains based on passive DNS data. The session will also include some useful insights Akamai has learned while handing out solutions from research to development, including the transition from small-scale to large-scale data consumption, model export/import using PMML and sampling techniques. This information is valuable for researchers and developers alike.

The Road to Uncovering Botnets
From Python Scikit-Learn
to Scala Spark

whoami
• Avi Aminov
– ~2 years Security Researcher at Akamai
– Physics PhD student
• Asaf Nadler
– ~1.5 years Security Researcher at Akamai
– CS PhD student

Enterprise Threat Protection
• Detect malware presence from outbound traffic
– Behavioral pattern analysis
– Domain blacklisting
• Availability – End of June ’17
Akamai
Recursive
DNS
Branch / HQ
Enterprise
DNS

Data
• Akamai Data
– 20-30% of internet traffic
– Customer ISP/Enterprise logs – 20B DNS queries/day
• Third party data
– e.g. Authoritative DNS log lines
• Open data sources
– e.g. WHOIS information

Bot Networks – IP Fluxing
• Goal – Evasion
– Regular bots: waiting for orders
– Proxies: concealing origin server
Command
& Control
server
Bots
Proxy Bots

Bot Networks Detection
• Detect illegitimate IP fluxing
• Features
– IP dispersity (Geo, systems)
– TTL features
– Lexical
Domain Description #Systems #Countries
astro-travels.net PoS CNC Host 157 11

Decision Tree Model
Malicious with high confidence
• Spread across systems
• Unpopular
Benign with high confidence
• IPs in the same system
• Contains meaningful words

Challenge – Going to Production
Feature
Extraction
Scoring Blacklist
Feature
Extraction
Model
Training Model
Model
Evaluation
Data
Sources

What have we done so far?
• Flow
– Researcher describes an algorithm (document + Hive query)
– Dev rewrites the code in MapReduce (now Scala/Spark)
• Problems
– Not applicable to ML pipelines
– Prone to mistakes
– Longer development cycle

Can We Do Better? Option #1
• Research side – Pipeline in Scala/Spark
• Dev side – Implement the algorithms
• Pros
– Greater flexibility
– Research scale
• Cons
– Learning curve
– Lose sklearn/R benefits

Can We Do Better? Option #2
• Research side – Train locally and export model
• Dev side – Transform data using imported model
• Pros
– Quick implementation
– Unified procedure
• Cons
– No support for all models

Export scheme
• Predictive Model Markup Language
• General scheme for ML pipelines
– Data transformations
– Scoring models
• XML format – Readable
• Supported by major data science / ML
frameworks using jPMML (R, sklearn)

PMML Simple Boilerplate
Python (Research side) Scala (Dev side)
Credit: jpmml lib http://openscoring.io/ , https://github.com/jpmml/
Maintained by Villu Ruusmann

Lessons Learned
• Work process adjusted to the task
– Training locally? Export the model
– Training on larger scales? Better to use Spark
• Use jpmml for model export
• When applicable, reduce workload in production
– Example – only look at domains with many IPs

Challenge solved
Feature
Extraction
Scoring Blacklist
Data
Collection
Model
Training Model
Model
Evaluation
Data
Sources PMML

This document provides an overview of Spark: Data Science as a Service by Sridhar Alla and Kiran Muglurmath of Comcast. It discusses Comcast's data science challenges due to massive data size and lack of scalable architecture. It introduces Roadrunner, Comcast's solution built on Spark, which provides a centralized processing system with SQL and machine learning capabilities to enable data ingestion, quality checks, feature engineering, modeling and workflow management. Roadrunner is accessed through REST APIs and helps multiple teams work with the same large datasets. Examples of transformations, joins, aggregations and anomaly detection algorithms demonstrated in Roadrunner are also included.

Apache Kylin: Speed Up Cubing with Apache Spark with Luke Han and Shaofeng ShiDatabricks

This document discusses speeding up OLAP cube building in Apache Kylin using Spark. Cubing with MapReduce can be slow due to serialization overhead and repeated job submissions. Spark allows caching data in memory across cuboid layers in one job, significantly reducing build times compared to MapReduce as shown in a benchmark on a 160 million row dataset. Spark simplifies Kylin development and brings capabilities for real-time OLAP and cloud integration.

SSR: Structured Streaming for R and Machine Learningfelixcss

Stepping beyond ETL in batches, large enterprises are looking at ways to generate more up-to-date insights. As we step into the age of Continuous Application, this session will explore the ever more popular Structure Streaming API in Apache Spark, its application to R, and building examples of machine learning use cases. Starting with an introduction to the high-level concepts, the session will dive into the core of the execution plan internals and examine how SparkR extends the existing system to add the streaming capability. Learn how to build various data science applications on data streams integrating with R packages to leverage the rich R ecosystem of 10k+ packages. Session hashtag: #SFdev2

Spark r under the hood with Hossein FalakiDatabricks

SparkR is a new and evolving interface to Apache Spark. It offers a wide range of APIs and capabilities to Data Scientists and Statisticians. Being a distributed system with a JVM core some R users find SparkR errors unfamiliar. In this talk we will show what goes on under the hood when you interact with SparkR. We will look at SparkR architecture, performance bottlenecks and API semantics. Equipped with those, we will show how some common errors can be eliminated. I will use debugging examples based on our experience with real SparkR use cases.

Huawei Advanced Data Science With Spark StreamingJen Aman

This document discusses streamDM, an open source machine learning library for stream mining in Spark Streaming. It summarizes streamDM's capabilities for incremental learning on data streams using algorithms like SGD, Naive Bayes, clustering and decision trees. Examples of using streamDM in Huawei's network alarm analysis and fault localization systems are provided, demonstrating improvements in efficiency, accuracy and ability to handle large volumes of streaming data. The document encourages researchers to apply for Huawei's Innovation Research Program grants to further collaborative work on stream mining algorithms and applications.

Spark Summit EU talk by Kent Buenaventura and Willaim LauSpark Summit

This document summarizes Unity Technologies' journey migrating their data pipeline from a legacy Hive-based system to using Spark. Some key points: - They moved to Spark for its scaling, performance, and ability to handle both batch and streaming workloads from a single stack. - The new Spark-based pipeline uses Airflow for workflow management and saves processed data to Parquet files stored in S3 for backup. - Taking a test-driven development approach with unit and integration tests helped ensure a smooth migration. Staging the pipeline in an environment similar to production also helped address issues early. - The new Spark pipeline completed analysis stages up to 2x faster than the previous Hive-based system and

Machine Learning as a Service: Apache Spark MLlib Enrichment and Web-Based Co...Databricks

Boosted by Apache Spark’s data processing engine, machine learning as a service (MLaaS) is now faster and more powerful. However, Spark MLlib is developing and is limited by data preprocessing algorithms. In this session, learn how Suning R&D’s MLaaS platform abstracted, standardized and implemented a very rich machine learning pipeline on top of Spark, from data pre-processing, supervised and unsupervised modeling, performance evaluation, to model deployment. Their feature Spark extensions are: 1) a rich function set of data pre-processing, such as missing data treatment, many types of sampling, outlier detecting, advanced binning, etc.; (2) time series analysis/modeling algorithms; (3) domain-specific library for finance, such as cost sensitive decision tree for fraud detection; (4) a user-friendly drag-and-play codeless modeling canvas.

Apache Spark MLlib's Past Trajectory and New Directions with Joseph BradleyDatabricks

- MLlib has rapidly developed over the past 5 years, growing from a few algorithms to over 50 algorithms and featurizers for classification, regression, clustering, recommendation, and more. - This growth has shifted from just adding algorithms to improving algorithms, infrastructure, and integrating ML workflows with Spark's broader capabilities like SQL, DataFrames, and streaming. - Going forward, areas of focus include continued scalability improvements, enhancing core algorithms, extensible APIs, and making MLlib a more comprehensive standard library.

Spark Summit EU talk by Jakub HavaSpark Summit

This document summarizes Sparkling Water 2.0, which is a new version of the Sparkling Water platform that integrates the H2O machine learning library with Apache Spark. Some key features of Sparkling Water 2.0 include the ability to use H2O data structures and algorithms within the Spark API, machine learning pipelines that allow embedding H2O algorithms within Spark ML pipelines, and high availability support to make the H2O cluster resilient to Spark executor failures. The document outlines how Sparkling Water can be used for tasks like data munging, model building, streaming data processing, and provides code examples.

A Journey into Databricks' Pipelines: Journey and Lessons LearnedDatabricks

With components like Spark SQL, MLlib, and Streaming, Spark is a unified engine for building data applications. In this talk, we will take a look at how we use Spark on our own Databricks platform throughout our data pipeline for use cases such as ETL, data warehousing, and real time analysis. We will demonstrate how these applications empower engineering and data analytics. We will also share some lessons learned from building our data pipeline around security and operations. This talk will include examples on how to use Structured Streaming (a.k.a Streaming DataFrames) for online analysis, SparkR for offline analysis, and how we connect multiple sources to achieve a Just-In-Time Data Warehouse.

Dynamic DDL: Adding Structure to Streaming Data on the Fly with David Winters...Databricks

At the end of day, the only thing that data scientists want is tabular data for their analysis. They do not want to spend hours or days preparing data. How does a data engineer handle the massive amount of data that is being streamed at them from IoT devices and apps, and at the same time add structure to it so that data scientists can focus on finding insights and not preparing data? By the way, you need to do this within minutes (sometimes seconds). Oh… and there are a lot of other data sources that you need to ingest, and the current providers of data are changing their structure. GoPro has massive amounts of heterogeneous data being streamed from their consumer devices and applications, and they have developed the concept of “dynamic DDL” to structure their streamed data on the fly using Spark Streaming, Kafka, HBase, Hive and S3. The idea is simple: Add structure (schema) to the data as soon as possible; allow the providers of the data to dictate the structure; and automatically create event-based and state-based tables (DDL) for all data sources to allow data scientists to access the data via their lingua franca, SQL, within minutes.

Random Walks on Large Scale Graphs with Apache Spark with Min ShenDatabricks

Random Walks on graphs is a useful technique in machine learning, with applications in personalized PageRank, representational learning and others. This session will describe a novel algorithm for enumerating walks on large-scale graphs that benefits from the several unique abilities of Apache Spark. The algorithm generates a recursive branching DAG of stages that separates out the “closed” and “open” walks. Spark’s shuffle file management system is ingeniously used to accumulate the walks while the computation is progressing. In-memory caching over multi-core executors enables moving the walks several “steps” forward before shuffling to the next stage. See performance benchmarks, and hear about LinkedIn’s experience with Spark in production clusters. The session will conclude with an observation of how Spark’s unique and powerful construct opens new models of computation, not possible with state-of-the-art, for developing high-performant and scalable algorithms in data science and machine learning.

Performance Optimization Case Study: Shattering Hadoop's Sort Record with Spa...Databricks

Spark Summit EU talk by Shay Nativ and Dvir VolkSpark Summit

This document discusses accelerating Spark ML models with Redis modules. It provides an overview of Redis and Spark, and describes how Redis modules can add new capabilities like secondary indexes, time series, and machine learning. The document demonstrates a Redis ML module that implements random forests and decision trees. It shows how Spark ML models can be trained, saved to Redis for low-latency serving, and evaluated directly in Redis for improved performance over Spark alone.

Use of Spark MLib for Predicting the Offlining of Digital Media-(Christopher ...Spark Summit

This document discusses using Spark MLlib to predict which digital media files should be offlined from storage to free up space. It describes using k-means clustering, naive Bayes classification, and support vector machines (SVM) on features like file size, age, and airing schedule. SVM performed best and allowed building a predictive system in under an hour. The system is run twice daily on a Spark cluster to select files for purging from a large storage system based on predictions. Some initial issues were addressed and the system is now running robustly in production.

Experiences Migrating Hive Workload to SparkSQL with Jie Xiong and Zhan ZhangDatabricks

At Facebook, millions of Hive queries are executed on a daily basis, and the workload contributes to important analytics that drive product decisions and insights. Spark SQL in Apache Spark provides much of the same functionality as Hive query language (HQL) more efficiently, and Facebook is building a framework to migrate existing production Hive workload to Spark SQL with minimal user intervention. Before Facebook began large-scale migration to SparkSQL, they worked on identifying the gap between HQL and SparkSQL. They built an offline syntax analysis tool that parses, analyzes, optimizes and generates physical plans on daily HQL workload. In this session, they’ll share their results. After finding their syntactic analysis encouraging, they built tooling for offline semantic analysis where they run HQL queries in their Spark shadow cluster and validate the outputs. Output validation is necessary since the runtime behavior in Spark SQL may be different from HQL. They have built a migration framework that supports HQL in both Hive and Spark execution engines, can shadow and validate HQL workloads in Spark, and makes it easy for users to convert their workloads.

Resource-Efficient Deep Learning Model Selection on Apache SparkDatabricks

Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan ZhuDatabricks

XGBoost (https://github.com/dmlc/xgboost) is a library designed and optimized for tree boosting. XGBoost attracts users from a broad range of organizations in both industry and academia, and more than half of the winning solutions in machine learning challenges hosted at Kaggle adopt XGBoost. While being one of the most popular machine learning systems, XGBoost is only one of the components in a complete data analytic pipeline. The data ETL/exploration/serving functionalities are built up on top of more general data processing frameworks, like Apache Spark. As a result, users have to build a communication channel between Apache Spark and XGBoost (usually through HDFS) and face the difficulties/inconveniences in data navigating and application development/deployment. We (Distributed (Deep) Machine Learning Community) develop XGBoost4J-Spark (https://github.com/dmlc/xgboost/tree/master/jvm-packages), which seamlessly integrates Apache Spark and XGBoost. The communication channel between Spark and XGBoost is established based on RDDs/DataFrame/Datasets, all of which are standard data interfaces in Spark. Additionally, XGBoost can be embedded into Spark MLLib pipeline and tuned through the tools provided by MLLib. In this talk, I will cover the motivation/history/design philosophy/implementation details as well as the use cases of XGBoost4J-Spark. I expect that this talk will share the insights on building a heterogeneous data analytic pipeline based on Spark and other data intelligence frameworks and bring more discussions on this topic.

Building, Debugging, and Tuning Spark Machine Leaning Pipelines-(Joseph Bradl...Spark Summit

This document discusses Spark ML pipelines for machine learning workflows. It begins with an introduction to Spark MLlib and the various algorithms it supports. It then discusses how ML workflows can be complex, involving multiple data sources, feature transformations, and models. Spark ML pipelines allow specifying the entire workflow as a single pipeline object. This simplifies debugging, re-running on new data, and parameter tuning. The document provides an example text classification pipeline and demonstrates how data is transformed through each step via DataFrames. It concludes by discussing upcoming improvements to Spark ML pipelines.

Apache Spark Performance is too hard. Let's make it easierDatabricks

Apache Spark is a dynamic execution engine that can take relatively simple Scala code and create complex and optimized execution plans. In this talk, we will describe how user code translates into Spark drivers, executors, stages, tasks, transformations, and shuffles. We will then describe how this is critical to the design of Spark and how this tight interplay allows very efficient execution. We will also discuss various sources of metrics on how Spark applications use hardware resources, and show how application developers can use this information to write more efficient code. Users and operators who are aware of these concepts will become more effective at their interactions with Spark.

Build, Scale, and Deploy Deep Learning Pipelines with Ease Using Apache SparkDatabricks

Deep Learning has shown a tremendous success, yet it often requires a lot of effort to leverage its power. Existing Deep Learning frameworks require writing a lot of code to work with a model, let alone in a distributed manner. In this talk, we’ll survey the state of Deep Learning at scale, and where we introduce the Deep Learning Pipelines, a new open-source package for Apache Spark. This package simplifies Deep Learning in three major ways: • It has a simple API that integrates well with enterprise Machine Learning pipelines. • It automatically scales out common Deep Learning patterns, thanks to Spark. • It enables exposing Deep Learning models through the familiar Spark APIs, such as MLlib and Spark SQL. In this talk, we will look at a complex problem of image classification, using Deep Learning and Spark. Using Deep Learning Pipelines, we will show: • how to build deep learning models in a few lines of code; • how to scale common tasks like transfer learning and prediction; and • how to publish models in Spark SQL.

Spark Summit EU talk by Heiko KorndorfSpark Summit

Heiko Korndorf presented on scaling SparkR in production and lessons from real-world projects. The presentation covered classifying SparkR as both a data science and data engineering tool. It discussed SparkR's architecture in versions 1.x and 2.x, approaches to parallelizing R code with Spark 1.5/1.6 and YARN, and dynamic R deployment including dependencies. Advanced techniques like 2-level parallelization using GPGPU were also presented. The talk concluded with an outlook on further integrating data engineering and data science and new technical approaches like simplifying data pipelines and moving calculations to GPUs.

Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman

The combination of Deep Learning with Apache Spark has the potential for tremendous impact in many sectors of the industry. This webinar, based on the experience gained in assisting customers with the Databricks Virtual Analytics Platform, will present some best practices for building deep learning pipelines with Spark. Rather than comparing deep learning systems or specific optimizations, this webinar will focus on issues that are common to deep learning frameworks when running on a Spark cluster, including: * optimizing cluster setup; * configuring the cluster; * ingesting data; and * monitoring long-running jobs. We will demonstrate the techniques we cover using Google’s popular TensorFlow library. More specifically, we will cover typical issues users encounter when integrating deep learning libraries with Spark clusters. Clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker. Setting up pipelines for efficient data ingest improves job throughput, and monitoring facilitates both the work of configuration and the stability of deep learning jobs.

Overview of Apache Spark 2.3: What’s New? with Sameer AgarwalDatabricks

Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community contributors have continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2. Continuing forward in that spirit, Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of Spark 2.3 features: Kubernetes Scheduler Backend PySpark Performance and Enhancements Continuous Structured Streaming Processing DataSource v2 APIs Spark History Server Performance Enhancements

Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit

Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.

Spark Summit EU talk by Berni SchieferSpark Summit

This document summarizes experiences using the TPC-DS benchmark with Spark SQL 2.0 and 2.1 on a large cluster designed for Spark. It describes the configuration of the "F1" cluster including its hardware, operating system, Spark, and network settings. Initial results show that Spark SQL 2.0 provides significant improvements over earlier versions. While most queries completed successfully, some queries failed or ran very slowly, indicating areas for further optimization.

Spark Summit EU talk by Kaarthik SivashanmugamSpark Summit

This document introduces Mobius, a C# API for building Apache Spark applications in .NET. Mobius allows organizations invested in .NET to develop Spark jobs using C#. It provides bindings between the Scala/Java Spark API and C#, enabling reuse of existing .NET libraries in Spark and development of Spark applications using C# and F#. The document demonstrates running and debugging Mobius applications, and discusses performance considerations and internals of the Mobius driver and worker architecture.

Spark Streaming and MLlib - Hyderabad Spark GroupPhaneendra Chiruvella

Presto as a Service - Tips for operation and monitoringTaro L. Saito

- Presto as a Service in Treasure Data involves deploying Presto using blue-green deployments with no downtime and automatic error recovery of failed queries. - Monitoring Presto involves using its JSON API to view queries and query plans as well as collecting Presto metrics with Fluentd and detecting anomalies. - Benchmarking compares query performance between Presto versions by running predefined query sets and aggregating the results.

East Bay Java User Group Oct 2014 Spark Streaming Kinesis Machine LearningChris Fregly

This document provides an overview and summary of Spark Streaming. It discusses Spark Streaming's architecture and APIs. Spark Streaming receives live input data streams and divides them into micro-batches, which it processes using Spark's execution engine to perform operations like transformations and actions. This allows for low-latency, high-throughput stream processing with fault tolerance. The document also covers Spark Streaming deployment and integrating it with sources like Kinesis, as well as monitoring and tuning Spark Streaming applications.

More Related Content

What's hot (20)