Sketching Data with T-Digest In Apache Spark: Spark Summit East talk by Erik Erlandson

Feb 15, 20173 likes2,437 views

Algorithms for sketching probability distributions from large data sets are a fundamental building block of modern data science. Sketching plays a role in diverse applications ranging from visualization, optimizing data encodings, estimating quantiles, data synthesis and imputation. The T-Digest is a versatile sketching data structure. It operates on any numeric data, models tricky distribution tails with high fidelity, and most crucially it works smoothly with aggregators and map-reduce. T-Digest is a perfect fit for Apache Spark; it is single-pass and intermediate results can be aggregated across partitions in batch jobs or aggregated across windows in streaming jobs. In this talk I will describe a native Scala implementation of the T-Digest sketching algorithm and demonstrate its use in Spark applications for visualization, quantile estimations and data synthesis. Attendees of this talk will leave with an understanding of data sketching with T-Digest sketches, and insights about how to apply T-Digest to their own data analysis applications.

Erik Erlandson
Sketching Data With
T-Digest in Apache Spark
Red Hat, Inc.

Introduction
Erik Erlandson
Software Engineer at Red Hat, Inc.
Emerging Technologies Group
Internal Data Science
Insightful Applications

Why Sketching?
● Faster
● Smaller
● Essential Features

We All Sketch Data
3.4
6.0
2.5
⋮
Mean = 3.97
Variance = 3.30
3.4, 5.0, 9.0
6.0, 2.1, 7.7
2.5, 4.4, 3.2
⋮

T-Digest
• Computing Extremely Accurate Quantiles Using
t-Digests
• Ted Dunning & Omar Ertl
• https://github.com/tdunning/t-digest
• Implementations in Java, Python, R, JS, C++
and Scala

What is T-Digest Sketching?
3.4
6.0
2.5
⋮
(3.4, 3)
(6.0, 2)
(2.5, 8)
⋮
or
Sketch of
CDF
P(X <= x)
X
Data Domain

Incremental Updates
Current
T-Digest
+ (x, w) = Updated
T-Digest
Large or
Streaming Data
Compact
“Running”
Sketch

The Payoff
REST
Service
Query
Latencies
What does my
latency distribution
look like?
I want to simulate
my latencies!
Are 90% of my
latencies under 1
second?

Representation
clusters
Distribution
CDF
(location, mass)
(x, m)

Update
(x, m)
Nearest
Cluster
Update
location
Increment
Mass

Cluster Mass Bounds
q=0 q=1
C∙M/4
Quantiles q(x)
M =
(masses)
B(x) =
C∙M∙q(x)∙(1-q(x))
C =
compression

Bounds Force New Clusters
(x,m)
mc
+ m?
(xc
,mc
)
mc
+ m > B(xc
)!
(xc
,mc
) (xu
,B(xc
))
(x, B(xc
)-(mc
+ m))
(x,m)

Resolution
q=0 q=1
More small
clusters
Fewer Large
Clusters

T-Digests are Monoidal
C1
∪ C2
D1
|+| D2
D1
≡ C1
D2
≡ C2
C1
∪ C2
⟹

Monoidal => Map-Reduce
P1
P2
Pn
|+|
Data in Spark t-digests
result
Map

7
|+| - Randomized Order
1
3
5
92 4
86 1110
7
1
3
5
9 24
86 1110D1
|+| D2
⟸

7
|+| - Merged Order
1
3
5
92 4
86 1110
7
1
3
5
92 4
86 1110D1
|+| D2
⟸

7
|+| - Large to Small
1
3
5
92 4
86 1110
7
1
3
5
924
8 611 10
D1
|+| D2
⟸

Algorithmic Considerations
• Clusters maintained in sorted order by location
• Clusters frequently inserted / deleted / updated
• Query the cluster nearest to an incoming (x,m)
• Given (x,m), query the prefix-sum of cluster mass
– (m’), over all clusters (x’,m’) where x’ <= x
• Do it all in logarithmic time!

Scala Considerations
• Immutable Red/Black Tree
• Extends Map and MapLike
• Capabilities are Mixable Traits
– Red/Black
– Ordered
– Incrementable-Values
– Nearest-Neighbor
– Prefix-Sum
• Interface to Algebird Monoids & Aggregators

$Discrete Distributions If (tdigest.clusters.size <= max_discrete) { // increment by m (or insert new) tdigest.clusters.increment(x, m) } else { // do full t-digest cluster updating algorithm tdigest.update(x, m) } Experim ental$

Applications
• Quantile Estimation
• Feature Data Characterization
• Building CoDecs
• Value-At-Risk Modeling
• Generative Data Models

Thank You
eje@redhat.com
@manyangled
https://github.com/isarn/isarn-sketches

This document provides an overview of Pandas, a Python library used for data analysis and manipulation. Pandas allows users to manage, clean, analyze and model data. It organizes data in a form suitable for plotting or displaying tables. Key data structures in Pandas include Series for 1D data and DataFrame for 2D (tabular) data. DataFrames can be created from various inputs and Pandas includes input/output tools to read data from files into DataFrames.

User Defined Aggregation in Apache Spark: A Love StoryDatabricks

This document summarizes a user's journey developing a custom aggregation function for Apache Spark using a T-Digest sketch. The user initially implemented it as a User Defined Aggregate Function (UDAF) but ran into performance issues due to excessive serialization/deserialization. They then worked to resolve it by implementing the function as a custom Aggregator using Spark 3.0's new aggregation APIs, which avoided unnecessary serialization and provided a 70x performance improvement. The story highlights the importance of understanding how custom functions interact with Spark's execution model and optimization techniques like avoiding excessive serialization.

Pandasmaikroeder

Pandas is a powerful Python library for data analysis and manipulation. It provides rich data structures for working with structured and time series data easily. Pandas allows for data cleaning, analysis, modeling, and visualization. It builds on NumPy and provides data frames for working with tabular data similarly to R's data frames, as well as time series functionality and tools for plotting, merging, grouping, and handling missing data.

GDG Helwan Introduction to pythonMohamed Hegazy

Python is a widely used programming language with a design philosophy that emphasizes code readability. The presentation covered Python installation, syntax, objects, conditions and loops, classes and functions, error handling, modules, working with files and databases. It also provided an overview of concepts in big data like volume, velocity and variety of data as well as Google Cloud tools for big data like BigQuery, Cloud Dataflow, Cloud Dataproc, and Cloud Datalab.

A Rusty introduction to Apache Arrow and how it applies to a time series dat...Andrew Lamb

Data Streaming Ecosystem Management at Booking.com confluent

This document provides an overview of the data streaming ecosystem at Booking.com. It discusses how Booking.com uses Apache Kafka, Kafka Connect, and related tools across over 300 clusters containing over 350 brokers to handle large volumes of streaming data from its various services and applications. Key aspects of Booking.com's data streaming infrastructure are highlighted, including its use of multiple data centers, global and local clusters, monitoring and alerting systems, and operational best practices.

cordova/electronの構造を知るYasuharu Seki

C++ programming functionVishalini Mugunen

Functions allow code to be reused by defining formulas that can be called from different parts of a program. Functions take in inputs, perform operations, and return outputs. They are defined outside of the main body with a function prototype, and can be called multiple times from within main or other functions. This document demonstrates how to define a FindMax function that takes in two numbers, compares them, and returns the maximum number. It shows function prototypes, defining the function outside of main, and calling the function from within main to find the maximum of two user-input numbers.

Python basicsJyoti shukla

The document discusses Python programming language. Some key points: - Python was invented by Guido Van Rossum and is a combination of various programming languages. - It is a general purpose, high-level programming language that is interpreted, interactive, object-oriented and extensible. - Python can be used for web applications, desktop applications, data analysis, artificial intelligence, and more. Popular companies that use Python include Google, YouTube, and NASA. - Python code is portable and can run on various platforms. It has a simple syntax and is easy to learn.

F stringsMariatta Wijaya

F-strings provide a new, concise way to format strings in Python 3.6. They allow embedding expressions inside curly braces within a string that starts with f. This allows printing variables directly into a string without needing string formatting or .format() methods. F-strings are faster than older string formatting methods and make multiline strings easier to read by allowing expressions across multiple lines. Documentation on f-strings can be found in the Python docs glossary under "formatted string literal".

Introduction to pandasPiyush rai

Spark And Cassandra: 2 Fast, 2 FuriousJen Aman

The document discusses the Datastax Spark Cassandra Connector. It provides an overview of how the connector allows Spark to interact with Cassandra data, including performing full table scans, pushing down filters and projections to Cassandra, distributed joins using Cassandra's partitioning, and writing data back to Cassandra in a distributed way. It also highlights some recent features of the connector like support for Cassandra 3.0, materialized views, and performance improvements from the Java Wildcard Cassandra Tester project.

Clickhouse at Cloudflare. By Marek VavrusaAltinity Ltd

Cloudflare uses ClickHouse to analyze over 1 million DNS queries per second from its global network. ClickHouse is a column-oriented database that allows Cloudflare to perform complex ad-hoc queries and aggregations over trillions of rows of DNS log data with dimensions like timestamp, zone, and location. They store raw logs for 3 months and aggregated data indefinitely to monitor trends and traffic over time. The multi-tenant ClickHouse cluster inserts over 8 million rows per second and has excellent query performance for common aggregations used by Cloudflare analysts.

Python pptMohita Pandey

Python programming : Classes objectsEmertxe Information Technologies Pvt Ltd

Introduction to pythonAyshwarya Baburam

This document provides an overview of the Python programming language. It discusses what Python is, its key features, who uses it, common applications, and how to download and install Python. It then covers Python syntax concepts like identifiers, keywords, multiline statements, docstrings, indentation, comments, and string formatting. The document also introduces Python data types like numbers, strings, lists, tuples, dictionaries, sets and how to work with them. It describes how to convert between number types and access/update strings and lists. Finally, it discusses Python development environments like Anaconda and Spyder.

Python Functions Tutorial | Working With Functions In Python | Python Trainin...Edureka!

** Python Certification Training: https://www.edureka.co/python ** This Edureka PPT on Python Functions tutorial covers all the important aspects of functions in Python right from the introduction to what functions are, all the way till checking out the major functions and using the code-first approach to understand them better. Agenda Why use Functions? What are the Functions? Types of Python Functions Built-in Functions in Python User-defined Functions in Python Python Lambda Function Conclusion Python Tutorial Playlist: https://goo.gl/WsBpKe Blog Series: http://bit.ly/2sqmP4s Follow us to never miss an update in the future. Instagram: https://www.instagram.com/edureka_learning/ Facebook: https://www.facebook.com/edurekaIN/ Twitter: https://twitter.com/edurekain LinkedIn: https://www.linkedin.com/company/edureka

Sizing MongoDB Clusters MongoDB

The document discusses sizing a MongoDB cluster for a large coffee chain called PlanetDollar. It describes collecting mobile app performance data, including 2 years of historical event data with 3000-5000 events per second. The key steps to size the MongoDB cluster are: 1) calculate the collection and index sizes based on the amount of data, 2) estimate the working set size based on frequently accessed data, 3) use a simplified model to estimate IOPS requirements and adjust based on factors like working sets, and 4) calculate the number of shards needed based on storage, memory and IOPS requirements.

Introduction to Python amiable_indian

PythonOOPVeera Pendyala

This document provides examples of built-in functions and decorators in Python like map, filter, all, any, getattr, hasattr, setattr, callable, isinstance, issubclass, closures, and memoization decorators. It demonstrates how to use these functions and decorators through examples. Built-in functions like map, filter and decorators allow extending functionality of functions. Closures enable functions to remember values in enclosing scopes. The @decorator syntax is demonstrated to be equivalent to applying a function to another function.

Spark SQLJoud Khattab

The document summarizes Spark SQL, which is a Spark module for structured data processing. It introduces key concepts like RDDs, DataFrames, and interacting with data sources. The architecture of Spark SQL is explained, including how it works with different languages and data sources through its schema RDD abstraction. Features of Spark SQL are covered such as its integration with Spark programs, unified data access, compatibility with Hive, and standard connectivity.

Introduction to python programmingSrinivas Narasegouda

Python in IndustryDharmit Shah

Python Basicstusharpanda88

This document provides an introduction to the Python programming language. It covers Python's background, syntax, types, operators, control flow, functions, classes, tools, and IDEs. Key points include that Python is a multi-purpose, object-oriented language that is interpreted, strongly and dynamically typed. It focuses on readability and has a huge library of modules. Popular Python IDEs include Emacs, Vim, Komodo, PyCharm, and Eclipse.

026 Neo4j Data Loading (ETL_ELT) Best Practices - NODES2022 AMERICAS Advanced...Neo4j

Neo4j graph databasePrashant Bhargava

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Parquet is a columnar storage format for Hadoop data. It was developed collaboratively by Twitter and Cloudera to address the need for efficient analytics on large datasets. Parquet provides more efficient compression and I/O compared to row-based formats by only reading and decompressing the columns needed by a query. It has been adopted by many companies for analytics workloads involving terabytes to petabytes of data. Parquet is language-independent and supports integration with frameworks like Hive, Pig, and Impala. It provides significant performance improvements and storage savings compared to traditional row-based formats.

Intro to Web Development Using Python and DjangoChariza Pladin

This document outlines an introductory course on web development using Python and the Django web framework. The course agenda includes introductions to Python programming, setting up a Django project, creating models and populating a database, configuring URLs and views, and using Django templates. The instructor will demonstrate how to set up a development environment, build a basic polls application with Django, and display data through the admin interface and web pages. Attendees will learn basic Python concepts, write simple programs, and create their first Django-powered websites over the course of discussions and hands-on coding labs.

Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...Spark Summit

The demand for stream processing is increasing a lot these days. Immense amounts of data have to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications include trading, social networks, Internet of things, system monitoring, and many other examples. A number of powerful, easy-to-use open source platforms have emerged to address this. But the same problem can be solved differently, various but sometimes overlapping use-cases can be targeted or different vocabularies for similar concepts can be used. This may lead to confusion, longer development time or costly wrong decisions.

IoT and the Autonomous Vehicle in the Clouds: Simultaneous Localization and M...Spark Summit

Processing real-time analytics of big data streams from sensor data will continue to be an important task as embedded technology increases and we continue to generate new types and ways of data analysis, particularly in regard to the Internet of Things (IoT). Robotics models many of these key challenges well and incorporates the possibility of high- throughput streams as well as complex online machine learning and analytics algorithms. These challenges make it an almost ideal candidate for in depth analysis of real-time streaming analytics. We look at a simultaneous localization and mapping (SLAM) problem, an ongoing research area in robotics for autonomous vehicles, and well recognized as a non-trivial problem space in both industry and research. We will use a new integrated framework on Kafka and Spark Streaming to explore a constrained SLAM problem using online algorithms to navigate and map a space in real time. We present benchmarks of our open-source robot’s integration with Kafka and Spark Streaming for performance against other SLAM algorithms currently in use, explore some of the challenges we faced in our implementation, and make recommendations for improvement of performance and optimization on our framework. Finally, new to this talk, we demo real-time usage of our implementation with the Turtlebot II and explore relevant benchmarks and their implications on the future of autonomous vehicles in the IoT and cloud analytics space.

More Related Content

What's hot (20)

Python basicsJyoti shukla

F stringsMariatta Wijaya

Introduction to pandasPiyush rai

Spark And Cassandra: 2 Fast, 2 FuriousJen Aman

Clickhouse at Cloudflare. By Marek VavrusaAltinity Ltd

Python pptMohita Pandey

Python programming : Classes objectsEmertxe Information Technologies Pvt Ltd

Introduction to pythonAyshwarya Baburam

Python Functions Tutorial | Working With Functions In Python | Python Trainin...Edureka!

Sizing MongoDB Clusters MongoDB

Introduction to Python amiable_indian

PythonOOPVeera Pendyala

Spark SQLJoud Khattab

Introduction to python programmingSrinivas Narasegouda

Python in IndustryDharmit Shah

Python Basicstusharpanda88

026 Neo4j Data Loading (ETL_ELT) Best Practices - NODES2022 AMERICAS Advanced...Neo4j

Neo4j graph databasePrashant Bhargava

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Intro to Web Development Using Python and DjangoChariza Pladin

Python basicsJyoti shukla

F stringsMariatta Wijaya

Introduction to pandasPiyush rai

Spark And Cassandra: 2 Fast, 2 FuriousJen Aman

Clickhouse at Cloudflare. By Marek VavrusaAltinity Ltd

Python pptMohita Pandey

Python programming : Classes objectsEmertxe Information Technologies Pvt Ltd

Introduction to pythonAyshwarya Baburam

Python Functions Tutorial | Working With Functions In Python | Python Trainin...Edureka!

Sizing MongoDB Clusters MongoDB

Introduction to Python amiable_indian

PythonOOPVeera Pendyala

Spark SQLJoud Khattab

Introduction to python programmingSrinivas Narasegouda

Python in IndustryDharmit Shah

Python Basicstusharpanda88

026 Neo4j Data Loading (ETL_ELT) Best Practices - NODES2022 AMERICAS Advanced...Neo4j

Neo4j graph databasePrashant Bhargava

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Intro to Web Development Using Python and DjangoChariza Pladin

Viewers also liked (20)

Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...Spark Summit

IoT and the Autonomous Vehicle in the Clouds: Simultaneous Localization and M...Spark Summit

High Resolution Energy Modeling that Scales with Apache Spark 2.0 Spark Summi...Spark Summit

As advanced sensor technologies are becoming widely deployed in the energy industry, the availability of higher-frequency data results in both analytical benefits and computational costs. To an energy forecaster or data scientist, some of these benefits might include enhanced predictive performance from forecasting models as well as improved pattern recognition in energy consumption across building types, economic sectors, and geographies. To a utility or electricity service provider, these benefits might include significantly deeper insights into their diverse customer base. However, these advantages can come with a high computational price tag. With Spark 2.0, User-Defined Functions can be applied across grouped SparkDataFrames in the SparkR API to solve the multivariate optimization and model selection problems typically required for fitting site-level models. This recently added feature of Spark 2.0 on Databricks has allowed DNV GL to efficiently fit predictive models that relate weather, electricity, water, and gas consumption across virtually any number of buildings.

Scalable Data Science with SparkR: Spark Summit East talk by Felix CheungSpark Summit

R is a very popular platform for Data Science. Apache Spark is a highly scalable data platform. How could we have the best of both worlds? How could a Data Scientist leverage the rich 9000+ packages on CRAN, and integrate Spark into their existing Data Science toolset? In this talk we will walkthrough many examples how several new features in Apache Spark 2.x will enable this. We will also look at exciting changes in and coming next in Apache Spark 2.x releases.

Using SparkR to Scale Data Science Applications in Production. Lessons from t...Spark Summit

R is a hugely popular platform for Data Scientists to create analytic models in many different domains. But when these applications should move from the science lab to the production environment of large enterprises a new set of challenges arises. Independently of R, Spark has been very successful as a powerful general-purpose computing platform. With the introduction of SparkR an exciting new option to productionize Data Science applications has been made available. This talk will give insight into two real-life projects at major enterprises where Data Science applications in R have been migrated to SparkR. • Dealing with platform challenges: R was not installed on the cluster. We show how to execute SparkR on a Yarn cluster with a dynamic deployment of R. • Integrating Data Engineering and Data Science: we highlight the technical and cultural challenges that arise from closely integrating these two different areas. • Separation of concerns: we describe how to disentangle ETL and data preparation from analytic computing and statistical methods. • Scaling R with SparkR: we present what options SparkR offers to scale R applications and how we applied them to different areas such as time series forecasting and web analytics. • Performance Improvements: we will show benchmarks for an R applications that took over 20 hours on a single server/single-threaded setup. With moderate effort we have been able to reduce that number to 15 minutes with SparkR. And we will show how we plan to further reduces this to less than a minute in the future. • Mixing SparkR, SparkSQL and MLlib: we show how we combined the three different libraries to maximize efficiency. • Summary and Outlook: we describe what we have learnt so far, what the biggest gaps currently are and what challenges we expect to solve in the short- to mid-term.

R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...Spark Summit

The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.

Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...Spark Summit

Spark 2.0 provided strong performance enhancements to the Spark core while advancing Spark ML usability to use data frames. But what happens when you run Spark 2.0 machine learning algorithms on a large cluster with a very large data set? Do you even get any benefit from using a very large data set? It depends.  How do new hardware advances affect the topology of high performance Spark clusters. In this talk we will explore Spark 2.0 Machine Learning at scale and share our findings with the community. As our test platform we will be using a new cluster design, different from typical Hadoop clusters, with more cores, more RAM and latest generation NVMe SSD’s and a 100GbE network with a goal of more performance, in a more space and energy efficient footprint.

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...Spark Summit

HP ships millions of PCs, Printers, and other devices every year to customers in all market segments. More customers are seeking services provided with our products enabling new opportunities for HP to create services from the data we can collect from our devices. Every device we ship is an IoT endpoint with powerful CPU to capture rich data. Insights from this data are used internally to improve our products and focus on customer needs. In this presentation, John will focus on HP’s journey to enabling Big Data analytics from within a large enterprise environment. He will review the challenges and how HP decided on AWS, Apache Spark and Databricks as the foundation for their entry into Big Data Analytics. John will also review how HP uses Spark to build analytic services from the data they generate from their devices.

Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...Spark Summit

Apache Spark MLlib provides scalable implementation of popular machine learning algorithms, which lets users train models from big dataset and iterate fast. The existing implementations assume that the number of parameters is small enough to fit in the memory of a single machine. However, many applications require solving problems with billions of parameters on a huge amount of data such as Ads CTR prediction and deep neural network. This requirement far exceeds the capacity of exisiting MLlib algorithms many of who use L-BFGS as the underlying solver. In order to fill this gap, we developed Vector-free L-BFGS for MLlib. It can solve optimization problems with billions of parameters in the Spark SQL framework where the training data are often generated. The algorithm scales very well and enables a variety of MLlib algorithms to handle a massive number of parameters over large datasets. In this talk, we will illustrate the power of Vector-free L-BFGS via logistic regression with real-world dataset and requirement. We will also discuss how this approach could be applied to other ML algorithms.

Clipper: A Low-Latency Online Prediction Serving System: Spark Summit East ta...Spark Summit

Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learning frameworks and systems only address model training and not deployment. In this talk, we present Clipper, a general-purpose low-latency prediction serving system. Interposing between end-user applications and a wide range of machine learning frameworks, Clipper introduces a modular architecture to simplify model deployment across frameworks. Furthermore, by introducing caching, batching, and adaptive model selection techniques, Clipper reduces prediction latency and improves prediction throughput, accuracy, and robustness without modifying the underlying machine learning frameworks. We evaluated Clipper on four common machine learning benchmark datasets and demonstrate its ability to meet the latency, accuracy, and throughput demands of online serving applications. We also compared Clipper to the Tensorflow Serving system and demonstrate comparable prediction throughput and latency on a range of models while enabling new functionality, improved accuracy, and robustness.

Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...Spark Summit

This talk will cover the tools we used, the hurdles we faced and the work arounds we developed with the help from Databricks support in our attempt to build a custom machine learning model and use it to predict the TV ratings for different networks and demographics. The Apache Spark machine learning and dataframe APIs make it incredibly easy to produce a machine learning pipeline to solve an archetypal supervised learning problem. In our applications at Cadent, we face a challenge with high dimensional labels and relatively low dimensional features; at first pass such a problem is all but intractable but thanks to a large number of historical records and the tools available in Apache Spark, we were able to construct a multi-stage model capable of forecasting with sufficient accuracy to drive the business application. Over the course of our work we have come across many tools that made our lives easier, and others that forced work around. In this talk we will review our custom multi-stage methodology, review the challenges we faced and walk through the key steps that made our project successful.

Powering Predictive Mapping at Scale with Spark, Kafka, and Elastic Search: S...Spark Summit

The document discusses powering predictive mapping at scale using the SMACK stack, which includes Spark, Kafka, and Elasticsearch. It describes how the SMACK stack can ingest millions of events per second from connected devices, store the data in Apache Spark, and allow real-time and batch processing of the data. It also provides an example of using the stack for real-time tracking of geo-enabled IoT devices and demonstrates the data flow and a demo of the system.

Parallelizing Existing R Packages with SparkRDatabricks

R is the latest language added to Apache Spark, and the SparkR API is slightly different from PySpark. With the release of Spark 2.0, the R API officially supports executing user code on distributed data. This is done through a family of apply() functions. In this talk, Hossein Falaki gives an overview of this new functionality in SparkR. Using this API requires some changes to regular code with dapply(). This talk will focus on how to correctly use this API to parallelize existing R packages. Most important topics of consideration will be performance and correctness when using the apply family of functions in SparkR. Speaker: Hossein Falaki This talk was originally presented at Spark Summit East 2017.

FIS: Accelerating Digital Intelligence in FinTech: Spark Summit East talk by...Spark Summit

In 2017, 60% of the US population will be a digital banking user. The challenges to meet the demands of a more engaged customer is increasing as the banking experience becomes less formal and easier to deliver. Building a better customer experience starts with how you handle your data to leverage accountable, actionable analytics. To take this on, FIS had to overcome the challenge posed by increasing data volumes and data velocity, enterprise level operational complexity and requirements, and work with antiquated traditional techniques. Within a year we transformed to Apache Spark and Databricks, providing thousands of financial institutions the ability to build better relationships with their customers by understanding the behaviors and interactions in digital banking.

ModelDB: A System to Manage Machine Learning Models: Spark Summit East talk b...Spark Summit

Building a machine learning model is an iterative process. A data scientist will build many tens to hundreds of models before arriving at one that meets some acceptance criteria. However, the current style of model building is ad-hoc and there is no practical way for a data scientist to manage models that are built over time. In addition, there are no means to run complex queries on models and related data. In this talk, we present ModelDB, a novel end-to-end system for managing machine learning (ML) models. Using client libraries, ModelDB automatically tracks and versions ML models in their native environments (e.g. spark.ml, scikit-learn). A common set of abstractions enable ModelDB to capture models and pipelines built across different languages and environments. The structured representation of models and metadata then provides a platform for users to issue complex queries across various modeling artifacts. Our rich web frontend provides a way to query ModelDB at varying levels of granularity. ModelDB has been open-sourced at https://github.com/mitdbg/modeldb.

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli Spark Summit

In the race to invent multi-million dollar business opportunities with exclusive insights, data scientists and engineers are hampered by a multitude of challenges just to make one use case a reality – the need to ingest data from multiple sources, apply real-time analytics, build machine learning algorithms, and intermix different data processing models, all while navigating around their legacy data infrastructure that is just not up to the task. This need has created the demand for Virtual Analytics, where the complexities of disparate data and technology silos have been abstracted away, coupled with a powerful range of analytics and processing horsepower, all in one unified data platform. This talk describes how Databricks is powering this revolutionary new trend with Apache Spark.

Realtime Analytical Query Processing and Predictive Model Building on High Di...Spark Summit

Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on row based columnar datasets through full scan but does not provide constructs for column indexing and time series analysis. For dealing with document datasets with timestamps where the features are represented as variable number of columns in each document and use-cases demand searching over columns and time to retrieve documents to generate learning models in realtime, a close integration within Spark and Lucene was needed. We introduced LuceneDAO in Spark Summit Europe 2016 to build distributed lucene shards from data frame but the time series attributes were not part of the data model. In this talk we present our extension to LuceneDAO to maintain time stamps with document-term view for search and allow time filters. Lucene shards maintain the time aware document-term view for search and vector space representation for machine learning pipelines. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms with filters on time. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. Our synchronous API uses Spark-as-a-Service to power analytical queries while our asynchronous API uses kafka, spark streaming and HBase to power time series prediction algorithms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms and configurable time stamp aggregate columns. We will demonstrate the latency of APIs on a suite of queries generated from terms. Key takeaways from the talk will be a thorough understanding of how to make Lucene powered time aware search a first class citizen in Spark to build interactive analytical query processing and time series prediction algorithms.

Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by...Spark Summit

Modeling Catastrophic Events in Spark: Spark Summit East Talk by Georg Hofman...Spark Summit

This document discusses using Spark to model catastrophic events more efficiently than MapReduce. It describes how catastrophe (cat) models involve large datasets but generate even larger intermediate datasets requiring complex analytics. Spark is better suited than MapReduce for this work due to its ability to share memory and resources across processes, providing faster performance at lower costs. The document advocates designing cat model workflows in Spark to take advantage of its flexible architecture and high code quality.