Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armbrust

Feb 9, 2017Download as PPTX, PDF5 likes2,281 views

In Spark 2.0, we introduced Structured Streaming, which allows users to continually and incrementally update your view of the world as new data arrives, while still using the same familiar Spark SQL abstractions. I talk about progress we’ve made since then on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications.

Insights Without
Tradeoffs Using
Structured Streaming
Michael Armbrust - @michaelarmbrust
Spark Summit East 2017

Parallelism
Split up the problem
to harness many
machines for computation
Complexity
Handle cross-machine
communication and failures

Developer Productivity
Quickly and concisely express
common computations
Efficiency
Hand-tune code to minimize
overheads and process the
most data per cycle
SQL

Throughput
Process large historical
repositories quickly
Latency
Up-to-date answers
as new data arrives
Structured Streaming

Streaming at
Collect logs and metrics from a variety of
sources to ensure the security,
availability and performance of our cloud
platform.

Engineer Office Hours
MY HOURS
• Spark Core
• R
• Data Science
• ML
• GraphFrames, Deep Learning
• Databricks
• Spark SQL
• Structured Streaming
• Spark SQL
• Structured Streaming
• Databricks
TODAY 4:30 –
5:15
OTHER ENGINEERS
TODAY 1:45 –
5:15
THURS 10:30 –
2:30
@ Databricks Booth

Thank you!
All code available at databricks.com/blog
and @michaelarmbrust

Apache Spark 2.0 introduced Structured Streaming which allows users to continually and incrementally update your view of the world as new data arrives while still using the same familiar Spark SQL abstractions. Michael Armbrust from Databricks talks about the progress made since the release of Spark 2.0 on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications. Speaker: Michael Armbrust Video: http://go.databricks.com/videos/spark-summit-east-2017/using-structured-streaming-apache-spark This talk was originally presented at Spark Summit East 2017.

Fighting Cybercrime: A Joint Task Force of Real-Time Data and Human Analytics...Spark Summit

Cybercrime is big business. Gartner reports worldwide security spending at $80B, with annual losses totalling more than $1.2T (in 2015). Small to medium sized businesses now account for more than half of the attacks targeting enterprises today. The threat actors behind these attacks are continually shifting their techniques and toolkits to evade the security defenses that businesses commonly use. Thanks to the growing frequency and complexity of attacks, the task of identifying and mitigating security-related events has become increasingly difficult. At eSentire, we use a combination of data and human analytics to identify, respond to and mitigate cyber threats in real-time. We capture all network traffic on our customers’ networks, hence ingesting a large amount of time-series data. We process the data as it is being streamed into our system to extract relevant threat insights and block attacks in real-time. Furthermore, we enable our cybersecurity analysts to perform in-depth investigations to: i) confirm attacks and ii) identify threats that analytical models miss. Having security experts in the loop provides feedback to our analytics engine, thereby improving the overall threat detection effectiveness. So how exactly can you build an analytics pipeline to handle a large amount of time-series/event-driven data? How do you build the tools that allow people to query this data with the expectation of mission-critical response times? In this presentation, William Callaghan will focus on the challenges faced and lessons learned in building a human-in-the loop cyber threat analytics pipeline. They will discuss the topic of analytics in cybersecurity and highlight the use of technologies such as Spark Streaming/SQL, Cassandra, Kafka and Alluxio in creating an analytics architecture with missions-critical response times.

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...Databricks

Workday Prism Analytics enables data discovery and interactive Business Intelligence analysis for Workday customers. Workday is a “pure SaaS” company, providing a suite of Financial and HCM (Human Capital Management) apps to about 2000 companies around the world, including more than 30% from Fortune-500 list. There are significant business and technical challenges to support millions of concurrent users and hundreds of millions daily transactions. Using memory-centric graph-based architecture allowed to overcome most of these problems. As Workday grew, data transactions from existing and new customers generated vast amounts of valuable and highly sensitive data. The next big challenge was to provide in-app analytics platform, which for the multiple types of accumulated data, and also would allow using blend in external datasets. Workday users wanted it to be super-fast, but also intuitive and easy-to-use both for the financial and HR analysts and for regular, less technical users. Existing backend technologies were not a good fit, so we turned to Apache Spark. In this presentation, we will share the lessons we learned when building highly scalable multi-tenant analytics service for transactional data. We will start with the big picture and business requirements. Then describe the architecture with batch and interactive modules for data preparation, publishing, and query engine, noting the relevant Spark technologies. Then we will dive into the internals of Prism’s Query Engine, focusing on Spark SQL, DataFrames and Catalyst compiler features used. We will describe the issues we encountered while compiling and executing complex pipelines and queries, and how we use caching, sampling, and query compilation techniques to support interactive user experience. Finally, we will share the future challenges for 2018 and beyond.

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Spark Summit

Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.

Spark Summit EU talk by Tug GrallSpark Summit

This document discusses how Spark is enabling converged applications by allowing for both batch and stream processing of data. It describes how stream processing is important for analyzing events as they occur in real-time. Spark can be used for both offline batch analytics and real-time stream processing. The MapR converged data platform brings together storage, streaming, SQL, and machine learning processing to allow organizations to do both batch and stream processing on a single platform without data movement between systems. This enables new applications like customer behavior prediction by combining real-time streaming data with offline historical data.

Going Real-Time: Creating Frequently-Updating Datasets for Personalization: S...Spark Summit

Streaming applications have often been complex to design and maintain because of the significant upfront infrastructure investment required. However, with the advent of Spark an easy transition to stream processing is now available, enabling personalization applications and experiments to consume near real-time data without massive development cycles. Our decision to evaluate Spark as our stream processing engine was primarily led by the following considerations: 1) Ease of development for the team (already familiar with spark for batch), 2) the scope/requirements of our problem, 3) re-usability of code from spark batch jobs, and 4) Spark support from infrastructure teams within the company. In this session, we will present our experience using Spark for stream processing unbounded datasets in the personalization space. The datasets consisted of, but were not limited, to the stream of playback events that are used as feedback for all personalization algorithms. These plays are used to extract specific behaviors which are highly predictive of a customer’s enjoyment of our service. This dataset is massive and has to be further enriched by other online and offline Netflix data sources. These datasets, when consumed by our machine learning models, directly affect the customer’s personalized experience, which means that the impact is high and tolerance for failure is low. We’ll talk about the experiments we did to compare Spark with other streaming solutions like Apache Flink , the impact that we had on our customers, and most importantly, the challenges we faced. Take-aways for the audience: 1) A great example of stream processing large, personalization datasets at scale. 2) An increased awareness of the costs/requirements for making the transition from batch to streaming successfully. 3) Exposure to some of the technical challenges that should be expected along the way.

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...Spark Summit

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Building Data Pipelines with Spark and StreamSetsPat Patterson

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Metadata in upstream sources can ‘drift’ due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an Apache 2.0 licensed open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we’ll look at how SDC’s “intent-driven” approach keeps the data flowing, with a particular focus on clustered deployment with Spark and other exciting Spark integrations in the works.

Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...Spark Summit

Across all assets globally, Shell carries a huge stock of spare part inventory which ties up large quantities of working capital. Over the past 2 years an interdisciplinary project team has produced a tool, Inventory Optimization Analytics solution (IOTA), based on advanced analytical methods, that helps assets optimise stock levels and purchase strategies. To calculate the recommended stocking inventory level requirement for a material the Data Science team have written a Markov Chain Monte Carlo (MCMC) bootstrapping statistical model in R. Cumulatively, the computational task is large but, fortunately, is one of an embarrassingly parallel nature because the model can be applied independently to each material. The original solution which utilised the R “parallel” package was deployed on a single 48 core PC and took 48 hours to run. In this presentation, we describe how we moved the original solution to a distributed cloud-based Apache Spark framework. Using the new R User Defined Functions API in Apache Spark and with only a minimal amount of code changes the computational run time was reduced to 4 hours. A restructuring of the architecture to “pipeline” the problem resulted in a run time of less than 1 hour. This use case is important because it verifies the scalability and performance of SparkR.

Trends for Big Data and Apache Spark in 2017 by Matei ZahariaSpark Summit

Big data remains a rapidly evolving field with new applications and infrastructure appearing every year. In this talk, I’ll cover new trends in 2016 / 2017 and how Apache Spark is moving to meet them. In particular, I’ll talk about work Databricks is doing to make Apache Spark interact better with native code (e.g. deep learning libraries), support heterogeneous hardware, and simplify production data pipelines in both streaming and batch settings through Structured Streaming.

Big Telco - Yousun JeongSpark Summit

This document provides an overview of SK Telecom's use of big data analytics and Spark. Some key points: - SKT collects around 250 TB of data per day which is stored and analyzed using a Hadoop cluster of over 1400 nodes. - Spark is used for both batch and real-time processing due to its performance benefits over other frameworks. Two main use cases are described: real-time network analytics and a network enterprise data warehouse (DW) built on Spark SQL. - The network DW consolidates data from over 130 legacy databases to enable thorough analysis of the entire network. Spark SQL, dynamic resource allocation in YARN, and integration with BI tools help meet requirements for timely processing and quick

Spark Summit EU talk by Pat PattersonSpark Summit

This document discusses building data pipelines with Spark and StreamSets. It describes how StreamSets Data Collector can be used to build pipelines that run on Spark today by leveraging Kafka RDDs and containers on Spark. It also outlines future directions for deeper Spark integration, including running pipelines on Databricks and developing a standalone Spark processor. The document concludes with a demo of StreamSets Data Collector capabilities.

Data Science at Scale by Sarah GuidoSpark Summit

This document summarizes Sarah Guido's talk on using Apache Spark for data science at Bitly. She discusses how Bitly uses Spark to extract, explore, and model subsets of their data including decoding Bitly links, performing topic modeling using LDA, and trend detection. While Spark provides performance benefits over MapReduce for these tasks, she notes issues with Hadoop servers, JVM, and lack of documentation that must be addressed for full production usage at Bitly.

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...Spark Summit

Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.

Accelerating Machine Learning and Deep Learning At Scale...With Apache Spark:...Spark Summit

Deep learning is a fast growing subset of machine learning. There is an emerging trend to conduct deep learning in the same cluster along with existing data processing pipelines to support feature engineering and traditional machine learning. As the leading framework for Distributed ML, we believe that the addition of deep learning to the super-popular Spark framework is important, because it allows Spark developers to perform a range of data analysis tasks within a single framework that helps avoid the complexity inherent in using multiple frameworks and libraries. As one of the early and top contributors to Apache Spark, Intel is thrilled to share with the community a big deal contribution to open source Spark…”BigDL” -… A distributed deep Learning framework organically built on Big Data (Apache Spark) platform. It combines the benefits of “high performance computing” and “Big Data” architecture for rich deep learning support. With BigDL on Spark, customers can eliminate large volume of unnecessary dataset transfer between separate systems, eliminate separate HW clusters and move towards a CPU cluster, reduce system complexity and the latency for end-to-end learning. Ultimately, customers can achieve better scale, higher resource utilization, ease of use/development, and better TCO. Feature parity with Caffe and Torch, significant performance boost when combined with Intel’s Math Kernel Library (MKL), scale-out, fault tolerance, elasticity and dynamic resource sharing are some of the prominent features of BigDL. BigDL open source project will be launched at the 2017 Spark Summit East and this keynote will help spotlight this new contribution and benefits to the Spark developer community and encourage their wide contribution and collaboration. We will also showcase some real world applications of Big DL from early customers’ adoption.

Spark Summit EU talk by Ahsan Javed AwanSpark Summit

The document discusses performance characterization of in-memory data analytics using Apache Spark on a scale-up server. It identifies problems like poor multicore scalability, thread load imbalance, I/O wait times, and GC overhead. Solutions proposed include NUMA awareness, hyperthreading, disabling next-line prefetchers, using parallel scavenge GC, multiple small executors, and a future node architecture based on a hybrid in-storage processing and 2D processing-in-memory design. The work aims to improve node-level performance through architecture support for emerging big data workloads.

A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...Databricks

This document discusses using Apache Spark for predictive analytics on DICOM medical images. It describes challenges in working with medical image and metadata, and how tools like Spark, Spark-TK, and TensorFlow can help analyze this data at scale. A live demo then shows building machine learning models on DICOM data to derive insights and predict patient outcomes or device performance. Performance tests analyze the impact of data size, partitions, executors, and cores on processing time.

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.

Spark and Online Analytics: Spark Summit East talky by Shubham ChopraSpark Summit

Apache Spark was designed as a batch analytics system. By caching RDDs, Spark speeds up jobs that iteratively process the same data. This pattern is also applicable to online analytics. We use Bloomberg’s Spark Server as a server runtime for online analytics. Our framework implements certain useful patterns applicable to online query processing and is centered on the idea of “Managed” DataFrames that can be refreshed and updated as per user requirements, without violating the immutability of RDDs/DataFrames. However, Spark presents significant challenges with respect to availability and resilience in an online setting where Spark is required to respond to queries with high SLAs. In this talk, we try to identify specific areas where slow-down or failures can result in the largest hits on online-query performance and potential solutions to address these.

Self-Service Analytics on Hadoop: Lessons LearnedDataWorks Summit/Hadoop Summit

1) The document discusses the evolution of self-service analytics from using spreadsheets with limited data and compute, to specialized databases with larger datasets and clustered storage and compute. 2) It provides two examples of use cases using Datameer's self-service analytics on Hadoop at Comcast. The first identified network abuse by analyzing large call detail records, and the second aimed to improve customer experience by analyzing millions of customer viewing experiences. 3) Benefits of the self-service analytics approach included empowering analysts without IT hand-offs, creating repeatable processes, discovering issues like network abuse, and enhancing the customer viewing experience through data-driven insights.

Announcing Databricks Cloud (Spark Summit 2014)Databricks

How Apache Spark Is Helping Tame the Wild West of Wi-FiSpark Summit

Spark is helping Wi-Fi provider iPass tame the unpredictability of Wi-Fi hotspots. iPass analyzes over 21 billion Wi-Fi scans to understand characteristics of over 500 million records and 100 million hotspots globally. Using Spark on AWS Databricks, iPass can automatically scale to handle real-time analytics on this large and growing data in a cost-effective way. This allows iPass to build an understanding of Wi-Fi network characteristics to improve their services.

Sparkler Presentation for Spark Summit East 2017Karanjeet Singh

Using SparkML to Power a DSaaS (Data Science as a Service): Spark Summit East...Spark Summit

The document discusses Sparkle, a solution built by Comcast to address challenges in processing massive amounts of data and enabling data science workflows at scale. Sparkle is a centralized processing system with SQL and machine learning capabilities that is highly scalable and accessible via a REST API. It is used by Comcast to power various use cases including churn modeling, price elasticity analysis, and direct mail campaign optimization.

A Production Quality Sketching Library for the Analysis of Big DataDatabricks

Building Robust Production Data Pipelines with Databricks DeltaDatabricks

"Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT YOU’LL LEARN: – Understand the key data reliability and performance data pipelines challenges – How Databricks Delta helps build robust pipelines at scale – Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements – How to deliver performance gains using Delta PREREQUISITES: – A fully-charged laptop (8-16GB memory) with Chrome or Firefox – Pre-register for Databricks Community Edition" Speakers: Steven Yu, Burak Yavuz

Enancing Threat Detection with Big Data and AIDatabricks

Enterprises today face a daily barrage of cyberattacks. Responding quickly to threats is crucial to avoiding a serious breach. To do this successfully, security teams need to monitor and analyze billions of data signals or events each day. They come in different form and format. Yet, existing security tools are struggling to keep up. Threats are going unnoticed, and remediation timelines are being pushed out. Overcoming these challenges requires a new approach to threat detection rooted in data science. In this talk, we will discuss the technical aspects of: * Why cyber security analysis and threat detection are a challenge in the big data era * How real-time, big data analytics and AI enhance threat detection and uncover anomalous behavior patterns * Showcase how Data Scientists can use the Databricks’ Unified Analytics Platform to improve an organization's security posture with highly scalable, real-time analytics, and AI while reducing ingress and retention costs

From Idea to Model: Productionizing Data Pipelines with Apache AirflowDatabricks

Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Spark Summit

Elasticsearch provides native integration with Apache Spark through ES-Hadoop. However, especially during development, it is at best cumbersome to have Elasticsearch running in a separate machine/instance. Leveraging Spark Cluster with Elasticsearch Inside it is possible to run an embedded instance of Elasticsearch in the driver node of a Spark Cluster. This opens up new opportunities to develop cutting-edge applications. One such application is Dataset Search. Oscar will give a demo of a Dataset Search Engine built on Spark Cluster with Elasticsearch Inside. Motivation is that once Elasticsearch is running on Spark it becomes possible and interesting to have the Elasticsearch in-memory instance join an (existing) Elasticsearch cluster. And this in turn enables indexing of Datasets that are processed as part of Data Pipelines running on Spark. Dataset Search and Data Management are R&D topics that should be of interest to Spark Summit East attendees who are looking for a way to organize their Data Lake and make it searchable.

Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...Spark Summit

We all dread “Lost task” and “Container killed by YARN for exceeding memory limits” messages in our scaled-up spark yarn applications. Even answering the question “How much memory did my application use?” is surprisingly tricky in the distributed yarn environment. Sqrrl has developed a testing framework for observing vital statistics of spark jobs including executor-by-executor memory and CPU usage over time for both the JDK and python portions of pyspark yarn containers. This talk will detail the methods we use to collect, store, and report spark yarn resource usage. This information has proved to be invaluable for performance and regression testing of the spark jobs in Sqrrl Enterprise.

More Related Content

What's hot (20)

Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...Spark Summit

Trends for Big Data and Apache Spark in 2017 by Matei ZahariaSpark Summit

Big Telco - Yousun JeongSpark Summit

Spark Summit EU talk by Pat PattersonSpark Summit

Data Science at Scale by Sarah GuidoSpark Summit

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...Spark Summit

Accelerating Machine Learning and Deep Learning At Scale...With Apache Spark:...Spark Summit

Spark Summit EU talk by Ahsan Javed AwanSpark Summit

A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...Databricks

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

Spark and Online Analytics: Spark Summit East talky by Shubham ChopraSpark Summit

Self-Service Analytics on Hadoop: Lessons LearnedDataWorks Summit/Hadoop Summit

Announcing Databricks Cloud (Spark Summit 2014)Databricks

How Apache Spark Is Helping Tame the Wild West of Wi-FiSpark Summit

Sparkler Presentation for Spark Summit East 2017Karanjeet Singh

Using SparkML to Power a DSaaS (Data Science as a Service): Spark Summit East...Spark Summit

A Production Quality Sketching Library for the Analysis of Big DataDatabricks

Building Robust Production Data Pipelines with Databricks DeltaDatabricks

Enancing Threat Detection with Big Data and AIDatabricks

From Idea to Model: Productionizing Data Pipelines with Apache AirflowDatabricks

Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...Spark Summit

Trends for Big Data and Apache Spark in 2017 by Matei ZahariaSpark Summit

Big Telco - Yousun JeongSpark Summit

Spark Summit EU talk by Pat PattersonSpark Summit

Data Science at Scale by Sarah GuidoSpark Summit

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...Spark Summit

Accelerating Machine Learning and Deep Learning At Scale...With Apache Spark:...Spark Summit

Spark Summit EU talk by Ahsan Javed AwanSpark Summit

A Predictive Analytics Workflow on DICOM Images using Apache Spark with Anahi...Databricks

Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit

Spark and Online Analytics: Spark Summit East talky by Shubham ChopraSpark Summit

Self-Service Analytics on Hadoop: Lessons LearnedDataWorks Summit/Hadoop Summit

Announcing Databricks Cloud (Spark Summit 2014)Databricks

How Apache Spark Is Helping Tame the Wild West of Wi-FiSpark Summit

Sparkler Presentation for Spark Summit East 2017Karanjeet Singh

Using SparkML to Power a DSaaS (Data Science as a Service): Spark Summit East...Spark Summit

A Production Quality Sketching Library for the Analysis of Big DataDatabricks

Building Robust Production Data Pipelines with Databricks DeltaDatabricks

Enancing Threat Detection with Big Data and AIDatabricks

From Idea to Model: Productionizing Data Pipelines with Apache AirflowDatabricks

Viewers also liked (9)

Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Spark Summit

Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...Spark Summit

A New “Sparkitecture” for Modernizing your Data Warehouse: Spark Summit East ...Spark Summit

Legacy enterprise data warehouse (EDW) architecture, geared toward day-to-day workloads associated with operational querying, reporting, and analytics, are often ill-equipped to handle the volume of data, traffic, and varied data types associated with a modern, ad-hoc analytics platform. Faced with challenges of increasing pipeline speed, aggregation, and visualization in a simplified, self-service fashion, organizations are increasingly turning to some combination of Spark, Hadoop, Kafka, and proven analytical databases like Vertica as key enabling technologies to optimize their EDW architecture. Join us to learn how successful organizations have developed real-time streaming solutions with these technologies for range of use cases, including IOT predictive maintenance.

What No One Tells You About Writing a Streaming App: Spark Summit East talk b...Spark Summit

So you know you want to write a streaming app but any non-trivial streaming app developer would have to think about these questions: How do I manage offsets? How do I manage state? How do I make my spark streaming job resilient to failures? Can I avoid some failures? How do I gracefully shutdown my streaming job? How do I monitor and manage (e.g. re-try logic) streaming job? How can I better manage the DAG in my streaming job? When to use checkpointing and for what? When not to use checkpointing? Do I need a WAL when using streaming data source? Why? When don’t I need one? In this talk, we’ll share practices that no one talks about when you start writing your streaming app, but you’ll inevitably need to learn along the way.

Lessons Learned from Dockerizing Spark Workloads: Spark Summit East talk by T...Spark Summit

Many initiatives for running applications inside containers have been scoped to run on a single host. Using Docker containers for large-scale production environments poses interesting challenges, especially when deploying distributed Big Data applications like Apache Spark. Some of these challenges include container lifecycle management, smart scheduling for optimal resource utilization, network configuration and security, and performance. At BlueData, we’re “all in” on Docker containers – with a specific focus on Spark applications. We’ve learned first-hand how to address these challenges for Fortune 500 enterprises and government organizations that want to deploy Big Data workloads using Docker. In this session, you’ll learn about networking Docker containers across multiple hosts securely. We’ll discuss ways to achieve high availability across distributed Big Data applications and hosts in your data center. And since we’re talking about very large volumes of data, performance is a key factor. So we’ll discuss some of the storage options we explored and implemented at BlueData to achieve near bare-metal I/O performance for Spark using Docker. We’ll share our lessons learned as well as some tips and tricks on how to Dockerize your Big Data applications in a reliable, scalable, and high-performance environment.

Tuning and Monitoring Deep Learning on Apache SparkDatabricks

Deep Learning on Apache Spark has the potential for huge impact in research and industry. This talk will describe best practices for building deep learning pipelines with Spark. Rather than comparing deep learning systems or specific optimizations, this talk will focus on issues that are common to many deep learning frameworks when running on a Spark cluster: optimizing cluster setup and data ingest, tuning the cluster, and monitoring long-running jobs. We will demonstrate the techniques we cover using Google’s popular TensorFlow library. More specifically, we will cover typical issues users encounter when integrating deep learning libraries with Spark clusters. Clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker. Setting up pipelines for efficient data ingest improves job throughput. Interactive monitoring facilitates both the work of configuration and checking the stability of deep learning jobs. Speaker: Tim Hunter This talk was originally presented at Spark Summit East 2017.

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...Spark Summit

Spark Streaming makes it easy to build scalable, robust stream processing applications — but only once you’ve made your data accessible to the framework. If your data is already in one of Spark Streaming’s well-supported message queuing systems, this is easy. If not, an ad hoc solution to import data may work for a single application, but trying to scale that approach to complex data pipelines integrating dozens of data sources and sinks with multi-stage processing quickly breaks down. Spark Streaming solves the realtime data processing problem, but to build large scale data pipeline we need to combine it with another tool that addresses data integration challenges. The Apache Kafka project recently introduced a new tool, Kafka Connect, to make data import/export to and from Kafka easier. This talk will first describe some data pipeline anti-patterns we have observed and motivate the need for a tool designed specifically to bridge the gap between other data systems and stream processing frameworks. We will introduce Kafka Connect, starting with basic usage, its data model, and how a variety of systems can map to this model. Next, we’ll explain how building a tool specifically designed around Kafka allows for stronger guarantees, better scalability, and simpler operationalization compared to other general purpose data copying tools. Finally, we’ll describe how combining Kafka Connect and Spark Streaming, and the resulting separation of concerns, allows you to manage the complexity of building, maintaining, and monitoring large scale data pipelines.

Spark SQL: Another 16x Faster After Tungsten: Spark Summit East talk by Brad ...Spark Summit

Apache Spark 2.1.0 boosted the performance of Apache Spark SQL due to Project Tungsten software improvements. Another 16x times faster has been achieved by using Oracle’s innovations for Apache Spark SQL. This 16x improvement is made possible by using Oracle’s Software in Silicon accelerator offload technologies. Apache Spark SQL In-memory performance is becoming more important due to many factors. Users are now performing more advanced SQL processing on multi-terabyte workloads. In addition on-prem and cloud servers are getting larger physical memory to enable storing these huge workloads be stored in memory. In this talk we will look at using Spark SQL in feature creation, feature generation within pipelines for Spark ML. This presentation will explore workloads at scale and with complex interactions. We also provide best practices and tuning suggestion to support these kinds of workloads on real applications in cloud deployments. In addition ideas for next generation Tungsten project will also be discussed.

Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit East ta...Spark Summit

Netflix is the world’s largest streaming service, with 80 million members in over 250 countries. Netflix uses machine learning to inform nearly every aspect of the product, from the recommendations you get, to the boxart you see, to the decisions made about which TV shows and movies are created. Given this scale, we utilized Apache Spark to be the engine of our recommendation pipeline. Apache Spark enables Netflix to use a single, unified framework/API – for ETL, feature generation, model training, and validation. With pipeline framework in Spark ML, each step within the Netflix recommendation pipeline (e.g. label generation, feature encoding, model training, model evaluation) is encapsulated as Transformers, Estimators and Evaluators – enabling modularity, composability and testability. Thus, Netflix engineers can build our own feature engineering logics as Transformers, learning algorithms as Estimators, and customized metrics as Evaluators, and with these building blocks, we can more easily experiment with new pipelines and rapidly deploy them to production. In this talk, we will discuss how Apache Spark is used as a distributed framework we build our own algorithms on top of to generate personalized recommendations for each of our 80+ million subscribers, specific techniques we use at Netflix to scale, and the various pitfalls we’ve found along the way.