Scaling spark
Download as PPTX, PDF2 likes1,155 views
This document discusses how a media platform scaled their use of Spark across AWS to process terabytes of data daily. They moved from two on-premise clusters to running analytics and streaming workloads on AWS while keeping their core workload on-premise, initially using Spark on EMR but then self-managing Spark on EC2 for more flexibility. They implemented auto-scaling of the AWS clusters to maintain utilization targets and handle fluctuating workload demands.
1 of 36
Downloaded 18 times
Ad
Recommended
Optimizing Flash at Scale
Optimizing Flash at Scaleinside-BigData.com In this deck from the DDN User Group at SC19, Gael Delbray from CEA presents: Optimizing Flash at Scale. CEA, a major player in research and innovation, has been recognized as an expert in HPC through the momentum of the "Simulation Programme" supported by its Direction des Applications Militaires (CEA / DAM) and implemented by the Department of Simulation Sciences and Information (DSSI).
"The major challenges that the HPC will face in the coming years are manifold, such as the development of hardware and software architectures able to deliver very high computing power, modelling methods combining different scales and physical models and the management of huge volumes of numerical data.
High performance computing for numerical simulation has become an essential tool in scientific and technological research, as well as for industrial applications. Simulation can replace experiments that are too dangerous (accidental situations), beyond reach in terms of time or scale (climate, astrophysics) or banned (nuclear tests). Simulation is also time-saving and leverages productivity in many situations."
Watch the video: https://wp.me/p3RLHQ-li0
Learn more: https://www.ddn.com/company/events/user-group-sc/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Bartolomeo_ASGSR
Bartolomeo_ASGSRDr. Per Christian Steimle The document describes commercial external payload hosting platforms on the International Space Station (ISS) for space research. It discusses Airbus Defense and Space and Teledyne Brown Engineering's platforms called NanoRacks External Platform and Bartolomeo. NanoRacks will host small payloads of up to 100kg starting in early 2016. Bartolomeo is designed to host multiple medium to large payloads up to 500kg and will provide power, data, cooling and pointing capabilities. The document provides details on payload accommodations, operations concepts and end-to-end commercial service for customers.
Sierra overview
Sierra overviewGanesan Narayanasamy Sierra will be LLNL's next advanced technology system and part of the CORAL collaboration between ORNL, ANL, and LLNL. Sierra will replace the current Sequoia system and feature an IBM POWER9 and NVIDIA Volta GPU accelerated architecture with over 125 PFLOPS of peak performance. Benchmark projections show the GPU-accelerated Sierra system is expected to deliver substantial performance gains compared to a CPU-only configuration. Sierra and its follow-on systems will usher in an accelerator-based computing era at LLNL.
Top FME Recipes: Raster
Top FME Recipes: RasterSafe Software The document describes 4 raster recipes that can be performed using FME:
1) Translate and reproject rasters between different formats and coordinate systems
2) Mosaic multiple rasters into a single raster
3) Tile or clip rasters to extract portions of raster data
4) Drape raster data onto 3D terrain models
OpenACC Monthly Highlights June 2017
OpenACC Monthly Highlights June 2017NVIDIA OpenACC June Highlights are full of the latest OpenACC news, events, user community, resources and more. Learn about OpenACC and join upcoming events.
GTC Taiwan 2017 如何在充滿未知的巨量數據時代中建構一個數據中心
GTC Taiwan 2017 如何在充滿未知的巨量數據時代中建構一個數據中心NVIDIA Taiwan This document discusses the evolution of data storage needs from traditional structured data to modern unstructured data like objects and machine data. It outlines the four industrial revolutions defined by major technological advances. Pure Storage's FlashBlade is introduced as the industry's first data hub purpose-built for AI and deep learning, with massively parallel architecture powered by Purity software to scale without limits. Real-world customer examples demonstrate how FlashBlade accelerates AI initiatives for autonomous vehicles and powers some of the world's most powerful AI supercomputers.
GTC 2017: Powering the AI Revolution
GTC 2017: Powering the AI RevolutionNVIDIA Jensen Huang, founder and CEO of NVIDIA, discusses the rise of GPU computing and artificial intelligence. He outlines how GPUs have enabled massive performance increases for deep learning workloads. NVIDIA is introducing new products like the Tesla V100 GPU and DGX-1 server to further accelerate AI research and commercial applications. These announcements position NVIDIA to power continued growth in AI and deep learning.
Circuit Simplifier
Circuit SimplifierVineet Markan This document describes an electric circuit solver program called Electrica that takes in a jumbled circuit diagram and outputs an organized 2D representation with reduced overlaps. It uses C++ and OpenGL and stores circuit components as nodes connected by edges in a data structure. The algorithm finds the longest chain in the input circuit to lay out the skeleton and draws connections orthogonally while color coding components. The program aims to simplify circuit analysis and has applications in PCB routing, mesh processing, and pathfinding.
Scaling graphite to handle a zerg rush
Scaling graphite to handle a zerg rushDaniel Ben-Zvi This document discusses scaling issues with Graphite and solutions implemented at Similarweb to handle high volumes of metrics. Key points:
1) Graphite struggled with high IOPS and a single-threaded carbon-cache. Replacing carbon-cache with the multi-threaded go-carbon and using SSDs helped address IOPS bottlenecks.
2) carbon-relay was replaced with the faster C implementation carbon-c-relay to load balance metrics among go-carbon instances.
3) statsd was replaced with the C implementation statsite for better performance and capabilities like quantiles.
4) The final setup consisted of statsite sending to multiple carbon-c-relay and go-carbon instances, handling
GTC Taiwan 2017 企業端深度學習與人工智慧應用
GTC Taiwan 2017 企業端深度學習與人工智慧應用NVIDIA Taiwan This document discusses the evolution of computing from PCs to mobile-cloud to AI and IoT. It highlights how deep learning using GPUs has become a new computing model, with neural network complexity exploding to tackle increasingly complex challenges. It introduces Nvidia's Volta GPU and how it delivers revolutionary performance for deep learning training and inference through new tensor cores and optimizations for deep learning frameworks and models.
An Update on Arm HPC
An Update on Arm HPCinside-BigData.com In this deck from the Linaro Connect conference, Brent Gorda presents an update on ARM for HPC.
"Arm-based systems are showing up in the HPC community and new silicon is coming. The architecture has also been selected for several of the exascale projects worldwide. Brent will talk about the aspects of Arm that are attractive to the HPC community, updates on projects and what we as a community can do to help accelerate adoption in this space."
Watch the video: https://insidehpc.com/2019/09/an-update-on-arm-in-hpc/
Learn more: https://developer.arm.com/tools-and-software/server-and-hpc
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Interconnects: Assessment & Rankings
High Performance Interconnects: Assessment & Rankingsinside-BigData.com In this deck from the HPC Advisory Council Spain Conference, Dan Olds from OrionX discusses the High Performance Interconnect (HPI) market landscape, plus provides ratings and rankings of HPI choices today.
"The HPI market is the very high-end of the networking equipment market where high bandwidth and low latency are non-negotiable. It started out as a specialist proprietary segment but has blossomed into an indispensable, large, and growing area. Products in this category are used to build extreme-scale computing systems. They are typically not used for traditional telco, enterprise, or service provider networking needs. In this talk, we’ll take a look at the technologies and performance of their high-end technology and the coming battle between onloading vs. offloading interconnect architectures."
Watch the video presentation: http://wp.me/p3RLHQ-fON
Learn more: http://orionx.net/wp-content/uploads/2016/06/HPI-Environment-OrionX-Constellation-DataCenter-20160626.pdf
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
OPTIMIZING THE TICK STACK
OPTIMIZING THE TICK STACKInfluxData In this session, you will learn how to tune your queries for performance plus strategies for effective schema design.
Cypher for Gremlin
Cypher for GremlinopenCypher Presented at the Seventh openCypher Implementers Group Meeting in March 2018 @ https://www.opencypher.org/ocig/2018/03/14/ocig7/
Programming Languages & Tools for Higher Performance & Productivity
Programming Languages & Tools for Higher Performance & ProductivityLinaro By Hitoshi Murai, RIKEN AICS
For higher performance and productivity of HPC systems, it is important to provide users with good programming environment including languages, compilers, and tools. In this talk, the programming model of the post-K supercomputer will be shown.
Hitoshi Murai Bio
Hitoshi Murai received a master's degree in information science from Kyoto University in 1996. He worked as a software developer in NEC from 1996 to 2010. He received a Ph.D degree in computer science from University of Tsukuba in 2010. He is currently a research scientist of the programming environment research team and the Flagship 2020 project in Advanced Institute for Computational Science, RIKEN. His research interests include compilers and parallel programming languages.
Email
[email protected]
For more info on The Linaro High Performance Computing (HPC) visit https://www.linaro.org/sig/hpc/
Enable IPv6 on Route53 AWS ELB, docker and node App
Enable IPv6 on Route53 AWS ELB, docker and node AppFyllo Fyllo Technical notes: Enable IPv6 support on AWS and docker machine.
We recently had to add IPv6 support to our APIs. Had to scratch head at multiple points. So, we want to help community with these notes. Happy building :)
Arm in HPC
Arm in HPCinside-BigData.com In this video from the Rice Oil & Gas Conference, Brent Gorda from ARM presents: ARM in HPC.
"With the recent Astra system at Sandia Lab (#203 on the Top500) and HPE Catalyst project in the UK, Arm-based architectures are arriving in HPC environments. Several partners have announced or will soon announce new silicon and projects, each of which offers something different and compelling for our community. Brent will describe the driving factors and how these solutions are changing the landscape for HPC."
Watch the video: https://wp.me/p3RLHQ-jXS
Learn more: https://developer.arm.com/hpc
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Debugging & Tuning in Spark
Debugging & Tuning in SparkShiao-An Yuan This document provides tips and best practices for debugging and tuning Spark applications. It discusses Spark concepts like RDDs, transformations, actions, and the DAG execution model. It then gives recommendations for improving correctness, reducing overhead from parallelism, avoiding data skew, and tuning configurations like storage level, number of partitions, executor resources and joins. Common failures are analyzed along with their causes and fixes. Overall it emphasizes the importance of tuning partitioning, avoiding shuffles when possible, and using the right configurations to optimize Spark jobs.
Spark After Dark - LA Apache Spark Users Group - Feb 2015
Spark After Dark - LA Apache Spark Users Group - Feb 2015Chris Fregly Spark After Dark is a mock dating site that uses the latest Spark libraries including Spark SQL, BlinkDB, Tachyon, Spark Streaming, MLlib, and GraphX to generate high-quality dating recommendations for its members and blazing fast analytics for its operators.
We begin with brief overview of Spark, Spark Libraries, and Spark Use Cases. In addition, we'll discuss the modern day Lambda Architecture that combines real-time and batch processing into a single system. Lastly, we present best practices for monitoring and tuning a highly-available Spark and Spark Streaming cluster.
There will be many live demos covering everything from basic topics such as ETL and data ingestion to advanced topics such as streaming, sampling, approximations, machine learning, textual analysis, and graph processing.
Scaling Machine Learning to Billions of Parameters - Spark Summit 2016
Scaling Machine Learning to Billions of Parameters - Spark Summit 2016Badri Narayan Bhaskar This document summarizes scaling machine learning to billions of parameters using Spark and a parameter server architecture. It describes the requirements for supporting both batch and sequential optimization at web scale. It then outlines the Spark + Parameter server approach, leveraging Spark for distributed processing and the parameter server for synchronizing model updates. Examples of distributed L-BFGS and Word2Vec training are provided to illustrate batch and sequential optimization respectively using this architecture.
Foundations for Scaling ML in Apache Spark
Foundations for Scaling ML in Apache SparkDatabricks Apache Spark has become the most active open source Big Data project, and its Machine Learning library MLlib has seen rapid growth in usage. A critical aspect of MLlib and Spark is the ability to scale: the same code used on a laptop can scale to 100’s or 1000’s of machines. This talk will describe ongoing and future efforts to make MLlib even faster and more scalable by integrating with two key initiatives in Spark. The first is Catalyst, the query optimizer underlying DataFrames and Datasets. The second is Tungsten, the project for approaching bare-metal speeds in Spark via memory management, cache-awareness, and code generation. This talk will discuss the goals, the challenges, and the benefits for MLlib users and developers. More generally, we will reflect on the importance of integrating ML with the many other aspects of big data analysis.
Scaling Analytics with Apache Spark
Scaling Analytics with Apache SparkQuantUniversity Since its debut in 2010, Apache Spark has become one of the most popular Big Data technologies in the Apache open source ecosystem. In addition to enabling processing of large data sets through its distributed computing architecture, Spark provides out-of-the-box support for machine learning, streaming and graph processing in a single framework. Spark has been supported by companies like Microsoft, Google, Amazon and IBM and in financial services, companies like Blackrock (http://bit.ly/1Q1DVJH ) and Bloomberg (http://bit.ly/29LXbPv ) have started to integrate Apache Spark into their tool chain and the interest is growing. Unlike other big-data technologies which require intensive programming using Java etc., Spark enables data scientists to work with a big-data technology using higher level languages like Python and R making it accessible to conduct experiments and for rapid prototyping.
In this talk, we will introduce Apache Spark and discuss the key features that differentiate Apache Spark from other technologies. We will provide examples on how Apache Spark can help scale analytics and discuss how the machine learning API could be used to solve large-scale machine learning problems using Spark’s distributed computing framework. We will also illustrate enterprise use cases for scaling analytics with Apache Spark.
Spark performance tuning - Maksud Ibrahimov
Spark performance tuning - Maksud IbrahimovMaksud Ibrahimov The talk by Maksud Ibrahimov, Chief Data Scientist at InfoReady Analytics. He is going to share with us how to maximise the performance of Spark.
As a user of Apache Spark from very early releases, he generally sees that the framework is easy to start with but as the program grows its performance starts to suffer. In this talk Maksud will answer the following questions:
- How to reach higher level of parallelism of your jobs without scaling up your cluster?
- Understanding shuffles, and how to avoid disk spills
- How to identify task stragglers and data skews?
- How to identify Spark bottlenecks?
Machine learning at Scale with Apache Spark
Machine learning at Scale with Apache SparkMartin Zapletal This document discusses scaling machine learning using Apache Spark. It covers several key topics:
1) Parallelizing machine learning algorithms and neural networks to distribute computation across clusters. This includes data, model, and parameter server parallelism.
2) Apache Spark's Resilient Distributed Datasets (RDDs) programming model which allows distributing data and computation across a cluster in a fault-tolerant manner.
3) Examples of very large neural networks trained on clusters, such as a Google face detection model using 1,000 servers and a IBM brain-inspired chip model using 262,144 CPUs.
Spark 1.6 vs Spark 2.0
Spark 1.6 vs Spark 2.0Sigmoid By Abhishek Sharma
CONTENT:
-Apache Spark
-Project Tungsten
-Whole Stage Code Generation
-Vectorization
Big Data Day LA 2016 Keynote - Reynold Xin/ Databricks
Big Data Day LA 2016 Keynote - Reynold Xin/ DatabricksData Con LA This document discusses scaling big data using Apache Spark. It provides an overview of Spark's philosophy of providing a unified engine to support end-to-end applications using high-level APIs. It outlines some of the new features in Apache Spark 2.0, including improvements to structured APIs, structured streaming, and new deep learning and graph processing libraries. It also discusses initiatives by Databricks to grow the Spark community through massive open online courses and a free community edition of the Databricks platform.
Building, Debugging, and Tuning Spark Machine Leaning Pipelines-(Joseph Bradl...
Building, Debugging, and Tuning Spark Machine Leaning Pipelines-(Joseph Bradl...Spark Summit This document discusses Spark ML pipelines for machine learning workflows. It begins with an introduction to Spark MLlib and the various algorithms it supports. It then discusses how ML workflows can be complex, involving multiple data sources, feature transformations, and models. Spark ML pipelines allow specifying the entire workflow as a single pipeline object. This simplifies debugging, re-running on new data, and parameter tuning. The document provides an example text classification pipeline and demonstrates how data is transformed through each step via DataFrames. It concludes by discussing upcoming improvements to Spark ML pipelines.
Building Robust, Adaptive Streaming Apps with Spark Streaming
Building Robust, Adaptive Streaming Apps with Spark StreamingDatabricks As the adoption of Spark Streaming increases rapidly, the community has been asking for greater robustness and scalability from Spark Streaming applications in a wider range of operating environments. To fulfill these demands, we have steadily added a number of features in Spark Streaming. We have added backpressure mechanisms which allows Spark Streaming to dynamically adapt to changes in incoming data rates, and maintain stability of the application. In addition, we are extending Spark’s Dynamic Allocation to Spark Streaming, so that streaming applications can elastically scale based on processing requirements. In my talk, I am going to explore these mechanisms and explain how developers can write robust, scalable and adaptive streaming applications using them. Presented by Tathagata "TD" Das from Databricks.
How to Boost 100x Performance for Real World Application with Apache Spark-(G...
How to Boost 100x Performance for Real World Application with Apache Spark-(G...Spark Summit This document summarizes work done by an Intel software team in China to improve Apache Spark performance for real-world applications. It describes benchmarking tools like HiBench and profiling tools like HiMeter that were developed. It also discusses several case studies where the team worked with customers to optimize joins, manage memory usage, and reduce network bandwidth. The overall goal was to help solve common issues around ease of use, reliability, and scalability for Spark in production environments.
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014Chris Fregly Spark Streaming allows for processing of real-time data streams using Spark. The document discusses using Spark Streaming with Amazon Kinesis for streaming data ingestion. It covers the Spark Streaming and Kinesis integration architecture, how the Spark Kinesis receiver works, scaling considerations, and fault tolerance mechanisms through checkpointing. Examples of monitoring and tuning Spark Streaming jobs on Kinesis data are also provided.
Ad
More Related Content
What's hot (9)
Scaling graphite to handle a zerg rush
Scaling graphite to handle a zerg rushDaniel Ben-Zvi This document discusses scaling issues with Graphite and solutions implemented at Similarweb to handle high volumes of metrics. Key points:
1) Graphite struggled with high IOPS and a single-threaded carbon-cache. Replacing carbon-cache with the multi-threaded go-carbon and using SSDs helped address IOPS bottlenecks.
2) carbon-relay was replaced with the faster C implementation carbon-c-relay to load balance metrics among go-carbon instances.
3) statsd was replaced with the C implementation statsite for better performance and capabilities like quantiles.
4) The final setup consisted of statsite sending to multiple carbon-c-relay and go-carbon instances, handling
GTC Taiwan 2017 企業端深度學習與人工智慧應用
GTC Taiwan 2017 企業端深度學習與人工智慧應用NVIDIA Taiwan This document discusses the evolution of computing from PCs to mobile-cloud to AI and IoT. It highlights how deep learning using GPUs has become a new computing model, with neural network complexity exploding to tackle increasingly complex challenges. It introduces Nvidia's Volta GPU and how it delivers revolutionary performance for deep learning training and inference through new tensor cores and optimizations for deep learning frameworks and models.
An Update on Arm HPC
An Update on Arm HPCinside-BigData.com In this deck from the Linaro Connect conference, Brent Gorda presents an update on ARM for HPC.
"Arm-based systems are showing up in the HPC community and new silicon is coming. The architecture has also been selected for several of the exascale projects worldwide. Brent will talk about the aspects of Arm that are attractive to the HPC community, updates on projects and what we as a community can do to help accelerate adoption in this space."
Watch the video: https://insidehpc.com/2019/09/an-update-on-arm-in-hpc/
Learn more: https://developer.arm.com/tools-and-software/server-and-hpc
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Interconnects: Assessment & Rankings
High Performance Interconnects: Assessment & Rankingsinside-BigData.com In this deck from the HPC Advisory Council Spain Conference, Dan Olds from OrionX discusses the High Performance Interconnect (HPI) market landscape, plus provides ratings and rankings of HPI choices today.
"The HPI market is the very high-end of the networking equipment market where high bandwidth and low latency are non-negotiable. It started out as a specialist proprietary segment but has blossomed into an indispensable, large, and growing area. Products in this category are used to build extreme-scale computing systems. They are typically not used for traditional telco, enterprise, or service provider networking needs. In this talk, we’ll take a look at the technologies and performance of their high-end technology and the coming battle between onloading vs. offloading interconnect architectures."
Watch the video presentation: http://wp.me/p3RLHQ-fON
Learn more: http://orionx.net/wp-content/uploads/2016/06/HPI-Environment-OrionX-Constellation-DataCenter-20160626.pdf
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
OPTIMIZING THE TICK STACK
OPTIMIZING THE TICK STACKInfluxData In this session, you will learn how to tune your queries for performance plus strategies for effective schema design.
Cypher for Gremlin
Cypher for GremlinopenCypher Presented at the Seventh openCypher Implementers Group Meeting in March 2018 @ https://www.opencypher.org/ocig/2018/03/14/ocig7/
Programming Languages & Tools for Higher Performance & Productivity
Programming Languages & Tools for Higher Performance & ProductivityLinaro By Hitoshi Murai, RIKEN AICS
For higher performance and productivity of HPC systems, it is important to provide users with good programming environment including languages, compilers, and tools. In this talk, the programming model of the post-K supercomputer will be shown.
Hitoshi Murai Bio
Hitoshi Murai received a master's degree in information science from Kyoto University in 1996. He worked as a software developer in NEC from 1996 to 2010. He received a Ph.D degree in computer science from University of Tsukuba in 2010. He is currently a research scientist of the programming environment research team and the Flagship 2020 project in Advanced Institute for Computational Science, RIKEN. His research interests include compilers and parallel programming languages.
Email
[email protected]
For more info on The Linaro High Performance Computing (HPC) visit https://www.linaro.org/sig/hpc/
Enable IPv6 on Route53 AWS ELB, docker and node App
Enable IPv6 on Route53 AWS ELB, docker and node AppFyllo Fyllo Technical notes: Enable IPv6 support on AWS and docker machine.
We recently had to add IPv6 support to our APIs. Had to scratch head at multiple points. So, we want to help community with these notes. Happy building :)
Arm in HPC
Arm in HPCinside-BigData.com In this video from the Rice Oil & Gas Conference, Brent Gorda from ARM presents: ARM in HPC.
"With the recent Astra system at Sandia Lab (#203 on the Top500) and HPE Catalyst project in the UK, Arm-based architectures are arriving in HPC environments. Several partners have announced or will soon announce new silicon and projects, each of which offers something different and compelling for our community. Brent will describe the driving factors and how these solutions are changing the landscape for HPC."
Watch the video: https://wp.me/p3RLHQ-jXS
Learn more: https://developer.arm.com/hpc
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Viewers also liked (20)
Debugging & Tuning in Spark
Debugging & Tuning in SparkShiao-An Yuan This document provides tips and best practices for debugging and tuning Spark applications. It discusses Spark concepts like RDDs, transformations, actions, and the DAG execution model. It then gives recommendations for improving correctness, reducing overhead from parallelism, avoiding data skew, and tuning configurations like storage level, number of partitions, executor resources and joins. Common failures are analyzed along with their causes and fixes. Overall it emphasizes the importance of tuning partitioning, avoiding shuffles when possible, and using the right configurations to optimize Spark jobs.
Spark After Dark - LA Apache Spark Users Group - Feb 2015
Spark After Dark - LA Apache Spark Users Group - Feb 2015Chris Fregly Spark After Dark is a mock dating site that uses the latest Spark libraries including Spark SQL, BlinkDB, Tachyon, Spark Streaming, MLlib, and GraphX to generate high-quality dating recommendations for its members and blazing fast analytics for its operators.
We begin with brief overview of Spark, Spark Libraries, and Spark Use Cases. In addition, we'll discuss the modern day Lambda Architecture that combines real-time and batch processing into a single system. Lastly, we present best practices for monitoring and tuning a highly-available Spark and Spark Streaming cluster.
There will be many live demos covering everything from basic topics such as ETL and data ingestion to advanced topics such as streaming, sampling, approximations, machine learning, textual analysis, and graph processing.
Scaling Machine Learning to Billions of Parameters - Spark Summit 2016
Scaling Machine Learning to Billions of Parameters - Spark Summit 2016Badri Narayan Bhaskar This document summarizes scaling machine learning to billions of parameters using Spark and a parameter server architecture. It describes the requirements for supporting both batch and sequential optimization at web scale. It then outlines the Spark + Parameter server approach, leveraging Spark for distributed processing and the parameter server for synchronizing model updates. Examples of distributed L-BFGS and Word2Vec training are provided to illustrate batch and sequential optimization respectively using this architecture.
Foundations for Scaling ML in Apache Spark
Foundations for Scaling ML in Apache SparkDatabricks Apache Spark has become the most active open source Big Data project, and its Machine Learning library MLlib has seen rapid growth in usage. A critical aspect of MLlib and Spark is the ability to scale: the same code used on a laptop can scale to 100’s or 1000’s of machines. This talk will describe ongoing and future efforts to make MLlib even faster and more scalable by integrating with two key initiatives in Spark. The first is Catalyst, the query optimizer underlying DataFrames and Datasets. The second is Tungsten, the project for approaching bare-metal speeds in Spark via memory management, cache-awareness, and code generation. This talk will discuss the goals, the challenges, and the benefits for MLlib users and developers. More generally, we will reflect on the importance of integrating ML with the many other aspects of big data analysis.
Scaling Analytics with Apache Spark
Scaling Analytics with Apache SparkQuantUniversity Since its debut in 2010, Apache Spark has become one of the most popular Big Data technologies in the Apache open source ecosystem. In addition to enabling processing of large data sets through its distributed computing architecture, Spark provides out-of-the-box support for machine learning, streaming and graph processing in a single framework. Spark has been supported by companies like Microsoft, Google, Amazon and IBM and in financial services, companies like Blackrock (http://bit.ly/1Q1DVJH ) and Bloomberg (http://bit.ly/29LXbPv ) have started to integrate Apache Spark into their tool chain and the interest is growing. Unlike other big-data technologies which require intensive programming using Java etc., Spark enables data scientists to work with a big-data technology using higher level languages like Python and R making it accessible to conduct experiments and for rapid prototyping.
In this talk, we will introduce Apache Spark and discuss the key features that differentiate Apache Spark from other technologies. We will provide examples on how Apache Spark can help scale analytics and discuss how the machine learning API could be used to solve large-scale machine learning problems using Spark’s distributed computing framework. We will also illustrate enterprise use cases for scaling analytics with Apache Spark.
Spark performance tuning - Maksud Ibrahimov
Spark performance tuning - Maksud IbrahimovMaksud Ibrahimov The talk by Maksud Ibrahimov, Chief Data Scientist at InfoReady Analytics. He is going to share with us how to maximise the performance of Spark.
As a user of Apache Spark from very early releases, he generally sees that the framework is easy to start with but as the program grows its performance starts to suffer. In this talk Maksud will answer the following questions:
- How to reach higher level of parallelism of your jobs without scaling up your cluster?
- Understanding shuffles, and how to avoid disk spills
- How to identify task stragglers and data skews?
- How to identify Spark bottlenecks?
Machine learning at Scale with Apache Spark
Machine learning at Scale with Apache SparkMartin Zapletal This document discusses scaling machine learning using Apache Spark. It covers several key topics:
1) Parallelizing machine learning algorithms and neural networks to distribute computation across clusters. This includes data, model, and parameter server parallelism.
2) Apache Spark's Resilient Distributed Datasets (RDDs) programming model which allows distributing data and computation across a cluster in a fault-tolerant manner.
3) Examples of very large neural networks trained on clusters, such as a Google face detection model using 1,000 servers and a IBM brain-inspired chip model using 262,144 CPUs.
Spark 1.6 vs Spark 2.0
Spark 1.6 vs Spark 2.0Sigmoid By Abhishek Sharma
CONTENT:
-Apache Spark
-Project Tungsten
-Whole Stage Code Generation
-Vectorization
Big Data Day LA 2016 Keynote - Reynold Xin/ Databricks
Big Data Day LA 2016 Keynote - Reynold Xin/ DatabricksData Con LA This document discusses scaling big data using Apache Spark. It provides an overview of Spark's philosophy of providing a unified engine to support end-to-end applications using high-level APIs. It outlines some of the new features in Apache Spark 2.0, including improvements to structured APIs, structured streaming, and new deep learning and graph processing libraries. It also discusses initiatives by Databricks to grow the Spark community through massive open online courses and a free community edition of the Databricks platform.
Building, Debugging, and Tuning Spark Machine Leaning Pipelines-(Joseph Bradl...
Building, Debugging, and Tuning Spark Machine Leaning Pipelines-(Joseph Bradl...Spark Summit This document discusses Spark ML pipelines for machine learning workflows. It begins with an introduction to Spark MLlib and the various algorithms it supports. It then discusses how ML workflows can be complex, involving multiple data sources, feature transformations, and models. Spark ML pipelines allow specifying the entire workflow as a single pipeline object. This simplifies debugging, re-running on new data, and parameter tuning. The document provides an example text classification pipeline and demonstrates how data is transformed through each step via DataFrames. It concludes by discussing upcoming improvements to Spark ML pipelines.
Building Robust, Adaptive Streaming Apps with Spark Streaming
Building Robust, Adaptive Streaming Apps with Spark StreamingDatabricks As the adoption of Spark Streaming increases rapidly, the community has been asking for greater robustness and scalability from Spark Streaming applications in a wider range of operating environments. To fulfill these demands, we have steadily added a number of features in Spark Streaming. We have added backpressure mechanisms which allows Spark Streaming to dynamically adapt to changes in incoming data rates, and maintain stability of the application. In addition, we are extending Spark’s Dynamic Allocation to Spark Streaming, so that streaming applications can elastically scale based on processing requirements. In my talk, I am going to explore these mechanisms and explain how developers can write robust, scalable and adaptive streaming applications using them. Presented by Tathagata "TD" Das from Databricks.
How to Boost 100x Performance for Real World Application with Apache Spark-(G...
How to Boost 100x Performance for Real World Application with Apache Spark-(G...Spark Summit This document summarizes work done by an Intel software team in China to improve Apache Spark performance for real-world applications. It describes benchmarking tools like HiBench and profiling tools like HiMeter that were developed. It also discusses several case studies where the team worked with customers to optimize joins, manage memory usage, and reduce network bandwidth. The overall goal was to help solve common issues around ease of use, reliability, and scalability for Spark in production environments.
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014Chris Fregly Spark Streaming allows for processing of real-time data streams using Spark. The document discusses using Spark Streaming with Amazon Kinesis for streaming data ingestion. It covers the Spark Streaming and Kinesis integration architecture, how the Spark Kinesis receiver works, scaling considerations, and fault tolerance mechanisms through checkpointing. Examples of monitoring and tuning Spark Streaming jobs on Kinesis data are also provided.
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016Anya Bida by Anya Bida and Rachel Warren from Alpine Data
https://spark-summit.org/east-2016/events/spark-tuning-for-enterprise-system-administrators/
Spark offers the promise of speed, but many enterprises are reluctant to make the leap from Hadoop to Spark. Indeed, System Administrators will face many challenges with tuning Spark performance. This talk is a gentle introduction to Spark Tuning for the Enterprise System Administrator, based on experience assisting two enterprise companies running Spark in yarn-cluster mode. The initial challenges can be categorized in two FAQs. First, with so many Spark Tuning parameters, how do I know which parameters are important for which jobs? Second, once I know which Spark Tuning parameters I need, how do I enforce them for the various users submitting various jobs to my cluster? This introduction to Spark Tuning will enable enterprise system administrators to overcome common issues quickly and focus on more advanced Spark Tuning challenges. The audience will understand the “cheat-sheet” posted here: http://techsuppdiva.github.io/ Key takeaways: FAQ 1: With so many Spark Tuning parameters, how do I know which parameters are important for which jobs? Solution 1: The Spark Tuning cheat-sheet! A visualization that guides the System Administrator to quickly overcome the most common hurdles to algorithm deployment. [1]http://techsuppdiva.github.io/ FAQ 2: Once I know which Spark Tuning parameters I need, how do I enforce them at the user level? job level? algorithm level? project level? cluster level? Solution 2: We’ll approach these challenges using job & cluster configuration, the Spark context, and 3rd party tools – of which Alpine will be one example. We’ll operationalize Spark parameters according to user, job, algorithm, workflow pipeline, or cluster levels.
Spark 2.0 What's Next （Hadoop / Spark Conference Japan 2016 キーノート講演資料）
Spark 2.0 What's Next （Hadoop / Spark Conference Japan 2016 キーノート講演資料）Hadoop / Spark Conference Japan Hadoop / Spark Conference Japan 2016 キーノート講演資料
『Spark 2.0 What's Next』
Reynold Xin (databricks)
▼イベントページ
http://hadoop.apache.jp/hcj2016-program/
http://hcj2016.eventbrite.com/
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...Spark Summit Apache Spark MLlib provides scalable implementation of popular machine learning algorithms, which lets users train models from big dataset and iterate fast. The existing implementations assume that the number of parameters is small enough to fit in the memory of a single machine. However, many applications require solving problems with billions of parameters on a huge amount of data such as Ads CTR prediction and deep neural network. This requirement far exceeds the capacity of exisiting MLlib algorithms many of who use L-BFGS as the underlying solver. In order to fill this gap, we developed Vector-free L-BFGS for MLlib. It can solve optimization problems with billions of parameters in the Spark SQL framework where the training data are often generated. The algorithm scales very well and enables a variety of MLlib algorithms to handle a massive number of parameters over large datasets. In this talk, we will illustrate the power of Vector-free L-BFGS via logistic regression with real-world dataset and requirement. We will also discuss how this approach could be applied to other ML algorithms.
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...
Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...Spark Summit Spark 2.0 provided strong performance enhancements to the Spark core while advancing Spark ML usability to use data frames. But what happens when you run Spark 2.0 machine learning algorithms on a large cluster with a very large data set? Do you even get any benefit from using a very large data set? It depends. How do new hardware advances affect the topology of high performance Spark clusters. In this talk we will explore Spark 2.0 Machine Learning at scale and share our findings with the community.
As our test platform we will be using a new cluster design, different from typical Hadoop clusters, with more cores, more RAM and latest generation NVMe SSD’s and a 100GbE network with a goal of more performance, in a more space and energy efficient footprint.
Making Structured Streaming Ready for Production
Making Structured Streaming Ready for ProductionDatabricks In mid-2016, we introduced Structured Steaming, a new stream processing engine built on Spark SQL that revolutionized how developers can write stream processing application without having to reason about having to reason about streaming. It allows the user to express their streaming computations the same way you would express a batch computation on static data. The Spark SQL engine takes care of running it incrementally and continuously updating the final result as streaming data continues to arrive. It truly unifies batch, streaming and interactive processing in the same Datasets/DataFrames API and the same optimized Spark SQL processing engine.
The initial alpha release of Structured Streaming in Apache Spark 2.0 introduced the basic aggregation APIs and files as streaming source and sink. Since then, we have put in a lot of work to make it ready for production use. In this talk, Tathagata Das will cover in more detail about the major features we have added, the recipes for using them in production, and the exciting new features we have plans for in future releases. Some of these features are as follows:
- Design and use of the Kafka Source
- Support for watermarks and event-time processing
- Support for more operations and output modes
Speaker: Tathagata Das
This talk was originally presented at Spark Summit East 2017.
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson Scala Days, Amsterdam, 2015: Lambda Architecture - Batch and Streaming with Spark, Cassandra, Kafka, Akka and Scala; Fault Tolerance, Data Pipelines, Data Flows, Data Locality, Akka Actors, Spark, Spark Cassandra Connector, Big Data, Asynchronous data flows. Time series data, KillrWeather, Scalable Infrastructure, Partition For Scale, Replicate For Resiliency, Parallelism
Isolation, Data Locality, Location Transparency
Apache Spark MLlib 2.0 Preview: Data Science and Production
Apache Spark MLlib 2.0 Preview: Data Science and ProductionDatabricks This talk highlights major improvements in Machine Learning (ML) targeted for Apache Spark 2.0. The MLlib 2.0 release focuses on ease of use for data science—both for casual and power users. We will discuss 3 key improvements: persisting models for production, customizing Pipelines, and improvements to models and APIs critical to data science.
(1) MLlib simplifies moving ML models to production by adding full support for model and Pipeline persistence. Individual models—and entire Pipelines including feature transformations—can be built on one Spark deployment, saved, and loaded onto other Spark deployments for production and serving.
(2) Users will find it much easier to implement custom feature transformers and models. Abstractions automatically handle input schema validation, as well as persistence for saving and loading models.
(3) For statisticians and data scientists, MLlib has doubled down on Generalized Linear Models (GLMs), which are key algorithms for many use cases. MLlib now supports more GLM families and link functions, handles corner cases more gracefully, and provides more model statistics. Also, expanded language APIs allow data scientists using Python and R to call many more algorithms.
Finally, we will demonstrate these improvements live and show how they facilitate getting started with ML on Spark, customizing implementations, and moving to production.
Spark 2.0 What's Next （Hadoop / Spark Conference Japan 2016 キーノート講演資料）
Spark 2.0 What's Next （Hadoop / Spark Conference Japan 2016 キーノート講演資料）Hadoop / Spark Conference Japan
Ad
Similar to Scaling spark (20)
Introduction to Apache Hivemall v0.5.0
Introduction to Apache Hivemall v0.5.0Makoto Yui Apache Hivemall is a scalable machine learning library for Apache Hive, Apache Spark, and Apache Pig.
Hivemall provides a number of machine learning functionalities across classification, regression, ensemble learning, and feature engineering through UDFs/UDAFs/UDTFs of Hive.
We have released the first Apache release (v0.5.0-incubating) on Mar 5, 2018 and the project plans to release v0.5.2 in Q2, 2018.
We will first give a quick walk-through of features, usages, what's new in v0.5.0, and future roadmaps of Apache Hivemall. Next, we will introduce Hivemall on Apache Spark in depth such as DataFrame integration and Spark 2.3 supports in Hivemall.
Ceph
CephHien Nguyen Van This document summarizes a presentation about Ceph, an open-source distributed storage system. It discusses Ceph's introduction and components, benchmarks Ceph's block and object storage performance on Intel architecture, and describes optimizations like cache tiering and erasure coding. It also outlines Intel's product portfolio in supporting Ceph through optimized CPUs, flash storage, networking, server boards, software libraries, and contributions to the open source Ceph community.
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas Après la petite intro sur le stockage distribué et la description de Ceph, Jian Zhang réalise dans cette présentation quelques benchmarks intéressants : tests séquentiels, tests random et surtout comparaison des résultats avant et après optimisations. Les paramètres de configuration touchés et optimisations (Large page numbers, Omap data sur un disque séparé, ...) apportent au minimum 2x de perf en plus.
AMD It's Time to ROC
AMD It's Time to ROCinside-BigData.com In this video from the HPC User Forum in Tucson, Gregory Stoner from AMD presents: It's Time to ROC.
"With the announcement of the Boltzmann Initiative and the recent releases of ROCK and ROCR, AMD has ushered in a new era of Heterogeneous Computing. The Boltzmann initiative exposes cutting edge compute capabilities and features on targeted AMD/ATI Radeon discrete GPUs through an open source software stack. The Boltzmann stack is comprised of several components based on open standards, but extended so important hardware capabilities are not hidden by the implementation."
Learn more: http://gpuopen.com/getting-started-with-boltzmann-components-platforms-installation/
and
http://hpcuserforum.com
Watch the video presentation: http://wp.me/p3RLHQ-fcJ
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Parallelism Processor Design
Parallelism Processor DesignSri Prasanna The document discusses parallel computing and multicore processors. It notes that Berkeley researchers believe multicore is the future of computing. It also discusses building an academic "manycore" research system using FPGAs to allow researchers to experiment with parallel algorithms, compilers, and programming models on thousands of processor cores. This would help drive innovation and avoid long waits between hardware and software iterations.
Red Hat Storage Day New York - New Reference Architectures
Red Hat Storage Day New York - New Reference ArchitecturesRed_Hat_Storage The document provides an overview and summary of Red Hat's reference architecture work including MySQL and Hadoop, software-defined NAS, and digital media repositories. It discusses trends toward disaggregating Hadoop compute and storage and various data flow options. It also summarizes performance testing Red Hat conducted comparing AWS EBS and Ceph for MySQL workloads, and analyzing factors like IOPS/GB ratios, core-to-flash ratios, and pricing. Server categories and vendor examples are defined. Comparisons of throughput and costs at scale between software-defined scale-out storage and traditional enterprise NAS solutions are also presented.
Copr HD OpenStack Day India
Copr HD OpenStack Day Indiaopenstackindia The document discusses CoprHD, an open source software-defined storage controller that automates storage provisioning across heterogeneous storage infrastructure. It summarizes CoprHD's key capabilities in automating storage lifecycle management and integrating with cloud stacks like OpenStack. The document also provides an overview of CoprHD architecture and describes how CoprHD can operate as a Cinder driver within OpenStack. It outlines CoprHD's interoperability with OpenStack through different integration methods and concludes with information on the CoprHD community.
CETH for XDP [Linux Meetup Santa Clara | July 2016] ![CETH for XDP [Linux Meetup Santa Clara | July 2016]](https://cdn.slidesharecdn.com/ss_thumbnails/ceth5overview1-160801192921-thumbnail.jpg?width=560&fit=bounds)
CETH for XDP [Linux Meetup Santa Clara | July 2016] IO Visor Project This document discusses CETH (Common Ethernet Driver Framework), which aims to improve kernel networking performance for virtualization. CETH simplifies NIC drivers by consolidating common functions. It supports various NICs and accelerators. CETH features efficient memory and buffer management, flexible TX/RX scheduling, and a customizable metadata structure. It is being simplified to work with XDP for even higher performance network I/O processing in the kernel. Next steps include further optimizations and measuring performance gains when using CETH with XDP and virtualized environments.
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...
Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...HostedbyConfluent Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Databricks In this session, the speakers will discuss their experiences porting Apache Spark to the Cray XC family of supercomputers. One scalability bottleneck is in handling the global file system present in all large-scale HPC installations. Using two techniques (file open pooling, and mounting the Spark file hierarchy in a specific manner), they were able to improve scalability from O(100) cores to O(10,000) cores. This is the first result at such a large scale on HPC systems, and it had a transformative impact on research, enabling their colleagues to run on 50,000 cores.
With this baseline performance fixed, they will then discuss the impact of the storage hierarchy and of the network on Spark performance. They will contrast a Cray system with two levels of storage with a “data intensive” system with fast local SSDs. The Cray contains a back-end global file system and a mid-tier fast SSD storage. One conclusion is that local SSDs are not needed for good performance on a very broad workload, including spark-perf, TeraSort, genomics, etc.
They will also provide a detailed analysis of the impact of latency of file and network I/O operations on Spark scalability. This analysis is very useful to both system procurements and Spark core developers. By examining the mean/median value in conjunction with variability, one can infer the expected scalability on a given system. For example, the Cray mid-tier storage has been marketed as the magic bullet for data intensive applications. Initially, it did improve scalability and end-to-end performance. After understanding and eliminating variability in I/O operations, they were able to outperform any configurations involving mid-tier storage by using the back-end file system directly. They will also discuss the impact of network performance and contrast results on the Cray Aries HPC network with results on InfiniBand.
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...Databricks In this session, the speakers will discuss their experiences porting Apache Spark to the Cray XC family of supercomputers. One scalability bottleneck is in handling the global file system present in all large-scale HPC installations. Using two techniques (file open pooling, and mounting the Spark file hierarchy in a specific manner), they were able to improve scalability from O(100) cores to O(10,000) cores. This is the first result at such a large scale on HPC systems, and it had a transformative impact on research, enabling their colleagues to run on 50,000 cores.
With this baseline performance fixed, they will then discuss the impact of the storage hierarchy and of the network on Spark performance. They will contrast a Cray system with two levels of storage with a “data intensive” system with fast local SSDs. The Cray contains a back-end global file system and a mid-tier fast SSD storage. One conclusion is that local SSDs are not needed for good performance on a very broad workload, including spark-perf, TeraSort, genomics, etc.
They will also provide a detailed analysis of the impact of latency of file and network I/O operations on Spark scalability. This analysis is very useful to both system procurements and Spark core developers. By examining the mean/median value in conjunction with variability, one can infer the expected scalability on a given system. For example, the Cray mid-tier storage has been marketed as the magic bullet for data intensive applications. Initially, it did improve scalability and end-to-end performance. After understanding and eliminating variability in I/O operations, they were able to outperform any configurations involving mid-tier storage by using the back-end file system directly. They will also discuss the impact of network performance and contrast results on the Cray Aries HPC network with results on InfiniBand.
Hadoop Performance Optimization at Scale, Lessons Learned at Twitter
Hadoop Performance Optimization at Scale, Lessons Learned at TwitterDataWorks Summit - Profiling Hadoop jobs at Twitter revealed that compression/decompression of intermediate data and deserialization of complex object keys were very expensive. Optimizing these led to performance improvements of 1.5x or more.
- Using columnar file formats like Apache Parquet allows reading only needed columns, avoiding deserialization of unused data. This led to gains of up to 3x.
- Scala macros were developed to generate optimized implementations of Hadoop's RawComparator for common data types, avoiding deserialization for sorting.
Accelerating EDA workloads on Azure – Best Practice and benchmark on Intel EM...
Accelerating EDA workloads on Azure – Best Practice and benchmark on Intel EM...Meng-Ru (Raymond) Tsai Benchmark EDA workloads on Azure Intel Emerald Rapids (EMR) VMs.
The article evaluates the performance of the latest Azure VMs using the 5th Gen Intel® Xeon® Platinum 8537C (Emerald Rapids) processor by comparing them to the previous Ice Lake generation. Using two EDA tools, Cadence Spectre-X and Synopsys VCS, the benchmarks involve real-world scenarios including single-threaded, multi-threaded, and multiple jobs running on one node.
Results show that Spectre-X performs 12 to 18% better on D64ds v6 instances and 22 to 29% better on FX64v2 instances compared to D64ds v5 instances. The D64ds v6 instances were found to be more cost-effective, while FX64mds v2 instances achieved the shortest total runtime. For Synopsys VCS, the benchmarks revealed a speedup of 17 to 43% for Emerald Rapids instances over Ice Lake instances across various parallel simulations. The findings offer EDA customers options on which Azure EMR instances to select based on the cost-efficiency analysis.
The Streaming Data Lake - What Do KIP-405 and KIP-833 Mean for Your Larger Da...
The Streaming Data Lake - What Do KIP-405 and KIP-833 Mean for Your Larger Da...HostedbyConfluent "KIP-405 and KIP-833 stand to make a big impact, not just in the world of event streaming but on the larger data infrastructure of your business too. Kafka no longer just powers the operational side of business but is making significant inroads into the analytical side too. Many are realising that a tiered storage enabled Kafka (containing the huge variety of data enabled by Kraft), shows many of the same characteristics as a traditional datalake and can be used to serve requests for raw historical data as well as linking services and applications.
In this talk we will explore the efficiencies that can be gained by using Kafka for this role. Kafka as a datalake can greatly simplify data infrastructure, improve consistency between operational and analytical data, dramatically reduce costs and provide advantages in terms of the freshness and variety of data available to analysts. We will also cover the critical features that are missing to make this possible and ways in which these may be achieved in today and future.
Join us for an exciting look into the future where we expand the role of Kafka in the data infrastructure and challenge the ETL/ELT status quo."
Avoiding Chaos: Methodology for Managing Performance in a Shared Storage A...
Avoiding Chaos: Methodology for Managing Performance in a Shared Storage A...brettallison Scope - The primary focus of this presentation is on the methodology we use for managing performance in a very large shared Storage Area Network environment with a Primary focus on Distributed Systems and IBM Enterprise Storage Server. The focus on this presentation is methodology and NOT measurement. There are numerous excellent presentations already out there on measurement. However, there are several references in the back of the presentation to measurement tools.
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...
Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Alex Levenson My talk at Hadoop Summit 2015
Video can be found here: https://www.youtube.com/watch?v=jupArYWxoq0
MARC ONERA Toulouse2012 Altreonic
MARC ONERA Toulouse2012 AltreonicEric Verhulst Altreonic was spun off in 2008 from Eonic Systems to focus on real-time operating systems using formal techniques. Their OpenComRTOS is a small, network-centric real-time OS that uses CSP concurrency and can scale from 1 to over 10,000 nodes. It provides priority-based communication and fault tolerance and has been implemented on many heterogeneous platforms from DSPs to many-core systems.
Ultimate journey towards realtime data platform with 2.5M events per sec
Ultimate journey towards realtime data platform with 2.5M events per secb0ris_1 The document discusses the journey towards building a real-time data platform capable of handling 2.5 million events per second. It describes migrating Spark processing from on-premises CDH to AWS EMR to improve scalability. Fault tolerance was added through batch processing in Spark and auto-recovery capabilities. Backpressure was enabled through Spark streaming, HDFS, and pulling data into Vertica to prevent overloading downstream systems. Monitoring was enhanced with a separate application to track pipeline metrics. The final platform achieved the performance goals through these architectural changes.
Roadrunner Tutorial: An Introduction to Roadrunner and the Cell Processor
Roadrunner Tutorial: An Introduction to Roadrunner and the Cell ProcessorSlide_N Roadrunner Tutorial: An Introduction to Roadrunner and the Cell Processor
Redefining Data Redundancywith RAID Offload
Redefining Data Redundancywith RAID OffloadDerze •
Storage Services Evolution
•
Data Redundancy Compute Challenges
•
Offload to SSD
•
Why xPUShould Leverage SSD Offload
Accelerating EDA workloads on Azure – Best Practice and benchmark on Intel EM...
Accelerating EDA workloads on Azure – Best Practice and benchmark on Intel EM...Meng-Ru (Raymond) Tsai
Ad
Recently uploaded (20)
computer organization and assembly language.docx
computer organization and assembly language.docxalisoftwareengineer1 computer organization and assembly language : its about types of programming language along with variable and array description..https://www.nfciet.edu.pk/
How iCode cybertech Helped Me Recover My Lost Funds
How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345 I was devastated when I realized that I had fallen victim to an online fraud, losing a significant amount of money in the process. After countless hours of searching for a solution, I came across iCode cybertech. From the moment I reached out to their team, I felt a sense of hope that I can recommend iCode Cybertech enough for anyone who has faced similar challenges. Their commitment to helping clients and their exceptional service truly set them apart. Thank you, iCode cybertech, for turning my situation around!
[email protected]
Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...
Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...Abodahab IHOY78T6R5E45TRYTUYIU
AI Competitor Analysis: How to Monitor and Outperform Your Competitors
AI Competitor Analysis: How to Monitor and Outperform Your CompetitorsContify AI competitor analysis helps businesses watch and understand what their competitors are doing. Using smart competitor intelligence tools, you can track their moves, learn from their strategies, and find ways to do better. Stay smart, act fast, and grow your business with the power of AI insights.
For more information please visit here https://www.contify.com/
Developing Security Orchestration, Automation, and Response Applications
Developing Security Orchestration, Automation, and Response ApplicationsVICTOR MAESTRE RAMIREZ Developing Security Orchestration, Automation, and Response Applications
FPET_Implementation_2_MA to 360 Engage Direct.pptx
FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d Engage Direct 360 marketing optimization sas entreprise miner
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS Prediction of Aluminium wire rod physical properties through AI, ML
or any modern technique for better productivity and quality control.
Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptx
Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptxPareaRusan planning and calculation monitoring project
Just-In-Timeasdfffffffghhhhhhhhhhj Systems.ppt
Just-In-Timeasdfffffffghhhhhhhhhhj Systems.pptssuser5f8f49 Just-in-time: Repetitive production system in which processing and movement of materials and goods occur just as they are needed, usually in small batches
JIT is characteristic of lean production systems
JIT operates with very little “fat”
Calories_Prediction_using_Linear_Regression.pptx
Calories_Prediction_using_Linear_Regression.pptxTijiLMAHESHWARI Calorie prediction using machine learning
Template_A3nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
Template_A3nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnncegiver630 Telangana State, India’s newest state that was carved from the erstwhile state of Andhra
Pradesh in 2014 has launched the Water Grid Scheme named as ‘Mission Bhagiratha (MB)’
to seek a permanent and sustainable solution to the drinking water problem in the state. MB is
designed to provide potable drinking water to every household in their premises through
piped water supply (PWS) by 2018. The vision of the project is to ensure safe and sustainable
piped drinking water supply from surface water sources
Scaling spark
- 2. ABOUT US
- 3. Alex Rovner, Director of Data Engineering Media Platform Processing Terabytes Daily
- 4. PRIOR STATE TWO CLUSTERS CORE & ANALYTICS BOTH IN COLO
- 5. CHALLENGES
- 7. SPARK
- 9. AWS
- 11. INSTANCES
- 12. D2.8XLARGE 48TB OF EPHEMERAL STORAGE 244 GB RAM 38 V-CPU INSTANCES
- 15. INSTANCES SPOT?
- 17. INSTANCES WAIT, WHAT ABOUT DATA LOCALITY?
- 18. HADOOP
- 20. HADOOP CDH 5.3 SPARK 1.2 ON YARN
- 21. HADOOP CDH 5.4 SPARK 1.3 ON YARN
- 22. HADOOP CDH 5.4 SPARK 1.5 ON YARN
- 23. HADOOP RUN THE LATEST VERSION! TECH.MAGNETIC.COM
- 24. AUTO SCALE
- 25. CALCULATE CLUSTER UTILIZATION QUERY CM API V-CORES AVAILABLE, USED & PENDING AUTO SCALE
- 26. CALCULATE TARGET CAPACITY TARGET 80% UTILIZATION LIMIT DOWNSIZING AUTO SCALE
- 28. SPEED BUMPS
- 29. SPEED BUMPS APPLICATION MASTER ON SPOT YARN LABELS
- 30. SPEED BUMPS USERS ARE IMPATIENT ITS NEVER ENOUGH
- 31. SPEED BUMPS I AM LOST! YARN LOGS SET YARN OVERHEAD CHECK GC TIME INCREASE EXECUTOR MEMORY TRY AGAIN
- 32. SPEED BUMPS BROADCASTING IS EVIL
- 33. SPEED BUMPS BROADCASTING “LARGE” DATASETS IS EVIL
- 34. CURRENT STATE THREE CLUSTERS ANALYTICS & STREAMING (AWS) CORE (COLO - MOVING SOON!)
- 35. BIG SUCCESS!
- 36. QUESTIONS?
Editor's Notes
- #7: Scalability - As our volumes increased, we were struggling to keep up. Looking for a more performant framework Elasticity - Team’s workloads vary, so do seasonal volumes Agility - Colo terms for hardware were not flexible. Multi year commitments, long turnaround times for getting new hardware.
- #9: Promises 100x improvement over M/R Existing code in PIG Python UDFs
- #13: Lucky brea
- #20: EMR easy to get started with, additional cost when compared to EC2 Existing expertise with CDH