Apache Spark on Kubernetes
Download as PPTX, PDF1 like443 views
How we can make use of Kubernetes as Resource Manager for Spark. What are the Pros and Cons of Spark Resource manager are discussed on this slides and the associated tutorial. Refer this github project for more details and code samples : https://github.com/haridas/hadoop-env
1 of 14
Download to read offline
Ad
Recommended
Bigdata and Hadoop with Docker
Bigdata and Hadoop with Dockerharidasnss This document provides an introduction to big data and Hadoop. It discusses how distributed systems can scale to handle large data volumes and discusses Hadoop's architecture. It also provides instructions on setting up a Hadoop cluster on a laptop and summarizes Hadoop's MapReduce programming model and YARN framework. Finally, it announces an upcoming workshop on Spark and Pyspark.
Apache spark on Hadoop Yarn Resource Manager
Apache spark on Hadoop Yarn Resource Managerharidasnss How we can configure the spark on apache hadoop environment, and why we need that compared to standalone cluster manager.
Slide also includes docker based demo to play with the hadoop and spark on your laptop itself. See more on the demo codes and other documentation here - https://github.com/haridas/hadoop-env
SF Big Analytics_20190612: Scaling Apache Spark on Kubernetes at Lyft
SF Big Analytics_20190612: Scaling Apache Spark on Kubernetes at LyftChester Chen Talk 1. Scaling Apache Spark on Kubernetes at Lyft
As part of this mission Lyft invests heavily in open source infrastructure and tooling. At Lyft Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark at Lyft has evolved to solve both Machine Learning and large scale ETL workloads. By combining the flexibility of Kubernetes with the data processing power of Apache Spark, Lyft is able to drive ETL data processing to a different level. In this talk, We will talk about challenges the Lyft team faced and solutions they developed to support Apache Spark on Kubernetes in production and at scale. Topics Include: - Key traits of Apache Spark on Kubernetes. - Deep dive into Lyft's multi-cluster setup and operationality to handle petabytes of production data. - How Lyft extends and enhances Apache Spark to support capabilities such as Spark pod life cycle metrics and state management, resource prioritization, and queuing and throttling. - Dynamic job scale estimation and runtime dynamic job configuration. - How Lyft powers internal Data Scientists, Business Analysts, and Data Engineers via a multi-cluster setup.
Speaker: Li Gao
Li Gao is the tech lead in the cloud native spark compute initiative at Lyft. Prior to Lyft, Li worked at Salesforce, Fitbit, Marin Software, and a few startups etc. on various technical leadership positions on cloud native and hybrid cloud data platforms at scale. Besides Spark, Li has scaled and productionized other open source projects, such as Presto, Apache HBase, Apache Phoenix, Apache Kafka, Apache Airflow, Apache Hive, and Apache Cassandra.
DIscover Spark and Spark streaming
DIscover Spark and Spark streamingMaturin BADO This document provides an overview of Spark, including how it differs from MapReduce by loading more data into memory and implementing caching mechanisms. It discusses Spark's ecosystem, features like RDDs, and how it can run on Hadoop clusters. It also summarizes Spark Streaming, describing how it processes data in micro-batches, provides windowing and transformation operations on DStreams, and unifies batch, streaming, and interactive analytics.
A Collaborative Data Science Development Workflow
A Collaborative Data Science Development WorkflowDatabricks Collaborative data science workflows have several moving parts, and many organizations struggle with developing an efficient and scalable process. Our solution consists of data scientists individually building and testing Kedro pipelines and measuring performance using MLflow tracking. Once a strong solution is created, the candidate pipeline is trained on cloud-agnostic, GPU-enabled containers. If this pipeline is production worthy, the resulting model is served to a production application through MLflow.
An Overview of Apache Spark
An Overview of Apache SparkYasoda Jayaweera This lecture was intended to provide an introduction to Apache Spark's features and functionality and importance of Spark as a distributed data processing framework compared to Hadoop MapReduce. The target audience was MSc students with programming skills at beginner to intermediate level.
Spark-on-Yarn: The Road Ahead-(Marcelo Vanzin, Cloudera)
Spark-on-Yarn: The Road Ahead-(Marcelo Vanzin, Cloudera)Spark Summit Spark on YARN provides resource management and security features through YARN, but still has areas for improvement. Dynamic allocation in YARN allows Spark applications to grow and shrink executors based on task demand, though latency and data locality could be enhanced. Security supports Kerberos authentication and delegation tokens, but long-lived applications face token expiration issues and encryption needs improvement for control plane, shuffle files, and user interfaces. Overall, usability, security, and performance remain areas of focus.
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan Volz
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan VolzDatabricks This document discusses using Hadoop for archiving, e-discovery, and supervision. It outlines the key components of each task and highlights traditional shortcomings. Hadoop provides strengths like speed, ease of use, and security. An architectural overview shows how Hadoop can be used for ingestion, processing, analysis, and machine learning. Examples demonstrate surveillance use cases. While some obstacles remain, partners can help address areas like user interfaces and compliance storage.
#GeodeSummit - Redis to Geode Adaptor
#GeodeSummit - Redis to Geode AdaptorPivotalOpenSourceHub This document discusses implementing a Redis adaptor using Apache Geode. It provides an overview of Redis data structures and commands, describes how Geode partitioned regions and indexes can be used to store and access Redis data, outlines advantages like scalability and high availability, and presents a roadmap for further development including supporting additional commands and performance optimization.
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedis Labs The Home Depot is transforming its architecture to use microservices and polyglot persistence to handle increasing online order volumes of 250,000 lines per hour. Redis is being used to turbo charge existing monolithic applications by offloading pieces to new processes using patterns like caching, concurrency management, and powering algorithms. This improves performance by reducing database degradation and wait times by over 95%. Next steps include setting up Redis clusters on-premises and off-premises to further reduce database CPU usage and onboard more patterns.
Apache Cassandra Lunch #72: Databricks and Cassandra
Apache Cassandra Lunch #72: Databricks and CassandraAnant Corporation In Cassandra Lunch #72, we will discuss how we can use Databricks with Cassandra.
Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-72-databricks-and-cassandra
Accompanying YouTube: https://youtu.be/5zCN27KHADo
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Cassandra.Lunch:
https://github.com/Anant/Cassandra.Lunch
Email:
[email protected]
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataMichael Stack This document discusses Apache Kylin, an OLAP engine for big data. It provides 3 key points:
1. Apache Kylin is designed to provide fast, interactive queries on large datasets stored in Hadoop. It uses pre-calculated cube structures stored in HBase to enable sub-second query performance on trillion row datasets.
2. HBase was selected as the storage engine because it is natively integrated with Hadoop, supports high throughput and low-latency queries, and can store very large datasets. Kylin leverages HBase's capabilities for cube storage, metadata storage, online calculation pushing, and caching.
3. Apache Kylin is used by over 1000 global companies across
Riak at shareaholic
Riak at shareaholicfreerobby Robby Grossman presented on Shareaholic's transition from MongoDB to Riak. Shareaholic needed a database with linear scalability, full-text search, and flexible indexing to support their growing product. They evaluated HBase, Cassandra, and Riak. Riak was chosen for its operational simplicity, linear scalability, integrated search, and secondary indices. Shareaholic migrated their data from MongoDB to Riak without downtime by writing to both databases simultaneously and verifying data integrity before decommissioning MongoDB. Riak has succeeded for Shareaholic's MapReduce queries, full text search, and publisher analytics use cases. Benchmarking showed vertical scaling on EC2 provides better latency than horizontal scaling.
HBaseConAsia2018 Track3-2: HBase at China Telecom
HBaseConAsia2018 Track3-2: HBase at China TelecomMichael Stack HBase is used at China Telecom for various applications including persistence for streaming jobs, online reading and writing, and as a data store for their core system. They operate several HBase clusters storing over 500 TB of data ingesting 1 TB per day. They monitor HBase using Ganglia for basic metrics and Zabbix for critical alerts. When issues arise, such as a system hang, they investigate debug cases and perform optimizations like changing the garbage collector from CMS to G1 and implementing read/write splitting.
Zabbix at scale with Elasticsearch
Zabbix at scale with ElasticsearchLeandro Totino Pereira Zabbix was experiencing performance issues due to large history tables in the database. To address this, the architecture was changed to store history data in Elasticsearch instead of database tables. This improved scalability and performance. The basic item and event data remained in the MariaDB database cluster. Zabbix proxies were also used to distribute load across multiple network segments. With this new architecture, history data is indexed in Elasticsearch without database tables, improving query speed and reducing database size.
Scylla Summit 2016: Graph Processing with Titan and Scylla
Scylla Summit 2016: Graph Processing with Titan and ScyllaScyllaDB Jason Plurad, Software Engineer, IBM Open Technology
PMC and Committer, Apache TinkerPop, on graph landscape, graph computing, and Titan and Scylla
Kafka website activity architecture
Kafka website activity architectureOmid Vahdaty Using Kafka in a website activity use-case. for disaster recovery, multi DC clusters, and multi cloud clusters.
Spark Core
Spark CoreTodd McGrath When learning Apache Spark, where should a person begin? What are the key fundamentals when learning Apache Spark? Resilient Distributed Datasets, Spark Drivers and Context, Transformations, Actions.
Running Scylla on Kubernetes with Scylla Operator
Running Scylla on Kubernetes with Scylla OperatorScyllaDB - The document discusses running Scylla, a NoSQL database, on Kubernetes using the Scylla Operator. The Operator allows Kubernetes to leverage for workload management and provides a management layer for Scylla.
- A demo shows deploying a Scylla cluster on Kubernetes with the Operator, stress testing the deployment, and performing common procedures like scaling up and upgrading Scylla versions.
- The Operator uses custom resources and controllers to map Scylla concepts like members, clusters, and datacenters to Kubernetes concepts like statefulsets and pods. This provides capabilities like topology changes and rolling upgrades.
Beyond Relational
Beyond RelationalLynn Langit The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.
Tachyon and Apache Spark
Tachyon and Apache Sparkrhatr Meet Tachyon: an exciting new project trying to unify in-memory computing and provide a universal data-exchange layer for Hadoop ecosystem projects.
Building a REST API with Cassandra on Datastax Astra Using Python and Node
Building a REST API with Cassandra on Datastax Astra Using Python and NodeAnant Corporation DataStax Astra provides the ability to develop and deploy data-driven applications with a cloud-native service, without the hassles of database and infrastructure administration. In this webinar, we are going to walk you through creating a REST API and exposing that to your Cassandra database.
Webinar Link: https://www.youtube.com/watch?v=O64pJa3eLqs&t=20s
Real time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.jsBen Laird This document discusses building a dynamic visualization of large streaming transaction data. It proposes using Apache Kafka to handle the transaction stream, Apache Spark Streaming to process and aggregate the data, MongoDB for intermediate storage, a Node.js server, and Socket.io for real-time updates. Visualization would use Crossfilter, DC.js and D3.js to enable interactive exploration of billions of records in the browser.
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...DataStax Academy The state of analytics has changed dramatically over the last few years. Hadoop is now commonplace, and the ecosystem has evolved to include new tools such as Spark, Shark, and Drill, that live alongside the old MapReduce-based standards. It can be difficult to keep up with the pace of change, and newcomers are left with a dizzying variety of seemingly similar choices. This is compounded by the number of possible deployment permutations, which can cause all but the most determined to simply stick with the tried and true. But there are serious advantages to many of the new tools, and this presentation will give an analysis of the current state–including pros and cons as well as what’s needed to bootstrap and operate the various options.
About Robbie Strickland, Software Development Manager at The Weather Channel
Robbie works for The Weather Channel’s digital division as part of the team that builds backend services for weather.com and the TWC mobile apps. He has been involved in the Cassandra project since 2010 and has contributed in a variety of ways over the years; this includes work on drivers for Scala and C#, the Hadoop integration, heading up the Atlanta Cassandra Users Group, and answering lots of Stack Overflow questions.
Spark Streaming and MLlib - Hyderabad Spark Group
Spark Streaming and MLlib - Hyderabad Spark GroupPhaneendra Chiruvella - A brief introduction to Spark Core
- Introduction to Spark Streaming
- A Demo of Streaming by evaluation top hashtags being used
- Introduction to Spark MLlib
- A Demo of MLlib by building a simple movie recommendation engine
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark MetricsDatabricks Luca Canali, a data engineer at CERN, presented on performance troubleshooting using Apache Spark metrics at the UnifiedDataAnalytics #SparkAISummit. CERN runs large Hadoop and Spark clusters to process over 300 PB of data from the Large Hadron Collider experiments. Luca discussed how to gather, analyze, and visualize Spark metrics to identify bottlenecks and improve performance.
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack This document summarizes a presentation on scaling a 30 TB data lake using Apache HBase and Scala. It introduces Apache HBase and Spark as technologies for building fast data platforms. It then describes a case study where they were used to architect a retail analytics platform capable of processing 4.6 billion events weekly from 30 TB of historical data. Key aspects included using HBase for data deduplication and as a master data management system, and connecting Spark to HBase using a Scala DSL for efficient querying and updates at scale. Performance was improved 5x by reengineering the data pipeline to be highly concurrent and asynchronous.
Apache Spark Acceleration Using Hardware Resources in the Cloud, Seamlessl wi...
Apache Spark Acceleration Using Hardware Resources in the Cloud, Seamlessl wi...Databricks InAccel provides high performance accelerators for your application based on novel hardware reconfigurable engines as IP blocks. The hardware accelerators can be deployed in the cloud, like Amazon AWS, using the f1 accelerators and can be integrated to widely used frameworks like Spark and PostgreSQL. The main novelty is that the users do not need to change the original code as the accelerators are deployed as software packages.
In this talk we will show how machine learning applications (e.g. logistic regression and K-means), based on Spark, can be accelerated by 3x-10x using hardware accelerators that are deployed in the Amazon AWS using f1 accelerators without any changes on the Spark code. InAccel provides all the required APIs for the integration on Spark using Java, Python or Scala. The utilization of hardware accelerators can also be used to reduce the OpEx as less resources and less time is required for the processing of the data.
Module01
Module01NPN Training This document provides an overview of Apache Spark, including its history, features, architecture and use cases. Spark started in 2009 at UC Berkeley and was later adopted by the Apache Foundation. It provides faster processing than Hadoop by keeping data in memory. Spark supports batch, streaming and interactive processing on large datasets using its core abstraction called resilient distributed datasets (RDDs).
APACHE SPARK.pptx
APACHE SPARK.pptxDeepaThirumurugan Spark is a fast, general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R for distributed tasks including SQL, streaming, and machine learning. Spark improves on MapReduce by keeping data in-memory, allowing iterative algorithms to run faster than disk-based approaches. Resilient Distributed Datasets (RDDs) are Spark's fundamental data structure, acting as a fault-tolerant collection of elements that can be operated on in parallel.
Ad
More Related Content
What's hot (20)
#GeodeSummit - Redis to Geode Adaptor
#GeodeSummit - Redis to Geode AdaptorPivotalOpenSourceHub This document discusses implementing a Redis adaptor using Apache Geode. It provides an overview of Redis data structures and commands, describes how Geode partitioned regions and indexes can be used to store and access Redis data, outlines advantages like scalability and high availability, and presents a roadmap for further development including supporting additional commands and performance optimization.
RedisConf17 - Home Depot - Turbo charging existing applications with Redis
RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedis Labs The Home Depot is transforming its architecture to use microservices and polyglot persistence to handle increasing online order volumes of 250,000 lines per hour. Redis is being used to turbo charge existing monolithic applications by offloading pieces to new processes using patterns like caching, concurrency management, and powering algorithms. This improves performance by reducing database degradation and wait times by over 95%. Next steps include setting up Redis clusters on-premises and off-premises to further reduce database CPU usage and onboard more patterns.
Apache Cassandra Lunch #72: Databricks and Cassandra
Apache Cassandra Lunch #72: Databricks and CassandraAnant Corporation In Cassandra Lunch #72, we will discuss how we can use Databricks with Cassandra.
Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-72-databricks-and-cassandra
Accompanying YouTube: https://youtu.be/5zCN27KHADo
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Cassandra.Lunch:
https://github.com/Anant/Cassandra.Lunch
Email:
[email protected]
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataMichael Stack This document discusses Apache Kylin, an OLAP engine for big data. It provides 3 key points:
1. Apache Kylin is designed to provide fast, interactive queries on large datasets stored in Hadoop. It uses pre-calculated cube structures stored in HBase to enable sub-second query performance on trillion row datasets.
2. HBase was selected as the storage engine because it is natively integrated with Hadoop, supports high throughput and low-latency queries, and can store very large datasets. Kylin leverages HBase's capabilities for cube storage, metadata storage, online calculation pushing, and caching.
3. Apache Kylin is used by over 1000 global companies across
Riak at shareaholic
Riak at shareaholicfreerobby Robby Grossman presented on Shareaholic's transition from MongoDB to Riak. Shareaholic needed a database with linear scalability, full-text search, and flexible indexing to support their growing product. They evaluated HBase, Cassandra, and Riak. Riak was chosen for its operational simplicity, linear scalability, integrated search, and secondary indices. Shareaholic migrated their data from MongoDB to Riak without downtime by writing to both databases simultaneously and verifying data integrity before decommissioning MongoDB. Riak has succeeded for Shareaholic's MapReduce queries, full text search, and publisher analytics use cases. Benchmarking showed vertical scaling on EC2 provides better latency than horizontal scaling.
HBaseConAsia2018 Track3-2: HBase at China Telecom
HBaseConAsia2018 Track3-2: HBase at China TelecomMichael Stack HBase is used at China Telecom for various applications including persistence for streaming jobs, online reading and writing, and as a data store for their core system. They operate several HBase clusters storing over 500 TB of data ingesting 1 TB per day. They monitor HBase using Ganglia for basic metrics and Zabbix for critical alerts. When issues arise, such as a system hang, they investigate debug cases and perform optimizations like changing the garbage collector from CMS to G1 and implementing read/write splitting.
Zabbix at scale with Elasticsearch
Zabbix at scale with ElasticsearchLeandro Totino Pereira Zabbix was experiencing performance issues due to large history tables in the database. To address this, the architecture was changed to store history data in Elasticsearch instead of database tables. This improved scalability and performance. The basic item and event data remained in the MariaDB database cluster. Zabbix proxies were also used to distribute load across multiple network segments. With this new architecture, history data is indexed in Elasticsearch without database tables, improving query speed and reducing database size.
Scylla Summit 2016: Graph Processing with Titan and Scylla
Scylla Summit 2016: Graph Processing with Titan and ScyllaScyllaDB Jason Plurad, Software Engineer, IBM Open Technology
PMC and Committer, Apache TinkerPop, on graph landscape, graph computing, and Titan and Scylla
Kafka website activity architecture
Kafka website activity architectureOmid Vahdaty Using Kafka in a website activity use-case. for disaster recovery, multi DC clusters, and multi cloud clusters.
Spark Core
Spark CoreTodd McGrath When learning Apache Spark, where should a person begin? What are the key fundamentals when learning Apache Spark? Resilient Distributed Datasets, Spark Drivers and Context, Transformations, Actions.
Running Scylla on Kubernetes with Scylla Operator
Running Scylla on Kubernetes with Scylla OperatorScyllaDB - The document discusses running Scylla, a NoSQL database, on Kubernetes using the Scylla Operator. The Operator allows Kubernetes to leverage for workload management and provides a management layer for Scylla.
- A demo shows deploying a Scylla cluster on Kubernetes with the Operator, stress testing the deployment, and performing common procedures like scaling up and upgrading Scylla versions.
- The Operator uses custom resources and controllers to map Scylla concepts like members, clusters, and datacenters to Kubernetes concepts like statefulsets and pods. This provides capabilities like topology changes and rolling upgrades.
Beyond Relational
Beyond RelationalLynn Langit The document discusses building data pipelines in the cloud. It covers serverless data pipeline patterns using services like BigQuery, Cloud Storage, Cloud Dataflow, and Cloud Pub/Sub. It also compares Cloud Dataflow and Cloud Dataproc for ETL workflows. Key questions around ingestion and ETL are discussed, focusing on volume, variety, velocity and veracity of data. Cloud vendor offerings for streaming and ETL are also compared.
Tachyon and Apache Spark
Tachyon and Apache Sparkrhatr Meet Tachyon: an exciting new project trying to unify in-memory computing and provide a universal data-exchange layer for Hadoop ecosystem projects.
Building a REST API with Cassandra on Datastax Astra Using Python and Node
Building a REST API with Cassandra on Datastax Astra Using Python and NodeAnant Corporation DataStax Astra provides the ability to develop and deploy data-driven applications with a cloud-native service, without the hassles of database and infrastructure administration. In this webinar, we are going to walk you through creating a REST API and exposing that to your Cassandra database.
Webinar Link: https://www.youtube.com/watch?v=O64pJa3eLqs&t=20s
Real time data viz with Spark Streaming, Kafka and D3.js
Real time data viz with Spark Streaming, Kafka and D3.jsBen Laird This document discusses building a dynamic visualization of large streaming transaction data. It proposes using Apache Kafka to handle the transaction stream, Apache Spark Streaming to process and aggregate the data, MongoDB for intermediate storage, a Node.js server, and Socket.io for real-time updates. Visualization would use Crossfilter, DC.js and D3.js to enable interactive exploration of billions of records in the browser.
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...
Cassandra Community Webinar: Apache Spark Analytics at The Weather Channel - ...DataStax Academy The state of analytics has changed dramatically over the last few years. Hadoop is now commonplace, and the ecosystem has evolved to include new tools such as Spark, Shark, and Drill, that live alongside the old MapReduce-based standards. It can be difficult to keep up with the pace of change, and newcomers are left with a dizzying variety of seemingly similar choices. This is compounded by the number of possible deployment permutations, which can cause all but the most determined to simply stick with the tried and true. But there are serious advantages to many of the new tools, and this presentation will give an analysis of the current state–including pros and cons as well as what’s needed to bootstrap and operate the various options.
About Robbie Strickland, Software Development Manager at The Weather Channel
Robbie works for The Weather Channel’s digital division as part of the team that builds backend services for weather.com and the TWC mobile apps. He has been involved in the Cassandra project since 2010 and has contributed in a variety of ways over the years; this includes work on drivers for Scala and C#, the Hadoop integration, heading up the Atlanta Cassandra Users Group, and answering lots of Stack Overflow questions.
Spark Streaming and MLlib - Hyderabad Spark Group
Spark Streaming and MLlib - Hyderabad Spark GroupPhaneendra Chiruvella - A brief introduction to Spark Core
- Introduction to Spark Streaming
- A Demo of Streaming by evaluation top hashtags being used
- Introduction to Spark MLlib
- A Demo of MLlib by building a simple movie recommendation engine
Performance Troubleshooting Using Apache Spark Metrics
Performance Troubleshooting Using Apache Spark MetricsDatabricks Luca Canali, a data engineer at CERN, presented on performance troubleshooting using Apache Spark metrics at the UnifiedDataAnalytics #SparkAISummit. CERN runs large Hadoop and Spark clusters to process over 300 PB of data from the Large Hadron Collider experiments. Luca discussed how to gather, analyze, and visualize Spark metrics to identify bottlenecks and improve performance.
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...
HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack This document summarizes a presentation on scaling a 30 TB data lake using Apache HBase and Scala. It introduces Apache HBase and Spark as technologies for building fast data platforms. It then describes a case study where they were used to architect a retail analytics platform capable of processing 4.6 billion events weekly from 30 TB of historical data. Key aspects included using HBase for data deduplication and as a master data management system, and connecting Spark to HBase using a Scala DSL for efficient querying and updates at scale. Performance was improved 5x by reengineering the data pipeline to be highly concurrent and asynchronous.
Apache Spark Acceleration Using Hardware Resources in the Cloud, Seamlessl wi...
Apache Spark Acceleration Using Hardware Resources in the Cloud, Seamlessl wi...Databricks InAccel provides high performance accelerators for your application based on novel hardware reconfigurable engines as IP blocks. The hardware accelerators can be deployed in the cloud, like Amazon AWS, using the f1 accelerators and can be integrated to widely used frameworks like Spark and PostgreSQL. The main novelty is that the users do not need to change the original code as the accelerators are deployed as software packages.
In this talk we will show how machine learning applications (e.g. logistic regression and K-means), based on Spark, can be accelerated by 3x-10x using hardware accelerators that are deployed in the Amazon AWS using f1 accelerators without any changes on the Spark code. InAccel provides all the required APIs for the integration on Spark using Java, Python or Scala. The utilization of hardware accelerators can also be used to reduce the OpEx as less resources and less time is required for the processing of the data.
Similar to Apache Spark on Kubernetes (20)
Module01
Module01NPN Training This document provides an overview of Apache Spark, including its history, features, architecture and use cases. Spark started in 2009 at UC Berkeley and was later adopted by the Apache Foundation. It provides faster processing than Hadoop by keeping data in memory. Spark supports batch, streaming and interactive processing on large datasets using its core abstraction called resilient distributed datasets (RDDs).
APACHE SPARK.pptx
APACHE SPARK.pptxDeepaThirumurugan Spark is a fast, general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R for distributed tasks including SQL, streaming, and machine learning. Spark improves on MapReduce by keeping data in-memory, allowing iterative algorithms to run faster than disk-based approaches. Resilient Distributed Datasets (RDDs) are Spark's fundamental data structure, acting as a fault-tolerant collection of elements that can be operated on in parallel.
Savanna - Elastic Hadoop on OpenStack
Savanna - Elastic Hadoop on OpenStackSergey Lukjanov Savanna is an OpenStack component that allows elastic provisioning of Hadoop clusters in OpenStack. It has a 3 phase roadmap - phase 1 allows basic cluster provisioning which is complete, phase 2 will add advanced configuration and tool integration currently in progress, and phase 3 will enable analytics as a service with a job execution framework. Savanna uses an extensible plugin architecture to provision Hadoop VMs and configure the clusters, integrating with other OpenStack components like Nova, Glance, and Swift.
Big data and Kubernetes
Big data and KubernetesAnirudh Ramanathan High level overview of work performed around the Kubernetes ecosystem to enable big data applications and pipelines
What's the Hadoop-la about Kubernetes?
What's the Hadoop-la about Kubernetes?DataWorks Summit There is increased interest in using Kubernetes, the open-source container orchestration system for modern, stateful Big Data analytics workloads. The promised land is a unified platform that can handle cloud native stateless and stateful Big Data applications. However, stateful, multi-service Big Data cluster orchestration brings unique challenges. This session will delve into the technical gaps and considerations for Big Data on Kubernetes.
Containers offer significant value to businesses; including increased developer agility, and the ability to move applications between on-premises servers, cloud instances, and across data centers. Organizations have embarked on this journey to containerization with an emphasis on stateless workloads. Stateless applications are usually microservices or containerized applications that don’t “store” data. Web services (such as front end UIs and simple, content-centric experiences) are often great candidates as stateless applications since HTTP is stateless by nature. There is no dependency on the local container storage for the stateless workload.
Stateful applications, on the other hand, are services that require backing storage and keeping state is critical to running the service. Hadoop, Spark and to lesser extent, noSQL platforms such as Cassandra, MongoDB, Postgres, and mySQL are great examples. They require some form of persistent storage that will survive service restarts...
Speakers
Anant Chintamaneni, VP Products, BlueData
Nanda Vijaydev, Director Solutions, BlueData
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...Alex Zeltov Introduction to Big Data Analytics using Apache Spark on HDInsights on Azure (SaaS) and/or HDP on Azure(PaaS)
This workshop will provide an introduction to Big Data Analytics using Apache Spark using the HDInsights on Azure (SaaS) and/or HDP deployment on Azure(PaaS) . There will be a short lecture that includes an introduction to Spark, the Spark components.
Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes.
The lecture will be followed by demo . There will be a short lecture on Hadoop and how Spark and Hadoop interact and compliment each other. You will learn how to move data into HDFS using Spark APIs, create Hive table, explore the data with Spark and SQL, transform the data and then issue some SQL queries. We will be using Scala and/or PySpark for labs.
Lessons learned from running Spark on Docker
Lessons learned from running Spark on DockerDataWorks Summit Today, most any application can be “Dockerized.” However, there are special challenges when deploying a distributed application such as Spark on containers. This session will describe how to overcome these challenges in deploying Spark on Docker containers, with many practical tips and techniques for running Spark in a container environment.
Containers are typically used to run stateless applications on a single host. There are significant real-world enterprise requirements that need to be addressed when running a stateful, distributed application in a secure multi-host container environment.
There are decisions that need to be made concerning which tools and infrastructure to use. There are many choices with respect to container managers, orchestration frameworks, and resource schedulers that are readily available today and some that may be available tomorrow including:]
• Mesos
• Kubernetes
• Docker Swarm
Each has its own strengths and weaknesses; each has unique characteristics that may make it suitable, or unsuitable, for Spark. Understanding these differences is critical to the successful deployment of Spark on Docker containers.
This session will describe the work done by the BlueData engineering team to run Spark inside containers, on a distributed platform, including the evaluation of various orchestration frameworks and lessons learned. You will learn how to apply practical networking and storage techniques to achieve high performance and agility in a distributed, container environment.
Speaker
Thomas Phelan, Chief Architect, Blue Data, Inc
Deploying Anything as a Service (XaaS) Using Operators on Kubernetes
Deploying Anything as a Service (XaaS) Using Operators on KubernetesAll Things Open This document discusses deploying software-as-a-service (XaaS) applications using operators on Kubernetes. It defines operators as collections of custom resource definitions and controllers that manage the lifecycle of those resources. Operators can deploy applications and dependencies within or outside the Kubernetes cluster. The document provides examples of when to use operators for internal resources like databases, as well as managed cloud services. It also discusses where to find operators and how to deploy common ones like Elasticsearch, AWS services, and Kafka.
Introduction to Apache Spark
Introduction to Apache Sparkdatamantra Apache Spark is a fast, general engine for large-scale data processing. It provides unified analytics engine for batch, interactive, and stream processing using an in-memory abstraction called resilient distributed datasets (RDDs). Spark's speed comes from its ability to run computations directly on data stored in cluster memory and optimize performance through caching. It also integrates well with other big data technologies like HDFS, Hive, and HBase. Many large companies are using Spark for its speed, ease of use, and support for multiple workloads and languages.
Apache Spark on HDinsight Training
Apache Spark on HDinsight TrainingSynergetics Learning and Cloud Consulting An Engine to process big data in faster(than MR), easy and extremely scalable way. An Open Source, parallel, in-memory processing, cluster computing framework. Solution for loading, processing and end to end analyzing large scale data. Iterative and Interactive : Scala, Java, Python, R and with Command line interface.
New Analytics Toolbox DevNexus 2015
New Analytics Toolbox DevNexus 2015Robbie Strickland The state of analytics has changed dramatically over the last few years. Hadoop is now commonplace, and the ecosystem has evolved to include new tools such as Spark, Shark, and Drill, that live alongside the old MapReduce-based standards. It can be difficult to keep up with the pace of change, and newcomers are left with a dizzying variety of seemingly similar choices. This is compounded by the number of possible deployment permutations, which can cause all but the most determined to simply stick with the tried and true. In this talk I will introduce you to a powerhouse combination of Cassandra and Spark, which provides a high-speed platform for both real-time and batch analysis.
Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek Alumni
Spark 101 – First Steps To Distributed Computing - Demi Ben-Ari @ Ofek AlumniDemi Ben-Ari The world has changed and having one huge server won’t do the job anymore, when you’re talking about vast amounts of data, growing all the time the ability to Scale Out would be your saviour.
This lecture will be about the basics of Apache Spark and distributed computing and the development tools needed to have a functional environment.
Bio:
Demi Ben-Ari, Sr. Data Engineer @Windward, Ofek Alumni
Has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems.
Co-Founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-i...
Hadoop and OpenStack - Hadoop Summit San Jose 2014
Hadoop and OpenStack - Hadoop Summit San Jose 2014spinningmatt This document discusses Hadoop and OpenStack Sahara. Sahara is an OpenStack project that allows users to provision and manage Hadoop clusters within OpenStack. It provides a plugin mechanism to support different Hadoop distributions like Hortonworks Data Platform (HDP). The HDP plugin fully integrates HDP clusters with Sahara using the Ambari API for cluster management. Sahara handles tasks like cluster scaling, integration with Swift for storage, and data locality. Its plugin architecture allows different Hadoop versions and distributions to be deployed and managed through Sahara.
Hadoop and OpenStack
Hadoop and OpenStackDataWorks Summit Sahara is an OpenStack project that provides an abstraction layer for provisioning and managing Apache Hadoop clusters and jobs in OpenStack clouds. It allows users to easily deploy and scale Hadoop clusters on demand without having to manage the underlying infrastructure. Sahara uses plugins to integrate various Hadoop distributions like Hortonworks Data Platform (HDP) and Cloudera Distribution including Apache Hadoop (CDH). It leverages other OpenStack services like Nova, Neutron, Swift, Cinder, Heat etc. to provision, configure and manage the Hadoop clusters and jobs.
Unit II Real Time Data Processing tools.pptx
Unit II Real Time Data Processing tools.pptxRahul Borate Apache Spark is a lightning-fast cluster computing framework designed for real-time processing. It overcomes limitations of Hadoop by running 100 times faster in memory and 10 times faster on disk. Spark uses resilient distributed datasets (RDDs) that allow data to be partitioned across clusters and cached in memory for faster processing.
Key trends in Big Data and new reference architecture from Hewlett Packard En...
Key trends in Big Data and new reference architecture from Hewlett Packard En...Ontico Динамичное развитие инструментов для обработки Больших Данных порождает новые подходы к повышению производительности. Ключевые новые технологии в Hadoop 2.0, такие как Yarn labeling и Storage Tiering, уже используются компаниями Yahoo и Ebay. Эти новые технологии открывают путь для серьезного повышения эффективности ИТ-инфраструктуры для Hadoop, достигая прироста производительности в несколько десятков процентов при одновременном снижении потребления памяти и электроэнергии.
Эталонная архитектура для Hadoop от HP — HP Big Data Reference Architecture — предлагает использование специализированных "микросерверов" HP Moonshot вкупе с высокоплотными узлами хранения HP Apollo для достижения лучших на сегодня показателей полезной отдачи от железа в Hadoop.
Etu Solution Day 2014 Track-D: 掌握Impala和Spark
Etu Solution Day 2014 Track-D: 掌握Impala和SparkJames Chen ESD (Etu Solution Day) 2014, Track D Slide. A presentation about Impala and Spark introduction and hands-on.
Getting Started with Spark Scala
Getting Started with Spark ScalaKnoldus Inc. The exponential growth of data is not a problem but processing/managing the huge diversity of data is a concern. In this session, we are gonna discuss about one of the most popular big data distributed processing framework "Spark" where we'll be developing API using "Scala", a gained prominence amid big data developers.
Ad
Recently uploaded (20)
Implementing promises with typescripts, step by step
Implementing promises with typescripts, step by stepRan Wahle Implementing promises with typescripts, step by step.
This slides were presented at Negev Web Development meetup
Best Practices for Collaborating with 3D Artists in Mobile Game Development
Best Practices for Collaborating with 3D Artists in Mobile Game DevelopmentJuego Studios Discover effective strategies for working with 3D artists on mobile game projects. Learn how top mobile game development companies streamline collaboration with 3D artists in Dubai for high-quality, optimized game assets.
Exploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the FutureICS Wayland is revolutionizing the way we interact with graphical interfaces, offering a modern alternative to the X Window System. In this webinar, we’ll delve into the architecture and benefits of Wayland, including its streamlined design, enhanced performance, and improved security features.
Get & Download Wondershare Filmora Crack Latest [2025]![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Get & Download Wondershare Filmora Crack Latest [2025]saniaaftab72555 Copy & Past Link 👉👉
https://dr-up-community.info/
Wondershare Filmora is a video editing software and app designed for both beginners and experienced users. It's known for its user-friendly interface, drag-and-drop functionality, and a wide range of tools and features for creating and editing videos. Filmora is available on Windows, macOS, iOS (iPhone/iPad), and Android platforms.
Microsoft Excel Core Points Training.pptx
Microsoft Excel Core Points Training.pptxMekonnen This is a short and precise training PowerPoint about Microsoft excel.
Odoo ERP for Education Management to Streamline Your Education Process
Odoo ERP for Education Management to Streamline Your Education ProcessiVenture Team LLP Odoo ERP for Education Management can streamline your education process such as admission, staff, lecture, payment and much more.
Expand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchangeFexle Services Pvt. Ltd. AgentExchange is Salesforce’s latest innovation, expanding upon the foundation of AppExchange by offering a centralized marketplace for AI-powered digital labor. Designed for Agentblazers, developers, and Salesforce admins, this platform enables the rapid development and deployment of AI agents across industries.
Email: [email protected]
Phone: +1(630) 349 2411
Website: https://www.fexle.com/blogs/agentexchange-an-ultimate-guide-for-salesforce-consultants-businesses/?utm_source=slideshare&utm_medium=pptNg
Secure Test Infrastructure: The Backbone of Trustworthy Software Development
Secure Test Infrastructure: The Backbone of Trustworthy Software DevelopmentShubham Joshi A secure test infrastructure ensures that the testing process doesn’t become a gateway for vulnerabilities. By protecting test environments, data, and access points, organizations can confidently develop and deploy software without compromising user privacy or system integrity.
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdfTechSoup In this webinar we will dive into the essentials of generative AI, address key AI concerns, and demonstrate how nonprofits can benefit from using Microsoft’s AI assistant, Copilot, to achieve their goals.
This event series to help nonprofits obtain Copilot skills is made possible by generous support from Microsoft.
What You’ll Learn in Part 2:
Explore real-world nonprofit use cases and success stories.
Participate in live demonstrations and a hands-on activity to see how you can use Microsoft 365 Copilot in your own work!
DVDFab Crack FREE Download Latest Version 2025
DVDFab Crack FREE Download Latest Version 2025younisnoman75 ⭕️➡️ FOR DOWNLOAD LINK : http://drfiles.net/ ⬅️⭕️
DVDFab is a multimedia software suite primarily focused on DVD and Blu-ray disc processing. It offers tools for copying, ripping, creating, and editing DVDs and Blu-rays, as well as features for downloading videos from streaming sites. It also provides solutions for playing locally stored video files and converting audio and video formats.
Here's a more detailed look at DVDFab's offerings:
DVD Copy:
DVDFab offers software for copying and cloning DVDs, including removing copy protections and creating backups.
DVD Ripping:
This allows users to rip DVDs to various video and audio formats for playback on different devices, while maintaining the original quality.
Blu-ray Copy:
DVDFab provides tools for copying and cloning Blu-ray discs, including removing Cinavia protection and creating lossless backups.
4K UHD Copy:
DVDFab is known for its 4K Ultra HD Blu-ray copy software, allowing users to copy these discs to regular BD-50/25 discs or save them as 1:1 lossless ISO files.
DVD Creator:
This tool allows users to create DVDs from various video and audio formats, with features like GPU acceleration for faster burning.
Video Editing:
DVDFab includes a video editing tool for tasks like cropping, trimming, adding watermarks, external subtitles, and adjusting brightness.
Video Player:
A free video player that supports a wide range of video and audio formats.
All-In-One:
DVDFab offers a bundled software package, DVDFab All-In-One, that includes various tools for handling DVD and Blu-ray processing.
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025mu394968 🌍📱👉COPY LINK & PASTE ON GOOGLE https://dr-kain-geera.info/👈🌍
PDF Reader Pro is a software application, often referred to as an AI-powered PDF editor and converter, designed for viewing, editing, annotating, and managing PDF files. It supports various PDF functionalities like merging, splitting, converting, and protecting PDFs. Additionally, it can handle tasks such as creating fillable forms, adding digital signatures, and performing optical character recognition (OCR).
Creating Automated Tests with AI - Cory House - Applitools.pdf
Creating Automated Tests with AI - Cory House - Applitools.pdfApplitools In this fast-paced, example-driven session, Cory House shows how today’s AI tools make it easier than ever to create comprehensive automated tests. Full recording at https://applitools.info/5wv
See practical workflows using GitHub Copilot, ChatGPT, and Applitools Autonomous to generate and iterate on tests—even without a formal requirements doc.
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxPortli Discover the top 10 client portal software solutions for 2025. Streamline communication, ensure security, and enhance client experience.
Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)Allon Mureinik Tests, especially unit tests, are the developers’ superheroes. They allow us to mess around with our code and keep us safe.
We often trust them with the safety of our codebase, but how do we know that we should? How do we know that this trust is well-deserved?
Enter mutation testing – by intentionally injecting harmful mutations into our code and seeing if they are caught by the tests, we can evaluate the quality of the safety net they provide. By watching the watchmen, we can make sure our tests really protect us, and we aren’t just green-washing our IDEs to a false sense of security.
Talk from SciFiDevCon 2025
https://www.scifidevcon.com/courses/2025-scifidevcon/contents/680efa43ae4f5
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxlaravinson24 This presentation is a beginner-friendly guide to developing a crypto wallet from scratch. It covers essential concepts such as wallet types, blockchain integration, key management, and security best practices. Ideal for developers and tech enthusiasts looking to enter the world of Web3 and decentralized finance.
Kubernetes_101_Zero_to_Platform_Engineer.pptx
Kubernetes_101_Zero_to_Platform_Engineer.pptxCloudScouts Presentacion de la primera sesion de Zero to Platform Engineer
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)Andre Hora Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including CONTRIBUTING files (58%), external documentation (24%), and README files (8%). Furthermore, test documentation commonly explains how to run tests (83.5%), but less often provides guidance on how to write tests (37%). It frequently covers unit tests (71%), but rarely addresses integration (20.5%) and end-to-end tests (15.5%). Other key testing aspects are also less frequently discussed: test coverage (25.5%) and mocking (9.5%). We conclude by discussing implications and future research.
Tools of the Trade: Linux and SQL - Google Certificate
Tools of the Trade: Linux and SQL - Google CertificateVICTOR MAESTRE RAMIREZ Tools of the Trade: Linux and SQL - Google Certificate
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with ...Eric D. Schabell It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).
Revolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptxnidhisingh691197 Discover why Wi-Fi 7 is set to transform wireless networking and how Router Architects is leading the way with next-gen router designs built for speed, reliability, and innovation.
Ad
Apache Spark on Kubernetes
- 1. Apache Spark on Kubernetes Haridas N
- 2. Agenda ● What’s Kubernetes ● Why we need to run park on Kubernetes ● How comparable kubernetes with other cluster managers ● Hands on with few spark jobs.
- 3. Kubernetes ● Container orchestrator ● Provision containers on multiple nodes and abstract the networks over multiple node. ● Supports multiple namespaces for better project isolation. ● User role and privilege management.
- 4. Spark on Kubernetes ● Support different deployment modes
- 6. Why spark on Kubernetes ● Kubernetes is now widely used container management for SOA and other application environment. ● Better isolation on different deployments.
- 8. RECAP
- 9. Hadoop, HDFS and Yarn ● HDFS for the storage layer, namenode and datanode services take care the data storage part. ● Yarn resource negotiator provide a compute framework over HDFS nodes. ● Map-reduce jobs are written on yarn framework. ● Best fit for batch-processing, big-data storage
- 10. Apache Spark on Yarn ● Using Yarn we deployed spark driver and slaves into hadoop cluster. ● Yarn provides more flexible resource management. ● Dynamic worker allocation or on demand allocation. ● Best fit, if you already have a hadoop cluster and want to run spark jobs on it.
- 11. Apache Drill ● SQL interface for bigdata, with spark like architecture. ● Interface with HDFS, NoSQL, Hive, Kafka etc. and provide unified standard SQL interface ● Exposes APIs JDBC, HTTP. ● Best fit for quick data analysis using SQL commands.
- 12. Apache Spark on Kubernetes ● Kubernetes is a widely used container orchestrator ● Major deployments outside big-data domain for different needs. ● Project supports big-data tools like spark and hadoop on top of it. ● Run spark job on existing kubernetes cluster. ● Got better feature set with resourcemanagement compared to all other cluster managers. ● Best fit if you already have kubernetes cluster in your environment.
- 13. QA
- 14. Thank you