How to deploy Apache Spark to Mesos/DCOS
26 likes46,378 views
The document provides an overview of deploying Apache Spark on Mesos/DCOS, highlighting the benefits of Spark as a lightning-fast big data solution and the efficiency of Mesos as a cluster manager. It discusses different execution modes, resource scheduling, the integration of Spark with various systems, and the importance of dynamic allocation in resource management. Additionally, it outlines support options available for organizations looking to implement Spark with Mesos.
How to deploy Apache Spark to Mesos/DCOS
- 1. How to deploy Apache Spark to Mesos/DCOS with Iulian Dragoș
- 2. Agenda • Intro Apache Spark • Apache Mesos • Why Spark on Mesos • A look under the hood 2
- 4. Spark - lightning-fast cluster computing • next generation Big Data solution • analytics and data processing • up to 100x faster than Hadoop MapReduce • built with Scala and Akka • Apache top-level project 4
- 5. Spark • It’s a next generation compute-engine • Does not replace the whole Hadoop ecosystem • just MapReduce • Integrates/works with HDFS, Hive, Hbase, etc. 5
- 6. Spark API • Scala distributed collections • also available from Python and Java • interactive shell and job submission • streaming and batch modes • flourishing ecosystem (SparkSQL, MLLib, GraphX) 6
- 8. Spark execution • local (for experimentation) • standalone (built-in cluster manager) • YARN (Hadoop cluster manager) • Mesos (general cluster manager) 8
- 9. Apache Mesos
- 10. Why Apache Mesos? • General (a “distributed kernel”) • Efficient resource management • Proven technology (in production at Apple and Twitter) • Typesafe & Mesosphere maintain the Spark/Mesos framework 10 “Program against your datacenter as a single pool of resources”
- 11. Frameworks running on Mesos • HDFS • Cassandra • ElasticSearch • Yarn (Myriad) • Marathon, etc. • and of course, Spark 11
- 12. Resource scheduling with Mesos • 2-level scheduling • Mesos offers resources to frameworks • Frameworks accept or reject offers • Offers include • CPU cores, memory, ports, disk 12
- 13. 13 Mesos Cluster master Mesos Master Key Mesos Spark HDFS master / client master / client node Mesos Slave Name Node Executor task1 … node DiskDiskDiskDiskDisk Mesos Slave Data Node Executor … … node … HDFS FW Sched. Job 1 Spark FW Sched. Job 1 1 (S1, 8CPU, 32GB, ...)
- 14. 14 Mesos Cluster master Mesos Master Key Mesos Spark HDFS master / client master / client node Mesos Slave Name Node Executor task1 … node DiskDiskDiskDiskDisk Mesos Slave Data Node Executor … … node … HDFS FW Sched. Job 1 Spark FW Sched. Job 1 2 (S1, 8CPU, 32GB, ...) 1
- 15. def foo(x: Int) 15 Mesos Cluster master Mesos Master Key Mesos Spark HDFS master / client master / client node Mesos Slave Name Node Executor task1 … node DiskDiskDiskDiskDisk Mesos Slave Data Node Executor … … node … HDFS FW Sched. Job 1 Spark FW Sched. Job 1 2 1 (S1, 2CPU, 8GB, ...) (S1, 2CPU, 8GB, ...) 3
- 16. def foo(x: Int) 16 Mesos Cluster master Mesos Master Key Mesos Spark HDFS master / client master / client node Mesos Slave Name Node Executor task1 … node DiskDiskDiskDiskDisk Mesos Slave Data Node Executor … … node … HDFS FW Sched. Job 1 Spark FW Sched. Job 1 2 1 (S1, 2CPU, 8GB, ...) (S1, 2CPU, 8GB, ...) 3 4 Spark Executor task1 …
- 17. 17 Spark Cluster Abstraction …NodeNode Spark Driver object MyApp { def main() { val sc = new SparkContext(…) … } } Cluster Manager Spark Executor task task task task Spark Executor task task task task …
- 18. 18 Mesos Coarse Grained Mode …Node Node Mesos Executor …Mesos Executor master Spark Executor task task task task Spark Executor task task task task … Mesos Master Spark Framework Spark Driver object MyApp { def main() { val sc = new SparkContext(…) … } } Scheduler
- 19. Mesos Coarse Grained Mode • Fast startup for tasks: • Better for interactive sessions. • But resources locked up in larger Mesos task. • (Dynamic allocation changes this is in 1.5) 19 …Node Node Mesos Executor …Mesos Executor master Spark Executor task task task task Spark Executor task task task task … Mesos Master Spark Framework Spark Driver object MyApp { def main() { val sc = new SparkContext(…) … } } Scheduler
- 20. Mesos Fine Grained Mode 20 …NodeNode Spark Framework Mesos Executor … master Spark Driver object MyApp { def main() { val sc = new SparkContext(…) … } } task task task task … Mesos Master Mesos Executor Spark Exec task Spark Exec task Spark Exec task Spark Exec task Mesos Executor Spark Exec task Spark Exec task Spark Exec task Spark Exec task … Scheduler
- 21. Mesos Fine Grained Mode • Better resource utilization. • Slower startup for tasks: • Fine for batch and relatively static streaming. 21 …NodeNode Spark Framework Mesos Executor … master Spark Driver object MyApp { def main() { val sc = new SparkContext(…) … } } task task task task … Mesos Master Mesos Executor Spark Exec task Spark Exec task Spark Exec task Spark Exec task Mesos Executor Spark Exec task Spark Exec task Spark Exec task Spark Exec task … Scheduler
- 22. Dynamic allocation • Mesos support was added in Spark 1.5 • adds and removes executors based on load • when executors are idle, kills them • when tasks queue up in the scheduler, adds executors • needs external-shuffle-service to be running on each node 22
- 23. Client vs Cluster mode • Where does the driver process run? • client-mode: on the machine that submits the job • cluster-mode: on a machine in the cluster 23
- 24. Demo
- 25. What’s next on Mesos • Oversubscription (0.23) • Persistence Volumes • Dynamic Reservations • Optimistic Offers • Isolations • More…. 25
- 26. Closing words on Spark Streaming • Spark 1.5 improves resiliency by adding back-pressure inside Spark Streaming • slow-down receivers dynamically, based on load • Spark 1.6 will add the ability to connect to Reactive Streams • propagate back-pressure outside of Spark 26
- 27. Key points • Spark is a next-generation compute engine for Big Data • Mesos is a next-generation cluster manager • better utilization of cluster resources across organization • Spark on Mesos is commercially supported by Typesafe • Typesafe&Mesosphere are the maintainers of Spark/Mesos 27
- 28. EXPERT SUPPORT Why Contact Typesafe for Your Apache Spark Project? Ignite your Spark project with 24/7 production SLA, unlimited expert support and on-site training: • Full application lifecycle support for Spark Core, Spark SQL & Spark Streaming • Deployment to Standalone, EC2, Mesos clusters • Expert support from dedicated Spark team • Optional 10-day “getting started” services package Typesafe is a partner with Databricks, Mesosphere and IBM. Learn more about on-site trainingCONTACT US