Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Spark on Mesos (20)
Ad
More from Jen Aman (15)
Recently uploaded (20)
Spark on Mesos
- 1. Spark on Mesos Tim Chen Mirantis [email protected] Dean Wampler Lightbend [email protected]
- 2. Dean Wampler • Architect for Big Data Products at Lightbend • Early advocate for Spark on Mesos • O’Reilly author –Programming Scala, 2nd Edition –Programming Hive –Functional Programming for Java Developers Timothy Chen • Principal Engineer at Mirantis • Previously lead engineer at Mesosphere • Apache Mesos PMC • Spark contributor, help maintain Spark on Mesos
- 3. What’s this all about, then? • Why Spark on Mesos? • What’s happened since last year? • Demo - GPU support •What’s next for Spark and Mesos?
- 4. Why Spark on Mesos • Hadoop is great, but ... –… resource management with YARN is limited to compute engines like MapReduce and Spark. •What if your clustering system could run everything?
- 5. Why Spark on Mesos • Hadoop is great, but ... –… Big Data is moving to streaming (“Fast Data”) and Spark offers mini-batch streaming. •What if your cluster system offered dynamic and flexible resource scheduling able to meet the needs of evolving, long-running streams?
- 6. Why Spark on Mesos • Hadoop is great, but ... –… it doesn’t support other popular tools like Cassandra, Akka, web frameworks, ... •Maybe you need the SMACK stack: –Spark –Mesos –Akka –Cassandra –Kafka There’s a Scheduler for that!
- 7. What’s happened since last year? • What’s new in Mesos • What’s new in Spark on Mesos • Getting rid of fine-grained mode?
- 8. What’s new in Mesos? • Maintenance primitives • Resource quotas, dynamic reservation *Beta* • CNI network Support • GPU Support • Unified Containerizer • More..
- 9. What’s new in Spark on Mesos? • Integration test suite • New scheduler • Mesos framework authentication • Cluster mode now supports Python
- 10. Integration Test Suite • A recent release candidate for Spark broke Mesos integration completely. –Better integration testing clearly needed. –Lightbend and Mesosphere collaborated on an automated integration test suite. https://github.com/typesafehub/mesos-spark-integration-tests
- 11. Integration Test Suite • “mesos-docker” subproject: –Builds Docker image with Ubuntu, Mesos, Spark, and HDFS. –Scripts to run cluster with 1 master and N slaves, configurable #s of CPUs, memory, etc. •(Not needed if you already have a Mesos cluster ;^)
- 12. Integration Test Suite • “test-runner” subproject: –Executes a suite of tests on your Mesos or DC/OS cluster. – Currently exercises dynamic allocation, coarse-grain and fine-grain modes, etc.
- 13. New Coarse Grain Scheduler How the old Coarse grain scheduler works? Launch 1 Spark executor per agent - Rough steps: - Evaluate offers as it comes in from the master - Offers that meets min cpu (1) and min memory requirements - Use as much cores until meets spark.cores.max - Every executor requests fixed memory
- 14. New Coarse Grain Scheduler How the old Coarse grain scheduler works? Mesos Agent 1 CPU: 8 Memory: 8gb Mesos Agent 2 CPU: 8 Memory: 8gb Mesos Agent 3 CPU: 8 Memory: 8gb CoarseMesosSchedulerBackend spark.cores.max=12 spark.executor.memory=4gb Spark Executor CPU 8 Memory 4gb Spark Executor CPU 4 Memory 4gb
- 15. New Coarse Grain Scheduler How the old Coarse grain scheduler works? Mesos Agent 1 CPU: 8 Memory: 8gb Mesos Agent 2 CPU: 2 Memory: 8gb Mesos Agent 3 CPU: 2 Memory: 8gb CoarseMesosSchedulerBackend spark.cores.max=12 spark.executor.memory=4gb Spark Executor CPU 8 Memory 4gb Spark Executor CPU 2 Memory 4gb Spark Executor CPU 2 Memory 4gb
- 16. New Coarse Grain Scheduler How the old Coarse grain scheduler works? Mesos Agent CPU: 8 Memory: 64gb Mesos Agent CPU: 2 Memory: 64gb Mesos Agent CPU: 2 Memory: 64gb CoarseGrainedMesosScheduler Spark Executor CPU 8 Memory 64gb Spark Executor CPU 2 Memory 64gb Spark Executor CPU 2 Memory 64gb spark.cores.max=12 spark.executor.memory=64gb
- 17. New Coarse Grain Scheduler Problems with the old scheduler: • Only allow one executor per slave • Unpredictable performance • Can skew allocation
- 18. New Coarse Grain Scheduler Mesos Agent 1 CPU: 8 Memory: 8gb Mesos Agent 2 CPU: 8 Memory: 8gb Mesos Agent 3 CPU: 8 Memory: 8gb CoarseMesosSchedulerBackend spark.cores.max=12 spark.executor.memory=4gb spark.executor.cores=4 Spark Executor CPU 4 Memory 4gb Spark Executor CPU 4 Memory 4gb Spark Executor CPU 4 Memory 4gb
- 19. Mesos Framework Authentication • Mesos supports framework authentication. • Roles can be set per framework –Impacts the relative weight of resource allocation •Optional authentication information to allow the framework to be connected to the master.
- 20. Getting rid of fine-grained mode? Coarse-grained Mode Fine-grained Mode
- 21. Getting rid of fine-grained mode? • Why two modes? –FG uses resources more efficiently, because of start- on-demand and Spark executor+task are removed when no longer needed. –CG holds onto all allocated tasks until the job finishes. –But that makes CG faster to start tasks; nice for interactive jobs (e.g., SQL queries). –While FG has a longer start up time.
- 22. Getting rid of fine-grained mode? • Why two modes? –Until recently, only ONE CG executor allowed per worker node. –Makes it harder to exploit all of the node’s resources.
- 23. Getting rid of fine-grained mode? • Today: –Dynamic Allocation reclaims unused executors. •(Although running this service on every node is a disadvantage) –Allows more than one CG executor per node. •Hence, the advantages of FG are becoming less important.
- 24. Getting rid of fine-grained mode? • Spark has lots of redundant code to implement both modes. •So, to simplify the code base and operations, FG is now deprecated, but it can’t be removed yet.
- 25. GPUsMesos Running _____________ on __________with _______ on top of ______ using _____ in the _____! Demo Deep Learning Tensorflow Spark Cloud
- 26. What’s Next for Mesos? • Pod support • Multiple roles support • Event Bus • Improved Container Security (capabilities, etc)
- 27. What’s Next for Spark on Mesos? • GPU Support on Mesos • Use revocable resources • Better scheduling –Strategies (e.g: Spread, Binpack) –Scheduling metrics • More integration test coverage: –More cluster and job configuration options. –Roles and authentication scenarios.
- 28. What’s Next for Spark on Mesos? • Make “production” easier: –Easier overriding of configuration with config files outside the jars. –Better documentation. –Easier access to Spark UIs and logs from Mesos UIs –Improved metrics. –Smarter acceptance of resources offered.