Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters
3 likes956 views
This document discusses the Cloud and Information Services Lab (CISL) and its vision of having one cluster that can run all workloads. It describes CISL's research into improving resource management in shared clusters through projects like Mercury, which aims to improve cluster utilization by opportunistically using otherwise idle resources. The document outlines Mercury's architecture and how it implements a hybrid of centralized and distributed scheduling to better schedule tasks with short durations.
![Centralized Resource Management
[YARN, Mesos, Omega, Borg]
Node
Manager
Node
Manager
Node
Manager
1. Request
2. Allocation
3. Start task
•
•
•](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-10-320.jpg)
![Distributed Resource Management
[Apollo, Sparrow]
Node
Manager
Node
Manager
Node
Manager
•
•](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-11-320.jpg)
![Container Types
GUARANTEED containers
•
•
QUEUEABLE containers
• opportunistically
•
•
•
•
central
•
distributed
[YARN-2882]
PATCH AVAILABLE](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-17-320.jpg)
![AMRMProxy
•
•
•
•
[YARN-2884,2885]](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-21-320.jpg)
![Task Execution: Conflict Resolution
two priorities
types of schedulers shared resources
[YARN-2883]](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-24-320.jpg)
![Application Policies
•
• container type to be
requested for each task
•
•
• Choosing QUEUEABLE at job level enables opportunistic jobs
[YARN-2887]](https://image.slidesharecdn.com/june111150ammicrosoftraov2-150629222520-lva1-app6891/85/Mercury-Hybrid-Centralized-and-Distributed-Scheduling-in-Large-Shared-Clusters-26-320.jpg)
Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters
- 2. CISL systems Cloud and Information Services Lab (CISL)
- 3. Vision: One cluster to rule them all Ambitious multi-person, multi-year agenda
- 5. Application Engines M/R AM REEFTezSpark Runtime Cluster-wide resource management: YARN++ YARN + Federation YARN + Rayon YARN + Mercury YARN + Mercury YARN + Mercury YARN + Mercury YARN + Mercury Per-job/framework Resource Management Hive …Storm Giraph PigSpark Big Picture
- 6. Research lab embedded in a Product organization doing Open-Source. The 3 hats we wear in CISL… (We are hiring… Come see us after the talk!)
- 7. Application Engines M/R AM REEFTezSpark Runtime Cluster-wide resource management: YARN++ YARN + Federation YARN + Rayon YARN + Mercury YARN + Mercury YARN + Mercury YARN + Mercury YARN + Mercury Per-job/framework Resource Management Hive …Storm Giraph PigSpark Big Picture
- 9. cluster ROI Consolidate workloads • Heterogeneity APIs centralized distributed high cluster utilization Resource Management in Shared Clusters • •
- 10. Centralized Resource Management [YARN, Mesos, Omega, Borg] Node Manager Node Manager Node Manager 1. Request 2. Allocation 3. Start task • • •
- 11. Distributed Resource Management [Apollo, Sparrow] Node Manager Node Manager Node Manager • •
- 12. Centralized vs. Distributed Scheduling Centralized Distributed Workload heterogeneity Task placement Enforcing scheduling invariants Allocation latency Slot utilization Scalability
- 13. “Sweet spot” we are after 1ms 100ms 1s 1m 1h “Executor” model Mercury sweet spot task duration
- 14. • “Trade performance guarantees for allocation latency” choose among scheduling types Based on application type (SLA job, ad-hoc job, service), job characteristics (task runtime, type of computation), cluster load, etc. Mercury provides a programmatic way to use otherwise idle resources Mercury achieves up to 40% task throughput and 66% mean job latency gain over stock YARN Mercury: Key Insight
- 16. Mercury Architecture (Conceptual) Mercury Runtime Mercury Runtime Mercury Runtime Mercury Resource Management Framework • • • •
- 17. Container Types GUARANTEED containers • • QUEUEABLE containers • opportunistically • • • • central • distributed [YARN-2882] PATCH AVAILABLE
- 18. GUARANTEED vs. QUEUEABLE Containers GUARANTEED containers • • QUEUEABLE containers • • • •
- 19. Hybrid Scheduling on Tez AM: Examples • • •
- 20. AMRMProxy queuing • Application • Framework Mercury Implementation over YARN Q G
- 22. GUARANTEED Request and Allocation start(GUARANTEED, …) request(GUARANTEED, …) allocate(…) rewriting a single parameter
- 23. QUEUEABLE Request and Allocation start(QUEUEABLE, …) request(QUEUEABLE, …)allocate(…) unique token we respect YARN’s security guarantees
- 24. Task Execution: Conflict Resolution two priorities types of schedulers shared resources [YARN-2883]
- 25. Issues with QUEUEABLE containers?
- 26. Application Policies • • container type to be requested for each task • • • Choosing QUEUEABLE at job level enables opportunistic jobs [YARN-2887]
- 28. rebalance reordering job arrival time QUEUEABLE containers per node Load Shaping Policies Mercury Runtime Mercury Runtime Mercury Runtime
- 31. Task Throughput for Increasing Task Duration • •
- 32. Cosmos-based Workload: Task Throughput •
- 35. Future Work
- 36. OSS Overview: Apache JIRA YARN-2877 Extend YARN to support distributed scheduling • • • • • • • •