What's new in Hadoop Yarn- Dec 2014
Download as PPTX, PDF•1 like•899 views
The document provides an overview of new features introduced in Hadoop YARN 2.6, including enhanced fault tolerance, support for long-running services, and improvements in workload scheduling. Key innovations include CPU scheduling, cgroup isolation, and security enhancements such as Kerberos token renewal. It also highlights the capability for rolling upgrades and log aggregation to ensure reliable and secure operations.
What's new in Hadoop Yarn- Dec 2014
- 1. Page1 © Hortonworks Inc. 2014 What’s new in YARN Hortonworks. We do Hadoop.
- 2. Page2 © Hortonworks Inc. 2014 Speaker Varun Vasudev Hortonworks Work on Hadoop YARN [email protected]
- 3. Page3 © Hortonworks Inc. 2014 Agenda • Overview of YARN • New YARN Innovation in Hadoop 2.6 – Rolling upgrades – Added fault tolerance – CPU scheduling in Capacity Scheduler – C-Group isolation – Node labels – Support for long running services • Q & A
- 4. Page4 © Hortonworks Inc. 2014 Overview of YARN
- 5. Page5 © Hortonworks Inc. 2014 What is YARN? YARN : Data Operating System °1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °° ° ° ° ° ° ° ° ° ° ° ° ° ° N Batch MapReduce Batch & Interactive Tez Real-Time Slider Direct Java .NET Scripting Pig SQL Hive Cascading Java Scala NoSQL HBase Accumulo Stream Storm Other ISV Other ISV Others Spark Other ISV
- 6. Page6 © Hortonworks Inc. 2014 What’s new in Hadoop YARN 2.6?
- 7. Page7 © Hortonworks Inc. 2014 YARN in Hadoop 2.6: What’s New • Security: Kerberos Token Renewal • Log Aggregation: View logs for running applications • Fault Tolerance: AM Retry and Container keep alive • Service Registry: Directory for services running in YARN Long Running Services Support THEME • CPU Scheduling in Capacity Scheduler • CPU Isolation through CGroups • Node Labels for scheduling constraints Workload Scheduling THEME • YARN Rolling Upgrades support • Work Preserving Restart • Timeline Server support in Secure clusters Reliable and Secure Operations THEME
- 8. Page8 © Hortonworks Inc. 2014 New in Hadoop YARN 2.6: Long Running Service Support
- 9. Page9 © Hortonworks Inc. 2014 YARN Supports Multiple Workloads YARN : Data Operating System °1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Batch MapReduce Batch & Interactive Tez Real-Time Slider Direct Java .NET Scripting Pig SQL Hive Cascading Java Scala NoSQL HBase Accumulo Stream Storm Other ISV Other ISV Application Others Spark Other ISV
- 10. Page10 © Hortonworks Inc. 2014 Enhancements to Support Long Running Services • YARN-941: YARN updates Kerberos token for a Long Running Service after the token expires Security Capability • YARN-2468: Aggregate and capture logs during the lifetime of a Long Running Service Log Aggregation Capability • YARN-1489: When ApplicationMaster(AM) restarts, do not kill all associated containers – reconnect to the AM • YARN-611/YARN-614: Tolerate AM failures for Long Running Services Fault Tolerance Capability • YARN-913: Service Registry that publishes host and ports that each service comes up on Service Registry Capability
- 11. Page11 © Hortonworks Inc. 2014 New in Hadoop YARN 2.6: Workload Scheduling
- 12. Page12 © Hortonworks Inc. 2014 Node Labels: Apply Node Constraints A App L1 L1 L1 Deploy/Allocate B App L1 L1 L1 Isolate A A A nodes labels
- 13. Page13 © Hortonworks Inc. 2014 CPU Scheduling in Capacity Scheduler What • Admin tells YARN how much CPU capacity is available in cluster • Applications specify CPU capacity needed for each container • YARN Capacity Scheduler schedules application taking CPU capacity availability into account Why • Applications (for example Storm, HBase, Machine Learning) need predictable access to CPU as a resource • CPU has become bottleneck instead of memory in certain clusters (128 GB RAM, 6 CPUs)
- 14. Page14 © Hortonworks Inc. 2014 CGroup Isolation What • Admin enables CGroups for CPU Isolation for all YARN application workloads Why • Applications need guaranteed access to CPU resources • To ensure SLAs, need to enforce CPU allocations given to an Application container
- 15. Page15 © Hortonworks Inc. 2014 New in Hadoop YARN 2.6: Reliable and Secure Operations
- 16. Page16 © Hortonworks Inc. 2014 YARN Rolling Upgrades Support • No Service Disruption – Highly Available Resource Manager – Job submission automatically retries during failover • No Service Degradation – Resource Manager Restart with Work Preservation – Node Manager Restart with Container Preservation • Preserve State – Preserve Application Queue – Preserve Application History – Preserve Node/Container state
- 17. Page17 © Hortonworks Inc. 2014 Secure Timeline Server Support App Timeline Server AMBARI Custom App Monitoring Client HTTPs HTTPs
- 18. Page18 © Hortonworks Inc. 2014 Q & A