Managing your Hadoop Clusters with Ambari
11 likes8,597 views
Apache Ambari provides a 100% open source and intuitive set of tools to monitor, manage and efficiently provision your Apache Hadoop cluster. Ambari simplifies the operation and hides the complexity of Hadoop, making Hadoop appear like a single, cohesive data platform. Hadoop cluster provisioning and ongoing management can be a complicated task, especially when there are hundreds or thousands of nodes involved. Ambari allows you to control Hadoop cluster services from a single point. In this session, we will provide an overview of the Apache Ambari key features, architecture and web service-based APIs.
![Discover.hdp2.2.ambari.final[1]](https://cdn.slidesharecdn.com/ss_thumbnails/discover-150114162536-conversion-gate01-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (7)
Ad
Similar to Managing your Hadoop Clusters with Ambari (20)
Ad
More from DataWorks Summit (20)
Recently uploaded (20)
Managing your Hadoop Clusters with Ambari
- 1. Apache Ambari – Management, Monitoring and Operations Mahadev Konar Co Founder & Software Engineer @mahadevkonar (@hortonworks) Pramod Thangali Director of Engineering @pramodthangali (@hortonworks) Page 1
- 2. Hello! • Hadoop – Contributor since 2006 – Committer/PMC • ZooKeeper – Contributor since 2008 – Committer/PMC • Ambari – Contributor since 2012 – Committer/PPMC Page 2
- 3. Ambari Easiest way to manage Hadoop Clusters Cluster Operations Job Diagnostics Extensible Platform Core capabilities to include the Insight into job performance and Integration and customization critical tasks associated with reduce the burden on points so Hadoop can provisioning and operating specialized Hadoop skills and interoperate with existing Hadoop clusters. knowledge. operational tooling. Page 3
- 4. Architecture Web Client REST API JS Ambari /clusters Web Configurable java Ambari Auth Provider Server Auth Provider Cluster REST API Configurations RDBMS Request Dispatcher LDAP DB User postgres Orchestrator Monitoring Repo Bootstrap or Manual install python Ambari Ganglia php puppet Agent/s Collector Page 4
- 5. API’s API’s and API’s http://{your.ambari.server}:8080 Clusters /api/v1/clusters Services /api/v1/clusters/mycluster/services Components /api/v1/clusters/mycluster/services/hdfs/components/namenode Hosts /api/v1/clusters/mycluster/hosts Host-Components /api/v1/clusters/mycluster/hosts/host1/host_components https://github.com/apache/ambari/blob/branch-1.2/ambari-server/docs/api/v1/index.md Page 5
- 6. How to plug into the API’s • Consistent front-end REST API for –Accessing Hadoop and System metrics –Managing cluster resources • Service provider plugin architecture to enable –Adding new components and resources –Providing alternative service providers to manage existing components < Place tittle Here by using Header and Footer Options > Page 6
- 7. Enabling Integration Scenarios • Expand REST API “ZERO TOUCH” INSTALLS Blueprint Ambari Cluster “LIGHTS OUT” WORKFLOWS Alarm Decom Inform AVOID CONFIGURATION “DRIFT” Change Apply Callback “BRING MY OWN” SCENARIOS Custom Provider Page 7
- 8. Ambari + Teradata Viewpoint Integration • Integrate with Ambari to provide Monitoring REST API • Use Ganglia for metrics collection • No Ambari mgmt controls • No Ambari UI • No Ambari Agents P HDP Cluster Ambari Custom P Page 8
- 9. Extending Ambari – Adding new services • Stack Definitions – Define which Services are available (services) – Define where to get the packages (repos) repos Stack A services S S S S S repos Stack B services S S S S repos Stack C extends Stack A S S S S S S services + Page 9
- 10. Demo Page 10
- 11. Ambari RoadMap • Releases –Ambari 1.2.0 –Ambari 1.2.1 –Ambari 1.2.2 (under vote) –Ambari 1.2.3 (under development for stability fixes) • Ambari 1.3.0 • Ambari 1.4.0 • Futures < Place tittle Here by using Header and Footer Options > Page 11
- 12. Ambari 1.2.2 – whats new • Bug Fixes to 1.2 line • Host-level Alerts • Paging Controls on Zoomed Graphs • Change Ambari Web Port # • Support for AD Authentication • Quicker Install/Start/Test progress display • Upgrade Paths – 1.2.0 1.2.2 and 1.2.1 1.2.2 • Performance – Optimized deployment + monitoring on clusters < Place tittle Here by using Header and Footer Options > Page 12
- 13. Host Level Alerts • Host-specific alerts • For example: – TaskTracker process, DataNode process, DataNode Storage < Place tittle Here by using Header and Footer Options > Page 13
- 14. Paging Controls on Zoomed Graphs Last 1 Hour Last 12 Hours Last 2 Hours Last 24 Hours Last 4 Hours Last 1 Week Page 14
- 15. Ambari 1.3 • Manage Kerberos Secure Cluster • Managed Stack Upgrade • Disaster protection service via HDFS mirroring • Multi-tenancy support via capacity Scheduler • Improved Configuration Mgmt with host-level controls • Master service mobility to ease hardware maintenance < Place tittle Here by using Header and Footer Options > Page 15
- 16. Ambari 1.3 continued • External group mappings (LDAP/AD) • Support API and external driven cluster Installation Management • Enhanced Job Diagnostic visualizations • HUE Support for Hadoop data workers Page 16
- 17. Multi Tenancy Support • Compute resource quotas – Managed via Capacity Scheduler and used in PROD at Yahoo! – Basic configuration support in Ambari 1.3 – Visualization for Queue Diagnostics in Ambari 1.4 (target) Page 17
- 18. Multi Tenancy Support – User and Data Management • Disk and Name space quotas • Support for quota mgmt in Ambari 2.x (target) Page 18
- 19. Release Schedule and Info • Ambari 1.2.2 (pending release) • Ambari 1.2.3 (2-3 weeks) • Ambari 1.3.0 (May) • Ambari 1.4.0 (TBD) • Links – Website – http://incubator.apache.org/ambari/ – Mailing Lists – http://incubator.apache.org/ambari/mail-lists.html – Development Wiki – https://cwiki.apache.org/confluence/display/AMBARI – Looking for Apache contributors < Place tittle Here by using Header and Footer Options > Page 19
- 20. Thank you. @mahadevkonar @pramodthangali Architecting the Future of Big Data Page © Hortonworks Inc. 2012