Managing Hadoop, HBase and Storm Clusters at Yahoo Scale
Download as PPTX, PDF4 likes1,296 views
The document discusses managing Hadoop, HBase and Storm clusters at Yahoo scale. It describes Yahoo's grid infrastructure which includes 3 data centers with over 45k nodes across 18 Hadoop clusters, 9 HBase clusters and 13 Storm clusters. It then provides details on the rolling upgrade processes for HDFS, YARN, HBase and Storm which involve minimizing downtime, upgrading components independently and verifying upgrades. CI/CD processes are used to automate software deployment and upgrades.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Managing Hadoop, HBase and Storm Clusters at Yahoo Scale (20)
![[Hadoop Meetup] Apache Hadoop 3 community update - Rohith Sharma](https://cdn.slidesharecdn.com/ss_thumbnails/apachehadoop3communityupdate-rohith-171222071357-thumbnail.jpg?width=560&fit=bounds)
Ad
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded (20)
Managing Hadoop, HBase and Storm Clusters at Yahoo Scale
- 1. Managing Hadoop, HBase and Storm Clusters at Yahoo Scale PRESENTED BY Dheeraj Kapur, Savitha Ravikrishnan⎪June 30, 2016
- 2. Agenda Topic Speaker(s) Introduction, HDFS RU, HBase RU & Storm RU Dheeraj Kapur YARN RU, Component RU, Distributed Cache & Sharelib Savitha Ravikrishnan Q&A All Presenters HadoopSummit 2016
- 4. Grid Infrastructure at Yahoo HadoopSummit 2016 ▪ A multi-tenant, secure, distributed compute and storage environment, based on Hadoop stack for large scale data processing ▪ 3 data centers, over 45k physical nodes. ▪ 18 YARN (Hadoop) clusters, having 350 to 5200 nodes. ▪ 9 HBase clusters, having 80 to 1080 nodes. ▪ 13 Storm clusters, having 40 to 250 nodes
- 5. Grid Stack Zookeeper Backend Support Hadoop Storage Hadoop Compute Hadoop Services Support Shop Monitoring Starling for logging HDFS Hbase as NoSql store Hcatalog for metadata registry YARN (Mapred) and Tez for Batch processing Storm for stream processing Spark for iterative programming PIG for ETL Hive for SQL Oozie for workflows Proxy services GDM for data Mang Café on Spark for ML
- 6. Deployment Model DataNode NodeManager NameNode R M DataNodes RegionServers NameNode HBase Master Nimbu s Supervisor Administration, Management and Monitoring ZooKeeper Pools HTTP/HDFS/GDM Load Proxies Applications and Data Data Feeds Data Stores Oozie Server HS2/ HCat HadoopSummit 2016
- 7. HDFS
- 8. Hadoop Rolling Upgrade ▪ Complete CI/CD for HDFS and YARN Upgrades ▪ Build software and config “tgz” and push to repo servers ▪ Installs software and configs in pre-deploy phase, activate during upgrade ▪ Slow upgrade 1 node per cycle ▪ Each component is upgraded independently i.e HDFS, YARN & Client HadoopSummit 2016 Release Configs/Bundles: --- doc: This file is auto generated packages: - label: hadoop version: 2.7.2.13.1606200235-20160620-000 - label: conf version: 2.7.2.13.1606200235-20160620-000 - label: gridjdk version: 1.7.0_17.1303042057-20160620-000 - label: yjava_jdk version: 1.8.0_60.51-20160620-000
- 9. Package Download (pre- deploy) RU process Git (release info) Namenode, Datanodes, Resourcemanager HBaseMaster, Regionserver, Gateways Repo Farm Jenkins Start Servers /Cluster ygrid-deploy-software
- 10. CI/CD process Git (release info) Jenkins Start HDFS Upgrade RU process Finalize RU Create Dir Structure Put NN in RU mode SNN Upgrade NN Failover SNN Upgrade foreach DN Select DN Check installed version Stop DN Activate new software Start DN Wait for DN to join Stop/termina te RU on X failures 1 2 3a 3b 3c 4a 4b 4c 4d 4e 4f After 100 hosts are successfully upgraded Check HDFS used %age, Live nodes consistency on NNs Terminate Upgrade incase of more than X failure Involves service and IP failover from NN to SNN and vice versa Safeupgrade-dn
- 11. Hadoop 2.7.x improvements over 2.6.x Performance ▪ Reduce NN failover by parallelizing the quota init ▪ Datanode layout inefficiency causing high I/O load. ▪ Use a offline upgrade script to speed up the layout upgrade. ▪ Adding fake metrics sink to subvert JMX cache fix, causing delays in datanode upgrade/health check. ▪ Improved datanode shutdown speed Failure handling ▪ Reduce the read/write failures by blocking clients until DN is fully initialized.
- 12. YARN
- 13. YARN Rolling Upgrade ▪ Minimize downtime, maximize service availability ▪ Work preserving restart on RM and NM ▪ Retains state for 10mins. ▪ Ensures that applications continuously run during a RM restart ▪ Save state, update software, restart and restore state. ▪ Uses leveldb as state store ▪ After RM restarts, it loads all the application metadata and other credentials from state-store and populates them into memory. HadoopSummit 2016
- 14. CI/CD process Git (release info) Jenkins Start YARN Upgrade RU process Create Dir Structure Resource Manager Upgrade HistoryServer Upgrade Foreach NM Select NM Check installed version Safestop NM (kill -9) Activate new software Start NM Wait for NM to join Stop/termina te RU on X failures Timeline Server Upgrade 1 2 2a 2b 2c 2d 2e 3 4 5 Terminate Upgrade incase of more than X failure
- 15. Distributed cache & Sharelib
- 16. Distributed Cache ▪ Distributed cache distributes application-specific, large, read-only files efficiently. ▪ Applications specify the files to be cached in URLs (hdfs://) in the Job ▪ DistributedCache tracks the modification timestamps of the cached files. ▪ DistributedCache can be used to distribute simple, read-only data or text files and more complex types such as archives and JAR files. HadoopSummit 2016
- 17. Sharelib ▪ "Sharelib" is a management system for a directory in HDFS named /sharelib, which exists on every cluster. ▪ Shared libraries can simplify the deployment and management of applications. ▪ The target directory is /sharelib, under which you will find various things: /sharelib/v1 - where all the packages are • /sharelib/v1/conf - where the unique metafile for the cluster is (and all previous versions) • /sharelib/v1/{tez, pig, ... } - where the package versions are kept ▪ The links/tags (metafile) are unique per cluster. ▪ Grid Ops maintains shared libraries on HDFS of each cluster ▪ Packages in shared libraries include mapreduce, pig, hbase, hcatalog, hive and oozie. HadoopSummit 2016
- 18. Jenkins Start Sharelib Uploader Git Bundles Verify Dist Cache Download toDo packages Dist repo Re-package and upload package Re-generate Meta info (HDFS) Upload to Oozie Sharelib Update Generate clients to update
- 19. Subsystems
- 20. Component Upgrade HadoopSummit 2016 ▪ New Releases : CI environment continuously releases certified builds & their versions. ▪ Generate state : Package rulesets contain the list of core packages and their dependencies for each & every cluster ▪ Deploy cookbooks : contain chef code and configuration that is pushed to Chef server ▪ Deploy pipelines : are YAML files that specify the flow & order of the deploy for every environment/cluster. ▪ Validation jobs : are run after a deploy completes on all the nodes which ensures end-to-end functionality is working as expected.
- 21. Components Upgrade CI process Component versions Git Bundles Certified Releases Rule set files (cluster: component specific) Git bundles Certified package version info Statefiles Build Farms Cookbook, Roles, Env, Attribute files Git (release info) Build Farms Artifactory Ruby (Rake) New Release Package Rulesets Deploy cookbooks A B Build Farms Rspec rubocop, state generate, compare & upload Validate increment version 1 2 3 Chef
- 22. CD process Components Upgrade cont.. Git (release info) Build Farms Statefiles Deploy Pipeline Component Node Ruby (Rake) Min size, zerodowntime check, targetsize, validate Chef-client, cookbook-converge, graceful shutdown and healthcheck 4 Chef A B
- 23. HBase
- 24. HBase Rolling Upgrade Release Configs: default: group: 'all' command: 'start' system: 'ALL' verbose: 'true' retry: 3 upgradeREST: 'false' upgradeGateway: 'true' dryrun: 'false' force: 'false' upgrade_type: 'rolling' skip_nn_upgrade: 'false' skip_master_upgrade: 'false' Workflow definitions: default: continue_on_failure: - broken - badnodes relux.red: - master - default - user - ca_soln-stage - perf,perf2,projects - restALL ▪ Workflow based system. ▪ Complete CI/CD for HDFS and HBase Upgrades ▪ Build tgz and push to repo servers ▪ Installs software before hand, activate new release during upgrade ▪ Each component and Region group is upgraded independently i.e HDFS, group of regionservers.
- 25. CI/CD process Git (release info) Jenkins Start Put NN in RU mode & Upgrade NN SNN Master Upgrade Region- server Upgrade process Stargate Upgrade Gateway Upgrade HBase Upgrade Foreach DN/RS Upgrade regionserver Repo Server Package + conf version Stop Regionserver DN Safeupgrade, Stop DN Upgrade and Start DN Upgrade and Start RS 1 2 3 4 3a 3c 3b 3d 3e 3f 3f 5 HDFS Rolling Upgrade process Iterate over each group Iterate over each server in a group
- 26. STORM
- 27. Storm Rolling Upgrade Release Configs: default: parallel: 10 verbose: 'true' retry: 3 dryrun: 'false' upgrade_type: 'rolling' quarantine: 'true' terminate_on_failure: 'true' sup_failure_threshold: 10 sendmail_to: '[email protected]' sendmail_cc: '[email protected], [email protected]' cluster_workflow: cluster1.colo1: pacemaker_drpc cluster2.colo2: default Workflow Defination: default: rolling_task: - upgradeNimbus - bounceNimbus - upgradeSupervisor - bounceSupervisor - upgradeDRPC - bounceDRPC - upgradeGateways - doGatewayTask - verifySupervisor - runDRPCTestTopology - verifySoftwareVersion full_upgrade_task: - killAllTopologies - specifyOperation_stop - sleep10 - bounceNimbus - bounceSupervisor - bounceDRPC - clearDiskCache - cleanZKP - upgradeNimbus - upgradeSupervisor - upgradeDRPC - specifyOperation_start - bounceNimbus - bounceSupervisor - bounceDRPC - upgradeGateways - doGatewayTask - verifySupervisor - runDRPCTestTopology - verifySoftwareVersion ▪ Complete CI/CD system. Statefiles are build per component and pushed to artifactory before upgrade ▪ Installs software before hand, activate new release during upgrade ▪ Each component is upgraded independently i.e Pacemaker, Nimbus, DRPC & Supervisor
- 28. Storm Upgrade CI/CD process Git (release info) Jenkins Start Artifactory (State files & Release info) RE Jenkins and SD process Pacemaker Upgrade Nimbus Upgrade Supervisor Upgrade Bounce Workers DRPC Upgrade DRPC Upgrade Verify Supervisors Run Test/Validatio n topology Audit All Components RE Jenkins lets to statefile generation for each component and updates git with release info Statefiles are published in artifactory and downloaded during upgrade Upgrade fails if more than X supervisors fails to upgrade
- 29. Rolling Upgrade timeline Component Parallelism Hadoop 2.6.x Hadoop 2.7.x Hbase 0.98.x Storm 0.10.1.x HDFS (4k nodes) 1 4 days 1 day X X YARN (4k nodes) 1 1 day 1 day X X HBase (1k nodes) 1-4 4-5 days X 4-5 days X Storm (350 nodes) 10 X X X 4-6 hrs Components 1 1-2 hrs 1-2 hrs 1-2 hrs X HadoopSummit 2016
- 30. 99.928 99.898 99.940 99.687 99.705 99.600 99.650 99.700 99.750 99.800 99.850 99.900 99.950 100.000 AB DB FB HB IB JB LB PB UB BT LT PT TT UT BR DR IR LR MR PR Rolling Upgrade Impact YTD Availability by Cluster 99.990