Virtualizing Hadoop
0 likes843 views
The document discusses virtualizing Hadoop and provides details on: 1) Current and projected usage of Hadoop shows a trend toward more virtualized deployments on-premises and in the public cloud. 2) Different virtualization platform scenarios for Hadoop including using shared storage, storage appliances, or local disks. Common virtualization platforms that can be used are VMware vSphere and OpenStack. 3) Example virtualization architectures showing how Hadoop could be deployed with shared NAS storage or using direct-attached storage on each virtual machine.
Virtualizing Hadoop
- 2. HADOOP USAGE 3 40% 28% 39% 51% 21% 21% Today In 2 Years On public cloud infrastructure such as AWS or Google Virtualized servers in your data center Unvirtualized servers in your data center Off-premise pCAGR: 1% On-premise, Virtualized pCAGR: 14% On-premise, Unvirtualized qCAGR: -16% 26% 21% 8% 30% 9% 5% 2% 0% Currently use Actively evaluating Have evaluated but decided not to use May consider it in the future No interest whatsoever Never heard of it Don't Know Other Source: Internal VMware Core Metrics Study, July 2015
- 3. COMMODITY VS. APPLIANCE VIRTUALIZATION HARDWARE
- 4. VIRTUALIZATION PLATFORM SCENARIO 1 ▸ SAN Storage (LUN) ▸ Generic Blade Servers for Compute ▸ 1/10Gbe Network ▸ vm sizes are typically small ▸ 4 vCPU ▸ 32GB vRAM
- 5. VIRTUALIZATION PLATFORM SCENARIO 2 ▸ Storage Appliance for Hadoop ▸ EMC Isilon ▸ NetApp Open Solution ▸ Purpose-built Virtualization Blade Servers for Compute ▸ Fabric Interconnect/Infiniband ▸ vm sizes are typically bigger ▸ up to 16 vCPU ▸ up to 120GB vRAM
- 6. VIRTUALIZATION PLATFORM SCENARIO 3 ▸ Local Storage for Hadoop ▸ Rack Mounted Servers ▸ 1/10Gbe Network ▸ vm sizes are typically bigger ▸ up to 16 vCPU ▸ up to 120GB vRAM
- 7. VIRTUALIZATION PLATFORM OF CHOICE COMMON CHOICE ▸ VMWare vSphere ▸ ahead of the curve, a lot more mature ▸ BDE provisions Hadoop ▸ OpenStack ▸ new, only open source choice which provides a lot of promise
- 8. CAN WE USE IT FOR POC, DEV, UAT, PROD??? THE ANSWER IS YES.
- 10. QUICK REVIEW ON HADOOP ARCHITECTURE HADOOP ARCHITECTURE Worker Node 1 Worker Node 2 Worker Node 3 Input File ResourcemanagerJob Datanode Nodemanager Split 1 – 64MB AppMaster - 1 Split 2 – 64MB Split 3 – 64MB Nodemanager Nodemanager Datanode Datanode Block 1 – 64MB Block 2 – 64MB Block 3 – 64MB Container - 2 Container - 3 Namenode Master Roles Image credit: VMware
- 11. VIRTUALIZATION ARCHITECTURE HADOOP WITH ISILON Shared storage/NAS Hadoop Virtual Node 2 NN NN NN NN NN NN data node Isilon Virtualization Host VMDKOS Image – VMDKOS Image – VMDK VMDK VMDK Hadoop Virtual Node 1 Ext4 Resourcemanager Ext4 Temp OS Image – VMDK Ext4 Nodemanager Ext4 Hadoop Virtual Node 3 Ext4 Nodemanager Ext4 Temp Image credit: VMware
- 12. VIRTUALIZATION ARCHITECTURE DAS WITH HADOOP Virtualization Host Server VMDK Hadoop Node 1 Virtual Machine Datanode Ext4 Nodemanager Ext4 Ext4 Ext4 Six Local DAS disks per Virtual Machine VMDK VMDK VMDK VMDK VMDK VMDK VMDK Hadoop Node 2 Virtual Machine Datanode Ext4 Nodemanager Ext4 Ext4 Ext4Ext4 VMDKVMDK VMDKVMDK Ext4Ext4Ext4 Image credit: VMware
- 13. VIRTUALIZATION ARCHITECTURE STORAGE DISK LAYOUT vSAN Ext4 Master Role VMDK OS image Hadoop master node Local Disks Hypervisor vSAN Ext4 Ext4 Ext4 Datanode Node Manager VMDK VMDK VMDK OS image Hadoop slave node Virtual machine Hardware Image credit: VMware
- 14. SOME BENCHMARKS SPHERE 6 RESULTS - 32 HOSTS, 23 DISKS PER HOST - 2014 REPORT CONFIDENTIALhttp://www.vmware.com/resources/techresources/10452
- 15. VM LAYOUT MATTERS LARGE DEPLOYMENT ARCHITECTURE
- 16. DEPLOYMENT LAYOUT LAYOUT 1: ONE VSPHERE CLUSTER PER RACK Rack01 Rack02 Rack03 Rack04 Rack05 Rack06 Rack07 Rack08 Cluster01 Cluster02 Cluster03 Cluster04 Cluster05 Cluster06 Cluster07 Cluster08 host001 host005 host009 host013 host017 host021 host025 host029 host002 host006 host010 host014 host018 host022 host026 host030 host003 host007 host011 host015 host019 host023 host027 host031 host004 host008 host012 host016 host020 host024 host028 host032 host033 host037 host041 host045 host049 host053 host057 host061 host034 host038 host042 host046 host050 host054 host058 host062 host035 host039 host043 host047 host051 host055 host059 host063 host036 host040 host044 host048 host052 host056 host060 host064 host065 host069 host073 host077 host081 host085 host089 host093 host066 host070 host074 host078 host082 host086 host090 host094 host067 host071 host075 host079 host083 host087 host091 host095 host068 host072 host076 host080 host084 host088 host092 host096 host097 host101 host105 host109 host113 host117 host121 host125 host098 host102 host106 host110 host114 host118 host122 host126 host099 host103 host107 host111 host115 host119 host123 host127 host100 host104 host108 host112 host116 host120 host124 host128 host129 host133 host137 host141 host145 host149 host153 host157 host130 host134 host138 host142 host146 host150 host154 host158 host131 host135 host139 host143 host147 host151 host155 host159 host132 host136 host140 host144 host148 host152 host156 host160 Image credit: VMware
- 17. DEPLOYMENT LAYOUT LAYOUT 2: CROSS-RACK CLUSTER LAYOUT Rack01 Rack02 Rack03 Rack04 Rack05 Rack06 Rack07 Rack08 Clusrter1 host001 host005 host009 host013 host017 host021 host025 host029 host002 host006 host010 host014 host018 host022 host026 host030 host003 host007 host011 host015 host019 host023 host027 host031 host004 host008 host012 host016 host020 host024 host028 host032 Cluster2 host033 host037 host041 host045 host049 host053 host057 host061 host034 host038 host042 host046 host050 host054 host058 host062 host035 host039 host043 host047 host051 host055 host059 host063 host036 host040 host044 host048 host052 host056 host060 host064 Cluster3 host065 host069 host073 host077 host081 host085 host089 host093 host066 host070 host074 host078 host082 host086 host090 host094 host067 host071 host075 host079 host083 host087 host091 host095 host068 host072 host076 host080 host084 host088 host092 host096 Cluster4 host097 host101 host105 host109 host113 host117 host121 host125 host098 host102 host106 host110 host114 host118 host122 host126 host099 host103 host107 host111 host115 host119 host123 host127 host100 host104 host108 host112 host116 host120 host124 host128 Cluster5 host129 host133 host137 host141 host145 host149 host153 host157 host130 host134 host138 host142 host146 host150 host154 host158 host131 host135 host139 host143 host147 host151 host155 host159 host132 host136 host140 host144 host148 host152 host156 host160 Image credit: VMware
- 18. DEPLOYMENT LAYOUT VIRTUAL MACHINE ROLES - MASTERS AND CLIENTS Master VMs host001 host037 host073 host109 host145 mst01 mst02 mst03 mst04 mst05 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 NAMENODE RESOURCEMANAGER HIVE_METASTORE OOZIE_SERVER NAGIOS_SERVER RESOURCEMANAGER NAMENODE HIVE_SERVER FALCON_SERVER GANGLIA_SERVER JOURNALNODE JOURNALNODE JOURNALNODE OOZIE_SERVER ZKFC ZKFC MYSQL_SERVER* APP_TIMELINE_SERVER* HISTORYSERVER WEBHCAT_SERVER* SECONDARY_NAMENODE* ZOOKEEPER_SERVER ZOOKEEPER_SERVER ZOOKEEPER_SERVER ZOOKEEPER_SERVER ZOOKEEPER_SERVER GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR cln01 cln01 cln01 cln01 cln01 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 Disk:8192GB, RAM:120GB, vCPU:16 PIG PIG PIG PIG PIG SQOOP SQOOP SQOOP SQOOP SQOOP HIVE_CLIENT HIVE_CLIENT HIVE_CLIENT HIVE_CLIENT HIVE_CLIENT MAPREDUCE2_CLIENT MAPREDUCE2_CLIENT MAPREDUCE2_CLIENT MAPREDUCE2_CLIENT MAPREDUCE2_CLIENT HDFS_CLIENT HDFS_CLIENT HDFS_CLIENT HDFS_CLIENT HDFS_CLIENT YARN_CLIENT YARN_CLIENT YARN_CLIENT YARN_CLIENT YARN_CLIENT ZOOKEEPER_CLIENT ZOOKEEPER_CLIENT ZOOKEEPER_CLIENT ZOOKEEPER_CLIENT ZOOKEEPER_CLIENT OOZIE_CLIENT OOZIE_CLIENT OOZIE_CLIENT OOZIE_CLIENT OOZIE_CLIENT FALCON_CLIENT FALCON_CLIENT FALCON_CLIENT FALCON_CLIENT FALCON_CLIENT GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR Image credit: VMware
- 19. DEPLOYMENT LAYOUT VIRTUAL MACHINE ROLES - WORKERS Workers host002 host003 host159 host160 wrk01 wrk01 wrk01 wrk01 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 NODEMANAGER NODEMANAGER NODEMANAGER NODEMANAGER DATANODE DATANODE DATANODE DATANODE GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR … wrk02 wrk02 wrk02 wrk02 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 NODEMANAGER NODEMANAGER NODEMANAGER NODEMANAGER DATANODE DATANODE DATANODE DATANODE GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR host225 host226 host239 host240 wrk01 wrk01 wrk01 wrk01 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 NODEMANAGER NODEMANAGER NODEMANAGER NODEMANAGER DATANODE DATANODE DATANODE DATANODE GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR … wrk02 wrk02 wrk02 wrk02 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 Disk:8192GB, RAM:120GB, vCPU: 16 NODEMANAGER NODEMANAGER NODEMANAGER NODEMANAGER DATANODE DATANODE DATANODE DATANODE GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR GANGLIA_MONITOR Image credit: VMware
- 20. DEPLOYMENT LAYOUT LAYOUT 3: EXPANDED RACK LAYOUT (HADOOP/ANALYTICS APPS) Rack09 Rack10 Rack11 Rack12 Rack13 Rack14 Rack15 Rack16 Cluster6 host161 host165 host169 host173 host177 host181 host185 host189 host162 host166 host170 host174 host178 host182 host186 host190 host163 host167 host171 host175 host179 host183 host187 host191 host164 host168 host172 host176 host180 host184 host188 host192 Cluster7 host193 host197 host201 host205 host209 host213 host217 host221 host194 host198 host202 host206 host210 host214 host218 host222 host195 host199 host203 host207 host211 host215 host219 host223 host196 host200 host204 host208 host212 host216 host220 host224 Cluster8 host225 host227 host229 host231 host233 host235 host237 host239 host226 host228 host230 host232 host234 host236 host238 host240 ESXi cluster Power rack Master Node Worker Node 1:1 High Mem Master Node (AnalyKcsApp) Worker Node (AnalyKcsApp) Image credit: VMware
- 21. ?