A New "Sparkitecture" for modernizing your data warehouse
Download as PPTX, PDF2 likes987 views
1) Enterprises struggle to manage big data with existing technologies due to more systems, complexity, and data to handle. 2) HPE proposes a new "Sparkitecture" called the HPE Elastic Platform for Analytics to address these issues. It uses a data-centric foundation to consolidate all data and applications on a single, elastic platform for analytics workloads. 3) The platform offers workload-optimized systems that provide better performance, scalability, and economics than traditional Hadoop architectures.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to A New "Sparkitecture" for modernizing your data warehouse (20)
Ad
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded (20)
A New "Sparkitecture" for modernizing your data warehouse
- 1. A New “Sparkitecture” for Modernizing Your Data Warehouse Ranga Nathan (Big Data Solutions Product Management) Jack Gudenkauf (Big Data Professional Services Architect) June 9, 2016
- 2. 40% of Enterprises struggle to identify, integrate, and Manage Big Data with existing technology More systems to manageMore complexity to integrateMore data to identify Barriers:
- 3. Hadoop and the data lake 3 App App App Hadoop Data Lake Vision: • Data-centric foundation for all data and apps • Elastic data management & compute platform for all data • Single platform for all analytical workloads Reality: • Data swamps due to lack of oversight and data governance • Dearth of skilled resources to extract value from the data • Sub-optimal performance with traditional architectures • Cannot scale to handle multi-tenant workload complexityData Ponds
- 4. Conventional Wisdom Regarding Deploying a Data Lake Infrastructure 4 Use Cases: ProLiant DL380 Apollo 4530 Apollo 4200 − Traditional Hadoop architecture − Batch workloads with predictable growth − Lowers Big Data Costs for larger deployments − Match compute to workload − Large internal storage − Ideal for large data volumes and batch workloads where density or cost per GB is key Density-OptimizedTraditional Data Lakes & Hubs • Ingestion of multiple types and sources of data • Aggregation, Transformation and Visualization • Batch, Interactive, Real-time workloads Data Warehouse Modernization • Data Staging & landing zone • Migration of operational data stores • Active archiving • Batch workloads
- 5. A Big Data Journey… ETL Offload Archival Deep Learning Event Processing In Memory Analytics
- 6. HPE Elastic Platform for Analytics Flexible Convergence for Big Data Workloads Low Latency Compute Event Processing Moonshot M710P Big Memory Compute In Memory Analytics Apollo xl170r w 512G memory Archival Storage Apollo 4200 w 6TB HDD High Latency Compute ETL Offload and Archival Apollo xl170 w 256G memory HPC Compute Deep Learning Apollo xl190r w GPUs HDFS Storage Apollo 4200 w 3TB HDD
- 7. Unique ValueHPE Workload- and Density-Optimized (WDO) Solution HPE Elastic Platform for Analytics Innovation delivering unique value to customers and the open source community Data Consolidation − Shared storage pool for multiple Big Data environments Maximum Elasticity − Dynamic cluster provisioning from compute pools without repartitioning data Flexible Scalability − Scale compute and storage independently Breakthrough Economics − Workload optimized components for better density, cost and power Ethernet HPE Apollo 4xx0 HPE Moonshot or HPE Apollo 7
- 8. Building blocks for the HPE Elastic Platform for Analytics 8 HP Apollo 2000 System HP Apollo 4200 Scalable System A density optimized compute platform that offers double the density of traditional 1U servers and high memory/core ratios A cost-effective industry standard storage server purpose built for big data with converged infrastructure that offers high density energy-efficient storage HP Moonshot System A complete server system engineered for specific workloads and delivered in a dense, energy-efficient package HP ProLiant DL300 System The industry’s most popular server balances the latest compute and memory technologies with internal storage and flash options, coupled with industry-leading management and serviceability Workload-optimized compute nodes for Spark, Hive/Tez, MapReduce, YARN, Vertica SQL on Hadoop, and other analytics and batch workload Workload-optimized compute nodes for Hbase, Kafka and other low-latency, streaming workloads. Ideal for highest density + lowest power requirements Workload-optimized storage nodes for HDFS, Kudu, and building a multi-temperate data lake environment. Ideal for EDW Offload workloads, and a foundation storage block for workload consolidation Balanced worker node for batch and single- function use cases. Scalable storage node, with smaller fault domain than Apollo 4200, for workload consolidation use cases.
- 9. Yes, But Does It Perform? ComputeNodes StorageNodes HDFS Hyperscale Hadoop Ethernet w/o RoCE StorageNodes HDFS Conventional Hadoop MapReduce Read 9.2GB/Sec 4 – worker nodes 4 - Storage nodes MapReduce Write 7.4GB/Sec Moonshot with 45 - M710 Read 4.9GB/Sec Write 3.4GB/Sec
- 10. Comparing Configurations – Single Subject Data Mart Use Case Normalized on CPU and list price Balanced 18 Worker Nodes (Symmetric) ProLiant DL380 Gen9 BDO 6 Worker Node Blocks (18 nodes) Apollo 4530 Same performance (SpecInt) 7% higher $/SpecInt WDO 16 Compute nodes + 4 Storage Nodes Apollo 2000/4200 6% better performance (SpecInt) 2% lower $/SpecInt 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 UID UID UID 21 UID 21 UID 21 UID Apollo 4500 Gen9 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A Normalized on: List price SpecInt
- 11. Hyperscale Price/Performance Compared with Conventional Cluster 11 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 Green=40Gbps, Yellow=10GbpsQSFP+4847 49 51 50 52 343332311817161521 1/10GBASE-TGreen=10Gbps, Yellow=1Gbps SYS HP 5900 Series Switch JG336A Green=40Gbps, Yellow=10GbpsQSFP+4847 49 51 50 52 343332311817161521 1/10GBASE-TGreen=10Gbps, Yellow=1Gbps SYS HP 5900 Series Switch JG336A UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID ProLiant DL380p Gen8 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric 32313029282726252423222120191817161514131211654321 SYS Green=40Gbps,Yellow=10GbpsQSFP+ 5930 Series Switch JG726A HP FlexFabric SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K SAS 1.2 TB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB SATA 7.2K 4.0 TB 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K 10 9 8 7 6 14 13 12 11 19 18 17 16 15 24 23 22 21 20 5 4 3 2 1 UID Apollo 2000 System SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K SATA 1.0 TB 7.2K 14% 31% 22% 14% 13% 18% 16% 15% 5% 7% 27% 2x -3% 70% 6x HadoopAggregation HadoopJoin HadoopScan HadoopSort HadoopTerasort HadoopWordcount HadoopGzip HadoopGunzip hperead HPeWrite ScalaSparkAggregat… ScalaSparkJoin ScalaSparkScan ScalaSparkBayes ScalaSparkPagerank Normalized on list price and storage capacity ProLiant DL380 Apollo 2000 + Apollo 4200
- 12. HOT Data BALANCED COLD Data Independent scaling of Compute and Storage HOT Data * − 2.8x compute − 97% of the storage capacity − 4x the memory Balanced * − 1.6x compute − 1.5x the storage capacity − 2.5x the memory Cold Data * − 0.9x of the compute − 2.1x the storage capacity − 1.5x the memory Hyperscale vs. Conventional Scale-out 12 * Compared with balanced, conventional full rack cluster
- 13. Hyperscale benefits for Big Data Hadoop Labels feature (jira Yarn-796) – Contributed node labels concepts into Hadoop (Apache 2.6 trunk) – Allows scheduling of YARN containers to specific pools of nodes – Combined with Hyperscale approach, compute nodes can be dynamically assigned because no data needs to be repartitioned Hadoop Cluster 1 Hive/Tez Vertica SQL on Hadoop Data prep (12am – 6am) Predictive analytics (6am – 12am) Spark Hadoop Cluster 1(High Power) Storage Node Storage Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node
- 14. 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 UID 2 1 4 3 6 5 7 8 ProLiant DL360 Gen9 SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K SAS 900 GB 10K UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 UID Tray 2 22191613 24211815 10741 12963 Tray 1 Pull for tray 2Pull for tray 2 Apollo 4200 Gen9 SYS161521 32311817 10/100/1000Base-T 48473433 5049 5251 5453 Speed: Green=1000Mbps, Yellow=10/100Mbps SFP+ QSFP+ Green=10Gbps, Yellow=1Gbps Green=40Gbps, Yellow=10Gbps HP 5900 Series Switch JG510A A Modern “Sparkitecture” for Real-time Analytics (SMACK) 14 Cassandra Mongo HDFS Kafka Orc OracleVertica Spark Sparksql Batch ML Streaming graph Parquet HANA Vora
- 15. Vertica Cluster MPP Columnar DW Vertica DW Vertica DW(s) Vertica Cluster Multi-DC DR Vertica DW Vertica DW(s) Kafka (Mirrored) Dual Ingestion Pipeline Vertica Spark Vertica Kafka Connector HDFS /AVRO/Topic/Y YYY/MM/DD/H H Semi-Structured Data HDFS /ORC/Topic/YY YY/MM/DD Structured Data Non-relational Hadoop Cluster (Hive) Hortonworks, Cloudera, AWS 1 2 3 4 5 6 78 1 0 11 9 Spark Parallel Streaming Transformations Near real-time Transform/ReShape Mapping to Vertica SOT Scala, Java, Python, SQL Hadoop™ Spark™ Cluster Spark Vertica Loader Web Mobile IoT MySQL Applications Centralized Data HubKafka™ Cluster DataCenterBoundar Parallel Streaming Transformation Loader Confluent™ REST Proxy JSON/AVRO Messages Kafka Connect
- 16. Infrastructure trends affecting Big Data architecture Workload optimization Low-power SoC’s and other accelerators giving rise to workload optimized servers Faster network fabric Dramatic increase in fabric speeds Multi-temperate storage Enterprise adoption of tiering accelerated (NVMe, flash, etc.) storage Container-based Apps Running multiple container apps while hosting a common resource management (YARN) Big Data
- 17. Elastic Platform for Analytics long term view Evolve to support multiple compute and storage blocks Low Cost Nodes SSD Nodes Disk Nodes Archive Nodes Multi-temperate, Density Optimized Storage using HDFS Tiering, NoSQLs and Objectstores GPU Nodes FPGA Nodes Big Memory Nodes Density and Workload Optimized compute nodes to accelerate various big data software
- 18. Delivering a Scalable Data Lake for the Enterprise • Unlock the most value and performance from Hadoop • Scale without compromising data security, reliability, and ROI • Enterprise-Grade, Trusted, and Proven HPE solution Optimize the Hadoop Data Lake for More Business Value Elastic Platform for Analytics (Workload and Density Optimized) High-Performing Analytics Engines for Hadoop Consulting & Implementation Services for Hadoop Data Security for Hadoop
- 19. Thank you 19