Hortonworks on IBM POWER Analytics / AI
0 likes359 views
In this session I will tell you what Hortonworks and IBM Power solutions are and how we can realize significant business value development and prompt use of open innovation in future cognitive utilization. In addition, I will introduce the value added unique to IBM that can be provided by IBM and Hortonworks partnership from the viewpoint of storage, analytics, data science and streaming analysis.
Ad
More Related Content
What's hot (20)
Similar to Hortonworks on IBM POWER Analytics / AI (20)
![[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene](https://cdn.slidesharecdn.com/ss_thumbnails/32apm-160226173205-thumbnail.jpg?width=560&fit=bounds)
Ad
More from DataWorks Summit (20)
Ad
Recently uploaded (20)
Hortonworks on IBM POWER Analytics / AI
- 2. A I 2@ IBM Corporation
- 3. Data Time Available Data Understood Data Enterprise Amnesia 80 million wearable health devices will be available by 2017. 2.5 quintillion bytes of data generated daily by connected machines. There will be 28 times more sensor- enabled devices than people by the year 2020. 25 gigabytes of data per hour is generated by a connected car. 90% of cars will be connected by 2020. 153 exabytes of healthcare data generated by devices in 2013. Increasing to 2,314 exabytes in 2020. 1.7 megabytes of data per second generated by every human being on the planet by 2020.
- 4. & ( ) 4@ IBM Corporation
- 5. 5 2011 2016 26% 3%5% @ IBM Corporation
- 7. Machine Learning Analytics Data AI EDW Optimization – • Hadoop ETL EDW Modernization – + • • EDW Hadoop Data Science – • • Data Science – • • 7@ IBM Corporation
- 8. IBM & Hortonworks: Our Journey f L Hc , C I Md a September 2016 , - C D X February 2017 , 3 B 32 & 32 P April 2017 - M- C D e . F 2 B 32 M - e December 2017 M- C D e June 2017 - F M 3 B 32 e Q W S X 2 2 2 3 3 June/July 2018 &- - M - C D e August 2017 2017 Global Partner of the Year IBM 8@ IBM Corporation
- 9. S P H A h a 9 IBM Consensus Leader in Data Science & Business Analytics Apache Committers Top 20 OPEN SOURCE B I M P h B H ke Bcd 9@ IBM Corporation
- 10. / / I H H D B ( )2 . , 2 ( A P O D C I C F N 10@ IBM Corporation
- 11. / / I H H D B ( )2 . , 2 ( A N P O D C I C F 11@ IBM Corporation
- 12. Freedom: Productivity: / Trust: / IBM Watson Studio Local ibm.com/datascience - 12@ IBM Corporation
- 13. - / Data Scientist - Browser / / A D A D I 13@ IBM Corporation
- 14. R P I E A O S W E G U 439 *s 7 F E A Ia d l g e L (o MT u S )o u t 12 1:E 2 A DD E --1 - F E - D 1 :E 4 I E 0 8 F 4 I E 0 EDE F -10 - D 1 :E 0 F F 14 p hL O 0g Li 0Rw V r bL W n V P Ll
- 16. Graphics Memory POWER8 CPU DDR4 GPU NVLink CPUDDR4 PCIe Graphics Memory PCIe Gen 3 Data Pipe POWER9 NVLink Data Pipe GPU / System Bottleneck Here 16@ IBM Corporation
- 17. 17 C - 5 1 TB Memory Power 9 CPU V100 GPU V100 GPU 170GB/s NVLink 150 GB/s 1 TB Memory Power 9 CPU V100 GPU V100 GPU 170GB/s NVLink 150 GB/s IBM AC922 Power System Deep Learning Server (4-GPU Config) Store Large Models in System Memory Operate on One Layer at a Time Fast Transfer via NVLink @ IBM Corporation
- 18. / / I H H D B ( )2 . , 2 ( A P O D C I C F N 18@ IBM Corporation
- 19. ACTIONABLE INTELLIGENCE MODERN DATA APPLICATIONS / Hortonworks DataFlow Hortonworks Data Platform @ IBM Corporation
- 20. • O D l Ac hg • % k D P P R I e I A hg • t A a r A pnI hg • l H N P Y m R fo A DR O P Aa 0 0 1 0 ns ® @ IBM Corporation
- 21. : S P E F v i c @ IBM Corporation • • // Eo t rp E a s F E e • / / / l n l lFwg m
- 22. / / I H H D B ( )2 . , 2 ( A P O D C I C F N 22@ IBM Corporation
- 23. ) ( ) e / ew / ) ( ) 2 S A / S A y m o w 5 P I A C ) ( s 5 ew tr +++ ++++++ ) ( ) S A 0ew
- 24. © 2018 International Business Machines Corporation 9 - ( I 4 23 OP - / / 9E O / 9 hi C 9 M (M )M M XU RTS VPae 8 28 e ZW RTS 9.5 ZW RTS P e 87>19 M ) BHG 8 EN AEB (1) 2X performance per core is based on IBM Internal measurements as of 2/28/18 on various system configuration and workload environments including (1) Enterprise Database (2.22X per core): 20c S922L (2x10-core/2.9 GHz/256 GB memory): 1,039,365 Ops/sec versus 2-socket Intel Xeon Skylake Gold 6148 (2x20-core/2.4 GHz/256 GB memory): 932,273 Ops/sec. (2) DB2 Warehouse (2.43X per core): 20c S922L (2x10-core/2.9 GHz/512 GB memory): 3242 QpH versus 2-socket Intel Xeon Skylake Platinum 8168 (2x24-core/2.7 GHz/512 GB memory): 3203 QpH. (3) DayTrader 7 (3.19X per core): 24c S924 (2x12-core/3.4 GHz/512 GB memory): 32221.4 tps versus 2-socket Intel Xeon Skylake Platinum 8180 (2x28-core/2.5 GHz/512 GB memory): 23497.4 tps. (2) 2.6X memory capacity is based on 4TB per socket for POWER9 and 1.5TB per socket for x86 Scalable Platform Intel product brief: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/xeon-scalable-platform-brief.pdf?asset=14606 (3) 1.8X bandwidthy is based on 230 GB/sec per socket for POWER9 and 128GB/sec per socket for x86 Scalable Platform Intel product brief: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/xeon-scalable-platform-brief.pdf?asset=14606 (4) 9.5X is based on POWER9 and next-generation NVIDIA NVLink peak transfer rate is 150 GB/sec = 48 lanes x 3.2265625 GB/sec x 64 bit/66 bit encoding compared to x86 PCI Express 3.0 (x16) peak transfer rate is 15.75 GB/sec = 16 lanes X 1GB/sec/lane x 128 bit/130 bit encoding. 87>19 L C 6 4 GE M ) BHG
- 25. / M L B 9 IA B 9 C2 25@ IBM Corporation
- 26. PVHR STHNS 92/(( § ( (A § z ( , ( / 6 § ( 1 , 38 S § ( 55%955 733% 3 W B H § T PODM DGG T PODM RHDR (W 55 0 % 0 0 IPR § ( 1 733 § , 1 3 § ,in 28H 6HO § ( 28H 6 W , 5759 20 8( § ) 28H 6 W. 5759 S FDMM W , § 28H 6 W. 9 § T ) HDR /W 2 A § 9 OUW z § AEUOTU . 9 § 749 - IPR PVHR/ 92/((o 92 n n y rii n08 mb l t ysi C4 / y y • – h nz y n m en y • W., – m 0 DF H DRLz n c , • 28H 6HO bup20 8( n n – • y a xh y k dwu m fxh 26@ IBM Corporation
- 27. 2 A 9 A 1. Results are based IBM Internal Measurements running four concurrent streams of 99 TPC-DS like queries against a 3TB dataset. Results valid as of 4/25/18 and conducted under laboratory condition with speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks on both systems, individual results can vary based on workload size, use of storage subsystems & other conditions 2. Hardware: 4 nodes IBM Power LC922 (2x20-core/2.7 GHz/512 GB memory) using 12 x 8TB HDD, 10 GbE two-port, RHEL 7.5 LE for Power9 and 4 nodes of Intel Xeon Gold 6140; 36 cores (2 x 18c chips) at 2.3 GHz; 512 GB memory, 12 x 8TB HDDs, 10Gbps NIC, Red Hat Enterprise Linux 7.5 3. Software: Apache Spark 2.3.0 located at http://spark.apache.org/downloads.html ; and open source Hadoop HDP 2.7.5 4. Pricing is based on Power LC922 http://www-03.ibm.com/systems/power/hardware/linux-lc.html and publicly available x86 pricing. 5. Apache®, Apache Spark®, and associated logos are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks. Intel Xeon SP Gold 6140 : 22.6 QpH Power LC922 29.5 QpH 30% MORE Performance1 Power LC922 $43,948 Intel Xeon SP Gold 6150 : $53,548 18% LOWER Price2,3,4 Power LC922 Delivers 1.6X +
- 28. . 3 9 Refer to the published Reference Architectures for Local Storage and Shared Storage configurations. 28
- 29. 2017 Global Server Hardware, Server, OS Reliability Report 2% IBM 6% HPE 8% Dell 10% OracleSPARC Servers
- 30. lw + N g W r u y + + ,% 3 6 RP U e cb H U B A M W dN e I nDW fa g p + + +-+ + ( - + + +-+ + ( - + + +-+ + ( - + + +-+ + ( - M b te e I U m so Si dWc W e F X X e O be U t i B ) / + - / - + ) + - N S DCb 0 S E e Xb j e A S S N b X b U ) ) + M W d 30@ IBM Corporation
- 31. D C 1 4 4 / 4 H P M M 6 I8 C S Et CB8 cPe a n ED pl CB8 IEOP o E e s x 8wm uEir E E InfiniBand (RDMA) / 40 GigE / 10 GigE Scale Compute Nodes • IBM Power Systems • Only Hadoop services and HDFS client ESS HDP HDP HDP HDP HDP ESS Elastic Storage Server (Powered by Spectrum Scale & Power Systems) C C C C CC C Spectrum Scale Client HDP Hortonworks Data Platform Scale Storage as Required 31@ IBM Corporation
- 32. sxM ) r v tM ) o u 02I • 7 u/ D C w ( P • u 09 /E C E 6 E P1 kc chanN 58E w U • 4 DC BE G chanN • c ch lbe • p6 B M02 6 02 6 Pkc SbiNb L Model 4U drawers Drives Raw Disk Capacity GL1C 1 104 1.04 PB GL2C 2 210 2.1 PB GL4C 4 422 4.22 PB GL6C 6 634 6.34 PB @ IBM Corporation 32
- 33. IBM Elastic Storage Server (ESS) Model GL4S: 4 Enclosures, 20U 334 NL-SAS, 2 SSD Model GL6S: 6 Enclosures, 28U 502 NL-SAS, 2 SSD Model GL2S: 2 Enclosures, 12U 166 NL-SAS, 2 SSD ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage 36 GB/s12 GB/s24 GB/s System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 EXP3524 8 9 16 17 Model GS1S 24 SSD EXP3524 8 9 16 17 System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 EXP3524 8 9 16 17 Model GS2S 48 SSD EXP3524 8 9 16 17 System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 EXP3524 8 9 16 17 EXP3524 8 9 16 17 EXP3524 8 9 16 17 Model GS4S 96 SSD 40 GB/s 14 GB/s 26 GB/s Model GL1S: 1 Enclosures, 9U 82 NL-SAS, 2 SSD ESS 5U84 Storage 6 GB/s ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage 38 GB/s 40 GB/s Model GH14S: 1 2U24 Enclosure SSD 4 5U84 Enclosure HDD 334 NL-SAS, 24 SSD Model GH24S: 2 2U24 Enclosure SSD 4 5U84 Enclosure HDD 334 NL-SAS, 48 SSD 1428TB - 3708TB 1520TB - 4077TB 92TB, 368TB 184TB, 737TB 368TB, 1474TB 664TB - 1660TB 328TB - 820TB 1336TB - 3340TB 2008TB - 5020TB ) ESS 5U84 Storage ESS 5U84 Storage Model GH12S: 1 2U24 Enclosure SSD 2 5U84 Enclosure HDD 166 NL-SAS, 24 SSD 756TB - 2029TB TBD GB/s ) Model 4U drawers Drives Raw Disk Capacity GL1C 1 104 1.04 PB GL2C 2 210 2.1 PB GL4C 4 422 4.22 PB GL6C 6 634 6.34 PB @ IBM Corporation 33
- 34. &A & A & & A D & & I & / E datascience.ibm.comESS HDF POWER9 34@ IBM Corporation
- 35. - -
- 36. • • • POWER8 (Power S822L x24 ) • IBM Elastic Storage Server (ESS) with Spectrum Scale: GL2S (x2) • Hortonworks Data Platform (HDP) • Linux HDP IBM Power • IBM Spectrum Scale based ESS • TCO (Intel Power ESS ) • HDP SAS (ESS ) IBM Spectrum Scale 36@ IBM Corporation
- 37. E B : B : E B : 37 Better Together datascience.ibm.com E B : OP L 0 x H , S ac e H W /R E B9 d h c r r k 6 r o H % c H Ap 8 @ IBM Corporation
- 38. Dfmen k P A I D l a A D H es B rv Bf Bt I S I P O M M FS w P eBd i mucB Bf P DS A dipPy i l taoBk E I S r o M ng Bf P O dip H MFS 38@ IBM Corporation