HPC Infrastructure To Solve The CFD Grand Challenge
1 like2,690 views
This document summarizes Anand Haridass' presentation on using HPC infrastructure to solve computational fluid dynamics (CFD) grand challenges. It discusses how CFD utilizes physics, mathematics, computational geometry, and computer science. Solving CFD problems is bound by memory usage, computation needs, and network requirements. The presentation outlines IBM's POWER processor roadmap and how the POWER9 will have stronger cores, enhanced caches, and improved interfaces like NVLink and CAPI to accelerate workloads like CFD. Case studies demonstrate how IBM systems using GPUs and NVLink can provide faster performance for CFD codes and reservoir simulations.
![OpenFOAM: Motorbike example
Multi-node runs with up to 500 cores / 25 nodes
Perfect scaling for 50M and 100M cells up to 500 cores
Smaller cases run into saturation
10
100
1000
10000
100000
16 64 256 1024
Runtime[s]
Cores
MotorBike on S822LC
0.50
0.60
0.70
0.80
0.90
1.00
1.10
1.20
16 64 256 1024
Efficiency Cores
Parallel Efficiency
12.8M
50M
100M](https://image.slidesharecdn.com/opcfd-180509163933/85/HPC-Infrastructure-To-Solve-The-CFD-Grand-Challenge-13-320.jpg)
![IBM Systems | 14
Code Optimization: Reservoir Simulation – Exploration for Oil & Gas
New Record IBM + Nvidia + Stone Ridge Technology (Echelon)
April ’17 - 1 Billion Cells simulated in 92 minutes using 30x 2 socket IBM OpenPower POWER8 Minsky servers (30 server nodes 60
POWER8 chips [600 cores]+ 120 Nvidia P100 GPU’s)
Previous Record
ExxonMobil – Feb ‘17 Used Full Blue Water facility at NCSA (716,800 cores & 22,400 server nodes) to simulate one billion cells
took 20hours
https://www-03.ibm.com/press/us/en/pressrelease/52164.wss
https://venturebeat.com/2017/04/25/ibm-nvidia-and-stone-ridge-technology-set-record-for-supercomputing-in-oil-and-gas-exploration/
Billion-cell simulation shatters previous published results
using one-tenth the power and 1/100th of the space
Results achieved in 92 minutes with 60 IBM POWER8
processors & 120 Nvidia P100 GPU accelerators, smash
previous published record of 20 hours using thousands of
processors](https://image.slidesharecdn.com/opcfd-180509163933/85/HPC-Infrastructure-To-Solve-The-CFD-Grand-Challenge-14-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to HPC Infrastructure To Solve The CFD Grand Challenge (20)
Ad
More from Anand Haridass (7)
Ad
Recently uploaded (20)
HPC Infrastructure To Solve The CFD Grand Challenge
- 1. HPC Infrastructure To Solve The CFD Grand Challenge Anand Haridass Senior Technical Staff Member India System Development Lab IBM India 19th Aeronautical Society of India CFD Symposium Bangalore Aug 10-11 2017
- 2. IBM Systems Aerospace Design Turbine Design Computational Fluid Dynamics - Layman View Physics – Fluid Dynamics Mathematics – Partial Diff. Eqn. Computational Geometry Computer Science / Numerical Analysis Systems + Hardware Huge Linear Solve Huge Non-Linear Solve Algorithms Memory Bound eg Data transportation costs; Direct Numerical Simulation Compute Bound eg. Grid property updates Networking Bound eg. Large cluster MPI implementation
- 3. Technology shrink alone no longer drive sufficient Cost/Performance improvements Driving Innovation Beyond The Microprocessor
- 4. OpenPOWER Foundation Platinum Members The goal of the OpenPOWER Foundation is to create an open ecosystem, using the POWER Architecture to share expertise, investment, and server-class intellectual property to serve the evolving needs of customers.
- 5. Enabling National Computing Agenda
- 6. POWER Processor Roadmap 2014 12 Cores SMT8 2X DPFP PCIE Gen 3 Coprocessor (CAPI) Enhanced Prefetch NVLink 1.0 2X CAPI 2020+ 24 Cores New µArchitecture Direct-attach DDR4 Gen4 PCIe CAPI 2.0 OpenCAPI 3.0 NVLink 2.0 650mm2 POWER8 22 nm POWER8 with NVLink 22 nm POWER9 14 nm 659mm2 2016 2017 POWER10 >>24 Cores New µArchitecture Enhanced Memory OpenCAPI 4.0 Future NVLink 695mm2 Memory bandwidth 230GBs or 115GBs sustained memory bandwidth in 8/4 Port configuration L4 cache – reducing access latency to larger data sets Strong Cores - Greater raw CPU throughput More performance from each core Superior Multi-threading & large caches 1, 2, 4 , 8 threads per core – improving parallelism & throughput Tighter accelerator integration Only Processor with integrated NVLink interface Memory coherency (GPUs & otherwise) I/O leadership Established OpenCAPI Open Industry Coherent Attach Latency / Bandwidth Improvement Removes Overhead from Attach Silicon FPGA / Parallel Compute Optimized Network/Memory/Storage Innovation
- 7. 7 | 7 POWER9 – Premier Acceleration Platform Extreme Processor / Accelerator Bandwidth and Reduced Latency Coherent Memory and Virtual Addressing Capability for all Accelerators OpenPOWER Community Enablement – Robust Accelerated Compute Options 7 State of the Art I/O and Acceleration Attachment Signaling • PCIe Gen 4 x 48 lanes – 192 GB/s duplex bandwidth • 25G Link x 48 lanes – 300 GB/s duplex bandwidth Robust Accelerated Compute Options with OPEN standards • On-Chip Acceleration – Gzip x1, 842 Compression x2, AES/SHA x2 • CAPI 2.0 – 4x bandwidth of POWER8 using PCIe Gen 4 • NVLink 2.0 – Next generation of GPU/CPU interconnect Up to 2x bandwidth of NVLink1.0 Easier programming model for complex analytic & cognitive applications • Coherency, virtual addressing, low overhead communication • OpenCAPI 3.0 – High bandwidth, low latency and open interface using 25G Link POWER9 PowerAccel
- 8. IBM Systems IBM POWER HPC Long-term Systems Roadmap | 8 2016 GPU Intensive S822LC-HPC P8 “Witherspoon” “Deep Eddy” Deep Eddy F.O. 2017-2018 2020+ HPC/HPA/DL GPU Lite S821LC P8 HPC Non- accelerated Nodes (2- 8sockets in 2U) S822LC-B.D. P8 Future “Boston” 1U “Boston” “Aero” Boston Follow-on Boston Follow-on Aero Follow-on Plans subject to change Fully Invested in Enabling Heterogeneous Computing
- 9. IBM Systems “Minsky” S822LC for HPC • 2.5X the CPU:GPU Interface Bandwidth • Tight coupling: strong CPU: strong GPU performance • Equalizing access to memory - for all kinds of programming • Closer programming to the CPU paradigm 115GB/S 115GB/S NVLink DDR4 P8’ InfiniBand Fabric DDR4 P8’ Tesla P100 Tesla P100 80GB/S Tesla P100 Tesla P100 80GB/S Why it Matters: Use Cases where NVLink will have the most Impact Stream Data at Same Rate as Computation Genomics, Cryptography, Video Processing, etc. Burst Data at Startup and Teardown . CFD/CAE, Machine Learning, Deep Learning, etc. Constant Data Transfers between adjacent GPUs Molecular Dynamics, Amber, etc. Mask Bus Transfers from Host-Device Accelerated Databases, Analytics, etc.
- 10. IBM Systems | 10 • Profiling result based on running Kinetica “Filter by geographic area” queries on data set of 280 million simulated 1 simultaneous query stream each with 0 think time. • Power System S822LC for HPC; 20 cores (2 x 10c chips) / 160 threads, POWER8 with NVLink; 2.86 GHz, 1024 GB memory, 2x 6Gb SSDs, 2-port 10 GbEth, 4x Tesla P100 GPU; Ubuntu 16.04. • Competitive stack: 2x Xeon E5-2640 v4; 20 cores (2 x 10c chips) / 40 threads; Intel Xeon E5-2640 v4; 2.4 GHz; 512GB memory 2x 6Gb SSDs, 2-port 10 GbEth, 4x Tesla P100 GPU, Ubuntu 16.04. 2.7X faster query response time on IBM Power Systems S822LC for HPC 87% of the total speedup (2.35x of 2.7x improvement) is due to the NVLink Interface from CPU:GPU • Performance dividend unique to POWER8 with NVLink platforms 100 ticks 79 ticks Data Transfer 21 ticks Calculation* 79% 21% 37 ticks 24 ticks 13 ticks 2.7x speedup Query Time: Competing System PCI-E x16 3.0 Query Time: S822LC for HPC, NVLink * Includes non-overlapping: CPU, GPU, and idle times. Data Transfer Calculation* 65% 35% “Minsky” S822LC for HPC
- 11. Reduce Data Transfer For CFD Codes CPU:GPU NVLink improves data transfer 2.6X in Nekbone Newly accelerated codes such as Nekbone which require faster data movement • Avoid latency that exceeds solver time • Support your largest datasets with the GPU accelerated versions of your code because data-transfer time improves • IBM Power System S822LC for HPC; 20 cores (2 x 10c chips) / 160 threads, POWER8 with NVLink; 2.86 GHz, 256 GB memory, 2 x 1TB SATA 7.2K rpm HDD, 2-port 10 GbEth, 2x Tesla P100 GPU; Ubuntu 16.04. • Competitive stack: 2x Xeon E5-2640 v4; 20 cores (2 x 10c chips) / 40 threads; Intel Xeon E5-2640 v4; 2.4 GHz; 256 GB memory, 2 x 1TB SATA 7.2K rpm HDD, 2-port 10 GbEth, 2x Tesla K80 GPUs, Ubuntu 16.04.
- 12. OpenFOAM: Motorbike example Provided with OpenFoam examples: incompressible/simpleFoam Different problem sizes: 12M – 100M points SimpleFoam, double precision
- 13. OpenFOAM: Motorbike example Multi-node runs with up to 500 cores / 25 nodes Perfect scaling for 50M and 100M cells up to 500 cores Smaller cases run into saturation 10 100 1000 10000 100000 16 64 256 1024 Runtime[s] Cores MotorBike on S822LC 0.50 0.60 0.70 0.80 0.90 1.00 1.10 1.20 16 64 256 1024 Efficiency Cores Parallel Efficiency 12.8M 50M 100M
- 14. IBM Systems | 14 Code Optimization: Reservoir Simulation – Exploration for Oil & Gas New Record IBM + Nvidia + Stone Ridge Technology (Echelon) April ’17 - 1 Billion Cells simulated in 92 minutes using 30x 2 socket IBM OpenPower POWER8 Minsky servers (30 server nodes 60 POWER8 chips [600 cores]+ 120 Nvidia P100 GPU’s) Previous Record ExxonMobil – Feb ‘17 Used Full Blue Water facility at NCSA (716,800 cores & 22,400 server nodes) to simulate one billion cells took 20hours https://www-03.ibm.com/press/us/en/pressrelease/52164.wss https://venturebeat.com/2017/04/25/ibm-nvidia-and-stone-ridge-technology-set-record-for-supercomputing-in-oil-and-gas-exploration/ Billion-cell simulation shatters previous published results using one-tenth the power and 1/100th of the space Results achieved in 92 minutes with 60 IBM POWER8 processors & 120 Nvidia P100 GPU accelerators, smash previous published record of 20 hours using thousands of processors
- 15. IBM Systems | 15https://www.ibm.com/us-en/marketplace/deep-learning-platform Deep Learning - PowerAI
- 16. Thinking Differently - Getting HPC to ‘Work Smart Not Hard’ • Beginning to see ML/DL techniques being applied to CFD codes • Typically HPC development is focused on increased speed. • The fastest calculation is the one which you don’t run! • Can we use machine learning to make better decisions on which simulations give the most value? • Can we use machine learning to improve resolution of information? ‘Cognitive’ workflow uses 1/3 of the calculations to achieve 4 orders of magnitude resolution increase Cognitive steering of an ensemble of simulations
- 17. BACKUP
- 18. POWER8 Compute 6/8/10/12 cores, ST/SMT2/SMT4/SMT8 Enhanced, Auto balancing threads 8 dispatch/16 execution pipes/224 instructions in flight Transactional Memory/ Crypto & CRC instructions Cache 64KB L1 + 512KB L2 / core 96MB L3 + up to 128MB L4 / socket System Interfaces 230 GB/s memory bandwidth / socket vs Intel 85GB/s Up to 48x Integrated PCI gen 3 / socket CAPI (over PCI gen 3) Robust, Large SMP Interconnect On chip Energy Mgmt, VRM / core IBM Journal of Research and Development Issue 1 Date Jan.-Feb. 2015 On IEEE Explore - Link
- 19. 19 POWER9 Processor New Core Microarchitecture • Stronger thread performance • Efficient agile pipeline • POWER ISA v3.0 Enhanced Cache Hierarchy • 120MB NUCA L3 architecture • 12 x 20-way associative regions • Advanced replacement policies • Fed by 7 TB/s on-chip bandwidth Cloud + Virtualization Innovation • Quality of service assists • New interrupt architecture • Workload optimized frequency • Hardware enforced trusted execution Leadership Hardware Acceleration • Enhanced on-chip acceleration • Nvidia NVLink 2.0: High bandwidth, advanced new features (25G Link) • CAPI 2.0: Coherent accelerator and storage attach (PCIe G4) • New CAPI: Improved latency and bandwidth, open interface (25G Link) State of the Art I/O Subsystem • PCIe Gen4 – 48 lanes High Bandwidth Signaling Technology • 16 Gb/s interface – Local SMP • 25 Gb/s interface – BlueLink – Accelerator, remote SMP 14nm finFET Semiconductor Process • Improved device performance and reduced energy • 17 layer metal stack and eDRAM • 8.0 billion transistors
- 20. InfiniBand remains our strategic interconnect IB is an open standard with a proven track record over a decade of forward and backward compatibility IB features rich hardware offload • “Intelligent” Mellanox IB NIC versus Intel Omni- Path NIC More CPU cycles available for application processing with less CPU frequency sensitivity, processor/memory occupancy, jitter • MPI collectives acceleration in the IB fabric, reducing CPU overhead • Additional intelligence in the switch • IB features GPU direct, reducing CPU overhead • IB features NVMe over Fabrics support, reducing CPU overhead • IB has hardware-controlled, end-to-end retry IB supports multiple network topologies for cost optimization • Fat tree, 3D torus, Dragonfly+
- 21. Open Industry Coherent Attach • Latency / Bandwidth Improvement • Removes Overhead from Attach Silicon • Eliminates “Von-Neumann Bottleneck” • FPGA / Parallel Compute Optimized • Network/Memory/Storage Innovation http://opencapi.org Consortium announced October 14, 2016 Open Forum to Manage the OpenCAPI Specification & Ecosystem