Tackling Network Bottlenecks with Hardware Accelerations: Cloud vs. On-Premise
1 like1,096 views
The ever-growing continuous influx of data causes every component in a system to burst at its seams. GPUs and ASICs are helping on the compute side, whereas in-memory and flash storage devices are utilized to keep up with those local IOPS. All of those can perform extremely well in smaller setups and under contained workloads. However, today's workloads require more and more power that directly translates into higher scale. Training major AI models can no longer fit into humble setups. Streaming ingestion systems are barely keeping up with the load. These are just a few examples of why enterprises require a massive versatile infrastructure, that continuously grows and scales. The problems start when workloads are then scaled out to reveal the hardships of traditional network infrastructures in coping with those bandwidth hungry and latency sensitive applications. In this talk, we are going to dive into how intelligent hardware offloads can mitigate network bottlenecks in Big Data and AI platforms, and compare the offering and performance of what's available in major public clouds, as well as a la carte on-premise solutions.
![Speeds
• 1, 10, 25, 40, 100, 200Gbps
• Faster network doesn’t
necessarily mean a faster
runtime
• Many workloads consist of
relatively short bursts rather
than sustainable throughput:
higher bandwidth may not have
any effect
10#UnifiedAnalytics #SparkAISummit
0
100
200
300
400
500
600
700
800
Flink
TeraSort
Flink
PageRank
PowerGraph
PageRank
Timely
PageRank
Effect of network speed
on workload runtime*
1GbE 10GbE 40GbE
* “On The [Ir]relevance of Network Performance for Data Processing” by Trivedi et al.](https://image.slidesharecdn.com/072020yuvaldeganijithinjose-190508203942/85/Tackling-Network-Bottlenecks-with-Hardware-Accelerations-Cloud-vs-On-Premise-10-320.jpg)
![[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/12skt-160226171513-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Tackling Network Bottlenecks with Hardware Accelerations: Cloud vs. On-Premise (20)
![[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene](https://cdn.slidesharecdn.com/ss_thumbnails/32apm-160226173205-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Databricks (20)
Ad
Recently uploaded (20)
Tackling Network Bottlenecks with Hardware Accelerations: Cloud vs. On-Premise
- 1. Yuval Degani, LinkedIn Dr. Jithin Jose, Microsoft Azure Tackling Network Bottlenecks with Hardware Accelerations: Cloud vs. On-Premise #UnifiedAnalytics #SparkAISummit
- 2. Intro • Infinite loop of removing performance road blocks • With faster storage devices (DRAM, NVMe, SSD) and stronger than ever processing power (CPU, GPU, ASIC), a traditional network just can’t keep up with I/O flow • Upgrading to higher wire speeds will rarely do the trick • This is where co-designed hardware acceleration can be used to truly utilize the power of a compute cluster 2#UnifiedAnalytics #SparkAISummit
- 3. Previous talks 3#UnifiedAnalytics #SparkAISummit Spark Summit Europe 2017 First open-source stand-alone RDMA accelerated shuffle plugin for Spark (SparkRDMA) Spark+AI Summit North America 2018 First preview of SparkRDMA on Azure HPC nodes, demonstrating x2.6 job speed-up on cloud VMs
- 4. Network Bottlenecks in the Wild 4#UnifiedAnalytics #SparkAISummit
- 5. Network Bottlenecks in the Wild • Not always caused by lack of bandwidth • Network I/O imposes overhead in many system components: – Memory management – Memory copy – Garbage Collection – Serialization/Compression/Encryption • Overhead=CPU cycles, cycles that are not available for the actual job at hand • Hardware acceleration can reduce overhead and allow better utilization of compute and network resources 5#UnifiedAnalytics #SparkAISummit
- 6. Network Bottlenecks: Shuffle • Most expensive non-storage network I/O in compute clusters • Blocking, massive movement of transient data • Acceleration opportunities: – Efficient serving with reduced server- side logic – Serialization/Compression/Encryption – Reduce I/O overhead and latency by employing modern transport protocols 6#UnifiedAnalytics #SparkAISummit Partitioning 4% Input 11% Shuffle Read 57% Output 28% HiBench TeraSort on Spark
- 7. Network Bottlenecks: Distributed Training • Model updates create massive network traffic • Model update frequency rises as GPUs get faster • Acceleration opportunities: – Inter-GPU RDMA communication – Lower latency network transport – Collectives offloads 7#UnifiedAnalytics #SparkAISummit K80 M60 V100 ResNet 269* Total Time GPU Active Time * “Parameter Hub: High Performance Parameter Servers for Efficient Distributed Deep Neural Network Training” by Luo et al.
- 8. Network Bottlenecks: Storage • Massive data movement • Premium devices (DRAM, Flash) provide storage access speeds that were never seen before • Acceleration opportunities: – Higher bandwidth – Reduced transport overhead – OS/CPU bypass – direct storage access from network devices 8#UnifiedAnalytics #SparkAISummit
- 9. Major Hardware Acceleration Technologies 9#UnifiedAnalytics #SparkAISummit
- 10. Speeds • 1, 10, 25, 40, 100, 200Gbps • Faster network doesn’t necessarily mean a faster runtime • Many workloads consist of relatively short bursts rather than sustainable throughput: higher bandwidth may not have any effect 10#UnifiedAnalytics #SparkAISummit 0 100 200 300 400 500 600 700 800 Flink TeraSort Flink PageRank PowerGraph PageRank Timely PageRank Effect of network speed on workload runtime* 1GbE 10GbE 40GbE * “On The [Ir]relevance of Network Performance for Data Processing” by Trivedi et al.
- 11. InfiniBand • De-facto standard in the HPC world • FDR: 56Gbps, EDR: 100Gbps, HDR: 200Gbps • Sub-microsecond latency • Native support for RDMA • HW accelerated transport layer • True SDN: standard fabric components are developed as open-source and are cross- platform • Native support for Switch collectives offload 11#UnifiedAnalytics #SparkAISummit Ethernet 23% InfiniBand 38% Custom 28% Omnipath 10% Proprietary 1% TOP500 Supercomputers Interconnect Performance Share* * www.top500.org
- 12. RDMA • Remote Direct Memory Access – Read/write from/to remote memory locations • Zero-copy • Direct hardware interface – bypasses the kernel and TCP/IP in IO path • Flow control and reliability is offloaded in hardware • Supported on almost all mid-range/high- end network adapters: both InfiniBand and Ethernet 12 Java app buffer OS Sockets TCP/IP Driver Network Adapter RDMA Socket Context switch #UnifiedAnalytics #SparkAISummit
- 13. NVIDIA GPUDirect • Direct DMA over PCIe • RDMA devices can write/read directly to/from GPU memory over the network • No CPU overhead • Zero-copy 13#UnifiedAnalytics #SparkAISummit GPUDirect Non-GPUDirect NIC GPU CPU
- 14. “Smart NIC” – FPGA/ASIC Offloads • FPGA – tailor-made accelerations • ASIC – less flexibility, better performance • Common use cases: – I/O: Serialization, compression, encryption offloads – Data: Aggregation, sorting, group-by, reduce • Deployment options: – Pipeline – Look-aside – Bump-on-the-wire 14#UnifiedAnalytics #SparkAISummit
- 15. “Smart Switch” • In-network processing – Data reduction during movement – Wire-speed • Generic: MPI Switch Collectives Offloads (e.g. Mellanox SHArP) • Per-workload: Programmable switches (e.g. Barefoot Tofino) – Example: Network-Accelerated Query Processing 15#UnifiedAnalytics #SparkAISummit
- 16. NVMeOF • Network protocol for NVM express disks (PCIe) • Uses RDMA to provide direct NIC<->Disk access • Completely bypasses the host • Minimal latency differences between local and remote access 16#UnifiedAnalytics #SparkAISummit NVMeOF Traditional NIC CPU
- 17. Azure Network Acceleration Offering 17#UnifiedAnalytics #SparkAISummit
- 18. Offer ‘Bare Metal’ Experience – Azure HPC Solution #UnifiedAnalytics #SparkAISummit 18 Eliminate Jitter Host holdback is a start, but must completely isolate guest from host Minroot & CPU Groups; separated host and guest VM sandboxes Full Network Experience Enable customers to use Mellanox or OFED drivers Supports all MPI types and versions Leverage hardware offload to Mellanox InfiniBand ASIC Transparent Exposure of Hardware Core N in guest VM should = Core N in silicon 1:1 between physical pNUMA topology and vNUMA topology
- 19. Latest Azure HPC Offerings – HB/HC HB Series (AMD EPYC) HC Series (Intel Xeon Platinum) Workloads Targets Bandwidth Intensive Compute Intensive Core Count 60 44 System Memory 240 GB 352 GB Network 100 Gbps EDR InfiniBand, 40 Gbps Ethernet Storage Support Standard / Premium Azure Storage, and 700GB Local SSD OS Support for RDMA CentOS/RHEL, Ubuntu, SLES 12, Windows MPI Support OpenMPI, HPC-X, MVAPICH2, MPICH, Intel MPI, PlatformMPI, Microsoft MPI Hardware Collectives Enabled Access Model Azure CLI, ARM template, Azure CycleCloud, Azure Batch, Partner Platform 19#UnifiedAnalytics #SparkAISummit
- 20. Other Azure HPC Highlights • SR-IOV going broad – All HPC SKUs will support SR-IOV – Driver/SKU Performance Optimizations • GPUs – Latest NDv2 Series • 8 Nvidia Tesla v100 NVLINK interconnected GPUs • Intel Skylake, 672 GB Memory • Excellent platform for HPC and AI workloads • Azure FPGA – Based on Project Brainwave – Deploy model to Azure FPGA, Reconfigure for different models – Supports ResNet 50, ResNet 152, DenseNet-121, and VGG-16 20#UnifiedAnalytics #SparkAISummit
- 21. Accelerate Your Framework 21#UnifiedAnalytics #SparkAISummit
- 22. MPI Microbenchmarks 22#UnifiedAnalytics #SparkAISummit • Experiments on HC cluster • OSU Benchmarks 5.6.1 • OpenMPI (4.0.0) + UCX (1.5.0) • MPI ranks pinned nearer to HCA 1.77 us 12 GB/s • MPI Latency (4 B) – 1.77us • Getting even better later this year • MPI Bandwidth (4 MB) – 12.06 GB/s 0 2000 4000 6000 8000 10000 12000 14000 1 2 4 8 16 32 64 128 256 512 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M Bandwidth(MB/s) Message Size (bytes) MPI Bandwidth Ethernet (40 Gbps) IPoIB (100 Gbps) RDMA (100 Gbps) 0 10 20 30 40 50 60 70 80 90 0 1 2 4 8 16 32 64 128 256 512 1K 2K Time(us) Message Size (bytes) MPI Latency Ethernet (40 Gbps) IPoIB (100 Gbps) RDMA (100 Gbps)
- 23. SparkRDMA • RDMA-powered ShuffleManager plugin for Apache Spark • Similarly spec 8 node cluster: – On-prem: 100GbE RoCE – Cloud: Azure ”h16mr” instances with 56Gbps InfiniBand • https://github.com/Mellanox/SparkRDMA 23#UnifiedAnalytics #SparkAISummit 0 1000 2000 TeraSort 320GB PageRank 19GB On-prem non-RDMA 100GbE On-prem RDMA 100GbE Azure IPoIB 56Gbps Azure RDMA 56Gbps
- 24. SparkRDMA on Azure • Azure HC cluster: – 100 Gbps InfiniBand – 16 Spark Workers/HDFS DataNodes – Separate NameNode – Data folder hosted on SSD – HiBench Benchmarks (gigantic) • Spark 2.4.0, Hadoop 2.7.7, SparkRDMA 3.1 24#UnifiedAnalytics #SparkAISummit 0 100 200 300 400 500 600 TeraSort - 320 GB PageRank - 19GB Execution Time (s) RDMA (100 Gbps) IPoIB (100 Gbps)
- 25. HDFS-RDMA on Azure 25#UnifiedAnalytics #SparkAISummit • OSU HDFS RDMA 0.9.1 • Based on Hadoop 3.0.0 • http://hibd.cse.ohio-state.edu/#hadoop3 • HDFS on HC cluster • 1 NameNode • 16 DataNodes • Data folder hosted on SSD • Packet Size: 128KB • Containers per Node: 32 0 50 100 150 200 250 300 350 400 512GB 640GB 768GB 896GB 1TB Time(sec) Size (bytes) TestDFSIO (Write) Execution Time Ethernet (40 Gbps) IPoIB (100 Gbps) RDMA (100 Gbps)
- 26. Memcached-RDMA on Azure 26#UnifiedAnalytics #SparkAISummit • OSU Memcached RDMA 0.9.6 • Based on Memcached 1.5.3 and libmemcached 1.0.18 • http://hibd.cse.ohio-state.edu/#memcached • Experiment run on HC Nodes • Memcached GET (8 B) Latency – 5.5us • Memcached SET (8 B) Latency – 6.45us 0 20 40 60 80 100 120 140 160 180 1 2 4 8 16 32 64 128 256 512 1K 2K 4K Latency(us) Message Size (bytes) Memcached GET 0 20 40 60 80 100 120 140 160 180 1 2 4 8 16 32 64 128 256 512 1K 2K 4K Latency(us) Message Size (bytes) Memcached SET Ethernet (40 Gbps) IPoIB (100 Gbps) RDMA (100 Gbps)
- 27. Kafka-RDMA on Azure 27#UnifiedAnalytics #SparkAISummit • OSU Kafka RDMA 0.9.1 • Based on Apache Kafka 1.0.0 • http://hibd.cse.ohio-state.edu/#kafka • HC cluster • Broker with 100 GB Ramdisk • Record Size – 100 bytes • Number of Records – 500000 0 50 100 150 200 250 300 350 400 Producer Time(s) Kafka Producer Latency IPoIB (100 Gbps) RDMA (100 Gbps) 0 10 20 30 40 50 60 70 Producer Bandwidth(MB/s) Kafka Producer Bandwidth IPoIB (100 Gbps) RDMA (100 Gbps)
- 28. Horovod on Azure 28#UnifiedAnalytics #SparkAISummit • Tensorflow 1.13 – ResNet-50 Training – Partial ImageNet Data – Batch Size = 64 per worker – 2 workers per node – Total batches 100 – CPU only version • HC Cluster – OpenMPI 4.0 + UCX 1.5 – Singularity container • ~97% Scaling efficiency 100.00 96.78 95.58 94.93 100.00 98.86 98.37 96.94 50.00 55.00 60.00 65.00 70.00 75.00 80.00 85.00 90.00 95.00 100.00 0 200 400 600 800 1000 1200 1400 1600 2 4 8 16 %Efficiency Images/second # nodes IPoIB (100 Gbps) RDMA (100 Gbps) IPoIB Efficiency RDMA Efficiency
- 30. What’s available on major clouds? Technology Azure AWS GCP Network speeds 100Gbps 100Gbps 20Gbps? InfiniBand ✔ ! ! RDMA ✔ (limited) ! GPUDirect ! (single host) ! Smart NIC ! ! ! Smart Switch ! ! ! NVMeOF ! ! ! 30#UnifiedAnalytics #SparkAISummit
- 31. Take-aways • Accelerated Frameworks: – SparkRDMA on GitHub – High Performance Big Data (From OSU) – Horovod • Azure instances – Azure HPC HB/HC – Azure NDv2 GPUs – Azure FPGA 31#UnifiedAnalytics #SparkAISummit
- 33. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT