Advanced GPU Topics #1

Jeremy Appleyard, September 2015

 • Profiling GPU applications
• Optimizing Data Movement

In this talk • Using MPI with GPUs

Ask questions at any point!

2
Profiling GPU Applications

3
Profiling Tools
Many options!

From
FromNVIDIA
NVIDIA From
ThirdNVIDIA
Party
• nvprof • nvprof
• nvprof • System
• NVIDIA Visual profiler • NVIDIA Visual profiler
• NVIDIA Visual profiler • VampirTrace
• Standalone (nvvp) • Standalone (nvvp)
• Standalone (nvvp)
• Integrated into Nsight Eclipse • Integrated into Nsight Eclipse
• Integrated into Nsight Eclipse • PAPI CUDA component
Edition (nsight) Edition (nsight)
Edition (nsight)
• Nsight Visual Studio Edition • HPC Toolkit
• Nsight Visual Studio Edition
• Nsight Visual Studio Edition

4
This talk

We will focus on nvprof and nvvp

nvprof => NVIDIA profiler
Command line

nvvp => NVIDIA Visual Profiler

GUI based

5
nvprof
Simple usage

• nvprof ./<executable> attributes(global) subroutine vecAdd_GPU(c, a, b, n)

INTEGER, value :: n
• Example: vector addition REAL, device, intent(in) :: a(n), b(n)
REAL, device, intent(out) :: c(n)
from yesterday’s talk
INTEGER :: i

i = (blockIdx%x – 1) * blockDim%x + threadIdx%x

if (i <= n) then
c(i) = a(i) + b(i)
end if
end subroutine vecAdd_GPU

6
nvprof ./vecAdd
==34092== NVPROF is profiling process 34092, command: ./vecAdd
==34092== Profiling application: ./vecAdd
==34092== Profiling result:
Time(%) Time Calls Avg Min Max Name
69.07% 3.8547ms 2 1.9274ms 1.9093ms 1.9454ms [CUDA memcpy HtoD]
29.04% 1.6205ms 1 1.6205ms 1.6205ms 1.6205ms [CUDA memcpy DtoH]
1.19% 66.178us 1 66.178us 66.178us 66.178us kernel_vecadd_gpu_
0.71% 39.650us 2 19.825us 19.617us 20.033us __pgi_dev_cumemset_4

==34092== API calls:

Time(%) Time Calls Avg Min Max Name
95.29% 321.93ms 3 107.31ms 215.28us 321.48ms cudaMalloc
2.70% 9.1355ms 3 3.0452ms 2.1550ms 3.5234ms cudaMemcpy
1.47% 4.9710ms 498 9.9810us 177ns 498.93us cuDeviceGetAttribute
0.22% 758.16us 3 252.72us 234.62us 284.11us cudaFree
0.15% 519.49us 6 86.581us 82.801us 89.206us cuDeviceTotalMem
0.13% 449.63us 6 74.938us 70.892us 81.049us cuDeviceGetName
0.02% 69.908us 3 23.302us 11.485us 43.331us cudaLaunch
0.00% 12.300us 10 1.2300us 179ns 8.0460us cudaSetupArgument
0.00% 3.4430us 12 286ns 193ns 716ns cuDeviceGet
0.00% 2.8280us 2 1.4140us 285ns 2.5430us cuDeviceGetCount
0.00% 2.5750us 3 858ns 448ns 1.6120us cudaConfigureCall
7
nvprof ./vecAdd
==34092== NVPROF is profiling process 34092, command: ./vecAdd
==34092== Profiling application: ./vecAdd
==34092== Profiling result:
Time(%) Time Calls Avg Min Max Name
69.07% 3.8547ms 2 1.9274ms 1.9093ms 1.9454ms [CUDA memcpy HtoD]
29.04% 1.6205ms 1 1.6205ms 1.6205ms 1.6205ms [CUDA memcpy DtoH]
1.19% 66.178us 1 66.178us 66.178us 66.178us kernel_vecadd_gpu_
0.71% 39.650us 2 19.825us 19.617us 20.033us __pgi_dev_cumemset_4

• Top half of the profile is runtime measured from the GPU perspective
• What can we see?
• Memcpy XtoX is memory copy to and from the GPU

• The vector addition is only 1.19%!

• This is valuable information!

8
nvprof ./vecAdd
==34092== NVPROF is profiling process 34092, command: ./vecAdd
==34092== Profiling application: ./vecAdd
==34092== Profiling result:
Time(%) Time Calls Avg Min Max Name
69.07% 3.8547ms 2 1.9274ms 1.9093ms 1.9454ms [CUDA memcpy HtoD]
29.04% 1.6205ms 1 1.6205ms 1.6205ms 1.6205ms [CUDA memcpy DtoH]
1.19% 66.178us 1 66.178us 66.178us 66.178us kernel_vecadd_gpu_
0.71% 39.650us 2 19.825us 19.617us 20.033us __pgi_dev_cumemset_4

• (PGI) OpenACC kernels will be named after the subroutine name and line number
• eg. vecadd_17_gpu
• Subroutine vecadd

• Line 17 in the file

• Compiled for the gpu

9
• Bottom half of the profile is runtime measured from the CPU
perspective
• What can we see?
• First allocation is expensive (cuda initialisation)

==34092== API calls:

Time(%) Time Calls Avg Min Max Name
95.29% 321.93ms 3 107.31ms 215.28us 321.48ms cudaMalloc
2.70% 9.1355ms 3 3.0452ms 2.1550ms 3.5234ms cudaMemcpy
1.47% 4.9710ms 498 9.9810us 177ns 498.93us cuDeviceGetAttribute
0.22% 758.16us 3 252.72us 234.62us 284.11us cudaFree
0.15% 519.49us 6 86.581us 82.801us 89.206us cuDeviceTotalMem
0.13% 449.63us 6 74.938us 70.892us 81.049us cuDeviceGetName
0.02% 69.908us 3 23.302us 11.485us 43.331us cudaLaunch
0.00% 12.300us 10 1.2300us 179ns 8.0460us cudaSetupArgument
0.00% 3.4430us 12 286ns 193ns 716ns cuDeviceGet
0.00% 2.8280us 2 1.4140us 285ns 2.5430us cuDeviceGetCount
0.00% 2.5750us 3 858ns 448ns 1.6120us cudaConfigureCall
10
nvprof
More advanced options

• nvprof -h
• There are quite a few options!
• Some useful ones:
• -o: creates an output file which can be imported into nvvp
• -m and -e: collect metrics or events
• --analysis-metrics: collect all metrics for import into nvvp
• --query-metrics and --query-events: query which metrics/events are available
11
nvprof
Events and Metrics

Most are quite in-depth, however some useful ones for quick analysis
In general, events are only for the expert. Rarely useful
(A few) useful metrics:
dram_read_throughput: Main GPU memory read throughput
dram_write_throughput: Main GPU memory write throughput
flop_count_sp: Number of single precision floating point operations
flop_count_dp : Number of double precision floating point operations
12
 nvprof
Metrics Example

nvprof --metrics dram_read_throughput,dram_write_throughput,flop_count_sp,flop_count_dp ./vecAdd

==32928== NVPROF is profiling process 32928, command: ./vecAdd
==32928== Warning: Some kernel(s) will be replayed on device 0 in order to collect all events/metrics.
==32928== Profiling application: ./vecAdd
==32928== Profiling result:
==32928== Metric result:
Invocations Metric Name Metric Description Min Max Avg
Device "Tesla K40m (0)"
Kernel: kernel_vecadd_gpu_
1 dram_read_throughput Device Memory Read Throughput 136.55GB/s 136.55GB/s 136.55GB/s
1 dram_write_throughput Device Memory Write Throughput 70.062GB/s 70.062GB/s 70.062GB/s
1 flop_count_sp Floating Point Operations(Single Precisi 1000000 1000000 1000000
1 flop_count_dp Floating Point Operations(Double Precisi 0 0 0

13
 nvprof
Metrics Example
Problem was addition of 1,000,000
element single precision vectors
1,000,000 single precision flop count
is therefore expected!
dram_read_throughput 136.55GB/s
dram_write_throughput 70.062GB/s ~50 GFLOP/s
flop_count_sp 1000000
flop_count_dp 0 About 1% peak FLOPs

Measured dram throughput is

~206GB/s
Peak for this machine is 288 GB/s
72% peak bandwidth
14
 Visual Profiler
nvvp

• Either import the output of nvprof –o ...

• Need to select metrics in advance

• Or run application through GUI

• Re-run on-the-fly to compute new metrics
based on requirements

15
Vecadd - timeline

16
NEMO - timeline

17
NEMO - details

18
Back to vecadd
More details!

• If we collected our profiling

information with –analysis-
metrics, or if we’re running the
application through nvvp
directly, more information is
available! This is what we want!

• -analysis-metrics may take some

time to run
• Run separately after
generating timeline info

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 19

20
• Matches our earlier calculations of ~72% peak bandwidth

21
Compute, Bandwidth or Latency Bound

 Other analysis options available

 The tool will guide you
 Becomes more useful the more experience
you get

22
Profiling tips
 The nvprof output is a very good place to start
 The timeline is a good place to go next
 Only dig deep into a kernel if it’s taking a significant amount of your time
 Where possible, try to match profiler output with theory
 For example, if I know I’m moving 1GB, and my kernel takes 10ms, I expect the
profiler to report 100GB/s.

 Discrepancies are likely to mean your application isn’t doing what you thought it was

23
Optimising Data Movement

24
Heterogeneous Computing

PCI Bus

25
Data movement
What are the costs?

Two costs:
Bandwidth
Latency

Large copies will be bandwidth bound.

Peak PCI-E bandwidth is 16 GB/s

Small copies will be high latency

Many small copies are expensive
26
Data movement
Several optimisation options

Several ways to optimise data movement:

1. Do less of it!
2. Fuse small copies into larger ones
3. Use pinned buffers
4. Overlap data movement with execution

Remember: profile before optimising!

27
Optimising Data Movement
1. Do less of it!

• Consider moving data movement outside of main loop

• For example, an iterative solver would not want data movement within the
iteration loop
• Often, during the process of making an application run on a GPU these additional
copies are unavoidable
• If doing intermediate performance analysis, bear in mind these copies may disappear
in the final application

• Consider if data can be passed as an argument (CUDA), or declared as private

(OpenACC)

28
Optimising Data Movement
2. Fuse small copies into larger ones

• Small data transfers are much less efficient

• If the opportunity arises, merging copies can be beneficial
• Can be difficult to accomplish in some applications
• Additional cost due to packing/unpacking may outweigh the savings

29
Optimising Data Movement
3. Use pinned buffers

• Page-locked (or pinned) allocations generally have higher bandwidth

• Can harm system performance if overused
• In CUDA:
• cudaHostAlloc() – allocate page-locked memory

• cudaHostRegister() – convert normal allocation to page-locked

• Pinned attribute in CUDA Fortran

• In OpenACC:
• (Hopefully) done automatically!
30
Optimising Data Movement
4. Overlap data movement with execution

• With pinned memory PCI-E transfers can occur at the same time as kernel
execution
• In CUDA:
• cudaMemcpyAsync

• In OpenACC:
• async clause on parallel or kernels directives

• This can greatly reduce the cost of memory copies – almost to none in the right
case

31
Optimising Data Movement
Summary

• There are good (and bad) ways of moving data between GPU and CPU
• For many applications this is not imporant
• Data resides on the GPU for the application’s lifetime

• Next-gen GPU Pascal will bring hardware improvements - NVLINK

32
NVLINK
Faster communications between processors

NVLINK
• High speed interconnect
• Alongside/replacing PCI-E
• 80-200 GB/s
• Improved energy efficiency
• Improved flexibility

33
 NVLink Unleashes Multi-GPU Performance
GPUs Interconnected with NVLink Over 2x Application Performance Speedup
When Next-Gen GPUs Connect via NVLink Versus PCIe
Speedup vs
PCIe based Server
2.25x

CPU
2.00x

PCIe Switch
1.75x

1.50x
TESLA TESLA
GPU GPU
1.25x

5x Faster than 1.00x

PCIe Gen3 x16 ANSYS Fluent Multi-GPU Sort LQCD QUDA AMBER 3D FFT

34
Using MPI with GPUs

35
 MPI+CUDA

System System System

GDDR5 Memory GDDR5 Memory GDDR5 Memory
Memory Memory Memory

…
GPU CPU GPU CPU GPU CPU

PCI-e PCI-e PCI-e

Network Network Network
Card Card Card

Node 0 Node 1 Node n-1

36
 MPI+CUDA

System System System

GDDR5 Memory GDDR5 Memory GDDR5 Memory
Memory Memory Memory

…
GPU CPU GPU CPU GPU CPU

PCI-e PCI-e PCI-e

Network Network Network
Card Card Card

Node 0 Node 1 Node n-1

37
 MPI+CUDA

//MPI rank 0
MPI_Send(s_buf_d, size, MPI_CHAR, n-1, tag, MPI_COMM_WORLD);

//MPI rank n-1

MPI_Recv(r_buf_d, size, MPI_CHAR, 0, tag, MPI_COMM_WORLD, &stat);
38
Message Passing Interface
MPI

• Standard to exchange data between processes via messages

• Defines API to exchanges messages
• Pt. 2 Pt.: e.g. MPI_Send, MPI_Recv

• Collectives, e.g. MPI_Reduce

• Multiple implementations (open source and commercial)

• Binding for C/C++, Fortran, Python, …

• E.g. MPICH, OpenMPI, MVAPICH, IBM Platform MPI, Cray MPT, …

39
MPI
Compiling and launching

$ mpicc –o myapp myapp.c

$ mpirun –np 4 ./myapp <args>

myapp myapp myapp myapp

40
Launch MPI + CUDA/OpenACC programs
 Launch one process per GPU
 MVAPICH: MV2_USE_CUDA

$ MV2_USE_CUDA=1 mpirun –np ${np} ./myapp <args>

 Open MPI: CUDA-aware features are enabled per default

 Cray: MPICH_RDMA_ENABLED_CUDA

 IBM Platform MPI: PMPI_GPU_AWARE

41
Unified Virtual Addressing

• One address space for all CPU and GPU memory

• Determine physical memory location from a pointer value

• Enable libraries to simplify their interfaces (e.g. MPI and cudaMemcpy)

• Support:
• 64-bit applications on Linux

• Windows using TCC mode

42
Unified Virtual Addressing
No UVA : Separate Address Spaces UVA : Single Address Space

System GPU System GPU

Memory Memory Memory Memory
0x0000 0x0000 0x0000

0xFFFF 0xFFFF 0xFFFF

CPU GPU CPU GPU

PCI-e PCI-e
43
MPI + CUDA

With UVA CUDA-aware MPI No UVA and regular MPI

//MPI rank 0
//MPI rank 0 s_buf_h = s_buf_d
call MPI_Send(s_buf_d,size,…) call MPI_Send(s_buf_h,size,…)

//MPI rank n-1 //MPI rank n-1

call MPI_Recv(r_buf_d,size,…) call MPI_Recv(r_buf_h,size,…)
r_buf_d = r_buf_h

44
 MPI + OpenACC

With UVA CUDA-aware MPI No UVA and regular MPI

!$acc host_data use_device (s_buf, r_buf) !$acc update host(s_buf)
!MPI rank 0 !MPI rank 0
call MPI_Send(s_buf,size,…) call MPI_Send(s_buf,size,…)

!MPI rank n-1 !MPI rank n-1

call MPI_Recv(r_buf,size,…) call MPI_Recv(r_buf,size,…)
!$acc end host_data !$acc update device(r_buf)

45
NVIDIA GPUDirectTM
Peer to Peer tranfers
GPU1 GPU2
Memory Memory

System
Memory

CPU
GPU GPU
1 2

PCI-e Chip
set
IB
46
NVIDIA GPUDirectTM
Peer to Peer tranfers
GPU1 GPU2
Memory Memory

System
Memory

CPU
GPU GPU
1 2

PCI-e Chip
set
IB
47
NVIDIA GPUDirectTM
Support for RDMA
GPU1 GPU2
Memory Memory

System
Memory

CPU
GPU GPU
1 2

PCI-e Chip
set
IB
48
NVIDIA GPUDirectTM
Support for RDMA
GPU1 GPU2
Memory Memory

System
Memory

CPU
GPU GPU
1 2

PCI-e Chip
set
IB
49
Performance Results two Nodes
OpenMPI 1.8.4 MLNX FDR IB (4X) Tesla K40 @ 875GHz
7000
6000
5000
BW (MB/s)

4000 GPU-aware MPI with

3000 GPUDirect RDMA
2000 GPU-aware MPI
1000
0 regular MPI

Message Size (byte)

Latency (1 byte) 19.79 us 17.97 us 5.70 us 50
 • Profiling GPU applications
• Optimizing Data Movement

In this talk • Using MPI with GPUs

Any questions?

51
Bonus slides - Pascal!

52
 GPU Roadmap

20
Pascal
Unified Memory
3D Memory
18 NVLink

16
SGEMM / W Normalized

14

Maxwell
12 DX12

10

8
Kepler
Dynamic Parallelism
6

Fermi
2 FP64
Tesla
CUDA
0
2008 2010 2012 2014 2016
53
Pascal

• Faster and larger global memory

• Faster communication between processors
• More powerful unified memory
• Mixed precision computing

54
Stacked Memory
High performance global memory

3D Stacked Memory
• 4x Higher Bandwidth (~1 TB/s)
• 3x Larger Capacity (up to 32GB)
• 4x More Energy Efficient per bit

55
NVLINK
Faster communications between processors

NVLINK
• High speed interconnect
• Alongside/replacing PCI-E
• 80-200 GB/s
• Improved energy efficiency
• Improved flexibility

56
 KEPLER GPU PASCAL GPU

NVLink

NVLink
High-Speed GPU
Interconnect POWER CPU

NVLink

PCIe PCIe

X86, ARM64, X86, ARM64,

POWER CPU POWER CPU

2014 2016
57
 Unified Memory: Simpler & Faster with NVLink

Traditional Developer View Developer View With Developer View With

Unified Memory Pascal & NVLink

NVLink
80 GB/s

System GPU Memory Unified Memory Unified Memory

Memory
Share Data Structures at
CPU Memory Speeds, not PCIe speeds
Oversubscribe GPU Memory

58
Mixed precision computing
IEEE 16-bit float support

• Halving precision results in:

• Half the memory footprint

• Half the bandwidth required

• Double the computational throughput

• Obviously comes at an accuracy penalty

• Application dependent

• 16/32/64 bit all supported

• Supported in software now

59