Introduction to

CUDA C/C++

Tom Papatheodore
HPC User Support Specialist/Programmer

June 19, 2017

ORNL is managed by UT-Battelle

for the US Department of Energy
 Outline
• Programming model for heterogeneous
architectures
• Structure of a basic CUDA program
– Data transfers & kernel launches
– Thread hierarchy
– Run vector addition program

• CUDA error checking

• Multi-D CUDA grids
– Run matrix addition program
– Create matrix-vector multiply kernel

• CUDA device queries

• Shared Memory
– Run dot product program
– Create matrix-vector multiply kernel (with shared memory)

Introduction to CUDA C/C++

 Logging in to Chester

• ssh [email protected]
• ssh [email protected]
• cd $MEMBERWORK/trn001

Introduction to CUDA C/C++

 Let’s run something…

• git clone https://github.com/tpapathe/intro_cuda.git

• module load cudatoolkit
• cd intro_cuda/vector_addition/
• nvcc vector_addition.cu -o run
• qsub submit.pbs

• qstat –u username
• Check output file vec_add.o${JOB_ID}
– If you see __SUCCESS__, you have successfully run on GPU
– If not, try again and/or ask for help

Introduction to CUDA C/C++

 CUDA Programming Model
Heterogeneous Architecture

GPU (device, accelerator)

• Thousands of compute cores
Tesla K20X

6 GB
Titan/Chester node

Each with its

own memory
16 cores

32 GB

CPU (host)
• Several compute cores

Introduction to CUDA C/C++

 CUDA Programming Model

• Heterogeneous Programming
– program separated into serial regions (run on CPU) & parallel regions (run on GPU)

Introduction to CUDA C/C++

 CUDA Programming Model

• Heterogeneous Programming
– program separated into serial regions (run on CPU) & parallel regions (run on GPU)

Introduction to CUDA C/C++

 CUDA Programming Model

• Parallel regions consist of many calculations that can be executed

independently
– Data Parallelism (e.g. vector addition)

A
+ + + … +
B
= = = … =
C

Introduction to CUDA C/C++

 CUDA Programming Model

From the CUDA Programming Guide:

At its core are three key abstractions – a hierarchy of thread

groups, shared memories, and barrier synchronization – that
are simply exposed to the programmer as a minimal set of
language extensions (to C programming language)

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

// Allocate memory for array on device

// Fill array on host

// Copy data from host array to device array

// Do something on device (e.g. vector addition)

// Copy data from device array to host array

// Check data for correctness

// Free Host Memory

// Free Device Memory

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

// Allocate memory for array on host

size_t bytes = N*sizeof(int);

int *A = (int*)malloc(bytes);

int *B = (int*)malloc(bytes);

int *C = (int*)malloc(bytes);

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Allocate memory for array on device

int *d_A, *d_B, *d_C;

cudaMalloc(&d_A, bytes);

cudaMalloc(&d_B, bytes);

cudaMalloc(&d_C, bytes);

. . .

cudaError_t cudaMalloc( void** devPtr, size_t size )

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Fill array on host

for(int i=0; i<N; i++)

A[i] = 1;

B[i] = 2;

C[i] = 0;

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Copy data from host array to device array

cudaMemcpy(d_A, A, bytes, cudaMemcpyHostToDevice);

cudaMemcpy(d_B, B, bytes, cudaMemcpyHostToDevice);

. . .

cudaError_t cudaMemcpy( void* dst, const void* src, size_t count,

cudaMemcpyKind kind )

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Do something on device (e.g. vector addition)

// We’ll come back to this soon

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Copy data from device array to host array

cudaMemcpy(C, d_C, bytes, cudaMemcpyDeviceToHost);

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Check data for correctness

for (int i=0; i<N; i++)

if(C[i] != 3)

// Error – value of C[i] is not correct!

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Free Host Memory

free(A);

free(B);

free(C);

. . .

Introduction to CUDA C/C++

 A Basic CUDA Program Outline

int main(){

. . .

// Free Device Memory

cudaFree(d_A);

cudaFree(d_B);

cudaFree(d_C);

cudaError_t cudaFree( void* devPtr )

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel?

• How is it different from a normal
function?
– When kernel is launched, a grid of threads
are generated
– Same code is executed by all threads
• Single-Program Multiple-Data (SPMD)

Serial – CPU

for (int i=0; i<N; i++){

C[i] = A[i] + B[i];

Parallel – GPU

C[i] = A[i] + B[i];

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

__global__

Indicates the function is a CUDA kernel function – called by the host and executed on the
device.

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

__void__

Kernel does not return anything.

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

int *a, int *b, int *c

Kernel function arguments

• a, b, c are pointers to device memory

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

Grid
{

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

}
Blocks

int i = blockDim.x * blockIdx.x + threadIdx.x;

This defines a unique thread id among all threads in a grid

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

Grid
{

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

}
Threads

int i = blockDim.x * blockIdx.x + threadIdx.x;

This defines a unique thread id among all threads in a grid

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

{
4
int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

blockDim

Gives the number of threads within each block (in the x-dimension in 1D case)
• E.g., 4 threads per block

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

{
(4) (0-3)
int i = blockDim.x * blockIdx.x + threadIdx.x; 0
if (i<N) c[i] = a[i] + b[i];

} 1

blockIdx 2
Specifies which block the thread belongs to (within the grid of blocks)
3

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

{
(4) (0-3) (0-3) 0
int i = blockDim.x * blockIdx.x + threadIdx.x; 0 1
2
3
if (i<N) c[i] = a[i] + b[i];
0

} 1 1
2
3

0
1
threadIdx 2 2
3

Specifies a local thread id within a thread block 0

3 1
2
3

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

int i = blockDim.x * blockIdx.x + threadIdx.x;

This defines a unique thread id among all threads in a grid

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

{
(4) (2) (1)
int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3

block 0 block 1 block 2 block 3

int i = 4 * 2 + 1 = 9

A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

if (i<N)

Number of threads in the grid might be larger than number of elements in array.

A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel? What does it look like?

__global__ void vector_addition(int *a, int *b, int *c)

int i = blockDim.x * blockIdx.x + threadIdx.x;

if (i<N) c[i] = a[i] + b[i];

int i

Local variables are private to each thread.

The loop was replaced by a grid of threads.

Introduction to CUDA C/C++

 CUDA Kernels

Ok, so what about the kernel?

• How is it called (launched)?

In general
kernel<<< blk_in_grid, thr_per_blk >>>(arg1, arg2, …);

Our specific problem

thr_per_blk = 128;
blk_in_grid = ceil( float(N) / thr_per_blk );
vec_add<<< blk_in_grid, thr_per_blk >>>(d_a, d_b, d_c);

Introduction to CUDA C/C++

 Now let’s run the vector_addition
program again.

Run the vector_addition program (nvcc vector_addition.cu -o run)

• Change execution configuration parameters (i.e. change thr_per_blk)

– What happens if you make thr_per_blk too large?

• Change thr_per_blk back to a value <= 1024 and change the size of d_A
– e.g., cudaMalloc(&d_A, 8e9*bytes);

6 GB

32 GB
Introduction to CUDA C/C++
 CUDA Error Checking
API calls
cudaError_t err = cudaMalloc(&d_A, 8e9*bytes);

if(err != cudaSuccess) printf(“Error: %s\n”, cudaGetErrorString(err));

Kernels (check for synchronous and asynchronous errors)

add_vectors<<<blk_in_grid, thr_per_blk>>>(d_A, d_B, d_C, N);

// Kernel does not return an error, so get manually

cudaError_t errSync = cudaGetLastError();

if(errSync != cudaSuccess) printf(“Error: %s\n”, cudaGetErrorString(errSync));

// After launch, control returns to the host, so errors can occur at seemingly
// random points later in the code. Calling cudaDeviceSynchronize catches these
// errors and allows you to check them

cudaError_t errAsync = cudaDeviceSynchronize();

if(errAsync != cudaSuccess) printf(”Error: %s\n”, cudaGetErrorString(errAsync));

Introduction to CUDA C/C++

 Multi-D CUDA Grids

Introduction to CUDA C/C++

 Multi-D CUDA Grids
In previous 1D example
thr_per_blk = 128
blk_in_grid = ceil( float(N) / thr_per_blk );
vec_add<<< blk_in_grid, thr_per_blk >>>(d_a, d_b, d_c);

In general
dim3 is c struct
dim3 threads_per_block( threads per block in x-dim, with member
threads per block in y-dim,
variables x, y, z.

threads per block in z-dim);

dim3 blocks_in_grid( grid blocks in x-dim,

grid blocks in y-dim,
grid blocks in z-dim );

Introduction to CUDA C/C++

 Multi-D CUDA Grids
In previous 1D example
thr_per_blk = 128
blk_in_grid = ceil( float(N) / thr_per_blk );
vec_add<<< blk_in_grid, thr_per_blk >>>(d_a, d_b, d_c);

So we could have used

dim3 is c struct
dim3 threads_per_block( 128, 1, 1 ); with member
variables x, y, z.

dim3 blocks_in_grid( ceil( float(N) / threads_per_block.x ), 1, 1 );

vec_add<<< blocks_in_grid, threads_per_block >>>(d_a, d_b, d_c);

Introduction to CUDA C/C++

 Map CUDA threads to 2D array
A0,0 A0,1 A0,2 A0,3 A0,4 A0,5 A0,6 A0,7 A0,8 A0,9 M = 7 rows
A1,0 A1,1 A1,2 A1,3 A1,4 A1,5 A1,6 A1,7 A1,8 A1,9
N = 10 columns

A2,0 A2,1 A2,2 A2,3 A2,4 A2,5 A2,6 A2,7 A2,8 A2,9
Assume a 4x4
blocks of threads…
A3,0 A3,1 A3,2 A3,3 A3,4 A3,5 A3,6 A3,7 A3,8 A3,9

Then to cover all

A4,0 A4,1 A4,2 A4,3 A4,4 A4,5 A4,6 A4,7 A4,8 A4,9
elements in the
A5,0 A5,1 A5,2 A5,3 A5,4 A5,5 A5,6 A5,7 A5,8 A5,9
array, we need 3
blocks in x-dim and
A6,0 A6,1 A6,2 A6,3 A6,4 A6,5 A6,6 A6,7 A6,8 A6,9 2 blocks in y-dim.

dim3 threads_per_block( 4, 4, 1 );
dim3 blocks_in_grid( ceil( float(N) / threads_per_block.x ),
ceil( float(M) / threads_per_block.y ) , 1 );
mat_add<<< blocks_in_grid, threads_per_block >>>(d_a, d_b, d_c);

Introduction to CUDA C/C++

 Row-Major Order

A0,0 A0,1 A0,2 A0,3 A0,4

A1,0 A1,1 A1,2 A1,3 A1,4

A2,0 A2,1 A2,2 A2,3 A2,4

A3,0 A3,1 A3,2 A3,3 A3,4

… …

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Introduction to CUDA C/C++

 Row-Major Order

A0,0 A0,1 A0,2 A0,3 A0,4

A1,0 A1,1 A1,2 A1,3 A1,4

A2,0 A2,1 A2,2 A2,3 A2,4

A3,0 A3,1 A3,2 A3,3 A3,4

… A0,0 A0,1 A0,2 A0,3 A0,4 …

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Introduction to CUDA C/C++

 Row-Major Order

A0,0 A0,1 A0,2 A0,3 A0,4

A1,0 A1,1 A1,2 A1,3 A1,4

A2,0 A2,1 A2,2 A2,3 A2,4

A3,0 A3,1 A3,2 A3,3 A3,4

… A0,0 A0,1 A0,2 A0,3 A0,4 A1,0 A1,1 A1,2 A1,3 A1,4 …

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Introduction to CUDA C/C++

 Row-Major Order

A0,0 A0,1 A0,2 A0,3 A0,4

A1,0 A1,1 A1,2 A1,3 A1,4

A2,0 A2,1 A2,2 A2,3 A2,4

A3,0 A3,1 A3,2 A3,3 A3,4

… A0,0 A0,1 A0,2 A0,3 A0,4 A1,0 A1,1 A1,2 A1,3 A1,4 A2,0 A2,1 A2,2 A2,3 A2,4 …

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Introduction to CUDA C/C++

 Row-Major Order

A0,0 A0,1 A0,2 A0,3 A0,4

A1,0 A1,1 A1,2 A1,3 A1,4

A2,0 A2,1 A2,2 A2,3 A2,4

A3,0 A3,1 A3,2 A3,3 A3,4

… A0,0 A0,1 A0,2 A0,3 A0,4 A1,0 A1,1 A1,2 A1,3 A1,4 A2,0 A2,1 A2,2 A2,3 A2,4 A3,0 A3,1 A3,2 A3,3 A3,4 …

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Introduction to CUDA C/C++

 Map CUDA threads to 2D array
A0,0 A0,1 A0,2 A0,3 A0,4 A0,5 A0,6 A0,7 A0,8 A0,9 M = 7 rows
A1,0 A1,1 A1,2 A1,3 A1,4 A1,5 A1,6 A1,7 A1,8 A1,9
N = 10 columns

A2,0 A2,1 A2,2 A2,3 A2,4 A2,5 A2,6 A2,7 A2,8 A2,9
Assume 4x4 blocks
of threads…
A3,0 A3,1 A3,2 A3,3 A3,4 A3,5 A3,6 A3,7 A3,8 A3,9

Then to cover all

A4,0 A4,1 A4,2 A4,3 A4,4 A4,5 A4,6 A4,7 A4,8 A4,9
elements in the
A5,0 A5,1 A5,2 A5,3 A5,4 A5,5 A5,6 A5,7 A5,8 A5,9
array, we need 3
blocks in x-dim and
A6,0 A6,1 A6,2 A6,3 A6,4 A6,5 A6,6 A6,7 A6,8 A6,9 2 blocks in y-dim.

__global__ void add_matrices(int *a, int *b, int *c){

int column = blockDim.x * blockIdx.x + threadIdx.x;
int row = blockDim.y * blockIdx.y + threadIdx.y;
if (row < M && column < N){
int thread_id = row * N + column;
c[thread_id] = a[thread_id] + b[thread_id];
}
}
Introduction to CUDA C/C++
 0 1 2 3 4 5 6 7 8 9 10 11

0 A0,0 A0,1 A0,2 A0,3 A0,4 A0,5 A0,6 A0,7 A0,8 A0,9 M = 7 rows
1 A1,0 A1,1 A1,2 A1,3 A1,4 A1,5 A1,6 A1,7 A1,8 A1,9
N = 10 columns

2 A2,0 A2,1 A2,2 A2,3 A2,4 A2,5 A2,6 A2,7 A2,8 A2,9
Assume 4x4 blocks
of threads…
3 A3,0 A3,1 A3,2 A3,3 A3,4 A3,5 A3,6 A3,7 A3,8 A3,9

Then to cover all

4 A4,0 A4,1 A4,2 A4,3 A4,4 A4,5 A4,6 A4,7 A4,8 A4,9
elements in the
5 A5,0 A5,1 A5,2 A5,3 A5,4 A5,5 A5,6 A5,7 A5,8 A5,9
array, we need 3
blocks in x-dim and
6 A6,0 A6,1 A6,2 A6,3 A6,4 A6,5 A6,6 A6,7 A6,8 A6,9 2 blocks in y-dim.
7

__global__ void add_matrices(int *a, int *b, int *c){

int column = blockDim.x * blockIdx.x + threadIdx.x; (0 – 11)
int row = blockDim.y * blockIdx.y + threadIdx.y; (0 – 7)
if (row < M && column < N){
int thread_id = row * N + column; (0 – 69)
c[thread_id] = a[thread_id] + b[thread_id];
} Ex: What element of the array does the highlighted
} thread correspond to?
Introduction to CUDA C/C++ thread_id = row * N + column
= 5 * 10 + 6 = 56
 Now let’s run the matrix_addition
program.

Run the matrix_addition program

• Change execution configuration parameters
– threads_per_block( 16, 16, 1 );

• NOTE: you cannot exceed 1024 threads per block (in total)
– threads_per_block( 16, 16, 1 ); 256 ✔
– threads_per_block( 32, 32, 1 ); 1024 ✔
– threads_per_block( 64, 64, 1 ); 4096 x

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

• Navigate into matVec_multiply_template/ directory

– Edit the file: matVec_multiply.cu

• Write CUDA kernel for this program

– HINT: Each thread will basically calculate the dot product of one row of the matrix A and
the vector x (i.e. calculate one element of resulting vector).

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

A0,0 A0,1 A0,2 A0,3 A0,4 x0 A0,0 * x0 + A0,1 * x1 + A0,2 * x2 + A0,3 * x3 + A0,4 * x4

A1,0 A1,1 A1,2 A1,3 A1,4 x1

A2,0 A2,1 A2,2 A2,3 A2,4 x2

=
A3,0 A3,1 A3,2 A3,3 A3,4 x3

x4

(4x5) (5x1) (4x1)

(MxN) (Nx1) (Mx1)

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

A0,0 A0,1 A0,2 A0,3 A0,4 x0 A0,0 * x0 + A0,1 * x1 + A0,2 * x2 + A0,3 * x3 + A0,4 * x4

A1,0 A1,1 A1,2 A1,3 A1,4 x1 A1,0 * x0 + A1,1 * x1 + A1,2 * x2 + A1,3 * x3 + A1,4 * x4

A2,0 A2,1 A2,2 A2,3 A2,4 x2

=
A3,0 A3,1 A3,2 A3,3 A3,4 x3

x4

(4x5) (5x1) (4x1)

(MxN) (Nx1) (Mx1)

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

A0,0 A0,1 A0,2 A0,3 A0,4 x0 A0,0 * x0 + A0,1 * x1 + A0,2 * x2 + A0,3 * x3 + A0,4 * x4

A1,0 A1,1 A1,2 A1,3 A1,4 x1 A1,0 * x0 + A1,1 * x1 + A1,2 * x2 + A1,3 * x3 + A1,4 * x4

A2,0 A2,1 A2,2 A2,3 A2,4 x2

= A2,0 * x0 + A2,1 * x1 + A2,2 * x2 + A2,3 * x3 + A2,4 * x4

A3,0 A3,1 A3,2 A3,3 A3,4 x3

x4

(4x5) (5x1) (4x1)

(MxN) (Nx1) (Mx1)

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

A0,0 A0,1 A0,2 A0,3 A0,4 x0 A0,0 * x0 + A0,1 * x1 + A0,2 * x2 + A0,3 * x3 + A0,4 * x4

A1,0 A1,1 A1,2 A1,3 A1,4 x1 A1,0 * x0 + A1,1 * x1 + A1,2 * x2 + A1,3 * x3 + A1,4 * x4

A2,0 A2,1 A2,2 A2,3 A2,4 x2

= A2,0 * x0 + A2,1 * x1 + A2,2 * x2 + A2,3 * x3 + A2,4 * x4

A3,0 A3,1 A3,2 A3,3 A3,4 x3 A3,0 * x0 + A3,1 * x1 + A3,2 * x2 + A3,3 * x3 + A3,4 * x4

x4

(4x5) (5x1) (4x1)

(MxN) (Nx1) (Mx1)

Introduction to CUDA C/C++

 2D CUDA Ex. – matrix vector multiply

• Navigate into matVec_multiply_template/ directory

– Edit the file: matVec_multiply.cu

• Write CUDA kernel for this program

– HINT: Each thread will basically calculate the dot product of one row of the matrix A and
the vector x (i.e. calculate one element of resulting vector).

• Assign the execution configuration parameters

– threads_per_block( ?, ?, ? );
– blocks_in_grid( ?, ?, ? );
– NOTE: This should be a 1D grid of threads

• When you finish, try changing values of M and N and adjust

threads_per_block accordingly

Introduction to CUDA C/C++

dim3 threads_per_block( 1, 128, 1 );
dim3 blocks_in_grid( 1, ceil( float(M) / threads_per_block.y ), 1 );

__global__ void multiply_mat_vec(int *a, int *b, int *c)

{
Each thread computes
int row = blockDim.y * blockIdx.y + threadIdx.y;
one element of
resulting vector
if(row < M) (i.e. sum of N element-
{ wise products)
for(int i=0; i<N; i++)
{ Move through the
y[row] = y[row] + a[row*N + i] * x[i]; 2D array with 1D
} indexing
}
(MxN) (Nx1) (Mx1)
}
A0,0 A0,1 A0,2 A0,3 A0,4 x0 A0,0 * x0 + A0,1 * x1 + A0,2 * x2 + A0,3 * x3 + A0,4 * x4 0

A1,0 A1,1 A1,2 A1,3 A1,4 x1 A1,0 * x0 + A1,1 * x1 + A1,2 * x2 + A1,3 * x3 + A1,4 * x4 1

A2,0 A2,1 A2,2 A2,3 A2,4 x2

= A2,0 * x0 + A2,1 * x1 + A2,2 * x2 + A2,3 * x3 + A2,4 * x4 2

A3,0 A3,1 A3,2 A3,3 A3,4 x3 A3,0 * x0 + A3,1 * x1 + A3,2 * x2 + A3,3 * x3 + A3,4 * x4 3

x4

Introduction to CUDA C/C++

 Device Queries

Introduction to CUDA C/C++

 Device Queries

• cudaDeviceProp
– C struct with many member variables

• cudaError_t cudaGetDeviceProperties(cudaDeviceProp *prop, int device)

– CUDA API: returns info about the device

• cudaError_t cudaGetDeviceCount(int *count)

– CUDA API: returns the number of CUDA-capable devices

Let’s look at an example problem

• intro_cuda/device_query/device_query.cu

Introduction to CUDA C/C++

 Device Queries

• Add output for memory clock frequency and memory bus width
– Google “cudaDeviceProp”, find the member variables and add print statements

• Add user-defined output for theoretical memory bandwidth (GB/s)

– BW (GB/s) = (memory clock rate) * (memory bus width)
GDDR5 - double data rate

Mem clock rate (kHz) Lanes in bus memory

“ticks per second” “bits per tick”
(returned from query) (returned from query)

BW (GB/s) = ((2.6×106 kHz)*(1×103 Hz / kHz)) * ((384 bits)*2)*(B / 8 bits)) * (GB / 1×109 B)

Convert from kHz to Hz Convert from bits to B Convert from B to GB

BW = ((memory clock rate in kHz)*1e3) * ((memory bus width in bits)*2) * (1/8) * (1/1e9) )

Introduction to CUDA C/C++

 Shared Memory

Introduction to CUDA C/C++

 Shared Memory

• Very fast on-chip memory

• Allocated per thread block

– Allows data sharing between threads in the same block
– Declared with __shared__ specifier

• Limited amount
– 49152 B per block

• Must take care to avoid race conditions. For example…

– Say, each thread writes the value 1 to one element of an array element.
– Then one thread sums up the elements of the array
– Synchronize with __syncthreads()
• Acts as a barrier until all threads reach this point

Introduction to CUDA C/C++

 Ex: intro_cuda/dot_product/dot_product.cu

x0 x1 x2 x3 x4 x5 y0
= x0 * y0 + x1 * y1 + x2 * y2 + x3 * y3 + x4 * y4 + x5 * y5
y1

y2

y3

y4

y5

Introduction to CUDA C/C++

 Ex: intro_cuda/dot_product/dot_product.cu
#define N 1024 NOTE: We are only using 1 thread block here!

int threads_per_block = 1024;

int blocks_in_grid = ceil( float(N) / threads_per_block);

__global__ void dot_prod(int *a, int *b, int *res)

Declare array of
{ shared memory,
__shared__ int products[N]; shared within a
int id = blockDim.x * blockIdx.x + threadIdx.x; grid block
products[id] = a[id] * b[id];
Each thread
__syncthreads(); Ensure all
calculates one
threads have
element-wise
if(id == 0) reached this
product
point before sum
{
int sum_of_products = 0;
for(int i=0; i<N; i++)
{ Thread 0 sums
sum_of_products = sum_of_products + products[i]; all elements of
} the array and
writes value to
*res = sum_of_products; *res.
}
}
Introduction to CUDA C/C++
 Multi-Block dot_product Program
The intro_cuda/dot_product/ program only works for a single block
• Can only perform dot product on arrays of at most 1024 elements (due to 1024 threads per
block limit)
• Shared memory is only shared among threads within the same block

In order to perform dot product on larger arrays, we need to use

multiple blocks
• Each block computes element-wise products and sum on at most 1024 elements
• Then, the results from each block can be summed together for final result

But when thread 0 from each block tries to update the (global) res
variable, thread 0 from another block might also be writing to it
• Data race condition!
• Solution: Atomic Functions

Introduction to CUDA C/C++

 Atomic Functions

From the CUDA Programming Guide:

An atomic function performs a read-modify-write atomic operation on one 32-bit or
64-bit word residing in global or shared memory.

int atomicAdd(int *address, int val)

• Reads a word at some address in global or shared memory, adds a number to it,
and writes the result back to the same address.
• The operation is atomic in the sense that it is guaranteed to be performed without
interference from other threads.
– No other thread can access this address until the operation is complete.

Introduction to CUDA C/C++

 Multi-Block dot_product Program
The program dot_product_multiBlock_template/ is currently identical
to dot_product/ program
• Edit the template version so it can use multiple blocks (for larger arrays)

To do so
• Edit the kernel so that each blocks computes only THREADS_PER_BLOCK
elements of the dot product (i.e. only a portion of the sum of products)
• Sum results from each block into global res variable using atomicAdd()

HINTS
• Each block’s (local) thread 0 should be computing a portion of the dot product.
– i.e. threadIdx.x instead of the global thread id

• In __shared__ int products[num_elements], num_elements must be

known at compile time
Introduction to CUDA C/C++
#define N 4096 Increase size of array
#define THREADS_PER_BLOCK 511 Set value of threads per block globally

int threads_per_block = THREADS_PER_BLOCK;

int blocks_in_grid = ceil( float(N) / threads_per_block);

__global__ void dot_prod(int *a, int *b, int *res) Since shared memory
{ is only shared among
__shared__ int products[THREADS_PER_BLOCK]; threads in same
int id = blockDim.x * blockIdx.x + threadIdx.x; block, only compute
products[threadIdx.x] = a[id] * b[id]; portion of dot product

__syncthreads(); Use thread 0 within each

block to sum its block’s
if(threadIdx.x == 0) portion of the dot product
{
int sum_of_products = 0;
for(int i=0; i<THREADS_PER_BLOCK; i++)
{ Sum results from
sum_of_products = sum_of_products + products[i]; individual blocks
} using atomicAdd()
to avoid race
atomicAdd(res, sum_of_products); condition
}
}
Introduction to CUDA C/C++
 Where to go from here
CUDA C Programming Guide
• https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

PARALLEL FORALL Blog (search through archives)

• https://devblogs.nvidia.com/parallelforall/

Programming Massively Parallel Processors: A Hands-on Approach

• David B. Kirk, Wen-mei Hwu (3rd Edition)

CUDA by Example: An Introduction to General-Purpose GPU Programming

• https://developer.nvidia.com/cuda-example

cuBLAS – Dense Linear Algebra on GPUs

• https://developer.nvidia.com/cublas

Things we didn’t cover

• Timing CUDA codes, NVIDIA visual profiler, many others

Introduction to CUDA C/C++