CS 3006

Parallel and Distributed Computing

Lecture 12
Danyal Farhat
FAST School of Computing
NUCES Lahore
Synchronization Constructs in OpenMP
Lecture’s Agenda
• Synchronization Clauses
• Barrier Directive
• Single Directive
• Master Directive
• Critical Section
Race Condition
• Atomic Directive
Lecture’s Agenda (Cont.)
• Environment Variables in OpenMP
OMP_NUM_THREADS
OMP_DYNAMIC
 OMP_Dynamic [dynamic.c]
OMP_SCHEDULE
OMP_NESTED
 OMP_NESTED [nested.c]

• Summary
• Additional Resources
Synchronization Clauses
• #pragma omp barrier
• #pragma omp single
• #pragma omp master
• #pragma omp critical
• #pragma omp atomic
• #pragma omp ordered
• #pragma omp flush[(list)]
• #pragma omp threadprivate(variable list)
Barrier Directive
• On encountering this directive, all threads in a team wait until
others have caught up, and then release
#pragma omp barrier
Single Directive
• A single directive specifies a structured block that is executed by a
single (arbitrary) thread in parallel region
• Implicit barrier
#pragma omp single [clause list]
structured block
Master Directive
• The master directive is a specialization of the single directive in
which only the master thread executes the structured block
• No implicit barrier
#pragma omp master
structured block
Critical Section
• A Critical Section is a code segment that has a shared variable and
need to be executed as an atomic action
It means that in a group of cooperating processes/threads, at a given point
of time, only one process must be executing its critical section

#pragma omp critical

• An operation acting on shared memory is atomic if it completes in a

single step relative to other threads (no other thread can observe
the modification half-complete}
Critical Section (Cont.)
• Forces threads to be mutex (mutually exclusive)
Only one thread at a time executes the given code section
double area, pi, x;
int i, n;
...
area = 0.0;
for (i = 0; i < n; i++) {
x += (i+0.5)/n; //can be calculated independently
area += 4.0/(1.0 + x*x); //requires mutex lock
}
pi = area / n;
Critical Section (Cont.)
• If we simply parallelize the loop... A race condition may occur
double area, pi, x;
int i, n;
...
area = 0.0;
#pragma omp parallel for private(x)
for (i = 0; i < n; i++) {
x = (i+0.5)/n;
area += 4.0/(1.0 + x*x); //not atomic
}
pi = area / n;
Critical Section (Cont.)
• Race Condition
Value of area Thread A Thread B

11.667
+ 3.765
11.667

15.432 + 3.563

15.230

• Thread A reads value of area first

• Thread B reads value of area before A can update its value
• Thread A updates value of area
• Thread B ignores update by A and writes its incorrect value to area
Critical Section (Cont.)
Race Condition
• A race condition is created when one process may “race ahead” of
another and overwrite the change made by the first process to the
shared variable
Critical Section (Cont.)
• Critical section: a portion of code that only thread at a time may
execute
• We denote a critical section by putting the pragma
#pragma omp critical [(name)]
• Optional identifier name can be used to identify a critical region
• Solves the problem but, as only one thread at a time may execute
the statement; it becomes sequential code
Critical Section (Cont.)
Critical Section:
double area, pi, x;
int i, n;
area = 0.0;
#pragma omp parallel for private(x)
for (i = 0; i < n; i++) {
x = (i+0.5)/n;
#pragma omp critical
area += 4.0/(1.0 + x*x);
}
pi = area / n;
Atomic Directive
• The atomic directive specifies that the single memory location
update should be performed as an atomic operation

#pragma omp atomic

Update instruction e.g., x++
Atomic Directive (Cont.)
#pragma omp parallel for
{
for (i=0; i++, i<n)
{
#pragma omp atomic // Shortcut for critical section
ic = ic + bigfunc(); // Little faster than critical overhead
}
printf("counter = %d\n", ic);
}
Supported functions
+, *, -, /, &, ^, |, <<, >>
Environment Variables in OpenMP
• OpenMP provides additional environment variables that help
control execution of parallel programs

• OMP_NUM_THREADS
• OMP_DYNAMIC
• OMP_SCHEDULE
• OMP_NESTED
Environment Variables in OpenMP (Cont.)
OMP_NUM_THREADS
• Specifies the default number of threads created upon entering a
parallel region
• The number of threads can be changed during run-time using:
omp_set_num_threads (int threads) routine [OR]
num_threads clause ---> num_threads(int threads)

• Setting OMP_NUM_THREADS to 4 using bash:

“export OMP_NUM_THREADS=4”
Environment Variables in OpenMP (Cont.)
OMP_DYNAMIC
• When set to TRUE, allows the number of threads to be controlled
at runtime
It means Openmp will use its dynamic adjustment algorithm to create
number of threads that may optimize system performance
In case of TRUE, total number of threads generated may not be equal to the
threads requested by using the omp_set_num threads() function or the
num_threads clause
In case of FALSE, usually total number of generated threads in a parallel
region become as requested by the num_threads clause
Environment Variables in OpenMP (Cont.)
OMP_DYNAMIC
• OpenMP routines for setting/getting dynamic status:

void omp_set_dynamic (int flag); //disables if flag=0

Should be called from outside of a parallel region

int omp_get_dynamic (); //return value of dynamic status

OMP_Dynamic [dynamic.c]
workers = omp_get_max_threads(); //can use num_procs
printf("%d maximum allowed threads\n", workers);
printf("total number of allocated cores are:%d\n", omp_get_num_procs());
omp_set_dynamic(1); // dynamic adjustment enabled
omp_set_num_threads(8);
printf("total number of requested when dynamic is true are:%d\n", 1);
#pragma omp parallel {
#pragma omp single nowait
printf("total threads in parallel region1=%d:\n", omp_get_num_threads());
#pragma omp for
for (i = 0; i < mult; i++)
{a = complex_func();}
}
OMP_DYNAMIC [dynamic.c] (Cont.)
omp_set_dynamic(0); // dynamic adjustment disabled
omp_set_num_threads(8);
printf("total number of requested when dynamic is false are:%d\n", 8);
#pragma omp parallel
{
#pragma omp single nowait
printf("total threads in parallel region2=%d:\n", omp_get_num_threads());
#pragma omp for
for (i = 0; i < mult; i++)
{a = complex_func();}
}
Environment Variables in OpenMP (Cont.)
OMP_SCHEDULE
• Controls the assignment of iteration spaces associated
with “for” directives that use the runtime scheduling class
• Possible values: static, dynamic, and guided
Can also be used along with chunk size [optional]
• If chunk size is not specified, then default chunk-size of 1 is
used
• Setting OMP_SCHEDULE to guided with minimum chunk
size of 4 using Ubuntu-based terminal:
“ export OMP_SCHEDULE= "guided,4” ”
Environment Variables in OpenMP (Cont.)
OMP_NESTED
• Default value is FALSE
While using nested parallel pragma inside another, the nested one is
executed by the original team instead of making new thread team
• When TRUE
Enables nested parallelism
While using nested parallel pragma code inside another, it makes a new
team of threads for executing the nested one.
• Use omp_set_nested(int val) with non-zero value to set
this variable to TRUE
When called with ‘0’ as argument, it set the variable to FALSE
OMP_NESTED [nested.c]
omp_set_nested(0);
#pragma omp parallel num_threads(2) {
#pragma omp single
printf("Level 1: number of threads in the team : %d\n",
omp_get_num_threads());

#pragma omp parallel num_threads(4)

{
#pragma omp single
printf("Level 2: number of threads in the team : %d\n",
omp_get_num_threads());
}
}
OMP_NESTED [nested.c] (Cont.)
omp_set_nested(1);
#pragma omp parallel num_threads(2) {
#pragma omp single
printf("Level 1: number of threads in the team : %d\n",
omp_get_num_threads());

#pragma omp parallel num_threads(4)

{
#pragma omp single
printf("Level 2: number of threads in the team : %d\n",
omp_get_num_threads());
}
}
Summary
• Synchronization Clauses
Barrier, single, master, critical, atomic etc.
• Barrier Directive
On encountering this directive, all threads in a team wait until others have
caught up, and then release
• Single Directive
A single directive specifies a structured block that is executed by a single
(arbitrary) thread in parallel region
• Master Directive
The master directive is a specialization of the single directive in which only
the master thread executes the structured block
Summary (Cont.)
• Critical Section
A Critical Section is a code segment that has a shared variable and need to
be executed as an atomic action

• Race Condition
A race condition is created when one process may “race ahead” of another
and overwrite the change made by the first process to the shared variable

• Atomic Directive
The atomic directive specifies that the single memory location update
should be performed as an atomic operation
Summary (Cont.)
• Environment Variables in OpenMP
OMP_NUM_THREADS
OMP_DYNAMIC
 OMP_Dynamic [dynamic.c]
OMP_SCHEDULE
OMP_NESTED
 OMP_NESTED [nested.c]
Additional Resources
• Introduction to Parallel Computing by Ananth Grama and
Anshul Gupta

Chapter 7: Programming Shared Address Space Platforms

• Introduction to OpenMP by Tim Mattson (Intel)

https://www.youtube.com/watch?v=nE-xN4Bf8XI&list=PLLX-
Q6B8xqZ8n8bwjGdzBJ25X2utwnoEG&index=1
Questions?