CPU Parallelism &

GPU Parallelism
By ADITYA SINGH CHAUHAN (21027105)
CPU
Parallelism

Understanding Parallelism in Computing

In the world of computing, parallelism plays a pivotal role in CPU Parallelism

achieving higher performance and efficiency. It refers to the The Core of Parallel Processing Modern CPUs (Central
simultaneous execution of multiple tasks, which can Processing Units) have evolved significantly from their single-
significantly enhance the throughput and capabilities of a core predecessors. Today, most CPUs come equipped with
system. Parallelism can be harnessed at various levels in multiple processing cores, each capable of executing
computing, with the most prominent ones being CPU instructions independently. This fundamental shift in CPU
parallelism, GPU parallelism etc. design has ushered in the era of CPU parallelism.
Multicore Architectures
The core component of CPU parallelism is the multicore
architecture. Traditional CPUs were single-core,
meaning they could execute one instruction at a time.
However, as the demand for increased computing
power grew, CPU manufacturers turned to multicore
designs. These designs incorporate two or more
individual processing cores on a single CPU chip.

Dual-Core and Beyond

The most common multicore CPUs are dual-core,
quad-core, hexa-core, octa-core, and even more. A
dual-core CPU, for example, has two processing cores
on a single chip, while an octa-core CPU boasts eight
cores. These cores are capable of running tasks in
parallel, greatly enhancing the CPU's overall
performance.
 Instruction-Level Thread-Level
Parallelism (ILP) Parallelism (TLP)

Types of CPU Parallelism:

 Instruction-Level Thread-Level
Parallelism (ILP) Parallelism (TLP)
Instruction-Level Parallelism focuses on the Thread-Level Parallelism involves running multiple
execution of multiple instructions from a single threads or processes concurrently. Threads are
program in parallel. It optimizes the utilization of individual sequences of instructions that can be
CPU resources by allowing various stages of scheduled and executed independently. In a
instruction execution to overlap. Several multicore CPU, different threads can run on
techniques enable ILP, including: separate cores, harnessing the power of
parallelism.

Pipelining Superscalar Execution Problem 3

This involves breaking down the
Superscalar CPUs can execute
execution of instructions into
more than one instruction per
stages, where each stage is
clock cycle by dispatching
handled by a different CPU
multiple instructions to various
component. As one instruction
execution units simultaneously.
proceeds to the next stage, the
CPU can start processing the
next instruction, effectively
increasing throughput.
 Multi-Threaded Software
Multi-threaded applications are designed to split their
workload into threads that can run in parallel. Common
examples include web browsers (where each tab can
run as a separate thread), multimedia processing, and
database management systems.

Scientific Computing
Tasks such as simulations, weather modeling, and
molecular dynamics calculations can greatly benefit
from CPU parallelism. These tasks often involve complex
mathematical computations that can be parallelized to

Applications of reduce processing time.

CPU Parallelism Server and Data Center Workloads

CPU parallelism finds applications in various
fields and scenarios, where tasks can be In data centers and servers, CPU parallelism is crucial

broken down into smaller sub-tasks for for handling multiple user requests simultaneously. Web

concurrent execution. Some notable servers, database servers, and virtualization

applications include: environments all rely on CPU parallelism to ensure

efficient resource utilization.
 Synchronization
In multi-threaded applications, threads often need to
access shared resources like memory or data structures.
Synchronization mechanisms are required to ensure that
these resources are accessed in a coordinated manner to
avoid conflicts. Common synchronization tools include
locks, semaphores, and barriers.

Data Consistency
Parallel execution can lead to issues with data consistency.
When multiple threads or cores read and write to the same
data simultaneously, it's essential to manage data consistency
to ensure that the results are accurate. Techniques like atomic

Challenges in operations and memory fences are employed to address this

challenge

CPU Parallelism
Race Conditions
While CPU parallelism offers significant A race condition occurs when multiple threads access and
performance improvements, it also modify shared data simultaneously, leading to unpredictable
presents several challenges that must and erroneous results. Detecting and preventing race
be addressed: conditions is a critical aspect of parallel programming. Tools
like thread-safe data structures and careful coding practices
are used to mitigate race conditions.
GPU
Parallelism
Graphics Processing Units (GPUs) are another critical component of modern computing, renowned for their exceptional parallel
processing capabilities. Originally designed for rendering graphics, GPUs have evolved into powerful general-purpose processors
capable of handling a wide range of parallel workloads.

GPU Architecture

Shader Cores SIMD (Single Instruction, Multiple Data) Execution

The heart of GPU parallelism lies in its shader cores. A shader GPUs excel in tasks that can be parallelized across many data
core is a small processing unit optimized for parallel elements. They use SIMD execution, which means that a
computation. GPUs contain a vast number of shader cores, single instruction is applied to multiple data points
often numbering in the thousands. These cores work in simultaneously. This design makes GPUs incredibly efficient
harmony to perform parallel calculations. for tasks like matrix multiplication, image processing, and
scientific simulations.
Programming Model for GPU Parallelism
To harness the power of GPU parallelism, developers use specialized programming models and APIs
(Application Programming Interfaces). Two of the most prominent GPU programming models are CUDA and
OpenCL:

CUDA (Compute Unified Device OpenCL (Open Computing

Architecture) Language)
Developed by NVIDIA, CUDA is a OpenCL is an open-standard
programming model and parallel programming framework supported
computing platform designed by various GPU vendors, including
specifically for NVIDIA GPUs. It AMD, Intel, and NVIDIA. It enables
provides a straightforward way to developers to write code that can
write GPU-accelerated run on a variety of GPUs and CPUs,
applications using C/C++ or making it a versatile choice for
Python. heterogeneous computing.

Market Size
 Machine Learning and
Graphics Rendering Deep Learning

Applications
The GPU's original purpose was The training and inference phases
graphics rendering. It can rapidly of machine learning and deep
process the multitude of pixels learning models involve performing

of GPU required for high-definition

graphics, making it indispensable
numerous matrix multiplications
and complex calculations. GPUs,

Parallelism for gaming, computer-aided

design (CAD), and video editing.
with their massive parallelism,
accelerate these tasks significantly,
enabling the rapid development of
GPU parallelism has found applications AI models.
in diverse fields due to its ability to
handle massively parallel workloads
efficiently. Some notable applications
include:
Scientific Simulations Cryptocurrency Mining
Scientific simulations, such as those Cryptocurrency mining relies on
in physics, chemistry, and climate solving complex mathematical
modeling, often require performing problems, which can be
extensive calculations on vast parallelized and executed
datasets. GPUs excel at these efficiently on GPUs. This has led to
simulations by parallelizing the widespread use of GPUs in the
computations and reducing cryptocurrency mining community.
processing times.
 Challenges in
GPU Parallelism
GPU parallelism is incredibly powerful, but it also
comes with its own set of challenges:

Data Transfer Bottlenecks Thread Divergence Limited Memory

Transferring data between the In SIMD execution, all threads within a GPUs have limited memory
CPU and GPU can be a warp (a group of threads) execute the compared to CPUs. Managing
bottleneck in GPU computing. same instruction, but they may take memory efficiently and
Efficient memory management different code paths. This can lead to avoiding memory-related
and minimizing data transfer thread divergence, where some issues, such as out-of-memory
overhead are crucial for threads are idle while others are errors, is vital in GPU
maximizing GPU performance. active. Optimizing code to minimize programming.
thread divergence is essential for
efficient GPU parallelism.
THANK YOU