Parallel processing

• Parallel computing means doing several task

simultaneously, thus improving the performance
of the computer system
• Parallel processing is a type of computing where
numerous instruction are implemented at same
time on the principle that larger problems may
be divided into smaller ones that are solved in
parallel
• If a computer system provide the technique for
simultaneous processing of different set of data ,
the compu te r s y s te m i s s a i d to be u s i n g a
parallel processing technique
Parallel processing
• The different type of parallel computing
are
⮚ Bit level parallelism
⮚ Data parallelism
⮚ Task parallelism
The various technique by which parallel
processing can be achieved are as
⮚ pipelining
⮚ Vector processing
⮚ Parallel processors
Parallel processing
⮚ Parallel processing is a term used to denote a large class of techniques that
are used to provide simultaneous data processing task for the purpose of
increasing computational speed of computer system
⮚ Instead of processing each instruction sequentially as in a conventional
computer, a parallel processing system is able to perform concurrent data
processing to achieve faster execution time
⮚ For example, while an instruction is being executed in the ALU , the next
instruction can be read from memory
⮚ The system may have two or more ALU and be able to execute two or
more instructions at the same time.
⮚ Furthermore the system may have two or more processors operating
concurrently
⮚ The purpose of parallel processing is to speed up the computer processing
capability and increase the throughput, that is the amount of processing
that can be accomplished during a given interval in time
⮚ The amount of hardware increase with parallel processing, and with it , the
cost of the system increases
Parallel processing
⮚ Parallel processing can be viewed from various levels of complexity. At
the lowest level , we distinguish between parallel and serial operations by
the type of registers used.
⮚ Shift register operate in serial fashion one bit at a time while registers with
parallel load operate with all the bits of the word simultaneously
⮚ Parallel processing at a higher level of complexity can be achieved by
having a multiplicity of functional units that perform identical or different
operations simultaneously
⮚ Parallel processing is established by distributing the data among the
multiple functional units
⮚ For example the arithmetic logic and shift operations can be separated into
three units and the operands diverted to each units under the supervision of
control unit
Parallel processing
⮚ The possible way of separating the execution unit into eight
functional units operating in parallel.
Parallel processing
⮚ The operands in the registers are applied to one of the units depending on
the operation specified by the instruction associated with the operands
⮚ The operation performed in each functional unit is indicated in each block
of the diagram
⮚ The adder and integer multiplier perform the arithmetic operations with
integer numbers
⮚ The floating point operations are separated into three circuits operating in
parallel
⮚ The logic shift and increment operations can be performed concurrently on
different data
⮚ All units are independent of each other , so one number can be shifted
while another number is being incremented
⮚ A multifunctional organization is usually associated with a complex
control unit to coordinate all the activities among the various components
Parallel processing
⮚ There are variety of ways that parallel processing can be classified
⮚ It can be considered from the internal organization of the processors from
the interconnection structure between processors, or from the flow of
information through the system
⮚ One classification introduced by M J Flyn considers the organization of a
computer system by the number of instructions and data items that are
manipulated simultaneously
⮚ The sequence of instructions read from memory constitutes an instruction
stream
⮚ The operation performed on the data in the processor constitutes a data
stream
⮚ Four major classification are
Multiple Processor Organization
• Single instruction stream, single data
stream - SISD
• Single instruction stream, multiple data
stream - SIMD
• Multiple instruction stream, single data
stream - MISD
• Multiple instruction stream, multiple data
stream- MIMD
 Flynn’s Taxonomy

⮚ Flynn’s Taxonomy uses two basic concepts:

Parallelism in instruction stream, and
parallelism in data stream.
⮚ A n CPU system has n program counter, so
there are n “instruction stream” that can
execute in parallel.
⮚ A data stream can be used as a sequence of
data, and there exist 4 possible combinations.
Flynn’s Classification
 Flynn’s Classification

Insrtuction Data Name Examples

Set Streams
1 1 SISD Von Neumann Machine
1 Multiple SIMD Vector Super Computer
Multiple 1 MISD Arguably None

Multiple Multiple MIMD Multiprocessor, Multicomputer

 SISD(Single Instruction Single Data)
⮚ A processor that can only do one job at a
time from start to finish.
Single Instruction, Single Data Stream -
SISD
• SISD represent the organization of a single
computer containing a control unit, a processor
unit and a memory unit
• Instructions are executed sequentially and the
system may or may not have internal parallel
processing capabilities
• Parallel processing in this case may be achieved
by means of multiple functional units
 SIMD(Single Instruction Multiple Data)

⮚ Single CU and multiple PEs

⮚ CU fetches an instruction from memory and
after decoding, broadcasts control signals to all
PEs.
⮚ That is, at any given time, all PEs are
Synchronously executing the same.
⮚ Instruction but on different setsof data; hence
the name SIMD
SIMD(Single Instruction
Multiple Data)
SIMD(Single Instruction Multiple Data)

⮚ SIMD represent an organization that include many

processing units under the supervision of a common control
unit
⮚ All processor receive the same instruction from the control
unit but operate on different items of data
⮚ The shared memory unit must contain multiple modules so
that it can communicate with all the processors
simultaneously
Single Instruction, Multiple Data Stream
- SIMD
• Single machine instruction
• Controls simultaneous execution
• Number of processing elements
• Lockstep basis
• Each processing element has associated
data memory
• Each instruction executed on different set
of data by different processors
• Vector and array processors
MISD(Multiple instructions single data)

⮚ A rare type, since data throughput is limited.

MISD structure is only of theoretical
interest since no practical system has been
constructed using this organization
Multiple Instruction, Single Data Stream
- MISD
• Sequence of data
• Transmitted to set of processors
• Each processor executes different
instruction sequence
• Never been implemented
MIMD(Multiple instruction Multiple Data)

⮚ A MIMD is a true multiprocessor

⮚ In contrast to SIMD, a MIMD is a general-
purpose machine.
⮚ When all the processor in MIMD are
running the same program, we call it Single
Program Multiple Data(SPMD) computation.
⮚ The SPMD model is widely used by many
parallel platforms.
MIMD(Multiple instruction Multiple Data)
Multiple Instruction, Multiple Data
Stream- MIMD
• Set of processors
• Simultaneously execute different
instruction sequences
• Different sets of data
• SMPs, clusters and NUMA systems
MIMD - Overview
• General purpose processors
• Each can process all instructions
necessary
• Further classified by method of processor
communication
Pipeline Computers
• Pipelining is a technique of decomposing a
sequential process into sub operations with each
sub process being executed in a special
dedicated segment that operate concurrently
with all other segments
• A pipeline can be visualized as a collection of
processing segments through which binary
information flows
• Each segment performs partial processing
dictated by the way the task is portioned
• The result obtained from the computation in
each segment is transferred to the next segment
in the pipeline
• The final result is obtained after the data have
passed throuhg all the segments
Pipeline Computers
• The name pipeline implies a flow of information
analogous to an industrial assembly line
• It is characteristics of pipelines that several
computations can be in progress in distinct
segment at the same time
• The overlapping of computation is made possible
by associating a register with each segment in
the pipeline
• The registers provide isolation between each
segment so that each can operate on distinct
data simultaneously
Pipeline Computers
• Pipeline structure is imagine that each segment
consist of an input register followed by a
combinational circuit
• The register holds the data and the
combinational circuit performs the sub operation
in the particular segment
• The output of the combinational circuit in a given
segment is applied to the input register of the
next segment
• clock is applied to all registers after enough time
has elapsed to perform all segment activity
• In this way the information flows through the
pipeline one step at a time
Pipeline Computers
• The pipeline organization example. Suppose we
want to perform the combined multiply and add
operations with a stream of numbers
• Ai * Bi + Ci i=1,2,3,4,5,6,7
• Each sub operation is to be implemented in a
segment within a pipeline
• Each segment has one or two registers and a
combinational circuit shown in the figure
• R1 through R5 are registers that receive new
data with every clock pulse. The multiplier and
adder are combinational circuit
• The sub operation performed in each segment of
the pipeline are as follows
Pipeline Computers
•R1<-Ai R2<-Bi input Ai and Bi
•R3<-R1*R2 R4<-Ci multiply and input Ci
•R5<-R3+R4 Add Ci to product
•The five registers are loaded with new data every
clock pulse.
•The first clock pulse transfers A1 and B1 into R1
and R2.
•The second clock pulse transfers the product of r1
and r2 into R3 and C1 into r4
•The same clock pulse transfer A2 and B2 into R1
and r2
•The third clock pulse operates on all three
segments simultaneously
Pipeline Computers
•It pluses A3 and B3 into r1 and r2 transfers the
product of r1 and R2 into R3 transfers C2 into r4
and places the sum of R3 and r4 into r5.
•It take 3 clock pulses to fill up the pipe and
retrieve the first output from r5
•From there on each clock produces a new output
and moves the data one step down the pipeline
•When no more input data available ,the clock
must continue until the last output emerges out of
the pipeline
Pipeline Computers
Pipeline Computers