Enhancing Performance

with
PIPELINING
 Pipelining

• Pipeline concepts
• Hazards
• Example
 Pipelined vs. Single-Cycle
Instruction Execution
Program
execution 2 4 6 8 10 12 14 16 18
order Time
(in instructions)
Instruction Data Single-cycle
lw $1, 100($0) fetch
Reg ALU
access
Reg

Instruction Data
lw $2, 200($0) 8 ns fetch
Reg ALU
access
Reg

Instruction
lw $3, 300($0) 8 ns fetch
...
8 ns

Assume 2 ns for memory access, ALU operation; 1 ns for register access:

therefore, single cycle clock 8 ns; pipelined clock cycle 2 ns.
Program
execution 2 4 6 8 10 12 14
Time
order
(in instructions)
Instruction Data
lw $1, 100($0) Reg ALU Reg
fetch access

Instruction Data
Pipelined
lw $2, 200($0) 2 ns Reg ALU Reg
fetch access

Instruction Data
lw $3, 300($0) 2 ns Reg ALU Reg
fetch access

2 ns 2 ns 2 ns 2 ns 2 ns
 Pipeline Implementation
• Idea:
– Goal of MIPS: CPI <= 1
– Some instructions take longer to execute than others
– Don’t want cycle time to depend on slowest instruction
– Want 100% hardware utilization
– Split execution of each instruction into several, balanced “stages”
– Each stage is a block of combinational logic
– Latency of each stage fits within 1 clock cycle
– Insert registers between each pipeline stage to hold intermediate results
– Execute each of these steps in parallel for a sequence of instructions

• This is called pipelining

 Pipelining MIPS

• What makes it easy with MIPS?

– all instructions are same length
• so fetch and decode stages are similar for all instructions
– just a few instruction formats
• simplifies instruction decode and makes it possible in one stage
– memory operands appear only in load/stores
• so memory access can be deferred to exactly one later stage
– operands are aligned in memory
• one data transfer instruction requires one memory access stage
 MIPS pipeline stages
• Fetch (F)
– read next instruction from memory, increment address counter
– assume 1 cycle to access memory
• Decode (D)
– read register operands, resolve instruction in control signals, compute branch
target
• Execute (E)
– execute arithmetic/resolve branches
• Memory (M)
– perform load/store accesses to memory, take branches
– assume 1 cycle to access memory
• Write back (W)
– write arithmetic results to register file
 Hazards
• Hazards are data flow problems that arise as a result of pipelining
– Limits the amount of parallelism, sometimes induces “penalties” that prevent one
instruction per clock cycle

– Structural hazards
• Two operations require a single piece of hardware e.g. Memory
• Structural hazards can be overcome by adding additional hardware

– Control hazards
• Need to worry about branch instructions, requiring subsequent instruction fetches to be
predicted
– Flushed if prediction does not hold (make sure no state change)
• Branch hazards can use dynamic prediction/speculation, branch delay slot

– Data hazards
• Instruction from one pipeline stage is “dependant” of data computed in previous pipeline
stage
 Structural Hazards
• Inadequate hardware to simultaneously support all instructions in
the pipeline in the same clock cycle
• E.g., suppose single – not separate – instruction and data
memory in pipeline below with one read port
 Structural Hazards
• E.g., suppose single – not separate – instruction and data memory in
pipeline below with one read port
– then a structural hazard between first and fourth lw instructions
Program
execution 2 4 6 8 10 12 14
Time
order
(in instructions)
Instruction Data
lw $1, 100($0) Reg ALU Reg
fetch access Pipelined
Instruction Data
lw $2, 200($0) 2 ns Reg ALU Reg
fetch access
Hazard if single memory
Instruction Data
lw $3, 300($0) 2 ns Reg ALU Reg
fetch access

Instruction Data
lw $4, 400($0) Reg ALU Reg
2 ns fetch access

2 ns 2 ns 2 ns 2 ns 2 ns

• Structural hazards are easy to avoid!

– by adding additional hardware
 Control Hazards
• Need to make a decision based on the result of a previous instruction still executing
in pipeline
 Control Hazards
• Control hazard: need to make a decision based on the result of a previous instruction
still executing in pipeline
• Solution 1 Stall the pipeline

Program
execution 2 4 6 8 10 12 14 16
order Time
(in instructions)
Instruction
Reg ALU
Data
Reg Note that branch outcome is
add $4, $5, $6 fetch access computed in ID stage with
Instruction Data added hardware (later…)
beq $1, $2, 40 Reg ALU Reg
2ns fetch access

Instruction Data
lw $3, 300($0) bubble fetch
Reg ALU
access
Reg

4 ns 2ns

Pipeline stall
 Control Hazards
• Solution 2 Predict branch outcome
– e.g., predict branch-not-taken :
Program
execution 2 4 6 8 10 12 14
order Time
(in instructions)
Instruction Data
add $4, $5, $6 fetch
Reg ALU
access
Reg

Instruction Data
beq $1, $2, 40 Reg ALU Reg
2 ns fetch access

Instruction Data
lw $3, 300($0) Reg ALU Reg
2 ns fetch access

Prediction success
Program
execution 2 4 6 8 10 12 14
order Time
(in instructions)
Instruction Data
add $4, $5 ,$6 Reg ALU Reg
fetch access

Instruction Data
beq $1, $2, 40 Reg ALU Reg
fetch access
2 ns
bubble bubble bubble bubble bubble

Instruction Data
or $7, $8, $9 Reg ALU Reg
fetch access
4 ns
Prediction failure: undo (=flush) lw
 Control Hazards
• Solution 3 Delayed branch: always execute the sequentially next statement with
the branch executing after one instruction delay
– compiler’s job to find a statement that can be put in the slot that is independent
of branch outcome

Program
execution 2 4 6 8 10 12 14
order Time
(in instructions)

beq $1, $2, 40 Instruction Data

Reg ALU Reg
fetch access

add $4, $5, $6 Instruction Data

Reg ALU Reg
(d elayed branch slot) 2 ns fetch access

Instruction Data
lw $3, 300($0) Reg ALU Reg
2 ns fetch access

2 ns

Delayed branch beq is followed by add that is

independent of branch outcome
 Data Hazards
• Data hazard: instruction needs data from the result of a previous instruction still
executing in pipeline
• Solution Forward data if possible…

2 4 6 8 10
Time
Instruction pipeline diagram:
add $s0, $t0, $t1 IF ID EX MEM WB shade indicates use –
left=write, right=read

Program
execution 2 4 6 8 10
order Time
(in instructions)
add $s0, $t0, $t1 IF ID EX MEM WB
Without forwarding – blue line –
data has to go back in time;
with forwarding – red line –
sub $t2, $s0, $t3
data is available in time
IF ID EX MEM WB
 Data Hazards
• Forwarding may not be enough
– e.g., if an R-type instruction following a load uses the result of the load –
called load-use data hazard
2 4 6 8 10 12 14
Program Time
execution
order
(in instructions)
Without a stall it is impossible
lw $s0, 20($t1) IF ID EX MEM WB
to provide input to the sub
instruction in time

sub $t2, $s0, $t3 IF ID EX MEM WB

2 4 6 8 10 12 14
Program Time
execution
order
(in instructions)
With a one-stage stall, forwarding
lw $s0, 20($t1) IF ID EX MEM WB
can get the data to the sub
instruction in time
bubble bubble bubble bubble bubble

sub $t2, $s0, $t3 IF ID EX MEM WB

 Reordering Code to Avoid Pipeline Stall
(Software Solution)
• Example:
lw $t0, 0($t1)
lw $t2, 4($t1)
Data hazard
sw $t2, 0($t1)
sw $t0, 4($t1)

• Reordered code:
lw $t0, 0($t1)
lw $t2, 4($t1)
sw $t0, 4($t1) Interchanged
sw $t2, 0($t1)
 Pipelined Datapath
Recall the 5 steps in instruction execution
1. Instruction Fetch & PC Increment (IF)
2. Instruction Decode and Register Read (ID)
3. Execution or calculate address (EX)
4. Memory access (MEM)
5. Write result into register (WB)
 Review - Single-Cycle
Datapath “Steps”
ADD

4 ADD

PC <<2
Instruction I
ADDR RD
32 16 32
5 5 5
Instruction
Memory RN1 RN2 WN
RD1 Zero
Register File ALU
WD
RD2 M
U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD
N
D

IF ID EX MEM WB
Instruction Fetch Instruction Decode Execute/ Address Calc. Memory Access Write Back
 Pipelined Datapath – Key Idea
• What happens if we break the execution into multiple cycles,
but keep the extra hardware?
– Answer: We may be able to start executing a new instruction at each
clock cycle - pipelining
• …but we shall need extra registers to hold data between
cycles – pipeline registers
 Pipelined Datapath
Pipeline registers wide enough to hold data coming in
ADD

4 ADD
64 bits 128 bits
PC <<2 97 bits 64 bits
Instruction I
ADDR RD
32 16 32
5 5 5
Instruction
Memory RN1 RN2 WN
RD1
Zero
Register File ALU
WD
RD2 M
U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD
N
D

IF/ID ID/EX EX/MEM MEM/WB

 Pipelined Datapath
Pipeline registers wide enough to hold data coming in
ADD

4 ADD
64 bits 128 bits
PC <<2 97 bits 64 bits
Instruction I
ADDR RD
32 16 32
5 5 5
Instruction
Memory RN1 RN2 WN
RD1
Zero
Register File ALU
WD
RD2 M
U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD
N
D

IF/ID ID/EX EX/MEM MEM/WB

Only data flowing right to left may cause hazard…, why?
 Bug in the Datapath

IF/ID ID/EX EX/MEM MEM/WB

ADD

4 ADD

PC <<2
Instruction I
ADDR RD
32 16 32
5 5 5
Instruction
Memory RN1 RN2 WN
RD1
Register File ALU
WD
RD2 M
U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD
N
D

Write register number comes from another later instruction!

 Corrected Datapath
IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4 64 bits 133 bits
102 bits 69 bits
<<2
PC
ADDR RD 5
RN1 RD1
32
ALU Zero
Instruction RN2
5
Memory Register
5
WN File RD2 M
WD U ADDR
X
Data
E Memory RD M
U
16 X 32 X
T WD
N
5 D

Destination register number is also passed through ID/EX, EX/MEM

and MEM/WB registers, which are now wider by 5 bits
 Pipelined Example
• Consider the following instruction sequence:
lw $t0, 10($t1)
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
LW
Clock Cycle 1
IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
SW
Clock Cycle 2 LW

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
ADD
Clock Cycle 3 SW LW

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
Clock Cycle 4 ADD SW LW
SUB

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
Clock Cycle 5 SUB ADD SW LW

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
Clock Cycle 6 SUB ADD SW

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
Clock Cycle 7 SUB ADD

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Single-Clock-Cycle Diagram:
Clock Cycle 8 SUB

IF/ID ID/EX EX/MEM MEM/WB

ADD
ADD
4
<<2
PC
ADDR RD RN1 RD1
32 5 Zero
Instruction RN2 ALU
5
Memory Register
WN File
5 RD2 M
WD U ADDR
X
Data
Memory RD M
E U
16 X 32 X
T WD

lw $t0, 10($t1) 5
N
D
sw $t3, 20($t4)
add $t5, $t6, $t7
sub $t8, $t9, $t10
 Tutorial Question
• Write the significant difference in the execution of an R-type
instruction between multicycle and pipelined implementations.
 Recall Single-Cycle Control –
the Datapath
0
M
u
x
ALU
Add result 1
Add Shift PCSrc
RegDst left 2
4 Branch
MemRead
Instruction [31 26] MemtoReg
Control
ALUOp
MemWrite
ALUSrc
RegWrite

Instruction [25 21] Read

PC Read register 1
address Read
Instruction [20 16] data 1
Read
register 2 Zero
Instruction 0 Registers Read ALU ALU
[31– 0] 0 Read
M Write data 2 result Address 1
Instruction u register M data
u M
memory Instruction [15 11] x u
1 Write x Data
data x
1 memory 0
Write
data
16 32
Instruction [15 0] Sign
extend ALU
control

Instruction [5 0]
Recall Single-Cycle – ALU Control
Instruction AluOp Instruction Funct Field Desired ALU control
opcode operation ALU action input
LW 00 load word xxxxxx add 010
SW 00 store word xxxxxx add 010
Branch eq 01 branch eq xxxxxx subtract 110
R-type 10 add 100000 add 010
R-type 10 subtract 100010 subtract 110
R-type 10 AND 100100 and 000
R-type 10 OR 100101 or 001
R-type 10 set on less 101010 set on less 111

ALUOp Funct field Operation

ALUOp1 ALUOp0 F5 F4 F3 F2 F1 F0
0 0 X X X X X X 010
0 1 X X X X X X 110
1 X X X 0 0 0 0 010
1 X X X 0 0 1 0 110
1 X X X 0 1 0 0 000
1 X X X 0 1 0 1 001
1 X X X 1 0 1 0 111
Truth table for ALU control bits
 Recall Single-Cycle – Control Signals
Effect of control bits
Signal Name Effect when deasserted Effect when asserted

RegDst The register destination number for the The register destination number for the
Write register comes from the rt field (bits 20-16) Write register comes from the rd field (bits 15-11)
RegWrite None The register on the Write register input is written
with the value on the Write data input
AlLUSrc The second ALU operand comes from the The second ALU operand is the sign-extended,
second register file output (Read data 2) lower 16 bits of the instruction
PCSrc The PC is replaced by the output of the adder The PC is replaced by the output of the adder
that computes the value of PC + 4 that computes the branch target
MemRead None Data memory contents designated by the address
input are put on the first Read data output
MemWrite None Data memory contents designated by the address
input are replaced by the value of the Write data input
MemtoReg The value fed to the register Write data input The value fed to the register Write data input
comes from the ALU comes from the data memory

Memto- Reg Mem Mem

Deter- Instruction RegDst ALUSrc Reg Write Read Write Branch ALUOp1 ALUp0
mining R-format 1 0 0 1 0 0 0 1 0
control lw 0 1 1 1 1 0 0 0 0
bits sw X 1 X 0 0 1 0 0 0
beq X 0 X 0 0 0 1 0 1
 Pipeline Control

• Initial design – motivated by single-cycle datapath control – use the

same control signals
• Observe:
Will be
– No separate write signal for the PC as it is written every cycle modified
– No separate write signals for the pipeline registers as they are written every by hazard
detection
cycle unit!!
– No separate read signal for instruction memory as it is read every clock cycle
– No separate read signal for register file as it is read every clock cycle
• Need to set control signals during each pipeline stage
• Since control signals are associated with components active during a
single pipeline stage, can group control lines into five groups according to
pipeline stage
 Pipelined Datapath with
Control I PCSrc

0
M
u
x
1

IF/ID ID/EX EX/MEM MEM/WB

Add

Add
4 Add
result
Branch
Shift
RegWrite left 2

Read MemWrite
Instruction

PC Address register 1 Read

Read data 1 ALUSrc
register 2 Zero
Zero MemtoReg
Instruction
Registers Read ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u M
Data u
Write x memory
data x
1
0
Write
data
Instruction
[15– 0] 16 32 6
Sign ALU
extend control MemRead

Same control Instruction

[20– 16]

signals as the
0
M ALUOp
Instruction u

single-cycle [15– 11]

1
x

datapath RegDst
 Pipeline Control Signals
• There are five stages in the pipeline
– instruction fetch / PC increment Nothing to control as instruction memory
read and PC write are always enabled
– instruction decode / register fetch
– execution / address calculation
– memory access
– write back
Write-back
Execution/Address Calculation Memory access stage stage control
stage control lines control lines lines
Reg ALU ALU ALU Mem Mem Reg Mem to
Instruction Dst Op1 Op0 Src Branch Read Write write Reg
R-format 1 1 0 0 0 0 0 1 0
lw 0 0 0 1 0 1 0 1 1
sw X 0 0 1 0 0 1 0 X
beq X 0 1 0 1 0 0 0 X
Pipeline Control Implementation

• Pass control signals along just like the data – extend each pipeline register to hold needed control bits for succeeding stages

WB

Instruction
Control M WB

EX M WB
• Note: The 6-bit funct field of the instruction required in the EX stage to generate ALU control can be retrieved as the 6 least
significant bits of the immediate field which is sign-extended and passed from the IF/ID register to the ID/EX register

IF/ID ID/EX EX/MEM MEM/WB

 Pipelined Datapath with
PCSrc
Control II
ID/EX
0
M
u WB
x EX/MEM
1
Control M WB
MEM/WB

EX M WB
IF/ID

Add

Add
4 Add result

RegWrite
Branch
Shift
left 2

MemWrite
ALUSrc
Read

MemtoReg
Instruction

PC Address register 1
Read
data 1
Read
register 2 Zero
Instruction
Registers Read ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction 16 32 6
[15– 0]
Control signals
Sign ALU MemRead
extend control

emanate from Instruction

[20– 16]
0 ALUOp
the control Instruction
M
u

portions of the [15– 11] x

1

pipeline registers
RegDst
 IF: lw $10, 20($1) ID: before<1> EX: before<2> MEM: before<3> WB: before<4>

Pipelined
IF/ID ID/EX EX/MEM MEM/WB
0
M 00 00
u WB
x
1 000 000 00
Control M WB
0 0 0

Execution
0000 00 0
EX M WB 0
0 0

Add

Add
4 Add result

RegWrite
and
Shift Branch
left 2

MemWrite
ALUSrc
Read

MemtoReg
Instruction
PC Address register 1 Read
Read data 1
register 2 Zero
Instruction

Control
Registers Read ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
[15– 0] Sign ALU MemRead
extend control

Instruction
[20– 16]

Clock cycle 1
0 ALUOp
M
Instruction u
[15– 11] x

• Instruction Clock 1 1
RegDst

sequence:
IF: sub $11, $2, $3 ID: lw $10, 20($1) EX: before<1> MEM: before<2> WB: before<3>

IF/ID ID/EX EX/MEM MEM/WB

0
M

lw $10, 20($1)
11 00
u WB
x
1 lw 010 000 00
Control M WB

sub $11, $2, $3 0001

EX
0
00
0
M
0
0
0
0
WB 0

and $12, $4, $7 Add

Add

or $13, $6, $7
4 Add result

RegWrite
Shift Branch
left 2

MemWrite
add $14, $8, $9
ALUSrc
1 Read

MemtoReg
Instruction

register 1
PC Address Read $1
X data 1
Read
register 2 Zero
Instruction
Registers Read $X ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
x

Label “before<i>” means

data 1
0
Write
data

i th instruction before
Instruction
20 [15– 0] Sign 20 ALU MemRead
extend control

Instruction

lw 10 [20– 16] 10
0 ALUOp

Clock cycle 2
M
Instruction u
X [15– 11] X x
1
Clock 2 RegDst
 IF: and $12, $4, $5 ID: sub $11, $2, $3 EX: lw $10, . . . MEM: before<1> WB: before<2>

Pipelined
IF/ID ID/EX EX/MEM MEM/WB
0
M 10 11
u WB
x
1 sub 000 010 00
Control M WB
0 0 0

Execution
1100 00 0
EX M WB 0
1 0

Add

Add
4 Add result

RegWrite
and
Shift Branch
left 2

MemWrite
ALUSrc
2 Read

MemtoReg
Instruction
PC Address register 1 Read $2 $1
3 Read data 1
register 2 Zero
Instruction

Control
Registers Read $3 ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
X [15– 0] Sign X 20 ALU MemRead
extend control

Instruction
X [20– 16] X 10
0

Clock cycle 3
ALUOp
M
Instruction u
11 [15– 11] 11 x

• Instruction Clock 3
1
RegDst

sequence: IF: or $13, $6, $7 ID: and $12, $2, $3 EX: sub $11, . . . MEM: lw $10, . . . WB: before<1>

IF/ID ID/EX EX/MEM MEM/WB

0
M 10 10
u WB

lw $10, 20($1)
x
1 and 000 000 11
Control M WB
1 0 0

sub $11, $2, $3 1100

EX
10
0
M
1
0
WB 0

and $12, $4, $7 4

Add

Add
Add result

or $13, $6, $7
RegWrite
Shift Branch
left 2

MemWrite
ALUSrc

add $14, $8, $9

4 Read

MemtoReg
Instruction

register 1
PC Address Read $4 $2
5 data 1
Read
register 2 Zero
Instruction
Registers Read $5 $3 ALU ALU
memory Write 0 Address Read
data 2 result 1
register M data
u Data M
Write x u
memory x
data 1
0
Write
data

Instruction
X [15– 0] Sign X ALU MemRead
extend control

Instruction
X [20– 16] X
0 ALUOp

Clock cycle 4
M 10
Instruction u
12 [15– 11] 12 11 x
1
Clock 4 RegDst
 IF: add $14, $8, $9 ID: or $13, $6, $7 EX: and $12, . . . MEM: sub $11, . . . WB: lw $10, . . .

Pipelined
IF/ID ID/EX EX/MEM MEM/WB
0
M 10 10
u WB
x
1 or 000 000 10
Control M WB
1 0 1

Execution
1100 10 0
EX M WB 1
0 0

Add

Add
4 Add result

RegWrite
and
Shift Branch
left 2

MemWrite
ALUSrc
6 Read

MemtoReg
Instruction
PC Address register 1 Read $6 $4
7 Read data 1
register 2 Zero
Instruction

Control
Registers Read $7 $5 ALU ALU
memory 10 Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
X [15– 0] Sign X ALU MemRead
extend control

Instruction
X [20– 16] X

Clock cycle 5
0 ALUOp
M 11 10
Instruction u
13 [15– 11] 13 12 x
Clock 5 1

• Instruction RegDst

sequence: IF: after<1> ID: add $14, $8, $9 EX: or $13, . . . MEM: and $12, . . . WB: sub $11, . . .

IF/ID ID/EX EX/MEM MEM/WB

0
M 10 10
u WB

lw $10, 20($1)
x
1 add 000 000 10
Control M WB
1 0 1

sub $11, $2, $3 1100

EX
10
0
M
0
0
WB 0

and $12, $4, $7 4

Add

Add
Add result

RegWrite
or $13, $6, $7 Shift
left 2
Branch

MemWrite
ALUSrc
8

add $14, $8, $9

Read

MemtoReg
Instruction

register 1
PC Address Read $8 $6
9 data 1
Read
register 2 Zero
Instruction
Registers Read $9 $7 ALU ALU
memory 11 Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
x

Label “after<i>” means

data 1
0
Write
data

i th instruction after add

Instruction
X [15– 0] Sign X ALU MemRead
extend control

Instruction
X [20– 16] X

Clock cycle 6
0 ALUOp
M 12 11
Instruction u
14 [15– 11] 14 13 x
1
Clock 6 RegDst
 IF: after<2> ID: after<1> EX: add $14, . . . MEM: or $13, . . . WB: and $12, . . .

Pipelined
IF/ID ID/EX EX/MEM MEM/WB
0
M 00 10
u WB
x
1 000 000 10
Control M WB
1 0 1

Execution
0000 10 0
EX M WB 0
0 0

Add

Add
4 Add result

RegWrite
and
Shift Branch
left 2

MemWrite
ALUSrc
Read

MemtoReg
Instruction
PC Address register 1 Read $8
Read data 1
register 2 Zero
Instruction

Control
Registers Read $9 ALU ALU
memory 12 Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
[15– 0] Sign ALU MemRead
extend control

Instruction
[20– 16]

Clock cycle 7
0 ALUOp
M 13 12
Instruction u
[15– 11] 14 x
1
Clock 7 RegDst

• Instruction
IF: after<3> ID: after<2> EX: after<1> MEM: add $14, . . . WB: or $13, . . .
sequence:
IF/ID ID/EX EX/MEM MEM/WB
0
M 00 00
u WB
x
1 000 000 10
Control M WB

lw $10, 20($1) 0000

EX
0
00
0
M
0
0
0
1
WB 0

sub $11, $2, $3 Add

Add

and $12, $4, $7 4 Add result

RegWrite
Shift Branch
left 2

MemWrite
or $13, $6, $7
ALUSrc
Read

MemtoReg
Instruction

PC Address register 1
Read
data 1

add $14, $8, $9

Read
register 2 Zero
Instruction
Registers Read ALU ALU
memory 13 Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
[15– 0] Sign ALU MemRead
extend control

Instruction
[20– 16]

Clock cycle 8
0 ALUOp
M 14 13
Instruction u
[15– 11] x
1
Clock 8 RegDst
 Pipelined Execution and Control
• Instruction IF: after<4> ID: after<3> EX: after<2> MEM: after<1> WB: add $14, . . .

sequence:
IF/ID ID/EX EX/MEM MEM/WB
0
M 00 00
u WB
x
1 000 000 00

lw $10, 20($1)
Control M WB
0 0 1
0000 00 0
EX M WB 0

sub $11, $2, $3 0 0

Add
and $12, $4, $7 4
Add
Add result

or $13, $6, $7

RegWrite
Shift Branch
left 2

MemWrite
ALUSrc
add $14, $8, $9 Read

MemtoReg
Instruction

PC Address register 1 Read

Read data 1
register 2 Zero
Instruction
Registers Read ALU ALU
memory 14 Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction
[15– 0] Sign ALU MemRead
extend control

Instruction
[20– 16]
0 ALUOp
M 14

Clock cycle 9 Instruction u

[15– 11] x
1
Clock 9 RegDst
 Revisiting Hazards
• So far our datapath and control have ignored hazards
• We shall revisit data hazards and control hazards and
enhance our datapath and control to handle them in
hardware…

sub $2, $1, $3

and $12, $2, $5 $2 = 10 before sub;
or $13, $6, $2 $2 = -20 after sub
add $14, $2, $2
sw $15, 100($2)
 Data Hazards and Forwarding

• Problem with starting an instruction before previous are finished:

– data dependencies that go backward in time – called data hazards
Time (in clock cycles)
$2 = 10 before sub; Value of CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
$2 = -20 after sub register $2: 10 10 10 10 10/– 20 – 20 – 20 – 20 – 20
Program
execution
order
(in instructions)
sub $2, $1, $3 IM Reg DM Reg

sub $2, $1, $3

and $12, $2, $5 and $12, $2, $5 IM Reg DM Reg

or $13, $6, $2
add $14, $2, $2 or $13, $6, $2 IM Reg DM Reg
sw $15, 100($2)
add $14, $2, $2 IM Reg DM Reg

sw $15, 100($2) IM Reg DM Reg

 Software Solution
• Have compiler guarantee never any data hazards!
– by rearranging instructions to insert independent instructions between
instructions that would otherwise have a data hazard between them,
– or, if such rearrangement is not possible, insert nops
sub $2, $1, $3 sub $2, $1, $3 sub $2, $1, $3
and $12, $2, $5 lw $10, 40($3) nop
or $13, $6, $2 slt $5, $6, $7 nop
add $14, $2, $2 and $12, $2, $5 or and $12, $2, $5
sw $15, 100($2) or $13, $6, $2 or $13, $6, $2
add $14, $2, $2 add $14, $2, $2
sw $15, 100($2) sw $15, 100($2)
• Such compiler solutions may not always be possible, and nops
slow the machine down

MIPS: nop = “no operation” = 00…0 (32bits) = sll $0, $0, 0

 Hardware Solution: Forwarding
• Idea: use intermediate data, do not wait for result to be finally written
to the destination register.
• Two steps:
1. Detect data hazard
2. Forward intermediate data to resolve hazard
 Pipelined Datapath with Control II sub $2, $1, $3
(as before) and
or
$12,
$13,
$2, $5
$6, $2
PCSrc add $14, $2, $2
sw $15, 100($2)
ID/EX
0
M
u WB
x EX/MEM
1
Control M WB
MEM/WB

EX M WB
IF/ID

Add

Add
4 Add result

RegWrite
Branch
Shift
left 2

MemWrite
ALUSrc
Read

MemtoReg
Instruction

PC Address register 1
Read
data 1
Read
register 2 Zero
Instruction
Registers Read ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction 16 32 6
[15– 0]
Control signals
Sign ALU MemRead
extend control

emanate from Instruction

[20– 16]
0 ALUOp
the control Instruction
M
u

portions of the [15– 11] x

1

pipeline registers
RegDst
 Hazard Detection
• Hazard conditions: sub $2, $1, $3
1a. EX/MEM.RegisterRd = ID/EX.RegisterRs and $12, $2, $5
or $13, $6, $2
1b. EX/MEM.RegisterRd = ID/EX.RegisterRt
add $14, $2, $2
2a. MEM/WB.RegisterRd = ID/EX.RegisterRs sw $15, 100($2)
2b. MEM/WB.RegisterRd = ID/EX.RegisterRt
– Eg., in the example, first hazard between
• sub $2, $1, $3 and
• and $12, $2, $5 is detected
– When the and is in EX stage and the sub is in MEM stage because
• EX/MEM.RegisterRd = ID/EX.RegisterRs = $2 (1a)
 Hazard Detection
Time (in clock cycles)

Value of CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
register $2: 10 10 10 10 10/– 20 – 20 – 20 – 20 – 20
Program
execution
order
(in instructions) sub $2, $1, $3
sub $2, $1, $3 IM Reg DM Reg
and $12, $2, $5
or $13, $6, $2
add $14, $2, $2
and $12, $2, $5 IM Reg DM Reg sw $15, 100($2)

When the sub is in WB

or $13, $6, $2 IM Reg DM Reg stage and the or is in EX
stage
MEM/WB.RegisterRd =
ID/EX.RegisterRt = $2 (2b)
add $14, $2, $2 IM Reg DM Reg

sw $15, 100($2) IM Reg DM Reg

 Hazard Detection
• Whether to forward also depends on:
– if the later instruction is going to write a register
• if not, no need to forward
– if the destination register of the later instruction is $0
• no need to forward value ($0 is always 0 and never overwritten)
 Data Forwarding
• Plan:
– allow inputs to the ALU not just from ID/EX, but also later
pipeline registers, and
– use multiplexors and control signals to choose appropriate
inputs to ALU
Time (in clock cycles)
CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
Value of register $2 : 10 10 10 10 10/– 20 – 20 – 20 – 20 – 20
Value of EX/MEM : X X X – 20 X X X X X
Value of MEM/WB : X X X X – 20 X X X X

Program
execution order
(in instructions)
sub $2, $1, $3 IM Reg DM Reg

sub $2, $1, $3

and $12, $2, $5 and $12, $2, $5 IM Reg DM Reg
or $13, $6, $2
add $14, $2, $2
or $13, $6, $2 IM Reg DM Reg
sw $15, 100($2)

add $14, $2, $2 IM Reg DM Reg

sw $15, 100($2) IM Reg DM Reg

Dependencies between pipelines move forward in time

 ID/EX EX/MEM MEM/WB

Forwarding
Hardware Registers ALU

Data
memory M
u
x

a. No forwarding Datapath before adding forwarding hardware

ID/EX EX/MEM MEM/WB

M
u
x
Registers
ForwardA ALU

M Data
u memory
x M
u
x

Rs ForwardB
Rt
Rt M
u EX/MEM.RegisterRd
Rd
x
Forwarding MEM/WB.RegisterRd
unit

b. With forwarding
Datapath after adding forwarding hardware
 Forwarding Hardware:
Multiplexor Control
Mux control Source Explanation
ForwardA = 00 ID/EX The first ALU operand comes from the register file
ForwardA = 10 EX/MEM The first ALU operand is forwarded from prior ALU result
ForwardA = 01 MEM/WB The first ALU operand is forwarded from data memory
or an earlier ALU result
ForwardB = 00 ID/EX The second ALU operand comes from the register file
ForwardB = 10 EX/MEM The second ALU operand is forwarded from prior ALU result
ForwardB = 01 MEM/WB The second ALU operand is forwarded from data memory
or an earlier ALU result

Depending on the selection in the rightmost multiplexor

(see datapath with control diagram)
Data Hazard: Detection and Forwarding

• Forwarding unit determines multiplexor control according to the

following rules:

1. EX hazard
if ( EX/MEM.RegWrite // if there is a write…
and ( EX/MEM.RegisterRd  0 ) // to a non-$0 register…
and ( EX/MEM.RegisterRd = ID/EX.RegisterRs ) ) // which matches, then…
ForwardA = 10

if ( EX/MEM.RegWrite // if there is a write…

and ( EX/MEM.RegisterRd  0 ) // to a non-$0 register…
and ( EX/MEM.RegisterRd = ID/EX.RegisterRt ) ) // which matches, then…
ForwardB = 10
 Data Hazard: Detection and Forwarding
2. MEM hazard
if ( MEM/WB.RegWrite // if there is a write…
and ( MEM/WB.RegisterRd  0 ) // to a non-$0 register…
and ( EX/MEM.RegisterRd  ID/EX.RegisterRs ) // and not already a register match
// with earlier pipeline register…
and ( MEM/WB.RegisterRd = ID/EX.RegisterRs ) ) // but match with later pipeline
register, then…
ForwardA = 01

if ( MEM/WB.RegWrite // if there is a write…

and ( MEM/WB.RegisterRd  0 ) // to a non-$0 register…
and ( EX/MEM.RegisterRd  ID/EX.RegisterRt ) // and not already a register match
// with earlier pipeline register…
and ( MEM/WB.RegisterRd = ID/EX.RegisterRt ) ) // but match with later pipeline
register, then…
ForwardB = 01

This check is necessary, e.g., for sequences such as add $1, $1, $2; add $1, $1, $3; add $1, $1, $4;
(array summing…), where an earlier pipeline (EX/MEM) register has more recent data
 Forwarding Hardware with Control
Called forwarding unit, not hazard detection unit,
because once data is forwarded there is no hazard!
ID/EX

WB
EX/MEM

Control M WB
MEM/WB

IF/ID EX M WB

M
Instruction

u
x
Registers
Instruction Data
PC ALU
memory memory M
u
M x
u
x

IF/ID.RegisterRs Rs
IF/ID.RegisterRt Rt
IF/ID.RegisterRt Rt
M EX/MEM.RegisterRd
IF/ID.RegisterRd Rd u
x
Forwarding MEM/WB.RegisterRd
unit

Datapath with forwarding hardware and control wires – certain details,

e.g., branching hardware, are omitted to simplify the drawing
Note: so far we have only handled forwarding to R-type instructions…!
 or $4, $4, $2 and $4, $2, $5 sub $2, $1, $3 before<1> before<2>

ID/EX
10 10

Forwarding
WB
EX/MEM

Control M WB
MEM/WB

IF/ID EX M WB

2 $2 $1
M

Instruction
5 u
x
Registers
Instruction Data
PC ALU
memory memory M
$5 $3
u
M x
u
x

2 1
5 3
M
4 2 u
x
Forwarding

Clock cycle 3 unit

• Execution Clock 3
example:
add $9, $4, $2 or $4, $4, $2 and $4, $2, $5 sub $2, . . . before<1>

ID/EX
10 10
WB

sub $2, $1, $3 Control M

EX/MEM

WB
10
MEM/WB
and $4, $2, $5 IF/ID EX M WB

or $4, $4, $2 4 $4 $2

add $9, $4, $2 M

Instruction

6 u
x
Registers
Instruction Data
PC ALU
memory memory M
$2 $5
u
M x
u
x

2 2
6 5
M 2
4 4 u
x
Forwarding

Clock cycle 4 unit

Clock 4
 after<1> add $9, $4, $2 or $4, $4, $2 and $4, . . . sub $2, . . .

ID/EX
10 10

Forwarding
WB
EX/MEM
10
Control M WB
MEM/WB
1
IF/ID EX M WB

4 $4 $4
M

Instruction
2 u
x
Registers
Instruction 2 Data
PC ALU
memory memory M
$2 $2
u
M x
u
x

4 4
2 2
M 4 2
9 4 u
• Execution x

Forwarding

example Clock cycle 5 unit

(cont.): Clock 5

after<2> after<1> add $9, $4, $2 or $4, . . . and $4, . . .

ID/EX
10
WB
sub $2, $1, $3 Control M
EX/MEM

WB
10
MEM/WB
and $4, $2, $5 IF/ID EX M WB
1

or $4, $4, $2 $4

add $9, $4, $2 M

Instruction

u
x
Registers
Instruction 4 Data
PC ALU
memory memory M
$2
u
M x
u
x

4
2

M 4 4
9 u
x
Forwarding

Clock cycle 6 unit

Clock 6
 Data Hazards and Stalls
• Load word can still cause a hazard:
– an instruction tries to read a register following a load instruction that writes to the same register

Time (in clock cycles)

lw $2, 20($1) Program CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
and $4, $2, $5 execution
order
or $8, $2, $6 (in instructions)
lw $2, 20($1) IM Reg DM Reg
add $9, $4, $2
Slt $1, $6,we need
– therefore, $7a hazard detection unit to stall the pipeline after the load instruction
and $4, $2, $5 IM Reg DM Reg

As even a pipeline
or $8, $2, $6 IM Reg DM Reg
dependency goes
backward in time
forwarding will not add $9, $4, $2 IM Reg DM Reg

solve the hazard

slt $1, $6, $7 IM Reg DM Reg
Pipelined Datapath with Control II (as before)
PCSrc

Hazard ID/EX
0
M
u Detection WB
EX/MEM
x
1
Unit Control M WB
MEM/WB

EX M WB
IF/ID

Add

Add
4 Add result

RegWrite
Branch
Shift
left 2

MemWrite
ALUSrc
Read

MemtoReg
Instruction

PC Address register 1
Read
data 1
Read
register 2 Zero
Instruction
Registers Read ALU ALU
memory Write 0 Read
data 2 result Address 1
register M data
u Data M
Write x memory u
data x
1
0
Write
data

Instruction 16 32 6
[15– 0]
Control signals
Sign ALU MemRead
extend control

emanate from Instruction

[20– 16]
0 ALUOp
the control Instruction
M
u

portions of the [15– 11] x

1

pipeline registers
RegDst
 Hazard Detection Logic to Stall
• Hazard detection unit implements the following check if to stall

if ( ID/EX.MemRead // if the instruction in the EX stage is a load…

and ( ( ID/EX.RegisterRt = IF/ID.RegisterRs ) // and the destination register
or ( ID/EX.RegisterRt = IF/ID.RegisterRt ) ) ) // matches either source register
// of the instruction in the ID stage, then…stall the pipeline
 Mechanics of Stalling
• If the check to stall verifies, then the pipeline needs to stall only 1
clock cycle after the load as after that the forwarding unit can
resolve the dependency
• What the hardware does to stall the pipeline 1 cycle:
– does not let the IF/ID register change (disable write!) – this will cause
the instruction in the ID stage to repeat, i.e., stall
– therefore, the instruction, just behind, in the IF stage must be stalled
as well – so hardware does not let the PC change (disable write!) –
this will cause the instruction in the IF stage to repeat, i.e., stall
– changes all the EX, MEM and WB control fields in the ID/EX pipeline
register to 0, so effectively the instruction just behind the load
becomes a nop – a bubble is said to have been inserted into the
pipeline
• note that we cannot turn that instruction into an nop by 0ing all the bits in
the instruction itself – recall nop = 00…0 (32 bits) – because it has
already been decoded and control signals generated
 Hazard Detection Unit
Hazard ID/EX.MemRead
detection
unit ID/EX

WB
IF/IDWrite
EX/MEM
M
Control u M WB
x MEM/WB
0
IF/ID EX M WB
PCWrite

M
Instruction

u
x
Registers
Instruction Data
PC ALU
memory memory M
u
M x
u
x

IF/ID.RegisterRs
IF/ID.RegisterRt
IF/ID.RegisterRt Rt M EX/MEM.RegisterRd
IF/ID.RegisterRd Rd u
x
ID/EX.RegisterRt Rs Forwarding MEM/WB.RegisterRd
Rt unit

Datapath with forwarding hardware, the hazard detection unit and

controls wires – certain details, e.g., branching hardware are omitted
to simplify the drawing
 Stalling Resolves a Hazard
• Same instruction sequence as before for which forwarding by
itself could
Program
notTime
resolve the hazard:
(in clock cycles)
execution CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9 CC 10
order
(in instructions)

lw $2, 20($1) lw $2, 20($1) IM Reg DM Reg

and $4, $2, $5

or $8, $2, $6
and $4, $2, $5 IM Reg Reg DM Reg
add $9, $4, $2
Slt $1, $6, $7
or $8, $2, $6 IM IM Reg DM Reg

bubble

add $9, $4, $2 IM Reg DM Reg

slt $1, $6, $7 IM Reg DM Reg

Hazard detection unit inserts a 1-cycle bubble in the pipeline, after

which all pipeline register dependencies go forward so then the
forwarding unit can handle them and there are no more hazards
 and $4, $2, $5 lw $2, 20($1) before<1> before<2> before<3>
Hazard
ID/EX.MemRead
detection
1

Stalling
unit ID/EX
X
11
WB

IF/IDWrite
EX/MEM
M
Control u M WB
x MEM/WB
0
IF/ID EX M WB

1 $1

PCWrite
M

Instruction
X u
x
Registers
Instruction Data
PC ALU
memory memory M
$X
u
M x
u
x

• Execution 1
X

example: 2
M
u
x
ID/EX.RegisterRt Forwarding
unit

Clock cycle 2
Clock 2

lw $2, 20($1) or $4, $4, $2 and $4, $2, $5 lw $2, 20($1) before<1> before<2>
Hazard
and $4, $2, $5 2
5
detection
unit
ID/EX.MemRead
ID/EX

or $4, $4, $2 00
WB
11
IF/IDWrite

EX/MEM

add $9, $4, $2

M
Control u M WB
x MEM/WB
0
IF/ID EX M WB

2 $2 $1
PCWrite

M
Instruction

5 u
x
Registers
Instruction Data
PC ALU
memory memory M
$5 $X
u
M x
u
x

2 1
5 X
2 M
4 u
x
ID/EX.RegisterRt Forwarding
unit

Clock cycle 3
Clock 3
 or $4, $4, $2 and $4, $2, $5 bubble lw $2, . . . before<1>
Hazard
ID/EX.MemRead
detection
2 unit ID/EX
5
10 00
WB

IF/IDWrite
EX/MEM

Stalling
M 11
Control u M WB
x MEM/WB
0
IF/ID EX M WB

2 $2 $2

PCWrite
M

Instruction
5 u
x
Registers
Instruction Data
PC ALU
memory memory M
$5 $5
u
M x
u
x

• Execution
2
5
2
5

M 2
example 4 4 u
x
ID/EX.RegisterRt Forwarding
(cont.): unit

Clock cycle 4
Clock 4

add $9, $4, $2 or $4, $4, $2 and $4, $2, $5 bubble lw $2, . . .
Hazard
ID/EX.MemRead
detection
4
lw $2, 20($1) 2
unit

10
ID/EX

WB
10
IF/IDWrite

and $4, $2, $5

EX/MEM
M 0
Control u M WB
MEM/WB
or $4, $4, $2
x
0
11
IF/ID EX M WB

add $9, $4, $2 4 $4 $2

PCWrite

M
Instruction

2 u
x
Registers
Instruction 2 Data
PC ALU
memory memory M
$2 $5
u
M x
u
x

4 2
2 5
M 2
4 4 u
x
ID/EX.RegisterRt Forwarding
unit

Clock cycle 5
Clock 5
 after<1> add $9, $4, $2 or $4, $4, $2 and $4, . . . bubble
Hazard ID/EX.MemRead
detection
4
unit ID/EX

Stalling
2
10 10
WB

IF/IDWrite
EX/MEM
M 10
Control u M WB
x MEM/WB
0
0
IF/ID EX M WB

4 $4 $4

PCWrite
M

Instruction
2 u
x
Registers
Instruction Data
PC ALU
memory memory M
$2 $2
u
M x
u
x

4 4

• Execution 2 2

M 4

example
9 4 u
x
ID/EX.RegisterRt Forwarding
unit
(cont.): Clock cycle 6
Clock 6

after<2> after<1> add $9, $4, $2 or $4, . . . and $4, . . .

Hazard
detection ID/EX.MemRead

lw $2, 20($1) unit

10
ID/EX
10
WB
and $4, $2, $5
IF/IDWrite

EX/MEM
M 10
Control u M WB
or $4, $4, $2 0
x

EX M
MEM/WB

WB
1
IF/ID

add $9, $4, $2 $4

PCWrite

M
Instruction

u
x
Registers
Instruction 4 Data
PC ALU
memory memory M
$2
u
M x
u
x

4
2

M 4 4
9 u
x
ID/EX.RegisterRt Forwarding
unit

Clock cycle 7
Clock 7
 Pipelining Exercise
Consider the following MIPS assembly code:
add $3, $2, $3
lw $4, 100($3)
sub $7, $6, $2
xor $6, $4, $3

Assume there is no forwarding or stalling circuitry in a pipelined processor that uses the
standard 5-stages (IF, ID, EX, Mem, WB). Instead, we will require the compiler to add no-
ops to the code to ensure correct execution. (Assume that if the processor reads and
writes to the same register in a given cycle, the value read out will be the new value that
is written in.)

1.Rewrite the code to include the no-ops that are needed. Do not change the order of the
four statements. Use as few no-ops as possible.

2.Suppose the complier is allowed to change the order of the four statements, provided it
doesn’t change the final answer. Is it possible to reduce the number of no-ops needed?
Why or why not?
 Another pipelining exercise
Consider (again) the following MIPS assembly code:
add $3, $2, $3
lw $4, 100($3)
sub $7, $6, $2
xor $6, $4, $3
Assume that there is forwarding and stalling hardware. And the text (the forwarding path is
shown in the diagram on the next page).
Draw an execution diagram that shows where forwarding and stalling would take place. Use
arrows to show forwarding and “bubbles” to show stalls, as in the following
(hypothetical) example.
Cycle 1 2 3 4 5 6 7 This arrow says the
value stored in flip-
IF ID Ex Mem WB flops between Mem and
lw
WB is forwarded to an
ALU input in the Ex
stage for the “or”.
IM Bubble ID Ex Mem WB
or

and IF ID Ex Mem WB

This bubble says the ID stage of the lw

Here, we forward from the Ex/Mem
doesn’t complete until a cycle later.
flip-flops to an ALU input.
 The pipelined implementation with
forwarding
(used in the question on the previous page)
Control (or Branch) Hazards
• Problem with branches in the pipeline we have so far is that the
branch decision is not made till the MEM stage – so what
instructions, if at all, should we insert into the pipeline following the
branch instructions?

• Possible solution: stall the pipeline till branch decision is known

– not efficient, slow the pipeline significantly!

• Another solution: predict the branch outcome

– e.g., always predict branch-not-taken – continue with next sequential
instructions
– if the prediction is wrong have to flush the pipeline behind the branch –
discard instructions already fetched or decoded – and continue
execution at the branch target
 Predicting Branch-not-taken:
Misprediction delay
Program Time (in clock cycles)
execution CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
order
(in instructions)

40 beq $1, $3, 7 IM Reg DM Reg

44 and $12, $2, $5 IM Reg DM Reg

48 or $13, $6, $2 IM Reg DM Reg

52 add $14, $2, $2 IM Reg DM Reg

72 lw $4, 50($7) IM Reg DM Reg

The outcome of branch taken (prediction wrong) is decided only when

beq is in the MEM stage, so the following three sequential instructions
already in the pipeline have to be flushed and execution resumes at lw
 Optimizing the Pipeline to
Reduce Branch Delay
• Move the branch decision from the MEM stage (as in our current
pipeline) earlier to the ID stage
– calculating the branch target address involves moving the branch adder
from the MEM stage to the ID stage – inputs to this adder, the PC value
and the immediate fields are already available in the IF/ID pipeline
register
– calculating the branch decision is efficiently done, e.g., for equality test,
by XORing respective bits and then ORing all the results and inverting,
rather than using the ALU to subtract and then test for zero (when there
is a carry delay)
• with the more efficient equality test we can put it in the ID stage without
significantly lengthening this stage – remember an objective of pipeline design
is to keep pipeline stages balanced
– we must correspondingly make additions to the forwarding and hazard
detection units to forward to or stall the branch at the ID stage in case the
branch decision depends on an earlier result
 Flushing on Misprediction
• Same strategy as for stalling on load-use data hazard…
• Zero out all the control values (or the instruction itself) in pipeline registers
for the instructions following the branch that are already in the pipeline –
effectively turning them into nops – so they are flushed
– in the optimized pipeline, with branch decision made in the ID stage, we have
to flush only one instruction in the IF stage – the branch delay penalty is then
only one clock cycle
 Optimized Datapath for Branch
IF.Flush

Hazard
detection IF.Flush control zeros out the instruction in the IF/ID
unit
M ID/EX
pipeline register (which follows the branch)
u
x
WB
EX/MEM
M
Control u M WB
x MEM/WB
0

IF/ID EX M WB

4 Shift
left 2
M
u
x
Registers =
Instruction Data
PC ALU
memory memory M
u
M x
u
x

Sign
extend

M
u
x
Forwarding
unit

Branch decision is moved from the MEM stage to the ID stage – simplified drawing
not showing enhancements to the forwarding and hazard detection units
 and $12, $2, $5 beq $1, $3, 7 sub $10, $4, $8 before<1> before<2>

Pipelined
IF.Flush

Hazard
detection
unit
72 ID/EX
M
u
48 x WB
EX/MEM
M

Branch
Control u M WB
x MEM/WB
28
0
IF/ID EX M WB
48 44 72

4
$1
Shift M $4
left 2 u
x
=
Registers
Instruction Data
PC ALU
memory memory M
72 44 $3
u
M $8 x
7 u
x

• Execution Sign
extend

example:
10

Forwarding

Clock cycle 3
unit

Clock 3
36 sub $10, $4, $8
lw $4, 50($7) bubble (nop) beq $1, $3, 7 sub $10, . . .
40 beq $1, $3, 7 before<1>

IF.Flush

44 and $12 $2, $5 Hazard

detection

48 or $13 $2, $6
unit
ID/EX
M
u
76 x WB

52 add $14, $4, $2

EX/MEM
M
Control u M WB
x MEM/WB

56 slt $15, $6, $7

0
IF/ID EX M WB
76 72

… 4

Shift
left 2
M
u
$1

72 lw $4, 50($7)
x
Registers
=
Instruction Data
PC ALU
memory memory M
76 72
u
M $3 x
u

Optimized pipeline with

x

Sign

only one bubble as a result

extend

of the taken branch 10

Forwarding
unit

Clock cycle
Clock 4
4
 Simple Example: Comparing
Performance
• Compare performance for single-cycle, multicycle, and pipelined
datapaths using the gcc instruction mix
– assume 2 ns for memory access, 2 ns for ALU operation, 1 ns for
register read or write
– assume gcc instruction mix 23% loads, 13% stores, 19% branches,
2% jumps, 43% ALU
– for pipelined execution assume
• 50% of the loads are followed immediately by an instruction that uses
the result of the load
• 25% of branches are mispredicted
• branch delay on misprediction is 1 clock cycle
• jumps always incur 1 clock cycle delay so their average time is 2 clock
cycles
Simple Example: Comparing Performance
• Single-cycle (p. 373): average instruction time 8 ns
• Multicycle (p. 397): average instruction time 8.04 ns
• Pipelined:
– loads use 1 cc (clock cycle) when no load-use dependency and 2 cc when
there is dependency – given 50% of loads are followed by dependency the
average cc per load is 1.5
– stores use 1 cc each
– branches use 1 cc when predicted correctly and 2 cc when not – given 25%
misprediction average cc per branch is 1.25
– jumps use 2 cc each
– ALU instructions use 1 cc each
– therefore, average CPI is
1.5  23% + 1  13% + 1.25  19% + 2  2% + 1  43% = 1.18
– therefore, average instruction time is 1.18  2 = 2.36 ns
• 50% of the loads are followed immediately by an instruction that uses the result of
the load
• 25% of branches are mispredicted
• branch delay on misprediction is 1 clock cycle
• jumps always incur 1 clock cycle delay so their average time is 2 clock cycles
 Pipelining Advantages
• Higher maximum throughput
• Higher utilization of CPU resources

• But, more hardware needed, perhaps complex control

 Tutorial Question

Draw an execution diagram that shows where forwarding and

stalling would take place, if any.
add $6,$5,$2
lw $7,0($6)
addi $7,$7,10
add $6,$4,$2
sw $7,0($6)
addi $2,$2,4
blt $2,$3,loop
add $6,$5,$2
 Pipeline Enhancements
• Assume we add branch predictor
– Branch predictor success rate = 85%
– Penalty for bad prediction = 3 cycles
– Profiler tells us that 10% of instructions executed are branches

– Branch speedup
= (cycles before enhancement) / (cycles after enhancement)
= 3 / [.15(3) + .85(1)] = 2.3
1
– Amdahl’s Law: Speedup 
1  Fractionenhanced   Fractionenhanced
Speedupenhanced

– Speedup = 1 / (.90 + .10/2.3) = 1.06

– 6% improvement