Pipelining Outline

• Introduction
– Defining Pipelining
– Pipelining Instructions
• Hazards
– Structural hazards \
– Data Hazards
– Control Hazards
• Performance
• Controller implementation

CSCE430/830 Pipeline Hazards

 Pipeline Hazards

• Where one instruction cannot immediately

follow another
• Types of hazards
– Structural hazards - attempt to use the same resource by
two or more instructions
– Control hazards - attempt to make branching decisions
before branch condition is evaluated
– Data hazards - attempt to use data before it is ready
• Can always resolve hazards by waiting

CSCE430/830 Pipeline Hazards

 Structural Hazards

• Attempt to use the same resource by two or

more instructions at the same time
• Example: Single Memory for instructions and
data
– Accessed by IF stage
– Accessed at same time by MEM stage
• Solutions
– Delay the second access by one clock cycle, OR
– Provide separate memories for instructions & data
» This is what the book does
» This is called a “Harvard Architecture”
» Real pipelined processors have separate caches

CSCE430/830 Pipeline Hazards

 Pipelined Example -
Executing Multiple Instructions

• Consider the following instruction sequence:

lw $r0, 10($r1)
sw $sr3, 20($r4)
add $r5, $r6, $r7
sub $r8, $r9, $r10

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 1

LW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 2

SW LW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 3

ADD SW LW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 4

SUB ADD SW LW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 5

SUB ADD SW LW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 6

SUB ADD SW

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 7

SUB ADD

CSCE430/830 Pipeline Hazards

 Executing Multiple Instructions
Clock Cycle 8
SUB

CSCE430/830 Pipeline Hazards

 Alternative View - Multicycle Diagram

CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8

lw $r0, 10($r1) IM REG ALU DM REG

sw $r3, 20($r4) IM REG ALU DM REG

add $r5, $r6, $r7 IM REG ALU DM REG

sub $r8, $r9, $r10 IM REG ALU DM REG

CSCE430/830 Pipeline Hazards

 Alternative View - Multicycle Diagram

CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8

lw $r0, 10($r1) IM REG ALU DM REG

Memory Conflict

sw $r3, 20($r4) IM REG ALU DM REG

add $r5, $r6, $r7 IM REG ALU DM REG

sub $r8, $r9, $r10 IM REG ALU DM REG

CSCE430/830 Pipeline Hazards

 One Memory Port Structural Hazards

Time (clock cycles)

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7

I Load Ifetch

ALU
Reg DMem Reg

n
s

ALU
t
Instr 1 Ifetch Reg DMem Reg

r.

ALU
Reg
Instr 2 Ifetch Reg DMem

O
r
Stall Bubble Bubble Bubble Bubble Bubble
d
e
r

ALU
Instr 3 Ifetch Reg DMem Reg

CSCE430/830 Pipeline Hazards

 Structural Hazards

Some common Structural Hazards:

• Memory:
– we’ve already mentioned this one.
• Floating point:
– Since many floating point instructions require many cycles, it’s easy
for them to interfere with each other.
• Starting up more of one type of instruction than there are
resources.
– For instance, the PA-8600 can support two ALU + two load/store
instructions per cycle - that’s how much hardware it has available.

CSCE430/830 Pipeline Hazards

 Structural Hazards
Dealing with Structural Hazards
Stall
• low cost, simple
• Increases CPI
• use for rare case since stalling has performance effect
Pipeline hardware resource
• useful for multi-cycle resources
• good performance
• sometimes complex e.g., RAM
Replicate resource
• good performance
• increases cost (+ maybe interconnect delay)
• useful for cheap or divisible resources

CSCE430/830 Pipeline Hazards

 Structural Hazards

• Structural hazards are reduced with these rules:

– Each instruction uses a resource at most once
– Always use the resource in the same pipeline stage
– Use the resource for one cycle only
• Many RISC ISAs are designed with this in mind
• Sometimes very difficult to do this.
– For example, memory of necessity is used in the IF and MEM
stages.

CSCE430/830 Pipeline Hazards

 Pipelining Outline

• Introduction
– Defining Pipelining
– Pipelining Instructions
• Hazards
– Structural hazards
– Data Hazards \
– Control Hazards
• Performance
• Controller implementation

CSCE430/830 Pipeline Hazards

 Pipeline Hazards

• Where one instruction cannot immediately

follow another
• Types of hazards
– Structural hazards - attempt to use same resource twice
– Control hazards - attempt to make decision before
condition is evaluated
– Data hazards - attempt to use data before it is ready
• Can always resolve hazards by waiting

CSCE430/830 Pipeline Hazards

 Data Hazards
• Data hazards occur when data is used before
it is ready
Time (in clock cycles)

Value of CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
register $2: 10 10 10 10 10/– 20 – 20 – 20 – 20 – 20
Program
execution
order
(in instructions)
sub $2, $1, $3 IM Reg DM Reg

and $12, $2, $5 IM Reg DM Reg

or $13, $6, $2 IM Reg DM Reg

add $14, $2, $2 IM Reg DM Reg

sw $15, 100($2) IM Reg DM Reg

The use of the result of the SUB instruction in the next three instructions causes a
data hazard, since the register $2 is not written until after those instructions read it.

CSCE430/830 Pipeline Hazards

 Data Hazards
Execution Order is:
Read After Write (RAW)
InstrI
InstrJ InstrJ tries to read operand before InstrI writes it

I: add r1,r2,r3
J: sub r4,r1,r3

• Caused by a “Dependence” (in compiler nomenclature). This

hazard results from an actual need for communication.

CSCE430/830 Pipeline Hazards

 Data Hazards
Execution Order is:
Write After Read (WAR)
InstrI
InstrJ InstrJ tries to write operand before InstrI reads i
– Gets wrong operand

I: sub r4,r1,r3
J: add r1,r2,r3
K: mul r6,r1,r7
– Called an “anti-dependence” by compiler writers.
This results from reuse of the name “r1”.

• Can’t happen in MIPS 5 stage pipeline because:

– All instructions take 5 stages, and
– Reads are always in stage 2, and
– Writes are always in stage 5

CSCE430/830 Pipeline Hazards

 Data Hazards
Execution Order is:
Write After Write (WAW)
InstrI
InstrJ InstrJ tries to write operand before InstrI writes it
– Leaves wrong result ( InstrI not InstrJ )

I: sub r1,r4,r3
J: add r1,r2,r3
K: mul r6,r1,r7

• Called an “output dependence” by compiler writers

This also results from the reuse of name “r1”.

• Can’t happen in MIPS 5 stage pipeline because:

– All instructions take 5 stages, and
– Writes are always in stage 5

• Will see WAR and WAW later in more complicated pipes

CSCE430/830 Pipeline Hazards

 Data Hazard Detection in MIPS (1)
Read after Write
Time (in clock cycles)

Value of CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
register $2: 10 10 10 10 10/– 20 – 20 – 20 – 20 – 20
Program
execution IF/ID ID/EX EX/MEM MEM/WB
order
(in instructions)
sub $2, $1, $3 IM Reg DM Reg

and $12, $2, $5 IM Reg DM Reg

or $13, $6, $2 IM Reg DM Reg

add $14, $2, $2 IM Reg DM Reg

sw $15, 100($2) IM Reg DM Reg

1a: EX/MEM.RegisterRd = ID/EX.RegisterRs

EX hazard
1b: EX/MEM.RegisterRd = ID/EX.RegisterRt
2a: MEM/WB.RegisterRd = ID/EX.RegisterRs MEM hazard
2b: MEM/WB.RegisterRd = ID/EX.RegisterRt

CSCE430/830 Pipeline Hazards

 Data Hazards

• Solutions for Data Hazards

– Stalling
– Forwarding:
» connect new value directly to next stage
– Reordering

CSCE430/830 Pipeline Hazards

 Data Hazard - Stalling

0 2 4 6 8 10 12 16 18

add $s0,$t0,$t1 W
IF ID EX MEM s0 $s0
written
here

STALL
BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE

STALL
BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE

sub $t2,$s0,$t3 R
IF s0 EX MEM WB

$s0 read
here

CSCE430/830 Pipeline Hazards

 Data Hazards - Stalling

Simple Solution to RAW

• Hardware detects RAW and stalls

• Assumes register written then read each cycle
+ low cost to implement, simple
-- reduces IPC
• Try to minimize stalls

Minimizing RAW stalls

• Bypass/forward/shortcircuit (We will use the word “forward”)

• Use data before it is in the register
+ reduces/avoids stalls
-- complex
• Crucial for common RAW hazards

CSCE430/830 Pipeline Hazards

 Data Hazards - Forwarding
• Key idea: connect new value directly to next stage
• Still read s0, but ignore in favor of new result

• Problem: what about load instructions?

CSCE430/830 Pipeline Hazards
 Data Hazards - Forwarding
• STALL still required for load - data avail. after MEM
• MIPS architecture calls this delayed load, initial
implementations required compiler to deal with this

0 2 4 6 8 10 12 16 18

ID W
lw $s0,20($t1) IF ID EX MEM s0

new value
of s0

STALL
BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE

R
sub $t2,$s0,$t3 IF s0 EX MEM WB

CSCE430/830 Pipeline Hazards

 This is another
representation
Data Hazards of the stall.

LW R1, 0(R2) IF ID EX MEM WB

SUB R4, R1, R5 IF ID EX MEM WB

AND R6, R1, R7 IF ID EX MEM WB

OR R8, R1, R9 IF ID EX MEM WB

LW R1, 0(R2) IF ID EX MEM WB

SUB R4, R1, R5 IF ID stall EX MEM WB

AND R6, R1, R7 IF stall ID EX MEM WB

OR R8, R1, R9 stall IF ID EX MEM WB

CSCE430/830 Pipeline Hazards

 Control Hazards
A control hazard is when we need to find the
destination of a branch, and can’t fetch any new
instructions until we know that destination.

A branch is either
– Taken: PC <= PC + 4 + Immediate
– Not Taken: PC <= PC + 4

CSCE430/830 Pipeline Hazards

Control Hazards Control Hazard on Branches
Three Stage Stall

ALU
10: beq r1,r3,36 Ifetch Reg DMem Reg

ALU
Ifetch Reg DMem Reg
14: and r2,r3,r5

ALU
Reg
18: or r6,r1,r7 Ifetch Reg DMem

ALU
Ifetch Reg DMem Reg
22: add r8,r1,r9

ALU
36: xor r10,r1,r11 Ifetch Reg DMem Reg

The penalty when branch take is 3 cycles!

CSCE430/830 Pipeline Hazards
 Branch Hazards
• Just stalling for each branch is not practical
• Common assumption: branch not taken
• When assumption fails: flush three
instructions
Program Time (in clock cycles)
execution CC 1 CC 2 CC 3 CC 4 CC 5 CC 6 CC 7 CC 8 CC 9
order
(in instructions)

40 beq $1, $3, 7 IM Reg DM Reg

44 and $12, $2, $5 IM Reg DM Reg

48 or $13, $6, $2 IM Reg DM Reg

52 add $14, $2, $2 IM Reg DM Reg

72 lw $4, 50($7) IM Reg DM Reg

(Fig. 6.37)

CSCE430/830 Pipeline Hazards

 Static Branch Prediction
For every branch encountered during execution predict whether the
branch will be taken or not taken.

Predicting branch not taken:

1. Speculatively fetch and execute in-line instructions following the branch
2. If prediction incorrect flush pipeline of speculated instructions
• Convert these instructions to NOPs by clearing pipeline registers
• These have not updated memory or registers at time of flush

Predicting branch taken:

1. Speculatively fetch and execute instructions at the branch target address
2. Useful only if target address known earlier than branch outcome
• May require stall cycles till target address known
• Flush pipeline if prediction is incorrect
• Must ensure that flushed instructions do not update memory/registers

CSCE430/830 Pipeline Hazards

 Control Hazard - Stall

0 2 4 6 8 10 12 16 18

add $r4,$r5,$r6 IF ID EX MEM WB

beq $r0,$r1,tgt IF ID EX MEM WB

STALL
BUBBLE BUBBLE BUBBLE BUBBLE BUBBLE

sw $s4,200($t5) IF ID EX MEM WB
beq
writes PC new PC
here used here

CSCE430/830 Pipeline Hazards

 Control Hazard - Correct Prediction

0 2 4 6 8 10 12 16 18

add $r4,$r5,$r6 IF ID EX MEM WB

beq $r0,$r1,tgt IF ID EX MEM WB

tgt:
sw $s4,200($t5) IF ID EX MEM WB

Fetch assuming
branch taken

CSCE430/830 Pipeline Hazards

 Control Hazard - Incorrect Prediction

0 2 4 6 8 10 12 16 18

add $r4,$r5,$r6 IF ID EX MEM WB

beq $r0,$r1,tgt IF ID EX MEM WB

tgt:
sw $s4,200($t5) IF
(incorrect - ST ALL) BUBBLE BUBBLE BUBBLE BUBBLE

or $r8,$r8,$r9 IF ID EX MEM WB

“Squashed”
instruction

CSCE430/830 Pipeline Hazards

 1-Bit Branch Prediction
• Branch History Table (BHT): Lower bits of PC address index
table of 1-bit values
– Says whether or not the branch was taken last time
– No address check (saves HW, but may not be the right branch)
– If prediction is wrong, invert prediction bit

1 = branch was last taken

0 = branch was last not taken
1 prediction bit

0
a31a30…a11…a2a1a0 branch instruction

1K-entry BHT

10-bit index

Instruction memory

Hypothesis: branch will do the same again. Pipeline Hazards

CSCE430/830
 1-Bit Branch Prediction

• Example:
Consider a loop branch that is taken 9 times in a
row and then not taken once. What is the prediction
accuracy of the 1-bit predictor for this branch
assuming only this branch ever changes its
corresponding prediction bit?

– Answer: 80%. Because there are two mispredictions – one

on the first iteration and one on the last iteration. Is this
good enough and Why?

CSCE430/830 Pipeline Hazards

 2-Bit Branch Prediction
(Jim Smith, 1981)
• Solution: a 2-bit scheme where prediction is changed
only if mispredicted twice
Red: stop, not taken
Green: go, taken
T
NT
Predict Taken 11 10 Predict Taken
T
T NT
NT
Predict Not 01 00 Predict Not
T Taken
Taken

NT
CSCE430/830 Pipeline Hazards
 n-bit Saturating Counter

• Values: 0 ~ 2n-1
• When the counter is greater than or equal to one-half
of its maximum value, the branch is predicted as
taken. Otherwise, not taken.
• Studies have shown that the 2-bit predictors do
almost as well, and thus most systems rely on 2-bit
branch predictors.

CSCE430/830 Pipeline Hazards

 2-bit Predictor Statistics

Prediction accuracy of 4K-entry 2-bit prediction buffer on SPEC89 benchmarks:

accuracy is lower for integer programs (gcc, espresso, eqntott, li) than for FP
CSCE430/830 Pipeline Hazards
 2-bit Predictor Statistics

Prediction accuracy of 4K-entry 2-bit prediction buffer vs. “infinite” 2-bit buffer:
increasing buffer size from 4K does not significantly improve performance
CSCE430/830 Pipeline Hazards