Basic Pipelining: and Intermediate
Pipelining:
Concepts
Concepts
Introduction
The Major Hurdle of Pipelining
Pipeline Hazards
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Scoreboarding
Conclusion
CDA 5155 Spring 2012
Copyright 2012 Prabhat Mishra
1
Introduction
In early CPUs, deep combinational logic
networks were used between state updates.
Signal delays may vary widely across paths.
New input cannot be provided to the network until
the slowest paths have finished.
Slow clock speed, slow processing rates.
Logic Gate
2
Introduction
In pipelined design, logic networks are divided
into shallow slices (pipeline stages).
Delays through the network are made uniform.
A new input can be provided to each slice as soon
as its quick, shallow network has finished.
Clock cycle is only as long as the slowest stage.
3
Pipelining
Pipelining is an implementation technique
whereby multiple instructions are overlapped
in execution
Takes
advantage of parallelism that exists among
the actions needed to execute an instruction
fetch instruction from memory
decode to figure out what to do
read source operands
execute
write results
4
Simple RISC Datapath
IF
ID
EX
MEM
WB
Next PC
Program
Counter
Inst.
Reg.
Load
fr. Mem.
Data
5
Basic RISC Pipelining
Basic idea:
Each
instruction spends 1 clock cycle in each of
the 5 execution stages.
During 1 clock cycle, the pipeline can process (in
different stages) 5 different instructions.
6
Alternative Visualization
7
Visualization with Pipeline Registers
8
Limits of Pipelining
Increasing the number of pipeline stages in a
given logic block by a factor of n
Generally
allows increasing clock speed and
throughput by a factor of almost n.
Usually less than n because of overheads such as
latches and balance of delay in each stage.
But, pipelining has a natural limit:
At
least 1 layer of logic gates per pipeline stage.
Practical minimum is usually several gates (2-10).
Commercial designs are rapidly nearing this point.
9
Few Related Terms
Clock Period
= Max { time delay of a stage }1k + other delay (e.g., skew, latch delay)
Frequency
Reciprocal
f =1
of the clock period
Speedup
k
stage pipeline, n instructions
n.k
Sk =
k + (n 1) k when n >> k.
= Sk k
Efficiency
Ratio
of its actual speedup to the ideal speedup
Throughput
Number
w =
of instructions that can be completed per unit time.0
1
Beyond Pipelining
There are some problems with clocked logic
High
power dissipation for clock signal distribution
throughout chip
up to 40% or more of total power
Clock
signal timing differences (clock skew) between
different portions of chip
Can be significant, interfere with proper execution
Each
clock cycle can only be as fast as the worst-case
delay over all pipeline stages in the entire design.
Many results end up waiting when they dont need to.
An alternative
Self-timed
logic, Asynchronous logic, Dataflow circuits
Each logic block uses explicit handshaking signals
Globally asynchronous locally synchronous (GALS)
11
Outline
Introduction
The Major Hurdle of Pipelining
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Conclusion
12
Pipeline Hazards
Hazards are circumstances which may lead
to stalls (delays, bubbles) in the pipeline if
not addressed.
Three major types:
Structural
hazards:
Not enough HW resources to keep all instrs. moving.
Data
hazards
Data results of earlier instructions not available yet.
Control
hazards
Control decisions resulting from earlier instr. (branches)
not yet made; dont know which new instr. to execute.
13
Structural Hazard Example
The processor has a combined instruction+data memory with only 1 read port
14
Hazards Produce Bubbles
15
Textual View
A pipeline stalled for a structural hazard a load with only one memory port
16
Three Types of Data Hazards
Let i be an earlier instruction, j a later one.
RAW (read after write)
j
tries to read a value before i writes it
WAW (write after write)
i
and j write to same place, but in the wrong order.
Only occurs if more than 1 pipeline stage can write
WAR (write after read)
j
writes a new value to a location before i has read
the old one.
Only occurs if writes can happen before reads in
pipeline (in-order).
17
Data Hazard Example
18
Forwarding for Data Hazards
19
Another Forwarding Example
20
An Unavoidable Stall
21
Stalling in midst of instruction
22
Data Hazard Prevention
A clever compiler can often reschedule
instructions to avoid a stall.
A
simple example:
Original code:
lw r2, 0(r4)
add r1, r2, r3 Stall happens here.
lw r5, 4(r4)
Transformed code:
lw r2, 0(r4)
lw r5, 4(r4)
add r1, r2, r3
No stall needed.
23
% of loads that cause a stall
Simple RISC Pipeline Stall Statistics
24
Control (Branch) Hazards
Suppose the new PC value is not computed
until the MEM stage.
Then we must stall 3 clocks after every
branch!
25
Performance of Pipelines with Stalls
Speedup
Avg.
inst. time (unpipelined) / Avg. inst. time (pipelined)
CPI
unpipelined / CPI pipelined
Pipeline
Depth / (1 + Pipeline stall cycles per instruction)
Pipeline
Depth / (1 + Branch frequency x Branch penalty)
26
Delayed Branches
Machine code sequence:
Branch instruction
Delay slot instruction(s)
Post-branch instructions
Branch is taken
(if taken) at this point
27
Scheduling the Branch-Delay Slot
28
Outline
Introduction
The Major Hurdle of Pipelining
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Conclusion
29
Simple RISC Datapath
IF
ID
EX
MEM
WB
Next PC
Program
Counter
Inst.
Reg.
Load
fr. Mem.
Data
30
Description of Pipe Stages
31
Data Hazard Detection
32
Hazard Detection Logic
Example: Detecting whether an instruction
that has just been fetched needs to be stalled
because of a preceding load.
ID/EX.IR[rt] == IF/ID.IR[rs]
ID/EX.IR[rt] == IF/ID.IR[rt]
ID/EX.IR[rt] == IF/ID.IR[rs]
33
Forwarding Situations in DLX
34
Implementing Forwarding in HW
35
Early Branch Resolution
36
Original RISC Datapath
IF
ID
EX
MEM
WB
Next PC
Program
Counter
Inst.
Reg.
Load
fr. Mem.
Data
37
New Pipeline Logic
38
Control Instruction Statistics
39
Statistics on Taken Branches
40
Predict-Not-Taken
41
Static Branch Prediction
Earlier we discussed predict-taken, predictnot-taken static prediction strategies
Applied
uniformly across all branches in program
Static analysis in compiler may be able to do
better, if it can non-uniformly predict whether
each specific branch is likely to be taken
One
way: backwards taken, forwards not taken.
If we can do better, it can help with static code
scheduling to reduce data hazard stalls
Also
may assist later dynamic prediction
42
Prediction Helps Static Scheduling
Some data dependences
Code
movements
to consider:
If
case
Else
case
else:
after:
LD
DSUBU
BEQZ
OR
DADDU
J
DADDU
R1,0(R2) Potential load delay to fill
R1,R1,R3
Which way will this
R1,else
branch go?
R4,R5,R6
R10,R4,E3
If-then-else
control flow
after
R7,R8,R9
43
Some Static Prediction Schemes
Always predict taken
34%
mispredict rate on SPEC (range 9%-54%)
Backwards predict taken, forwards not taken
In
SPEC, more than of forwards are taken!
This does worse than always predict taken strategy
Usu.
not better than 30-40% misprediction rate
Better than either: Use profile information
Collect
statistics on earlier program runs.
Works
well because individual branches tend to be
strongly biased (taken or not) given average data
Bias remains stable across multiple runs
44
Profile-Based Predictor Statistics
Floating-Point
45
Predict-Taken vs. Profile-Based
Instructions between mis-predictions
Floating-point
46
Types of Exceptions (Interrupts, Faults)
I/O device request, timer event
Invoking OS services from a user program
Tracing (single-stepping) through program
Breakpoints
Integer arithmetic overflow, divide by zero
FP arithmetic anomaly (overflow, underflow, , NaN, etc.)
Page fault (page not in physical memory)
Misaligned memory access
Memory-protection violation (acc. mem. not alloced to proc.)
Illegal (undefined or unimplemented) instruction
Hardware malfunction
Power-related interrupt (e.g. battery low, power failure)
47
Terminology Across Architectures
48
Exception Categorizations
Synchronous vs. asynchronous
Event
synchronized with program execution?
User requested vs. coerced
Event
caused intentionally by user program?
User maskable (can be disabled) or not
Can
event be disabled?
Within instructions or between instructions
Does
event prevent instruction from completing?
Resume vs terminate
Does
the program continue from where it left off
after exception is handled, or does it stop?
49
Restartable Exceptions
Requirements:
Exception
may occur within instruction.
Program must continue after exception is handled.
Examples:
Virtual
memory page fault.
Difficult because:
Pipeline
state must be saved.
One approach, for easy cases:
1.
Force a trap inst. into pipeline on next IF.
2. Clear pipeline behind faulting instruction.
3. Exception handler saves PC of faulting instr.
50
Precise vs. Imprecise Handling
Machines may support either or both modes of
exception handling:
Precise exception handling:
Correctly implement all possible combinations of
exceptions in all circumstances.
May be a requirement for some systems/applications.
May be 10x slower!
Easier for integer than floating-point.
Useful for debugging code.
Imprecise
exception handling:
Only correctly implement the most common cases.
Software may avoid some exceptions.
Only statistical guarantee of correctness, through testing.1
5
Exceptions in MIPS pipeline
Instruction Fetch, & Memory stages
Page
fault on instruction/data fetch
Misaligned memory access
Memory-protection violation
Instruction Decode stage
Undefined/illegal
opcode
Execution stage
Arithmetic
exception
Write-Back stage
None!
52
Out-of-Order Exceptions
Consider the following code sequence:
LW
IF
ADD
ID
IF
EX
ID
MEM WB
EX MEM WB
The ADD may cause an exception during IF,
before LW causes an exception during MEM!
Cant
restart PC on the ADD!
Solution:
Note
the exception in a status vector, carried
along.
Disable writes for that instruction.
Resolve all exceptions at a late stage ( e.g. WB).
53
Outline
Introduction
The Major Hurdle of Pipelining
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Conclusion
54
Multi-cycle Operations for FP
55
Pipelined Multiple-Issue FPU
56
FPU Pipelining Issues in DLX
Notice instructions may complete out-of-order:
MULTD
ADDD
LD
SD
IF ID M1 M2
IF ID A1
IF ID
IF
M3
A2
EX
ID
M4
A3
ME
EX
M5 M6 M7 ME WB
A4 ME WB
WB
ME WB
Raises the possibility of WAW hazards, and
structural in hazards MEM & WB stages.
Structural hazards may occur especially often
with non-pipelined DIV unit.
Out-of-order completion impacts exception
handling.
57
Typical FP Code Seq. with Stalls
MUL.D stalls
in ID 1 cycle waiting for new value of F4 from MEM stage of L.D
ADD.D stalls
1 cycle in IF waiting for MUL.D to leave ID,
then 6 cycles in ID waiting for new F0 to be returned by MUL.D stage
M7.
S.D stalls
6 cycles in IF waiting for ADD.D to leave ID,
then 2 cycles in EX waiting for new F2 to be returned by ADD.D stage
A4, then 1 more cycle in EX waiting for ADD.D to clear MEM stage.
Clock Cycle Number
Instruction
1
2
L.D
IF
ID EX ME WB
F4,0(R2)
MUL.D F0,F4,F6
ADD.D F2,F0,F8
S.D
F2,0(R2)
IF
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17
ID stall M1 M2 M3 M4 M5 M6 M7 ME WB
IF stall ID stall stall stall stall stall stall A1 A2 A3 A4 ME WB
IF stall stall stall stall stall stall ID EX stall stall stall ME
58
Issues in Multi-Cycle Operations
Stall for RAW is longer and more frequent (Fig. A33)
WAW is possible; WAR is not (why?)
Structural Hazard possible for non-pipelined unit
Multiple WBs are likely (Fig. A.34)
Handling hazards at Issue (ID) stage:
Check structural hazards: functional unit, WB port
Check RAW hazards: Issue with forwarding
Check WAW hazards: Not issue to make sure write in order
Detect
and stall instruction before MEM and WB stages
59
FP Stall Statistics
per FP operation
60
ISA Design Impacts Pipelining
Variable instruction lengths & run times:
Introduces
delays due to pipeline inequities.
Complicates hazard-detection & precise exceptions
Sophisticated addressing modes:
Post-autoincrement
complicates hazard detection,
restarting, introduces WAR & WAW hazards.
Multiple-indirect modes complicate pipeline control
& timing.
Self-modifying code:
What
if you overwrite an instruction in the pipe?
Implicit condition codes: WAR hazards, restarts
61
Outline
Introduction
The Major Hurdle of Pipelining
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Conclusion
62
Real MIPS R4000 SuperPipeline
IF,IS - Instruction cache fetch, First & Second halves.
RF - Inst. decode, Register Fetch, hazard check
EX - Execution (EA calc, ALU op, target calc)
DF,DS - Data cache access, First & Second halves.
TC - Tag Check, did cache access hit?
WB - Write-Back for loads & register-register ops.
63
R4000: Two-Cycle Load Delay
64
R4000: Three-Cycle Branch Delay
65
R4000 FP Functional Unit Stages
U Unpack floating-point numbers
FP adder functional unit stages:
A
Mantissa ADD stage
R
Rounding stage
S
Operand shift stage
FP multiplier functional unit stages:
E
Exception test stage
M
First stage of multiplier
N
Second stage of multiplier
FP divider function unit stages:
D
Divide pipeline stage
66
Latency and Initiation Interval
Initiation Both units used in same clock cycle
FP Instruction Latency interval
MIPS R4000 Pipe stages
Add, subtract
4
3
U, S+A, A+R, R+S
Multiply
8
4
U, E+M, M, M, M, N, N+A, R
Divide
36
35
U, A, R, D27, D+A, D+R, D+A, D+R, A, R
Square root
112
111
U, E, (A+R)108, A, R
Negate
2
1
U, S
Absolute value
2
1
U, S
FP compare
3
2
U, A, R
U unpack
A mantissa add
R round
S shift
E exception test
M multiply 1st stage
Pair of units used on
108 consecutive cycles
N multiply 2nd stage
D divide
67
FP Multiply followed by Add
Clock cycle
Issue/
Op Stall
0
1
2
3
4
5
6
7
mul
Issue
U
EM
M
M
M
N
NA
R
add
Issue
U
SA
AR RS
Issue
Issue
Stall
Stall
Issue
Issue
U
SA
U
8
9
10
11
12
AR RS
SA
U
AR RS
SA AR RS
U
SA AR RS
U
SA
U
AR RS
SA
AR RS
68
The MIPS R4300 Pipeline
Manufactured by NEC
64-bit processor implements MIPS64 ISA
Used in embedded applications
Nintendo-64
game processor, network router,
Multiple EX stages for floating-point pipeline
Out-of-order
completion, precise exceptions
NEC VR 4122: Integer datapath, software for FP operations
69
Outline
Introduction
The Major Hurdle of Pipelining
Implementation
Multi-cycle Operations
The MIPS R4000 Pipeline
Crosscutting Issues
Conclusion
70
RISC ISA and Efficiency of Pipelines
Simple instruction set
Easier
to schedule code to improve performance
Static scheduling by compiler
Dynamic scheduling by hardware
Leads
to out-of-order execution (completion)
Requires
mechanism to ensure correct execution
Scoreboarding
71
Scoreboarding
Technique for implementing an instruction
queue that supports dynamic reordering.
Developed on CDC 6600 (decades ago).
Reordering must check WAR/WAW hazards:
DIVD F0,F2,F4
ADDD F10,F0,F8
SUBD F8,F8,F14
Long-running
Depends on DIV
Anti-depends on ADD
Goal: Begin execution of instructions as early
as possible
72
Simple scoreboarded datapath
73
Pipeline with Scoreboarding
0.
1.
2.
3.
4.
(F) Fetch instruction from cache or prefetch buffer
(I) Issue inst. to an execution path (when no structural/WAW hazards)
(R) Read operands (when no RAW hazards remain)
(E) Execute instruction (possibly multi-cycle)
(W) Write results (when no WAR hazards remain)
Scoreboard /
Control Unit
Pre-execution
buffers
Read
operands
Instruction
Fetch
Instruction
Issue
Instruction Decode
Execution
unit 1
Execution
unit 2
Write
results
Pre-issue
buffer
Read
operands
Post-execution
buffers
74
1. Instruction Issue (IS) Stage
Receive newly-fetched instruction
Decode binary instruction format
Check for structural hazards:
Replaces first half
of ID stage
Instruction
needs execution unit currently in use, whose
initiation interval hasnt passed?
Check for WAW hazards:
Instruction
wants to write to a register that an active
instruction (issued, but not yet finished) wants to write to?
Bad if they finish out-of-order!
Stall all current (& future) instruction issuing, until
none of these hazards remain.
Issue instructions (in-order) to the appropriate
execution units & track status on scoreboard.
75
2. Read Operands (RO) Stage
Receive instruction issued to functional unit.
Check for RAW hazards: Are all source
operands available yet?
If
no: Hold instruction in a pre-execution buffer.
If buffer has only 1 entry, this and all not-yet-issued
instructions using this functional unit must wait.
If
yes: Read operands from register file, & start
instruction down the execution units pipeline.
Replaces second half
of ID stage
76
3. Execution (EX) Stage
Once operands are received, begin execution
of the instruction in the execution unit.
Execution may take multiple cycles.
When result is ready, notify scoreboard of
instruction completion.
Replaces old EX stage
77
4. Write Result (WR) stage
Receive completed instruction & its result from
execution unit.
Check for WAR hazards:
Does
any previously-issued instruction that has not
yet read its operands depend on the old value we
are about to overwrite? (Does it anti-depend on
us?)
Replaces WB stage
While yes:
Stall instruction in a post-execution buffer.
When no:
Write instruction result to register file.
78
Scoreboard Implementation
One typical implementation uses three tables:
Instruction
status, for each instruction on the
scoreboard
Which stage of execution is the instruction currently in?
Functional
Unit (FU) status, for each FU:
What instruction (if any) is being processed?
If inst. is in RO stage, then for each operand:
What register is the operand coming from?
Is the operand ready?
If not, which FU will produce the operand?
Register
result status, for each reg. in the ISA:
Which currently-running FU (if any) is scheduled to
overwrite the given register?
79
Functional Unit Status Table
For each functional unit, the following fields:
Busy
Op
Fi
Is the unit busy (Yes/No)?
Which exact opcode to perform in the FU?
Destination register of instruction in the FU
Fj,Fk
Source registers of instruction
These
fields are only needed during RO stage:
Qj,Qk FUs to write new values of source registers, or 0
Rj,Rk Are operands Fj,Fk ready? (Yes/No)
Register result status table has only 1 field:
Result
Which currently-executing FU will write its
80
result to this register?
#1: Scoreboard After 2nd LDs EX
81
#2: Just before MULTDs WR stage
82
#3: Just before DIVDs WR stage
83
Scoreboarding Logic
In the table below:
= functional unit used by given instruction
D,S1,S2 = given instrs destination & source regs
op = operation to be performed
Result[reg] = Register result status table entry
for register identified by reg
FU
(true @ start
of cycle)
(completed by end of cycle)
(Avoids WAW)
(Avoids RAW)
(Avoids WAR)
(Release other writers of operand regs.)
(Lock into
operands)
84
A Problem with that Implementation
Note: The following artifact is introduced:
Only
one instruction per execution unit at a time!
Even if it is a pipelined, multi-cycle unit!
Think: What would you need to change in
order to fix this problem?
Change
the register result & FU status tables:
Specify which recently-issued instruction (not just FU) is
responsible for producing a given register value.
Instructions can be assigned unique ids when issued
Add
structural hazard detection logic, if needed:
To determine exactly when it is safe for a new
instruction to actually enter the pipe for each FU.
85
Another problem w. Scoreboarding
Since WAW hazards are avoided by stalling
on instruction issue, we lose time during
which we could go ahead and begin
executing.
But
note: If we just went ahead and issued the
instruction, then we havent ensured that results
will get written in the right order!
Think: How might you fix this deficiency?
86
Limitations on Scoreboarding
The following factors limit the number of stalls
that can be eliminated by scoreboarding:
Available parallelism among instructions
Cross-basic-block
rescheduling helps with this
Number of scoreboard instruction table
entries
Number and types of functional units
Presence of name dependences (lead to
WAR/WAW hazards)
Can
fix with register renaming
Static, or dynamic (well see how later)
87
Fallacies and Pitfalls
Why compiler generates two writes w/o read WAW
instruction in the delay slot from fall through
assuming branch is not taken.
Extensive pipelining can impact other aspects of a design
leading to overall worse cost-performance
8600 80 ns cycle time; 8650 55 ns
8700 operates at micro-instruction level - 45 ns
simpler pipeline, smaller CPU, faster clock 45 ns
8650 has CPI adv. of 20%; 8700 has clock adv. 20%
8700 achieves same performance with less hardware
Evaluating a compile time scheduler on the basis of unoptimized code
Idle cycles increases 18% in gcc/R3000 from unopt. to optimized
Optimized version is faster since it has less instructions.
To fairly evaluate a scheduler use optimized code.
88