Computer COSC3330 Architecture
Lecture 8. Pipelining
Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston
Next Pipelining Hazard
Instruction Execution
Sign-extended immediate
ROM
Offset Offset
Instruction Register
Target addr ext Target addr ext
4 32 data
32 1 32 0 Ydo RegFile 32 Xdo 32 Zdi we 32 32 A B 2 4 ALU 2 32
msel r/w Memory Address
beq mux bne (if true) + + jr j
Program Counter
Data
jr/j
I-Fetch 32 addr
Decode
Execute Memory
Write Result
Instruction Memory
Instruction Execution
5 basic steps fetch instruction (F) decode instruction/read registers (R) execute (X) access memory (M) store result (W)
Execution Datapath
5 5 5 Zwa Xra Yra RegFile Zdi Xdo we 2
Sign-extended immediate
imm enable 1 0
32 32 32 32 A
st enable
msel
r/w
Clock 32
Ydo
Memory Address
32
v/s LF ALS ALU ST SD
B
4 2
32
Data
ld enable
Execution Datapath
1 x 0
4
+
Xra Yra Zra Zdi
RegFile
Shift Left 2
+
Address Write data
Address Instruction
Xdo 1 0
Ydo
Data Memory
Read data
PC
Instruction Memory
1 u 0 0
1
Sign Ext
Address
Control Flow Support
rs
Target addr Target addr (from ROM) (from Offset (from ROM) Offset
ext
4
ext 1 jr/j mux 0 jr j
1 mux 0
32
32x32 RegFile
Datapath
+
+
Program Counter Instruction Register addr data
Instruction Memory
32
beq 1 mux 0 bne (if true)
Microcode ROM
Control Flow Support
1 x 0
4
+
Shift Left 2
+
Data Memory
RegFile Address Instruction
PC
Instruction Memory
Xra Yra Zra Zdi
Xdo 1 0
Ydo
Address Write data Read data
1 u 0 0
1
Sign Ext
Address
Datapath Timing
Instruction Fetch
2ns 1 x 0
Instruction Decode/ Execute/Addr Cal RegFile Read
1ns 2ns
Mem Access
Write Back
2ns
1ns
Clock rate = 10
+
9
/ 8 = 125MHz
4
+
Xra Yra Zra Zdi
RegFile
Shift Left 2
Address Instruction
Xdo 1 0
Ydo
Address Write data
Data Memory
Read data
PC
Instruction Memory
1 u 0 0
1
Sign Ext
Instruction execution time = ? Clock frequency = ?
9
Processor Performance
Performance of single-cycle processor is limited by the long critical path delay The critical path limits the operating clock frequency Can we do better? New semiconductor technology will reduce the critical path delay by manufacturing small-sized transistors Core 2 Duo is manufactured with 65nm technology Core i7 is manufactured with 45nm technology Can we increase the processor performance with a different microarchitecture? Yes! Pipelining
10
Revisiting Performance
Laundry Example
Ann, Brian, Cathy, Dave each has one load of clothes to wash, dry, and fold Washer takes 30 minutes Dryer takes 40 minutes Folder takes 20 minutes A B C D
11
Sequential Laundry
6 PM 7 8 9 Time 10 11 Midnight
30
T a s k O r d e r
40 20 30
40 20 30
40 20 30
40 20
A B C D Response time: 90 mins Throughput: 0.67 tasks / hr (= 90mins/task, 6 hours for 4 12 loads)
Pipelined Laundry
6 PM 7 8 9 Time 10 11 Midnight
30
T a s k O r d e r
40
40
40
40 20
A B C D
Response time: 90 mins Throughput: 1.14 tasks / hr (= 52.5 mins/task, 3.5 hours for 4 13 loads)
Pipelining doesnt help latency (response time) of a single task Pipelining helps throughput of entire workload Multiple tasks operating simultaneously Unbalanced lengths of pipeline stages reduce speedup Potential speedup = # of pipeline stages
Pipelining
Improve performance by increasing instruction throughput
Instruction Fetch Register File Access (Read) ALU Operation Data Access Register Access (Write)
2ns
1ns
2 4 6
2ns
8 10
2ns
12 14 16
1ns
18
P ro g ra m e x e c u t io n T im e o rd e r ( i n in s t r u c t i o n s ) lw $ 1 , 1 0 0 ( $ 0 ) lw $ 2 , 2 0 0 ( $ 0 ) lw $ 3 , 3 0 0 ( $ 0 ) P r o g ra m e x e c u t io n T im e o rd e r ( in i n s t r u c tio n s ) lw $ 1 , 1 0 0 ($ 0 ) lw $ 2 , 2 0 0 ($ 0 ) lw $ 3 , 3 0 0 ($ 0 )
Sequential Execution
2
4
6
8
10
12
14
Pipelined Execution
14
Pipelining (Cont.)
P ro g ra m e x e c u tio n T im e o rd e r (in in s tru c tio n s ) lw $ 1 , 1 0 0 ($ 0 ) lw $ 2 , 2 0 0 ($ 0 ) lw $ 3 , 3 0 0 ($ 0 )
Multiple instructions are being executed simultaneously
2 4 6 8 10 12 14
15
Pipeline Speedup
If all stages are balanced (meaning that each stage takes the same amount of time) If not balanced, speedup is less Speedup comes from increased throughput (the latency of instruction does not decrease) Time to execute an instructionsequential Number of stages
Time to execute an instructionpipeline
=
Pipelining and ISA Design
MIPS ISA is designed for pipelining
All instructions are 32-bits (4 bytes) Compared with x86 (CISC): 1- to 17-byte instructions Regular instruction formats Can decode and read registers in one cycle Load/store addressing Calculate address in 3rd stage Access memory in 4th stage Alignment of memory operands in memory Memory access takes only one cycle For example, 32-bit data (word) is aligned at word address, 0x0, 0x4, 0x8, 0xC
17
Basic Idea
Instruction Fetch Instruction Decode/ Execute/Addr Cal RegFile Read
1 x 0
Mem Access Write Back
4
+
Xra Yra Zra Zdi
RegFile
Shift Left 2
+
Address Write data
Address Instruction
Xdo 1 0
Ydo
Data Memory
Read data
PC
Instruction Memory
1 u 0 0
1
Sign Ext
What do we have to add to actually split the datapath into stages?
18
Basic Idea
Instruction Fetch Instruction Decode/ Execute/Addr Cal RegFile Read
1 x 0
Mem Access Write Back
4
+
Xra Yra Zra Zdi
RegFile
Shift Left 2
+
Address Write data
Address Instruction
Xdo 1 0
Ydo
Data Memory
Read data
PC
Instruction Memory
1 u 0 0
1
Sign Ext clock
D
F/F
Q Q
D
F/F
Q Q
D
F/F
Q Q
D
F/F
Q Q
19
Pipeline Depth vs. Performance
Adding more pipelines allows using higher clock speeds.
Fmax=1/Tdmax
Tdmax: delay of stage with the greatest delay
S1 S2 S3 S4 S5
Tda
More delay per stage
S 1
S 2
S 3
S 4
S 5
S 6
S 7
S 8
S 9
S Less delay 1 per stage 0
Tdb
ARM Pipeline Stages
x86 Pipeline Stages
1 2 3 4 5 6 7 8 9 10 Fetch Fetch Decode Decode Decode Rename ROB Rd Rdy/Sch Dispatch Exec
P3 Pipeline The P4 CPUs and derivatives were designed to reach high clock speeds. 20 pipelines initially 31 Pipeline stages in later revisions
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
TC Nxt IP TC FetchDriveAlloc Que Rename Sch Sch Sch Disp Disp RF RF Ex Flgs BrCkDrive
?
Pentium 4 Pipeline Core 2: It has only 14 pipeline stages
Pipeline Depth vs. Performance
Studies by Harstein and Puzak shows the effect of pipeline depth on the performance. It was found that there is an optimal number of pipeline stages which maximizes performance.
Graphically Representing Pipelines
Time lw add
IF
2 4 6 8 10
ID
EX
MEM
WB
IF
ID
EX
MEM
WB
Shading indicates the unit is being used by the instruction Shading on the right half of the register file (ID or WB) or memory means the element is being read in that stage Shading on the left half means the element is being written in that stage
24
Hazards
It would be happy if we split the datapath into stages and the CPU works just fine But, things are not that simple as you may expect There are hazards! Hazard is a situation that prevents starting the next instruction in the next cycle Structure hazards Conflict over the use of a resource at the same time Data hazard Data is not ready for the subsequent dependent instruction Control hazard Fetching the next instruction depends on the previous branch outcome
25
Structure Hazards
Structural hazard is a conflict over the use of a resource at the same time Suppose the MIPS CPU with a single memory Load/store requires data access in MEM stage Instruction fetch requires instruction access from the same memory Instruction fetch would have to stall for that cycle Would cause a pipeline bubble Hence, pipelined datapaths require either separate ports to memory or separate memories for instruction and data
Address Bus Address Bus
MIPS CPU
Data Bus
Memory
MIPS CPU
Data Bus Address Bus Data Bus
Memory
26
Structure Hazards (Cont.)
Time lw add sub
IF
2 4 6 8 10
ID
EX
MEM
WB
IF
ID
EX
MEM
WB
IF
ID
EX
MEM
WB
add
IF
ID
EX
MEM
WB
Either provide separate ports to access memory or to provide instruction memory and data memory separately
27
Data Hazards
Data is not ready for the subsequent dependent instruction
add $s0,$t0,$t1 sub $t2,$s0,$t3
IF
ID
EX
MEM
WB
IF
ID
Bubble EX
Bubble MEM
WB
To solve the data hazard problem, the pipeline needs to be stalled (typically referred to as bubble) Then, the performance is penalized A better solution? Forwarding (or Bypassing)
28
Forwarding
add $s0,$t0,$t1
IF ID EX MEM WB
sub $t2,$s0,$t3
IF
ID
Bubble
Bubble
EX
MEM
WB
29
Data Hazard - Load-Use Case
Cant always avoid stalls by forwarding Cant forward backward in time! Hardware interlock is needed for the pipeline stall
lw $s0, 8($t1)
IF
ID
EX
MEM
WB
sub $t2,$s0,$t3
IF
ID
Bubble EX
MEM
WB
This bubble can be hidden by proper instruction scheduling
30
Code Scheduling to Avoid Stalls
Reorder code to avoid use of load result in the next instruction
A = B + E; // B is loaded to $t1, E is loaded to $t2 C = B + F; // F is loaded to $t4
lw lw add sw lw add sw $t1, $t2, $t3, $t3, $t4, $t5, $t5, 0($t0) 4($t0) $t2 $t1, $t2 12($t0) 8($t0) $t1, $t4 $t4 16($t0) lw lw lw add sw add sw $t1, $t2, $t4, $t3, $t3, $t5, $t5, 0($t0) 4($t0) 8($t0) $t2 $t1, $t2 12($t0) $t1, $t4 $t4 16($t0)
stall
stall
13 cycles
31
11 cycles
Control Hazard
Branch determines the flow of instructions Fetching the next instruction depends on the branch outcome
Pipeline cant always fetch correct instruction Branch instruction is still working on ID stage when fetching the next instruction
Taken target address is known here
ID
Branch is resolved here
EX ID IF Bubble MEM WB MEM
beq $1,$2,L1 add $1,$2,$3 sw $1, 4($2)
IF
IF Bubble
EX ID
WB MEM WB
EX
L1: sub $1,$2, $3
IF
32
ID
EX
MEM
WB
Fetch the next instruction based on the comparison result
Reducing Control Hazard
To reduce 2 bubbles to 1 bubble, add hardware in ID stage to compare registers (and generate branch condition) But, it requires additional forwarding logic
Taken target address is known here
Branch is resolved here
EX MEM WB
beq $1,$2,L1 add $1,$2,$3
IF
ID
Bubble IF
ID
EX
MEM
WB
L1: sub $1,$2, $3
IF
ID
EX
MEM
WB
Fetch instruction based on the comparison result
33
Delayed Branch
Many CPUs adopt a technique called the delayed branch to further reduce the stall
Delayed branch always executes the next sequential instruction The branch takes place after that one instruction delay Delay slot is the slot right after a delayed branch instruction
Taken target address is known here Branch is resolved here
ID EX MEM WB
beq $1,$2,L1 add $1,$2,$3
IF
(delay slot)
IF
ID
EX
MEM
WB
L1: sub $1,$2, $3
IF
ID
EX
MEM
WB
Fetch instruction based on the comparison result
34
Delay Slot (Cont.)
Compiler needs to schedule a useful instruction in the delay slot, or fills it up with nop (no operation)
// $s1 = a, $s2 = b, $3 = c // $t0 = d, $t1 = f a = b + c; if (d == 0) {f = f + 1;} f = f + 2;
add $s1,$s2, $s3 bne $t0,$zero, L1 nop //delay slot addi $t1, $t1, 1 L1: addi $t1, $t1, 2
Can we do better? bne $t0, $zero, L1 add $s1,$s2,$s3 // delay slot addi $t1, $t1, 1 L1: addi $t1, $t1, 2
35
Fill the delay slot with a useful and valid instruction
Branch Prediction
Longer pipelines (implemented in Core 2 Duo, for example) cant readily determine branch outcome early Stall penalty becomes unacceptable since branch instructions are used so frequently in the program Solution: Branch Prediction Predict the branch outcome in hardware Flush the instructions (that shouldnt have been executed) in the pipeline if the prediction turns out to be wrong Modern processors use sophisticated branch predictors In MIPS pipeline, we are going to design hardware (as if branchesnot-taken) with no delayed branch Fetch the next instruction after branch If the prediction turns out to be wrong, flush out the instruction fetched
36
MIPS with Predict-Not-Taken
Prediction correct
Flush the instruction that shouldnt be executed
Prediction incorrect
37
Pipeline Summary
Pipelining improves performance by increasing instruction throughput Executes multiple instructions in parallel Pipelining is subject to hazards Structure hazard Data hazard Control hazard ISA affects the complexity of the pipeline implementation
38