749 ELE A Superscalar Pipeline
Ali Ziya Alkar
Adapted from Mary Jane Irwin ( www.cse.psu.edu/~mji ) [Computer Organization and Design, Patterson & Hennessy, 2005 and Instruction Issue Logic, IEEETC, 39:3, Sohi, 1990]
ELE 749 L10&11 SS Model.1
Alkar 2007
Review: Extracting More Performance
To achieve high performance, need both machine parallelism and instruction level parallelism (ILP) by
Superpipelining Static multiple-issue (VLIW) Dynamic multiple-issue (superscalar)
A processors instruction issue and completion policies impact available ILP
In-order issue with in-order completion In-order issue with out-of-order completion
- Creates output dependencies (write before write)
Out-of-order issue with out-of-order completion
- Creates antidependency (write before read)
Out-of-order issue with out-of-order completion and in-order commit
Register renaming can solve these storage dependencies
ELE 749 L10&11 SS Model.2 Alkar 2007
Speedup Measurements
The speedup of the SS processor is
- Assumes scalar and superscalar machines have the same IC & CR
# scalar cycles speedup = sn = -------------------------------# superscalar cycles To compute average speedup performance can use
Arithmetic mean Harmonic mean
AM =
1/n
i=1
n
si
HM =
n / ( 1/si )
i=1
n
- assigns a larger weighting to the programs with the smallest speedup
EX: two programs with same scalar cycles, with a SS speedup of 2 for program1 and 25 for program2
- AM = - HM =
ELE 749 L10&11 SS Model.3 Alkar 2007
Speedup Measurements
The speedup of the SS processor is
- Assumes scalar and superscalar machines have the same IC & CR
# scalar cycles speedup = sn = -------------------------------# superscalar cycles To compute average speedup performance can use
Arithmetic mean Harmonic mean
AM =
1/n
i=1
n
si
HM =
n / ( 1/si )
i=1
n
- assigns a larger weighting to the programs with the smallest speedup
EX: two programs with same scalar cycles, with a SS speedup of 2 for program1 and 25 for program2
- AM = * (2 + 25) = 13.5 - HM = 2 / (.5 + .04) = 2 /.54 = 3.7
ELE 749 L10&11 SS Model.4 Alkar 2007
Maximum (Theoretical) SS Speedups
The highest speedup that can be achieved with ideal machine parallelism (ignoring resource conflicts, storage dependencies, and procedural dependencies)
HM of 5.4 is the highest average speedup for these benchmarks that can be achieved even with ideal machine parallelism!
12 10 Speedup 8 6 4 2 0
From Johnson, 1992
cc om do du c gn uc he ss irs im lin pa ck si m pl e
ff
tr of f tw ol f
Alkar 2007
ELE 749 L10&11 SS Model.5
5d i
Baseline Superscalar MIPS Processor Model
Fetch Decode & Dispatch Issue & Execute Writeback Commit Load/Store Queue 1 2 BHT BTB LSQ FP NL RegFile I Integer NLI I I RegFile D$
P C
I$
3 4
FPALU
IALU IALU
RUU_Head
RUU
RUU_Tail
5 6
IMULT
Result Bus Register Update Unit (managed as a queue)
ELE 749 L10&11 SS Model.6 Alkar 2007
Typical Functional Unit Latencies
Result latency number of cycles taken by a functional unit (FU) to produce a result Issue latency minimum number of cycles between the issuing of an instruction to a FU and the issuing of the next instruction to the same FU
ELE 749 L10&11 SS Model.7
Issue Latency Integer ALU Integer multiply Load (on hit) Load (on miss) Store FltPt Add FltPt Multiply FltPt Divide FltPt Convert 1 1 1 1 1 1 1 12 1
Result Latency 1 2 1 40 n/a 2 4 12 2
Alkar 2007
Additional RegFile Fields
Each register in the general purpose RegFile has two associated n-bit counters (n of 3 is typical)
NI (number of instances) the number of instances of a register as a destination register in the RUU LI (latest instance) the number of the latest instance
When an instruction with destination register address Ri is dispatched to the RUU, both its NI and LI are incremented
Dispatch is blocked if a destination registers NI is 2n -1, so only up to 2n 1 instances of a register can be present in the RUU at any one time
When an instruction is committed (updates the Ri value) the associated NI is decremented
When NI = 0 the register is free (there are no instruction in the RUU that are going to write to that register) and LI is cleared
ELE 749 L10&11 SS Model.8 Alkar 2007
Register Update Unit (RUU)
A hardware data structure that is used to resolve data dependencies by keeping track of an instructions data and execution needs and that commits completed instructions in program order An entry in the RUU
speculative src operand 1 src operand 2 destination issued functional unit executed Yes/No Unit Number Yes/No
PC
Spec Instr Addr
Tag = RegFile addr || LI
Alkar 2007
ELE 749 L10&11 SS Model.9
Address
Yes/No
Ready
Ready
Tag
Content
Tag
Content
Content
Tag
Basic Instruction Flow Overview
Fetch (in program order): Fetch multiple instructions in parallel from the I$ Decode & Dispatch (in program order):
In parallel, decode the instrs just fetched and schedule them for execution by dispatching them to the RUU Loads and stores are dispatched as two (micro)instrs one to compute the effective addr and one to do the memory operation
Issue & Execute (out of program order): As soon as the RUU has the instrs source data and the FU is free, the instrs are issued to the FU for execution Writeback (out of program order): When done the FU puts its results on the Result Bus which allows the RUU and the LSQ to be updated the instr completes Commit (in program order): When appropriate, commit the instrs result data to the state locations (i.e., update D$ and RegFile)
ELE 749 L10&11 SS Model.10 Alkar 2007
Managing the RUU as a Queue
By managing the RUU as a queue, and committing instruction from RUU_Head, instruction are committed (aka retired) in the order they were received from the Decode & Dispatch logic (in program order)
Stores to state locations (RegFile and D$) are buffered (in the RUU and LSQ) until commit time Supports precise interrupts (the only state locations updated are those written by instructions before the interrupting instr)
The counter (LI) allows multiple instances of a specific destination register to exist in the RUU at the same time via register renaming
Solves write before write hazards if results from the RUU are returned to the RegFile in program order
Managing the RUU as a queue and committing from the head of the queue provides just this!
ELE 749 L10&11 SS Model.11 Alkar 2007
Major Functions of the RUU
Each of the tasks are done in parallel every cycle
1.
Accepts new instructions from the Decode & Dispatch logic Monitors the Result Bus to resolve true dependencies and to do write back of result data to the RUU Determines which instructions are ready for execution, reserves the Result Bus, and issues the instruction to the appropriate FU Determines if an instruction can commit (i.e., change the machine state) and commits the instruction if appropriate
2.
3.
4.
ELE 749 L10&11 SS Model.12
Alkar 2007
The First Function of the RUU
1.
Accepts new instructions from the Decode & Dispatch logic for each instruction in the fetch packet
The dispatch logic gets an entry in the RUU (a circular queue)
The RUU_Tail entry (currently empty) is allocated to the instruction and RUU_Tail is updated Then if RUU_Head = RUU_Tail, the RUU is full and further instruction fetch stalls until the RUU_Head advances (as a result of a commit)
For each source operand, if the contents of the source register is available, then it is copied to the source Content field of the RUU entry and its Ready bit is set. If not, the source RegFile addr || LI is copied to the source Tag field and the Ready bit is reset. For the destination operand, the RegFile destination addr || LI is copied to the allocated RUU destination Tag field The issued bit and executed bit are set to No, the number of the FU needed for the operation is entered, and the PC address of the instruction is copied to the PC Address field
ELE 749 L10&11 SS Model.13 Alkar 2007
Aside: Content Addressable Memories (CAMs)
Memories that are addressed by their content. Typical applications include RUU source tag field comparison logic, cache tags, and translation lookaside buffers
Memory hardware that compares the Search Data to the Match Field entries for each word in the CAM in parallel ! On a match the Data Field for that entry is output to Match Data on read or Match Data is written into the Data Field on write and the Hit bit is set. If no match occurs, the Hit bit is reset. CAMs can be designed to accommodate multiple hits.
ELE 749 L10&11 SS Model.14 Alkar 2007
Search Data
Match Field
Data Field
Hit
Match Data
The Second Function of the RUU
2.
Monitors the Result Bus to resolve true dependencies and to do write back of result data to the RUU
The Result Bus destination addr || LI is compared associatively to the source Tag fields (for those source operands that are not Ready). If a match occurs, the data on the Result Bus is gated into the Content field for matching source operands.
The Result Bus contains the result data, its RUU entry address, and its RegFile destination addr || LI
The result data is gated into the destination Content field of the RUU entry that matches the RUU entry addr on the Result Bus The executed bit is set to Yes
Resolves true dependencies through the Ready bit (i.e., must wait for the source operands before issue) Solves anti-dependencies through LI (making sure that the source fields get updated only for the correct version of the data)
ELE 749 L10&11 SS Model.15 Alkar 2007
The Third Function of the RUU
3.
Determines which instructions are ready for execution, reserves the Result Bus, and issues the ready instructions to the appropriate FUs for execution
When both source operands of an RUU entry are Ready, the RUU issues the highest priority instruction priority is given to load and store instrs and then to the instrs that entered the RUU the earliest (i.e., the ones closest to RUU_Head). Reserves the Result Bus Issues the instruction (sends to the FU the source operands, RUU entry addr, and the destinations RegFile addr || LI) and sets the issued bit to Yes
Multiple instructions can be issued in parallel, if they are ready, if they can reserve the Result Bus, and if they are destined for different FUs
ELE 749 L10&11 SS Model.16 Alkar 2007
The Fourth Function of the RUU
4.
Determines if an instruction can commit (i.e., change the machine state) and commits the instruction if appropriate
Monitors the executed bit of the RUU_Head entry. If the bit is set, the destination content data is written into the RegFile at the destinations RegFile address Matches the destination RegFile addr || LI against the RUUs source Tag fields and on match copies the destination content data into source Content fields Decrements the associated RegFile entrys NI counter Releases the RUU entry by incrementing the RUU_Head pointer
Solves output dependencies by writing to RegFile in program order Multiple instructions can commit in parallel if they are ready to commit, if they are writing to different RegFile registers, and if there are multiple RegFile write ports
ELE 749 L10&11 SS Model.17 Alkar 2007
Timing of RUU Activities
Need to add a slide like this but the timing on the following is slides not consistent with this one, so would need to change the following slides (or change the timing on this one)
decoded instrs dispatched to the RUU decode instrs
commit instrs completed in the previous cycle if appropriate & issue newly ready instrs to FUs for execution in the next cycle & write back result of completed instrs to RUU
FUs executing issued instrucions FUs output on Result Bus
ELE 749 L10&11 SS Model.18 Alkar 2007
Baseline Superscalar MIPS Processor Model
Fetch I$ mul $1,$1,$2 addi $1,$1,1 addi $3,$2,1 add $2,$3,$1 RegFile NI LI 0 0 1 0 # LI 3 1 3 0 0 1 0 0 3 Decode & Dispatch
LSQ_Head LSQ_Tail
Issue & Execute # LI
Writeback 1&2 D$ 2 9
Commit
LSQ 0 2,1 3+3 # LI 21 4 y4 5
IALU IALU
# LI
RUU_Head RUU_Tail
1
RUU
ELE 749 L10&11 SS Model.19
IMULT 6 Result Bus
Alkar 2007
Baseline Superscalar MIPS Processor Model
Fetch I$ mul $1,$1,$2 addi $1,$1,1 addi $3,$2,1 add $2,$3,$1 RegFile NI 0 0 1 0 LI 0 0 1 0 0 3 Decode & Dispatch
LSQ_Head LSQ_Tail
Issue & Execute # LI
Writeback 1&2 D$ 2 9
Commit
LSQ
# LI
RUU_Head
# LI 3 1 3
1
# LI 21
4 y4 5
IALU
IALU
RUU_Tail
RUU
ELE 749 L10&11 SS Model.20
IMULT 6 Result Bus 0 2,1 6
Alkar 2007
Baseline Superscalar MIPS Processor Model
Fetch I$ mul $1,$1,$2 addi $1,$1,1 addi $3,$2,1 add $2,$3,$1 RegFile NI 0 2 0 0 LI 0 2 0 0 0 LSQ 6
IALU
Decode & Dispatch
LSQ_Head LSQ_Tail
Issue & Execute # LI
Writeback 1&2 D$ 2 9
Commit
# LI 1
RUU_Head
# LI 3 3 1 121 1 3 6 1
# LI 21 11 12
4 6 y 4y y6
1 11
RUU_Tail
RUU
ELE 749 L10&11 SS Model.21
5 4 1 1,1 3*6 IMULT 6 Result Bus 0 2,1 6
Alkar 2007
IALU
Baseline Superscalar MIPS Processor Model
Fetch I$ mul $1,$1,$2 addi $1,$1,1 addi $3,$2,1 add $2,$3,$1 RegFile NI 0 2 0 0 LI 0 2 0 0 0 LSQ 6
IALU
Decode & Dispatch
LSQ_Head LSQ_Tail
Issue & Execute # LI
Writeback 1&2 D$ 2 9
Commit
# LI
RUU_Head
# LI 3 3 1 121 1 3 6 1
1 1 11
# LI 21 11 12
4 6 y 4y y6 4 5
IALU
RUU_Tail
RUU
ELE 749 L10&11 SS Model.22
IMULT 6 Result Bus
Alkar 2007
Baseline Superscalar MIPS Processor Model
Fetch I$ mul $1,$1,$2 addi $1,$1,1 addi $3,$2,1 add $2,$3,$1 RegFile NI 0 2 1 1 LI 0 2 1 1 0 LSQ Decode & Dispatch
LSQ_Head LSQ_Tail
Issue & Execute # LI
Writeback 1&2 D$ 2 9
Commit
# LI 31
RUU_Tail RUU_Head
# LI 12 3 121 1 6 1 6 1 1
# LI 21 11 12 31
4 y6 4 y5
4 3 3,1 6+1 5
IALU
IALU
1 11 1
RUU
ELE 749 L10&11 SS Model.23
6 Result Bus
1 1,1 3*6 IMULT
Alkar 2007
RUU with Bypass Logic
Consider an instruction that is to be added to the RUU whose source operand(s) have already been computed (i.e., exist somewhere in the RUU) but which have not yet been committed to the RegFile
The source Tag field comparison logic must also monitor the RUU to RegFile bus (in addition to the Result Bus)
With result bypass logic, can expedite the number of instrs ready to execute by doing an associative comparison of source operand Tags with the dst Tag of each RUU entry on instr Dispatch
ELE 749 L10&11 SS Model.24
From Sohi, 1990 # of RUU entries 4 8 15 30 Without bypass logic Speed up 0.912 1.082 1.225 1.393 Issue rate With bypass logic Speed up Issue rate 0.419 0.557 0.706 0.758
0.407 0.940 0.483 1.248 0.546 1.584 0.621 1.700
Alkar 2007
MicroOperations of Load and Store
Recall that loads and stores are dispatched to the RUU as two (micro)instrs one to compute the effective addr and one to do the memory operation
Load Store lw R1,2(R2) sw R1,6(R2) becomes becomes addi lw addi sw R0,R2,2 R1,R0 R0,R2,6 R1,R0
At the same time a LSQ entry is allocated
Each LSQ entry consists of a Tag field (RegFile addr || LI) and a Content field. The LI counter allows for multiple instances of stores (writes) to a memory address When a load completes (the D$ returns the data on the Result Bus) or a store commits (in program order) the LSQ entry is released Instruction dispatch is blocked if there is not a free LSQ entry and two free RUU entries
ELE 749 L10&11 SS Model.25 Alkar 2007
Load & Store RUU & LSQ Operation
Load addi R0,R2,2 lw R1,R0
NI 2 1 0 0
RUU_Tail RUU_Head
Store addi R0,R2,6 sw R1,R0
LI 2 1 0 0
0 2
LSQ_Head LSQ_Tail
# LI 11 02
D$ 1 2 2 9
# LI 1 1 1 11 2 1 2 1
# LI 2 01 6 02
# LI 01 11 02 02
LSQ y4 1 y5 2 Result Bus
Alkar 2007
RUU
ELE 749 L10&11 SS Model.26
Memory Location Data Dependencies
We can have true, anti- and output dependencies on memory locations as well
Memory storage conflicts are infrequent since memory locations are not reused in the same way that registers are and since the number of true dependencies is small (loads and stores are less than 30% of the instruction mix)
Can improve performance if loads are allowed to bypass previous stores (load bypassing) but can do so only if the memory addresses of all previous stores dispatched to the RUU are known since must first check for load data dependency on previous uncommitted stores
Facilitated by having a common load and store queue
Stores are not allowed to bypass previous loads, so there are no antidependencies between loads and stores Stores are committed to the D$ in program order, so there can be no output dependencies between stores
ELE 749 L10&11 SS Model.27 Alkar 2007
Loads from Memory
When a loads address becomes known, the address is compared (associatively) to see if it matches an entry already in the LSQ (i.e., if there is a pending operation to the same memory address)
If the match in the LSQ is for a load, the current load does not need to be issued (or executed) since the matching pending load will load in the data If the match in the LSQ is for a store, the current load does not need to be issued (or executed) since the matching pending store can directly supply the destination Content for the current load
If there is no match, the load is issued to the LSQ and executed when the D$ is next available When the RUU# of the load instr appears on the Result Bus (along with the memory data), the load completes by updating the RUU and releasing the LSQ entry (the RUU entry is released on load commit)
ELE 749 L10&11 SS Model.28 Alkar 2007
Load Bypassing with Forwarding
Load forwarding when a load is satisfied directly from the LSQ
The most recent matching LSQ data entry is supplied, since the LSQ may have more than one matching entry
Speedup
Load bypassing gives 19% speedup improvement with a 4-way decoder Load forwarding gives an additional 4% speedup improvement
ELE 749 L10&11 SS Model.29
3
From Johnson, 1992
2
1
Low HM High
0 total order load byp load fwd
Alkar 2007
Stores to Memory
When a stores address (and the store data) becomes known, the address is compared (associatively) to see if it matches an entry already in the LSQ (i.e., if there is a pending operation to the same memory address)
If the match in the LSQ is for a load, the current store is issued to the LSQ If the match in the LSQ is for a store, the current store is issued to the LSQ with an incremented LI If there is no match, the store is dispatched to the LSQ
Stores are held in the LSQ until the store is ready to commit (i.e., until its partner instr reaches the RUU_Head) at which time the store is executed (i.e., the data and address are sent to the D$) and the RUU and LSQ entries are released
ELE 749 L10&11 SS Model.30 Alkar 2007
SimpleScalar Structure
sim-outorder: supports out-of-order issue and execution (with in-order commit) with a Register Update Unit (RUU)
Uses a RUU for register renaming and to hold the results of pending instructions. The RUU (aka reorder buffer (ROB)) retires completed instructions in program order to the RegFile Uses a LSQ for store instructions not ready to commit and load instructions waiting for access to the D$ Loads are satisfied by either the memory or by an earlier store value residing in the LSQ if their addresses match
- Loads are issued to the memory system only when addresses of all previous loads and stores are known
ELE 749 L10&11 SS Model.31
Alkar 2007
Simulated SimpleScalar Pipeline
ruu_fetch(): fetches instrs from one I$ line, puts them in the fetch queue, probes the cache line predictor to determine the next I$ line to access in the next cycle
- fetch:ifqsize<size>: fetch width (default is 4) - fetch:speed<ratio>: ratio of the front end speed to the execution core (<ratio> times as many instructions fetched as decoded per cycle) - fetch:mplat<cycles>: branch misprediction latency (default is 3)
ruu_dispatch(): decodes instrs in the fetch queue, puts them in the dispatch (scheduler) queue, enters and links instrs into the RUU and the LSQ, splits memory access instructions into two separate instrs (one to compute the effective addr and one to access the memory), notes branch mispredictions
- decode:width<insts>: decode width (default is 4)
Alkar 2007
ELE 749 L10&11 SS Model.32
SimpleScalar Pipeline, cont
ruu_issue()and lsq_refresh(): locates and marks the instrs ready to be issued by tracking register and memory dependencies, ready loads issued to D$ unless there are earlier stores in LSQ with unresolved addrs, forwards store values with matching addr to ready loads
- issue:width<insts>: maximum issue width (default is 4) - ruu:size<insts>: RUU capacity in instrs (default is 16, min is 2) - lsq:size<insts>: LSQ capacity in instrs (default is 8, min is 2)
and handles instrs execution collects all the ready instrs from the scheduler queue (up to the issue width), check on FU availability, checks on access port availability, schedules writeback events based on FU latency (hardcoded in fu_config[])
- res:ialu | imult | memport | fpalu | fpmult<num>: number of FUs (default is 4 | 1 | 2 | 4 | 1)
ELE 749 L10&11 SS Model.33 Alkar 2007
SimpleScalar Pipeline, cont
ruu_writeback(): determines completed instrs, does data forwarding to dependent waiting instrs, detects branch misprediction and on misprediction rolls the machine state back to the checkpoint and discards erroneously issued instructions ruu_commit(): in-order commits results for instrs (values copied from RUU to RegFile or LSQ to D$), RUU/LSQ entries for committed instrs freed; keeps retiring instructions at the head of RUU that are ready to commit until the head instr is one that is not ready
ELE 749 L10&11 SS Model.34
Alkar 2007
In Order FETCH Fetch multiple instructions
ELE 749 L10&11 SS Model.35
SS Pipeline
DECODE & DISPATCH
In Order Out of Order In Order
Decode instructions 1st RUU function (RUU allocation, src operand copying (or Tag field set), dst Tag field set)
ISSUE & EXECUTE WRITE BACK RESULT COMMIT 4th RUU function (for instr at RUU_Head (if executed), write dst Contents to RegFile) 2nd RUU function (copy Result Bus data to matching srcs and to RUU dst entry)
3nd RUU function (when both src operands Ready and FU free, schedule Result Bus and issue instr for execution)
Alkar 2007
Our SS Model Performance
Out of order issue has consistently the best performance for the benchmark programs
3
From Johnson, 1992
Speedup
2
1
IOC OOC OOI
0
cc om do du gn c uc he ss irs im li n pa ck si m pl e tr of f tw ol f 5d i ff
ELE 749 L10&11 SS Model.36
Alkar 2007
Next Lecture and Reminders
Thank you!
ELE 749 L10&11 SS Model.37
Alkar 2007