Data Memory Flow
Prof. Mikko H. Lipasti University of Wisconsin-Madison
Lecture notes based on notes by John P. Shen Updated by Mikko Lipasti
Memory Data Flow
Memory Data Flow
Memory Data Dependences Load Bypassing Load Forwarding Speculative Disambiguation The Memory Bottleneck
Cache Hits and Cache Misses
Memory Data Dependences
Besides branches, long memory latencies are one of the biggest performance challenges today. To preserve sequential (in-order) state in the data caches and external memory (so that recovery from exceptions is possible) stores are performed in order. This takes care of antidependences and output dependences to memory locations.
However, loads can be issued out of order with respect to stores if the out-of-order loads check for data dependences with respect to previous, pending stores. WAW WAR RAW store X load X store X : store X : store X : load X
Memory Data Dependences
"Memory Aliasing" = Two memory references involving the same memory location (collision of two memory addresses). "Memory Disambiguation" = Determining whether two memory references will alias or not (whether there is a dependence or not). Memory Dependency Detection:
Must compute effective addresses of both memory references Effective addresses can depend on run-time data and other instructions Comparison of addresses require much wider comparators
Example code:
(1) (2) (3) (4) (5) (6) (7) STORE ADD LOAD LOAD W X V V
RA W
WA LOAD R
ADD STORE
W
Total Order of Loads and Stores
Keep all loads and stores totally in order with respect to each other. However, loads and stores can execute out of order with respect to other types of instructions. Consequently, stores are held for all previous instructions, and loads are held for stores.
I.e. stores performed at commit point Sufficient to prevent wrong branch path stores since all prior branches now resolved
Illustration of Total Order
Decoder Store v Load v Add Add Load w Load x Store w Cycle 2
Load Load x x Load Load Load Load x x x x
Cycle 1 Cycle 2 Cycle 3 data x Load
Load Load Load x x x
Cycle 1 Load x Load Load/Store x
Load Load Load Load
Reservation x Load Station
x x Load Load
x
Load
x x x x
Load x x Load Load w Load x Store v x Load Address Unit
data x Load
Store w Load x Load v Load x Load x x Load Load w Load x Store v x Load Address Unit
data x Load
Store w Load x Load v Load x Load x x Load Load w Load x Store v x Load Address Unit
Cycle 4 data x Load
Load Load Load x x x
data x Load
Address Unit cache addr cache write data Cycle 5 Store w Load x Load v x Load Load x Load x Load w x Load Address Unit Store v data
Load x Load Load Load Load Load
Store v released Cycle 6
Load Load x x Load Load x x Load Load Load x x x
Cycle7
Load Load Load x x x
Cycle 8
data x
x x x
x
Store w x Load Load v Load x Load x x Load Address Unit Load w
data x Load
Load Load x x
Store w Load x Load v Load x Address Unit Load x
data x Load
Load x
Store w Address Unit Load v
data
ISSUING LOADS AND STORES WITH TOTAL ORDERING
Load Bypassing
Loads can be allowed to bypass stores (if no aliasing). Two separate reservation stations and address generation units are employed for loads and stores. Store addresses still need to be computed before loads can be issued to allow checking for load dependences. If dependence cannot be checked, e.g. store address cannot be determined, then all subsequent loads are held until address is valid (conservative). Stores are kept in ROB until all previous instructions complete; and kept in the store buffer until gaining access to cache port.
Store buffer is "future file" for memory How would you build "history file" for memory?
Decoder Store v Load v Load Reservation Station Address Unit
Illustration of Load Bypassing
Add Add
Load Load
Load w Load x Cycle 1 Store w Cycle 2 Cycle 1 Cycle 2
Load Load Load Load x x x x Load Load Load Load Load x x x x Load Load Load Load x x x x
Cycle 3
Load Load Load Load Load x x x x
x x Load Store x x Load Reservation x Load x Load x x LoadStation Load Load x Load x
Loadx Loadw Address Unit
Storev Load x Address Unit
data x
Loadv Loadx Address Unit
Storew Load x Address Unit Store v
data x
Address Unit Store Buffer
data
cache addr
cache write data Cycle 4 Cycle 5
Load w Cycle 6
Load Load Load Load Load x x x x x Load Load Load Load Load x x x x x
Load Load Load Load Load
x x x x x
Load Load Load Load Load
x x x x x
Load v Address Unit
Load v Address Unit Address Unit Store w Store v data data Address Unit
Address Unit Storew Store v data data Storev v Store released released
Address Unit
Storew Load v
data
Load x
LOAD BYPASSING OF STORES
Load Forwarding
If a subsequent load has a dependence on a store still in the store buffer, it need not wait till the store is issued to the data cache. The load can be directly satisfied from the store buffer if the address is valid and the data is available in the store buffer. Since data is sourced from the store buffer:
Could avoid accessing the cache to reduce power/latency
Decoder Store v Load v Load Reservation Station Address Unit
Illustration of Load Forwarding
Add Add
Load Load Load
Load w Load x Cycle 1 Store w Cycle 2 Cycle 1 x Load x Store
x Load x
Cycle 2
Load Load x x x x Load Load Load Load Load x x x x Load Load x x x x
Cycle 3
Load Load Load Load x x x x
Reservation x x Load x LoadStation Load x
Load x Load x
Loadx Loadw Address Unit
Load Load
Storev Load x Address Unit
data x
Loadv Loadx Address Unit
Load Load
Storew Load x data Load x Address Unit
Address Unit Store Buffer
Storev Load w Cycle 5
Load Load Load x x x x x Load Load Load Load Load x x x x x Load Load Load Load Load x x x x x
data
cache addr
cache write data Cycle 4
Cycle 6
Load Load Load Load Load x x x x x
Load Load Load
x x x x x
Load Load Load Load Load
x x x x x
Load v Address Unit
Load Load
Loadv Address Unit
Load Load
Address Unit Store w Store v data data Storevv Store released released
Address Unit Store w data Store v data
Address Unit
Address Unit
Store w data Store v data
Load x
Load v
Forward Store datav released
LOAD BYPASSING OF STORES WITH FORWARDING
The DAXPY Example
Y(i) = A * X(i) + Y(i)
LD ADDI Loop: LD LD ADDD SD ADDI ADDI SUB BNZ F2, 0(Rx) F4, 0(Ry) F4, F2, F4 F4, 0(Ry) Rx, Rx, #8 Ry, Ry, #8 R20, R4, Rx R20, loop ; load X(i) ; A*X (i) ; load Y(i) ; A*X (i) + Y(i) ; store into Y(i) ; inc. index to X ; inc. index to Y ; compute bound ; check if done SD ADDD M ULTD LD MULTD F2, F0, F2 F0, a R4, Rx, #512 ; last address LD
Total Order
Performance Gains From Weak Ordering
Load Bypassing:
CODE: ST X : : LD Y LD Y ST X
Load Forwarding:
CODE: LD X
Reservation Station
ST X : : LD X
Completion Buffer
Load/S tore Unit
ST X
Store Buffer
Performance gain:
L o a d b y p a s s in g : L o a d f o r w a r d in g :
1 1 % - 1 9 % i n c r e a s e o v e r t o t a l o r d e r in g 1 % - 4 % i n c r e a s e o v e r lo a d b y p a s s in g
Optimizing Load/Store Disambiguation
Non-speculative load/store disambiguation
1. Loads wait for addresses of all prior stores 2. Full address comparison 3. Bypass if no match, forward if match
(1) can limit performance:
cache miss RAW for agen, stalled independent load stalled
load r5,MEM[r3] store r7, MEM[r5] ... load r8, MEM[r9]
Speculative Disambiguation
What if aliases are rare?
1. 2. 3. 4. Loads don't wait for addresses of all prior stores Full address comparison of stores that are ready Bypass if no match, forward if match Check all store addresses when they commit
No matching loads speculation was correct Matching unbypassed load incorrect speculation
Load/Store RS Agen Mem Load Queue Store Queue
Reorder Buffer
5.
Replay starting from incorrect load
Speculative Disambiguation: Load Bypass
i1: st R3, MEM[R8]: ?? i2: ld R9, MEM[R4]: ??
Agen Mem
Load Queue
Store Queue
i2: ld R9, MEM[R4]: x400A
Reorder Buffer
i1: st R3, MEM[R8]: x800A
i1 and i2 issue in program order i2 checks store queue (no match)
Speculative Disambiguation: Load Forward
i1: st R3, MEM[R8]: ?? i2: ld R9, MEM[R4]: ??
Agen Mem
Load Queue
Store Queue
i2: ld R9, MEM[R4]: x800A
Reorder Buffer
i1: st R3, MEM[R8]: x800A
i1 and i2 issue in program order i2 checks store queue (match=>forward)
Speculative Disambiguation: Safe Speculation i1: st R3, MEM[R8]: ld ??
i2: R9, MEM[R4]: ??
Agen Mem
Load Queue
Store Queue
i2: ld R9, MEM[R4]: x400C
Reorder Buffer
i1: st R3, MEM[R8]: x800A
i1 and i2 issue out of program order i1 checks load queue at commit (no match)
Speculative Disambiguation: Violation
i1: st R3, MEM[R8]: ?? i2: ld R9, MEM[R4]: ??
Agen Mem
Load Queue
Store Queue
i2: ld R9, MEM[R4]: x800A
Reorder Buffer
i1: st R3, MEM[R8]: x800A
i1 and i2 issue out of program order i1 checks load queue at commit (match)
i2 marked for replay
If aliases are rare: static prediction
Predict no alias every time
Why even implement forwarding? PowerPC 620 doesn't
Use of Prediction
If aliases are more frequent: dynamic prediction
Use PHT-like history table for loads
If alias predicted: delay load If aliased pair predicted: forward from store to load
More difficult to predict pair [store sets, Alpha 21264]
Pay misprediction penalty rarely
Memory cloaking [Moshovos, Sohi]
Predict load/store pair Directly copy store data register to load target register Reduce data transfer latency to absolute minimum
Pay misprediction penalty rarely
Load/Store Disambiguation Discussion
RISC ISA:
Many registers, most variables allocated to registers Aliases are rare Most important to not delay loads (bypass) Alias predictor may/may not be necessary Few registers, many operands from memory Aliases much more common, forwarding necessary Incorrect load speculation should be avoided If load speculation allowed, predictor probably necessary
CISC ISA:
Address translation:
Can't use virtual address (must use physical) Wait till after TLB lookup is done Or, use subset of untranslated bits (page offset)
Safe for proving inequality (bypassing OK) Not sufficient for showing equality (forwarding not OK)
The Memory Bottleneck
Dispatch Buffer Dispatch Reg. Write Back Reg. File Ren. Reg.
RS's Branch Integer Integer Float.Point Load/ Eff. Addr. Gen.
Store Addr. Translation D-cache Access
Reorder Buff. Complete Store Buff. Retire
Data Cache
Load/Store Processing
For both Loads and Stores:
1. Effective Address Generation:
Must wait on register value Must perform address calculation
1.
Address Translation:
Must access TLB Can potentially induce a page fault (exception)
For Loads: D-cache Access (Read)
Can potentially induce a D-cache miss Check aliasing against store buffer for possible load forwarding If bypassing store, must be flagged as "speculative" load until completion
For Stores: D-cache Access (Write)
When completing must check aliasing against "speculative" loads After completion, wait in store buffer for access to D-cache Can potentially induce a D-cache miss
Easing The Memory Bottleneck
Dispatch Buffer Dispatch Reg. Write Back Reg. File Ren. Reg.
RS's Branch Integer Integer Float.Point Load/ Store Load/ Store
Reorder Buff. Complete Store Buff. Retire Data Cache
Missed loads
Memory Bottleneck Techniques
Dynamic Hardware (Microarchitecture):
Use Multiple Load/Store Units (need multiported D-cache) Use More Advanced Caches (victim cache, stream buffer) Use Hardware Prefetching (need load history and stride detection) Use Non-blocking D-cache (need missed-load buffers/MSHRs) Large instruction window (memory-level parallelism)
Static Software (Code Transformation):
Insert Prefetch or Cache-Touch Instructions (mask miss penalty) Array Blocking Based on Cache Organization (minimize misses) Reduce Unnecessary Load/Store Instructions (redundant loads)
Caches and Performance
Caches
Enable design for common case: cache hit
Cycle time, pipeline organization Recovery policy
Uncommon case: cache miss
Fetch from next level
Apply recursively if multiple levels
What to do in the meantime?
What is performance impact? Various optimizations are possible
Performance Impact
Cache hit latency
Included in "pipeline" portion of CPI
E.g. IBM study: 1.15 CPI with 100% cache hits
Typically 1-3 cycles for L1 cache
Intel/HP McKinley: 1 cycle
Heroic array design No address generation: load r1, (r2)
IBM Power4: 3 cycles
Address generation Array access Word select and align Register file write (no bypass)
Cache Hit continued
Cycle stealing common
Address generation < cycle Array access > cycle Clean, FSD cycle boundaries violated
AGEN CACHE
AGEN
CACHE
Speculation rampant
"Predict" cache hit Don't wait for (full) tag check Consume fetched word in pipeline Recover/flush when miss is detected
Reportedly 7 (!) cycles later in Pentium 4
Cache Hits and Performance
Cache hit latency determined by:
Cache organization
Associativity
Parallel tag checks expensive, slow Way select slow (fan-in, wires)
Block size
Word select may be slow (fan-in, wires) Word Line
Number of block (sets x associativity)
Wire delay across array "Manhattan distance" = width + height Word line delay: width Bit line delay: height
Bit Line
Array design is an art form
Detailed analog circuit/wire delay modeling
Cache Misses and Performance
Mis...