and Multiprocessors Multithreading
why multiprocessors and/or multithreading?
what workloads do they run well
Readings
H+P chapter 6
6.16.4, 6.76.9
types of MT/MP
Flynn's taxonomy message passing vs. shared memory newer stuff: CMP, SMT
Recent Research Papers
SMT (Simultaneous Multithreading) Multiscalar
interconnection networks why caching shared memory is challenging
cache coherence, synchronization, and consistency
just the tip of the iceberg -- take ECE 259/CPS 221 for more!
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
1
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
2
Threads, Processes, Processors, etc.
some terminology to keep straight process thread processor (we should know what this is!) thread context multithreaded processor multiprocessor many issues are the same for MT and MP will discuss in terms of MPs, but will point out MT diffs
Why Parallel Processing?
multiple processors working together, why? performance: break physical limits of uniprocessing
ILP (branch prediction, RAW dependences, etc.)
cost and cost effectiveness
build big systems from commodity parts (ordinary uniprocessors) enough transistors to make MT processors enough transistors to make MP on a single chip (CMP)!
other
smooth upgrade path (keep adding processors) fault tolerance (one processor fails, still have P-1 working) power-effective (remember Mudge's paper on power)
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
3
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
4
Chip Multiprocessors
trend today: multiprocessors on a single chip (CMPs) can't spend all of the transistors on just one processor
with limited ILP, single processor would not exploit it
Multithreaded Processors
another trend: multithreaded processors processor utilization: IPC / processor width
decreases as processor width increases (~50% on 4 wide) why? cache misses, branch mis-predictions, RAW dependences
e.g., IBM POWER4
1 chip contains: 2 1GHz processors, L2, L3 tags, interconnect can connect 4 chips on 1 MCM to create 8 processor system targets threaded server workloads p1 p1 bus control
idea: two (or more) processes (threads) share one pipeline replicate process (thread) state
PC, register file, bpred history, page table pointer, etc.
one copy of stateless (or naturally tagged) structures
caches, functional units, buses, etc.
L2
L3 tags
hardware thread switch must be fast
multiple on-chip contexts no need to load from memory
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
5
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
6
Two Multithreading Paradigms
coarse-grained
in-order processor with short pipeline switch threads on long stalls (e.g., L2 cache misses) instructions from one thread in stage per cycle + threads don't interfere with each other much can't improve utilization on L1 misses, or branch mispredictions e.g., IBM Northstar/Pulsar (2 threads)
Why Parallel Processing Is Hard
in a word: software difficult to parallelize applications
compiler parallelization hard by-hand parallelization maybe harder (very error prone, not fun)
difficult to make parallel applications run fast
communication very expensive (must be aware of it) synchronization very complicated
fine-grained: simultaneous multithreading (SMT)
out-of-order processor with deep pipeline instructions from multiple threads in stage at same time + improves utilization in all scenarios individual thread performances suffer due to interference e.g., Pentium4 = 2 threads, Alpha 21464 (R.I.P.) = 4 threads
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
7
IT'S THE SOFTWARE, STUPID!
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
8
Amdahl's Law Revisited
speedup = 1/ [fracparallel/speedupparallel + 1 fracparallel] example
achieve speedup of 80 using 100 processors 80 = 1 / [fracparallel/100 + 1 fracparallel] fracparallel = 0.9975 only 0.25% work can be serial!
Application Domain 1: Parallel Programs
true parallelism in one job
regular loop structures data usually tightly shared automatic parallelization called "data-level parallelism" can often exploit vectors as well for (i=0;i<1000;i++){ A[i] = B[i]*C[i]; }
good application domains for parallel processing
problems where parallel parts scale faster than serial parts e.g., O(N2) parallel vs. O(N) serial interesting programs require communication between parallel parts problems where computation scales faster than communication
workloads
scientific simulation codes (e.g., FFT, weather, fluid dynamics, etc.) was the dominant market segment of 1015 years ago
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
9
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
10
Parallel Program Example: Matrix Multiply
N P parameters
N = size of matrix (N*N) P = number of processors
Application Domain 2: Parallel Tasks
parallel independent-but-similar tasks
irregular control structures loosely shared data locked at different granularities programmer defines & fine-tunes parallelism cannot exploit vectors
growth functions
computation grows as f(N ) computation per processor grows as data size grows as f(N ) data size per processor grows as f(N2/P) communication grows as f(N2/P1/2) computation/communication = f(N/P1/2)
2
3
f(N3/P)
called "thread-level parallelism" or "throughput-oriented parallelism"
workload
transaction processing, OS, databases, web-servers e.g., assign a thread to handle each request to server dominant MP/MT market segment today (by far)
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
11
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
12
Parallel Task Example: Bank Database
parameters
D = number of accounts P = number of processors in central server N = number of ATMs (parallel transactions)
Taxonomy of Processors
Flynn Taxonomy [1966] not universal, but simple dimensions
instruction streams: single (SI) or multiple (MI) data streams: single (SD) or multiple (MD)
growth functions
computation: f(N) computation per processor: f(N/P) what is communication? have to lock records while changing them communication: f(N) computation/communication: f(1) + but no serial parts!
cross-product
SISD: uniprocessor (been there) SIMD: vectors (skimmed that - refresher on next slide) MISD: no practical examples (won't do that) MIMD: multiprocessors + multithreading (doing it now)
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
13
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
14
SIMD Vector Architectures
different type of ISA that has vector instructions addv $v1, $v2, $v3 # v1 = v2 + v3 vector registers are like V N-bit scalar registers
for example, 128 32-bit values
SIMD vs. MIMD
why are MPs (much) more common than vector processors? programming model flexibility
can simulate vectors with an MP, but not the other way around dominant market segment cannot exploit vectors
can operate on all entries of vector at once many Cray systems were vector architectures useful for "vector code" for i=1 to 128 { sum [i] = addend1 [i] + addend2 [i]; }
cost effectiveness
commodity part: high volume (translation: cheap) component MPs made up of commodity parts (i.e., microprocessors) can match size of MP to your budget can't do this for a vector processor
footnote: vectors are making a comeback
for graphics/multimedia applications (MMX, SSE, Tarantula) NEC's EarthSimulator is an MP of vector processors
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
15
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
16
Taxonomy of Parallel (MIMD) Processors
again, two dimensions
focuses on organization of main memory (shared vs. distributed) m0 m1 m2 m3 long latency p0 interconnect p1 p2 p3
UMA vs. NUMA
UMA: uniform memory access
from p0, same latency to m0 as to m3 + data placement unimportant (software is easier) latency long, gets worse as system grows interconnect contention restricts bandwidth typically used in small multiprocessors only
dimension I: appearance of memory to hardware
Q: is access to all memory uniform in latency? shared (UMA): yes where you put data doesn't matter distributed (NUMA): no where you put data really matters
dimension II: appearance of memory to software
Q: can processors communicate via memory directly? shared (shared memory): yes communicate via loads/stores distributed (message passing): no communicate via messages long latency short latency interconnect m0 m1 m2 m3 p0 p1 p2 p3
NUMA: non-uniform memory accesss
from p0 faster to m0 (local) than m3 (non-local) + low latency to local memory helps performance data placement important (software is harder) + less contention (non-local only) more scalable typically used in larger multiprocessors
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
18
dimensions are orthogonal
e.g., DSM: (physically) distributed (logically) shared memory
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
17
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
Interlude: What Is "Interconnect"?
connects processors/memories to each other
direct: endpoints (i.e., procs, mems) connected directly (e.g., mesh) indirect: endpoints connected via switches/routers (e.g., tree) m0 m1 m2 m3
Interconnect 1: Bus
direct interconnect + cost
f(1) wires p0 p1 p2 p3
interconnect issues
latency: average latency most important (locality optimizations?) bandwidth: per processor (also, bisection bandwidth) cost: # wires, # switches, # ports per switch scalability: how latency, bandwidth, cost grow with # processors (P)
+ latency: f(1)
no neighbor/locality optimization
bandwidth: not scalable at all, f(1/P)
only used in small systems (P <= 8) m0 m1 m2 m3 p0 p1 p2 p3
we're mainly concerned with interconnect topology can have separate interconnects for addresses and data
+ capable of ordered broadcast
incapable of anything else
new: logical buses w/point-to-point links
tree = logical bus, if all messages go to root e.g., Sun UltraEnterprise E10000
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
19
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
20
Interconnect 2: Crossbar Switch
indirect interconnect + latency: f(1)
m0 m1 m2 m3 p0 p1 p2 p3 p0 no locality/neighbor optimizations
Interconnect 3: Multistage Network
indirect interconnect
routing done by address bit decoding k: switch arity (# inputs and outputs per switch) m0 m1 m2 m3 d: number of network stages = logkP
+ bandwidth: f(1) cost
f(2P) wires f(P2) switches 4 wires per switch p1 p2 p3
+ cost
f(d*P/k) switches f(P*d) wires f(k) wires per switch
+ latency: f(d) + bandwidth: f(1) commonly used in large UMA systems
a.k.a. butterfly, banyan, omega
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
21
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
22
Interconnect 4: 2D Torus
direct interconnect +
p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m p/m no dedicated switches latency: f(P1/2) locality/neighbor optimization p/m p/m p/m p/m
Interconnect 5: Hypercube
direct interconnect
k: arity (# nodes per dimension) d: dimension = logkP in figure: P = 16, k = 2, d = 4 p/m p/m p/m p/m
+ bandwidth: f(1), scales with P + cost
f(2P) wires 4 wires per switch
+ latency: f(k/d)
locality/neighbor optimized
+ bandwidth: f((k1)*d)
p/m p/m p/m p/m p/m p/m p/m p/m
cost
f((k1)*d*P) wires f((k1)*d) wires per switch
good scalability widely used
variants: 3D, mesh (no "wraparound")
e.g., Alpha 21364-based MPs
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
good scalability, expensive switches
switch changes as nodes are added
23
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
24
Interconnect Routing
store-and-forward routing
switch buffers entire message before passing it on latency = [(message length / bandwidth) + fixed overhead] * # hops
Avoiding Deadlock in Interconnect
two types of deadlock routing deadlock
circular dependence on buffers
full full full full
wormhole routing
pipeline message through interconnect switch passes message on before completely arrives latency = (message length / bandwidth) (fixed + overhead * # hops) + no buffering needed at switch + latency (relatively) independent of number of intermediate hops
solutions
routing restrictions (turn model) virtual channels
request/response deadlock
circular dependence on messages
A resp to B req A B
req B resp to A
solutions
separate networks virtual networks
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
25
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
26
Shared Memory vs. Message Passing
MIMD dimension II: appearance of address space to software message passing (multicomputers, clusters)
each processor has its own address space (and unique processor #) processors send (receive) messages to (from) each other communication pattern explicit and precise (only way) used for scientific codes (explicit communication patterns) message passing systems: PVM, MPI + simple hardware difficult programming model (in general)
Shared Memory vs. Message Passing
shared memory (multiprocessors)
one shared address space processors use conventional loads/stores to access shared data communication can be complex/dynamic + simpler programming model (compatible with uniprocessors) but with its own nasties (e.g., synchronization) more complex hardware... (we'll see soon) + but more room for hardware optimization
aside: software shared virtual memory (SVM) exists
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
27
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
28
(Not Too) Recent Parallel Systems
machine SPARCcenter SGI Challenge Cray T3D Convex SPP KSR-1 TMC CM-5 Intel Paragon IBM SP-2 communication shared memory shared memory shared memory (nc) shared memory shared memory messages messages messages interconnect bus bus 3D torus X-bar/ring bus/ring fat tree 2-d mesh multistage #cpus <=20 <= 32 64-1024 8-64 32 64-1024 32-2048 32-256 remote latency (us) 1 1 1 2 2-6 10 10-30 30-100
Multiprocessor Industry Trends
shared memory
easier, more dynamic program model (it IS the software, stupid!) can do more to optimize the hardware
small-to-medium size UMA systems (28 processors)
processors + memory + switch on single board (e.g., quad Pentium) same thing on a single chip (e.g., IBM POWER4) commodity part of the future (present?) glueless MP: slap these together and MP just works! e.g., Opteron
we will concentrate on shared memory systems more hardware oriented market is going this way
speaking of which...
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
29
larger NUMA systems built from smaller (N)UMA systems
exploit commodity nature of small systems use commodity interconnect (e.g., gigabit Ethernet, Myrinet) called NUMA clusters
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
30
Caching Shared Memory
three issues
cache coherence synchronization memory consistency model example processor 0 ----------read A
Cache (In)Coherence
most common cause: sharing of writeable data
processor 1 ----------read A write A read A correct value of A is in.. -------------------------memory memory, p0 cache memory, p0 cache, p1 cache p0 cache, memory (if wthru) p1 gets stale value on hit
not completely unrelated to each other not issues for message passing machines
why not?
other causes
process migration (even if jobs are independent) I/O (can be fixed by OS cache flushes)
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
31
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
32
Solutions to Coherence Problem
no caches
not a good solution - caches are important!
Cache Coherence Protocols
absolute coherence
all copies of each block have same data at all times not necessary
make shared-data non-cacheable
+ simplest software solution low performance if a lot of data is shared
what is required is appearance of absolute coherence
temporary incoherence is OK (e.g., write-back cache) as long as all loads get "correct" values
software flush at strategic times: e.g., after critical sections
+ relatively simple low performance if synchronization is frequent
cache coherence protocol: FSM that runs at every cache
and usually a FSM at every memory, too
hardware cache coherence
make memory and caches coherent (consistent) with each other in other words: let memory and other processors see writes invisible to software
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
33
two kinds of protocols: depends on how writes handled
invalidate protocol: invalidate copies in other caches update protocol: update copies in other caches
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
34
Bus-Based Protocols (Snooping)
bus-based cache coherence protocol (snooping)
ALL caches/memories see and react to ALL bus events protocol relies on global visibility of requests (ordered broadcast) owner (either proc or mem) responds to request with data proc proc cache proc cache proc cache BUS memory
Snooping Protocol Events
requests from proc/cache to cache coherence controller
load (Ld) store (St) writeback (WB)
bus events (= cache coherence transactions)
GetShared (GETS) - broadcast request for read-only data GetExclusive (GETX) - broadcast request for read-write data PutExclusive (PUTX) - broadcast request to write data back to mem
cache
coherence transactions on bus can be from self or others we'll assume atomic bus transactions
thus, we have atomic cache coherence transactions
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
35
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
36
Two-State (MI) Invalidate Protocol
two states
invalid: either don't have block or have it but not allowed to use it modified: have block with read-write access
Three-State (MSI) Invalidate Protocol
three states
idea: add new "read-only" state (shared) - allows multiple readers! invalid modified: have block with read-write access shared: have block with read-only access
St/OwnGETX
problem
block can be in only one cache at a time not efficient, especially if data is only being read {Ld,St}/OwnGETX invalid -/OtherGETX WB/OwnPUTX modified
notation "a/b": a = proc request b = coherence transaction blue = upgrade red = downgrade
invalid
-/OtherGETX WB/OwnPUTX
w St /O
modified
ET nG X
S ET G X wn ET /O G Ld er th -/O B/ W
shared
-/O
th
G er
S ET
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
37
2006 by Lebeck, Sorin, Roth, Hill, Wood, Sohi, Smith, Vijaykumar, Lipasti
ECE 252 / CPS 220 Lecture Notes Multiprocessors and Multithreading
38
Scalable Coherence Protocols: Directories
bus-based protocols (i.e., broadcast) are not scalable!
not enough bus b/w for everyone's coherence traffic not enough processor snooping b/w to handle everyone's traffic
Directory Protocol in Action (MI)
Node 1 = requestor (I -> M) proc/mem St/GETX proc/mem Node 2 = home of block
directories: scalable cache coherence for large MPs
each memory entry (cache line) has a bit vector (1 bit per processor) bit vector tracks which processors have cached copies of line send all requests to directory at home memory if no other cached copies, memory is owner and returns data otherwise, memory forwards request to current owner processor + low b/w consumption (communicate only with processors that care) + works with general interconnect (bus not needed) longer latency (3-hop transactions: p0 directory p1 p0)
Data proc/mem
ForwardedGETX (N1)
Node 3 = current owner of blo...