Big The Picture: Where are We Now?
Processor Control Datapath Processor Output Memory Input Input Control Datapath
Output Memory
Multiprocessor multiple processors with a single shared address space Cluster multiple computers (each with their own address space) connected over a local area network (LAN) functioning as a single system
MultiIntro.1
Applications Needing "Supercomputing"
Energy (plasma physics (simulating fusion reactions), geophysical (petroleum) exploration) DoE stockpile stewardship (to ensure the safety and reliability of the nation's stockpile of nuclear weapons) Earth and climate (climate and weather prediction, earthquake, tsunami prediction and mitigation of risks) Transportation (improving vehicles' airflow dynamics, fuel consumption, crashworthiness, noise reduction) Bioinformatics and computational biology (genomics, protein folding, designer drugs) Societal health and safety (pollution reduction, disaster planning, terrorist action detection)
MultiIntro.2
Encountering Amdahl's Law
Speedup due to enhancement E is Exec time w/o E Speedup w/ E = ---------------------Exec time w/ E Suppose that enhancement E accelerates a fraction F (F <1) of the task by a factor S (S>1) and the remainder of the task is unaffected
ExTime w/ E = ExTime w/o E Speedup w/ E =
MultiIntro.3
Encountering Amdahl's Law
Speedup due to enhancement E is Exec time w/o E Speedup w/ E = ---------------------Exec time w/ E Suppose that enhancement E accelerates a fraction F (F <1) of the task by a factor S (S>1) and the remainder of the task is unaffected
ExTime w/ E = ExTime w/o E ((1-F) + F/S) Speedup w/ E = 1 / ((1-F) + F/S)
MultiIntro.4
Examples: Amdahl's Law
Speedup w/ E =
Consider an enhancement which runs 20 times faster but which is only usable 25% of the time.
Speedup w/ E =
What if its usable only 15% of the time?
Speedup w/ E =
Amdahl's Law tells us that to achieve linear speedup with 100 processors, none of the original computation can be scalar! To get a speedup of 99 from 100 processors, the percentage of the original program that could be scalar would have to be 0.01% or less
MultiIntro.5
Examples: Amdahl's Law
Speedup w/ E = 1 / ((1-F) + F/S)
Consider an enhancement which runs 20 times faster but which is only usable 25% of the time.
Speedup w/ E = 1/(.75 + .25/20) = 1.31
What if its usable only 15% of the time?
Speedup w/ E = 1/(.85 + .15/20) = 1.17
Amdahl's Law tells us that to achieve linear speedup with 100 processors, none of the original computation can be scalar! To get a speedup of 99 from 100 processors, the percentage of the original program that could be scalar would have to be 0.01% or less
MultiIntro.6
Supercomputer Style Migration (Top500)
http://www.top500.org/lists/2005/11/
500 400 300 200 100 0
1993994995 996997998999000001 002003004005 1 1 1 1 1 1 2 2 2 2 2 2
Nov data
Clusters Constellations SIMDs MPPs SMPs Uniproc's
Cluster whole computers interconnected using their I/ O bus Constellation a cluster that uses an SMP multiprocessor as the building block
In the last 8 years uniprocessor and SIMDs disappeared while Clusters and Constellations grew from 3% to 80%
MultiIntro.7
Multiprocessor/Clusters Key Questions
Q1 How do they share data? Q2 How do they coordinate? Q3 How scalable is the architecture? How many processors can be supported?
MultiIntro.8
Flynn's Classification Scheme
SISD single instruction, single data stream
q
aka uniprocessor - what we have been talking about all semester single control unit broadcasting operations to multiple datapaths no such machine (although some people put vector machines in this category) aka multiprocessors (SMPs, MPPs, clusters, NOWs)
SIMD single instruction, multiple data streams
q
MISD multiple instruction, single data
q
MIMD multiple instructions, multiple data streams
q
MultiIntro.9
SIMD Processors
PE PE PE PE
PE Control PE
PE
PE
PE
PE
PE
PE
PE
PE
PE
PE
Single control unit Multiple datapaths (processing elements PEs) running in parallel
q
Q1 PEs are interconnected (usually via a mesh or torus) exchange/share and data as directed by the control unit Q2 Each PE performs the same operation on its own local data
q
MultiIntro.10
Example SIMD Machines
Maker Year # PEs # b/ Max PE System PE memory clock BW (MB/ (MB) (MHz) s) 64 1 1 2 2 512 1024 13 5 10 7 25 2,560 2,560 20,480 16,384 23,000
Illiac IV DAP MPP CM-2
UIUC ICL Thinking Machines
1972 64 1980 4,096
Goodyear 1982 16,384 1 1987 65,536 1 1989 16,384 4
MP-1216 MasPar
MultiIntro.11
Multiprocessor Basic Organizations
Processors connected by a single bus Processors connected by a network
# of Proc Communication Message passing 8 to 2048 model Shared NUMA 8 to 256 address UMA 2 to 64 Physical connection Network Bus 8 to 256 2 to 36
MultiIntro.12
Shared Address (Shared Memory) Multi's
Q1 Single address space shared by all the processors Q2 Processors coordinate/communicate through shared variables in memory (via loads and stores)
q
Use of shared data must be coordinated via synchronization primitives (locks)
UMAs (uniform memory access) aka SMP (symmetric multiprocessors)
q
all accesses to main memory take the same amount of time no matter which processor makes the request or which location is requested some main memory accesses are faster than others depending on the processor making the request and which location is requested can scale to larger sizes than UMAs so are potentially higher performance
NUMAs (nonuniform memory access)
q
q
MultiIntro.13
N/UMA Remote Memory Access Times (RMAT)
Year Sun Starfire Cray 3TE HP V Type Max Proc 64 Interconnection RMAT Network (ns) Address buses, 500 data switch 2-way 3D torus 8 x 8 crossbar Fat tree Switched bus Switched bus Switched bus Fat tree 300 1000 500 400 240 275 ???
1996 SMP
1996 NUMA 2048 1998 SMP 32 32 8 64
SGI Origin 3000 1999 NUMA 512 Compaq 1999 SMP AlphaServer GS Sun V880 HP Superdome 9000 2002 SMP 2003 SMP
NASA Columbia 2004 NUMA 10240
MultiIntro.14
Single Bus (Shared Address UMA) Multi's
Processor Processor Processor
Cache
Cache
Cache
Single Bus
Memory
I/O
Caches are used to reduce latency and to lower bus traffic Must provide hardware to ensure that caches and memory are consistent (cache coherency Must provide a hardware mechanism to support process synchronization
MultiIntro.15
Summing 100,000 Numbers on 100 Processors
Processors start by running a loop that sums their subset of vector A numbers (vectors A and sum are shared variables, Pn is the processor's number, i is a private varia...