# Lect02 - Parallelprogramming paradigmsandtheir performances...

This preview shows pages 1–6. Sign up to view the full content.

1 Parallel programming paradigms and their performances A high level exploration of the HPC world George Bosilca University of Tennessee, Knoxville Innovative Computer Laboratory Overview • Definition of parallel application • Architectures taxonomy • Laws managing the parallel domain • Models in parallel computation • Examples Formal definition Bernstein { I1 O2 = and I2 O1 = and O1 O2 = } General case: P1… Pn are parallel if and only if each for each pair Pi, Pj we have Pi || Pj. 3 limit to the parallel applications: 1. Data dependencies 2. Flow dependencies 3. Resources dependencies P1 I1 O1 P2 I2 O2

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 Data dependencies I1: A = B + C I2: E = D + A I3: A = F + G I1 I2 I3 Dataflow dependency Anti-dependency Output dependency How to avoid them? Which can be avoided ? Flow dependencies I1: A = B + C I2: if( A ) { I3: D = E + F } I4: G = D + H I1 I2 I3 I4 Flow dependency Dataflow dependency How to avoid ? Resources dependencies I1: A = B + C I2: G = D + H I1 I2 + How to avoid ?
3 Flynn Taxonomy 1 Instruction flow > 1 Instruction flow 1 data stream SISD Von Neumann MISD pipeline > 1 data stream SIMD MIMD •Computers classified by instruction delivery mechanism and data stream •4 characters code: 2 for instruction stream and 2 for data stream Flynn Taxonomy: Analogy • SISD: lost people in the desert • SIMD: rowing • MISD: pipeline in the car construction chain • MIMD: airport facility, several desks working at their own pace, synchronizing via a central database. • First law of parallel applications (1967) • Limit the speedup for all parallel applications Amdahl Law s p ( N a a speedup N p s p s speedup + = + + = 1 1 N = number of processors

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4 Amdahl Law Speedup is bound by 1/a. Amdahl Law • Bad news for parallel applications • 2 interesting facts: – We should limit the sequential part – A parallel computer should be a fast sequential computer to be able to resolve the sequential part quickly • What about increasing the size of the initial problem ? Gustafson Law • Less constraints than the Amdahl law. • In a parallel program the quantity of data to be processed increase, so the sequential part decrease. n P s t / + = n a P * = a s n a s speedup + + = * n speedup a
5 Gustafson Law • The limit of Amdahl Law can be transgressed if the quantity of data to be processed increase. s n n speedup ) 1 ( + Rule stating that if the size of most problems is scaled up sufficiently, then any required efficiency can be achieved on any number of processors. Speedup • Superlinear speedup ? Sub-linear Superlinear Sometimes superlinear speedups can be observed! •Memory/cache effects •More processors typically also provide more memory/cache. •Total computation time decreases due to more page/cache hits. •Search anomalies •Parallel search algorithms.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}