INTRODUCTION
GENERAL CSCI 5451 Spring 2007 Introduction to Parallel Computing
Why parallel computing? A short history Levels of parallelism
Class time : 8:15am 9:30 MW Room Instructor URL : EE/CSci 3-111 : Yousef Saad : www-users.itlabs.umn.edu/classes/Spring-2007/csci5451/
A little background: algorithms, complexity for a simple example. Obstacles to efficiency of parallel programs Types of parallel computer organizations January 16, 2007
Streaming video : http://www.unite.umn.edu/streaming-video/index.html
1 2
Parallel computing Two important points: 1. The demand for computational speed is always in creasing. The trend is accelerating with a number of new developments: demand in biology, genomics, as well as in physics/chemistry. 2. It was realized around the mid 1970s that gains in raw speed (clock cycle) were becoming harder to achieve. From 1950 to the mid 70s: speed gained a factor of 100,000 (!) 3 orders of magnitude due to clock cycle. The other 2 to architecture and design. A factor of about 10 every 5 years.
In 1975 CRAY (a Minnesota company!) unveiled its first commercial supercomputer. Clock cycle: 12.5 ns. (i.e., freq. of 80MHz.) Today: Pentium IV-HT chips available with 3.8 GHz. A gain of a factor of 47.5 in 32 years or, 13 % per year. Moore's law says that speed would double every 18 months [1975] In 8 years, the speed of my laptop increased from 330 MHz to 1.8Ghz or a gain of 23% per year. [stalled from the last time I taught this course in 2003!] Suggested hw: visit the web site of the top 500 fastest computers: www.top500.org
4
3 4
Csci 5451 January 16, 2007
A few computationally intensive applications Weather forecasting - Partial differential equations Time dependent problems involving several unknowns per mesh point leads to very large nonlinear models. Example: Assume a region of 10, 000km 5, 000km and a height of 10km. If cells of 1 cubic km are used: what is the total number of cells? Challenge: complete the calculation for tomorrow's weather before tomorrow. Device/ circuit simulation. A chip such as the Pentium IV has over 100 Million gates. Numerical simulation of flow of electrons and holes is exceedingly complex. Simplifications are common to reduce size.
5
5
Reservoir simulation (petroleum engineering) - Typical reservoir size: a few tens of kms, a few tens of meters deep. Equations involve Oil saturation, Water saturation, and pressure. Simulations are over long periods of time. Oil companies were the first commercial buyers of supercomputers. Example: Calculate the system size for the case of a 10km 10km 500m reservoir, when a mesh-size of 5m is used (one node every 5m in each direction). Electronic structures calculations / quantum mechanics Chemists and physicists doing electronic calculations are the biggest users of supercomputers.
Csci 5451 January 16, 2007
6
6
Csci 5451 January 16, 2007
Fundamentals of parallel computing We take a simple example of calculating the sum of n numbers
n
Divide and conquer: Split the sum in p subsums. As sume n = p m. ALGORITHM : 2 Parallel Sum of n numbers
s=
i=1
xi
Sum is traditionally computed as : ALGORITHM : 1 Sum of n numbers
s = 0; for (i = 0; i < n; i + +) s = s + x(i) ; (or s+ = x(i)) n - 1 "sequential" operations How can we exploit parallelism?
7
7
for (j = 0; j < p; j + +) { // Parallel Loop tmp(j) = 0; for (i = j m; i < (j + 1) m; i + +) tmp(j) = tmp(j) + x(i); } s = 0; for (j = 0; j < p; j + +) // Sequential loop s = s + tmp(j);
Csci 5451 January 16, 2007
8
8
Csci 5451 January 16, 2007
+ + + +
Assume n = 2k and n/2 arithmetic units ("processors") are available. Then, can apply the idea recursively. Divide into 2 subsums each subsum is divided in two subsums etc.. Yields the cascade sum: ALGORITHM : 3 Cascade Sum of n numbers
Number of operations = the same p (m - 1) + p - 1 = n - 1. Each subsum is done independently.
// Sequential loop: for (stride = 1; stride < n; stride = 2) { // Parallel Loop: for (j = 0; j + stride < n; j + = 2 stride) x(j) = x(j) + x(j + stride); } Final result in x(0). Number of steps needed: log2 n. Can be generalized to case where n not a power of 2.
9
9
Csci 5451 January 16, 2007
10
10
Csci 5451 January 16, 2007
Issues: Is it always worth it?
x0 x0 x0 x2 x4 x4 x6
Example: Assume 64 students in this class. Can compute the sum of 128 numbers by cascade algorithm.
Stride = 4 Stride = 2 Stride = 1
x7
Questions: (1) (2) How to organize the calculations? What are the other times involved (i.e., other than
x0
x1
x2
x3 x4
x5
x6
the times needed for adding)? (3) (4) Will it be cost-effective (time-wise) Assume now that we need to add 128 matrices of
A sum of n numbers can be computed in Order log(n) time units where a time unit is the time to perform an add. Similarly for the product of n numbers.
size 10 10 each - by hand. Can the cascade algorithm be used? What has changed?
12
12
11
11
Csci 5451 January 16, 2007
Csci 5451 January 16, 2007
Levels of parallelism Five different types of parallelism are commonly exploited: 1. At job level, i.e.; between different running jobs. 2. At `Macrotask' level: execution of different parallel sections of a given program; (often termed "coarse-grain" parallelism) 3. `Microtask' level; for example parallelism related to different steps of a loop; 4. Data-parallelism; Same operation performed on similar data sets (e.g. adding two matrices, two vectors). Hardware support provided for such operations. 5. At the arithmetic level (pipelining, multiple functional units, etc..)
13
13
There is a common distinction between Fine grain parallelism concerns the arithmetic operations or the loop, i.e., levels 3, 4 and 5 in the above list. Coarse grain parallelism : bigger tasks (`macrotasks') are executed independently. Typically coarse-grain parallelism implies that the user defines the macro-tasks and programs the parallelism explicitly. In contrast fine-grain parallelism is self-scheduled or inherent to the language. Example: Fine grain parallelism: the SAXPY loop [BLAS1] c FORTRAN-90 CONSTRUCT: x(1:n) = x(1:n) + alp*d(n1:n1+n-1)
Csci 5451 January 16, 2007
14
Csci 5451 January 16, 2007
14
Example: Coarse grain parallelism: parallel execution of calls to a given subroutine: ParDo 10 i = 1, n_regions call pde_solv(neq,domain,...) continue
Barriers to Parallel Efficiency The example of adding n numbers can give an indication to a few of the potential barriers to efficiency: 1. Data cessors. or other shaking, cost.. movement. Data to be exchanged between proData may have to move though several stages, processors, to reach its destination. Also: handexplicit programming of data exchange, increase
10
Each step solves a Partial Differential Equation on a different subregion.
2. Poor load balancing. Ideally: all processors should perform an equal share of the work all the time. However, processors are often idle waiting data from others. 3. Shynchronization. Some algorithms may require synchronization. This global operation (requires participation of all processors) may be costly.
15
15
Csci 5451 January 16, 2007
16
4. Impact of Architecture. Often, in parallel processing, processors will be competing for resources, such as access to memory, or cache.. 5. Inefficient parallel algorithm. Some sequential algorithms do not easily parallelize. New algorithms will be required which may be inefficient in sequential context.
The two main parallel programming paradigms 1. Shared global memory viewpoint programs will execute in parallel (parallelization with or without help from user) and access a shared address space. [example: Open MP.] 2. Message passing viewpoint codes are programmed to run in parallel. Data exchange is explicitly programmed by user. [example: MPI] Drawbacks with (1) : Need to make sure data being used is the latest one - need to synchronize. Drawbacks with (2) : difficult to program.
17
17
Csci 5451 January 16, 2007
18
Message passing
Shared memory
PE1 PE2 . . . PEp
PARALLEL COMPUTING PLATFORMS
Global Bus
History Parallel computing systems
Shared Global Memory
Current computational paradigms
Message exchange
In the 70's and 80's (1) was very popular ["paralleliz ing compilers"] Then message passing gained ground with libraries such as PVM and MPI.. In this course we will mainly discuss (2).
19
19
Csci 5451 January 16, 2007
20
A brief history of computing Essentially three stages: 1. Mechanization of arithmetic (abacus, Blaise adder Pascal's [1623-1662], various calculators, Leibnitz [1646-1716]). 2. Stored Program Concept [very important] Automatic looms [1801] punched cards.
3. Merging of Mechanized Arithmetic and Stored Programs Milestones: Babbage (17921871) Hollerith (census) IBM (1924) Aiken: Mark 1 computer electromechanical computer. Memory = relays. ENIAC : electronic computer (1946) - Upenn. UNIVAC: first commercial computer ...
22
21 22
Csci 5451 January 16, 2007
The five generations: 1. First: vacuum tubes; 2. Second: Transistors (1959-1965) 3. Third: Integrated circuits (19651970's) 4. Fourth: Very Large Scale Integration. (VLSI); (1975 1990) 5. Five: (current) Ultra Large Scale Integration, Very High Speed Integrated Circuits technology; Massive parallelism; Heteregenous systems.
The Von Newman architecture Central Processing Unit
MEMORY
6 6
Internal bus
Control Unit Arithmetic Logic (ALU)
Unit
* CPU *
-
Peripherals
CU
ALU
Memory Unit (or units) Main memory, cache, ROM, ... Peripherals: External memories, I/O devices, terminals, printers, ...
All components linked via one internal bus. Bottleneck
23
23
Csci 5451 January 16, 2007
24
24
Csci 5451 January 16, 2007
Execution of Von Neuman programs Fetch Next Instruction from memory into instruction registers Fetch Operands from memory into data registers Execute instruction Load result into memory Fetch Next Instruction from memory into instruction registers ...
Why parallelism? Main argument: Harder and harder to increase clock speeds. For sustainable gains in speed, the only alternative is through better software, better utilization of hardware, and parallelism. Parallelism is cost effective: multiplying hardware is cheap; fast components are expensive. Parallelism also helps from memory stand-point (multi plying memory is cheap, building systems with large memories is expensive).
25
25
Csci 5451 January 16, 2007
26
26
Csci 5451 January 16, 2007
Flynn's Taxonomy of parallel computers Flynn's classication distinguishes architectures by the way processors execute their instructions on the data. Four categories : 1. Single Instruction stream Single Data stream (SISD) 2. Single Instruction stream Multiple Data stream (SIMD) 3. Multiple Instruction stream Single Data stream (MISD) 4. Multiple Instruction stream Multiple Data stream (MIMD) SISD Architecture This is the standard von Neuman organization MISD Architecture Same data simultaneously exploited by several processors that execute different operations on it. Little practical interest but can include special purpose computers.
27
SIMD Architecture The same instruction is executed simultaneously on different "data streams". Single control unit dispatches a single stream of instructions. Data Streams to Memory
CU
Processing Units
Data Streams from Memory Includes pipelined vector computers - and a number of other machines (ILLIAC IV in the 60s, the CM2 in early 90s are just 2 examples).
28
MIMD Architecture MIMD machines simultaneously execute different instructions on different data streams. Processing elements have their own control units - but coordinate computational effort with the other processors. MIMD computers further classified in two important subcategories Shared memory models : processors have very little local or `private' memory; they exchange data and cooperate by accessing a global shared memory. Distributed memory models : No shared global address space. Each processor has its own local memory. Interconnections between processors allow exchange of data and control information.
30
29
Typical SIMD (left) and MIMD (right) organizations.
CPU CPU+Control Unit
CPU CPU+Control Unit Global Control Unit
CPU+Control Unit CPU Interconnection Network Interconnection Network
Csci 5451 January 16, 2007
30
Architecture options for exploiting parallelism 1. Multiple functional units , +, .. 2. Pipelining, Superscalar pipelines 3. Vector processing 4. Multiple Vector pipelines 5. Multiprocessing 6. Distributed computing
Multiple Functional units Present in early computers [CDC 6600, IBM 360/91] Advantage: Sharing control units, registers. Can chain operations between functional units. Detecting parallelism requires a `dependence analysis' Exploited in many of the japanese supercomputers + Dependence Analysis tree for / \ (a+b) + (c*d + d*e) / \ / \ / \ / \ + + / \ / \ / \ / \ a b / \ * * / \ /\ c d d e
31
31
Csci 5451 January 16, 2007
32
32
Csci 5451 January 16, 2007
Pipelining Idea of the assembly line. Assume an operation takes s stages to complete. Then we pass the operands through the s stages instead of waiting for all stages to be complete for first two operands. Arithmetic operations are always comprised of several stages. Example, for arithmetic adders we may have: 1 Compare exponents 3 Add fractions; 2 Align fractions accordingly 4 Normalize fraction; Assume the operation must be applied to a stream of data Then: no need to wait for all the stages to be completed on one pair of operands before starting the next one. Can "pipeline" the stream through the 4 stages....