Parallel Data Programming and the M AS PAR
1
Data Parallel Programming and the M AS PAR
P.Y. Wang Department of Computer Science 4A5 George Mason University Fairfax VA 22030-4444 U.S.A.
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
2
A Data Parallel Programming Model
Massive data level parallelism Homogeneous, ne-grain processors Local memory, arithmetic-logic unit Interconnection network for data routing Single control sequence Instructions are sequentially broadcast from a main control unit to all processors PEs are masked in order for effective computation to the carried out Two sets programming instructions Host instruction set (sequential) PE instruction set (data parallel)
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
3
Major Features of High-Level Data Parallel Languages
Data representation issues What are the basic language declarations that are used to declare single and parallel data structures? Parallel instructions How are parallel instructions coded? Processor selection How are processing elements masked out so that they do not participate in the execution of a parallel instruction?
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
4
Looping constructs What constructs in the languages can be used to repeat sections parallel code? Data aggregation and reduction What kinds of data aggregation and reduction operations are supported by data parallel languages? How are these operations coded? Data distribution and broadcasting What are the data distribution facilities provided by these languages? How are they implemented? Data routing What are the data routing facilities provided by these languages?
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
5
Maspar MP-2 Architecture Overview
Consists of a front-end unit and a Data Parallel Unit (DPU) The front-end is a Unix graphics workstation with windowing capability and standard I/O devices The DPU consists of an Array Control Unit (ACU) and a processor element (PE) array
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
6
memory
processor running UNIX OS
ACU
ACUPE bus
PE Array
Router
standard I/O keyboard display Ethernet disk tape
DPU
FrontEnd
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
7
The ACU contains a processor with its own registers as well as data and instruction memory Five 32-bit registers for register variables, 128 KB of data memory, 1 MB RAM that expands to 4 GB of virtual instruction memory Controls the PE array and performs operations on singular data Sends data and instructions to each PE simultaneously
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
8
The PE (processor element) array: Each PE has sixteen 32-bit registers available for register variables Each PE has 16 KB of RAM PEs that are enabled (i.e. active) will execute the instructions on variables that reside on PEs PEs are arranged in matrices size 1 K, 2 K, 4 K, 8 K or 16 K that are either square or such that the number of columns is twice the number of rows (xed or toroidal wrap)
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
9
A PE cluster is a non-overlapping square matrix of 16 PEs in the PE array A 1 K PE array = sixty-four 4 4 PE clusters Clusters are important in global router communications
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
10
Communication types: Between ACU and PE array reect broadcast and reduction Between PE and PE occur in the PE array x-net communications Direct between and PE and any other PE that lies on a a straight line in the N/S/E/W or NE/NW/SE/SW directions router communications Arbitrary permutations of data among the PEs
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
11
MPL-Maspar Parallel Application Language
MPL keywords: all K & R keywords are supported, including asm, but not entry and fortran all, globalor, plural, proc, router, visible, xnet[p,c]{N,S,E,W,NW,NE,SW,SE} plural type: Variable being declared is allocated identically to all PEs Declaration affects all PE s, in contrast to plural statements and functions
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
12
long long type: 64-bit integer data type unsigned or signed can be singular or plural plural types can be char, short, int, long long, long, unsigned, oat, double, enum, <struct-specier>, <union-specier>
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
13
Pointers: Singular pointer to singular data Singular pointer to plural data Plural pointers to plural data A PE can only point to its own memory Communications constructs xnet, router, proc must be used to access memory of some other PE The address of a plural variable is singular
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
14
Examples
int lions; plural int tigers; plural int * bears; int *plural ohMy; plural int k; plural int *plural toto; lions is a singular variable ACU memory tigers is a plural variable in the PEs memory bears is a singular variable in the ACU memory and points to PE memory all values pointed to by bears have the same address
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
15
int *p int lions int a[5]
plural int tigers
plural int b[4]
plural int * plural toto plural int *bears plural int k
ACU Memory
int * plural OhMy
PE Array Memory
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
16
Note: int *plural ohMy; plural int k;
k = *ohMy; *ohMy = k;
The rst statement is well-dened. The last statement is undened when the value of ohMy is the same on two or more PEs.
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
17
register attribute: use with the plural variable advises the compiler to allocate the variable in a PE register cannot take the address of such a variable Plural arrays: array space is replicated across all PEs (we already saw this) Structure and union speciers: If a structure variable is declared it as plural, the entire structure is allocated on the PEs. Otherwise, the entire structure is allocated on the ACU. Structure and union members cannot be explicitly declared as plural
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
18
Example
struct gangster { int legal member; }; struct gangster bonnie; plural struct gangster clyde;
bonnie and bonnie.legal member are singular clyde and cyde.legal member are plural
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
19
struct members may be pointers to singular or plural data: struct { int * sopwith; plural int * fokker; } airplane; plural struct { int * redstone; plural int * viking; } rocket; plural struct farkle { struct farkle * frob; plural struct farkle * gleep; } x; x is a plural variable with members frob and gleep frob points to ACU memory while gleep points to PE memory
c 2004 P.Y. Wang G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
20
Function denitions: plural int fun(i) int i; { ... } This function has a singular argument and produces plural results.
plural oat fun(x) plural oat x; { ... } This function has a plural arguments and produces plural results.
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
21
Front-End DPU Data Transfer
The visible keyword makes the address of the identiers associated with it known (visible) to both the front-end and the DPU Any identiers that you pass by reference using callRequest(), copyIn(), copyOut(), blockIn(), blockOut() must be declared visible. Example: The front-end calling an MPL subroutine Front-End extern mpfun(); callRequest(mpfun,nnn,data); MPL Code visible mpfun() { ... }
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
22
Another Example
Front-End int A[2048]; extern int X; blockOut(A,X, . . . ); blockIn(X,A, . . . );
MPL Code visible extern int A[2048]; visible plural int X; blockIn(A,&X, . . .); blockOut(&X,A, . . .);
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
23
Writing Expressions
The arithmetic operators of K & R C have been extended to operate on plural operands It looks more or less like what you would expect int s1, s2; plural int p1; p1 = s1 + s2; p1 = p1 + s1; p1 = p1 + (plural int) s1; The operation singular object plural value is illegal.
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
24
Array subscripting plural int i, j; int wizard[4]; j = wizard[i];
PE 0 PE 1 PE 2 PE 3 PE 4 PE 5 i j 1 3 0 3 2 0 ACU 6 7 8 9 wizard[3] wizard[0]
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
25
plural int i, j; int wizard[4]; wizard[i] = j;
PE 0 PE 1 PE 2 PE 3 PE 4 PE 5 i j 1 0 3 1 0 2 3 3 2 4 0 5 ACU
undef
wizard[0]
undef
wizard[3]
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
26
Dot and arrow: if a struct or union is plural, then dot ( . ) returns plural if the left operand is a pointer to a plural struct or union, then arrow ( ) returns plural
AND, OR (||, &&): the result is plural if either operand is plural
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
27
Conditional expressions if any of the operands are plural, then the PEs for which the condition is true dene the active set
globalor performs a bitwise OR operation on all values of a plural expression in the active set and returns a singular result
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
28
Processing Element Addressing
MPL Predened variables are used to give the size and shape of the PE array. These permit each PE to know its relative position (self-address).
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
29
x 0 0 1 y 2 3
PE PE PE PE
1
PE PE PE PE
2
PE
3
PE PE PE
PE PE
PE PE
PE
PE
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
30
Size (singular): nproc = number of PEs nxproc = number of PEs in the x-direction = nyproc number of PEs in the y-direction Representation: lnproc = number of bits needed to represent nproc lnxproc = number of bits needed to represent nxproc lnyproc = number of bits needed to represent nyproc Self-addresses (plural): iproc = one-dimensional self-address: 0 . . . nproc-1 (row-wise) ixproc = x self-address: 0 . . . nxproc-1 iyproc = y self-address: 0 . . . nyproc-1
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
31
Self-address Example
if ( ixproc = = 2) { /* active set */ ... }
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
32
Processing Element (PE) Communications
router uses linear view of the PE array proc: to access the value on a single PE int x; plural int pixel; x = proc[4].pixel; x = proc[0][63].pixel; xnet: PE to PE communications same direction and distance with toroidal wrap
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
33
XNET
Example
plural int i, j; ... i = xnetN[1].j; i = xnetN[1].j + xnetS[2].k; i = *xnetN[1].p; xnetN[1].j = i; xnetN[1].j = xnetSW[1].i; Remember: xnet on left hand side send right hand side fetch
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
34
XNETP
xnetp like xnet except it is faster and pipelines its values through intermediate PEs Any PE along the pipeline that is active causes the pipelining to stop Example: external plural int iyproc; plural int coeff; if (iyproc == 5) { coeff = coeff + xnetpS[8].coeff; } row 13 is added to row 5
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
35
XNETC
xnetc like xnetp but it leaves copies in intermediate PEs along the way Example: plural oat coeff; if (ixproc == 6) { xnetcE[3].coeff = coeff; } column 6 is copied into columns 7, 8, and 9
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
36
XNET
for Non-regular Topologies
plural int linear_shift (x) plural int x; { x = iproc; xnetE[1].x = x; if (ixproc == 0) xnetS[1].x. = x; return x; }
0 4 8 12
1 5 9 13
2 6 10 14
3 7 11 15
3 7 11 15
0 4 8 12
1 5 9 13
2 6 10 14
15 3 7 11
0 4 8 12
1 5 9 13
2 6 10 14
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
37
ROUTER
Communication Statement
Each active PE will participate in general communication If router construct is on the left hand side of an assignment, then you are doing a send from the active set to the connected set If router construct is on the right hand side of an assignment, then you are doing a fetch from the connected set to the active set
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
38
ROUTER
plural int i, j, k; iproc active i j k router[i].j = k; j k = router[i].j; k 0 1 2 17 19
Examples
1 0 1 23 29
2 0 3 31 37
3 1 0 41 43
...
43 ___
23
___
41
31 ___
29
37
___
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
39
MPL Special Parallel Primitives
Purpose: to compare/order PE instances of a plural variable (plural) i = enumerate (); depends on the active set unlike iproc, ixproc, iyproc which are static i = rankN (plural any type); integer order of argument on each PE returned into i of that PE N denotes data type (8, 8u, 16, 16u, 32, 32u, 64, 64u, f, d) psortN(plural any type); arranges the plural argument, putting the smallest value on PE with lowest self-address after doing psort(x), a rank(x) returns the list of integers matching enumerate
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
40
A Parallel Primitives Example
PE active 0 1 1 0 2 0 3 1 4 0 5 1
i = enumerate(); i ___ 0 ___ ___
y = rank16(x); x y 4 ___ 5 ___ 8 ___ 6 ___ 1 0 3 1
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
41
MPL Reduction Functions
Denition: reduction is the singular result of performing an operation cumulatively on all elements of an array M AS PAR implementation details 1. size of array is determined by the active set (traversed linearly) 2. result is initialized to the operations identity element (but always returns 0 if there are no active PEs)
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
42
Conventions for use: #include < reduce.h > plural double reduceMuld(); double y; plural double x; ... y = reduceMuld(x); /* needed for type defs */ /* dened in reduce.h */
Sufxes are 8, 8u, 16, 16u, 32, 32u, 64, 64u, f, d Reduction functions are Mul, Add, And, Or, Max, Min
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
43
MPL SENDWITH Functions
Motivation: with router, several PEs can send to the same destination When multiple PEs send data to the ACU, we can reduce them to a single value (using globalOR, reduceADD, etc). If a PE receives from many senders, the type of reduction should be specied by the programmer sendwithAdd8 is a hybridization of reduceAdd8 and the router construct Each receiving PE adds together all the data elements it gets during the sendwithAdd8 procedure
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
44
Note: if a PE is in the active set but receives no messages, the value returned on that PE is 0, regardless of the type of operation specied as part of the with in sendwith... Same reductions of Mul, Add, And, Or, Max, Min Same sufxes of 8, 8u, 16, 16u, 32, 32u, 64, 64u, f, d
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
45
Examples
Sum from all active PEs into ACU receiver: sr = reduceAdd (pv); Here pv is plural and sr is singular. Sum from all active PEs to the receivers who are active PEs matching some senders destination: pr = sendwithAdd (pv,dest); Here pv, dest, result are plural.
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
46
MPL SCAN Functions
scan performs an operation cumulatively on a list of PEs segmented scan performs an operation cumulatively on all PEs within a segment Within each segment, it is similar to reduction But unlike reduceAddd, for example, scanAddd returns a plural double result and intermediate sums are saved on intermediate PEs Currently supported functions are scanMul, scanAdd, scanAnd, scanOr, scanMax, scanMin sufxes are 8, 8u, 16, 16u, 32, 32u, 64, 64u, f, d
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
47
A segment is a series of PEs with a plural char variable containing 0, 0, 0, . . ., 0, 1 respectively Cumulative result is returned in each PE (by ascending iproc) Segment is conned to PEs in the active set For an unsegmented scan across all active PEs, set all seg = 0; Segment wraps from highest active PE to lowest active PE if (seg==1) on at least one active PE and (seg==0) on highest PE
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
48
Two SCAN Examples
PE x seg 0 3 0 1 4 0 2 8 0 3 1 0 4 2 0 5 6 1
i = scanAdd16(x,seg); 3 7 i
15
16
___
___
PE x seg
0 3 0
1 4 0
2 8 1
3 1 0
4 2 0
5 6 1
i = scanAdd16(x,seg); i 3 7 15 1 3 ___
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
49
MPL Statements (Summary)
K & R used for singular statements/expressions The active set is the set of PEs enabled at the time if, switch, while, do, and for become plural when the controlling expressions is plural if statement if (plural expression) statement if (plural expression) statement1 else statement2 Both blocks may be executed by a different set of PEs
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
50
while statement the body is executed by the active PEs at each iteration, the active set is either the same as the previous active set or smaller than the previous active set can contain return(), break, and continue statements, but not goto() do statement: similar to while for statement: its plural if expr2 is plural for (expr1; expr2; expr3) statement NOTE: while, do and for restore to the original active set before execution, when they are nished
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
51
MasPar MPL Sample Program
Find the average of all 8 neighbors using Toroidal connections Non-torioidal connections
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
52
The main.m Code
/* main program for the average example */ #include <stdio.h> #include <stdlib.h> extern plural int avg( plural int src ); extern plural int nowrapAvg( plural int src ); main () { plural int i; plural int avg_value; plural int nowrapAvg_value;
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
53
i = 10*iproc;
/* Give i some data on each PE*/
avg_value = avg( i ); if( iproc == 0 ) p_printf( " The average on processor 0", " with toroidal wrap : %d\n", avg_value ); nowrapAvg_value = nowrapAvg( i ); if( iproc == 0 ) p_printf( " The average on processor 0", " with no toroidal wrap: %d\n", nowrapAvg_value ); exit(0); }
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
54
The average.m Code
#include <mpl.h> #define nowrapXN(d,val) #define nowrapXS(d,val) #define nowrapXW(d,val) #define nowrapXE(d,val)
((iyproc ((iyproc ((ixproc ((ixproc
< d) ? 0 : xnetN[d].val) >= nyproc-d) ? 0 : xnetS[d].val) < d) ? 0 : xnetW[d].val) >= nxproc-d) ? 0 : xnetE[d].val)
/* nowrapAvg() averages each pixel with the values of its 8 neighbors but doesnt wrap around the PE Array. It turns out that only 4 Xnets are necessary since the first line accumulates the NW and NE values into the N neighbor and the SW and SE values into the S neighbor. */ plural int nowrapAvg( plural int src ) { src += nowrapXW(1,src) + nowrapXE(1,src); src += nowrapXN(1,src) + nowrapXS(1,src); return(src/9); }
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY
Data Parallel Programming and the M AS PAR
55
/* avg() averages each pixel with the values of its 8 neighbors. It turns out that only 4 Xnets are necessarY since the first line accumulates the NW and NE values into the N neighbor and the SW and SE values into the S neighbor. */ plural int avg( plural int src ) { src += xnetW[1].src + xnetE[1].src; src += xnetN[1].src + xnetS[1].src; return(src/9); }
c 2004 P.Y. Wang
G EORGE M ASON U NIVERSITY