292 CS Introduction to Parallel Computing
Spring 2008
Announcements
Read Sections 9.3-9.4 Programming assignment #2
Two versions of matrix multiplication using OpenMP Due Friday, 4/4/08
Exam make-up project description posted to Oak ACCRE is back up Questions??
Chapter 9 Topic Overview
Parallel Sorting: Issues in Sorting on Parallel Computers Sorting Networks Bubble Sort and its Variants Quicksort Bucket and Sample Sort Other Sorting Algorithms
Sorting Networks: Bitonic Sort
The comparator network that transforms an input sequence of 16 unordered numbers into a bitonic sequence.
Sorting Networks: Bitonic Sort
A bitonic merging network for n = 16. The input wires are numbered 0,1,..., 0,1,..., n - 1, and the binary representation of these numbers is shown. shown. Each column of comparators is drawn separately; the entire figure represents a BM[16] bitonic merging network. The network takes a bitonic sequence and outputs it in sorted order.
Mapping Bitonic Sort to Hypercubes
Consider the case of one item per processor. The question becomes one of how the wires in the bitonic network should be mapped to the hypercube interconnect. What can we say about the pairs of wires that are the inputs to the various comparators? Note from our earlier examples that the compare-exchange operation is performed between two wires only if their labels differ in exactly one bit! This implies a direct mapping of wires to processors. All communication is nearest neighbor!
Mapping Bitonic Sort to Hypercubes
Communication during the last stage of bitonic sort. Each wire is mapped to a hypercube process; each connection represents a compare-exchange between processes.
Mapping Bitonic Sort to Hypercubes
Communication characteristics of bitonic sort on a hypercube. During each stage of the algorithm, processes communicate along the dimensions shown.
Mapping Bitonic Sort to Hypercubes
Parallel formulation of bitonic sort on a hypercube with n = 2d processes.
Mapping Bitonic Sort to Hypercubes
During each step of the algorithm, every process performs a compare-exchange operation (single nearest neighbor communication of one word). Since each step takes (1) time, the parallel time is Tp = (log2n) This algorithm is cost optimal w.r.t. its serial counterpart, but not w.r.t. the best sorting algorithm.
Mapping Bitonic Sort to Meshes
The connectivity of a mesh is lower than that of a hypercube, so we must expect some overhead in this mapping. We will not cover this mapping in class in the interest of time. See the text for details. But for completeness sake, the parallel runtime on a mesh is:
Which is not cost optimal.
Block of Elements Per Processor
Each process is assigned a block of n/p elements. The first step is a local sort of the local block. Each subsequent compare-exchange operation is replaced by a compare-split operation. (see last lecture) We can effectively view the bitonic network as having (1 + log p)(log p)/2 or (log2p) steps.
Block of Elements Per Processor: Hypercube
Initially the processes sort their n/p elements (using merge sort) in time ((n/p)log(n/p)) and then perform (log2p) compare-split steps. The parallel run time of this formulation is
Comparing to an optimal sort, the algorithm can efficiently use up to p = (2 log nprocesses. ) The isoefficiency function due to both communication and extra work is (plog plog2p) -- thus this strategy does does not scale well.
Bubble Sort and its Variants
The sequential bubble sort algorithm compares and exchanges adjacent elements in the sequence to be sorted:
Sequential bubble sort algorithm.
Bubble Sort and its Variants
The complexity of bubble sort is (n2). Bubble sort is difficult to parallelize since the algorithm has no concurrency. A simple variant, though, uncovers the concurrency.
Odd-Even Transposition
Sequential transposition odd-even sort algorithm.
Odd-Even Transposition
Sorting n = 8 elements, using the odd-even transposition sort algorithm. During each phase, n = 8 elements are compared.
Odd-Even Transposition
After n phases of odd-even exchanges, the sequence is sorted. Each phase of the algorithm (either odd or even) requires (n) comparisons. Serial complexity is (n2).
Parallel Odd-Even Transposition
Consider the one item per processor case. There are n iterations; in each iteration, each processor does one compare-exchange which can all be done in parallel. The parallel run time of this formulation is (n). This is cost optimal with respect to the base serial algorithm but not the optimal serial algorithm.
Parallel Odd-Even Transposition
Parallel formulation of odd-even transposition.
Parallel Odd-Even Transposition
Consider a block of n/p elements per processor. The first step is a local sort. In each subsequent step, the compare exchange operation is replaced by the compare split operation. There are p phases with each phase performing (n/p) compares and (n/p) communication. The parallel run time of the formulation is
Parallel Odd-Even Transposition
The parallel formulation is cost-optimal for p = O(log n).
The isoefficiency function of this parallel formulation is (p2p).
Exponential, thus poor scalability
Quicksort
Quicksort is one of the most common sorting algorithms for sequential computers because of its simplicity, low overhead, and optimal average complexity. Quicksort selects one of the entries in the sequence to be the pivot and divides the sequence into two - one with all elements less than the pivot and other greater. The process is recursively applied to each of the sublists.
Quicksort
The sequential quicksort algorithm.
Quicksort
Example of the quicksort algorithm sorting a sequence of size n = 8.
Quicksort
The performance of quicksort depends critically on the quality of the pivot. In the best case, the pivot divides the list in such a way that the larger of the two lists does not have more than n elements (for some constant ). In this case, the complexity of quicksort is O(nlog n).
Parallelizing Quicksort
Lets start with recursive decomposition - the list is partitioned by a single process and then each of the subproblems is handled by a different processor. The time for this algorithm is lower-bounded by (n)!
Not cost optimal as the process-time product is (n2).
Can we parallelize the partitioning step - in particular, if we can use n processors to partition a list of length n around a pivot in O(1) time, we have a winner. This is difficult to do on real machines, though.
Parallelizing Quicksort: PRAM Formulation
We assume a CRCW (concurrent read, concurrent write) PRAM with concurrent writes resulting in an arbitrary write succeeding. The formulation works by creating pools of processors. Every processor is assigned to the same pool initially and has one element. Each processor attempts to write its element to a common location (for the pool). Each processor tries to read back the location. If the value read back is greater than the processor's value, it assigns itself to the 'left' pool, else, it assigns itself to the 'right' pool. Each pool performs this operation recursively. Note that the algorithm generates a tree of pivots. The depth of the tree is the expected parallel runtime. The average value is O(log n).
Parallelizing Quicksort: PRAM Formulation
A binary tree generated by the execution of the quicksort algorithm. Each level of the tree represents a different array-partitioning iteration. If pivot selection is optimal, then the height of the tree is (log n), which is also the number of iterations.
Parallelizing Quicksort: PRAM Formulation
The execution of the PRAM algorithm on the array shown in (a).