310 CS Exam 3 Review
Furman Haddix Ph.D. Assistant Professor Minnesota State University, Mankato Spring 2008
Dynamic Programming Objectives
What is Dynamic Programming Fibonacci algorithm top-down, bottom-up, memoized Factory Example Matrix Multiplication Example Memoization Example Text, Chapter 15.1-15.3
Dynamic Programming
Dynamic programming is a methodology rather than an algorithm Like Divide and Conquer, dynamic programming is a means of solving problems efficiently and effective Dynamic Programming is generally concerned with optimization problems Dynamic Programming applications have two characteristics: Overlapping subproblems: subproblem a is used in solving both subproblem b and subproblem c. A recursive solution will contain many instances of the same subproblem. The (optimal) solution contains (optimal) solutions to a subset of subproblems BEWARE the fallacy of suboptimization: Not all problems can be decomposed into subproblems for which optimal subproblem solutions are parts of the optimal problem solution Where applicable, dynamic programming may provide a more efficient solution than divide and conquer because it avoids repetitive subproblem solution
An Approach to Dynamic Programming
1. Characterize the structure of an optimal solution (Return the nth Fibonacci number) 2. Recursively define the value of an optimal solution (Recursive definition of Fibonacci series) 3. Compute the value of an optimal solution in bottom-up fashion (Convert top-down solution to bottom-up solution) 4. Construct an optimal solution from computed information (Solution of subproblems is solution to problem) Although Fibonacci problem is not seeking an optimal solution it has many of the characteristics of the class of problem for which dynamic programming provides good solutions.
Factory Production Line Example
Factory has two production lines composed of multiple stations Each station performs the same function Production lines were established at different times with different technologies Over time, some stations have been replaced For each station, the faster production method may be on a different line Because of different technologies employed, production times differ at each station In general, transfer times between stations of the same line are insignificant; however, transfer times between stations of different lines are not insignificant. Also, the entry times and exit times are different for the two production lines because of their physical locations.
Factory Production Line Example
Mathematically, we can represent the production time on a line i, as follows: Pi = Ei + Sij + Xi where E refers to entry time, S is station time, X is exit time, and j is one of n stations On occasion, it may be important to have a production run that is the fastest possible. The faster of the two stations for each production stage may be on different lines; however, transfer times, as well as entry and exit times, must also be taken into consideration. If there are k production lines and n production stages, the possible permutations of paths through production is kn, or 26, if there are 2 lines and 6 stations on each line.
Dynamic Programming Analysis of Factory Example
1. Characterize the structure of an optimal solution:
Sum of minimum station processing times including entry, transfer, and exit times. Optimal problem solution contains the optimal subproblem solution since if it did not, it could be reduced by including the optimal subproblem solution. Optimal problem solution is optimal solution through an exit f() = min(f1(n) + X1, f2(n) + X2) Optimal through an exit solution includes optimal solution to the final stage. Optimal final stage solution includes the optimal solution to the penultimate stage. ...
Dynamic Programming Analysis of Factory Example
2. Recursively define the value of an optimal solution: Solution is based on an optimal solutions for subproblems for each line. In this case, our two recursive functions are f1 and f2 Base cases: f1(1) = E1 + S1,1 f2(1) = E2 + S2,1 Reflexive cases (i > 1): f1(i) = min(f1(i-1), f2(i-1) + T2,i-1,1) + S1,i f2(i) = min(f2(i-1), f1(i-1) + T1,i-1,2) + S2,i Recursive function: Ei + Si,k if k = 1 fi(k) = min(fi(k-1), fj(k-1) + Tj,k-1,j) + Si,k if k > 1
3. Compute the value of an optimal solution in bottom-up fashion: f() = min(f1(n) + X1, f2(n) + X2) E1 + S1, E2 + S2 if i=1
f1(i), f2(i)
min(f1(i-1), f2(i-1) + T2,i-1,1) + S1,i, min(f2(i-1), f1(i-1) + T1,i-1,2) + S2,i if i>1 Detailed Pseudocode:
int f() { int n = 6, f1, f2; fi(n, f1, f2); return min(f1 + X[1], f2 + X[2]); } void fi(int n, int &f1, int &f2) { if(n == 1) { f1 = S[1][1] + E[1]; f2 = S[2][1] + E[2]; } else { fi(n-1, f1, f2); int t1 = f1, t2 = f2; f1 = min(t1, t2 + T[2][i-1][1]) + S[1][i]; f2 = min(t2, t1 + T[1][i-1][2]) + S[2][i]; } }
Dynamic Programming Analysis of Factory Example
4.
Construct an optimal solution from computed information: The previous algorithm computes the optimal time but does not capture the stations actually traversed for the optimal solution. The following routines capture the information and print it out. Modified f() and fi() routines which produce the l, l1[], and l2[] data used by printl() are on the following slide. Detailed Pseudocode:
void printl(int n, int l, int l1[], int l2[]) { int i = l; cout << "Optimal Path:" << endl; cout << "Station Line" << endl; for(int j = n; j > 1; j--) { if(i == 1) { i = l1[j]; cout << " " << j << " " << i << endl; } else { i = l2[j]; cout << " " << j << " " << i << endl; } }
Analysis of Factory Example
Factory Example Conclusions
Nave top-down recursion is O(2n) Dynamic programming (bottom-up) solution is O(n)
An Approach to Dynamic Programming
1. Characterize the structure of an optimal solution
f() = min(f1(n) + X1, f2(n) + X2)
Ei + Si,k min(fi(k-1), fj(k-1) + Tj,k-1,j) + Si,k if k > 1 if k = 1
2. Recursively define the value of an optimal solution
fi(k) =
3. Compute the value of an optimal solution in bottom-up fashion
Convert top-down solution to bottom-up solution
4. Construct an optimal solution from computed information
Modified f() and fi() routines which produce the l, l1[], and l2[] data
Matrix-chain Multiplication
1. Characterize the structure of an optimal solution: Multiplication of a chain of matrices Issue: Order of performing multiplications has large effect on number of scalar multiplications High level statement: Determine optimal order of multiplying matrices in order to minimize number of scalar multiplications Example: Three matrix chain (2 x 3, 3 x 4, 4 x 5) Two ways to order multiplications (parenthesize): ((2 x 3, 3 x 4), 4 x 5), (2 x 3, (3 x 4, 4 x 5)) Number of scalar multiplications 2 x 3 x 4 = 24, 2 x 4 x 5 = 40 Total = 64 3 x 4 x 5 = 60, 2 x 3 x 5 = 30 Total = 90
Top-down Approach to Matrix-chain Multiplication
2. Recursively define the value of an optimal solution: Concept: n-1 multiplications must be performed Dimensions of a multiplication are dependent (usually) on order of multiplication Detailed Pseudocode MMC(start, end) if(start = end) return 0 if(start + 1 = end) return mult(start, end) else min = MMC(start+1, end) + mult(start, start+1) for(i = start + 1, i < end, i++) cost = MMC(start, i) + MMC(i+1, end) + mult(i, i+1) if(cost < min) min = cost return min
Matrix-chain Multiplication
3. Compute the value of an optimal solution in bottom-up fashion: Some views of the n-matrix problem n+1 dimensions n matrices n-1 matrix multiplications subproblem is determining the number of scalar multiplications necessary to multiply a chain of matrices of various dimensions
Bottom-Up Approach to Matrix-chain Multiplication
Approach is to build a table which has optimal subproblem solutions for each subprobleme First row of table will have optimal solutions for all chains of length 1 (n-1 chains of 2 matrices each) ith row of table will have optimal solutions for all chains of length i n-1st row of table will have optimal problem solution, where n is number of matrices in chain to be solved Method of attack is to compute row 1 first Within row 1 compute cell 1 first Note that number of computations to compute a cell is i where i is row of cell
Promulgating the results
4. Construct an optimal solution from computed information: Detailed pseudocode augmented to record optimal ordering of multiplications:
for(i = 1, i < n, i++) a[0,n] = 0; Initialize used part of first row of n x n+1 array for(i = 1, i < n, i++) // i is number of matrix multiplications in subproblem row for(j = 1, j n-i, j++) // j is number of cells in subproblem row // subproblem is a chain of matrix multiplications for(k = 1, k i, k++) // k is kth permutation of i permutations for cell // (possible parenthesizations of chain (j, j+i) temp = a[k-1,j]+p(j,j+k,j+i+1) + a[i-k,j+k] if(k = 1 or temp < a[i,j]) a[i, j] = temp b[i,j] = k
a = array of subtree optimums i\j 0 1 2 3 4 5 5 36750 32375 15125 21875 26875 4 28125 18500 27250 10500 11875 18125 15375 24625 3 14875 13000 5375 22875 7125 9500 9375 11375 10000 2 7875 6000 2500 6250 18000 4375 3750 3500 1 15750 2625 750 1000 5000 0 0 0 0 0 0
6
Example Run: n=6
b = array of optimum subtree indices i\j 5 1 3 2 3 4 5
4
3
3
3
2
2 1
2
1
1
1
2
1
1
1
2
1 1
0
printParens(i, j) // i is number of matrices // j is current matrix if(i = j) print `A', j else print `(` printParens(b[i-j,j]+j-1,j) printParens(i, b[i-j,j]+j) print `)' return i\j
5 4
Print Parentheses Routine Detailed pseudocode:
pP(6,1)
pP(3,1) {b[5,1]}
( pP(1,1) {b[2,1]} pP(3,2) pP(2,2) {b[1,2]} pP(3,3)
(
A1 (
A2
A3 ) )
return pP(6,4)
return
b = array of optimum subtree indices
pP(5,4) {b[2,4]}
(
pP(4,4) {b[1,4]} pP(5,5)
(
A4 A5 ) A6 ) )
1
3 3
2
2
3
4
5
return pP(6,6)
3
2
3
1
2
2
1
1 2 return
1
1
1
1
1
1
return ((A1(A2 A3))((A4 A5)A6))
Recall Recursive Solution:
MMC(start, end)
if(start = end) return 0 if(start + 1 = end) return mult(start, end) else min MMC(start+1, end) + mult(start, start+1) for(i = start + 1, i < end, i++) cost MMC(start, i) + MMC(i+1, end) + mult(i, i+1) if(cost < min) lookUpChain(m, start, end) min cost if(m[start,end] 0 return min return m[start,end] if(start = end) Recursive Solution m[start, end] = 0 modified if(start + 1 = end) m[start, end] mult(start, end) for Memoization: else mMMC(m, n) min = lookupChain(m, start+1, end) + mult(start, start+1) for(i=1, i<n, i++) for(i = start + 1, i < end, i++) for(j=1, j<n, j++) cost lookupChain(m, start, i) m[i,j] -1 + lookupChain(m, i+1, end) + mult(i, i+1) r lookUpChain(m,1,n-1) if(cost < m[start, end]) return r m[start, end] cost return m[start, end]
Memoized Matrixchain Multiplication
Homework 13
a) b) c) 15.1-1 (See next slide) 15.2-1 A program for Towers of Hanoi follows. Write code for a memoized version which counts how many unique and redundant recursive calls are made for n = 5. Include the original call to towers(5, 1, 3). Note that you are not changing the flow of the program but capturing values in a three dimensional array[n][i][j]. The trick is to accumulate the first accesses of a cell and separately accumulate the subsequent accesses. The new program is actually not a memoization, but collecting metrics which would measure the benefit of memoization.
void towers (int n, int i, int j) { if (n == 1){ cout << "Move disk from tower " << i; cout << " to tower " << j << "." << endl; } else { k = 6 i j; towers(n-1,i,k); towers(1,i,j); towers(n-1,k,j); } }
Unit 16 Objectives
Optimal Binary Search Trees Elements of the Greedy Strategy Huffman Codes
Variable Length Codes Prefix Codes Constructing a Huffman Code Justification of Optimality of Huffman Codes
Dynamic Programming compared to Greedy Methods Knapsack Problems Text Chapters 15.5, 16.3, 16.2
Optimality Metric for Binary Search Tree
The cost of a search may be approximated as the number of nodes traversed (level of termination + 1). A search terminating at the root (level 0) would cost 1 A search terminating at level 3 would cost 4 A search terminating at level 4 would cost 5 Whether the search is successful (terminating at a key node) or unsuccessful (terminating at a dummy node does not affect the cost computation. Based on the above the following relationships exist: np + nq = 1, where p = probability of searching for i=1 i i=0 i i key ki and qi = probability of searching for an invalid key and encountering dummy di (in between ki and ki+1) E = Expected search cost = i=1n((li + 1)*pi ) + i=0n((mi + 1)* qi), where li is the level of key ki and mi is the level of dummy di
The Structure of an Optimal Binary Search Tree
1. Characterize the structure of an optimal solution There are n possible roots if n is the number of keys For each possible root, the tree is composed of the root and two optimal binary search subtrees Each optimal binary search subtree has j i + 1 possible roots, where i is the index of the smallest key and j is the index of the largest key. Similarly, the optimal subtree must be composed of a root and two optimal subtrees. k-level optimal subtrees may be components of multiple optimal subtrees.
Recursive Formulation of Optimal Binary Search Tree
2. Recursively define the value of an optimal solution Define the expected search cost e(i, j) for the subtree which includes keys i through j. If we have a leaf node, then j = i-1 and e(i, j) = qj For a subtree, the sum of the probabilities is w(i, j) = k=ijpk + k=i-1jqk This is the amount by which a subtree's search cost increases when it becomes a subtree of a node. Thus, a reflexive definition of the cost e(i,j) if rooted at a node r, such that i < r < j, is e(i,j) = pr + e(i, r-1) + w(i, r-1) + e(r+1, j) + w(r+1,j) Since w(i, j) = pr + w(i, r-1) + w(r+1, j) e(i, j) = e(i, r-1) + e(r+1, j) + w(i,j) Recursive Form: qj if j = i 1 e(i, j) = minr=ij{e(i, r-1) + e(r+1, j) + w(i, j)} if i j
Optimal Binary Search Tree
3. Compute the value of an optimal solution in bottom-up fashion Note the similarities in problem formulation to the matrixchain multiplication This solution will also utilize 3 loops and solve in O(n3)
optimalBST(p, q, n) // p & q are arrays holding // key and dummy probabilities, respectively // solution uses working array w // stores optimal subtree results in e
Optimal (frequency) Binary Search Tree
optimalBST(p, q, n) for (j 0, j n, j++) e[0, j] q[j] w[0, j] q[j] for(i 1, i n, i++) for (j i, j n, j++) w[i, j] = w[i-1, j-1] + p[j] + q[j] // in e & w, i refers to width of tree // and j refers to last data node for(k 1, k i , k++) temp = e[k-1, j-i+k-1] + e[i-k, j]+w[i, j] if(k = 1 or temp > e[i, j]) e[i,j] = temp return e[1, n]
Example Run optimalBST: n = 5
probabilities of success and failure j 0 1 2 3 4 5
i\j 5
e: expected optimal cost 0 1 2 3 4
5 3.05 2.75 2.80 2.75 2.85 2.20 2.10 2.00 2.10 1.55 1.35 1.30 1.05 . 90 .50 .10
p[j]
.15 .10 .05 .10 .20
4 1.95 1.75 1.90 2.00 1.30 1.20 1.25 1.20 1.50 1.35 .70 .65 .80 .60 .25 .30 .05 .05
q[j] .05 .10 .05 .05 .05 .10
w: sum of p and q for subtree(i, j) i\j 0 1 2 3 4 5 5 1.00 4 .70 .80 3 .55 .50 .60 2 .45 .35 .30 .50 1 .30 .25 .15 .20 .35 0 .05 .10 .05 .05 .05 .10
3 2
1 0 .45 .10
.05
.90 .95 .40 .05
Example Run optimalBST: n = 5
i\j 5
e: expected optimal cost 0 1 2 3 4
5 3.05 2.75 2.80 2.75 2.85 2.20 2.10 2.00 2.10 1.55 1.35 1.30 1.05 . 90 .50 .10
4 i\j 5 4 3 2 1 0
root: root of optimal subtree 0 1 2 3 4 5 2 2 3 2 1 3 1 1 2 2 1 1 1 1 1
3 2
1 0 .45 .10
.05
.90 .95 .40 .05
1.95 1.75 1.90 2.00 1.30 1.20 1.25 1.20 1.50 1.35 .70 .65 .80 .60 .25 .30 .05 .05
An Approach to Greedy Algorithms
1. Determine the optimal substructure of the problem (similar to dynamic programming) 2. Develop a recursive solution (similar to dynamic programming) 3. Show that the greedy choice is an optimal choice 4. Show that making a greedy choice reduces the number of subproblems to one 5. Develop a recursive algorithm for the greedy strategy 6. Convert the recursive algorithm to iterative (if desired)
Building a Huffman Code Set with a Greedy Algorithm
1. Determine the optimal substructure of the problem Once we know the frequencies (O(n) pass on file), we can construct a code scheme, which is equivalent to constructing a decoding tree. Once the code scheme is constructed, we can build the encoding tree as an optimal binary search tree with all q's = 0. There are n possible roots if n is the number of keys The principal problem is deriving an effective code set efficiently. Hopefully, the Huffman Codes can be derived in O(n) time.
Developing a Greedy Approach
2. Develop a recursive solution The optimal binary search tree algorithm provides an O(n 4) methodology. Although the optimal binary search tree algorithm is iterative, it is a straight-forward exercise to convert it to recursive
Is This a Greedy Problem?
3. Show that the greedy choice is an optimal choice First, what is our greedy strategy. One approach would be to take the largest frequencies first, since we want these to be close to the root. We can then follow an approach of balancing (as best we can) the frequencies so that for each node the sum of the frequencies in the left subtree will be as close as possible to the sum of the frequencies in the right subtree. Note that this approach will result in a decreasing frequency tree (of no particular value for codes). This approach is know as the Shannon-Fano algorithm The preceding approach is conceptually a top-down approach. Why not use a bottom-up approach? This was Huffman's insight, which gives a better result. Note that an efficient prefix tree is a full tree Consider a forest of trees, each tree having one node We can recursively take the two smallest nodes and construct a tree. We continue until the forest contains only one tree.
Only One Subproblem
4. Show that making a greedy choice reduces the number of subproblems to one By the same logic used above, making the greedy choice will always give an optimal solution. Any other choice will not improve the result.
A Recursive Greedy Algorithm
5. Develop a recursive algorithm for the greedy strategy: Procedure:
Determine frequencies of tokens (or characters) Rank frequencies from lowest to highest (forest of onenode trees) Iteratively combine two smallest trees until entire forest is combined into one binary tree. Encodings are created from tree traversal to target leaf node: 0 for left child and 1 for right child
Huffman Coding
huffman(F, n) tree z; if (n > 1) z.setLeft(extractMin(F)); z.setRight(extractMin(F)); z.setFreq(z.left()->freq() + z.right()->freq()); insert(F, z); return(huffman(F, n-1)); else return(extractMin(F));
An O(n) Procedure for Producing Huffman Codes
6. Convert the recursive algorithm to iterative (if desired) Determine frequencies of tokens: O(n) Perform counting sort on frequencies: O(n) Enque sorted token frequencies in forest F, implemented as a queue of trees: O(n) The modified Huffman algorithm utilizes a second forest G (initially empty), also implemented as a queue trees: of O(n) Procedure total is (O(n) + O(n) + O(n) + O(n)) = O(n)
Modified Huffman Code Algorithm
huffman(queue F, queue G, int n) // F is sorted and dequeing doesn't // change order // Each element enqued into G is greater // than or equal to any previous element; // thus, G is always sorted tree z; if (n > 1) z.setLeft(extractMin(F, G)); z.setRight(extractMin(F, G)); z.setFreq(z.left()->freq() + z.right()->freq()); enque(G, z); return(huffman(F, G, n-1)); else return(deque(G));
Component A candidate set from which a solution is created A selection function to choose best candidate for addition to the solution A feasibility function to determine if a candidate contributes to a solution An objective function assigning values to solutions A solution function to indicate discovery of a final solution Subproblem Persistence Analytic Approach
A Comparison of Methodologies Divide and Dynamic Memoization Greedy Conquer Programming Algorithms Yes Yes Yes Yes
Yes Yes Yes Yes
Yes Yes Yes No Top-down, but Bottom-up okay
Yes Yes Yes Yes Bottom-up
Yes Yes Yes Yes Top-down
No One candidate at a time Yes Yes No Top-down / Bottom-up
Knapsack Problems Analysis
Fractional Knapsack Problem is very amenable to a greedy approach.
Simply start with the most valuable per unit of weight and continue until W is reached The last item may (is likely to) be a fractional amount
1,0 (true, false) Knapsack Problem is not amenable to a greedy approach.
Only an exhaustive method, e.g., divide and conquer, dynamic programming will yield an optimal solution
Homework 14
Due Date: TBD 15.5-1 15.5-2 16.2-2 (Give detailed pseudocode) Hint: Consider a recursive function c[i, w], which provides the optimal solution for weight w, including items i. The first case covers i = 0. The second case covers if wi > w. The third case covers the remaining situations, namely, if i > 0 and w wi. Recall that for item i, the weight is wi and the value vi. w is a weight parameter defining a call criterion. 16.3-2 (Give detailed pseudocode)
Unit 17 Objectives
Seven bridges of Konigsberg problem Graph notation and representation Breadth-first search Depth-first search Text Chapter 22.1, 22.2, 22.3, Appendix B.4
Hamiltonian Paths and Circuits
Hamiltonian path a tour of a graph which visits every vertex once. Hamiltonian circuit a tour of a graph which visits every vertex once, starting and ending at the same vertex. Compare (Seven bridges of Konigsberg): Eulerian path a tour of a graph which traverses every edge once. Eulerian circuit a tour of a graph which traverses every edge once, starting and ending at the same vertex.
Graphic Notation
General graphs have two types of components, nodes, aka vertices, and links, aka edges. The nodes are usually the elements containing data and identifiers, while the edges are the connections between nodes. Vertex-edge terminology is considered more formal in some circles. Two major types of graphs directed and undirected, perhaps more precisely, graphs with unidirectional links and graphs with bidirectional links. There are also graphs with both directed and undirected links. A multigraph is a graph which allows multiple edges between nodes. So, the Bridges of Konigsberg problem is actually a multigraph problem. An edge for which the starting and ending vertices are the same is called a self-loop. Traditionally, these are legal in undirected graphs and multigraphs (undirected or directed) but not in directed graphs. A complete graph is a graph in which there is an edge between every pair of vertices. For directed graphs, there must be an edge in each direction.
Graphic Notation
Degree refers to the number of connected nodes. With an undirected graph, degree is the number of links incident on that node. With a directed graph, in-degree is the number of incoming links, out-degree is the number of outgoing links, and degree is the sum of in-degree and out-degree. A selfloop adds two to the degree of an undirected graph. For a directed graph, a self-loop adds one each to in-degree and out-degree (or two to the degree). Diameter refers to the longest shortest path between any two nodes of the graph. A simple cycle is a cycle in which no vertex is visited more than once. If a simple cycle contains all the vertices of a graph, it is a Hamiltonian circuit, and the graph is Hamiltonian. An elementary cycle is a cycle in which no edge is traversed more than once. If an elementary cycle contains all the edges of a graph, it is a Eulerian circuit, and the graph is Eulerian. Circumference refers to the longest simple cycle in a graph.
Two principal representations adjacency lists and adjacency matrices. With adjency lists, the representation is often a list of vertices, each of which has an adjacency list of all vertices directly reachable (with no intervening vertices) from it. An adjacency matrix is a tabular representation with the first column having the vertex identifier, and each of the remaining columns referring to the adjacency of another vertex. With unweighted graphs, a one indicates adjacency and a 0 non-adjacency. The decision of which to use generally depends on the relationship |E|/|V|2 for a graph G(V, E), where E is the set of edges and V is the set of vertices and || refers to the cardinality of the sets, respectively. As this relationship approaches 1, the graph is considered dense and more appropriate for adjacency matrix representation; as it approaches 0, the graph is considered sparse and more appropriate for adjacency list representation. Sparseness / denseness is somewhat dependent on whether a graph is directed or undirected.
Graphic Representation
Weighted Graphs
For the adjacency-list representation, the edge weight is simply associated with the destination vertex. For the adjacency-matrix representation, the edge weight can also be associated with the destination vertex; however, there may be an issue if there are multiple edges. If there are multiple weighted edges, many algorithms would only be interested in the sum of the weights, in which case, a simple matrix representation still suffices. In other cases, the algorithm is interested only in the minimum or maximum weight, in which cases, a simple matrix representation will suffice.
Breadth-first Search
Starting with a single vertex s, a breadth-first search gradually moves outward. Distance is measure in terms of edge traversals required to reach a vertex from s. Moving outward from s, a breadth-first search first discovers all the vertices at distance 1, then distance 2, ... The algorithm we will study uses three colors, with all nodes initially white, becoming gray upon discovery and black when processing is completed black indicates that the instant node has no white reachable neighbors. When a node is discovered, it is added to a queue. Upon removal from the queue, all of its reachable neighbors are "discovered" and it is colored black. Nodes are augmented with color, distance, and parent pointer, allowing the construction of a breath-first search tree with root s. If the objective is a graph traversal, the search tree is unnecessary. This algorithm assumes that edges are kept in adjacency lists
BFS(G, s) for each vertex u G.V s u.color white s.color gray s.d 0 s.p null Q null Enque(Q, s) while Q not empty u = deque(Q) for each v Adj(u) if v.color = white v.color gray v.d u.d + 1 v.p u enque(Q, v) u.color = black
High-level Pseudocode for BFS()
Note that if a vertex is white, neither its distance nor its parent is known
printPath()
for a node v, printPath prints the path from s to v by ascending recursively through parents until s is reached. printPath(G, s, v) if(v = s) print s; else if(p.v = null) print "no path exists from " s " to " v else printPath(G, s, p.v) print " " v
Starting with a single vertex s, a depth-first search moves outward possibly toward the extremes. Distance d, is measured in terms of edge traversals used to reach a vertex from s; in depth-first search, this is measured by time, a global variable. A second distance f, is also measured by time, indicating that a node is no longer part of the search Moving outward from s, a depth-first search discovers all the vertices but a nearby vertex may be the last discovered. The algorithm we will study uses three colors, with all nodes initially white, becoming gray upon discovery, with black indicating that the instant node has had all of its neighbors discovered (no white neighbors). d is the time when the node becomes gray, and f, black. Nodes are augmented with color, distance, and parent pointer, allowing the construction of depth-search tree with root s. If the objective is a graph traversal, the search tree is unnecessary. This algorithm assumes that edges are kept in adjacency lists
Depth-First Search
DFS Algorithm
DFS(G) for each vertex u in G color.u white p.u null time 1 for each vertex u in G if color.u = white visit(u) visit(u) color.u gray d.u time ++ for each v reachable from u if color.v = white p.v u visit(v) color.u black f.u time ++
printPath()
for a node v, printPath prints the path from s to v by ascending recursively through parents until s is reached. printPath(G, s, v) if(v = s) print s; else if(p.v = null) print "no path exists from " s " to " v else printPath(G, s, p.v) print v " "
Homework
22.1-3 Hint: A transpose graph is a graph with all of the edges reversed. It is named because its adjacency matrix is the transpose of the adjacency matrix of the original graph. 22.2-1 22.2-2 22.3-2 22.3-9
Unit 18 Objectives
Topological sort Strongly connected components Text Chapter 22.4, 22.5
Topological Sort
Idea of a topological sort is to produce an ordered set from a partially ordered set, if possible. The basic logic is provided by DFS. In conclusion, listing the ordered set is based on the finish times from the DFS in inverse order.
Strongly Connected Components
Informal definitions: Strongly connected graph a graph in which there is a path from every vertex to every other vertex Strongly connected component, strongly connected subgraph a subset of the vertices of a graph which has two properties: there is a path from every vertex in the subgraph to every other vertex in the subgraph, and no other vertex of the graph can be added to the subset without violating the first property The above properties are generally applied only to directed graphs since in an undirected graph, a subgraph is either connected or not.
Strongly Connected Components
Observation 1: Consider GSCC, the graph of strongly connected components of graph G(V, E). In GSCC, each node is a strongly connected component (composed of one or more nodes of G, and there is an edge between two nodes of GSCC iff (if and only if) there is an edge between member nodes of the GSCC nodes in G. Observation: GSCC is a directed acyclic graph (DAG), because if there were cycles in GSCC, one or more nodes in GSCC would not satisfy the definition of an SCC. Conclusion: Since GSCC is a DAG, we know that it has at least one source node (node with either no links or only outgoing links), and one sink node (node with either no links or only incoming links).
Strongly Connected Components
Observation 2: Observation 2a: If we were to start from a graph G node contained in a graph GSCC sink node, and do a DFS, it would terminate when all nodes within the sink node had been visited, thus defining an SCC. Observation 2b: If we remove the above GSCC node, the remaining nodes retain the DAG characteristic, and thus, there is at least one sink node remaining, from which an initiated DFS in G will terminate when all nodes in the SCC have been visited. Our only difficulty then is starting our DFS from a node in graph G contained within a sink node, and when that DFS terminates, starting the next DFS from a sink node of the remainder graph, ... .
Strongly Connected Components
Observation 3: Consider GT(V, ET), the transpose of G(V, E), where ET = {(u, v) : (v, u) E; u, v V} Observation 3a: If we perform a DFS on GT, the highest f.u will be a node within a source node of GT-SCC. Observation 3b: A source node within GT-SCC will be a sink node with GSCC. Observation 4: Since we have a procedure for identifying a sink node within GSCC, and once we remove a node, identifying a sink node in the remainder graph, all that we have to do is perform a series of DFS's each one of which will identify all of the components of an SCC.
SCC(G) Construct GT from G. Perform DFS on GT. Perform descending counting sort on f.u. Perform modified DFS on G. Each DFS is conducted in order of decreasing f.u. Each DFS traverses the nodes of exactly one SCC. // Thus, each component of G can be marked, // listed, ..., as to its SCC composition in GSCC. Note that the algorithm in the text actually finds the SCC's in GT. Nevertheless, it is correct since the SCC's in GT have identical composition to the SCC's in G; however, the edges of GT-SCC will be in reverse direction to those of GSCC. The vertices in GT-SCC are identical to the vertices in GSCC; however, the edges go in the opposite direction.
SCC Algorithm
Unit 19 Minimum Spanning Trees Objectives
Minimum Spanning Tree Problem Minimum Spanning Tree Examples (After Kruskal) Kruskal's Algorithm and Analysis Minimum Spanning Tree Example (After Jarnik) Jarnik's Algorithm and Analysis Text Chapter 23.1, 23.2
Minimum Spanning Tree Problem
A spanning tree is a tree which connects n vertices with n-1 edges. In a graph with unweighted or equal weight edges, any spanning tree is a minimum spanning tree. In a graph with weighted edges a minimum spanning tree is the spanning tree with the minimum weight for the included edges totalWeight(MST) = (u, v) MST w(u, v) If an edge is added to a MST, a cycle will result. If an edge of the MST is replaced with another edge totalWeight(T) totalWeight(MST)
Minimum Spanning Tree Algorithm (after Kruskal)
MSTkruskal(G) A null For all vertices v, leader.v v size.v 1 Sort edges by weight While sorted edges remain For smallest weight remaining edge (u, v) If (leader.u != leader.v) A A (u, v) If(size.leader.u size.leader.v) size.leader.u size.leader.u + size.leader.v for all i, leader.i = v leader.i u Else size.leader.v size.leader.u + size.leader.v for all i, leader.i = u leader.i v
Analysis of kruskalMST()
Sort of edges is O(E lg E) Main loop processing has E iterations Iteration processing is O(1) except for changing leaders Maximum number of leader changes is V lg V. Worst case is the following: V/2 edge additions result in V/2 components each of size 2 Maximum of V/2 leader changes V/4 edge additions result in V/4 components each of size 4 Maximum of V/2 leader changes ... 2 edge additions result in 2 components of size V Maximum of V/2 leader changes 1 edge addition results in 1 component of size V Maximum of V/2 leader changes Summarizing above, T(n) = O(E lg E) + O(E) + O(V lg V) Since |V| < |E + 2|, T(n) = O(E lg E) Since |E| < |V2|, T(n) = O(E lg V)
Jarnik's Algorithm (aka Prim's)
MSTjarnik(G) for each u V.G weight.u p[u] null weight.r 0 PQ V.G while PQ u deque.PQ for each v in adj.u if(v PQ weight.(u, v) < weight.v p.v u weight.v weight.(u, v)
Analysis of Jarnik's Algorithm
Initialization is O(V) while loop executes V times, each iteration is based on size of adjacency list and heapify when weight changes or deque which is lg V total size of adjacency list is 2|E|. So for loop is executed O(E) times, with worst case cost of lg V operations per iteration So while loop is greater of O(E lg V) or O(V lg V) If fibonacci heap is used, time for weight changes can be amortized to O(1) and total cost reduces to O(E + V lg V) (deletes are still lg V)
Single Source Shortest Paths Objectives
Introduction to Single Source Shortest Paths (SSSP) SSSP in Directed Acyclic Graph (DAG) Dijkstra's Algorithm Bellman-Ford Algorithm Text 24.1, 24.2, 24.3
Introduction to Single Source Shortest Paths
For single source shortest paths problems we consider a weighted graph, G(V, E) with a weight function w(E) = the set of real numbers Although shortest paths can be defined in multiple ways, we limit ourselves here to the path between two vertices for which the sum of the weights for the edges traversed is minimal For individual algorithms we may impose further limitations on the graph:
Directed Acyclic Non-negative or positive weights
For all algorithms, we require that the graph have no negative weight cycles
This would lead to repeated traversals of the negative weight cycle with each traversal yielding a shorter path.
Related Problems
Related problems solvable with the single source algorithms:
Single source shortest path in an unweighted graph Recall that the breadth-first search algorithm provides a single source shortest path in an unweighted directed (or undirected graph) Single-destination shortest path reverse direction of each edge Single-pair shortest path no asymptotically better solution than single source exists for this problem All-pairs shortest path Unit 21 addresses this problem better solutions exist than iterating single source algorithms
Overview of three algorithms
SSSP in DAG Algorithm
Time = O(E) Requires DAG Requires no negative weight-cycles
Dijkstra's Algorithm
Time = O(E lg V) Requires no negative weight edges (if cyclic graph)
Bellman-Ford Algorithm
Time = O(VE) Handles all conditions Generates error if negative-weight cycle exists
SSSP in DAG Algorithm
DAGsssp(G, w, s) topSort(G) d.s 0 for each v V{s} d.v p.v null for each u V // in order of decreasing finishing times for each v Adj[u] if d.v > d.u + w(u, v) d.v d.u + w(u, v) p.v u
Alternative SSSP in DAG Algorithm
DAGsssp(G, w, s) topSort(G) INITsssp(G, s) for each u V // in order of // decreasing finishing times for each v Adj[u] ` RELAXsssp (G, u, v, w) INITsssp(G, s) d.s 0 for each v V{s} d.v p.v null RELAXsssp (G, u, v, w) if d.v > d.u + w(u, v) d.v d.u + w(u, v) p.v u
Implementation of PERT
Two straight-forward methods of adapting DAGsssp() to implement PERT applications are the following:
Use negative edge weights Reverse the relational operator in RELAXsssp(), and use instead of in INITsssp().
DIJKsssp() // initialization INITsssp(G, s) S QV // Q is a priority queue INITsssp(G, s) // maintaining VS d.s 0 // relaxation step for each v V{s} while Q d.v u Deque(Q) p.v null S S{u} RELAXsssp (G, u, v, w) for each v Adj[u] if d.v > d.u + w(u, v) RELAXsssp (G, u, v, w) d.v d.u + w(u, v) // Implicit DECREASE-KEY p.v u // Time = O(E lg V) 2 loops // execute aggregate of 2E times
Alternative Dijkstra's Algorithm
Alternative Bellman-Ford Algorithm
BFsssp(G, s) INITsssp(G, s) // initialization d.s 0 INITsssp(G, s) for each v V{s} // relaxation step d.v for i 1 to|V|1 p.v null for each edge (u, v) E RELAXsssp (G, u, v, w) RELAXsssp (G, u, v, w) if d.v > d.u + w(u, v) // check negative-weight cycle d.v d.u + w(u, v) for each edge (u, v) E p.v u if d.v > d.u + w(u, v) print error message // d.v = (s, v) or // a negative-weight cycle // is reachable from s // Time = O(VE).
Homework 18
24.2-1 24.3-1 24.1-1