lecture24 - CS 473: Algorithms Chandra Chekuri...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: CS 473: Algorithms Chandra Chekuri chekuri@cs.illinois.edu 3228 Siebel Center University of Illinois, Urbana-Champaign Fall 2009 Chekuri CS473 Part I Heuristics Chekuri CS473 Coping with Intractability Some general things that people do. Consider special cases of the problem which may be tractable. Run inefficient algorithms (for example exponential time algorithms for NP-hard problems) augmented with (very) clever heuristics stop algorithm when time/resources run out use massive computational power Exploit properties of instances that arise in practice which may be much easier. Give up on hard instances, which is ok. Settle for sub-optimal solutions, especially for optimization problems Chekuri CS473 NP and EXP EXP : all problems that have an exponential time algorithm. Proposition NP ⊆ EXP Chekuri CS473 NP and EXP EXP : all problems that have an exponential time algorithm. Proposition NP ⊆ EXP Proof. Let X ∈ NP with certifier C . To prove X ∈ EXP , here is an algorithm for X . Given input s , For every t , with |t | ≤ p (|s |) run C (s , t ); answer “yes” if any one of these calls returns “yes”, otherwise say “no”. Every problem in NP has a brute-force “try all possibilities” algorithm that runs in exponential time. Chekuri CS473 Examples SAT: try all possible truth assignment to variables Independent set: try all possible subsets of vertices Vertex cover: try all possible subsets of vertices Chekuri CS473 Improving brute-force via intelligent backtracking Backtrack search: enumeration with bells and whistles to “heuristically” cut down search space. Works quite well in practice for several problems, especially for small enough problem sizes. Chekuri CS473 Backtrack Search Algorithm for SAT Input: CNF Formula ϕ on n variables x1 , . . . , xn and m clauses Output: Is ϕ satisfiable or not. 1 Pick a variable xi 2 ϕ is CNF formula obtained by setting xi = 0 and simplifying Run a simple (heuristic) check on ϕ : returns “yes”, “no” or “not sure” 3 If “not sure” recursively solve ϕ If ϕ is satisfiable, return “yes” 4 5 ϕ is CNF formula obtained by setting xi = 1 Run simple check on ϕ : returns “yes”, “no” or “not sure” If “not sure” recursively solve ϕ If ϕ is satisfiable, return “yes” 6 Return “no” Certain part of the search space is pruned. Chekuri CS473 Example Algorithms 286 Figure 9.1 Backtracking reveals that φ is not satisfiable. (w ∨ x ∨ y ∨ z ), (w ∨ x), (x ∨ y ), (y ∨ z ), (z ∨ w), (w ∨ z ) w=0 w=1 (x ∨ y ), (y ∨ z ), (z ), (z ) (x ∨ y ∨ z ), (x), (x ∨ y ), (y ∨ z ) x=1 (y ∨ z ), (y ), (y ∨ z ) y=0 () (x ∨ y ), () z=1 (x ∨ y ), (y ), () y=1 () (z ), (z ) z=0 z=0 (), (y ∨ z ) x=0 z=1 () Figure: Backtrack search. Formula b not satisfiable. happens to be a singleton, then at least one of the resulting isranches will be terminated. (If there is a tie in choosing subproblems, one reasonable policy is to pick the one lowest in the tree, in the hope that it is close to a satisfying assignment.) See Figure 9.1 for the conclusion of our earlier example. More abstractly, a backtracking algorithm requires a test that looks at a subproblem and quickly declares one of three outcomes: Figure from Dasgupta etal book. 1. Failure: the subproblem has no sChekuri . olution CS473 Backtrack Search Algorithm for SAT How do we pick the order of variables? Chekuri CS473 Backtrack Search Algorithm for SAT How do we pick the order of variables? Heuristically! Examples: pick variable that occurs in most clauses first pick variable that appears in most size 2 clauses first ... Chekuri CS473 Backtrack Search Algorithm for SAT How do we pick the order of variables? Heuristically! Examples: pick variable that occurs in most clauses first pick variable that appears in most size 2 clauses first ... What are quick tests for satisfiability? Chekuri CS473 Backtrack Search Algorithm for SAT How do we pick the order of variables? Heuristically! Examples: pick variable that occurs in most clauses first pick variable that appears in most size 2 clauses first ... What are quick tests for satisfiability? Depends on known special cases and heuristics. Examples. Obvious test: return “no” if empty clause, “yes” if no clauses left and otherwise “not sure” Run obvious test and in addition if all clauses are of size 2 then run 2-SAT polynomial time algorithm ... Chekuri CS473 Branch-and-Bound: backtracking for optimization probs Intelligent backtracking can be used also for optimization problems. Consider a minimization problem. Notation: for instance I , opt (I ) is optimum value on I . P0 initial instance of given problem. Keep track of the best solution value B found so far. Initialize B to be crude upper bound on opt (I ). Let P be a subproblem at some stage of exploration. If P is a complete solution, update B . Else use a lower bounding heuristic to quickly/efficiently find a lower bound b on opt (P ). If b ≥ B then prune P Else explore P further by breaking it into subproblems and recurse on them. Output best solution found. Chekuri CS473 Example: Vertex Cover Given G = (V , E ), find a minimum sized vertex cover in G . Initialize B = n − 1. Pick a vertex u . Branch on u : either choose u or discard it. Let b1 be a lower bound on G1 = G − u . If 1 + b1 < B , recursively explore G1 Let b2 be a lower bound on G2 = G − u − N (u ) where N (u ) is the set of neighbours of u . If |N (u )| + b2 < B , recursively explore G2 Output B . Chekuri CS473 Example: Vertex Cover Given G = (V , E ), find a minimum sized vertex cover in G . Initialize B = n − 1. Pick a vertex u . Branch on u : either choose u or discard it. Let b1 be a lower bound on G1 = G − u . If 1 + b1 < B , recursively explore G1 Let b2 be a lower bound on G2 = G − u − N (u ) where N (u ) is the set of neighbours of u . If |N (u )| + b2 < B , recursively explore G2 Output B . How do we compute a lower bound? One possibility: solve an LP relaxation that we saw in previous lecture. Chekuri CS473 Local Search Local Search: a simple and broadly applicable heuristic method Start with some arbitrary solution s Let N (s ) be solutions in the “neighbourhood” of s obtained from s via “local” moves/changes If there is a solution s ∈ N (s ) that is better than s , move to s and continue search with s Else, stop search and output s . Chekuri CS473 Local Search Main ingredients in local search: Initial solution Definition of neighbourhood of a solution Efficient algorithm to find a good solution in the neighbourhood Chekuri CS473 Example: TSP TSP: Given a complete graph G = (V , E ) with cij denoting cost of edge (i , j ), compute a Hamiltonian cycle/tour of minimum edge cost. Chekuri CS473 Example: TSP TSP: Given a complete graph G = (V , E ) with cij denoting cost of edge (i , j ), compute a Hamiltonian cycle/tour of minimum edge cost. 2-change local search: Start with an arbitrary tour s0 For a solution s define s to be a neighbour if s can be obtained from s by replacing two edges in s with two other edges. For a solution s at most O (n2 ) neighbours and one can try all of them to find an improvement. Chekuri CS473 ey can’t differ in just one edge (do you see why?), so we will consider differe s. TSP: fine the 2-change neighborhood of tour s as being the set of tours tha We de 2-change example by removing two edges of s and then putting in two other edges. Here’s an e move: w have a well-defined local search procedure. How does it measure up under criteria for algorithms—what is its overall running time, and does it always olution? rrassingly, neither of these questions has a satisfactory answer. Each iter fast, because a tour has only O(n 2 ) neighbors. However, it is not clear how s will be needed: whether for instance, there might be an exponential nu kewise, all we can easily say about the final tour is that it is locally optima perior to the tours in its immediate neighborhood. There might be better so way. For instance, the following picture shows a possible final answer that is al; the range of local moves is simply too limited to improve upon it. Chekuri CS473 ey can’t differ in just one edge (do you see why?), so we will consider differe ave a well-defined examplerch procedure. How does it measure up u local sea s. TSP: fine the 2-change neighborhood of tour s as being the set of tours tha We de 2-change eyiremovalg oriohms—what dstitenoputtanl runwinothemedgen.dHeres ianae r a for ing twt edges of s ani h s ver i l g in t n o g ti r e, a s do e’s t l b tmov? : ion e ssingly, neither of these questions has a satisfactory answer. Each t, because a tour has only O(n 2 ) neighbors. However, it is not clea ll be needed: whether for instance, there might be an exponentia ise, all we can easily say about the final tour is that it is locally op ior to the tours in its immediate neighborhood. There might be bet .w havina taell-e,efihed oolowsearchictucedshe.wsoa possiiblmeanure up wndeth For e s w nc d t n f l l cal ing p prore ur o H w doe t e fi sal ans u er r Figure below ms v a is mp o era l m n 2-change and o p critraingeroallocatshows es badsiis itslyvtoo lliruitningo iimprovedues n at. ays the er a fo f gori lhmo—what local optimum fored t t me, heuristic oit i lw olution? rrassingly, neither of these questions has a satisfactory answer. Each iter fast, because a tour has only O(n 2 ) neighbors. However, it is not clear how s will be needed: whether for instance, there might be an exponential nu kewise, all we can easily say about the final tour is that it is locally optima perior to the tours in its immediate neighborhood. There might be better so way. For instance, the following picture shows a possible final answer that is ale t e s, we ma c l y a m i r s g pl e t ous ne ed to improve u r n t. m; thhirange of loyatrmoves os eimeny roo limitighborhood, fopoinistance 3- rs that differ on up to three edges. And indeed, the preceding b Chekuri CS473 TSP: 3-change example overcome this, we may try a more generous neighborhood, for instance 3-change 3-change iffer search: swap 3 edges An of tours that dlocal on up to three edges. out. d indeed, the preceding bad cas t there is a downside, in that the size of a neighborhood becomes O(n 3 ), making on moNeighbourhood ofreohas, nowre may stilto e suboof iΩ(n3l) cal minima, alt re expensive. Mo s ver the increased l b a size pt mal o than before. To av-change e, we would have edges up tswapped ge, or higher. I Can define k oid thes heuristic where k to go are o 4-chan out. r, efficIncreasesdneighbourhoodusize ut to makes each local siderations in a local s iency an quality often t rn o and be competing con improvement ncy destep ds neighborhoods that can be searched quickly, but smaller neighbor man less efficient. Chekuri CS473 Local Search Variants Local search terminates with a local optimum which may be far from a global optimum. Many variants to improve plain local search. Randomization and restarts. Initial solution may strongly influence the quality of the final solution. Try many random initial solutions. Simulated annealing is a general method where one allows the algorithm to move to worse solutions with some probability. At the beginning this is done more agressively and then slowly the algorithm converges to plain local search. Controlled by a parameter called “temperature”. Tabu search. Store already visited solutions and do not visit them again (they are “taboo”). Chekuri CS473 Heuristics Several other heuristics used in practice. Heuristics for solving integer linear programs such as cutting planes, branch-and-cut etc are quite effective. Heuristics to solve SAT (SAT-solvers) have gained prominence in recent years Genetic algorithms ... Heuristics design is somewhat adhoc and depends heavily on the problem and the instances that are of interest. Rigorous analysis is sometimes possible. Chekuri CS473 Part II Finals and Closing Thoughts Chekuri CS473 Topics for Finals Recursion: reduce problem to smaller instance(s) of itself Divide and Conquer: divide into multiple pieces, solve recursively and build solution to the original instance Dynamic Programming: recursion with memoization, runs in polynomial time if number of sub-problems is polynomial. Graph Algorithms: DFS/BFS, directed graphs, DAGs, strong components, MST, shortest paths Greedy Algorithms: always think of a proper proof! Network Flows and Applications Polynomial time Reductions, P, NP, co-NP, NP-Completeness Light touch: Linear Programming, Integer Programming, Approximation Algorithm Chekuri CS473 Topics I wish I had time for Data structures - hashing, splay trees etc. Randomization in algorithms Basic lower bounds More on heuristics and applications Experimental evaluation Chekuri CS473 Theoretical Computer Science Algorithms: find efficient ways to solve particular problems Computational Complexity: understand nature of computation — classification of problems into classes (P, NP, coNP) and their relationships, limits of computation. Logic, Languages and Formal Methods Form the foundations for computer “science” Chekuri CS473 The Computational Lens The Algorithm: Idiom of Modern Science by Bernard Chazelle http://www.cs.princeton.edu/chazelle/pubs/algorithm.html Computation has gained ground as fundamental artifact in mathematics and science. nature of proofs, P vs NP , complexity, . . . quantum computation and information computational biology and the biological processes, . . . Standard question in math and sciences: Is there a solution/algorithm? Chekuri CS473 The Computational Lens The Algorithm: Idiom of Modern Science by Bernard Chazelle http://www.cs.princeton.edu/chazelle/pubs/algorithm.html Computation has gained ground as fundamental artifact in mathematics and science. nature of proofs, P vs NP , complexity, . . . quantum computation and information computational biology and the biological processes, . . . Standard question in math and sciences: Is there a solution/algorithm? New: Is there an efficient solution/algorithm? Chekuri CS473 Questions? Chekuri CS473 Questions? Final Exam: Thursday, Dec 17th, 1.30 - 4.30pm in 1404 Siebel. Thanks! Chekuri CS473 ...
View Full Document

This note was uploaded on 01/22/2012 for the course CS 573 taught by Professor Chekuri,c during the Fall '08 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online