Unformatted text preview: 18.415 6.854 Advanced Algorithms
Lecturer: Michel X. Goemans Linear Programming October 1994 1 An Introduction to Linear Programming
Linear programming is a very important class of problems, both algorithmically and combinatorially. Linear programming has many applications. From an algorithmic pointofview, the simplex was proposed in the forties soon after the war, and was motivated by military applications and, although it has performed very well in practice, is known to run in exponential time in the worstcase. On the other hand, since the early seventies when the classes P and NP were de ned, it was observed that linear programming is in NP coNP although no polynomialtime algorithm was known at that time. The rst polynomialtime algorithm, the ellipsoid algorithm, was only discovered at the end of the seventies. Karmarkar's algorithm in the mideighties lead to very active research in the area of interiorpoint methods for linear programming. We shall present one of the numerous variations of interiorpoint methods in class. From a combinatorial perspective, systems of linear inequalities were already studied at the end of the last century by Farkas and Minkovsky. Linear programming, and especially the notion of duality, is very important as a proof technique. We shall illustrate its power when discussing approximation algorithms. We shall also talk about network ow algorithms where linear programming plays a crucial role both algorithmically and combinatorially. For a more indepth coverage of linear programming, we refer the reader to 1, 4, 7, 8, 5 . A linear program is the problem of optimizing a linear objective function in the decision variables, x1 : : : xn, subject to linear equality or inequality constraints on the xi's. In standard form, it is expressed as: n X Min cj xj objective function j =1 subject to: n X aij xj = bi; i = 1 : : : m constraints j =1 xj 0; j = 1 : : : n nonnegativity constraints where faij ; bi; cj g are given. A linear program is expressed more conveniently using matrices: Ax = b T x subject to min c x 0 LP1 where x = b = c = A = 0 1 B x..1 C B . C @ A 0 xn 1 B b..1 C B . C @ A 0 bm 1 B c..1 C B . C @ A 0 cn B a11 . . B . @ 2 Rn1 2 Rm1 2 Rn1 amn 1 C C 2 Rmn A 2 Basic Terminology De nition 1 If x satis es Ax = b; x 0, then x is feasible. De nition 2 A linear program LP is feasible if there exists a feasible solution, otherwise it is said to be infeasible. De nition 3 An optimal solution x is a feasible solution s.t. cT x = minfcT x : Ax = b; x 0g. De nition 4 LP is unbounded from below if 8 2 R; 9 a feasible x s.t. cT x . 3 Equivalent Forms
A linear program can take on several forms. We might be maximizing instead of minimizing. We might have a combination of equality and inequality contraints. Some variables may be restricted to be nonpositive instead of nonnegative, or be unrestricted in sign. Two forms are said to be equivalent if they have the same set of optimal solutions or are both infeasible or both unbounded. 1. A maximization problem can be expressed as a minimization problem. max cT x , min ,cT x 2. An equality can be represented as a pair of inequalities. T , aiT x bii ai x b aT x = bi i T , ,aiT x ,bii ai x b LP2 3. By adding a slack variable, an inequality can be represented as a combination of equality and nonnegativity constraints. aT x bi , aT x + si = bi; si 0: i i
4. Nonpositivity constraints can be expressed as nonnegativity constraints. To express xj 0, replace xj everywhere with ,yj and impose the condition yj 0. 5. x may be unrestricted in sign. If x is unrestricted in sign, i.e. nonpositive or nonnegative, everywhre replace xj by x+ , x,, adding the constraints x+ ; x, 0. j j j j In general, an inequality can be represented using a combination of equality and nonnegativity constraints, and vice versa. n n Using these rules, min cT x s.t. Ax bg can be transformed into min cT x+ , cT x, s.t. Ax+ , Ax, , Is = b, x+; x,; s 0g. The former LP is said to be in canonical form, the latter in standard form. n Conversely, an LP in standard form may be written in canonical form. min cT x n s.t. Ax = b; x 0g is equivalent to min cT x 0 Ax b, ,Ax 1,b, Ix 0g. s.t. 1 0 A b 0 0 0 B C and b0 = B b C. This may be rewritten as A x b , where A = @ A A @ A
I 0 4 Example
Consider the following linear program: 8 x 1 min x2 subject to 3x1 , x2 2 : ,x1 + xx2 x1 + 2 2 0 6 0 The optimal solution is 4; 2 of cost 2 see Figure 1. If we were maximizing x2 instead of minimizing under the same feasible region, the resulting linear program would be unbounded since x2 can increase arbitrarily. From this picture, the reader should be convinced that, for any objective function for which the linear program is bounded, there exists an optimal solution which is a corner" of the feasible region. We shall formalize this notion in the next section. LP3 3x1x20 x2 (2,6) (2,4) x1+2x20
(4,2) (2,1) x12 x1+x26 x1 Figure 1: Graph representing primal in example. An example of an infeasible linear program can be obtained by reversing some of the inequalities of the above LP: x1 3x1 , x2 x1 + x2 ,x1 + 2x2 2 0 6 0: 5 The Geometry of LP
Let P = fx : Ax = b; x 0g Rn. De nition 5 x is a vertex of P if 6 9y 6= 0 s:t: x + y; x , y 2 P . Theorem 1 Assume minfcT x : x 2 P g is nite, then 8x 2 P; 9 a vertex x0 such that cT x0 cT x. Proof: 0 If x is a vertex, then take x = x. If x is not a vertex, then, by de nition, 9y 6= 0 s.t. x + y; x , y 2 P . Since Ax + y = b and Ax , y = b, Ay = 0. WLOG, assume cT y 0 take either y or ,y. If cT y = 0, choose y such that 9j s.t. yj 0. Since y 6= 0 and cT y = cT ,y = 0, this must be true for either y or ,y. Consider x + y; 0. cT x + y = cT x + cT y cT x, since cT y is assumed nonpositive. LP4 x2 (0,1) (0,0) x1 Figure 2: A polyhedron with no vertex. Case 1 9j such that yj 0 As increases, component j decreases until x + y is no longer feasible. Choose = minfj:yj 0gfxj = , yj g = xk = , yk . This is the largest such that x + y 0. Since Ay = 0, Ax + y = Ax + Ay = Ax = b. So x + y 2 P , and moreover x + y has one more zero component, x + yk , than x. Replace x by x + y. Case 2 yj 0 8j By assumption, cT y 0 and x + y is feasible for all 0, since Ax + y = Ax + Ay = Ax = b, and x + y x 0. But cT x + y = cT x + cT y ! ,1 as ! 1, implying LP is unbounded, a contradiction. Case 1 can happen at most n times, since x has n components. By induction on the number of nonzero components of x, we obtain a vertex x0 . Remark: The theorem was described in terms of the polyhedral set P = fx : Ax = b : x 0g. Strictly speaking, the theorem is not true for P = fx : Ax bg. Indeed, such a set P might not have any vertex. For example, consider P = fx1; x2 : 0 x2 1g see Figure 2. This polyhedron has no vertex, since for any x 2 P , we have x + y, x , y 2 P , where y = 1; 0. It can be shown that P has a
vertex i RankA = n. Note that, if we transform a program in canonical form into standard form, the nonnegativity constraints imply that the resulting matrix A has full column rank, since 2 3 A 6 A 7 = n: Rank 4 5
I LP5 Corollary 2 If minfcT x : Ax = b; x 0g is nite, There exists an optimal solution,
x, which is a vertex. Proof: Suppose not. Take an optimal solution. By Theorem 1 there exists a vertex costing no more and this vertex must be optimal as well. Corollary 3 If P = fx : Ax = b; x 0g 6= ;, then P has a vertex. Theorem 4 Let P = fx : Ax = b; x 0g. For x 2 P , let Ax be a submatrix of A
corresponding to j s.t. xj 0. Then x is a vertex i Ax has linearly independent columns. i.e. Ax has full column rank. Proof: 223 2 3 2 3 2 1 3 0 2 3 607 Example A = 6 7 3 2 1 7 x = 6 1 7 Ax = 6 7 2 7, and x is a vertex. 6 7 4 5 4 5 4 5 0 0 0 5 0 0 0 Show :i ! :ii. Assume x is not a vertex. Then, by de nition, 9y 6= 0 s.t. x + y; x , y 2 P . Let Ay be submatrix corresponding to nonzero components of y. As in the proof of Theorem 1, Ax + Ay = b Ay = 0: Ax , Ay = b Therefore, Ay has dependent columns since y 6= 0. Moreover, x + y 0 y = 0 whenever x = 0: j j x , y 0 Therefore Ay is a submatrix of Ax. Since Ay is a submatrix of Ax, Ax has linearly dependent columns. Show :ii ! :i. Suppose Ax has linearly dependent columns. Then 9y s.t. Axy = 0, y 6= 0. Extend y to Rn by adding 0 components. Then 9y 2 Rn s.t. Ay = 0; y 6= 0 and yj = 0 wherever xj = 0. Consider y0 = y for small 0. Claim that x + y0 ; x , y0 2 P , by argument analogous to that in Case 1 of the proof of Theorem 1, above. Hence, x is not a vertex. LP6 6 Bases
Let x be a vertex of P = fx : Ax = b; x 0g. Suppose rst that jfj : xj 0gj = m where A is m n. In this case we denote B = fj : xj 0g. Also let AB = Ax; we use this notation not only for A and B , but also for x and for other sets of indices. Then AB is a square matrix whose columns are linearly independent by Theorem 4, so it is nonsingular. Therefore we can express x as xj = 0 if j 62 B , and since AB xB = b, it follows that xB = A,1b. The variables corresponding to B will be called B basic. The others will be referred to as nonbasic. The set of indices corresponding to nonbasic variables is denoted by N = f1; : : : ; ng , B . Thus, we can write the above as xB = A,1b and xN = 0. B Without loss of generality we will assume that A has full row rank, rankA = m. Otherwise either there is a redundant constraint in the system Ax = b and we can remove it, or the system has no solution at all. If jfj : xj 0gj m, we can augment Ax with additional linearly independent columns, until it is an m m submatrix of A of full rank, which we will denote AB. In other words, although there may be less than m positive components in x, it is convenient to always have a basis B such that jB j = m and AB is nonsingular. This enables us to always express x as we did before, xN = 0, xB = A,1b. B Summary x is a vertex of P i there is B f1; : : : ; ng such that jB j = m and
1. xN = 0 for N = f1; : : : ; ng , B 2. AB is nonsingular 3. xB = A,1b 0 B In this case we say that x is a basic feasible solution . Note that a vertex can have several basic feasible solution corresponding to it by augmenting fj : xj 0g in di erent ways. A basis might not lead to any basic feasible solution since A,1b is B not necessarily nonnegative. Example: x1 + x2 + x3 = 5 2x1 , x2 + 2x3 = 1 x1; x2; x3 0
LP7 We can select as a basis B = f1; 2g. Thus, N = f3g and ! 1 1 AB = 2 ,1 1 1! ,1 = 3 ,1 3 AB 2 3 !3 2 ,1 b = AB 3 x = 2; 3; 0 n m . This number m . We can come up with a tighter approxis exponential itis upper bounded by n m imation of n,2 2 , though this is still exponential. The reason why the number is m much smaller is that most basic solutions to the system Ax = b which we counted are not feasible, that is, they do not satisfy x 0. Remark. A crude upper bound on the number of vertices of P is 7 The Simplex Method
The Simplex algorithm Dantzig,1947 2 solves linear programming problems by focusing on basic feasible solutions. The basic idea is to start from some vertex v and look at the adjacent vertices. If an improvement in cost is possible by moving to one of the adjacent vertices, then we do so. Thus, we will start with a bfs corresponding to a basis B and, at each iteration, try to improve the cost of the solution by removing one variable from the basis and replacing it by another. We begin the Simplex algorithm by rst rewriting our LP in the form: min cB xB + cN xN s.t. AB xB + AN xN = b xB ; xN 0 Here B is the basis corresponding to the bfs we are starting from. Note that, for any solution x, xB = A,1b , A,1AN xN and that its total cost, cT x can be speci ed B B as follows: cT x = cB xB + cN xN = cB A,1b , A,1AN xN + cN xN B B ,1 b + c , c A,1 A x = cB AB N B B N N We denote the reduced cost of the nonbasic variables by cN , cN = cN , cB A,1AN , ~ ~ B i.e. the quantity which is the coe cient of xN above. If there is a j 2 N such that
LP8 cj 0, then by increasing xj up from zero we will decrease the cost the value of ~ the objective function. Of course xB depends on xN , and we can increase xj only as long as all the components of xB remain positive. So in a step of the Simplex method, we nd a j 2 N such that cj 0, and increase ~ it as much as possible while keeping xB 0. It is not possible any more to increase xj , when one of the components of xB is zero. What happened is that a nonbasic variable is now positive and we include it in the basis, and one variable which was basic is now zero, so we remove it from the basis. If, on the other hand, there is no j 2 N such that cj 0, then we stop, and ~ the current basic feasible solution is an optimal solution. This follows from the new expression for cT x since xN is nonnegative. Remarks: 1. Note that some of the basic variables may be zero to begin with, and in this case it is possible that we cannot increase xj at all. In this case we can replace say j by k in the basis, but without moving from the vertex corresponding to the basis. In the next step we might replace k by j , and be stuck in a loop. Thus, we need to specify a pivoting rule" to determine which index should enter the basis, and which index should be removed from the basis. 2. While many pivoting rules including those that are used in practice can lead to in nite loops, there is a pivoting rule which will not known as the minimal index rule  choose the minimal j and k possible Bland, 1977 . This fact was discovered by Bland in 1977. There are other methods of breaking ties" which eliminate in nite loops. 3. There is no known pivoting rule for which the number of pivots in the worst case is better than exponential. 4. The question of the complexity of the Simplex algorithm and the last remark leads to the question of what is the length of the shortest path between two vertices of a convex polyhedron, where the path is along edges, and the length of the path in measured in terms of the number of vertices visited. Hirsch Conjecture: For m hyperplanes in d dimensions the length of the shortest path between any two vertices of the arrangement is at most m , d. This is a very open question  there is not even a polynomial bound proven on this length. On the other hand, one should note that even if the Hirsch Conjecture is true, it doesn't say much about the Simplex Algorithm, because Simplex generates paths which are monotone with respect to the objective function, whereas the shortest path need not be monotone. LP9 Recently, Kalai and others has considered a randomized pivoting rule. The idea is to randomly permute the index columns of A and to apply the Simplex method, always choosing the smallest j possible. In this way, it is possible to show a subexponential bound on the expected number of pivots. This leads to a subexponential bound for the diameter of any convex polytope de ned by m hyperplanes in a d dimension space. The question of the existence of a polynomial pivoting scheme is still open though. We will see later a completely di erent algorithm which is polynomial, although not strongly polynomial the existence of a strongly polynomial algorithm for linear programming is also open. That algorithm will not move from one vertex of the feasible domain to another like the Simplex, but will con ne its interest to points in the interior of the feasible domain. A visualization of the geometry of the Simplex algorithm can be obtained from considering the algorithm in 3 dimensions see Figure 3. For a problem in the form minfcT x : Ax bg the feasible domain is a polyhedron in R3, and the algorithm moves from vertex to vertex in each step or does not move at all. Objective function Figure 3: Traversing the vertices of a convex body here a polyhedron in R3. LP10 ...
View
Full
Document
This note was uploaded on 02/13/2012 for the course CSE 4101 taught by Professor Mirzaian during the Winter '12 term at York University.
 Winter '12
 Mirzaian
 Algorithms, Data Structures

Click to edit the document details