Unformatted text preview: 8 When is a Linear Program Feasible ?
We now turn to another question which will lead us to important properties of linear programming. Let us begin with some examples. We consider linear programs of the form Ax = b, x 0. As the objective function has no e ect on the feasibility of the program, we ignore it. We rst restrict our attention to systems of equations i.e. we neglect the nonnegativity constraints. Example: Consider the system of equations: x1 + x2 + x3 = 6 2x1 + 3x2 + x3 = 8 2x1 + x2 + 3x3 = 0 and the linear combination ,4 x1 + x2 + x3 = 6 1 2x1 + 3x2 + x3 = 8 1 2x1 + x2 + 3x3 = 0 The linear combination results in the equation
0x1 + 0x2 + 0x3 = ,16 which means of course that the system of equations has no feasible solution. In fact, an elementary theorem of linear algebra says that if a system has no solution, there is always a vector y such as in our example y = ,4; 1; 1 which proves that the system has no solution. Theorem 5 Exactly one of the following is true for the system Ax = b:
1. There is x such that Ax = b. 2. There is y such that AT y = 0 but yT b = 1. This is not quite enough for our purposes, because a system can be feasible, but still have no non-negative solutions x 0. Fortunately, the following lemma establishes the equivalent results for our system Ax = b; x 0. Theorem 6 Farkas' Lemma Exactly one of the following is true for the system Ax = b, x 0: 1. There is x such that Ax = b, x 0. 2. There is y such that AT y 0 but bT y 0.
LP-11 Proof: We will rst show that the two conditions cannot happen together, and then than at least one of them must happen. Suppose we do have both x and y as in the statement of the theorem. Ax = b = yT Ax = yT b = xT AT y = yT b but this is a contradiction, because yT b 0, and since x 0 and AT y 0, so xT AT y 0. The other direction is less trivial, and usually shown using properties of the Simplex algorithm, mainly duality. We will use another tool, and later use Farkas' Lemma to prove properties about duality in linear programming. The tool we shall use is the Projection theorem, which we state without proof: Theorem 7 Projection Theorem Let K be a closed convex see Figure 4 non- empty set in Rn, and let b be any point in Rn. The projection of b onto K is a point p 2 K that minimizes the Euclidean distance kb , pk. Then p has the property that for all z 2 K , z , pT b , p 0 see Figure 5 non-empty set. not convex convex Figure 4: Convex and non-convex sets in R2. We are now ready to prove the other direction of Farkas' Lemma. Assume that there is no x such that Ax = b, x 0; we will show that there is y such that AT y 0 but yT b 0. Let K = fAx : x 0g Rm A is an m n matrix. K is a cone in Rm and it is convex, non-empty and closed. According to our assumption, Ax = b, x 0 has no solution, so b does not belong to K . Let p be the projection of b onto K . Since p 2 K , there is a w 0 such that Aw = p. According to the Projection Theorem, for all z 2 K , z , pT b , p 0 That is, for all x 0 Ax , pT b , p 0 We de ne y = p , b, which implies Ax , pT y 0. Since Aw = p, Ax , AwT y 0. x , wT AT y 0 for all x 0 remember that w was xed by choosing b. LP-12 z p b Figure 5: The Projection Theorem. 0 1 B0C B0C B.C B .. C B C Set x = w + B 1 C w plus a unit vector with a 1 in the i-th row. Note that x B C B C B .. C B.C @ A 0 is non-negative, because w 0. This will extract the i-th column of A, so we conclude that the i-th component of T y is non-negative AT y i 0, and since this is true for all i, AT y 0: A Now it only remains to show that yT b 0. ytb = p , yT y = pT y , yT y Since Ax , pT y 0 for all x 0, taking x to be zero shows that pT y 0. Since b 62 K , y = p , b 6= 0, so yT y 0. So yT b = pT y , yT y 0.
Using a very similar proof one can show the same for the canonical form: Theorem 8 Exactly one of the following is true for the system Ax b: 1. There is x such that Ax b. 2. There is y 0 such that AT y = 0 but y T b 0.
The intuition behind the precise form for 2. in the previous theorem lies in the proof that both cannot happen. The contradiction 0 = 0x = yT Ax = yT Ax = yT b 0 is obtained if AT y = 0 and yT b 0. LP-13 9 Duality
Duality is the most important concept in linear programming. Duality allows to provide a proof of optimality. This is not only important algorithmically but also it leads to beautiful combinatorial statements. For example, consider the statement In a graph, the smallest number of edges in a path between two speci ed vertices s and t is equal to the maximum number of s , t cuts i.e. subsets of edges whose removal disconnects s and t. This result is a direct consequence of duality for linear programming. Duality can be motivated by the problem of trying to nd lower bounds on the value of the optimal solution to a linear programming problem if the problem is a maximization problem, then we would like to nd upper bounds. We consider problems in standard form: min cT x s.t. Ax = b x0 Suppose we wanted to obtain the best possible upper bound on the cost function. By multiplying each equation Amx = bm by some number ym and summing up the resulting equations, we obtain that yT Ax = bT y. if we impose that the coe cient of xj in the resulting inequality is less or equal to cj then bT y must be a lower bound on the optimal value since xj is constrained to be nonnegative. To get the best possible lower bound, we want to solve the following problem: max bT y s.t. AT y c This is another linear program. We call this one the dual of the original one, called the primal. As we just argued, solving this dual LP will give us a lower bound on the optimum value of the primal problem. Weak duality says precisely this: if we denote the optimum value of the primal by z, z = min cT x, and the optimum value of the dual by w, then w z. We will use Farkas' lemma to prove strong duality which says that these quantities are in fact equal. We will also see that, in general, the dual of the dual is the problem. Example: z = min x1 + 2x2 + 4x3 x1 + x2 + 2x3 = 5 2x1 + x2 + 3x3 = 8 The rst equality gives a lower bound of 5 on the optimum value z, since x1 + 2x2 + 4x3 x1 + x2 + 2x3 = 5 because of nonnegativity of the xi. We can get an even
LP-14 better lower bound by taking 3 times the rst equality minus the second one. This gives0x1 + 2x2 + 3x3 = 7 x1 + 2x2 + 4x3, implying a lower bound of 7 on z. For 1 3 x = B 2 C, the objective function is precisely 7, implying optimality. The mechanism @ A 0 of generating lower bounds is formalized by the dual linear program: max 5y1 + 8y2 y1 + 2y2 1 y1 + y2 2 2y1 + 3y2 4 y1 represents the multiplier for the rst constraint and y2 the multiplier for the second constraint, This LP's objective function also achieves a maximum value of 7 at y = ! 3 . ,1 We now formalize the notion of duality. Let P and D be the following pair of dual linear programs:
P z = minfcT x : Ax = b; x 0g D w = maxfbT y : AT y cg: P is called the primal linear program and D the dual linear program. In the proof below, we show that the dual of the dual is the primal. In other words, if one formulates D as a linear program in standard form i.e. in the same form as P , its dual DD can be seen to be equivalent to the original primal P . In any statement, we may thus replace the roles of primal and dual without a ecting the statement. Proof: The dual problem D is equivalent to minf,bT y : AT y + Is = c; s 0g. Changing forms we get minf,bT y+ + bT y, : AT y+ , AT y, + Is = c; and y+; y,; s 0g. Taking the dual of this we obtain: maxf,cT x : A,x ,b; ,A,x b; I ,x 0g. But this is the same as minfcT x : Ax = b; x 0g and we are done. We have the following results relating w and z. Lemma 9 Weak Duality z w. Proof: Suppose x is primal feasible and y is dual feasible. Then, cT x yT Ax = yT b, thus z = minfcT x : Ax = b; x 0g maxfbT y : AT y cg = w. From the preceding lemma we conclude that the following cases are not possible these are dual statements: 1. P is feasible and unbounded and D feasible. LP-15 2. P is feasible and D is feasible and unbounded. We should point out however that both the primal and the dual might be infeasible. To prove a stronger version of the weak duality lemma, let's recall the following corollary of Farkas' Lemma Theorem 8: Corollary 10 Exactly one of the following is true: 1. 9x0 : A0x0 b0. 2. 9y 0 0 : A0T y 0 = 0 and b0T y 0 0. Theorem 11 Strong Duality If P or D is feasible then z = w. Proof: We only need to show that z w. Assume without loss of generality by duality
that P is feasible. If P is unbounded, then by Weak Duality, we have that z = w = ,1. Suppose P is bounded, and let x be an optimal solution, i.e. Ax = b, x 0 and cT x = z. We claim that 9y s.t. AT y c and bT y z. If so we done. are ! AT Suppose no such y exists. Then, by the preceding corollary, with A0 = ,bT , ! ! c , x0 = y, y0 = x , 9x 0, 0 such that 0= b ,z Ax = b and cT x z:
We have two cases Case 1: 6= 0. Since we can normalize by we can assume that = 1. This means that 9x 0 such that Ax = b and cT x z. But this is a contradiction with the optimality of x. Case 2: = 0. This means that 9x 0 such that Ax = 0 and cT x 0. If this is the case then 8 0; x + x is feasible for P and its cost is cT x + x = cT x + cT x z, which is a contradiction. 9.1 Rules for Taking Dual Problems If P is a minimization problem then D is a maximization problem. If P is a maximization problem then D is a minimization problem. In general, using the rules for transforming a linear program into standard form, we have that the dual of P : z = min cT x1 + cT x2 + cT x3 1 2 3 LP-16 s.t. A11x1 + A12x2 + A13x3 = b1 A21x1 + A22x2 + A23x3 b2 A31x1 + A32x2 + A33x3 b3 x1 0 ; x2 0 ; x3 UIS where UIS means unrestricted in sign" to emphasize that no constraint is on the variable is D w = max bT y1 + bT y2 + bT y3 1 2 3 s.t. AT y1 + AT y2 + AT y3 c1 11 21 31 T y + AT y + AT y c A12 1 22 2 32 3 2 T y + AT y + AT y = c A13 1 23 2 33 3 3 y1 UIS ; y2 0 ; y3 0 10 Complementary Slackness
Let P and D be P z = minfcT x : Ax = b; x 0g D w = maxfbT y : AT y cg; and let x be feasible in P , and y be fesible in D. Then, by weak duality, we know that cT x bT y. We call the di erence cT x , bT y the duality gap. Then we have that the duality gap is zero i x is optimal in P , and y is optimal in D. That is, the duality gap can serve as a good measure of how close a feasible x and y are to the optimal solutions for P and D. The duality gap will be used in the description of the interior point method to monitor the progress towards optimality. It is convenient to write the dual of a linear program as D w = maxfbT y : AT y + s = c for some s 0g Then we can write the duality gap as follows: cT x , bT y = cT x , xT AT y = xT c , AT y = xT s since AT y + s = c. The following theorem allows to check optimality of a primal and or a dual solution. LP-17 Theorem 12 Complementary Slackness Proof: Let x, y ; s be feasible for P , D respectively. The following are equivalent: 1. x is an optimal solution to P and y ; s is an optimal solution to D. 2. sT x = 0. 3. xs = 0, 8 j = 1; : : : ; n. j j 4. If s 0 then x = 0. j j Suppose 1 holds, then, by strong duality, cT x = bT y. Since c = AT y + s and Ax = b, we get that yT Ax + sT x = xT AT y, and thus, sT x = 0 i.e 2 holds. It follows, since x, s 0, that xs = 0, 8 j = 1; : : : ; n i.e. 3 holds. j j j j Hence, if s 0 then x = 0, 8 j = 1; : : : ; n i.e. 4 holds. The converse also holds, j j and thus the proof is complete. In the example of section 9, the complementary slackness equations corresponding to the primal solution x = 3; 2; 0T would be: y1 + 2y2 = 1 y1 + y2 = 2 Note that this implies that y1 = 3 and y2 = ,1. Since this solution satis es the other constraint of the dual, y is dual feasible, proving that x is an optimum solution to the primal and therefore y is an optimum solution to the dual. 11 Size of a Linear Program
11.1 Size of the Input
If we want to solve a Linear Program in polynomial time, we need to know what would that mean, i.e. what would the size of the input be. To this end we introduce two notions of the size of the input with respect to which the algorithm we present will run in polynomial time. The rst measure of the input size will be the size of a LP, but we will introduce a new measure L of a LP that will be easier to work with. Moreover, we have that L sizeLP , so that any algorithm running in time polynomial in L will also run in time polynomial in sizeLP. Let's consider the linear program of the form: min cT x s:t: Ax = b x0 LP-18 where we are given as inputs the coe cients of A an m n matrix, b an m 1 vector, and c an n 1 vector, whith rationial entries. We can further assume, without loss of generality, that the given coe cients are all integers, since any LP with rational coe cients can be easily transformed into an equivalent one with integer coe cients just multiply everything by l.c.d.. In the rest of these notes, we assume that A; b; c have integer coe cients. For any integer n, we de ne its size as follows:
4 sizen = 1 + dlog2jnj + 1e where the rst 1 stands for the fact that we need one bit to store the sign of n, sizen represents the number of bits needed to encode n in binary. Analogously, we de ne the size of a p 1 vector d, and of a p l matrix M as follows: 4 sizev = Pp=1 sizevi i 4 Pp Pl sizeM = sizem i=1 j =1 ij De nition 6 Size of a linear program We are then ready to talk about the size of a LP.
4 sizeLP = sizeA + sizeb + sizec: De nition 7
where A more convenient de nition of the size of a linear program is given next.
4 L = sizedetmax + sizebmax + sizecmax + m + n 4 detmax = maxj detA0j A0 4 cmax = maxjcj j j 4 bmax = maxjbij i and A0 is any square submatrix of A. Proposition 13 L sizeLP; 8A; b; c.
Before proving this result, we rst need the following lemma: Lemma 14 1. If n 2 Z then jnj 2sizen,1 , 1. LP-19 2. If v 2 Zn then kvk kv k1 2sizev,n , 1. 3. If A 2 Znn then jdetAj 2sizeA,n2 , 1. Proof:
1. By de nition.
n n n X Y Y 2. 1 + kvk 1 + kvk1 = 1 + jvij 1 + jvij 2sizevi ,1 = 2sizev,n where i=1 i=1 i=1 we have used 1. 3. Let a1; : : : ; an be the columns of A. Since jdetAj represents the volume of the parallelepiped spanned by a1; : : : ; an, we have jdetAj Hence, by 2, 1 + jdetAj 1 +
n Y i=1 n Y n Y i=1 kaik:
n Y i=1 kaik 1 + kaik i=1 2sizeai ,n = 2sizeA,n2 : Proof:
1 2 We now prove Proposition 13. If B is a square submatrix of A then, by de nition, sizeB sizeA. Moreover, by lemma 14, 1 + jdetB j 2sizeB,1 . Hence, dlog1 + jdetB je sizeB , 1 sizeB sizeA:
sizeb + sizec dlog1 + max jcj je + dlog1 + max jbije + m + n: j i Let v 2 Zp. Then sizev sizemaxj jvj j + p , 1 = dlog1 + maxj jvj je + p: Hence, Combining equations 1 and 2, we obtain the desired result. Remark 1 detmax bmax cmax 2m+n 2L , since for any integer n, 2sizen jnj.
In what follows we will work with L as the size of the input to our algorithm. LP-20 ...
View Full Document