Optimality Recap: conditions for LP
Math 171A: Linear Programming
Lecture 14 Farkas' Lemma and its Implications
Philip E. Gill
c 2011
LP
minimize
x
c Tx Ax b
subject to
with A an m n matrix, b an m-vector and c an n-vector. The vector x is a solution of LP if and only if : (a) Ax b (b) c = AT for some vector 0, where Aa is the active a a a constraint matrix at x Monday, February 7th, 2011
http://ccom.ucsd.edu/~peg/math171a
UCSD Center for Computational Mathematics
Slide 2/41, Monday, February 7th, 2011
x2
Example: minimize 2x1 + x2 subject to the constraints:
constraint #1: constraint #2: constraint #3:
x1 + x2 1 x2 0 x1 0
a1 c a3
Written in the form min c Tx subject 1 2 c= , A=0 1 1
to Ax b, we get 1 1 b=0 1 , 0 0
x
a2 a1 c
x
x1
UCSD Center for Computational Mathematics
Slide 3/41, Monday, February 7th, 2011
At the point x = with Aa = 1 1 1 0 , ba = 1 0 0 1 , the active set is A = {1, 3}
At the point x= with Aa = 1 0 1 1 , ba = 1 0 1 0 , the active set is A = {1, 2}
Solving for the Lagrange multipliers gives AT a = c a x is optimal. 1 1 1 0 a = 2 1 a = 1 1 0
Solving for the Lagrange multipliers gives AT a = c a 1 1 0 1 a = 2 1 a = 2 -1 0
x is not optimal.
UCSD Center for Computational Mathematics
Slide 5/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 6/41, Monday, February 7th, 2011
Proof of Farkas' lemma
Proof: We have two statements that we must show are equivalent:
Result (Farkas' Lemma)
(A) c Tp 0 for all p such that Aa p 0 if and only if (B) c = AT for some 0 a a a
(A) (B)
c Tp 0 for all p such that AT p 0 a c = AT for some 0 a a a
We must show that (A) (B) and (B) (A)
UCSD Center for Computational Mathematics
Slide 7/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 8/41, Monday, February 7th, 2011
Next we must show that (A) (B). It is "EASY" to show that (B) (A) If (B) holds, then c = Then, c p=
T
This is the same as showing that: 0. If (B) does not hold, then (A) does not hold. We must show that
AT a a
for some
a
(AT )Tp a a
=
( )T (Aa p) a
0 for all p with Aa p 0
(B) is not true (B) not true (A) not true
(A) is not true
c Tp 0 for all p with Aa p 0 (A) holds.
there is no 0 such that c = AT a a a c Tp < 0 for some p such that Aa p 0
UCSD Center for Computational Mathematics
Slide 9/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 10/41, Monday, February 7th, 2011
We must show that there is no 0 such that c = AT a a a c Tp < 0 for some p such that Aa p 0 Two possibilities for there being no 0 with c = AT . a a a Case I: Case II: c does not lie in range(AT ) a c range(AT ), but c = AT with 0 a a a a Case I: c does not lie in range(AT ) a Every c Rn can be written uniquely as c = cR + cN with cR range(AT ) and cN null(Aa ) a
T The definition of cR and cN imply that cR cN = 0.
If c range(AT ), then a c = cR + cN with cN = 0
In both cases we construct a p such that c Tp < 0 and Aa p 0.
UCSD Center for Computational Mathematics
Slide 11/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 12/41, Monday, February 7th, 2011
Define p = -cN . Then Aa p = -Aa cN = 0 since cN null(Aa ). c Tp = -c TcN = -(cR + cN )TcN
T T = -cR cN - cN cN
Case II: c range(AT ), but c = AT with 0 a a a a This is the "HARD" part of Farkas' lemma. However, it is "quite EASY" if we assume that the rows of Aa are linearly independent.
= -cN cN
T
= - cN
2
<0 Aa = with linearly independent rows.
p = -cN satisfies Aa p 0 and c Tp < 0. (A) does not hold.
UCSD Center for Computational Mathematics
Slide 13/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 14/41, Monday, February 7th, 2011
Case II: c range(AT ), but c = AT with 0 a a a a In this case, exists, but ( )s < 0 for some index s. a a Let p be a solution of Aa p = es , where es is the unit vector 0 . . . 0 es = 1 row s 0 . . . 0 The equations Aa p = es are compatible (why?). Moreover, p is a feasible descent direction, as we now show: c Tp = (AT a )Tp = (T Aa )p = T (Aa p) = T es = (a )s < 0 a a a a Hence we have constructed a direction p such that c Tp < 0 and we are done! and Aa p ( = es ) 0
UCSD Center for Computational Mathematics
Slide 15/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 16/41, Monday, February 7th, 2011
Implications of Farkas' Lemma
In the proof we defined a feasible descent direction p such that Aa p = es where s is an index such that ( )s < 0 a
The direction p satisfies:
Result (Farkas' Lemma)
(A) c Tp 0 for all p such that Aa p 0 if and only if (B) c = AT for some 0 a a a ajTp = 0 for j = s, with ajT the jth row of Aa
T 1 for j = s, with as the sth row of Aa
A x step + p keeps all active constraint residuals fixed at zero except the sth, which increases. i.e., x + p "stays on" all the active constraints and "moves off" of a constraint with a negative component of a .
UCSD Center for Computational Mathematics
Slide 17/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 18/41, Monday, February 7th, 2011
x2 a1 c a3
At the point x= with Aa =
a2 a1 c
1 0
,
the active set is
A = {1, 2}
x
1 0
1 1
,
ba =
1 0
Solving for the Lagrange multipliers gives AT a = c a
x1
x
1 1
0 1
a =
2 1
a =
2 -1
UCSD Center for Computational Mathematics
Slide 20/41, Monday, February 7th, 2011
x2
a =
2 -1
s=2
We can construct a feasible descent direction p such that Aa p = e2 1 0 1 1 p= 0 1 p= -1 1
a2 p a1 c
Observe that
T a1 p = 0, T a2 p = 1,
c Tp = 2 1
-1 1
= -1
x1
p is a feasible descent direction.
we "move off" constraint #2, but "stay on" constraint #1.
UCSD Center for Computational Mathematics Slide 21/41, Monday, February 7th, 2011
Complications of degeneracy
Unfortunately, defining a feasible descent direction from Farkas' Lemma is known as a theorem of the alternative EITHER OR c = AT for some 0 a a a there is a p such that c Tp < 0 and Aa p 0 Aa p = es will not work if the active constraints are dependent, i.e., Aa = i.e., if Aa defines a degenerate vertex.
In this case, Aa p = es is incompatible and no p exists.
UCSD Center for Computational Mathematics
Slide 23/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 24/41, Monday, February 7th, 2011
Example: minimize 2x1 + x2 subject to the constraints:
constraint #1: constraint #2: constraint #3: constraint #4:
x2
a4
x1 + x2 1 x2 0 x1 0 x1 + 2x2 1 to Ax b, we get 1 1 0 1 , b= 0 0 2 1
x
a2
Written in the form min
c Tx
c=
2 1
,
subject 1 0 A= 1 1
a1
c
x
x1
UCSD Center for Computational Mathematics
Slide 25/41, Monday, February 7th, 2011
At the point x= with 1 0 The equations Aa p = e3 are , the active set is 1 1 , 2 A = {1, 2, 4} 1 Aa = 0 1 p1 + p2 = 0 p2 = 0 p1 + 2p2 = 1 Yet p = -1 1 1 1 2 incompatible equations! p1 p2 0 =0 1
1 Aa = 0 1
1 ba = 0 1
Solving AT a = c for the Lagrange multipliers gives a 1 1 0 1 1 2 2 1 3 a = 0 -1 s=3
a =
is still a feasible descent direction.
UCSD Center for Computational Mathematics
Slide 27/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 28/41, Monday, February 7th, 2011
x2
a4
x
-1 p= 1
a2
a1
c
There is no direction that moves off one active constraint while staying on all the others.
x
x1
UCSD Center for Computational Mathematics
Slide 30/41, Monday, February 7th, 2011
Geometric ideas behind Farkas' proof
Consider the active-set matrix at x = 1 Aa = 0 1 i.e., a1 = 1 1 , a2 = 0 1 and a4 = 1 2 1 1 2 1 0 2 1
y2
a4
with
c=
a2
a1
c
In this case c CN (AT ), so the corresponding x is not optimal. a
y1
UCSD Center for Computational Mathematics Slide 31/41, Monday, February 7th, 2011
y2
a4
As CN (AT ) is convex, we can draw a line through the origin that a separates every point y CN (AT ) from c. a This line has the equation p Ty = 0 with normal vector p. i.e., p1 y1 + p2 y2 = 0
p
p1 y1 + p2 y2 = 0
a2
a1
c
y1
UCSD Center for Computational Mathematics Slide 33/41, Monday, February 7th, 2011
The hyperplane splits Rn into two half-spaces: In the general case, p Ty = 0 separates p Ty H1 = {y : p Ty 0} = 0 is called a separating hyperplane H2 = {y : p Ty < 0} from c. With the vector p pointing "into" the half-space H1 . By definition, we have CN (AT ) H1 a c H2
CN (AT ) a
The existence of this hyperplane is the "beef" of Farkas' lemma (see the class notes).
UCSD Center for Computational Mathematics
Slide 35/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 36/41, Monday, February 7th, 2011
y2
a4
p
H1 = {y : p Ty 0} H2 = {y : p Ty < 0}
H1 a2 a1 H2
p1 y1 + p2 y2 = 0
Since CN (AT ) H1 , any y CN (AT ) satisfies p Ty 0. a a Now a1 , a2 and a4 must be in CN (AT ), so that a
T a1 CN (AT ) a1 p 0 a T a2 CN (AT ) a2 p 0 a T a4 CN (AT ) a4 p 0 a
c
y1
Aa p 0
UCSD Center for Computational Mathematics Slide 38/41, Monday, February 7th, 2011
H1 = {y : p Ty 0} H2 = {y : p Ty < 0} Since c H2 , it must hold that p Tc < 0 (i.e., c Tp < 0 ) Putting all this together implies that the normal vector p satisfies c Tp < 0 and Aa p 0 One particular separating hyperplane may be found by finding the point in CN (AT ) that is closest to c. a Then, define p = cp - c
UCSD Center for Computational Mathematics
Slide 39/41, Monday, February 7th, 2011
UCSD Center for Computational Mathematics
Slide 40/41, Monday, February 7th, 2011
y2
a4
p
H1 a2 a1 H2
cp c
y1