Linear Recap: programs in standard form
Math 171A: Linear Programming
Lecture 21 Linear Programs in Standard Form
Philip E. Gill
c 2011
minimize n
xR
c Tx Ax = b , x 0
simple bounds
subject to
equality constraints
The matrix A is m n with shape A = We apply "mixed-constraint" simplex with full matrix A I
http://ccom.ucsd.edu/~peg/math171a
Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 2/44, Friday, February 25th, 2011
Example
2 3 2 1 1 1 0 constraint #1 constraint #2 constraint #3 constraint #4 constraint #5 constraint #6 constraint #7 1 1 = 0 -1 1 1
x1 ,x2 ,x3 ,x4 ,x5
minimize
-6x1 - 9x2 - 5x3 2x1 + 3x2 + x3 + x4 =5 x1 + 2x2 + x3 - x5 = 3 x1 0 x2 0 x3 0 x4 0 x5 0
subject to
A I
1 1
Consider the vertex defined by: the two rows of A rows 3, 4, and 5 of D = I , i.e., rows 5, 6 and 7 of A I
UCSD Center for Computational Mathematics
Slide 3/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 4/44, Friday, February 25th, 2011
A Dk
2 1 =
3 2
1 1 1
constraint #1 0 -1 constraint #2 constraint #5 constraint #6 1 constraint #7 1 1 0
2 1
3 2
1 1 1
1
0
x1
5
0 -1 x 2 x3 x4 1 1 x5
3 =0 0 0
These equations are block upper-triangular, with structure: B 0 N In-m 3 2 xB xN = b 0
This defines a vertex since rank A Dk =5=n with B= 2 1
and N =
1 1
1 0 0 -1
UCSD Center for Computational Mathematics
Slide 5/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 6/44, Friday, February 25th, 2011
2
3 2
1 1 1
1
0
x1
5
1 1 x =0 0 0
1
0 -1 x 2 x3 x4 1 1 x5 and = 5 3
3 =0 0 0
x is a basic solution of Ax = b
Then, x 3 = x 4 = x 5 = 0, 2 3 1 2 x1 x2
Every vertex is like this, i.e., it has n - m zero components and m nonnegative components. x1 x2 = 1 1 e.g., consider the vertex defined by rows 3, 5, and 7 of A . I
UCSD Center for Computational Mathematics
Slide 7/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 8/44, Friday, February 25th, 2011
From the previous slide: 2 3 2 1 1 1 1 0 constraint #1 0 -1 constraint #2 constraint #3 constraint #5 1 constraint #7 x1 5 2 3 2 1 1 1 1 0 x1 5
1 1
1 1
0 -1 x 2 x3 x4 1 x5
3 =0 0 0
2
3 2
1 1 1
1
0
Swap variables x 1 and x 4 . 1 0 3 2 1 1 1 2 0 x4 5
1 1
0 -1 x 2 x3 x4 1 x
5
3 =0 0 0
1 -1 x 2 x3 1 x1 1 x5
3 =0 0 0
UCSD Center for Computational Mathematics
Slide 9/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 10/44, Friday, February 25th, 2011
From the previous slide: 1 3 2 1 1 1 2 0 x4 5 From the previous slide: 1 3 2 1 1 1 2 0 x4 5
0
1 -1 x 2 x3 1 x1 1 x5
3 =0 0 0
0
Swap equations 3 and 4. 1 3 2 1 1 1 2 0 x4 5
1 -1 x 2 x3 x1 1 1 x5 and 5 3
3 =0 0 0
In this case, x 1 = x 3 = x 5 = 0, 1 0 3 2 x4 x2 =
0
1 -1 x 2 x3 x1 1 1 x5
3 =0 0 0
x4 x2
=
1 2 3 2
UCSD Center for Computational Mathematics
Slide 11/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 12/44, Friday, February 25th, 2011
From the previous slide: 1 3 2 1 1 1 x4 1 -1 x 2 x3 x1 1 1 x5 2 0 5 3 =0 0 0
Recap: Finding a basic solution of Ax = b
0
Define a column basis {1 , 2 , . . . , m } Gather the basis B of m linearly independent columns from A: B = a 1 Solve BxB = b for xB Rm . a2 am
0
Scatter xB into a zero x, i.e., xj = x is another basic solution of Ax = b [ xB ]i 0 if aj is the ith column of B; otherwise
3 2 x =0 1 2 0
UCSD Center for Computational Mathematics
Slide 13/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 14/44, Friday, February 25th, 2011
x i = 0 at a vertex x means that the constraint xi 0 is active.
Result
A vertex of the feasible region: F = {x : Ax = b, x 0} is a nonnegative basic solution of Ax = b. In other words, if B is the basis associated with the vertex, then BxB = b with xB 0.
Result
A nondegenerate vertex of the feasible region: F = {x : Ax = b, x 0} is a nonnegative basic solution of Ax = b with precisely n - m zero components. i.e., if B is the basis associated with a nondegenerate vertex, then BxB = b with xB > 0.
UCSD Center for Computational Mathematics
Slide 15/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 16/44, Friday, February 25th, 2011
Result
A degenerate vertex of the feasible region: F = {x : Ax = b, x 0} is a nonnegative basic solution of Ax = b with more than n - m zero components. i.e., if B is the basis associated with a degenerate vertex, then BxB = b with [ xB ]i = 0 for some i. Our aim is to restate the simplex method so that equations of size m m are solved The restated method computes the same sequence of vertices as the simplex method.
UCSD Center for Computational Mathematics
Slide 17/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 18/44, Friday, February 25th, 2011
We assume that A has rank m, i.e., A has independent rows.
A mn
At a vertex we can partition the variables (i.e., columns) into two sets B and N , where:
1
B is the basic set: B = {1 , 2 , . . . , m }
column indices of A
The basis is B = a1
2
a 2
am
N is the nonbasic set N = {1 , 2 , . . . , n-m }
B mm nonsingular
UCSD Center for Computational Mathematics Slide 19/44, Friday, February 25th, 2011
column indices of A not in B
N = a1
a2
an-m is the matrix of columns not in B.
UCSD Center for Computational Mathematics
Slide 20/44, Friday, February 25th, 2011
A variable is either basic (i.e., "free") or nonbasic (at its bound). Variables xi with i B are called basic variables. Variables xi with i N are called nonbasic variables. At a nondegenerate vertex, the terms "basic" and "active" are opposites. xi being basic constraint xi 0 is inactive. active.
The working set for the mixed-constraint simplex method has the form: Ak = A Ik b 0 rows of In
and
bk =
components from 0n
xi being nonbasic xi constraint 0 is
UCSD Center for Computational Mathematics
Slide 21/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 22/44, Friday, February 25th, 2011
For example: 2 1 = 1 3 2 1 1 1 0 0 -1 1 2 3 2 1 1 1 1 0 1 1
column # of A #2
A Ik
0 -1 1
#3 #4
1
#5
#1
Basic set B = {2, 5}, nonbasic set N = {1, 3, 4}. Define the column permutation matrix P such that Ak P = B 0 N In-m
P=
0
0 0 0 0 1
1 0 0 0 0
0 0 1 0 0
0
1 0 0 0
0 0 1 0
UCSD Center for Computational Mathematics
Slide 23/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 24/44, Friday, February 25th, 2011
2
3 2
1 1 1
1
0
0
0 0 0 0 1
1 0 0 0 0
0 0 1 0 0
0
3
0
2 1 1
1 1 1
1
Matlab commands to find the permuted matrix AP. First method (preferred): Perm = [ 2 5 1 3 4 ] Aperm = A(:,Perm); % column list in order B-N % compute the permuted A
1 1
0 -1 1 0 0 1 0
0 2 -1 0 = 1 0
0 1
This gives B= 3 0 2 -1 and N = 2 1 1 1 1 0
Second method: I = eye(5,5); P = I(:,Perm); Aperm = A*P; % define the identity matrix % Define the column permutation matrix: % compute the permuted A
UCSD Center for Computational Mathematics
Slide 25/44, Friday, February 25th, 2011
The column permutation matrix P is such that A Ik P = Ak P = B 0 N In-m
The column permutation applied to a row vector produces a row vector with basic part first.
T c TP = cB T cN
P Tc =
cB cN xB xN pB pN
The permutation moves the basic variables to the front.
T x TP = xB T xN
P Tx =
Result
The matrix Ak is nonsingular if and only if B is nonsingular. Proof: Homework.
T p TP = pB T pN
P Tp =
UCSD Center for Computational Mathematics
Slide 27/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 28/44, Friday, February 25th, 2011
The same goes for matrices: AP = B N and I k P = 0 In-m so that A Ik P= B N 0 IN
=
Result
P is an orthogonal matrix, i.e., P TP = I = PP T .
T Applying the column permutation P T to cB T cN returns c T :
0 IN Also
T cB
T cN = c TP
T cB
T cN P T = c TPP T = c T
P
cB cN
= PP T c = c
UCSD Center for Computational Mathematics
Slide 29/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 30/44, Friday, February 25th, 2011
The simplex multipliers satisfy ATk = c. k The vertex xk defined by the simplex working set satisfies Ak xk = bk Ak PP T xk (Ak P)(P T xk ) B N 0 IN xB xN = bk = bk = b 0 BT NT Multiplying by P T gives P TATk = P Tc k (Ak P)Tk = P Tc 0 k = IN cB cN
If the multipliers are partitioned as k = then zN BT NT multipliers for Ax = b multipliers for x 0 0 IN zN cB cN
BxB + NxN = b xN = 0
BxB = b
Note: to avoid clutter, we don't put a suffix on B or N.
=
UCSD Center for Computational Mathematics
Slide 31/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 32/44, Friday, February 25th, 2011
Observe that BT NT 0 IN zN cB cN P Tz = zB zN = P T(c - AT) = P Tc - P TAT = P Tc - (AP)T c BT = B - cN NT = but B T = cB cB - B T = 0. the reduced costs associated with the basic variables are zero. cB - B T cN - N T
=
B T = cB zN = cN - N T Recall that z = c - AT are the reduced costs. zN are the reduced costs for the nonbasic variables.
UCSD Center for Computational Mathematics
Slide 33/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 34/44, Friday, February 25th, 2011
kth iteration
STEP 1: Check for optimality Solve the equations AT k = c. k Solve B T = cB .
Simplex Method for Standard Form
Compute zN = cN - N T. If [ zN ]i 0 for i = 1, 2, . . . , n - m, then STOP. Otherwise, define [ zN ]s = min(zN ). The s-th nonbasic variable will become a basic variable. N = {1 , 2 , . . . , s , . . . , n-m }
sth element of N
UCSD Center for Computational Mathematics
Slide 36/44, Friday, February 25th, 2011
kth iteration
kth iteration
STEP 2: (continued) Compute the search direction STEP 2: Compute the search direction Solve the equations Ak pk = em+s . B N 0 IN pB pN 0 es Solve BpB + NpN = 0 pN = es = BpB = -NpN = -Nes = -(sth column of N) = -(column s of A) we solve BpB = -as
Increase xs while keeping other nonbasics fixed at 0
UCSD Center for Computational Mathematics
Slide 37/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 38/44, Friday, February 25th, 2011
kth iteration
kth iteration
STEP 3: (continued) Step to an adjacent vertex What happens to the nonbasic variables?
STEP 3: Step to an adjacent vertex If we take a step along the vector pB pN = pB es
Then the new basic variables are: xB + pB all the basic variables change
0 0 . . . . . . 0 0 xN + pN = xN + es = xN + = row s 0 0 . . . . . . 0 0 all nonbasics remain at 0 except xs , which wants to increase.
UCSD Center for Computational Mathematics
Slide 39/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 40/44, Friday, February 25th, 2011
kth iteration
STEP 3: (continued) Step to an adjacent vertex The changed variables are xB and xs (which increases from 0 to ) We must ensure that xB + pB 0 [ xB ]i -[ pB ]i i = + This is known as the min ratio test. Define = min{i } If = +, the LP is unbounded, STOP. if [ pB ]i < 0 if [ pB ]i 0
kth iteration
STEP 3: (continued) Step to an adjacent vertex If = t then [ xB ]t + t [ pB ]t = 0 the t-th basic becomes nonbasic t points to index t of B s points to index s of N [ xB ]t 0 xt goes from basic to nonbasic [ xN ]s xs goes from nonbasic to basic
UCSD Center for Computational Mathematics
Slide 41/44, Friday, February 25th, 2011
UCSD Center for Computational Mathematics
Slide 42/44, Friday, February 25th, 2011
kth iteration
STEP 3: (continued) Step to an adjacent vertex
Summary
B {1 , 2 , . . . , s-1 , s , s+1 , . . . , m }
moved from nonbasic set
Two m m systems are solved: B T = cB BpB = -as Only the quantities , zN , xB pB need be computed
N {1 , 2 , . . . , t-1 , t , t+1 , . . . , n-m }
moved from basic set
B and N exchange indices swap [ xB ]t = 0 and [ xN ]s = [ xB ]t = since xN is not stored (or re-solve BxB = b) column t of B is replaced by column s of A
UCSD Center for Computational Mathematics Slide 43/44, Friday, February 25th, 2011 UCSD Center for Computational Mathematics Slide 44/44, Friday, February 25th, 2011