25: Algorithms
Lecture Linear Programming [Fa '10]
The greatest flood has the soonest ebb; the sorest tempest the most sudden calm; the hottest love the coldest end; and from the deepest desire oftentimes ensues the deadliest hate. -- Socrates Th' extremes of glory and of shame, Like east and west, become the same. -- Samuel Butler, Hudibras Part II, Canto I (c. 1670) Extremes meet, and there is no better example than the haughtiness of humility. -- Ralph Waldo Emerson, "Greatness", in Letters and Social Aims (1876)
25
Linear Programming
The maximum flow/minimum cut problem is a special case of a very general class of problems called linear programming. Many other optimization problems fall into this class, including minimum spanning trees and shortest paths, as well as several common problems in scheduling, logistics, and economics. Linear programming was used implicitly by Fourier in the early 1800s, but it was first formalized and applied to problems in economics in the 1930s by Leonid Kantorovich. Kantorivich's work was hidden behind the Iron Curtain (where it was largely ignored) and therefore unknown in the West. Linear programming was rediscovered and applied to shipping problems in the early 1940s by Tjalling Koopmans. The first complete algorithm to solve linear programming problems, called the simplex method, was published by George Dantzig in 1947. Koopmans first proposed the name "linear programming" in a discussion with Dantzig in 1948. Kantorovich and Koopmans shared the 1975 Nobel Prize in Economics "for their contributions to the theory of optimum allocation of resources". Dantzig did not; his work was apparently too pure. Koopmans wrote to Kantorovich suggesting that they refuse the prize in protest of Dantzig's exclusion, but Kantorovich saw the prize as a vindication of his use of mathematics in economics, which his Soviet colleagues had written off as "a means for apologists of capitalism". A linear programming problem asks for a vector x d that maximizes (or equivalently, minimizes) a given linear function, among all vectors x that satisfy a given set of linear inequalities. The general form of a linear programming problem is the following:
d
maximize
j=1 d
cj x j for each i = 1 .. p for each i = p + 1 .. p + q for each i = p + q + 1 .. n
subject to
j=1 d
ai j x j bi ai j x j = bi
j=1 d
ai j x j bi
j=1
Here, the input consists of a matrix A = (ai j ) nd , a column vector b n , and a row vector c d . Each coordinate of the vector x is called a variable. Each of the linear inequalities is called a constraint.
c Copyright 2010 Jeff Erickson. Released under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License (http://creativecommons.org/licenses/by-nc-sa/3.0/). Free distribution is strongly encouraged; commercial distribution is expressly forbidden. See http://www.cs.uiuc.edu/~jeffe/teaching/algorithms for the most recent revision.
1
Algorithms
Lecture 25: Linear Programming [Fa '10]
The function x c x is called the objective function. I will always use d to denote the number of variables, also known as the dimension of the problem. The number of constraints is usually denoted n. A linear programming problem is said to be in canonical form1 if it has the following structure:
d
maximize
j=1 d
cj x j for each i = 1 .. n for each j = 1 .. d
subject to
j=1
ai j x j bi xj 0
We can express this canonical form more compactly as follows. For two vectors x = (x 1 , x 2 , . . . , x d ) and y = ( y1 , y2 , . . . , yd ), the expression x y means that x i yi for every index i. max c x s.t. Ax b x0 Any linear programming problem can be converted into canonical form as follows: For each variable x j , add the equality constraint x j = x + - x - and the inequalities x + 0 and j j j x - 0. j Replace any equality constraint j ai j x j bi . Replace any upper bound
j j
ai j x j = bi with two inequality constraints
j
ai j x j bi and
ai j x j bi with the equivalent lower bound
j
-ai j x j -bi .
This conversion potentially double the number of variables and the number of constraints; fortunately, it is rarely necessary in practice. Another useful format for linear programming problems is slack form2 , in which every inequality is of the form x j 0: max c x s.t. Ax = b x0 It's fairly easy to convert any linear programming problem into slack form. Slack form is especially useful in executing the simplex algorithm (which we'll see in the next lecture).
25.1
The Geometry of Linear Programming
A point x d is feasible with respect to some linear programming problem if it satisfies all the linear constraints. The set of all feasible points is called the feasible region for that linear program. The feasible region has a particularly nice geometric structure that lends some useful intuition to the linear programming algorithms we'll see later.
1 2
Confusingly, some authors call this standard form. Confusingly, some authors call this standard form.
2
Algorithms
Lecture 25: Linear Programming [Fa '10]
Any linear equation in d variables defines a hyperplane in d ; think of a line when d = 2, or a plane when d = 3. This hyperplane divides d into two halfspaces; each halfspace is the set of points that satisfy some linear inequality. Thus, the set of feasible points is the intersection of several hyperplanes (one for each equality constraint) and halfspaces (one for each inequality constraint). The intersection of a finite number of hyperplanes and halfspaces is called a polyhedron. It's not hard to verify that any halfspace, and therefore any polyhedron, is convex--if a polyhedron contains two points x and y, then it contains the entire line segment x y.
A two-dimensional polyhedron (white) defined by 10 linear inequalities.
By rotating d (or choosing a coordinate frame) so that the objective function points downward, we can express any linear programming problem in the following geometric form: Find the lowest point in a given polyhedron. With this geometry in hand, we can easily picture two pathological cases where a given linear programming problem has no solution. The first possibility is that there are no feasible points; in this case the problem is called infeasible. For example, the following LP problem is infeasible: maximize x - y subject to 2x + y 1 x+ y 2 x, y 0
An infeasible linear programming problem; arrows indicate the constraints.
The second possibility is that there are feasible points at which the objective function is arbitrarily large; in this case, we call the problem unbounded. The same polyhedron could be unbounded for some objective functions but not others, or it could be unbounded for every objective function.
3
Algorithms
Lecture 25: Linear Programming [Fa '10]
A two-dimensional polyhedron (white) that is unbounded downward but bounded upward.
25.2
Example 1: Shortest Paths
We can compute the length of the shortest path from s to t in a weighted directed graph by solving the following very simple linear programming problem. maximize subject to dt ds = 0 d v - du
u v
for every edge u v
Here, u v is the length of the edge u v. Each variable d v represents a tentative shortest-path distance from s to v. The constraints mirror the requirement that every edge in the graph must be relaxed. These relaxation constraints imply that in any feasible solution, d v is at most the shortest path distance from s to v. Thus, somewhat counterintuitively, we are correctly maximizing the objective function to compute the shortest path! In the optimal solution, the objective function d t is the actual shortest-path distance from s to t, but for any vertex v that is not on the shortest path from s to t, d v may be an underestimate of the true distance from s to v. However, we can obtain the true distances from s to every other vertex by modifying the objective function: maximize subject to
v
dv ds = 0
u v
d v - du
for every edge u v
There is another formulation of shortest paths as an LP minimization problem using an indicator variable x u v for each edge u v. minimize
u v u v
xu xs
v
subject to
u
xu xu
u
s
-
w
w
=1 = -1 =0 0 for every vertex v = s, t for every edge u v
t
-
w
xt xv
w
w
xu
u
v
-
w v
xu
Intuitively, x u v = 1 means u v lies on the shortest path from s to t, and x u v = 0 means u v does not lie on this shortest path. The constraints merely state that the path should start at s, end at t, and either pass through or avoid every other vertex v. Any path from s to t--in particular, the shortest path--clearly implies a feasible point for this linear program. 4
Algorithms
Lecture 25: Linear Programming [Fa '10]
However, there are other feasible solutions, possibly even optimal solutions, with non-integral values that do not represent paths. Nevertheless, there is always an optimal solution in which every x e is either 0 or 1 and the edges e with x e = 1 comprise the shortest path. (This fact is by no means obvious, but a proof is beyond the scope of these notes.) Moreover, in any optimal solution, even if not every x e is an integer, the objective function gives the shortest path distance!
25.3
Example 2: Maximum Flows and Minimum Cuts
Recall that the input to the maximum (s, t)-flow problem consists of a weighted directed graph G = (V, E), two special vertices s and t, and a function assigning a non-negative capacity ce to each edge e. Our task is to choose the flow f e across each edge e, as follows: maximize
w
fs fv
w
w
-
u
fu fu
u
s
subject to
w
-
v v v
=0 cu 0
v
for every vertex v = s, t for every edge u v for every edge u v
fu fu
Similarly, the minimum cut problem can be formulated using `indicator' variables similarly to the shortest path problem. We have a variable S v for each vertex v, indicating whether v S or v T , and a variable X u v for each edge u v, indicating whether u S and v T , where (S, T ) is some (s, t)-cut.3 minimize
u v
cu Xu
v
v
Xu
v
subject to
+ S v - Su 0 Xu
v
for every edge u v for every edge u v
0
Ss = 1 St = 0 Like the minimization LP for shortest paths, there can be optimal solutions that assign fractional values to the variables. Nevertheless, the minimum value for the objective function is the cost of the minimum cut, and there is an optimal solution for which every variable is either 0 or 1, representing an actual minimum cut. No, this is not obvious; in particular, my claim is not a proof!
25.4
Linear Programming Duality
Each of these pairs of linear programming problems is related by a transformation called duality. For any linear programming problem, there is a corresponding dual linear program that can be obtained by a mechanical translation, essentially by swapping the constraints and the variables. The translation is simplest when the LP is in canonical form: Primal () max c x s.t. Ax b x0
3
Dual ( )
min y b s.t. yA c y0
These two linear programs are not quite syntactic duals; I've added two redundant variables Ss and S t to the min-cut program to increase readability.
5
Algorithms
Lecture 25: Linear Programming [Fa '10]
We can also write the dual linear program in exactly the same canonical form as the primal, by swapping the coefficient vector c and the objective vector b, negating both vectors, and replacing the constraint matrix A with its negative transpose.4 Primal () max c x s.t. Ax b x 0 Dual ( )
max -b y s.t. -A y -c y 0
Written in this form, it should be immediately clear that duality is an involution: The dual of the dual linear program is identical to the primal linear program . The choice of which LP to call the `primal' and which to call the `dual' is totally arbitrary.5 The Fundamental Theorem of Linear Programming. A linear program has an optimal solution x if and only if the dual linear program has an optimal solution y such that c x = y Ax = y b. The weak form of this theorem is trivial to prove. Weak Duality Theorem. If x is a feasible solution for a canonical linear program and y is a feasible solution for its dual , then c x yAx y b. Proof: Because x is feasible , for we have Ax b. Since y is positive, we can multiply both sides of the inequality to obtain yAx y b. Conversely, y is feasible for and x is positive, so yAx c x. It immediately follows that if c x = y b, then x and y are optimal solutions to their respective linear programs. This is in fact a fairly common way to prove that we have the optimal value for a linear program.
25.5
Duality Example
Before I prove the stronger duality theorem, let me first provide some intuition about where this duality thing comes from in the first place.6 Consider the following linear programming problem: maximize subject to 4x 1 + x 2 + 3x 3 x 1 + 4x 2 2 3x 1 - x 2 + x 3 4 x1, x2, x3 0 Let denote the optimum objective value for this LP The feasible solution x = (1, 0, 0) gives us a lower . bound 4. A different feasible solution x = (0, 0, 3) gives us a better lower bound 9. We could play this game all day, finding different feasible solutions and getting ever larger lower bounds. How do we know when we're done? Is there a way to prove an upper bound on ?
For the notational purists: In these formulations, x and b are column vectors, and y and c are row vectors. This is a somewhat nonstandard choice. Yes, that means the dot in c x is redundant. Sue me. 5 For historical reasons, maximization LPs tend to be called `primal' and minimization LPs tend to be called `dual'. This is a pointless religious tradition, nothing more. Duality is a relationship between LP problems, not a type of LP problem. 6 This example is taken from Robert Vanderbei's excellent textbook Linear Programming: Foundations and Extensions [Springer, 2001], but the idea appears earlier in Jens Clausen's 1997 paper `Teaching Duality in Linear Programming: The Multiplier Approach'.
4
6
Algorithms
Lecture 25: Linear Programming [Fa '10]
In fact, there is. Let's multiply each of the constraints in our LP by a new non-negative scalar value yi : maximize subject to 4x 1 + x 2 + 3x 3 ) 2 y1
y1 ( x 1 + 4x 2
y2 (3x 1 - x 2 + x 3 ) 4 y2 x1, x2, x3 0 Because each yi is non-negative, we do not reverse any of the inequalities. Any feasible solution (x 1 , x 2 , x 3 ) must satisfy both of these inequalities, so it must also satisfy their sum: ( y1 + 3 y2 )x 1 + (4 y1 - y2 )x 2 + y2 x 3 2 y1 + 4 y2 . Now suppose that each yi is larger than the ith coefficient of the objective function: y1 + 3 y2 4, 4 y1 - y2 1, y2 3.
This assumption lets us derive an upper bound on the objective value of any feasible solution: 4x 1 + x 2 + 3x 3 ( y1 + 3 y2 )x 1 + (4 y1 - y2 )x 2 + y2 x 3 2 y1 + 4 y2 . ()
In particular, by plugging in the optimal solution (x 1 , x 2 , x 3 ) for the original LP we obtain the following , upper bound on : = 4x 1 + x 2 + 3x 3 2 y1 + 4 y2 .
Now it's natural to ask how tight we can make this upper bound. How small can we make the expression 2 y1 + 4 y2 without violating any of the inequalities we used to prove the upper bound? This is just another linear programming problem. minimize subject to 2 y1 + 4 y2 y1 + 3 y2 4 4 y1 - y2 1 y2 3 y1 , y2 0 In fact, this is precisely the dual of our original linear program! Moreover, inequality () is just an instantiation of the Weak Duality Theorem.
25.6
Strong Duality
The Fundamental Theorem can be rephrased in the following form: Strong Duality Theorem. If x is an optimal solution for a canonical linear program , then there is an optimal solution y for its dual , such that c x = y Ax = y b. Proof (sketch): I'll prove the theorem only for non-degenerate linear programs, in which (a) the optimal solution (if one exists) is a unique vertex of the feasible region, and (b) at most d constraint hyperplanes pass through any point. These non-degeneracy assumptions are relatively easy to enforce in practice and can be removed from the proof at the expense of some technical detail. I will also prove the theorem only for the case n d; the argument for under-constrained LPs is similar (if not simpler). 7
Algorithms
Lecture 25: Linear Programming [Fa '10]
To develop some intuition, let's first consider the very special case where x = (0, 0, . . . , 0). Let ei denote the ith standard basis vector, whose ith coordinate is 1 and all other coordinates are 0. Because x i = 0 for all i, our non-degeneracy assumption implies the strict inequality ai x < bi for all i. Thus, any sufficiently small ball around the origin does not intersect any other constraint hyperplane ai x = bi . Thus, for all i, and for any sufficiently small > 0, the vector ei is feasible. Because x is the unique optimum, we must have ci = c (ei ) < c x = 0. We conclude that ci < 0 for all i. Now let y = (0, 0, . . . , 0) as well. We immediately observe that yA c and y 0; in other words, y is a feasible solution for the dual linear program . But y b = 0 = c x , so the weak duality theorem implies that y is an optimal solution to , and the proof is complete for this very special case. Now let us consider the more general case. Let x be the optimal solution for the linear program ; our non-degeneracy assumption implies that this solution is unique, and that exactly d of the n linear constraints are satisfied with equality. Without loss of generality (by permuting the constraints and possibly changing coordinates), we can assume that these are the first d constraints. Thus, we have ai x = bi ai x < bi
for all i d, for all i d + 1,
where ai denotes the ith row of A. Let A denote the d d matrix containing the first d rows of A. Our non-degeneracy assumption implies that A has full rank, and thus has a well-defined inverse V = A-1 . Now define a vector y n by setting y j := c v j y j := 0 for all j d, for all j d + 1,
where v j denotes the jth column of V = A-1 . Note that ai v j = 0 if i = j, and ai v j = 1 if i = j. To simplify notation, let y = ( y1 , y2 , . . . , yd ) and let b = (b1 , b2 , . . . , bd ) = A x . Because yi = 0 for all i d + 1, we immediately have y b = y b = cV b = cA-1 b = c x and yA = y A = cVA = cA-1 A = c. The point x lies on exactly d constraint hyperplanes; moreover, any sufficiently small ball around x ~ intersects only those d constraint hyperplanes. Consider the point x = x - v j , for some index ~ 1 j d and some sufficiently small > 0. We have ai x = ai x - (ai v j ) = bi for all i = j, and ~ ~ a j x = a j x - (a j v j ) = b j - < b j . Thus, x is a feasible point for . Because x is the unique ~ optimum for , we must have c x = c x - (c v j ) < c x . We conclude that y j = c v j > 0 for all j. We have shown that yA c and y 0, so y is a feasible solution for the dual linear program . We have also shown that y b = c x , so by the Weak Duality Theorem, y is also an optimal solution for , and the proof is complete! We can also give a useful geometric interpretation to the vector y d . Each linear equation ai x = bi defines a hyperplane in d with normal vector ai . The normal vectors a1 , . . . , ad are linearly independent (by non-degeneracy) and therefore describe a coordinate frame for the vector space d . d The definition of y implies that c = y A = i=1 yi ai . In other words, y lists the coefficients of the objective vector c in the coordinate frame a1 , . . . , ad .
8
Algorithms
Lecture 25: Linear Programming [Fa '10]
25.7
Complementary Slackness
Complementary Slackness Theorem. Let x be an optimal solution to a canonical linear program , and let y be an optimal solution to its dual . Then for every index i, we have yi > 0 if and only if ai x = bi . Symmetrically, for every index j, we have x > 0 if and only if y a j = c j . j
To be written
Exercises
1. (a) Describe how to transform any linear program written in general form into an equivalent linear program written in slack form.
d
maximize
j=1 d
cj x j ai j x j bi
j=1 d j=1 d
subject to
for each i = 1 .. p = for each i = p + 1 .. p + q for each i = p + q + 1 .. n
ai j x j = bi ai j x j bi
max c x s.t. Ax = b x0
j=1
(b) Describe precisely how to dualize a linear program written in slack form. (c) Describe precisely how to dualize a linear program written in general form: In all cases, keep the number of variables in the resulting linear program as small as possible. 2. A matrix A = (ai j ) is skew-symmetric if and only if a ji = -ai j for all indices i = j; in particular, every skew-symmetric matrix is square. A canonical linear program max{c x | Ax b; x 0} is self-dual if the matrix A is skew-symmetric and the objective vector c is equal to the constraint vector b. (a) Prove that any self-dual linear program is syntactically equivalent to its dual program . (b) Show that any linear program with d variables and n constraints can be transformed into a self-dual linear program with n + d variables and n + d constraints. The optimal solution to the self-dual program should include both the optimal solution for (in d of the variables) and the optimal solution for the dual program (in the other n variables). 3. (a) Give a linear-programming formulation of the maximum-cardinality bipartite matching problem. The input is a bipartite graph G = (U V ; E), where E U V ; the output is the largest matching in G. Your linear program should have one variable for each edge. (b) Now dualize the linear program from part (a). What do the dual variables represent? What does the objective function represent? What problem is this!? 4. Give a linear-programming formulation of the minimum-cost feasible circulation problem. Here you are given a flow network whose edges have both capacities and costs, and your goal is to find a feasible circulation (flow with value 0) whose cost is as small as possible. 9
Algorithms
Lecture 25: Linear Programming [Fa '10]
5. An integer program is a linear program with the additional constraint that the variables must take only integer values. (a) Prove that deciding whether an integer program has a feasible solution is NP-complete. (b) Prove that finding the optimal feasible solution to an integer program is NP-hard. [Hint: Almost any NP-hard decision problem can be formulated as an integer program. Pick your favorite.] 6. Helly's theorem states that for any collection of convex bodies in d , if every d + 1 of them intersect, then there is a point lying in the intersection of all of them. Prove Helly's theorem for the special case where the convex bodies are halfspaces. Equivalently, show that if a system of linear inequalities Ax b does not have a solution, then we can select d + 1 of the inequalities such that the resulting subsystem also does not have a solution. [Hint: Construct a dual LP from the system by choosing a 0 cost vector.] 7. Given points (x 1 , y1 ), (x 2 , y2 ), . . . , (x n , yn ) in the plane, the linear regression problem asks for real numbers a and b such that the line y = a x + b fits the points as closely as possible, according to some criterion. The most common fit criterion is minimizing the L2 error, defined as follows:7
n 2 (a, b) = i=1
( yi - a x i - b)2 .
But there are several other fit criteria, some of which can be optimized via linear programming. (a) The L1 error (or total absolute deviation) of the line y = a x + b is defined as follows:
n 1 (a, b) = i=1
yi - a x i - b .
Describe a linear program whose solution (a, b) describes the line with minimum L1 error. (b) The L error (or maximum absolute deviation) of the line y = a x + b is defined as follows:
(a, b)
= max yi - a x i - b .
i=1
n
Describe a linear program whose solution (a, b) describes the line with minimum L error.
7 This measure is also known as sum of squared residuals, and the algorithm to compute the best fit is normally called (ordinary/linear) least squares fitting.
c Copyright 2010 Jeff Erickson. Released under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License (http://creativecommons.org/licenses/by-nc-sa/3.0/). Free distribution is strongly encouraged; commercial distribution is expressly forbidden. See http://www.cs.uiuc.edu/~jeffe/teaching/algorithms for the most recent revision.
10