Duality 4.pdf - Linear Programming Leo Liberti \u00b4 LIX Ecole Polytechnique [email protected] Operations research courses LP theory \u2013 p 1

Duality 4.pdf - Linear Programming Leo Liberti u00b4 LIX...

This preview shows page 1 out of 47 pages.

Unformatted text preview: Linear Programming Leo Liberti ´ LIX, Ecole Polytechnique [email protected] Operations research courses / LP theory – p. 1 Contents LP formulations and examples The simplex method Optimality conditions Duality Operations research courses / LP theory – p. 2 Lecture material Lecture notes: ˜liberti/ isic/isc612-07/linear programming.pdf J.-B. Hiriart-Urruty, Optimisation et analyse convexe, PUF, Paris 1998 (Ch. 5) C. Papadimitriou, K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity, Dover, New York, 1998 Operations research courses / LP theory – p. 3 Definitions Mathematical programming formulation: ) minx f (x) [P ] s.t. g(x) ≤ 0 (1) A point x∗ is feasible in P if g(x∗ ) ≤ 0; F (P ) = set of feasible points of P A feasible x∗ is a local minimum if ∃B(x∗ , ε) s.t. ∀x ∈ F (P ) ∩ B(x∗ , ε) we have f (x∗ ) ≤ f (x) A feasible x∗ is a global minimum if ∀x ∈ F (P ) we have f (x∗ ) ≤ f (x) Thm.: if f and F (P ) convex, any local min. is also global If gi (x∗ ) = 0 for some i, gi is active at x∗ Operations research courses / LP theory – p. 4 Canonical form P is a linear programming problem (LP) if f : Rn → R, g : Rn → Rm are linear forms LP in canonical form: minx cT x s.t. Ax ≤ b [C] x≥0 (2) Can reformulate inequalities to equations by adding a non-negative slack variable xn+1 ≥ 0: n X j=1 aj xj ≤ b ⇒ n X aj xj + xn+1 = b ∧ xn+1 ≥ 0 j=1 Operations research courses / LP theory – p. 5 Standard form LP in standard form: all inequalities transformed to equations T minx (c′ ) x s.t. A′ x = b [S] x≥0 (3) where x = (x1 , . . . , xn , xn+1 , . . . , xn+m ), A′ = (A, Im ), c′ = (c, 0, . . . , 0) | {z } m Standard form useful because linear systems of equations are computationally easier to deal with than systems of inequalities Used in simplex algorithm Operations research courses / LP theory – p. 6 Diet problem I Consider set M of m nutrients (e.g. sugars, fats, carbohydrates, proteins, vitamins, . . . ) Consider set N of n types of food (e.g. pasta, steak, potatoes, salad, ham, fruit, . . . ) A diet is healthy if it has at least bi units of nutrient i ∈ M Food j ∈ N contains aij units of nutrient i ∈ M A unit of food j ∈ N costs cj Find a healthy diet of minimum cost Operations research courses / LP theory – p. 7 Diet problem II Parameters: m × n matrix A = (aij ), b = (b1 , . . . , bm ), c = (c1 , . . . , cn ) Decision variables: xj = quantity of food j in the diet Objective function: min x Constraints: ∀i ∈ M n P cj xj j=1 n P aij xj ≥ bi j=1 Limits on variables: ∀j ∈ N xj ≥ 0 Canonical form: min{cT x | − Ax ≤ −b} Standard form: add slack variables yi = surplus quantity of i-th nutrient, get min{cT x | − Ax + Im y = −b} Operations research courses / LP theory – p. 8 Geometry of LP A polyhedron is the intersection of a finite number of closed halfspaces. A bounded, non-empty polyhedron is a polytope x2 11111111111111111 00000000000000000 row 1 Q 111111111111 000000000000 2x1 + x2 ≤ 2 00000000000000000 11111111111111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 R 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 x1 + 2x2 ≤ 2 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 000000 111111 row 2 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 000000000000 x1 P 111111111111 S 000000 111111 000000 111111 F (C) = n | Ax ≤ b ∧ x ≥ 0} {x ∈ R 0 1 Canonical feas. polyhedron: [email protected] 1 2 2 1 A, bT = (2, 2) F (S) = | Ax + Im y = Standard feas. polyhedron: {(x, y) ∈ Rn+m b ∧ (x, y) ≥ 0} P = (0, 0, 2, 2), Q = (0, 1, 0, 1), R = ( 23 , 23 , 0, 0), S = (1, 0, 1, 0) Each vertex corresponds to an intersection of at least n hyperplanes ⇒ ≥ n coordinates are zero Operations research courses / LP theory – p. 9 Basic feasible solutions Consider polyhedron in “equation form” K = {x ∈ Rn | Ax = b ∧ x ≥ 0}. A is m × n of rank m (N.B. n here is like n + m in last slide!) A subset of m linearly independent columns of A is a basis of A If β is the set of column indices of a basis of A, variables xi are basic for i ∈ β and nonbasic for i 6∈ β Partition A in a square m × m nonsingular matrix B (columns indexed by β ) and an (n − m) × m matrix N Write A = (B|N ), xB ∈ Rm basics, xN ∈ Rn−m nonbasics Given a basis (B|N ) of A, the vector x = (xB , xN ) is a basic feasible solution (bfs) of K with respect to the given basis if Ax = b, xB ≥ 0 and xN = 0 Operations research courses / LP theory – p. 10 Fundamental Theorem of LP Given a non-empty polyhedron K in “equation form”, there is a surjective mapping between bfs and vertices of K For any c ∈ Rn , either there is at least one bfs that solves the LP min{cT x |x ∈ K}, or the problem is unbounded Proofs not difficult but long (see lecture notes or Papadimitriou and Steiglitz) Important correspondence between bfs’s and vertices suggests geometric solution method based on exploring vertices of K Operations research courses / LP theory – p. 11 Simplex Algorithm: Summary Solves LPs in form min cT x where K = {Ax = b ∧ x ≥ 0} x∈K Starts from any vertex x Moves to an adjacent improving vertex x′ (i.e. x′ is s.t. ∃ edge {x, x′ } in K and cT x′ ≤ cT x) Two bfs’s with basic vars indexed by sets β, β ′ correspond to adjacent vertices if |β ∩ β ′ | = m − 1 Stops when no such x′ exists Detects unboundedness and prevents cycling ⇒ convergence K convex ⇒ global optimality follows from local optimality at termination Operations research courses / LP theory – p. 12 Simplex Algorithm I Let x = (x1 , . . . , xn ) be the current bfs, write Ax = b as BxB + N xN = b Express basics in terms of nonbasics: xB = B −1 b − B −1 N xN (this system is a dictionary) Express objective function in terms of nonbasics: cT x = cTB xB + cTN xN = cTB (B −1 b − B −1 N xN ) + cTN xN ⇒ ⇒ cT x = cTB B −1 b + c¯TN xN (c¯TN = cTN − cTB B −1 N are the reduced costs) Select an improving direction: choose a nonbasic variable xh with negative reduced cost; increasing its value will decrease the objective function value If no such h exists, no improving direction, local minimum ⇒ global minimum ⇒ termination Operations research courses / LP theory – p. 13 Simplex Algorithm II Iteration start: xh is out of basis ⇒ its value is zero We want to increase its value to strictly positive to decrease objective function value . . . corresponds to “moving along an edge” We stop when we reach another (improving) vertex . . . corresponds to setting a basic variable xl to zero x2 11111111111111111 00000000000000000 P = (0, 0, 2, 2) Q 111111111111 000000000000 00000000000000000 11111111111111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 R: optimum 000000000000 111111111111 0 1 00000000000000000 11111111111111111 000000 111111 0 1 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 000000 111111 row 2 000000000000 111111111111 000000 111111 000000000000 111111111111 0 1 000000 111111 0 000000000000 111111111111 x1 P1 S 000000 increase x111111 1 000000 111111 x2 11111111111111111 00000000000000000 row 1 Q 111111111111 000000000000 00000000000000000 11111111111111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 R: optimum 000000000000 111111111111 0 1 00000000000000000 11111111111111111 000000 111111 0 1 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 000000 111111 S = (1, 0, 1, 0) 000000000000 111111111111 000000 111111 000000000000 111111111111 0 1 000000 111111 0 1 000000000000 x1 enters, x111111 exits P 111111111111 000000 S x1 4 000000 111111 xh enters the basis, xl exits the basis Operations research courses / LP theory – p. 14 Simplex Algorithm III How do we determine l and new positive value for xh ? Recall dictionary xB = B −1 b − B −1 N xN , write ¯b = B −1 b and A¯ = (¯aij ) = B −1 N P ¯ For i ∈ β (basics), xi = bi − a ¯ij xj j6∈β Consider nonbasic index h of variable entering basis (all the other nonbasics stay at 0), get xi = ¯bi − a¯ih xh , ∀i ∈ β Increasing xh may make xi < 0 (infeasible), to prevent this enforce ∀i ∈ β (¯bi − a¯ih xh ≥ 0) ¯bi a ¯ih for i ∈ β and a¯ih > 0: ¯bi l = argmin{ |i∈β∧a ¯ih > 0}, a ¯ih Require xh ≤ ¯bl xh = a ¯lh If all a¯ih ≤ 0, xh can increase without limits: problem unbounded Operations research courses / LP theory – p. 15 Simplex Algorithm IV Suppose > n hyperplanes cross at vtx R (degenerate) May get improving direction s.t. adjacent vertex is still R Objective function value does not change Seq. of improving dirs. may fail to move away from R ⇒ simplex algorithm cycles indefinitely Use Bland’s rule: among candidate entering / exiting variables, choose that with least index x2 3x1 + 3x2 ≤ 4 111111111111111111 000000000000000000 000000000000000000 111111111111111111 Q 111111111111 000000000000 2x1 + x2 ≤ 2 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 R: crossing of three lines 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 x1 + 2x2 ≤ 2 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000000000 111111111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 x1 P S 000000 111111 000000 111111 Operations research courses / LP theory – p. 16 Example: Formulation Consider problem: max x1 + x2 x1 ,x2 Standard form: s.t. x1 + 2x2 ≤ 2 2x1 + x2 ≤ 2 x≥0 − minx −x1 − x2 s.t. x1 + 2x2 + x3 = 2 2x1 + x2 + x4 = 2 x≥0 Obj. fun.: max f = − min −f , simply solve for min f Operations research courses / LP theory – p. 17 Example, itn 1: start Objective function vector cT = (−1, −1, 0, 0) Constraints in matrix form: 1 2 1 0 2 1 0 1 ! x1 x2 x3 x4 = 2 2 ! Choose obvious starting basis with ! ! 1 0 1 2 B= ,N = , β = {3, 4} 0 1 2 1 Corresponds to point P = (0, 0, 2, 2) Operations research courses / LP theory – p. 18 Example, itn 1: dictionary Start the simplex algorithm with basis in P x2 1111111111111111 0000000000000000 −∇f Q 111111111111 000000000000 2x + x ≤ 2 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 R 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 x + 2x ≤ 2 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 0 1 000000000000 111111111111 000000 111111 0 1 x S P 000000 111111 000000 111111 row 1 1 2 1 2 row 2 1 ¯ N, Compute dictionary xB = B −1 b − B −1 N xN = ¯b − Ax where ! ! 1 2 ¯b = 2 ¯ ; A= 2 2 1 Operations research courses / LP theory – p. 19 Example, itn 1: entering var Compute reduced costs c¯N = cTN − cTB A¯: (¯ c1 , c¯2 ) = (−1, −1) − (0, 0)A¯ = (−1, −1) All nonbasic variables {x1 , x2 } have negative reduced cost, can choose whichever to enter the basis Bland’s rule: choose entering nonbasic with least index in {x1 , x2 }, i.e. pick h = 1 (move along edge P S ) x2 1111111111111111 0000000000000000 Q 111111111111 000000000000 2x1 + x2 ≤ 2 −∇f 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 R 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 x1 + 2x2 ≤ 2 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 0 1 000000000000 111111111111 000000 111111 0 1 P x1 enters the basis S x1 000000 111111 000000 111111 row 1 row 2 Operations research courses / LP theory – p. 20 Example, itn 1: exiting var Select exiting basic index l ¯bi ¯b1 ¯b2 l = argmin{ , } |i∈β∧a ¯ih > 0} = argmin{ a ¯ih a ¯11 a ¯21 2 2 = argmin{ , } = argmin{2, 1} = 2 1 2 Means: “select second basic variable to exit the basis”, i.e. x4 Select new value x1 ): ¯bl a ¯lh for xh (recall h = 1 corrresponds to ¯bl ¯b2 2 = =1 = a ¯lh a ¯21 2 x1 enters, x4 exits (apply swap (1, 4) to β ) Operations research courses / LP theory – p. 21 Example, itn 2: start Start of new iteration: basis is β = {1, 3} ! ! 1 1 1 0 2 B= ; B −1 = 1 − 21 2 0 xB = (x1 , x3 ) = B −1 b = (1, 1), thus current bfs is (1, 0, 1, 0) = S x2 1111111111111111 0000000000000000 Q 111111111111 000000000000 2x1 + x2 ≤ 2 −∇f 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 R 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 x1 + 2x2 ≤ 2 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 0 1 000000000000 111111111111 000000 111111 0 1 x1 P S 000000 111111 000000 111111 row 1 row 2 Operations research courses / LP theory – p. 22 Example, itn 2: entering var Compute dictionary: ¯b = B −1 b = (1, 1)T , ! ! 1 2 0 0 −1 2 ¯ = A=B N = 1 1 −2 1 1 1 2 3 2 1 2 − 12 ! Compute reduced costs: (¯ c2 , c¯4 ) = (−1, 0) − (−1, 0)A¯ = (−1/2, 1/2) Pick h = 1 (corresponds to x2 entering the basis) x2 1111111111111111 0000000000000000 Q 111111111111 000000000000 2x1 + x2 ≤ 2−∇f 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 R 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 x1 + 2x2 ≤ 2 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 x2 enters basis 000000000000 111111111111 000000 111111 0 1 000000000000 111111111111 000000 111111 0 1 x1 P S 000000 111111 000000 111111 row 1 Operations research courses / LP theory – p. 23 Example, itn 2: exiting var Compute l and new value for x2 : ¯b1 ¯b2 1 1 , }= l = argmin{ , } = argmin{ a ¯11 a ¯21 1/2 3/2 = argmin{2, 2/3} = 2 l = 2 corresponds to second basic variable x3 New value for x2 entering basis: 2 3 x2 enters, x3 exits (apply swap (2, 3) to β ) Operations research courses / LP theory – p. 24 Example, itn 3: start Start of new iteration: basis is β = {1, 2} ! ! 2 1 1 2 − 3 3 B= ; B −1 = 2 1 − 2 1 3 3 xB = (x1 , x2 ) = B −1 b = ( 23 , 32 ), thus current bfs is ( 23 , 32 , 0, 0) = R x2 1111111111111111 0000000000000000 Q 111111111111 000000000000 2x1 + x2 ≤ 2 −∇f 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 R 000000000000 111111111111 0 1 000000 111111 0000000000000000 1111111111111111 0 x1 + 2x2 ≤ 2 1 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 0000000000000000 1111111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 x1 S P 000000 111111 000000 111111 row 1 row 2 Operations research courses / LP theory – p. 25 Example, itn 3: termination Compute dictionary: ¯b = B −1 b = (2/3, 2/3)T , ! ! ! 1 2 2 1 1 0 − − 3 3 3 3 = A¯ = B −1 N = 2 2 1 1 − 0 1 − 3 3 3 3 Compute reduced costs: (¯ c3 , c¯4 ) = (0, 0) − (−1, −1)A¯ = (1/3, 1/3) No negative reduced cost: algorithm terminates Optimal basis: {1, 2} Optimal solution: R = ( 23 , 32 ) Optimal objective function value f (R) = − 43 Permutation to apply to initial basis {3, 4}: (1, 4)(2, 3) Operations research courses / LP theory – p. 26 Optimality Conditions I If we can project improving direction −∇f (x′ ) on an active constraint g2 and obtain a feasible direction d, point x′ is not optimal x2 1111111111111 0000000000000 000000000000 111111111111 0000000000000 1111111111111 000000000000 111111111111 000000 111111 0000000000000 1111111111111 g 000000000000 111111111111 000000 111111 0000000000000 1111111111111 000000000000 111111111111 000000 111111 0000000000000 1111111111111 000000000000 111111111111 000000 111111 0000000000000 1111111111111 000000000000 111111111111 000000 111111 0000000000000 1111111111111 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 d 000000000000 111111111111 000000 111111 000000000000 111111111111 000000 111111 000000000000 111111111111 0 1 000000 111111 0 1 g 111111 000000 x 000000 111111 2 1 ′ −∇f (x′ ) ∇g1 (x′ ) x1 C ∇g1 (x′ ) Implies −∇f (x′ ) 6∈ C (cone generated by active constraint gradients) Operations research courses / LP theory – p. 27 Optimality Conditions II Geometric intuition: situation as above does not happen when −∇f (x∗ ) ∈ C , x∗ optimum x2 11111111111111111 00000000000000000 g ∇g 000000000000 111111111111 00000000000000000 11111111111111111 C 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 ∇g (x ) 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 0 1 000000 111111 0 1 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 x 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 00000000000000000 11111111111111111 000000 111111 000000000000 111111111111 000000 111111 000000000000 111111111111 g 000000 111111 000000000000 111111111111 000000 x 111111 000000000000 111111111111 000000 111111 000000 111111 1 1 −∇f (x∗ ) (x∗ ) 1 ∗ ∗ 2 1 Projection of −∇f (x∗ ) on active constraints is never a feasible direction Operations research courses / ...
View Full Document

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture