This preview shows page 1. Sign up to view the full content.
Unformatted text preview: IE 411 Optimization of LargeScale Linear Systems Xin Chen Spring 2010 Course Information
Lecture hours: MWF 22:50 pm Classroom: 203 TB Instructor: Professor Xin Chen Office: 216C TB Phone: 2448685 Office hours: M 11am12pm W 3pm4pm Email: [email protected] Website: https://netfiles.uiuc.edu/xinchen/www/
Course Website: Illinois Compass Textbook Text : Introduction to Linear Optimization D. Bertsimas and J. Tsitsiklis Athena Scientific (1997) Linear Programming: Foundations and Extensions (second edition) Robert Vanderbei http://www.princeton.edu/~rvdb/LPbook/ Linear Programming and Network Flows (second edition) M.S. Bazaraa, J.J. Jarvis and H. D. Sherali John Wiley and Sons Matlab Excel Solver Reference: Software NEOS Solver http://wwwneos.mcs.anl.gov/neos/ Course Content
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Linear Programming Models and Applications Properties of Linear Programs The Simplex Method Duality Theory and Applications, Dual Simplex Method Parametric Analysis and Sensitivity Analysis The Decomposition Principle and Column Generation Complexity of the Simplex Method Interior Point Methods Conic Linear Optimization Robust Optimization Course Grade Based on Homework Midterm Final 30% 35% 35% Extra Credit Graduate students opting for 1 unit of course work will be required to do extra work. This will involve extra homework assignments and a programming project. Linear Programming Models The Linear Programming Problem
Min subject to c1 x1 + c2 x2 + ...... + cn xn a11 x1 + a12 x2 + ... + a1n xn b1 a21 x1 + a22 x2 + ... + a2 n xn b2 . . . ai1 x1 + + aij x j + ... + a2 n xn bi . . . . am1 x1 + am 2 x2 + ... + amn xn bm x1 , x2 ,...xn 0 . } (1) Objective Function
Right hand side (rhs) vector (2) "Cost" coefficient Constraints } (3) Nonnegativity Technological Decision coefficient Variable Constraint Matrix Constraint Matrix A= a11 a21 : am1 a12 ... a1n a22 ... a2 n : : : am 2 ... amn A set of values assigned to x which satisfy constraints (2) and nonnegativity restrictions (3) gives a feasible point/solution/vector. The set of all feasible points forms the feasible space Among all feasible vectors, find one that minimizes (or maximizes) the objective function. Example
Min subject to
x1 0 2 x1 + 5 x2 x1 + x2 6  x1  2 x2 18 x1 , x2 0 (0,9) (0,6) (18,0) (0,0) (6,0) x2 0 Assumptions Proportionality For each variable, contribution to cost and each constraint is proportional to its value Additivity The total cost is the sum of individual costs. The total contribution is the sum of individual contributions of the variables. Divisibility A variable can take on any fractional value Determinism All problem parameters are known deterministically Standard Form Minimize (Maximize) c x
j =1 j n j a x
j =1 ij n j = bi i = 1,...,m j = 1,...,n xj 0 Canonical Form
Minimize c j x j
j =1 n a
j =1 n ij x j bi xj 0 i = 1,..., m j = 1,..., n Maximize c x
j=1 j n j a
j =1 n ij x j bi xj 0 i = 1,..., m j = 1,..., n Problem Manipulation  Inequalities & Equalities
i) aij x j bi aij x j  xn +1 = bi j =1 j =1 n n xn +1 0 where xn +1 surplus or slack variable (sometimes denoted by Si ) ii ) aij x j bi aij + xn +1 = bi j=1 j =1 n n xn +1 0 aij x j bi n j =1 iii ) aij x j = bi n never in practice j =1 aij x j bi j=1 n Minimization and Maximization Problems Maximum c x
j =1 j n j =  Minimum c x
j =1 j n j Nonnegativity restrictions Unrestricted Variables
x j = ( x j  x j ) x j 0 x j 0 Variable Bounds
xj j x j = x j  j 0 xj uj x j = u j  x j 0 Linear Programming Applications A Production Problem Maximize c x
j=1 j n j a x
j =1 ij n j bi i = 1,..., m j = 1,..., n xj 0 Table Production Example Table 1 10 Wood 1 Wood 2 Wood 3 2 1 3 Table 2 14 1 3 2 Resource 10 10 17 LP Model of Table Production Example Decision variables Table Production Example Now assume wood i is controlled by agent i, i=1,2,3 Do the agents put the wood together to make the production? If they do, how should they share the profit? l1+l2+ l3=V({1,2,3}) l1+l2 V({1,2}) l1+l3 V({1,3}) l2+l3 V({2,3}) l1 V({1}) l2 V({2}) l V({3}) Cooperative Game Production Game Network Flow Problems Cutting Stock Problem Data Fitting Piecewise Linear Convex Functions Pattern Classification Economic Lot Sizing Problem TwoPerson ZeroSum Games
Rock Rock Paper Scissor 0 1 1 Paper 1 0 1 Scissor 1 1 0 I I II III 5 3 3 II 1 2 0 III 3 4 1 Fischer's Exchange market Buyers have money wi to buy goods and maximize individual utility functions ui; Producers sell their goods bj for money. The equilibrium price is an assignment of prices to goods so as when every buyer buys an maximal bundle of goods then the market clears, meaning that all the money is spent and all goods are sold. Geometry of Linear Programming Mathematical Preliminaries Vector x
x k the jth component of the kth vector. j
2 3 Example : x1 = ; x 2 = ; x1 = 1 ; x12 = 3 2 1  1 ^ 0  zero vector (0,0,...,0) ^ 1  sum vector (1,1,...,1) e i  i th unit vector ( 0,0,..,0,13 33 ) 3 3 3 3 ,0,...0
1 is in the ith position Norm Direction n x 2 = ( xi ) 2 i =1 x if x 1/ 2 x 0 Linear Combinations A vector b is said to be a linear combination of vectors { x1 , x 2 ,...x k } if there exist scalars { 1 , 2 , ..., k } s.t. b = 1 x1 + 2 x 2 + ... + k x k .
. is said to be the representation of b in terms of the vectors { x1 , x 2 ,...x k } {1 , 2 , ..., k } Special Combinations The combination is called "proper" if ^ ^ ^ 0 and e i i
If then the linear combination is said to be : trivial nonnegative affine convex i = 0 i i 0 i i =1 k i =1 i =1 k i = 1 and i 0 i Linear Combinations Given a set of vectors
Combination Linear Affine Nonnegative Convex Result Linear Hull of or span of Affine Hull of NonNegative Hull of or Cone Convex Hull. Linear Combinations = x1 , x 2 R 2 { } x1 x 2 x1 x 2 Span Linear Hull Affine Hull Plane through x1,x2 line R2 Convex Hull line segment Cone = nonnegative hull Wedge shaped region of the plane Linear Combinations = e1 , e 2 R 2
Span Linear Hull Affine Hull Convex Hull R2 { } = e1 , e 2 , e 3 R 3
R3 { } Cone = nonnegative hull +ve quadrant +ve orthant Linear Independence { x1 , x 2 ,...x k } are said to be linearly independent if the only linear combination of these vectors which results in 0 is the trivial combination. ^ Otherwise, dependent Alternatively, none of its members is a proper linear combination of { x1 , x 2 ,...x k } Affine independence none of the members of { x1 , x 2 ,...x k } is a proper affine combination of its members Spanning Set The set x 1 , x 2 ,...x k is a spanning set for a vector space X if every vector in X can be written as a linear combination of x 1 , x 2 ,...x k { } { } For X = R n , kn A minimal spanning set for a vector space X is called a basis Any linearly independent set of vectors which spans Rn is said to be a basis for Rn Any n linearly independent vectors in Rn is a basis for Rn n x 1 , x 2 ,...x k , the representation Given a basis of R say of any vector b Rn in terms of the basis is unique. { } If we have a basis B o = {a1 ,...a j ,...a n } , what is the condition that will guarantee that if a j is replaced by another vector, say b, then B 1 = {a 1 ,...b,...a n } is a basis? Replacing a vector in the basis by another vector j =0
b = i a i = i a i
i j b  i a i = 0 linearly dependent.
i j j 0
Suppose u,u i (i j ) s.t.
i j i j u i a i + ub = 0 i j i j u i a i + ub = u i a i + u i a i = ( u i + u i ) a i + u j a j = 0 a i are linearly independent. u j = 0
i j 0 u = 0 u i + u i = 0 u i = 0 b and a are linearly independent. Importance of Hulls
A system of linear equations is said to be homogenous if all the right hand sides are zero Ax = 0 where A: m x n x: n x 1 (1) If x1, x2 are feasible to (1) i.e. Ax1 = 0 ; Ax 2 = 0 then A ( 1 x1 + 2 x 2 ) = 0 1 , 2 any linear combination of solutions of a homogenous system of linear equations is also a solution of this system. Every point in linear hull or span is feasible. Importance of Hulls
Consider a nonhomogenous system of linear equations: ^ Ax = b b 0 (2) If x1, x2 are feasible to (2) i.e. Ax1= b; Ax2 = b, then
A 1 x1 + 2 x 2 = ( 1 + 2 )b ( ) ( x
1 1 +2x2 ) is a solution of (2) if ( 1 + 2 ) = 1 i.e. if and only if (iff) it is an affine combination of x1, x2. Importance of Hulls
A nonhomogenous system of linear equations in nonnegative variables: (3) Ax = b ; x 0 x1,x2 are feasible to (3) i.e 1 2 For 1 x + 2 x ( Ax1 = Ax 2 = b; x 1 ,x 2 0 ) to be feasible to (3) we need a) Ax = b which holds if 1 b) x 0 holds if 1 , 2 0 ( + 2 ) = 1 ; and A linear combination of x1,x2 is feasible if it is a convex combination Requirement Space
Let A j represent the
n j th column of A Then Ax = A j x j is a linear combination of the j =1 columns of A Since x 0 Ax is the nonnegative hull or cone formed by columns of A Hence, for a feasible solution to exist to Ax = b x0 b must lie in the cone formed by cols. of A Requirements Space
The cone formed by columns of A is called the Requirements Space of the problem
b2 b1 Convex Sets
R
n 1 2 A set is a convex set iff for any two points x ,x :
1 2 The line segment joining x and x is in ; or Every convex combination of x and x is in ; or Convex hull of x and x is in ;or For any
1 0 1 ; x2 + (1  ) x 2 1
1 2 Equivalent Convex Sets Extreme Points A point x is an extreme point if there do not exist distinct points x1 and x2 and a scalar 0< <1 such that x = (1  ) x 1 + x 2 Ray A ray with a vertex x and direction d is the set of points {x + d: 0 } Bounded Set A set is said to be bounded if a scalar s.t. x < x Otherwise unbounded Convex Sets Direction of a Set A nonzero vector d is called a direction of the set iff and any scalar >0, ( x + d ) x Note: it is not necessary that d d d Extreme directions of a set A direction of a set is an extreme direction iff it cannot be written as a positive linear combination of two distinct directions of the set. Convex Polyhedral Set The intersection of a finite number of halfspaces a i x bi is called a convex polyhedral set or a convex polyhedron. { x Ax b} A bounded convex polyhedron is called a convex polytope. The intersection of a finite number of half spaces that pass through the origin is called a polyhedral cone. { x Ax 0} polyhedron polytope Convex Polyhedral Set Simplical Cones 1 2 r Let {a , a ,...a } be a linearly independent set of r vectors in R n then the cone of {a 1 , a 2 ,...a r } i.e. {x x = 1 a 1 + 2 a 2 + ... + r a r , 1 0,... 2 0} is called a simplical cone of dimension r.
a1 a1 a3 a2 a2 a1 a3 a3 2d 3d ? Simplex Let {a 1 , a 2 ,...a r } be a set of affinely independent vectors in R n (alternatively, {a 2  a1 , a 3  a1 , , a r  a1 } is a linearly independent set) then, the convex hull of {a1 , a 2 ,...a r } is called a simplex of dimension (r1).
a1 a1 a2 a2 a3 a2 a1 a4 a3 1d 2d 3d Extreme Points and Vertices A point x X is said to be an extreme point if there are no two distinct points x1, x2 X and 2 (0,1) such that x=(1 )x1+ x2. A point x X is said to be a vertex if there exists some vector c such that c'x<c'y for all y X with y x. Basic Solutions of Polyhedral Sets Consider X = { x Amn x b, x 0} a i x = bi ; and then the m hyperplanes the n hyperplanes xi = 0 are called defining hyperplanes. A set of hyperplanes is said to be linearly independent if the coefficient matrix associated with the corresponding set of equations has full row rank. Basic Solutions of Polyhedral Sets Assume a polyhedron X is defined by linear equality and inequality constraints. A point x is said to be a basic solution if All equality constraints are satisfied (active); Out of the constraints that are active at x, there are n of them that are linearly independent. Two basic solutions are called adjacent if they share n1 common linearly independent active constraints Basic Feasible Solutions of Polyhedral Sets Assume a polyhedron X is defined by linear equality and inequality constraints. If x is basic solution and feasible, then it Is a basic feasible solution (BFS). If it lies on more than n active constraints (defining hyperplanes) it is called a degenerate BFS. The excess number of planes is called the order of degeneracy. A polyhedron with a degenerate extreme point is called a degenerate polyhedron. A defining hyperplane is said to be redundant if the removal of the hyperplane won't change the set. Redundancy and Degeneracy Redundancy does not imply degeneracy
a 3 c b 2 1 d Extreme Points, Vertices, BFS of Polyhedral Sets Vertex Extreme point BFS Existence of Extreme Points Theorem Suppose that the polyhedron
P={x2 <nai'x bi, i=1,2,...,m} is nonempty. Then, the following are equivalent: P has at least one extreme point. There exist n vectors out of the family a1, ..., am, which are linearly independent. P does not contain a line. Does a polyhedron in standard form have at least one extreme point? Recession Cone and Extreme Directions Consider X={x  Ax b, x 0} then {d  Ad 0, d 0, d 0} defines the recession cone or characteristic cone of X
^ D = d Ad 0; d 0; 1d = 1 { } Recession cone Directions of X Extreme Directions of X Representation of Polyhedral Set
x3 x2 x x1 x5 _ x x4 d1 x2 _ x x x1 x3 d2 Representation Theorem Let X={x  Ax b, x 0} be nonempty. Then the set of extreme points is not empty and is finite say { x1 , x 2 ,...x k } If X is not bounded the set of extreme directions 1 2 is not empty and is finite, say {d , d ,...d } The set of extreme directions is empty if X is a polytope. x X if it can be represented as a convex { x1 , x 2 ,...x k } plus combination of {d 1 , d 2 ,...d } a nonnegative combination of k i.e.
x = j x j + u jd j
j =1 j =1 k j j =1 = 1, j 0; u j 0 Optimality of Extreme Point Theorem Consider the LP problem
min c'x s.t. x2 P ;. Then, either the optimal cost is 1 or there exists an optimal solution. If P has at least one extreme point, then there exists an extreme point which is optimal in the second case. ...
View
Full
Document
This note was uploaded on 09/08/2010 for the course IESE IE411 taught by Professor Xinchen during the Fall '09 term at University of Illinois, Urbana Champaign.
 Fall '09
 XinChen

Click to edit the document details