This **preview** has intentionally **blurred** parts. Sign up to view the full document

**Unformatted Document Excerpt**

of Principles Mathematics in Operations Research Recent titles in the I NTERNATIONAL SERIES IN OPERATIONS RESEARCH & MANAGEMENT SCIENCE Frederick S. Hillier, Series Editor, Stanford University T alluri & van Ryzin/ THE THEORY AND PRACTICE OF REVENUE MANAGEMENT K avadias & Lochy'PROJECT SELECTION UNDER UNCERTAINTY: Dynamically Allocating Resources to Maximize Value B randeau, Sainfort & Pierskalla/ OPERATIONS RESEARCH AND HEALTH CARE: A Handbook of Methods and Applications C ooper, Seiford & Zhu/ HANDBOOK OF DATA ENVELOPMENT ANALYSIS: Models and Methods L uenberger/ LINEAR AND NONLINEAR PROGRAMMING, 2'"1 Ed. S herbrooke/ OPTIMAL INVENTORY MODELING OF SYSTEMS: Multi-Echelon Techniques, Second Edition C hu, Leung, Hui & Cheung/ 4th PARTY CYBER LOGISTICS FOR AIR CARGO S imchi-Levi, Wu & Shen/ HANDBOOK OF QUANTITATIVE SUPPLY CHAIN ANALYSIS: Modeling in the E-Business Era G ass & Assad/ AN ANNOTATED TIMELINE OF OPERATIONS RESEARCH: An Informal History G reenberg/ TUTORIALS ON EMERGING METHODOLOGIES AND APPLICATIONS IN OPERATIONS RESEARCH W eber/ UNCERTAINTY IN THE ELECTRIC POWER INDUSTRY: Methods and Models for Decision Support F igueira, Greco & Ehrgott/ MULTIPLE CRITERIA DECISION ANALYSIS: State of the Art Surveys R eveliotis/ REAL-TIME MANAGEMENT OF RESOURCE ALLOCATIONS SYSTEMS: A Discrete Event Systems Approach K ail & Mayer/ STOCHASTIC LINEAR PROGRAMMING: Models, Theory, and Computation S ethi, Yan & Zhang/ INVENTORY AND SUPPLY CHAIN MANAGEMENT WITH FORECAST UPDATES C ox/ QUANTITATIVE HEALTH RISK ANALYSIS METHODS: Modeling the Human Health Impacts of Antibiotics Used in Food Animals C hing & Ng/ MARKOV CHAINS: Models, Algorithms and Applications Li & Sun/NONLINEAR INTEGER PROGRAMMING K aliszewski/ SOFT COMPUTING FOR COMPLEX MULTIPLE CRITERIA DECISION MAKING B ouyssou et al/ EVALUATION AND DECISION MODELS WITH MULTIPLE CRITERIA: Stepping stones for the analyst B lecker & Friedrich/ MASS CUSTOMIZATION: Challenges and Solutions A ppa, Pitsoulis & Williams/ HANDBOOK ON MODELLING FOR DISCRETE OPTIMIZATION H errmann/ HANDBOOK OF PRODUCTION SCHEDULING A xsater/ INVENTORY CONTROL, 2'"1 Ed. H all/ PATIENT FLOW: Reducing Delay in Healthcare Delivery J ozefowska & Wgglarz/ PERSPECTIVES IN MODERN PROJECT SCHEDULING T ian & Zhang/ VACATION QUEUEING MODELS: Theory and Applications Y an, Yin & Zhang/STOCHASTIC PROCESSES, OPTIMIZATION, AND CONTROL THEORY APPLICATIONS IN FINANCIAL ENGINEERING, QUEUEING NETWORKS, AND MANUFACTURING SYSTEMS S aaty & Vargas/ DECISION MAKING WITH THE ANALYTIC NETWORK PROCESS: Economic, Political, Social & Technological Applications w. Benefits, Opportunities, Costs & Risks Y u/ TECHNOLOGY PORTFOLIO PLANNING AND MANAGEMENT: Practical Concepts and Tools A list of the early publications in the series is at the end of the book * Levent Kandiller Principles of Mathematics in Operations Research 4y Springer Levent Kandiller Middle East Technical University Ankara, Turkey Library of Congress Control Number: ISBN-10: 0-387-37734-4 (HB) ISBN-10: 0-387-37735-2 (e-book) ISBN-13: 978-0387-37734-6 (HB) ISBN-13: 978-0387-37735-3 (e-book) Printed on acid-free paper. © 2007 by Springer Science+Business Media, LLC All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science + Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now know or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks and similar terms, even if the are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. 987654321 springer.com To my daughter, Deniz P reface T he aim of this book is to provide an overview of mathematical concepts a nd their relationships not only for graduate students in the fields of Operations Research, Management Science and Industrial Engineering but also for p ractitioners and academicians who seek to refresh their mathematical skills. T he contents, which could broadly be divided into two as linear algebra and real analysis, may also be more specifically categorized as linear algebra, convex analysis, linear programming, real and functional analysis. The book h as been designed to include fourteen chapters so that it might assist a 14week graduate course, one chapter to be covered each week. T he introductory chapter aims to introduce or review the relationship between Operations Research and mathematics, to offer a view of mathematics as a language and to expose the reader to the art of proof-making. T he chapters in Part 1, linear algebra, aim to provide input on preliminary linear algebra, orthogonality, eigen values and vectors, positive definiteness, condition numbers, convex sets and functions, linear programming and duality theory. The chapters in Part 2, real analysis, aim to raise awareness of number systems, basic topology, continuity, differentiation, power series and special functions, and Laplace and z-transforms. T he book has been written with an approach that aims to create a snowball effect. To this end, each chapter has been designed so that it adds to what the reader has gained insight into in previous chapters, and thus leads the reader t o the broader picture while helping establish connections between concepts. T he chapters have been designed in a reference book style to offer a concise review of related mathematical concepts embedded in small examples. T he remarks in each section aim to set and establish the relationship between concepts, to highlight the importance of previously discussed ones or those c urrently under discussion, and to occasionally help relate the concepts under scrutiny to Operations Research and engineering applications. The problems a t the end of each chapter have been designed not merely as simple exercises requiring little time and effort for solving but rather as in-depth problem solving tasks requiring thorough mastery of almost all of the concepts pro- VIII Preface vided within that chapter. Various Operations Research applications from deterministic (continuous, discrete, static, dynamic) modeling, combinatorics, regression, optimization, graph theory, solution of e quation systems as well as geometric a nd conceptual visualization of a bstract mathematical concepts have been included. As opposed t o supplying t he r eaders with a reference list or bibliography a t t he end of the book, active web resources have been provided a t the end of each chapter. T he r ationale behind this is t hat despite t he volatility of I nternet sources, which has recently proven t o be less so w ith t he necessary solid maintenance being ensured, t he availability of web references will enable t he ambitious reader t o access materials for further study without delay a t t he end of each chapter. I t will also enable t he a uthor t o keep this list of web m aterials updated t o exclude those that can no longer b e accessed a nd to include new ones after screening relevant web sites periodically. I would like t o acknowledge all those who have contributed t o the completion a nd p ublication of t his book. Firstly, I would like t o e xtend my g ratitude t o Prof. Fred Hillier for agreeing t o add t his book t o his series. I a m also i ndebted t o G ary Folven, Senior Editor a t Springer, for his speedy processing a nd encouragement. I owe a g reat deal t o my professors a t Bilkent University, Mefharet Kocatepe, Erol Sezer a nd my Ph.D. advisor Mustafa Akgiil, for t heir contributions t o my development. Without their impact, this book could never have materialized. I would also like t o extend my heartfelt thanks t o Prof. Caglar Giiven a nd Prof. Halim Dogrusoz from Middle East Technical University for the insight that they provided as regards O R methodology, t o Prof. M urat Koksalan for his encouragement a nd guidance, a nd to Prof. Nur Evin Ozdemirel for her mentoring a nd friendship. T he contributions of my g raduate students over t he years it t ook t o complete this book a re u ndeniable. I t hank them for t heir continuous feedback, invaluable comments a nd endless support. My special thanks go to Dr. Tevhide Altekin, former student current colleague, for sharing with me her view of the course content a nd conduct as well as for her suggestions as to the p resentation of t he m aterial within t he book. Last b ut not least, I am grateful t o my family, my p arents in p articular, for t heir continuous encouragement a nd s upport. My final words of a ppreciation go t o my local editor, my wife Sibel, for her faith in w hat started o ut as a far-fetched project, a nd most importantly, for her faith in me. Ankara, Turkey, J une 2006 Levent Kandiller C ontents 1 I ntroduction 1.1 Mathematics and OR 1.2 Mathematics as a language 1.3 The art of making proofs 1.3.1 Forward-Backward method 1.3.2 Induction Method 1.3.3 Contradiction Method 1.3.4 Theorem of alternatives P roblems Web material P reliminary Linear Algebra 2.1 Vector Spaces 2.1.1 Fields and linear spaces 2.1.2 Subspaces 2.1.3 Bases 2.2 Linear transformations, matrices and change of basis 2.2.1 Matrix multiplication 2.2.2 Linear transformation 2.3 Systems of Linear Equations 2.3.1 Gaussian elimination 2.3.2 Gauss-Jordan method for inverses 2.3.3 The most general case 2.4 The four fundamental subspaces 2.4.1 The row space of A 2.4.2 The column space of A 2.4.3 The null space (kernel) of A 2.4.4 The left null space of A 2.4.5 The Fundamental Theorem of Linear Algebra P roblems Web material 1 1 2 5 5 7 8 9 9 10 13 13 13 14 16 17 17 18 20 20 23 24 25 25 26 26 27 27 28 29 2 X 3 Contents O rthogonality 3.1 Inner Products 3.1.1 Norms 3.1.2 Orthogonal Spaces 3.1.3 Angle between two vectors 3.1.4 Projection 3.1.5 Symmetric Matrices 3.2 Projections and Least Squares Approximations 3.2.1 Orthogonal bases 3.2.2 Gram-Schmidt Orthogonalization 3.2.3 Pseudo (Moore-Penrose) Inverse 3.2.4 Singular Value Decomposition 3.3 Summary for Ax = b P roblems Web material E igen Values and Vectors 4.1 Determinants 4.1.1 Preliminaries 4.1.2 Properties 4.2 Eigen Values and Eigen Vectors 4.3 Diagonal Form of a Matrix 4.3.1 All Distinct Eigen Values 4.3.2 Repeated Eigen Values with Full Kernels 4.3.3 Block Diagonal Form 4.4 Powers of A 4.4.1 Difference equations 4.4.2 Differential Equations 4.5 The Complex case P roblems Web material P ositive Definiteness 5.1 Minima, Maxima, Saddle points 5.1.1 Scalar Functions 5.1.2 Quadratic forms 5.2 Detecting Positive-Definiteness 5.3 Semidefinite Matrices 5.4 Positive Definite Quadratic Forms P roblems Web material 33 33 33 35 36 37 37 38 39 40 42 43 44 47 47 51 51 51 52 54 55 55 57 58 60 61 62 63 65 66 71 71 71 73 74 75 76 77 77 4 5 Contents 6 C omputational Aspects 6.1 Solution of Ax = b 6.1.1 Symmetric and positive definite 6.1.2 Symmetric and not positive definite 6.1.3 Asymmetric 6.2 Computation of eigen values P roblems Web material C onvex Sets 7.1 Preliminaries 7.2 Hyperplanes and Polytopes 7.3 Separating and Supporting Hyperplanes 7.4 Extreme Points P roblems Web material L inear Programming 8.1 The Simplex Method 8.2 Simplex Tableau 8.3 Revised Simplex Method 8.4 Duality Theory 8.5 Farkas' Lemma P roblems Web material N umber Systems 9.1 Ordered Sets 9.2 Fields 9.3 The Real Field 9.4 ' The Complex Field 9.5 Euclidean Space 9.6 Countable and Uncountable Sets P roblems Web material XI 81 81 81 83 83 86 89 90 93 93 95 97 98 99 100 103 103 107 110 Ill 113 115 117 121 121 123 125 127 128 129 133 134 137 137 146 150 151 152 154 7 8 9 1 0 Basic Topology 10.1 Metric Spaces 10.2 Compact Sets 10.3 The Cantor Set 10.4 Connected Sets P roblems Web material XII Contents 157 I57 159 160 161 164 166 166 169 169 170 172 173 173 175 175 175 177 177 178 179 180 182 184 185 186 188 191 191 192 197 199 201 202 205 293 1 1 Continuity 11.1 Introduction 11.2 Continuity and Compactness 11.3 Uniform Continuity 11.4 Continuity and Connectedness 11.5 Monotonic Functions P roblems Web material 12 Differentiation 12.1 Derivatives 12.2 Mean Value Theorems 12.3 Higher Order Derivatives P roblems Web material 1 3 Power Series and Special Functions 13.1 Series 13.1.1 Notion of Series 13.1.2 Operations on Series 13.1.3 Tests for positive series 13.2 Sequence of Functions 13.3 Power Series 13.4 Exponential and Logarithmic Functions 13.5 Trigonometric Functions 13.6 Fourier Series 13.7 Gamma Function P roblems Web material 1 4 Special Transformations 14.1 Differential Equations 14.2 Laplace Transforms 14.3 Difference Equations 14.4 Z Transforms P roblems Web material S olutions I ndex 1 I ntroduction O perations Research, in a narrow sense, is the application of scientific models, especially mathematical and statistical ones, to decision making problems. T he present course material is devoted to parts of mathematics that are used in Operations Research. 1.1 Mathematics and OR In order to clarify the understanding of the relation between two disciplines, let us examine Figure 1.1. The scientific inquiry has two aims: • • cognitive: knowing for the sake of knowing instrumental: knowing for the sake of doing If A is Bis a proposition, and if B belongs to A, t he proposition is analytic. It can be validated logically. All analytic propositions are a priori. T hey are tautologies like "all husbands are married". If B is outside of A, t he proposition is synthetic and cannot be validated logically. It can be a posteriori like "all African-Americans have dark skin" and can be validated empirically, b ut there are difficulties in establishing necessity and generalizability like "Fenerbahce b eats Galatasaray". M athematics is purely analytical and serves cognitive inquiry. Operations Research is (should be) instrumental, hence closely related to engineering, m anagement sciences and social sciences. However, like scientific theories, Operations Research • • • refers to idealized models of the world, employs theoretical concepts, provides explanations and predictions using empirical knowledge. T he purpose of this material is to review the related mathematical knowledge t hat will be used in graduate courses and research as well as to equip the s tudent with the above three tools of Operations Research. 1 Introduction cognitive interest ANALYTIC H pure logic methodology mathematics physics chemistry biology psychology astronomy y O.R. '• l" /.I EMPIRICAL i /'i applied instrumental interest social sciences' ' management sciences medicine ' agriculture engineering - - - - F i g . 1 . 1 . T h e scientific inquiry. 1.2 Mathematics as a language T he main objective of mathematics is to state certainty. Hence, the main role of a mathematician is to communicate truths but usually in its own language. One example is V* e 5, 3j e T 3 ilj => Vj G T, 3i e S 3 i±j <=> S±T. T hat is, if for all i in S there exists an element j of T such that i is orthogonal t o j then for all elements j of T there is an element j of S such that j is o rthogonal to i; if and only if, S is orthogonal to T. To help the reader appreciate the expressive power of modern mathematical language, and as a tribute to those who achieved so much without it, a few samples of (original but translated) formulation of theorems and their equivalents have been collected below. (a + bf = a 2 + b2 + lab If a straight line be cut at random, the square on the whole is equal to the squares on the segments and twice the rectangle contained by the segments (Euclid, Elements, II.4, 300B.C). 1 + 2 + • • • + 2 " is prime => 2 n (l + 2 + • • • + 2") is perfect 1.2 Mathematics as a language 3 If as many numbers as we please beginning from a unit be set out continuously in double proportion, until the sum of all becomes prime, and if the sum multiplied into the last make some number, the product will be perfect (Euclid, Elements, IX.36, 300B.C). 2nr-r , A - — — = 7rrJ — The area of any circle is equal to a right-angled triangle in which one of the sides about the right angle is equal to the radius, and the other to the circumference, of the circle (Archimedes, Measurement of a Circle, 225B.C). S = 4wr2 The surface of any sphere is equal four times the greatest circle in it (Archimedes, On the Sphere and the Cylinder, 220B.C). 3 In n2 m3 3/ n n2 m3 Rule to solve x3 + mx = n: Cube one-third the coefficient of x; add to it the square of one-half the constant of the equation; and take the square root of the whole. You will duplicate this, and to one of the two you add one-half the number you have already squared and from the other you subtract one-half the same... Then, subtracting the cube root of the first from the cube root of the second, the remainder which is left is the value of x (Gerolamo Cardano, Ars Magna, 1545). However, the language of mathematics does not consist of formulas alone. T he definitions and terms are verbalized often acquiring a meaning different from the customary one. In this section, the basic grammar of mathematical language is presented. D efinition 1.2.1 Definition is a statement that is agreed on by all parties concerned. They exist because of mathematical concepts that occur repeatedly. E xample 1.2.2 A prime number is a natural integer which can only be (integer) divided by itself and one without any remainder. P roposition 1.2.3 A Proposition or Fact is a true statement of interest that is being attempted to be proven. Here are some examples: Always true Two different lines in a plane are either parallel or they intersect a t exactly one point. Always false —1 = 0. Sometimes true 2x — 1, by < 1, z > 0 and x,y,z e K. 4 1 Introduction Needs proof! T here is an angle t such that cos t = t. Proof. Proofs should not contain ambiguity. However, one needs creativity, intuition, experience and luck. The basic guidelines of proof making is tutored in the next section. Proofs end either with Q.E.D. ("Quod Erat Demonstrandum"), means "which was to be demonstrated" or a square such as the one here. • T heorem 1.2.4 Theorems are important propositions. L emma 1.2.5 Lemma is used for preliminary propositions that are to be used in the proof of a theorem. C orollary 1.2.6 Corollary is a proposition that follows almost immediately as a result of knowing that the most recent theorem is true. A xiom 1.2.7 Axioms are certain propositions that are accepted without formal proof. E xample 1.2.8 The shortest distance between two points is a straight line. C onjecture 1.2.9 Conjectures are propositions that are to date neither proven nor disproved. R emark 1 .2.10 A remark is an important observation. T here are also quantifiers: 3 V € 3 : t here is/are, exists/exist for all, for each, for every in, element of, member of, choose such that, that is member definition An example to the use of these delimiters is ~iy G S = {x e Z+ : x is odd } , y2 e S, t hat is the square of every positive odd number is also odd. Let us concentrate on A => B, i.e. if A is true, then B is true. This s tatement is the main structure of every element of a proposition family which is to be proven. Here, statement A is known as a hypothesis whereas B is t ermed as a conclusion. T he operation table for this logical statement is given in Table 1.1. This statement is incorrect if A is true and B is false. Hence, t he main aim of making proofs is to detect this case or to show that this case c annot happen. 1.3 The art of making proofs Table 1.1. Operation table for A => B A T rue T rue False False B T rue False T rue False A=>B T rue False T rue T rue 5 Formally speaking, A=> B means 1. 2. 3. 4. 5. whenever A is true, B must also be true. B follows from A. B is a necessary consequence of A. A is sufficient for B. A only if B. T here are related statements to our primal assertion A =>• B: B =>• A: converse A =>• B: inverse B => A: contrapositive where A is negation (complement) of A. 1.3 T h e art of making proofs T his section is based on guidelines of how to read and make proofs. Our p attern here is once again A =>• B. We are going to start with the forwardbackward method. After discussing the special cases defined in A or B in terms of quantifiers, we will see proof by Contradiction, in particular contraposition. Finally, we will investigate uniqueness proofs and theorem of alternatives. 1 .3.1 Forward-Backward method If t he statement A => B is proven by showing that B is true after assuming 4 A is true (A -t B), t he method is called full forward technique. Conversely, if we first assume that B is true and try to prove that A is true (A <- B), t his is the full backward m ethod. P roposition 1.3.1 If the right triangle XYZ with sides x, y and hypotenuse of length z has an area of ^- (A), then the triangle XYZ is isosceles (B). See Figure 1.2. 6 1 Introduction X y Z x F ig. 1.2. Proposition 1.3.1 Proof. Backward: B: x = y (a; - y = 0) < > F X Z = .XTZ (triangle is equilateral) = Forward: A -(i) Area: ^a;j/ = ^A-(ii) Pythagorean Theorem: x2 + y2 = z2 <£• \xy = ^ ± ^ < > a;2 - 2xj/ + y2 = 0 « • (a: - y)2 = 0 < > a; - y = 0. £ = • T he above proof is a good example of how forward-backward combination can be used. There are special cases defined by the forms of A or B with the use of quantifiers. The first three out of four cases are based on conditions on s tatement B and the last one arises when A has a special form. C o n s t r u c t i o n (3) If there is an object (3a; € N) with a certain property(a: > 2) such that something happens (x2 — 5x + 6 = 0 ), this is a construction. Our objective here is to first construct the object so that it possesses the certain property and then to show that something happens. S election (V) If something (3a; E I 3 2* = j ) happens for every object (Vj/ € R+) with a c ertain property (y > 0), this is a selection. Our objective here is to first make a list (set) of all objects in which something happens (T — {y € M+ : 3a; e R 3 2X — y}) and show that this set is equivalent to the set whose elements has the property (S = R + ) . In order to show an equivalence of two sets (S — T), one usually has to show (S C T) a nd (T C S) by choosing a generic element in one set and proving that it is in the other set, and vice versa. S pecialization If A is of the form "for all objects with a certain property such that something happens", then the method of specialization can be used. Without loss 1.3 The art of making proofs 7 of generality, we can fix an object with t he p roperty. If we can show that something happens for t his particular object, we can generalize t he result for all t he objects with t he same property. P roposition 1 .3.2 Let T C S C R, and u be an upper bound for S; i.e. Va; £ S, x < u. Then, u is an upper bound for T. Proof. Let u b e an u pper bound for S, so Vx £ S, x < u. Take any element yoiT.TCS=>y£S=>y<u. T hus, Vy £ T, y < u. T hen, u is an u pper b ound for T. D U niqueness W hen statement B h as the word unique in it, the proposition is more restrictive. We should first show t he existence then prove t he uniqueness. T he s tandard way of showing uniqueness is to assume two different objects with t he property a nd to conclude that they are the same. P roposition 1 .3.3 W £ R +, 3 unique i £ R 3 i 3 = r. Proof. Existence: Let y = r 3 , 1 /6K. Uniqueness: Let x, y £ M 3 x ^ y, x3 = r = y3 => x3 — y3 — 0 => (a; — y){x2 + xy + y2) = 0 => (x2 + xy + y2) = 0, since x ^ y. T he r oots of t he last equation (if we take y as p arameter a nd solve for a;) are -y ± \Jv2 - V 2 = -y ± \/-3y2 2 • Hence, y = 0 => y3 — 0 = r g R + . C ontradiction. Thus, x = y. 1 .3.2 Induction Method Proofs of the form "for every integer n > 1, something happens" is made by induction. Formally speaking, induction is used when B is t rue for each integer beginning with an initial one (n0). If the base case (n = n0) is t rue, it is assumed that something happens for a generic intermediate case ( n = nk). Consequently, t he following case ( n = n^+i) is shown, usually using t he p roperties of the i nduction hypothesis (n — nk). I n some instances, one may r elate any previous case ( nj, 0 < / < k). Let us give t he following example. T heorem 1 .3.4 1 + 2+ • • • + n — > k = — ^ r-f 2 -. 8 1 Introduction Proof. Base: n = 1 = -1-2 jHypothesis: n = j , E L i * = = L ^ 2 -J + I. Conciusion: n = j + 1, Efc=il & _ (.7 + D(J+ ) — 1 = ( i+1) [ 1 + f ] _ (j+l)(j+2) > 2 T hus, l + 2 + --- + n = £ ?c=l' , f c = ^ i . f_ 1 .3.3 Contradiction Method D W hen we examine t he o peration table for A =*• B in Table 1.2, we immediately conclude that t he only circumstance under which A =4- B is not correct is when A is t rue a nd B is false. C ontradiction Proof by C ontradiction assumes t he condition (A is t rue B is false) a nd t ries t o reach a legitimate condition in which this cannot happen. Thus, t he only way A =$• B being incorrect is ruled o ut. Therefore, A => B is correct. This proof method is q uite powerful. P roposition 1 .3.5 n 6 N, n is even =$• n is even. Proof. Let us assume that n 6 N, n 2 is even b ut n is o dd. Let n = 2k -1, A 6 ; N. T hen, n 2 = 4k2 - 4/c + 1 which is definitely o dd. C ontradiction. • C ontraposition In contraposition, we assume A a nd B a nd go forward while we assume A and come backward in order t o reach a C ontradiction. I n t hat sense, contraposition is a special case of C ontradiction where all the effort is directed towards a specific type of C ontradiction (^4 vs. A). T he m ain motivation under c ontrapositivity is the following: A=> B = AVB = (A\/ JB) V A = (A/\B)^> A. One can prove t he above fact simply by examining Table 1.2. T able 1.2. Operation table for some logical operators. A T T F F A F F T T B T F T F B A^B F T T F F T TT Av B AAB A/\B^ T F T F T F T F T F T T A 1.3 Problems P roposition 1 .3.6 p,qeR+3 proof. A: y/pq^^ y/pq^ V 9 -~Y- => P hi- a nd hence A: ^pq = *f*. Similarly, B: p £ q a nd B: = ^fpq. However, this E 2 p = q. Let us assume B a nd go forward 2±2 = p = ^ is nothing b ut A: ^/pq = y . C ontradiction. D 1 .3.4 Theorem o f a lternatives If t he p attern of the proposition is A => either C or (else) D is t rue ( but not b oth), we have a t heorem of a lternatives. In order t o prove such a proposition, we first assume A a nd C a nd try to reach D. T hen, we should interchange C and D, do the same operation. P roposition 1 .3.7 If x2 - 5x + 6 > 0, then either x < 2 or x > 3 . Proof. Let x > 2. T hen, a;2 - 5a; + 6 > 0 => (a; - 2)(x - 3) > 0 =» (a: - 3) > 0 => a; > 3. Let a; < 3. T hen, x2 - 5x + 6 > 0 => (a; - 2 )(x - 3 ) > 0 ^ ( a ; - 2 ) < 0 ^ a ; < 2 . D P roblems 1 .1. Prove t he following two propositions: (a) If / and g are two functions that are continuous * a t x, t hen t he function / + g is also continuous a t x, where ( / + g)(y) = f(y) + g(y). (b) If / is a function of one variable that (at point a;) satisfies 3 c > 0, 5 > 0 such that Vy 3 \x - y\ < 6, \f(x) - f(y)\ t hen / is continuous a t x. 1.2. Assume you have a chocolate b ar consisting, as usual, of a number of squares arranged in a r ectangular pattern. Your task is to split t he bar into small squares (always breaking along t he lines between t he squares) with a minimum number of b reaks. How many will it t ake? Prove 2 . A function / of one variable is continuous at point x if Ve > 0, 35 > 0 such that Vy B \x - y\ < S =^ | /(x) - f(y)\ < e. www.cut-the-knot.org/proofs/chocolad.shtml <c\x-y\ 2 10 1 Introduction 1 .3. Prove t he following: (a) (") = L%)1 (b ) C) = ("; ) + (":!)• (c)( + (d) ( m ) ( 7 ) = (r)(m-r)- ? J" ) + ;"nP = 2"' (e)(s) + r r)+---+rr)-r; + 1 )http://acept.la.asu.edu/courses/phsllO/si/chapterl/main.html http://cas.umkc.edu/math/MathUGcourses/Mathl05.htm http://cresst96.cse.ucla.edu/Reports/TECH429.pdf http://descmath.com/desc/language.html http://economictimes.indiatimes.com/articleshow/1024184.cms http://en.wikipedia.org/wiki/Mathematical_proof http://en.wikipedia.org/wiki/Mathematics_as_a_language http: //ids. oise .utoronto. ca/~ghanna/educationabstracts .html http:I Itcis.oise.utoronto.ca/~ghanna/philosophyabstracts.html http://germain.umemat.maine.edu/faculty/wohlgemuth/DMAltIntro.pdf http://interactive-mathvision.com/PaisPortfolio/CKMPerspective/ Constructivism(1998).html http://mathforum.org/dr.math/faq/faq.proof.html http://mathforum.org/library/view/5758.html http://mathforum.org/mathed/mtbib/proof.methods.html http://mtcs.truman.edu/"thammond/history/Language.html http://mzone.mweb.co.za/residents/profmd/proof.pdf http://online.redwoods.cc.ca.us/instruct/mbutler/BUTLER/ mathlanguage.pdf http://pass.maths.org.uk/issue7/features/proofl/index.html http://pass.maths.org.uk/issue8/features/proof2/index.html http://plus.maths.org/issue9/features/proof3/index.html http://plus.maths.org/issuelO/features/proof4/ http://research.microsoft.com/users/lamport/pubs/ lamport-how-to-write.pdf http://serendip.brynmawr.edu/blog/node/59 http://teacher.nsr1.rochester.edu/phy_labs/AppendixE/ AppendixE.html http://weblog.fortnow.com/2005/07/understanding-proofs.html http://www-didactique.imag.fr/preuve/ICME9TG12 http://www-didactique.imag.fr/preuve/indexUK.html http://www-leibniz.imag.fr/DIDACTIQUE/preuve/ICME9TG12 http://www-logic.Stanford.edu/proofsurvey.html http://www-personal.umich.edu/~tappen/Proofstyle.pdf http://www.4to40.com/activities/mathemagic/index.asp? article=activities_mathemagic_mathematicalssigns http://www.ams.org/bull/pre-1996-data/199430-2/thurston.pdf Web m aterial 1.4 W e b material 11 http://www.answers.com/topic/mathematics-as-a-language http://www.bisso.com/ujg_archives/000158.html http://www.bluemoon.net/~watson/proof.htm http://www.c3.lanl.gov/mega-math/workbk/map/mptwo.html http://www.cal.org/ericcll/minibibs/IntMath.htm http://www.chemistrycoach.com/language.htm http://www.cis.upenn.edu/"ircs/mol/mol.html http://www.crystalinks.com/math.html http://www.culturaleconomics.atfreeweb.com/Anno/Boulding •/.20Limitations7.20of'/.20Mathematics'/.201955.htm http://www.cut-the-knot.com/language/index.shtml http://www.cut-the-knot.org/ctk/pww.shtml http://www.cut-the-knot.org/language/index.shtml http://www.cut-the-knot.org/proofs/index.shtml http://www.education.txstate.edu/epic/mellwebdocs/ SRSUlitreview.htm http://www.ensculptic.com/mpg/fields/webpages/GilaHomepage/ philosophyabstracts.html http://www.fdavidpeat.com/bibliography/essays/maths.htm http://www.fiz-karlsruhe.de/fiz/publications/zdm/zdm985r2.pdf http://www.iigss.net/ http://www.indiana.edu/"mf1/cg.html http://www.isbe.state.il.us/ils/math/standards.htm http://www.lettredelapreuve.it/ICME9TG12/index.html http://www.lettredelapreuve.it/TextesDivers/ICMETGProof96.html http://www.maa.org/editorial/knot/Mathematics.html http://www.maa.org/reviews/langmath.html http://www.math.csusb.edu/notes/proofs/pfnot/nodelO.html http://www.math.csusb.edu/notes/proofs/pfnot/pfnot.html http://www.math.lamar.edu/MELL/index.html http://www.math.montana.edu/mathl51/ http://www.math.rochester.edu/people/faculty/rarm/english.html http://www.math.toronto.edu/barbeau/hannaj oint.pdf http://www.mathcamp.org/proofs.php http://www.mathemat icallycorrect.com/allen4.htm http://www.mathmlconference.org/2002/presentations/naciri/ http://www.maths.ox.ac.uk/current-students/undergraduates/ study-guide/p2.2.6.html http://www.mtholyoke.edu/courses/rschwart/mac/writing/language.shtml http://www.nctm.org/about/position_statements/ position_statement_06.htm http://www.nwrel.org/msec/science_inq/ http://www.quotedb.com/quotes/3002 http://www.righteducation.org/id28.htm http://www.sciencemag.org/cgi/content/full/307/5714/1402a http://www.sciencemag.org/sciext/125th/ http://www.southwestern.edu/"sawyerc/math-proofs.htm http://www.theproofproj ect.org/bibliography http://www.uoregon.edu/~moursund/Math/language.htm 12 1 Introduction http://www.utexas.edu/courses/bio301d/Topics/Scientific.method/ Text.html http://www.w3.org/Math/ http://www.Warwick.ac.uk/staff/David.Tall/themes/proof.html http://www.wmich.edu/math-stat/people/faculty/chartrand/proofs http://www2.edc.org/makingmath/handbook/Teacher/Proof/Proof.asp http://www2.edc.org/makingmath/mathtools/contradiction/ contradiction.asp http://www2.edc.org/makingmath/mathtools/proof/proof.asp https://www.theproofproj ect.org/bibliography/ 2 P reliminary Linear Algebra T his chapter includes a rapid review of basic concepts of Linear Algebra. After d enning fields and vector spaces, we are going to cover bases, dimension and linear transformations. The theory of simultaneous equations and triangular factorization are going to be discussed as well. The chapter ends with the fundamental theorem of linear algebra. 2.1 V e c t o r S p a c e s 2 .1.1 Fields and linear spaces D efinition 2.1.1 A set F together with two operations + :FxF^F •:FXFHF Addition Multiplication is called a field if 1. a) a + 0 — 0 + a, Va, 0 G F (Commutative) b) (a + 0) + 7 — a + (0 + 7 ), Va, 0,7 6 F (Associative) c) 3 a distinguished element denoted by 0 B Va E F , a + 0 = a (Additive identity) d) Va €W 3 — a s F 3 a + (—a) = 0 (Existence of an inverse) 2. a) a • 0 — 0 • a , Va,/3 € F (Commutative) b) ( a • 0) • 7 = a • (0 • 7 ), Va, 0,7 e F (Associative) c) 3 an element denoted by 1 B Va e F, a • 1 = a (Multiplicative identity) ^ V a ^ 0 e F 3 a _ 1 e F 3 a - a _ 1 = l (Existence of an inverse) 3. a • (/3 + 7) = (a • /?) + (a • 7 ), Va, 0,7 e F (Distributive) 14 2 Preliminary Linear Algebra D efinition 2.1.2 Let ¥ be a field. A set V with two operations + :V xV ^V Addition • : F x V H-> V Scalar multiplication is called a vector space (linear space) over the field F if the following axioms are satisfied: 1. a) b) c) d) 2. a) b) c) d^ u + v = u + v, Vu, v G V (u + v) + w = u + (v + w), Vu, v, w G V 3 a distinguished element denoted by 8 3 W G V, v + 6 = v Vw G V 3 unique - v eV B v + (-v) = 6 a • (0 • u) = ( a • /3) • u , Va,^ G F , VM G V a • (u + v) = ( a • u) + (a • v), Va G F , Vu,v eV (a + p) • u = ( a • u) + (p • u), Va, p G F , VM G F 1 • w = w, VM G V, where 1 is the multiplicative identity ofW E xample 2.1.3 Mn = { ( a i , a 2 , . . . , Q „ ) J ' : Q i , a 2 , . . . , « r l 6 R } is a vector space overR with(aci,a2,-. .,an)+{Pi,P2,---,Pn) = (ai+Pi,oi2+P2,--,an+ Pn); c- (cti,a2,-.. , a „ ) = (cai,ca2,. ..,can); and 6 — ( 0,0,. . . , 0 ) r . E xample 2.1.4 The set of all m by n complex matrices is a vector space over C with usual addition and multiplication. P roposition 2.1.5 In a vector space V, i. 0 is unique. ii. 0 • v = 6, Mv G V. Hi. (—1) • v = —v, V G V. w iv. -6 = 6. v. a-v = 6<&a = 0orv = 8. Proof. Exercise. 2 .1.2 Subspaces D efinition 2.1.6 Let V be a vector space overW, and let W C V. W is called a subspace ofV ifW itself is a vector space over F . P roposition 2.1.7 W is a subspace of V if and only if it is closed under vector addition and scalar multiplication, that is u>i, w2 G W, a i , c*2 € F < > ai • w± + a2 • w2 G W. ^ Proof. (Only if: =>) Obvious by definition. (If: <=) we have to show that 6 G W and Vw G W, -w G W. i. Let a i = 1, a>2 = — 1, and w\ = W2- T hen, l-wi + ( -1) •wi=w1 + (-wi) = 9 eW. • 2.1 Vector Spaces ii. Take any w. Let e*i = - 1 , a 2 = 0, and wi = w. Then, (-l)-w + (0)-w2 =-w eW. D 15 E xample 2.1.8 S C R 2 x 3 , consisting of the matrices of the form 0P 7 is a subspace of j>2x3 a a - P a + 27 P roposition 2.1.9 IfWx,W2 are subspaces, then so is W\ l~l W2. Proof. Take u>i, u>2 € Wi n W2, a i , a2 £ F . i. wi, w2 G Wi =>• a i • wi + a 2 • w2 € Wi ii. wi,w2 e W2 => cti • Wi + a2 • w2 £ W2 T hus, a itui + a2w2 € Wi n W 2 . • R emark 2.1.10 IfW\, W2 are subspaces, then W\ UW2 is not necessarily a subspace. D efinition 2.1.11 Let V be a vector space over ¥, X C V. X is said to be linearly dependent if there exists a distinct set of xi,x2,... ,Xk £ X and scalars a\,a2, ...,atk 6 F not all zero 3 5 ^ i = 1 o^Xi = 9. Otherwise, for any subset of size k, k X\,X2,...,Xk £ X , 2_2aixi — ® => al — a2 = ••• = <*k = 0. In this case, X is said to be linearly independent. We term an expression of the form $ Z i = 1 ot{Xi as linear combination. In particular, if JZi=i ai — 1» we ca^ ^ affine combination. Moreover, if Si=i ai = 1 and ai > 0, Vi = 1,2, ...,k, it becomes convex combination. On the other hand, if a* > 0, Vi = 1 ,2,..., k; then X )=i said to be canonical combination. E xample 2.1.12 In Rn, let E = { e;}" = 1 where ef = (0, • • • 0 ,1,0, • • • , 0) is the ith canonical unit vector that contains 1 in its ith position and 0s elsewhere. Then, E is an independent set since «i aiei H ha„en = a„ at = 0, Vi Let X = {xi}"=1 where xf = (0, • • - 0,1,1, • • • , 1) is the vector that contains 0s sequentially up to position i, and it contains Is starting from position i onwards. X is also linearly independent since 16 2 Preliminary Linear Algebra 8 = a\X\ + V anxn => a ; = 0, Mi. Let Y = {Vi}"=1 where yf = (0, • • - 0, - 1 , 1 , 0 , • • • , 0) is the vector that contains -1 in ith position, 1 in(i + l)st position, and 0s elsewhere. Y is not linearly independent since y\ + • • • + yn — #• D efinition 2.1.13 Let X C V. The set Span(X)= \v=YlaiXi £V : xi,x2,..-,xk€ X; ai,a2,---,ak eF; k€N> is called the span of X. If the above linear combination is of the affine combination form, we will have the affine hull of X; if it is a convex combination, we will have the convex hull of X; and finally, if it is a canonical combination, what we will have is the cone of X. See Figure 2.1. Cone(x) , Affine(p,q)v Affine b Convex / Span(p.q)=R Span(x) Convex(p,q) Fig. 2 .1. The subspaces defined by {a;} and {p, q}. P roposition 2.1.14 Span(X) is a subspace ofV. Proof. Exercise. 2 .1.3 Bases D efinition 2.1.15 A set X is called a basis for V if it is linearly independent and spans V. • 2.2 Linear transformations, matrices and change of basis 17 R emark 2.1.16 Since Span(X) C V, in order to show that it covers V, we only need to prove that Vv € V, v € Span(X). E xample 2.1.17 In K n , E = { ej}" =1 is a basis since E is linearly independent andVa = ( a i , a 2 , . . -,an)T € K n , a = a^ei -\ 1- ane„ € Span(E). X — {xi}™=1 is also a basis for Rn since Va = ( a i , a 2 , . . . ,an)T € R n , a = aixi + ( a 2 - " l ) ^ H 1- K - an-i)xn £ Span(X). P roposition 2.1.18 Suppose X = {a?i}7=i *s a ^0Sl'5 / o r ^ ower ^- ^ e n » a j Vw £ l^ can be expressed as v = E ?=i aixi where cti 's are unique. b) Any linearly independent set with exactly n elements forms a basis. c) All bases for V contain n vectors, where n is the dimension ofV. R emark 2.1.19 Any vector space V of dimension n and an field F™ have an isomorphism. Proof. Suppose X = {xi}"=1 is a basis for V over F. Then, a) Suppose v h as two different representations: v = Y17=iaix' = Y^i=i&iXiThen, 6 — v — v = E i = i ( a i ~ Pi)xi =^ °-% — ft, Vz — 1 ,2,..., n. C ontradiction, since X is independent. b) Let Y = {j/i}7=i be linearly independent. Then, yi = Yl^ixi (40> where at least one S{ ^ 0. Without loss of generality, we may assume that Si ^ 0. Consider Xi = {yi,x?,... ,xn}. Xi is linearly independent since 6 = n-dimensional fttfi+E?=2#** = / M E W * * + E r = 2 f t ^ = ft^^i+£r=2(ft^ + fr)xi =*• ft<5i = 0; PiSi + ft = 0, Vi = 2 , . . . , n =*• )8i = 0 (<Ji # 0); and ft = 0, Vi = 2 , . . . , n. Any o e K can be expressed as v = E ?=i 7*:c* = 7 iai + E_r=27iffi 1 <J 1 a; ( ) 1 T hus, S pan(Xi) = V. Similarly, X2 = {yi,y2,x3,...,xn} u = 7i(<^r 2/i - E r=2 r ^ i) * = (7i^r )yi + E"=2(7i - n s ^ s ^ . is a basis. Xn = {2/1,2/2, • • • ,2/n} = Y is a basis. c) Obvious from part b). • R emark 2.1.20 Since bases for V are not unique, the same vector may have different representations with respect to different bases. The aim here is to find the best (simplest) representation. 2 .2 Linear transformations, matrices and change of basis 2 .2.1 Matrix multiplication Let us examine another operation on matrices, matrix multiplication, with t he help of a small example. Let A e K 3 x 4 , B G R 4 x 2 , C € R 3 x 2 18 2 Preliminary Linear Algebra C l l C12 C 21 C22 C31 C32 C = AB = _ a n a i 2 a i 3 014 0 21 022 ^23 «24 ^ 31 «32 O33 034 j fell &12 &21 &22 631 fe32 641 642 _ O llfell + 012621 + 013&31 + 014&41 OH&12 + 012622 + 0 13632 + O14642 021&11 + 022fe21 + 023fe31 + 024641 O21612 + 022&22 + O23632 + 024642 031&11 + «32fe21 + 033631 + 034641 0 3 1 6 ^ + 032622 + 033632 + O34642 Let us list the properties of this operation: P roposition 2.2.1 Let A, B, C, D fee matrices and x be a vector. 1. 2. 3. 4. 5. {AB)x = A(Bx). {AB)C = A{BC). A(B + C) = AB + AC and (B + C)D = BD + CD. AB = BA does not hold (usually AB ^ BA) in general. Let In be a square n by n matrix that has Is along the main diagonal and Os everywhere else, called identity matrix. Then, AI = IA = A. 2 .2.2 Linear transformation D efinition 2.2.2 Let A e R m x n , i e l " . The map x i-> Ax describing a transformation K i-> K m with property (matrix multiplication) ™ Vx, y € R "; Vo, 6 € K, A(bx + cy) = b(Ax) + c(Ay) is called linear. R emark 2.2.3 Every matrix A leads to a linear transformation A. Conversely, every linear transformation A can be represented by a matrix A. Suppose the vector space V has a basis {vi,t>2> • • • ,vn} and the vector space W has a basis {u>i,W2, • • •, wm}. Then, every linear transformation A from V to W is represented by an m by n matrix A. Its entries atj are determined by applying A to each Vj, and expressing the result as a combination of the w's: AVJ = ^2 aHwi, j = 1 ,2,..., n. i=i E xample 2.2.4 Suppose A is the operation of integration of special polynomials if we take l,t,t2,t3, • • • as a basis where Vj and Wj are given by V~x. Then, AVJ = / V~x dt = — = -Wj . v J J 3 j+1 2.2 Linear transformations, matrices and change of basis "0 0 10 0\ For example, if dim V = 4 and dim W = 5 then A = 00 to integrate v(t) = 2t + 8t3 = 0«i + 2u2 + 0u3 + 8v4: "0 0 10 0\ 00 0 0" 00 00 \0 "0" 2 0 8 "0" 0 1 <^ y (2* + 8t3) dt = t2 + 2t4 = w3 + 2w5. 0 2 0 0" 00 00 |0 19 Let us try o oo \ o oo \ P roposition 2 .2.5 If the vector x yields coefficients ofv when it is expressed in terms of basis {v\, V2, • • •, vn}, then the vector y = Ax gives the coefficients of Av when it is expressed in terms of the basis {w\,W2, • • • ,wm}. Therefore, the effect of A on any v is reconstructed by matrix multiplication. m Av = Y2yiWi i=\ = 5Z aijXJWii,3 Proof. n n n V = J2 i i ^ j=l xv Av = A (52 i ^ 1 xv = Z ] xiAvi 1 = X) xi X j ai Wi i i- D P roposition 2 .2.6 / / the matrices A and B represent the linear transformations A and B with respect to bases {vi} in V, {u>i} in W, and {zi} in Z, then the product of these two matrices represents the composite transformation BA. Proof. A : v i->- Av B : Av i-> BAv => BA : v >-> BAv. D E xample 2 .2.7 Let us construct 3 x 5 matrix that represents the second derivative J J I , taking P4 (polynomial of degree four) to Pit4 ^ 4tz, t3 M- 3t 2 , t2 >> 2t, 11-> 1 -• 0020 0 0006 0 0 0 0 0 12 =*> B = 0 1000 0 0200 0 0030 0 0004 A= 0100 0020 0003 AB = Let v(t) = 2t + 8t3, then 2 d v(t) _ dt2 0020 0 0006 0 0 0 0 0 12 ' 0' 2 " 0" 0 = 48 = 48*. 8 0 0 20 2 Preliminary Linear Algebra P roposition 2 .2.8 Suppose {vi,v2,. ..,vn} and { wi, w2, • • •, wn} are both bases for the vector space V, and let v € V, v = Y^lxivi ~ J2"yjwj- V sx Vj = ]T™ SijWi, then yt = YJl ij jProof. y ] XjVj - ] P ^2 XjSijWi is equal t o ] P y{Wi J ^ ^ j i i i i j SijXjWi. • P roposition 2 .2.9 Let A : V ^ V. Let Av be the matrix form of the transformation with respect to basis {vi,v2,. •. ,vn) and Aw be the matrix form of the transformation with respect to basis {wi,W2,.-.,wn}. Assume that Vj = J2i sijwj- Then, Proof. Let v € V, v — J2xjvj- ^x g i y e s t he coefficients with respect t o w's, t hen AwSx yields t he coefficients of Av w ith respect t o original w's, a nd finally S~1AwSx gives t he coefficients of Av w ith respect t o original u 's. 0 R emark 2.2.10 Suppose that we are solving the system Ax = b. The most appropriate form of A is In so that x = b. The next simplest form is when A is diagonal, consequently Xi = £-. In addition, upper-triangular, lowertriangular and block-diagonal forms for A yield easy ways to solve for x. One of the main aims in applied linear algebra is to find a suitable basis so that the resultant coefficient matrix Av = 5 _ 1 >l l „5 has such a simple form. 2 .3 Systems of L inear Equations 2 .3.1 Gaussian elimination Let us t ake a system of linear m e quations with n unknowns Ax = b. In p articular, 2u + v + w— 1 "211" 4u + v=-2 <& 4 10 -2u + 2v + w= 7 -2 21 Let us apply some elementary row o perations: 5 1. S ubtract 2 times t he first equation from t he second, 52. S ubtract —1 times t he first equation from t he t hird, 5 3. S ubtract —3 times t he second equation from t he t hird. T he result is an equivalent b ut simpler system, Ux t riangular: "2 1 1" u 1" 0-1 -2 V -4 = 0 0-4 w -4 c where U is upperu V = w -2 7 r 2.3 Systems of Linear Equations 21 D efinition 2.3.1 A matrix U (L) is upper(lower)-triangular if all the entries below (above) the main diagonal are zero. A matrix D is called diagonal if all the entries except the main diagonal are zero. R emark 2.3.2 / / the coefficient matrix of a linear system of equations is either upper or lower triangular, then the solution can be characterized by backward or forward substitution. If it is diagonal, the solution is obtained immediately. Let us name the matrix that accomplishes SI (£21), subtracting twice the first row from the second to produce zero in entry (2,1) of the new coefficient m atrix, which is a modified J 3 such that its (2,l)st entry is - 2 . Similarly, t he elimination steps S2 and S3 can be described by means of £31 and £32, respectively. £•21 100 "100" 100" 2 1 0 , £ 31 — 0 1 0 , £ 32 — 0 1 0 101 031 001 These are called elementary matrices. Consequently, E32E31E21A = U and £ 3 2 £3i£2ib = c, 100" - 2 1 0 is lower triangular. If we undo the steps of - 5 3 1_ Gaussian elimination t hrough which we try to obtain an upper-triangular system Ux = c t o reach the solution for the system Ax = b, we have where £32 £31 £21 = A - # 3 2 -^31 E2\ U: LU, where p—1171—1171—1 • ^21 ^ 3 1 -^32 "100" 2 10 001 100' 0 10 -10 1 '100' 0 10 = 03 1 1 00 2 10 -1 - 3 1 is again lower-triangular. Observe that the entries below the diagonal are exactly the multipliers 2 , - 1 , and - 3 used in the elimination steps. We term L as the matrix form of the Gaussian elimination. Moreover, we have Lc = b. Hence, we have proven the following proposition that summarizes the Gaussian elimination or triangular factorization. P roposition 2.3.3 As long as pivots are nonzero, the square matrix A can be written as the product LU of a lower triangular matrix L and an upper triangular matrix U. The entries of L on the main diagonal are Is; below the main diagonal, there are the multipliers Uj indicating how many times of row j is subtracted from row i during elimination. U is the coefficient matrix, which appears after elimination and before back-substitution; its diagonal entries are the pivots. 22 2 Preliminary Linear Algebra In order to solve x = A~~xb — U~1c = U~1L~1b we never compute inverses t hat would take n 3 -many steps. Instead, we first determine c by forwardsubstitution from Lc = b, t hen find x by backward-substitution from Ux = c. T his takes a total of n2 o perations. Here is our example, 1 00 2 10 1 -3 1 211 0-1 -2 0 0-4 C\ Cl = cz Zl 1 -2 7 1 -4 -4 C\ =» C2 C3 = 1 -4 -4 -1 2 1 xx => Z2 23 X2 = = xz R emark 2.3.4 Once factors U and L have been computed, the solution x' for any new right hand side b' can be found in the similar manner in only n2 operations. For instance b' = 8 11 3 c\ =» c'i = 4 8 -5 -4 x\ => x 2 = x3 2 3 1 R emark 2.3.5 We can factor out a diagonal matrix D from U that contains pivots, as illustrated below. I di " 12 d\ "13 di .., 1 u d2 d„ 1 H2a d2 ... d2 1 Consequently, we have A = LDU, where L is lower triangular with Is on the main diagonal, U is upper diagonal with Is on the main diagonal and D is the diagonal matrix of pivots. LDU factorization is uniquely determined. R emark 2.3.6 What if we come across a zero pivot? ities: We have two possibil- Case (i) If there is a nonzero entry below the pivot element in the same column: We interchange rows. For instance, if we are faced with "0 2" 34 u V - V 01 _ 10_ represents the exchange. A permutation matrix P^i is the modified identity we will interchange row 1 and 2. The permutation matrix, P\2 2.3 Systems of Linear Equations 23 matrix of the same order whose rows k and I are interchanged. Note that Pki — P[^ (exercise!). In summary, we have PA = LDU. Case (ii) If the pivot column is entirely zero below the pivot entry: The current matrix (so was A) is singular. Thus, the factorization is lost. 2 .3.2 Gauss-Jordan method for inverses D efinition 2.3.7 The left (right) inverse B of A exists ifBA = I (AB = I). P roposition 2.3.8 BA = I and AC = I < > B = C. £ Proof. B(AC) = (BA)C &BI = IC&B = C. O P roposition 2.3.9 If A and B are invertible, so is AB. (AB)'1 Proof. (AB^B^A-1) (B^A-^AB = AiBB-^A'1 = B~l{A~lA)B = AIA'1 = B^IB = AA~X = I. = B~XB = 7. • = B-1A~1. R emark 2.3.10 Let A = LDU. A-1 = U^D^L-1 is never computed. If we consider AA_1 — I, one column at a time, we have AXJ = e j,Vj. When we carry out elimination in such n equations simultaneously, we will follow the Gauss-Jordan method. E xample 2.3.11 In our example instance, [A\eie2e3] = " 2 1 1 1 0 0" 410 010 -2 2 1 001 "2 1 1 1 0 0 ' 0 - 1 - 2 - 2 10 0 3 2 101 I _I 8 8 8 ->• "2 1 1 1 0 0 " - » 0 - 1 - 2 - 2 1 0 = \U\L-1] 0 0-4 -5 3 1 1 0 0| 1 oioj OOll k 2 4 k 2 4 k 2 4 = m^1] 5 _3 _I 24 2 Preliminary Linear Algebra 2 .3.3 The most general case In this subsection, we are going to concentrate on the equation system, Ax = b, where we have n unknowns and m equations. A xiom 2.3.12 The system Ax = b is solvable if and only if the vector b can be expressed as the linear combination of the columns of A (lies in Spanfcolumns of A] or geometrically lies in the subspace defined by columns of A). D efinition 2.3.13 The set of non-trivial solutions x ^ 8 to the homogeneous system Ax = 8 is itself a vector space called the null space of A, denoted by R emark 2.3.14 All the possible cases in the solution of the simple scalar equation ax = /? are below: • a 7^ 0: V/3 e R, 3a; = £ € K (nonsingular case), • a = (3 = 0: Vx € R are the solutions (undetermined case), • a — 0, (3 ^ 0 : there is no solution (inconsistent case). Let us consider a possible LU decomposition of a given A 6fl£»™xnw ith t he help of the following example: 1 332" "1332' "1332" 2 6 95 -» 0 031 -> 0 031 1 -330 0 062 0 000 T he final form of U is upper-trapezoidal. U. D efinition 2.3.15 An upper-triangular (lower-triangular) rectangular matrix U is called upper- (lower-)trapezoidal if all the nonzero entries Uij lie on and above (below) the main diagonal, i < j (i > j). An upper-trapezoidal matrices has the following "echelon" form: © ~ol© 0 0 0 0 000 *** ** 0 0 0 000 © * 00000000 00000000 In order to obtain such an U, we may need row interchanges, which would i ntroduce a permutation matrix P. T hus, we have the following theorem. T heorem 2.3.16 For any A 6 R m x n , there is a permutation matrix P, a lower-triangular matrix L, and an upper-trapezoidal matrix U such that PA = LU. 2.4 The four fundamental subspaces 25 D efinition 2.3.17 In any system Ax = b < > Ux = c, we can partition the £ unknowns Xi as basic (dependent) variables those that correspond to a column with a nonzero pivot 0 , and free (nonbasic,independent) variables corresponding to columns without pivots. We can state all the possible cases for Ax = b as we did in the previous r emark without any proof. T heorem 2.3.18 Suppose the m by n matrix A is reduced by elementary row operations and row exchanges to a matrix U in echelon form. Let there be r nonzero pivots; the last m — r rows of U are zero. Then, there will be r basic variables and n — r free variables as independent parameters. The null space, Af(A), composed of the solutions to Ax = 8, has n — r free variables. If n — r, then null space contains only x = 6. Solutions exist for every b if and only if r = m (U has no zero rows), and Ux = c can be solved by back-substitution. If r < m, U will have m — r zero rows. If one particular solution x to the first r equations of Ux = c (hence to Ax = b) exists, then x + ax, \/x G Af(A) \ {6} , Va S R is also a solution. D efinition 2.3.19 The number r is called the rank of A. 2.4 The four fundamental subspaces R emark 2.4.1 If we rearrange the columns of A so that all basic columns containing pivots are listed first, we will have the following partition of U: A = [B\N] -> U = mr UB\UN o -^v = Ir\VN o where B € R * , N € M™x("-'-)j \jB < Rrxr^ Uff £ R rx(n-r) > o is an = (m-r) x n matrix of zeros, VN £ K r x ( n - r > , and Ir is the identity matrix of order r. UB is upper-triangular, thus non-singular. If we continue from U and use elementary row operations to obtain Ir in the UB part, like in the Gauss-Jordan method, we will arrive at the reduced row echelon form V. 2 .4.1 T h e r ow s pace of A D efinition 2.4.2 The row space of A is the space spanned by rows of A. It is denoted by 1Z(AT). Tl(AT) = Spandat}^) =lyeRm:y =f>a< j = {d G R m : 3y € R m 9 yTA = dT) . 26 2 Preliminary Linear Algebra P roposition 2 .4.3 The row space of A has the same dimension r as the row space of U and the row space of V. They have the same basis, and thus, all the row spaces are the same. Proof. Each elementary row o peration leaves t he row space unchanged. 2 .4.2 T he c olumn s pace o f A D efinition 2 .4.4 The column space of A is the space spanned by the columns of A. It is denoted by H(A). 71(A) = Span {a^}nj=1 = \y € R" : y = ^ / 3 , - a ' = {b e R n : 3x E R" 3 Ax = b} . P roposition 2 .4.5 The dimension of column space of A equals the rank r, which is also equal to the dimension of the row space of A. The number of independent columns equals the number of independent rows. A basis for 71(A) is formed by the columns of B. D efinition 2 .4.6 The rank is the dimension of the row space or the column space. 2 .4.3 T he n ull space (kernel) o f A P roposition 2 .4.7 N(A) = {x G R n : Ax = 0(Ux = 6,Vx = 9)} = Af(U) = tf(V). • P roposition 2 .4.8 The dimension of J\f(A) is n — r, and a base for Af(A) \ -VN~ is the columns ofT = InProof. Ax = 6 « • Ux = 0 <£• Vx - 6 « • xB + VNxN = 0. The columns of T Let y = EjajTi, -VN~ *n—r is linearly independent because of t he last (n — r) Q coefficients. Is t heir span Af(A)? Ay = £ , - « ; ( - * # + V&) = 6. T hus, Span{{Ti}nZD DM(A)1 Let x xB xN XB XN M{A). Is Span({Ti}n=l) Ax - 6 <& xB + VNXN eM{A). xN T hen, Span({Ti}".:;) = 8 <> x = ^ ~-VN *n — r G T hus, Span({Ti}n=l)DAf(A). D 2.4 The four fundamental subspaces 2 .4.4 T he left null space o f A 27 D efinition 2 .4.9 The subspace of Rm that consists of those vectors y such that yTA = 6 is known as the left null space of A. M(AT) = { ! / e R m : yTA = 9} . P roposition 2.4.10 The left null space M{AT) is of dimension m - r, where the basis vectors are the lastm-r rows ofL~xP of PA = LU orL~lPA = U. Proof. A = [A\Im] - • V •• T hen, ( L _ 1 P) SUA = 6. • Si Sn Ir\VN o L~lP where Sn is the last m - r rows of L lP. T hen Fig. 2 .2. The four fundamental subspaces defined by A G 2 .4.5 T he F undamental Theorem o f L inear Algebra T heorem 2.4.11 TZ(AT)= row space of A with dimension r; N{A)= null space of A with dimension n — r; 11(A) = column space of A with dimension r; Af(AT)— left null space of A with dimension m — r; R emark 2.4.12 From this point onwards, we are going to assume that n> m unless otherwise indicated. 28 2 Preliminary Linear Algebra P roblems 2 .1. G raph spaces D efinition 2.4.13 Let GF(2) be the field with + and x (addition and multiplication modulo 2 on I?) 01 0 1 and 10 01 00 01 Fig. 2 .3. The graph in Problem 2.1 Consider t he n ode-edge incident matrix of the given graph G = (V, E) over G,F(2), A G RII^HXIISH: 12 3456 789 10 11 12 13 1 1 0000000 0 0 0 0 1 00000001 0 0 0 0 01 1000000 0 0 0 0 0011000010 0 0 0 0 001 10000 1 0 0 0 0 0001 1000 0 0 1 1 0 00001 1 0 0 0 1 0 0 000000010 1 1 0 1 0000001100 0 1 0 a b c A= d e f 9 h T he addition + o perator helps t o point o ut the end points of the p ath formed by the added edges. For i nstance, if we a dd the first a nd n inth columns of A, we will have [1,0,0,1,0,0,0,0,0] T , which indicates t he end points (nodes a a nd d) of the p ath formed by edges one and nine. (a) Find t he reduced row echelon form of A working over GF(2). I nterpret 2.5 Web material t he meaning of the bases. (b) Let T = { 1,2,3,4,5,6,7,8} and Tx = E \ T = { 9,10,11,12,13}. Let A = ® 29 . Let Z - [h\N]. For each row, zt,i € T, color the edges color the edges with non-zero w ith non-zero entries. Interpret z, (c) Let y = , . For each column yj, j £TX, e ntries. Interpret j/j. (d) Find a basis for the four fundamental subspaces related with A. 2 .2. D erivative of a polynomial Let us concentrate on a (n - k + 1) x (n + 1) real valued matrix A(n, k) t hat represents "taking kth derivative of nth order polynomial" P(t) =a0 + ait + --- + a„tn. (a) Let n = 5 a nd k = 2. Characterize bases for the four fundamental subspaces related with .4(5,2). (b) Find bases for and the dimensions of the four fundamental subspaces related with A(n, k). (c) Find B(n, k), t he right inverse of A(n, k). C haracterize the meaning of the underlying transformation and the four fundamental subspaces. 2 .3. As in Example 2.1.12, let Y = {2/j}™=1 be defined as yf = ( 0 , - " 0 , - l , l , 0 , - - - , 0 ) , t he vector that contains -1 in ith position, 1 in (i + l)st position, and 0s elsewhere. Let A = [2/1I2/2I • • • \yn]- C haracterize the four fundamental subspaces of A W e b material http://aigebra.math.ust.hk/matrix_iinear_trans/02_iinear_transform/ lecture5.shtml http://algebra.math.ust.hk/vector_space/ll_changebase/lecture4.shtml http://archives.math.utk.edu/topics/linearAlgebra.html http://calculusplus.cuny.edu/linalg.htm http://ceee.rice.edu/Books/CS/chapter2/linear43.html http://ceee.rice.edu/Books/CS/chapter2/linear44.html http: //dictionary. reference. com/search?q=vector'/,20space http://distance-ed.math.tamu.edu/Math640/chapterl/node6.html http://distance-ed.math.tamu.edu/Math640/chapter4/node2.html http://distance-ed.math.tamu.edu/Math640/chapter4/node4.html http://distance-ed.math.tamu.edu/Math640/chapter4/node6.html 30 2 Preliminary Linear Algebra http://en.wiklbooks.org/wiki/Algebra/Linear_transformations http://en.wikibooks.org/wiki/Algebra/Vector_spaces http://en.wikipedia.org/wiki/Examples_of_vector_spaces http://en.wikipedia.org/wiki/Fundamental_theorem_of_linear_algebra http://en.wikipedia.org/wiki/Gauss-Jordan_elimination http://en.wikipedia.org/wiki/Gaussian_elimination http://en.wikipedia.org/wiki/Linear_transformation http://en.wikipedia.org/wiki/Vector_space http://encyclopedia.laborlawtalk.com/Linear_transformation http: //eom. springer. de/L/1059520. htm http: //eom. springer. de/t/t093180 .htm http://eom.springer.de/v/v096520.htm http://euler.mcs.utulsa.edu/~class_diaz/cs2503/Spring99/lab7/ node8.html http: //everything2. com/index. pl?node=vector'/.20space http://graphics.cs.ucdavis.edu/~okreylos/ResDev/Geometry/ VectorSpaceAlgebra.html http://kr.cs.ait.ac.th/~radok/math/mat5/algebral2.htm http://math.postech.ac.kr/~kwony/Math300/chapter2P.pdf http://math.rice.edu/"hassett/teaching/221fall05/linalg5.pdf http://mathforum.org/workshops/sum98/participants/sinclair/ outline.html http://mathonweb.com/help/backgd3e.htm http://mathworld.wolfram.com/Gauss-JordanElimination.html http://mathworld.wolfram.com/GaussianElimination.html http://mathworld.wolfram.com/LinearTransformation.html http://mathworld.wolfram.com/VectorSpace.html http://mizar.uwb.edu.pl/JFM/Voll/vectsp_l.html http://planetmath.org/encyclopedia/GaussianElimination.html http://planetmath.org/encyclopedia/ ProofOfMatrixInverseCalculationByGaussianElimination.html http://planetmath.org/encyclopedia/VectorField.html http://planetmath.org/encyclopedia/VectorSpace.html http://rkb.home.cern.ch/rkb/AN16pp/nodel01.html http://thesaurus.maths.org/mmkb/entry.html?action=entryById&id=2243 http://triplebuffer.devmaster.net/file.php?id=5&page=l http: //tutorial .math. lamar. edu/AHBrowsers/2318/ LinearTransformations.asp http://uspas.fnal.gov/materials/3_LinearAlgebra.doc http://vision.unige.ch/~marchand/teaching/linalg/ http://web.mit.edu/18.06/www/Video/video-fall-99.html http://www-math.cudenver.edu/~wbriggs/5718s01/notes2/notes2.html http://www-math.mit.edu/~djk/18_022/chapterl6/section01.html http://www.absoluteastronomy.com/v/vector_space http://www.amath.washington.edu/courses/352-spring-2001/Lectures/ lecture7_print.pdf http://www.answers.com/topic/linear-transformation http://www.answers.com/topic/vector-space http: //www. biost at. umn. edu/" sudiptob/pubb.8429/ 2.5 W e b material 31 MatureLinearAlgebra.pdf http://www.bookrags.com/sciences/mathematics/vector-spaces-wom.html http://www.cap-lore.com/MathPhys/Vectors.html http://www.cartage.org.lb/en/themes/Sciences/Mathematics/Algebra/ foci/topics/transformations/transformations.htm http: //www. cee.umd.edu/menufiles/ence203/fall01/Chapter'/.205c,/.20 (Simultaneous'/,20Linear'/,2http: //www. sosmath.com/matrix/systeml/ systeml.html http://www.cs.berkeley.edu/~demmel/cs267/lectureSparseLU/ lectureSparseLUl.html http://www.cs.cityu.edu.hk/~luoyan/mirror/mit/ocw.mit.edu/18/ 18.013a/f01/required-readings/chapter04/section02.html http://www.cs.nthu.edu.tw/~cchen/CS2334/ch4.pdf http://www.cs.ut.ee/~toomas_l/linalg/linl/node6.html http://www.cs.ut.ee/"toomas_l/linalg/linl/node7.html http://www.cse.buffalo.edu/"hungngo/classes/2005/Expanders/notes/ LA-intro.pdf http://www.dc.uba.ar/people/materias/ocom/apuntel.doc http://www.eas.asu.edu/"aar/classes/eee598S98/4vectorSpaces.txt http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/vector.html http://www.ee.nchu.edu.tw/~minkuanc/courses/2006_01/LA/Lectures/ Lecture'/.205,/.20-'/.202006. pdf http://www.eng.fsu.edu/~cockburn/courses/eel5173_f01/four.pdf http://www.everything2.com/index.pl?node_id=579183 http://www.fact-index.com/v/ve/vector_space_l.html http://www.faqs.org/docs/sp/sp-129.html http://www.fismat.umich.mx/~htej eda/aa/AS,L24.pdf http://www.geometrictools.com/Books/GeometricTools/BookSample.pdf http://www.krellinst.org/UCES/archive/classes/CNA/dir1.6/ u cesL6.html http://www.lehigh.edu/~brha/m43fall2004notes5_rev.pdf http://www.library.Cornell.edu/nr/bookcpdf/c2-2.pdf http://www.ltcconline.net/greenl/courses/203/Matrix0nVectors/ kernelRange.htm http://www.ltcconline.net/greenl/courses/203/Matrix0nVectors/ matrix_of_a_linear_transformatio.htm http: //www. m a. umist. ac. uk/t v/Teaching/Linear*/.20algebra'/.20B/ Spring'/.202003/lecture2 .pdf http://www.math.byu.edu/"schow/work/GEnoP.htm http://www.math.gatech.edu/"bourbaki/math2601/Web-notes/8.pdf http://www.math.gatech.edu/~mccuan/courses/4305/notes.pdf http://www.math.grin.edu/~stone/events/scheme-workshop/gaussian.html http://www.math.harvard.edu/"elkies/M55a.99/field.html http://www.math.hmc.edu/calculus/tutorials/lineartransformations/ http://www.math.hmc.edu/~su/pcmi/topics.pdf http://www.math.jhu.edu/"yichen/teaching/2006spring/linear/ review2.pdf http://www.math.niu.edu/~beachy/aaol/f ields.html http://www.math.nps.navy.mil/~art/ma3046/handouts/Mat_Fund_Spa.pdf 32 2 Preliminary Linear Algebra http://www.math.poly.edu/courses/ma2012/Notes/GeneralLinearT.pdf http://www.math.psu.edu/xu/45l/H0MEW0RK/computer6/node5.html http://www.math.rutgers.edu/"useminar/basis.pdf http://www.math.rutgers.edu/"useminar/lintran.pdf http://www.math.sfu.ca/~lunney/macm316/hw05/nodel.html http://www.math.ubc.ca/"carrell/NB.pdf http://www.math.uiuc.edu/documenta/vol-01/04.ps.gz http://www.math.uiuc.edu/Software/magma/text387.html http://www.math.uiuc.edu/"bergv/coordinates.pdf http://www.mathcs.emory.edu/~rudolf/mathl08/summl-2-3/nodel9.html http://www.mathematik.uni-karlsruhe.de/mi2weil/lehre/stogeo2005s/ media/eg.pdf http://www.mathonweb.com/help/backgd3.htm http://www.mathonweb.com/help/backgd3e.htm http://www.mathreference.com/fid,intro.html http://www.mathreference.com/la,lxmat.html http://www.mathreference.com/la,xform.html http: //www .mathresource. iitb. ac. in/linear'/,20algebra/ mainchapter6.2.html http://www.maths.adelaide.edu.au/people/pscott/linear_algebra/lapf/ 24.html http://www.maths.adelaide.edu.au/pure/pscott/linear_algebra/lapf/ 21.html http://www.maths.nottingham.ac.uk/personal/sw/HG2NLA/gau.pdf http://www.maths.qmul.ac.uk/~pj c/class_gps/chl.pdf http://www.mathwords.eom/g/gaussian_elimination.htm http://www.matrixanalysis.com/DownloadChapters.html http://www.met.rdg.ac.uk/~ross/DARC/LinearVectorSpaces.html http://www.numbertheory.org/courses/MP274/lintrans.pdf http://www.phy.auckland.ac.nz/Staff/smt/453707/chap2.pdf http://www.ping.be/~pingl339/lintf.htm http://www.purplemath.com/modules/systlin5.htm http://www.reference.com/browse/wiki/Vector_space http://www.rsasecurity.com/rsalabs/node.asp?id=2370 http://www.sosmath.com/matrix/systeml/systeml.html http://www.Stanford.edu/class/ee387/handouts/lect07.pdf http: //www. swgc .mun. ca/-richards/M2051/M2051'/.20-March,/.2010'/.20-7. 20Vector'/.20Spaces,/.20and'/.20Subspaces. doc http://www.ucd.ie/math-phy/Courses/MAPH3071/nummeth6.pdf http://www.what-means.com/encyclopedia/Vector http://www2.pare.com/spl/members/hhindi/reports/CvxOptTutPaper.pdf http://xmlearning.maths.ed.ac.uk/eLearning/linear_algebra/ binder.php?goTo=4-5-1-1 https://www.cs.tcd.ie/courses/baict/bass/4ictlO/Michealmas2002/ Handouts/12_Matrices.pdf 3 O rthogonality I n this chapter, we will analyze distance functions, inner products, projection a nd orthogonality, the process of finding an orthonormal basis, QR and singular value decompositions and conclude with a final discussion about how to solve the general form of Ax = b. 3 .1 Inner P r o d u c t s Following a rapid review of norms, an operation between any two vectors in the same space, inner product, is discussed together with the associated geometric implications. 3 .1.1 N o r m s Norms (distance functions, metrics) are vital in characterizing the type of network optimization problems like the Travelling Salesman Problem (TSP) w ith the rectilinear distance. D efinition 3.1.1 A norm on a vector space V is a function that assigns to each vector, v € V, a nonnegative real number \\v\\ satisfying i. \\v\\ >0,Vvy£9 and \\6\\ = 0, ii. \\av\\ - \a\ \\v\\, Ma € K; v £ V. Hi. \\u + v\\ < \\u\\ + \\v\\, Vu, v € V (triangle inequality). D efinition 3.1.2 Vrc G C n , the most commonly used norms, H-l^ , ||.|| 2 , H-H^, are called the li, li and l^ norms, respectively. They are defined as below: 1. \\x\\x = | xi| + --- + |arn|, 2. | |x|| 2 = ( |x 1 | 2 + --- + |o ; n | 2 )i ; 3 - H alloo z=max {la;l|)--->l;En|}- 34 3 Orthogonality Furthermore, we know the following relations: <x y/n < l oo — 1 2' l2< y/n < li<IMI2XI < X 1• R e m a r k 3 .1.3 The good-old Euclidian distance is the l<z norm that indicates the bird-flight distance. In Figure 3.1, for instance, a plane's trajectory between two points (given latitude and longitude pairs) projected on earth (assuming that it is flat!) is calculated by using the Pythagoras Formula. The rectilinear distance (l\ norm) is also known as the Manhattan distance. It indicates the mere sum of the distances along the canonical unit vectors. It assumes the dependence of the movements along with the coordinate axes. In Figure 3.1, the length of the pathway restricted by blocks, of the car from the entrance of a district to the current location is calculated by adding the horizontal movement to the vertical. The Tchebychev's distance (1^) simply picks the maximum distance among all movements along the coordinate axes, and thus, assumes total independence. The forklift in Figure 3.1 can move sideways by its main engine, and it can independently raise or lower its fork by another motor. The total time it takes for the forklift to pick up an object 10m. away from a rack lying on the floor and place the object on a rack shelf 3m. above the floor is simply the maximum of the travel time and the raising time. A detailed formal discussion of metric spaces is located in Section 10.1. - x1 I Jl • F ig. 3.1. Metric examples: ||.| 2 ' 11*111 ' ll-lloo D efinition 3.1.4 The length \\x\\2 of a vector x in K is the positive square ™ root of 3.1 Inner Products 35 R emark 3 .1.5 \\x\\l geometrically amounts to the Pythagoras formula applied (n-1) times. D efinition 3 .1.6 The quantity xTy is called inner product of the vectors x and y in K" n xTy = » =i ^x^ji. P roposition 3 .1.7 xTy = 0 # i l j . Proof. (<=) P ythagoras Formula: ||x|| + ||y|| = ||a; — y\\ , T h e last t w 0 \\x ~ y\\2 = T,7=i(xi ~Vi)2 = \\x\? + \\y\\2-2xTyidentities yield T t he conclusion, x y = 0. (=») xTy = 0 =*- II^H2 + \\yf = \\x - y\\2 =>x±y. • T heorem 3 .1.8 ( Schwartz Inequality) \xTy\ < \\x\\2 \\y\\2 , Proof. T he following holds Va € R: 0 < ||x + ay\\l =xTx + 2 \a\ xTy + o?yTy = \\x\\22 + 2 \a\ xTy + a2 \\y\\22 , ( *) x,y£Rn. Case (x A. y): In t his case, we have =>• xTy = 0 < \\x\\2 \\y\\2. Case (x JL y): Let us fix a = l^f. T hen, ( *) 0 < - ||a;||2 + ' ffljffi. • 3 .1.2 Orthogonal Spaces D efinition 3 .1.9 Two subspaces U and V of the same space R are called ™ orthogonal ifMu 6 J/,Vu G V, u Lv. P roposition 3.1.10 Af(A) andlZ(AT) are orthogonal subspaces of and H(A) are orthogonal subspaces of K m . W,M(AT) Proof. Let w G M(A) a nd v G H(AT) such that Aw = 6, a nd v = ATx for some x G R ". wTv = wT(ATx) = (wTAT)x — 9Tx — 0. • D efinition 3.1.11 Given a subspace V o /R n , the space of all vectors orthogonal to V is called the orthogonal complement of V, denoted by V1-. T heorem 3.1.12 (Fundamental Theorem o f L inear Algebra, Part 2 ) Af(A) = (n(AT))^, Af(A ) = (K(A))^, T K(AT) = 11(A) = (Af(A))\ (Af(AT))±. 36 3 Orthogonality R emark 3.1.13 The following statements are equivalent, i. a v =w . Hi. W ± V and dimV + dimW — n. P roposition 3.1.14 The following are true: i. N{AB)2M{B). ii. Tl(AB) C 11(A). iii.Af((AB)T)DAf(AT). iv. Tl{(AB)T) C Tl{BT). Proof. Consider the following: i. Bx = 0 => ABx = 0. Thus, Vx € M(B), x £ Af(AB). ii. Let b 3 ABx = b for some x, hence 3y = Bx 3 Ay = b. iii. I tems (iii) and (iv) are similar, since (AB)T = BTAT. C orollary 3.1.15 rank(AB) rank(AB) 3 .1.3 Angle between two vectors See Figure 3.2 and below to prove the following proposition. c = b — a =$• cos c = cos(6 — a) = cos b cos a + sin b sin a — J i L J^L 4. Jf2__^2_ _ "1^1 + U2V2 cosc W^V1. ± O < rank(A), < rank(B). ~ U\U\ U\ Nl " IHIIHI " U=(U Lfe) I v=(v„v2) X -Axis Fig. 3.2. Angle between vectors 3.1 Inner Products 37 P roposition 3.1.16 The cosine of the angle between any two vectors u and v is U C OSC : T V i u \\v\\ R emark 3.1.17 The law of cosines: \\u — v\\ = ||u|| + \\v|| — 2 ||u|| llvll cose. 3 .1.4 Projection Let p = xv where W = x 6 R is the scale factor. See Figure 3.3. (u - p) J- v «=> vT(u — p) = 0 & x = T Vx U X-Axis Fig. 3 .3. Projection D efinition 3.1.18 The projection p of the vector u onto the line spanned by T the vector v is given by p — %^-v. The distance from the vector u to the line is (Schwartz inequality) therefore u ir u 1 =-v vv uTu - 2^- + &?vTv V 1 = (»r»)("r;)-(«r«)a, V1 V V V 1 V 3 .1.5 Symmetric Matrices D efinition 3.1.19 A square matrix A is called symmetric if AT = A. P roposition 3.1.20 Let A e R m x n , rank{A) = r . The product ATA symmetric matrix and rank(ATA) = r . is a 38 3 Orthogonality Proof. (ATA)T = AT{AT)T = ATA. T Claim: N{A) = H{A A). i. M{A) C M{ATA) : x e M(A) =• Ax = 6 =>• 4 r A r - i T 9= »4i£ ii. Af(ATA) C M{A) : x e M{ATA) => A7'Ax = 6 => a; r i4 r Ac = 0 4* | |Ar|| 2 = 0 & Ax = 9, x € M(A). D R emark 3.1.21 ATA has n columns, so does A. Since Af(A) = dimhf(A) = n — r => dimR(ATA) = n - {n - r) = r. C orollary 3.1.22 / / rank(A) = n =>• ATA vertible (non-singular) matrix. N(ATA), is a square, symmetric, and in- 3.2 Projections and Least Squares Approximations Ax = 6 is solvable if b e R(A). If b £ R(A), t hen our problem is choose x 3 \\b — Ax\\ is as small as possible. Ax - b J. R(A) < > (Ay)T(Ax S yT[ATAx - b) = 0 <> ^ = 9^ ATAx = ATb. - .4T6] = 0 (yT jt 6) => ATAx -ATb P roposition 3.2.1 The least squares solution to an inconsistent system Ax — b of m equations and n unknowns satisfies ATAx = ATb (normal equations). If columns of A are independent, then AT A is invertible, and the solution is x= (ATA)-1ATb. The projection of b onto the column space is therefore p = Ax = A{ATA)~lATb where the matrix P = A(ATA)"1 as projection matrix. = Pb, AT that describes this construction is known R emark 3.2.2 (I — P) is another projection matrix which projects any vector b onto the orthogonal complement: (I — P)b — b — Pb. P roposition 3.2.3 The projection matrix P = A(ATA)~1AT properties: a. it is idempotent: P2 — P. b. it is symmetric: PT — P. has two basic 3.2 Projections and Least Squares Approximations 39 Conversely, any matrix with the above two properties represents a projection onto the column space of A. Proof. T he projection of a projection is itself. P2 = A[{ATA)-1ATA](ATA)-1AT We know that ( S " 1 ) 7 = (BT)-\ PT = (AT)T[(ATA)-1}TAT 3 .2.1 Orthogonal bases D efinition 3.2.4 A basis V = V7V. = { VJ}" = 1 = A(ATA)~lAT = P. Let B = ATA. = A(ATA)~lAT = P. D = A[AT(AT)T}'1AT is called orthonormal if (ortagonality) (normalization) (°^^Jj — E xample 3.2.5 E — {ej}™=1 is an orthonormal basis for M", whereas X = {xi}"=1 in Example 2.1.12 is not. P roposition 3.2.6 If A is an m by n matrix whose columns are orthonormal (called an orthogonal matrix), then ATA = In. P = AAT = aiaj H is the least squared solution for Ax = b. C orollary 3.2.7 An orthogonal matrix Q has the following properties: 1. QTQ = I = QQT> 2. QT = Q~\ 3. QT is orthogonal. E xample 3.2.8 Suppose we project a point aT = (a,b,c) into R 2 plane. Clearly, p — (a, b, 0) as it can be seen in Figure 3.4T h anaTn => x = ATb 4 e\ex a = a 0 0 i e2e|,Q = "0" b 0 P = eiej + e2e2 = "100' 010 0 0() a b c "100" 010 0 0 0_ = a b 0 Pa = 40 3 Orthogonality Pa=(a,b,0) F ig. 3.4. Orthogonal projection R emark 3.2.9 When we find an orthogonal basis that spans the ground vector space and the coordinates of any vector with respect to this basis is on hand, the projection of this vector into a subspace spanned by any subset of the basis has coordinates 0 in the orthogonal complement and the same coordinates in the projected subspace. That is, the projection operation simply zeroes the positions other than the projected subspace like in the above example. One main aim of using orthogonal bases like E = {ej}™=1 for the Cartesian system, W1, is to have the advantage of simplifying projections, besides many other advantages like preserving lengths. P roposition 3.2.10 Multiplication by an orthogonal Q preserves lengths \\Qx\\ = \\x\\, \fx; and inner products (Qx)T(Qy)=xTy,Vx,y. 3 .2.2 Gram-Schmidt Orthogonalization Let us take two independent vectors a and b. We want to produce two perpendicular vectors v\ and v2: , > V T° r Tp — vi =>• v{ v2 = 0 =>• vi ± v2. v(vx If we have a third independent vector c, then V3 = C vi = a, v2 = b — p = b v7f c Vi i V{ V\ vfA c V2 => V3 -L V2, V3 V$ V2 ±Vi. If we scale Vi,v2,v3, we will have orthonormal vectors: Vi Qi = i i — M , a2 v2 «2 v3 93 = "3 3.2 Projections and Least Squares Approximations 41 P roposition 3.2.11 Any set of independent vectors ai,a,2,- • • ,an can be converted into a set of orthogonal vectors v\, V2, • • •, vn by the Gram-Schmidt process. First, Vi = a\, then each Vi is orthogonal to the preceding v\, v-i,..., « i_i : Vi = a,vj a% u vfvi Vi i-lui Vi-l. For every choice of i, the subspace spanned by original a i , a 2 , - . . , a j is also spanned by v\, v-i,..., Vi. The final vectors « illJi=i { * i&} are orthonormal. E x a m p l e 3.2.12 Let ai = vi = a%, and a~ v\ 1 = Vj_ "1" 0 , a2 = 1 1 1 , as = 0 0 1 1 v(vi 2 * a2 - \vi -- = f i => v3 = a3 r!i V2 iui \v2 = Then, 1 1 • _ q\ " 1" l 0 V2 1 • 2 0 l 2" ?2 1" 1 . 2 1 2. vG 1 V6. and 03 9 12 . Ol 3 2 3 2 3. V3 — 1 v /3 1 . v s. 2 °2 i>i = -v/2oj «2 = §«i + «2 = \J\qi + as = \vx + \v2 + v3 = yj\qi + yj\q2 + yj §<»3 <£> [ a i , a 2 , a 3 ] = [gi, 02,03] <> A = QR. ^ P roposition 3.2.13 A — QR where the columns of Q are orthonormal vectors, and R is upper-triangular with \\vi\\ on the diagonal, therefore is invertible. If A is square, then so are Q and R. 42 3 Orthogonality D efinition 3.2.14 A = QR is known as Q~R decomposition. R emark 3.2.15 If A = QR, then it is easy to solve Ax — b: x = (ATA)-1ATb = (RTQTQR)-1RTQTb Rx = QTb. = {RTR)-lRTQTb = R~1QTb. 3 .2.3 Pseudo (Moore-Penrose) Inverse Ax = b<->Ax=p = Pb<&x = (ATA)~lATb. Ax = p have only one solution o- The columns of A are linearly independent <$• N{A) contains only 6 <& rank(A) = n •& ATA is invertible. Let A^ b e pseudo inverse of A. If A is invertible, then A^ = A"1. O therwise, A^ — (ATA)~*AT', if the above conditions hold. Then, x = A%. Otherwise, the optimal solution is the solution of Ax — p which is the one t hat has the minimum length. Let x~o 9 j4afo = P> x"o = xr + w where xr G TZ(AT) and w £ N{A). We have the following properties: i. Axr = A{xr + w) = Ax~o = p. ii. VS 9 Ax — p, x = xr + w w ith a variation in w p art only, where xr is fixed. 2 iii. \\xr +w\\ — \\xr\\ w P roposition 3.2.16 The optimal least squares solution to Ax = b is xr (or simply x), which is determined by two conditions 1. Ax — p, where p is the projection ofb onto the column space of A. 2. x lies in the row space of A. Then, x = A^b. E xample 3.2.17 A = "00 0 0 0/300 where a > 0, /3 > 0. [o 0 a 0 Then, K(A) = R 2 and p = Pb = (0, b2, b3)T "0 0 0 0' Ax =p •& 0 /3 0 0 0 0 a0 Xi X2 X~4 "0" = &2 b3 x2 — -r, X3 — —, £ i = a ; 4 = 0, with the minimum length! a a 3.2 Projections and Least Squares Approximations 0" => x = "0 0 0 OiO "0 0 0" 0 Thus, A* = 00 .0 0 0 . 43 L a = A*b = ooi 000 0 3 .2.4 Singular Value Decomposition D efinition 3.2.18 A 6 R m x n , A — Q\EQ% is known as singular value decomposition, where Qx G R m x m orthogonal, Q2 £ E""*"1 orthogonal, and E has a special diagonal form E= with the nonzero diagonal entries called singular values of A. P roposition 3.2.19 A* = Q2E^Ql where £+ Proof. | |Ac - 6|| = \\Qi2QZx - b\\ = \\EQ$x - Qfb\\. This is multiplied by Qf y = Q\x = Q2~1x w ith \\y\\ = ||a min \\Ey - Q\b\\ - » y = E^Qjb. ^x = Q2y = Q2E^Qjb => A^ = Q2E^Qj O R emark 3.2.20 A typical approach to the computation of the singular value decomposition is as follows. If the matrix has more rows than columns, a QR decomposition is first performed. The factor R is then reduced to a bidiagonal matrix. The desired singular values and vectors are then found by performing a bidiagonal QR iteration (see Remarks 6.2.3 and 6.2.8). 44 3 Orthogonality 3 .3 S u m m a r y for Ax = b Let us start with the simplest case which is illustrated in Figure 3.5. A G R " x " is square, nonsingular (hence invertible), rank(yl) = n = r. T hus, A represents a change-of-basis transformation from R" onto itself. Since n = r, we have V6 G Ti{A) = R™. Therefore, there exists a unique solution x = A~lb. If we have a decomposition of A (PA = LU, A = QR, A = QiEQ^), we follow an easy way to obtain the solution: (A = LU) => Lc = b, Ux = c using forward/backward substitutions as illustrated in the previous chapter; {A = QR) => Rx = QTb using backward substitution after multiplying the right hand side with QT; (A = Q\EQT) => x = Q-2E~lQjb using matrix multiplication operations after we take the inverse of the diagonal matrix E simply by inverting the diagonal elements. ••r:-&::-< 1-1 -:••:'•.•'.•'.'.•'•'• • : ; : -\ ; ;' : l ' GASE"' n ... 2 1 v F ig. 3.5. Unique solution: b £ 11(A), A : n x n, and r = n If A £ R m x n has full rank r = m < n, we choose any basis among the columns of A = [B\N] t o represent 11(A) = R m t hat contains b. In this case, we have a p = n — m dimensional kernel M(A) whose elements, being the solutions to the homogeneous system Ax = 0, extend the solution. Thus, we have infinitely many solutions XB = B~lb — B~1NXN, given any basis B. One such solution is obtained by .z'/v = 0 => XB = B~lb is called a basic 4 solution. In this case, we may use decompositions of B (B = LU, B = QR, B = Q1EQ2) t o speed up the calculations. mx If A G ! " has rank r < m < n as given in Figure 3.6, we have dim(M(A)) = p = n-r, dim(Af(AT)) = q = m-r and 11(A) = K(AT) = W. BN T he elementary row operations yield A . T here exists solution(s) Oqxn , only if b G 1Z(A). Assuming that we are lucky to have b G 1Z(A), and if x is a solution to the first r equations of Ax = b (hence to [B\N]x = b), t hen x + ax, V.x G N(A) \ {0} , VQ G R is also a solution. Among all solutions l l XB = B~ b — B~XNXN, XJV = 9 => XB = B~ b is a basic solution. We may use decompositions of B t o obtain XB as well. 3.3 Summary for Ax = b 45 Fig. 3.6. Parametric solution: b 6 TZ(A), A : m x n, and r — rank(A) W hat if b $. 11(A)? We cannot find a solution. For instance, it is quite h ard to fit a regression line passing through all observations. In this case, we are interested in the solutions, x, yielding the least squared error ||6 — J 4X|| 2 . If & € Af(AT), t he projection of b over TZ(A) is the null vector 0. Therefore, Af(A) is the collection of the solutions we seek. Fig. 3.7. Unique least squares solution: (A1A) is invertible and A^ = (ATA) l7 A If b is contained totally in neither TZ(A) nor J\f(AT), we are faced with the n on-trivial least squared error minimization problem. If ATA is invertible, t he unique solution is x = (ATA)~1ATb as given in Figure 3.7. The regression line in Problem 3.2 is such a solution. We may use A = QR or A = Q1SQ2 decompositions to find this solution easily, in these ways: Rx — QTb or x — Q2^Qfb, respectively. Otherwise, we have many x € R leading to the least squared solution as ™ in Figure 3.8. Among these solutions, we are interested in the solution with 46 3 Orthogonality w F ig. 3 .8. Least norm squared solution: (ATA) is not invertible and A^ = QiE'Qj T able 3.1. How to solve Ax = b, where A G R m x " C ase S ubcase r=n=m x=A~1b XB = Solution T ype E xact u nique Special Forms Inverse r=m < n b e - £(4) A=[B\N] r=m < n [A\\b B\N 6 ' B_16B~1Nxn XB = B-'bB^NXn A=LU => Lc=b,Ux=c A=QR => Rx=QTb A^A-1 A = QtEQl =» x=Q2Z-xQlb B=LU => Lc=b,UxB=c E xact B=QR => RxB=QTb m any A^B-1 B=Q-,EQl => xB=Q2S-1Qjb B=LU =• Lc=b,UxB-c E xact B=QR => RxB=QTb m any B=Q1EQ2r => xB=Q2Z~1QTb A* « B-1 o M(AT) A^ 0 r <m be - m any Trivial L east S quares x=a T I\N~\ O\ ' -N' I M.n~r ) n one (A A): b <jL 11(A) invertible (ATA): bgAf(A) n ot i nvertible T Unique x=A*b L east S quares m any L east N orm x=A*b m in.norm S quares V= A=QR aG> Rx=QTb A = QxEQl => x=Q2E-lQjb A^QiEQZ =* x=Q2E1iQjb A*= (ATA)~1AT A*= Q2E^Ql 3.4 Web material 47 t he smallest magnitude, in some engineering applications. We may use the singular value decomposition in this process. T he summary of the discussions about Ax = b is listed in Table 3.1. P roblems 3 .1. Q -R Decomposition 1 2 0-1 2 1-13 2 1-13 1 1 -3 1 F ind QR decomposition of A — 3 .2. L east Squares Approximation: Regression Assume that you have sampled n pairs of data of the form (x,y). Find the regression line that minimizes the squared errors. Give an example for n=5. 3 .3. A x = b Solve the following Ax = b using the special decomposition forms. "132" 8" (a) Let Ai= 2 1 3 and b\ = 19 using LU decomposition. 32 1 3 8 "2 1 3 1 0 (b) A2= 1 3 2 0 1 and 62 = 19 using LU decomposition. Find at least 32 1 1 0 3 two solutions. 12 45 using QR decomposition. and 63 = (c) Az = 78 10 11 -10 01 1 - 1 00 and 64 = using singular value decomposition. (d) A, = 0 1-10 0 0-11 "1 W eb material http://abel.math.harvard.edu/~knlll/math2lb2002/10-orthogonal/ orthogonal.pdf http://astro.temple.edu/~dhill001/modern/l-sect6-2.pdf http://avalon.math.neu.edu/~bridger/lschwart/lschwart.html http://ccrma-www.Stanford.edu/~j os/mdft/ 48 3 O rthogonality Norm_Induced_Inner_Product.html http://ccrma-www.Stanford.edu/~jos/r320/Inner_Product.html http://ccrma-www.Stanford.edu/"j os/sines/ Geometric_Signal_Theory.html http://ccrma.Stanford.edu/"j os/mdft/Inner_Product.html http://cnx.org/content/ml0561/latest/ http://elsa.berkeley.edu/~ruud/cet/excerpts/PartIOverview.pdf http://en.wikipedia.org/wiki/Inner_product_space http://en.wikipedia.org/wiki/Lp_space http://en.wikipedia.org/wiki/Moore-Penrose_inverse http://en.wikipedia.org/wiki/QR_decomposition http://en.wikipedia.org/wiki/Singular_value_decomposition http://engr.smu.edu/emis/8371/book/chap2/node8.html http://eom.springer.de/P/p074290.htm http://eom.springer.de/R/r130070.htm http://epoch.uwaterloo.ca/"ponnu/syde312/algebra/page09.htm http://epubs.s iam.org/sam-bin/dbq/art icle/30478 http://genome-www.Stanford.edu/SVD/ http://geosci.uchicago.edu/~gidon/geos31415/genLin/svd.pdf http://info.wlu.ca/"wwwmath/facuity/vaughan/ma255/ ma255orthogproj 05.pdf http://ingrid.ldeo.Columbia.edu/dochelp/StatTutorial/SVD/ http://iria.pku.edu.cn/~jiangm/courses/IRIA/nodell9.html http://isolatium.uhh.hawaii.edu/linear/lectures.htm http://kwon3d.com/theory/jkinem/svd.html http://library.lanl.gov/numerical/bookcpdf/c2-10.pdf http://linneus20.ethz.ch:8080/2_2_1.html http://lmb.informatik.uni-freiburg.de/people/dkats/ DigitallmageProcessing/pseudoInvNew.pdf http://mathnt.mat.jhu.edu/matlab/5-15.html http://mathnt.mat.jhu.edu/matlab/5-6.html http://maths.dur.ac.uk/"dmaOwmo/teaching/lh-la/LAnotes/node18.html http://mathworld.wolfram.com/InnerProductSpace.html http://mathworId.wolfram.com/Matrixlnverse.html http://mathworld.wolfram.com/Moore-PenroseMatrixlnverse.html http://mathworld.wolfram.com/QRDecomposition.html http://mathworld.wolfram.com/SchwarzsInequality.html http://mathworld.wolfram.com/SingularValueDecomposition.html http://mcraefamily.com/MathHelp/BasicNumberIneqCauchySchwarz.htm http://mymathlib.webtrellis.net/matrices/vectorspaces.html http://planetmath.org/encyclopedia/CauchySchwarzInequality.html http://planetmath.org/encyclopedia/InnerProduct.html http://planetmath.org/encyclopedia/InnerProductSpace.html http://planetmath.org/encyclopedia/ MoorePenroseGeneralizedlnverse.html http://planetmath.org/encyclopedia/NormedVectorSpace.html http://planetmath.org/encyclopedia/OrthogonalityRelations.html http://planetmath.org/encyclopedia/QRDecompositlon.html http://planetmath.org/encyclopedia/SingularValueDecompos it ion.html 3.4 W e b material 49 http://psblade.ucdavis.edu/papers/ginv.pdf http://public.lanl.gov/mewall/kluwer2002.html http://rkb.home.cern.ch/rkb/AN16pp/node224.html http://robotics.caltech.edu/~jwb/courses/ME115/handouts/pseudo.pdf http://staff.science.uva.nl/~brandts/NW2/D0WNL0ADS/hoofdstukl.pdf http: //tutorial .math. lamar. edu/AHBrowsers/2318/ InnerProductSpaces.asp http://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm http://wks7.itlab.tamu.edu/Math640/notes7b.html http://world.std.com/~sweetser/quaternions/quantum/bracket/ bracket.html http://www-ccrma.stanford.edu/~jos/mdft/Inner_Product.html http://www.axler.net/Chapter6.pdf http://www.ccmr.Cornell.edu/~muchomas/8.04/1995/ps5/node17.html http://www.ceo.caltech.edu/~mihai/Ma8-Fall2004/Notes/Notes4/n4.pdf http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/ SingularValueDecompos it ion.ps http: //www. cs. hartf ord. edu/~bpollina/m220/html/7.1/ 7.l.InnerProducts.html http://www.cs.rpi.edu/~flaherje/pdf/linll.pdf http://www.cs.unc.edu/~krishnas/eigen/node6.html http://www.cs.ut.ee/~toomas_l/linalg/linl/nodelO.html http://www.csit.fsu.edu/~gallivan/courses/NLA2/set9.pdf http://www.ctcms.nist.gov/~wcraig/variational/node2.html http://www.ctcms.nist.gov/~wcraig/variational/node3.html http://www.davidson.edu/math/will/svd/index.html http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/decomp.html http://www.emis.de/j ournals/AM/99-l/cruells.ps http://www.eurofreehost.com/ca/Cauchy-Schwartz_inequality.html http://www.everything2.com/index.pl?node_id=53160 http://www.fiu.edu/~economic/wp2004/04-08.pdf http://www.fmrib.ox.ac.uk/"tkincses/j c/SVD.pdf http://www.fon.hum.uva.nl/praat/manual/ generalized_singular_value_decomposition.html http://www.free-download-soft.com/info/sdatimer.html http://www.iro.umontreal.ca/~ducharme/svd/svd/index.html http://www.library.Cornell.edu/nr/bookepdf/c2-6.pdf http://www.mast.queensu.ca/~speicher/Section6.pdf http://www.math.duke.edu/education/ccp/materials/linalg/leastsq/ leas2.html http://www.math.duke.edu/education/ccp/materials/linalg/orthog/ http://www.math.harvard.edu/~knill/teaching/math21b2002/ 10-orthogonal/orthogonal.pdf http://www.math.ohio-state.edu/~gerlach/math/BVtypset/node6.html http://www.math.psu.edu/~anovikov/math436/h-out/gram.pdf http://www.math.sfu.ca/"ralfw/math252/weekl3.html http://www.math.ucsd.edu/~gnagy/teaching/06-winter/Math20F/w9-F.pdf http://www.math.umd.edu/~hck/461/s04/461s04m6.pdf http://www.mathes.duq.edu/larget/math496/qr.html 50 3 Orthogonality http://www.mathreference.com/top-ms,csi.html http://www.maths.adelaide.edu.au/people/pscott/linear_algebra/lapf/ 32.html http://www.maths.qmw.ac.uk/"sm/LAII/01ch5.pdf http://www.mathworks.com/access/helpdesk/help/techdoc/math/ mat_linalg25.html http://www.matrixanalysis.com/Chapter5.pdf http://www.mccormick.northwestern.edu/jrbirge/lec31_14nov2000.ppt http://www.mccormick.northwestern.edu/j rbirge/lec33_17nov2000.ppt http://www.nada.kth.se/kurser/kth/2D1220/Hsvd.pdf http://www.nasc.snu.ac.kr/sheen/nla/html/node19.html http://www.nationmaster.com/encyclopedia/Inner-product http://www.netlib.org/lapack/lug/node53.html http://www.public.asu.edu/~sergei/classes/mat242f99/LinAlg4.doc http://www.reference.com/browse/wiki/Inner_product_space http://www.sciencedaily.com/encyclopedia/lp_space http://www.uwlax.edu/faculty/will/svd/ http://www.vias.org/tmdatanaleng/cc.matrix.pseudoinv.html http://www.wooster.edu/math/linalg/LAFact s04.pdf www.chu.edu.tw/~ chlee/NA2003/NA2003-1.pdf www.math.umn.edu/"olver/appl_/ort.pdf www.math.uwo.ca/"aricha7/courses/283/weekl0.pdf www.maths.lse.ac.uk/Personal/mart in/fme5a.pdf www.maths.qmw.ac.uk/~sm/LAII/01ch5.pdf 4 E igen Values and Vectors In this chapter, we will analyze determinant and its properties, definition of eigen values and vectors, different ways how to diagonalize square matrices and finally the complex case with Hermitian, unitary and normal matrices. 4.1 Determinants 4 .1.1 Preliminaries P roposition 4.1.1 det A ^ 0 => A is nonsingular. R e m a r k 4 .1.2 Is A — XI (where X is the vector of eigen values) invertible? det(A - XI) = ? 0 where det(A — XI) is a polynomial of degree n in X, thus it has n roots. P roposition 4.1.3 (Cramer's Rule) Ax = b where A is nonsingular. Then, the solution for the jth unknown is j _ det(A(j <- b)) ~ det ,4 ' where A(j <— b) is the matrix obtained from A by interchanging column j with the right hand side b. P roposition 4.1.4 d et A = ± [product of pivots]. P roposition 4.1.5 \detA\ = Vol(P), where P=conv{Y^"=1eiai, e^ is the jth unit vector} is parallelepiped whose edges are from rows of A. See Figure 4-1Corollary 4.1.6 | detvl| = H?=1 \\a,i\\. D efinition 4.1.7 Let det A " 1 = g ^ . 52 4 Eigen Values and Vectors Volume=det(A) A= a n ai2 ai3 0 2 1 <J22 0,23 ^31 «32 «33 Fig. 4.1. |det A| = Volume(P). 4 .1.2 P r o p e r t i e s 1. T he determinant of J is 1. E xample 4.1.8 10 01 E xample 4.1.9 det ab cd ab = ad — cb. cd 1. 2. T he determinant is a linear function of any row, say the first row. ta tb ab = tad — ted = t cd cd 3. If A h as a zero row, then det .4 = 0. E xample 4.1.10 00 = 0. cd 4. T he determinant changes sign when two rows are exchanged. E xample 4.1.11 cd — cb — ad ab ab cd 5. T he elementary row operations of subtracting a multiple of one row from a nother leaves the determinant unchanged. E xample 4.1.12 a — acb — ad c d (ad — acd) — (be — acd) bc = ab cd 6. If two rows are equal (singularity!), then det A = 0. E xample 4.1.13 ab h=o. ab 4.1 Determinants 7. d et AT = d et A. E xample 4.1.14 a c = ad — cb a b cd bd 8. If A is t riangular, then det A = Yi7=i a" ( d e t / = 1). E xample 4.1.15 ab «0 = ad, = ad. Od cd 9. A,B € R n x n , nonsingular, d e t ( 4 5 ) = (det A)(det B). E xample 4.1.16 a b ef = (ad — cb)(eh — gf) = adeh — adgf — cbeh + cbgf. cd 9 h ae + bg af + bh = (ae + bg){cf + dh) - (af + bh)(ce + dg) ce + dg cf + dh = aecf + aedh + bgcf + bgdh — afce — afdg — bhce — bhdg = adeh — adgf — cbeh + cbgf. 10. Let A b e nonsingular, A = P-lLDU. T hen, d et A = d et P _ 1 det L d et D det U — ± (product of p ivots). 53 T he sign ± is the d eterminant of P _ 1 (or P) depending on w hether t he n umber of row exchanges is even or odd. We know det L = d et U = 1 from p roperty 7. E xample 4.1.17 By one Gaussian elimination step, we have ab cd 10a a1n 0 ad—be a ol since ab c d -» ab Od-^ . Thus, ab = ad — bc — det D. cd 11. d et A = anAii + aiiAa + ••• + ainAin ( property 1!) where A^'s are cofactors Atj = (-l)i+i det Mij where t he minor Mij is formed from A by deleting row i a nd column j . E xample 4.1.18 a n « i2 a i 3 O 21 0 2 2 ^ 2 3 0-31 0-32 0-33 an 0,22 0 2 3 ai2 ai3 a 1 32 ^ 33 ^ 21 ^ 31 23 a 33 + O21 0 22 &31 « 32 o22a3i) = 0 11(022033 - 0 23032) + 0 12(023031 - 0 21033) + 0 13(021032 - : 0 11022033 + 0 12023031 + 0 13021032 — 0 11023032 — 0 12021033 — 0 13022031. 54 4 Eigen Values and Vectors 4.2 Eigen Values and Eigen Vectors D efinition 4.2.1 The number X is an eigen value of A, with a corresponding nonzero eigen vector v such that Av = Xv. T he last equation can be organized as (XI—A)v = 0. In order to have a nontrivial solution v 7^ 9, t he corresponding null space (kernel) M(XI — A) should contain vectors other than 9. T hus, the kernel has dimension larger than 0, which means we get at least one zero row in Gaussian elimination. Therefore, (XI — A) is singular. Hence, A should be chosen such that det(AJ — A) = 0. T his equation is known as characteristic equation for A. d(s) = det(sl - A) = s11 + dxsn+1 + • • • + dn = 0. T hen, the eigen values are the roots. k d(s) = (s- X,r (s - A 2 ) " . . . ( « - Xk) - = Y[(s - 2 n K)ni- The sum of multiplicities should be equal to the dimension, i.e. Y^i ni — nThe sum of n-eigen values equals the sum of n-diagonal entries of A. Ai + • • • + An = n iAi + • • • + nfcAfc = o u + • • • + ann. T his sum is known as trace of A. F urthermore, the product of the n-eigen values equals the determinant of A. i=l n *<=n A "' =detA I 00 II o u R emark 4.2.2 If A is triangular, the eigen values X\,. ., Xn are the diagonal entries an,... ,ann. E xample 4.2.3 A= 4 4 d et.4 = i ( l ) | = |(property 8). 8~\ d(s) = - i 0 8-1 0 0 s— = 1-2 (*-l) 3 S ~4 So, Ai = \ = a n , A2 = 1 = a2i, A3 = | = a 33 . Finally, 4.3 Diagonal Form of a Matrix 55 4 .3 Diagonal Form of a M a t r i x P roposition 4.3.1 Eigen vectors associated with distinct eigen values form a linearly independent set. Proof. Let Aj •H- vi, i = 1 , . . . , k. Consider YA=I a*Vi ~ ®- Multiply from the left by Ili=2(^ — - ^1). Since (A — Xil) = 9, we obtain (A — Xil)vj = (Xj — Xi)vj, which yields a i(Aj - A2)(Ai - A3) • • • (Ai - Xk)Vl = 6. = v\ 7^ 9, Ai - A2 7^ 0, ..., Ai - Ac ^ 0 => Qi = 0. Then, we have J27=2 aiv' f Repeat by multiplying J^ i=3 (yl — Xil) t o get a2 = 0, and so on. D 4 .3.1 All Distinct Eigen Values d{s) — n r = i (s~^i)- T he n eigen vectors vi,... s et. Choose them as a basis: {wi}™=1Avi = Aii>i + 0v2 + Av2 = 0^! + A2i>2 H ®- ,vn form a linearly independent 1- 0vn h 0u„ Avn = Owi + 0v2 + "Ai 0 ••• h Xnvn 0" 0 A2 • • • 0 T hus, A h as representation A = 0 0 • • • A„ Alternatively, let S — [f i|f2| • • • \vn] AS = [Avi\Av2\---\Avn] = "Ai AS = [vi\v2\ X2 Xn T hus, S t heorem. X [X1v1\X2v2\---\Xnvn} SA. AS = A (Change of basis). Hence, we have proven the following T heorem 4.3.2 Suppose the n by n matrix A has n linearly independent eigen vectors. If these vectors are columns of a matrix S, then A, S , _ 1 yl5 = yl = A2 An 56 4 Eigen Values and Vectors E xample 4.3.3 From the previous example, MO A= i 10 0I3 "44 = *Ai = 1, A 3 = ± X1 1*1" = Ax = X\x <& \x\ + x2 ;X2 + f x 3 _ 1*2 1*3. \x\ + | x2 = 0 < > x\ + x2 = 0 £ I x 2 + | x 3 = 0 & x2 + x 3 = 0 |*i :} Thus, v\ = Xi Ax = X2x < > = 5 X1 + X 2 fX2 |*3 x2 X3 H = 0.1 7 *2 - 7 X3 = 0 <4> X2 - X3 = 0. J Thus, v2 = " fail" 1*1 Ax = A 3 x < > = \x\ + x2 [X2 + f x 3 _ = fx2 O- §xi - \x2 100 - 1 10 1 11 .1*3. xx = 0 . = 0 =>• 2a:i + x 2 = 0. \ Thus, v3 = x 2 = 0. Therefore, S = [S\I] Then, 100 100' - 1 10 0 1 0 111 001 -4 "100 100' 0 1 0 1 10 0 11 -10 1 -» ' 1 0 0 1 00" 0 1 0 1 1 0 = [I\S - i i 001 - 2 - 1 1 S^AS •• 1 00' 1 10 2 -1 1 1 00" ."44. \ 10 012 100' - 1 10 111 l oo" = 01 0 = A. oof R emark 4.3.4 Any matrix with distinct eigen values can be diagonalized. However, the diagonalization matrix S is not unique; hence neither is the basis { «}"_!• If we multiply an eigen vector with a scalar, it will still remain an eigen vector. Not all matrices posses n linearly independent eigen vectors; therefore, some matrices are not dioganalizable. 4.3 Diagonal Form of a Matrix 4 .3.2 Repeated Eigen Values with Full Kernels 57 In t his case, (recall that d(s) = Yii=i(s ~ A ;) ni )i w e have dimAf([A — Xil]) — Tii,Vz. Thus, there exists n* linearly independent vectors in Af ([A — Xil]), each of which is an eigen vector associated with A;, Vi. A i •HA 2 f* Vn,Vl2,...,Vini V2l,V22,---,V2n2 Afc *-t v 1S Vkl,Vk2,...,Vknk U r=i { »i}?ii linearly independent (Exercise). Thus, we have obtained n linearly independent vectors, which constitute a basis. Consequently, we get Ai S~ AS 1 = E xample 4.3.5 A= 3 1 -1 13-1 00 2 1 1 =0 s-2 s-3 -1 d(s) = det(sl - A) = - 1 s - 3 0 0 = (* - 3) 2 (s - 2) - (a - 2) = (a - 2)[(* - 3) 2 - 1] = (* - 2)(s - 4)(a - 2) = (a - 2) 2 (s - 4). => Ai = 2, ni = 2 and A2 = 4, 712 = 1. A-\iI = 1 1 -1 1 1 -1 00 0 dim(Af([A - Aj/])) = 2. v ii = ( l ) - l , 0 ) r , « i 2 = ( 0 , l , l ) T . -1 A-\2I= 1 -1 1-1-1 0 0-2 4 Eigen Values and Vectors V2 = (l,l,0)r • I I I s= 10 r 1 1 1 , 5 -- 1 010 • _ 2 1 .2 2 1 2 2 0 0 1 2. 1 1 , S~ AS = "200" 020 004 4 .3.3 Block Diagonal Form In this case, we have 3i 9 iii > 1, dim(Af[A - A;/]) < n. D efinition 4.3.6 The least degree monic (the polynomial with leading coefficient one) polynomial m(s) that satisfies m(A)=0 is called the minimal polynomial of A. P roposition 4.3.7 The following are correct for the minimal polynomial. i. m(s) divides d(s); ii. m(Xi) = 0 , Vi = 1 ,2,..., k; Hi. m(s) is unique. E xample 4.3.8 A= c 10 O cO 00c , d{s) = d et(s7 - A) s-c 0 0 -1 s-c 0 0 3 0 = (s- c ) = 0. s-c Ax = c, ni = 3. m(s) = ? (s — c), (s — c ) 2 , (s — c ) 3 [A - Xil] = 010 000 000 ^03=> 010 000 000 m(s) ^(sc). [A - A i/] 2 = 0 10 000 000 = 0 3 => m(s) - (s - c)2. Then, to find the eigen vectors (A - d)x = <?<£> P roposition 4.3.9 d(s) = nU(* - ^ ) " % m(s) = n?=1(a - Xi)m\ 1 < rrn < m, i = 1,2,. Xil)ni] ,k. 010 000 000 x = 6 =>• Dn " 1" 0 0 "0" 0 1 , V\2 = *W - Ai/)] § MU - V)2] i • • • i MU - v n = j V p - A*/)™^1] = • • • = N{{A - 4.3 Diagonal Form of a Matrix P roposition 4.3.10 m(s) = ITf=1(s - Xi)mi, then Xk)mk], 59 C " = Af[(A - A i) m i ] e • • -®AT[(A where © is the direct sum of vector spaces. T heorem 4.3.11 d(s) = n^=1(s - A*)"*, m(s) = n*=1{s - A*)mi. i. dim(Af[{A - A*)mi]) = m; ii. If columns of n x ni matrices Bi form bases for Af[(A — Aj)m*] and B = [Bi\ • • • |.Bfc], then B is nonsingular and M B'XAB Ak where Ai are ni x n ,; Hi. Independent of the bases chosen for Af[(A — A ;) mi ], det(sl - A\) = (a - Xi)"*; iv. Minimal polynomial of Ai is (s — Xi)mi. E xample 4.3.12 A= "0 1 0 ' s-1 0 0 0 1 , d(s) = 0 s -1 = (s-l)2(s-2)=0. -2 5s-4 2-5 4 Ai = 1, m = 2; A2 = 2, n 2 = 1. [A- - AiJ] = " - 1 10" 0 - 1 1 , dim{Af[{A - A:)]) = 1 < 2 = m(!) 2-5 3 m i > 1 => mi = 2 => m(s) = (s - l) 2 (s - 2) = d(s). 2 [A- - A i / ] = "1 - 2 1" 2 - 4 2 , dim(M[(A - Aj)2]) = 2 4-8 4 10 01 - 12 « 11 = ( l , 0 , - l ) r ) «i2 = ( 0 , l , 2 ) T , B 1 = A2 = 2, [;4 - A27] = " - 2 1 0" 0 -2 1 2 -5 2 dim(N[{A - Aa)]) = 1. 60 4 Eigen Values and Vectors B2 = 10 01 -12 01 -1 2 . Therefore, v2 = (l,2,4)T, B= B- 0 2-1 -2 5 - 2 1 -2 1 =>r'AB 01 -12 where Ai and A2 = [2]. 4 .4 P o w e r s of A E xample 4.4.1 (Compound Interest) Let us take an example from engineering economy. Suppose you invest $ 500 for six years at 4 % in Citibank. Then, Pk+i = 1.04Pfc, P 6 = (1.04)6, P 0 = (1.04)6500 = $632.66. Suppose, the time bucket is reduced to a month: Pk+i= ( l + ^ W Pr2= (l 0.04 ~12~ 72 Po = (1.003)72500 = $635.37. What if we compound the interest daily? 0.04 Pk+l = ( 1 + ^ J Pk, i>6(364) + 1.5 =(1+ Thus, we have dP Pk+i - Pk = 0.04Pfc - » At dt 0 .04P ^> P(t) = eomtP0. 2185.5 3 6^ J P 0 = $635.72. In the above simplest case, what we have is a difference/differential equation with one scalar variable. What if we have a matrix representing a set of difference/differential equation systems? What is e~Ai1 E xample 4.4.2 (Fibonacci Sequence) Fk+2 = Fk+i + Fk, Fi = 0, F2 — 1. Uk = Fk+i Fk Uk+l — Fk+2 Fk+i 11 10 11 10 Fk+1 Fk Auk. uk = Aku0 Hence, we sometimes need powers of a matrix! 4.4 Powers of A 61 4 .4.1 Difference equations T heorem 4.4.3 If A can be diagonalized (A — SAS"1), uk = Aku0 = (SAS-^iSAS-1) Remark 4.4.4 • • • {SAS-^uo = then SAkS~1u0- xk Uk [vi,- S A_ 1 u0 = aiXiVi H hanX„vn. The general solution is a combination of special solutions XkVi and the coefficients a , that match the initial condition UQ are aiXlvi + • • • + a „A°u n = Uo or Sa — uo or a = 5 _ 1 MO- Thus, we have three different forms to the same equation. E xample 4.4.5 (Fibonacci Sequence, continued) 11 10 d(s) = s - 1 -1 = s* - s - 1 = 0. -1 s A2 = 1-\/5 2' 1 -A2 A2 Ai A2 11 1 A, = A= •ffc+i l + \ /5 SAS-1 Ai A2 11 - A x 1 Ai — A2 - 1 x1 Fk Fk = uk = AkuQ Xk A x k At 1 -1 1 M A2 2 1+ v^ '1-VET 1 000 Ai — A2 k X\ — A2 71 Since -4? (^—f^-J <\, Fww = the nearest integer to 4= ( ^ f l - ) Note that the ratio - J^ 1 = 1 - y 5 = 1.618 is known as the Golden Ratio, which represents the ratio of the lengths of the sides of the most elegant rectangle. E xample 4.4.6 (Markov Process) Assume that the number of people leaving Istanbul annually is 5 % of its population, and the number of people entering is 1 % of Turkey's population outside Istanbul. Then, #inside jfcoutside A 0.95 0.01 0.05 0.99 d(s) 0.95 0.01 0.05 0.99 s - 0 . 9 5 - 0.01 = (s-1.0)(s-0.94). - 0.05 s - 0 . 9 9 62 4 Eigen Values and Vectors Ai = 1.0, A2 = 0.94 =>• V! SAS'1 Vk Zk , V2 5 -1 5 = |1 -1 1 1.00 0.94 1.00* 5 _I 6 6 0.95 0.01 0.05 0.99 -ik |1 -1 1 0.94* II I -I /5 5 = ^2/o + ^ o Since 0.94* -» 0 as A -)• oo, ; 2/oo •2 CO 1 + I | »b " ^ o ) 0.94* - 1 = l^2/o + | « b L6 6 J TVie steady-state probabilities are computed as in the classical way, Auoo — 1 • Moo, corresponding to the eigen value of one. Thus, the steady-state vector is the eigen vector of A corresponding to A = 1, after normalization to have legitimate probabilities (see Remark 4-3-4): 5 "I ' [1 Moo = avi = - 5 6 L 1J "1" 6 5 6 4 .4.2 Differential Equations E xample 4.4.7 du ~dl Au • 23 u < > u(t) = eAtu0= 14 A2 = l , «2 = ( - 3 , l ) a 1 t 1 + a2e 1 -3 11 J>t OLi Xx = 5,vi = (l,lf, „ u(t) = a 1 e A l S 1 + a2e*2tv2 = axe 5t -3 1 U0 = «1 + a2 a5t -3 1 Oil u(t) 1 -3 11 =S 5-^0- The power series expansion of the exponentiation of one scalar is 4.5 The Complex case and if we generalize to the matrices 63 eM = I + At + {At)2 2! (At)3 3! +• If we take the derivative of both sides, we have deM dt = I +A+ , T A A2{2t) A3{it2) „. + + — )^— L + 2! 3! 2 = A I + At+ ±-J- + ^J2! 3! UA = SAS-1, ' (At) (At) 3 + ••• = AeAt eAt = / + SAS At r OAO-I x + SA2S-H2 -. 2! + SA3S~H3 -. + 3! S"1 = SeAtS~1. ' , {At)2 {At)3 i+At+^-^- 2! + ^ -T '—+• 3! T hus, we have the following theorem. T heorem 4.4.8 / / A can be diagonalized as( A = SAS-1), then ^ — Au has the solution u(i) = eAtuo = SeAtS~1uo, or equivalently u(i) = a ie A l 'vi + • • • + aneXntvn, where a — 5 _ 1 «o- 4.5 T h e C omplex case In this section, we will investigate Hermitian and unitary matrices. The complex field C is denned over complex numbers (of the form x+iy where x,y E.R and i2 — —1) with the following operations: {a + ib) + {c + id) = {{a + c) + i(b+d)) (a + ib)(c + id) = {{ac-bd) + i(cb+ad)). D efinition 4.5.1 The complex conjugate ofa + ibsC Figure 4-2. Properties: i. (a + ib){c + id) — {a + ib)(c + id), ii. {a + ib) + (c + id) — {a + ib) + (c + id), Hi. (a + ib)a + ib = a2 + b2 — r2 where r is called modulus of a + ib. We have a = y/a2 + b2 cos 9 and b = \/a2 + b2 sin 9 and a + ib = \]a? + b2{cos9 + ism 9) = re10 (Polar Coordinates), where re1,6 — cos 9 + i sin 9. is a + ib = a — ib. See 64 4 Eigen Values and Vectors Im a +ib= r (coxO + i sinO) * - Re a a +ib=a-ib F ig. 4.2. Complex conjugate D efinition 4.5.2 A = AH with entries (AH)ij = (A)^ is known as conjugate transpose (Hermitian transpose). Properties: i. < x, y >— xHy, x A. y < > xHy = 0, = ii. \\x\\ = (xHx)z, in. {AB)H = BHAH. D efinition 4.5.3 A is Hermitian if AH = A. Properties: i. A" = A,~ix<E C ", xHAx € K". ii. Every eigen value of a Hermitian matrix is real. Hi. The eigen vectors of a Hermitian matrix, if they correspond to different eigen values, are orthogonal to each other, iv. (Spectral Theorem) A = AH, there exists a diagonalizing unitary (complex matrix of orthonormal vectors as columns) U such that U'1AU = UHAU = A. Therefore, any Hermitian matrix can be decomposed into A = USUH = Xivxv? + ••• + \„vnv%. D efinition 4.5.4 If B = M~1AM (change of variables), then A and B have the same eigen values with the same multiplicities, termed as A is similar to B. 4.5 Problems Properties: 65 i. A 6 C m X n , 3 unitary M — U 9 U~lAU = T is upper-triangular. The eigen values of A must be shared by the similar matrix T and appear along the main diagonal. ii. Any Hermitian matrix A can be diagonalized by a suitable U. D efinition 4.5.5 The matrix N is called normal if NNH normal matrices, T = U"1NU = A where A is diagonal. = NHN. Only for P roblems 4 .1. D eterminant Prove property 11 in Section 4.1.2. 4 .2. J ordan form 1 1-1-1 -1 2 112 1 0 110-1 1-113 1 2-2224 "2 1 2 F ind S such that S^AS = 21 2 Let A = 2_ Hint: Choose V2 € M[{A - XI)2}, v\ = [A - A/]w2. Similarly, choose v± and v3Finally, Finally, choose v* e M[{A -- XI)}. XI)}. 5 6 N "UA 4 .3. U sing Jordan Decomposition -1 I 10 10 0 ' F ind A10. Let A = 0 1l0 1l0 00 I 10 . 4 .4. D ifferential Equation S y s t e m Let the Blue (allied) forces be in a combat situation with the Red (enemy) forces. There are two Blue units (Xi, X 2 ) and two Red military units (Yi, Y^)At the start of the combat, the first Blue unit has 100 (X° = 100) combatants, t he second Blue unit has 60 (X° = 60) combatants. The initial conditions for the Red force are Y° = 40 and Y2° = 30. Since the start of the battle (t = 0 ), the number of surviving combatants (less than the initial values due t o attrition) decrease monotonically and the values are denoted by X\, X\, Y{, and Y\. T he first Blue unit is subjected to directed fire from all the Red forces, w ith an attrition rate coefficient of 0.03 Blue 1 targets/Red 1 firer per unit t ime and 0.02 Blue 1 targets/Red 2 firer per unit time. The second Blue unit is also subjected to directed fire from all the Red forces, with an attrition rate 66 4 Eigen Values and Vectors coefficient of 0.04 Blue 2 targets/Red 1 firer per unit time and 0.01 Blue 2 t argets/Red 2 firer per unit time. The first Red unit is under directed fire from b oth Blue units, with an attrition rate coefficient of 0.05 Red 1 targets/Blue 1 firer per unit time and 0.02 Red 1 targets/Blue 2 firer per unit time. The second Red unit is subjected to directed fire from only Blue 1, with an attrition rate coefficient of 0.03 Red 2 targets/Blue 1 firer per unit time. (a) Write down the differential equation system to represent the combat dynamics. (b) Find the closed form values as a function of time t for X{, X\, Y±, Y$. (c) Calculate X{, X\, Y{, Yj,t = 0 ,1,2,3,4,5. W eb material http://149.170.199.144/multivar/eigen.htm http://algebra.math.ust.hk/determinant/03_properties/lecturel.shtml http://algebra.math.ust.hk/eigen/01_definition/lecture2.shtml http://bass.gmu.edu/ececourses/ece521/lecturenote/chapl/node3.html http://c2.com/cgi/wiki?EigenValue http://ceee.rice.edu/Books/LA/eigen/ http://cepa.newschool.edu/het/essays/math/eigen.htm http://cio.nist.gov/esd/emaildir/lists/opsftalk/msg00017.html http://cnx.org/content/m2116/latest/ http://cnx.rice.edu/content/ml0742/latest/ http://college.hmco.com/mathematics/larson/elementary_linear/4e/ shared/downloads/c08s5.pdf http://college.hmco.com/mathematics/larson/elementary_linear/5e/ students/ch08-10/chap_8_5.pdf http://ece.gmu.edu/ececourses/ece521/lecturenote/chapl/node3.html http://en.wikipedia.org/wiki/Determinant http://en.wikipedia.org/wiki/Eigenvalue http://en.wikipedia.org/wiki/Hermitian_matrix http: //en. wikipedia. org/wiki/Jordan_normal_f orm http://en.wikipedia.org/wiki/Skew-Hermitian_matrix http://encyclopedia.laborlawtalk.com/Unitary_matrix http://eom.springer.de/C/c023840.htm http://eom.springer.de/E/e035150.htm http://eom.springer.de/H/h047070.htm http://eom.springer.de/J/j 054340.htm http://eom.springer.de/L/1059520.htm http://everything2.com/index.pl?node=determinant http://fourier.eng.hmc.edu/el61/lectures/algebra/node3.html http://fourier.eng.hmc.edu/el61/lectures/algebra/node4.html http://gershwin.ens.fr/vdaniel/Doc-Locale/Cours-Mirrored/ Methodes-Maths/white/math/s3/s3spm/s3spm.html http://home.iitk.ac.in/~arlal/book/nptel/mthl02/node57.html http://homepage.univie.ac.at/Franz.Vesely/cp0102/dx/node28.html http://hyperphysics.phy-astr.gsu.edu/hbase/deter.html 4.6 W e b material 67 http://kr.cs.ait.ac.th/~radok/math/mat/51.htm http://kr.cs.ait.ac.th/~radok/math/mat3/ml32.htm http://kr.cs.ait.ac.th/"radok/math/mat3/ml33.htm http://kr.cs.ait.ac.th/"radok/math/mat3/ml46.htm http://kr.cs.ait.ac.th/~radok/math/mat7/stepl7.htm http://linneus20.ethz.ch:8080/2_2_1.html http://math.carleton.ca:16080/"daniel/teaching/114W01/117_EigVal.ps http://math.fullerton.edu/mathews/n2003/JordanFormBib.html http://mathworId.wolfram.com/Determinant.html http://mathworId.wolfram.com/DeterminantExpansionbyMinors.html http://mathworld.wolfram.com/Eigenvalue.html http://mathworld.wolfram.com/Eigenvector.html http://mathworld.wolfram.com/Hermit ianMatrix.html http://mathworld.wolfram.com/JordanCanonicalForm.html http://mathworld.wolfram.com/UnitaryMatrix.html http: //meru. m e t . mis sour i. edu/people/hai/research/j acobi. c http://mpec.sc.mahidol.ac.th/radok/numer/STEP17.HTM http://mysoftwear.com/go/0110/10406671133e894dl72cd42.html http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-241Fall2003/A685C9EE-6FF0-4ElA-81AC-04A8981C4FD9/0/ rec5.pdf http://oonumerics.org/MailArchives/oon-list/2000/06/0486.php http://oonumerics.org/MailArchives/oon-list/2000/06/0499.php http://orion.math.iastate.edu/hentzel/class.510/May.23 http://ourworld.compuserve.com/homepages/fcfung/mlaseven.htm http://planetmath.org/encyclopedia/Determinant2.html http://planetmath.org/encyclopedia/ DeterminantlonTermsOfTracesOfPowers.html http://planetmath.org/encyclopedia/Eigenvalue.html http://planetmath.org/encyclopedia/JordanCanonicalForm.html http://planetmath.org/encyclopedia/ ProofOfJordanCanonicalFormTheorem.html http://psroc.phys.ntu.edu.tw/cjp/v41/221.pdf http://rakaposhi.eas.asu.edu/cse494/f02-hwl-qnl.txt http://rkb.home.cern.ch/rkb/AN16pp/node68.html http://schwehr.org/software/density/html/Eigs_8C.html http://sherry.if i.unizh.ch/mehrmann99structured.html http://sumantsumant.blogspot.com/2004/12/one-of-beauty-of-matrixoperation-is. html http://www-gap.dcs.st-and.ac.uk/"history/Search/historysearch.cgi? SUGGESTION=Determinant&CONTEXT=l http://www-history.mcs.st-andrews.ac.uk/history/Biographies/ Jordan.html http://www-history.mcs.st-andrews.ac.uk/history/HistTopics/ Matrices_and_determinants.html http://www-math.mit.edU/18.013A/HTML/chapter04/section01.html# DeterminantVectorProducts http://www.bath.ac.uk/mech-eng/units/xxl0118/eigen.pdf http://www.caam.rice.edu/software/ARPACK/UG/node46.html 68 4 Eigen Values and Vectors http://www.cap-lore.com/HathPhys/Implicit/eigen.html http://www.cs.berkeley.edu/~wkahan/MathHl10/j ordan.pdf http://www.cs.ucf.edu/courses/cap6411/cot6505/Lecture-2.PDF http://www.cs.ucf.edu/courses/cap6411/cot6505/spring03/Lecture-2.pdf http://www.cs.uleth.ca/~holzmann/notes/eigen.pdf http://www.cs.ut.ee/"toomas_l/linalg/linl/nodel4.html http://www.cs.ut.ee/~toomas_l/linalg/linl/nodel6.html http://www.cs.ut.ee/"toomas_l/linalg/lin2/nodel8.html http://www.cs.ut.ee/~toomas_l/linalg/lin2/node20.html http://www.cs.utk.edu/~dongarra/etemplates/ http://www.dpmms.cam.ac.uk/site2002/Teaching/IB/LinearAlgebra/ jordan.pdf http://www.ece.tamu.edu/"chmbrlnd/Courses/ELEN601/ELEN601-Chap7.pdf http://www.ece.uah.edu/courses/ee448/appen4_2.pdf http://www.ee.bilkent.edu.tr/~sezer/EEE501/Chapter8.pdf http://www.ee.ic.ac.uk/hp/staff/www/matrix/decomp.html http://www.emunix.emich.edu/"phoward/f03/416f3fh.pdf http://www.freetrialsoft.com/free-download-1378.html http://www.gold-software.com/MatrixTCL-reviewl378.htm http://www.itl.nist.gov/div898/handbook/pmc/section5/pmc532.htm http://www.mat.univie.ac.at/"kratt/artikel/detsurv.html http://www.math.colostate.edu/~achter/369/help/jordan.pdf http://www.math.ku.dk/ma/kurser/symbolskdynamik/konjug/node14.html http://www.math.lsu.edu/~verrill/teaching/linearalgebra/linalg/ linalg8.html http://www.math.missouri.edu/courses/math4140/331eigenvalues.pdf http://www.math.missouri.edu/~hema/331eigenvalues.pdf http://www.math.poly.edu/courses/ma2012/Notes/Eigenvalues.pdf http://www.math.sdu.edu.cn/mathency/math/u/u062.htm http://www.math.tamu.edu/~dallen/m640_03c/lectures/chapter8.pdf http://www.math.uah.edu/mathclub/talks/ll-9-2001.html http://www.math.ucdavis.edu/~daddel/linear_algebra_appl/ Applications/Determinant/Determinant/Determinant.html http://www.math.ucdavis.edu/~daddel/linear_algebra_appl/ Applications/Determinant/Determinant/node3.html http://www.math.ucdavis.edu/~daddel/Math22al_S02/LABS/LAB9/lab9_w00/ nodel5.html http://www.math.umd.edu/~hck/Normal.pdf http://www.mathreference.com/la-det,eigen.html http://www.mathreference.com/la-j f,canon.html http://www.maths.gla.ac.uk/~tl/minimal.pdf http://www.maths.lanes.ac.uk/~gilbert/m306c/nodel6.html http://www.maths.liv.ac.uk/~vadim/M298/108.pdf http://www.maths.lse.ac.uk/Personal/james/old_ma201/lectll.pdf http://www.maths.mq.edu.au/~wchen/lnlafolder/lal2.pdf http://www.maths.surrey.ac.uk/interactivemaths/emmaspages/ option3.html http://www.mathwords.com/d/determinant.htm http://www.mines.edu/~rtankele/cs348/LAy.207.doc 4.6 W e b material http://www.nova.edu/"zhang/OlCommAlgJordanForm.pdf http://www.numbertheory.org/courses/MP274/realjord.pdf http://www.numbertheory.org/courses/MP274/uniq.pdf http://www.oonumerics.org/MailArchives/oon-list/2000/05/0481.php http://www.oonumerics.org/oon/oon-list/archive/0502.html http://www.perfectdownloads.com/audio-mp3/other/ download-matrix-tcl.htm http://www.ping.be/~pingl339/determ.htm http://www.ppsw.rug.nl/"gladwin/eigsvd.html http://www.reference.com/browse/wiki/Hermitian_matrix http://www.reference.com/browse/wiki/Unitary.matrix http://www.riskglossary.com/link/eigenvalue.htm http://www.sosmath.com/matrix/determO/determO.html http://www.sosmath.com/matrix/determ2/determ2.html http://www.sosmath.com/matrix/inverse/inverse.html http://www.Stanford.edu/class/ee263/j cf.pdf http://www.Stanford.edu/class/ee263/j cf2.pdf http://www.techsoftpl.com/matrix/doc/eigen.htm http://www.tversoft.com/computer/eigen.html http://www.wikipedia.org/wiki/Determinant http://www.wikipedia.org/wiki/Unitary_matrix http://www.yotor.com/wiki/en/de/Determinant.htm http://www.zdv.uni-tuebingen.de/static/hard/zrsinfo/x86_64/nag/ mark20/NAGdoc/f1/html/indexes/kwic/determinant.html http://wwwl.mengr.tamu.edu/aparlos/MEEN651/ EigenvaluesEigenvectors.pdf http://www2.maths.unsw.edu.au/ForStudents/courses/math2509/ch9.pdf 69 5 P ositive Definiteness Positive definite matrices are of both theoretical and computational importance in a wide variety of applications. They are used, for example, in optimization algorithms and in the construction of various linear regression models. As an initiation of our discussion in this chapter, we investigate first the p roperties for maxima, minima and saddle points when we have scalar functions with two variables. After introducing the quadratic forms, various tests for positive (semi) definiteness are presented. 5 .1 Minima, Maxima, Saddle points 5 .1.1 Scalar Functions Let us remember the properties for maxima, minima and saddle points when we have scalar functions with two variables with the help the following examples. F ig. 5.1. Plot of f(x, y) = x2 + y2 72 5 Positive Definiteness = x2 + y2. Find the extreme points of ay f(x,y): E xample 5.1.1 Let f(x,y) ox Since we have only one critical point, it is either the maximum or the minimum. We observe that f(x,y) takes only nonnegative values. Thus, we see that the origin is the minimum point. Fig. 5.2. Plot of f(x, y) = xy - x2 - y2 - 2x - 2y + 4 E xample 5.1.2 Find the extreme points off(x,y) = xy-x2—y2-2x The function is differentiate and has no boundary points. 1 — 2y+4. dx v dy y Thus, x = y = — 2 is the critical point. Jxx _82f(x,y)_ ~ dx2 n_d 2 ~ f(x,y)_ dy2 ~' Jvy Jxy _d2f(x,y) ~ Dxdy _1 ~ The discriminant (Jacobian) of f at (a,b) = ( —2,-2) is Jxx Jxy Jxy fyy — fxxfyy — fXy = 4 — 1 = 0. Since fxx < 0, fxxfyy — f2y > 0 =>• / has a local maximum at (—2, —2). 5.1 Minima, Maxima, Saddle points T heorem 5.1.3 The extreme values for f(x,y) can occur only at 73 i. Boundary points of the domain of f. ii. Critical points (interior points where fx = fy = 0, or points where fx or fy fails to exist). If the first and second order partial derivatives of f are continuous throughout an open region containing a point (a,b) and fx(a,b) = fy(a,b) — 0, you may be able to classify (a, b) with the second derivative test: local maximum; i- fxx < 0, fxxfyy - fL > 0 at (a,b) local minimum; ii- fxx > 0, fxxfyy ~ / J y > 0 at (a,b) 2 saddle point; in- fxxfyy - fc2<0 at (a,b) y test is inconclusive (f is singular). iv- fxxfyy- fL = 0 at (a,b) 5 .1.2 Quadratic forms D efinition 5.1.4 The quadratic term f(x,y) = ax2 + 2bxy + cy2 is positive definite (negative definite) if and only if a > 0 (a < 0) and ac—b2>0.f has a minimum (maximum) at x = y = 0 if and only if fxx(0,0) > 0 ( / X x(0,0) < 0) and fxx(0,0)fyy{0,0) > f2y{0,0). If f(0,0) = 0, we term f as positive (negative) semi-definite provided the above conditions hold. Now, we are able to introduce matrices to the quadratic forms: ax2 + 2bxy + cy2 — [x, y) ab be T hus, for any symmetric A, t he product / = x7 Ax is a pure quadratic form: it has a stationary point at the origin and no higher terms. an xATx - [xi,x2,--,xn] an\ = a n Z j +a12xix2 H 0-n2 • aX2 • • • 0-in Xi X2 021 "22 • «2n ann _ ^ j=l %n V annx2n = ^ i=l ai}-XjXj. D efinition 5.1.5 If A is such that a^- = dxJx. (hence symmetric), it is called the Hessian matrix. If A is positive definite (xTAx > 0, Vx ^ 6) and if f has a stationary point at the origin (all first derivatives at the origin are zero), then f has a minimum. 74 5 Positive Definiteness R emark 5 .1.6 Let f : M™ M- R andx* G R n be the local minimum, Vf(x*) = 0 and V2f(x*) is positive definite. We are able to explore the neighborhood of x* by means of x* + Ax, where \\Ax\\ is sufficiently small (such that the second order Taylor's approximation is pretty good) and positive. Then, f(x* + Ax) S f(x*) + AxTVf(x*) + ^AxTV2f(x*)Ax. The second term is zero since x* is a critical point and the third term is positive since the Hessian evaluated at x* is positive definite. Thus, the left hand side is always strictly greater than the right hand side, indicating the local minimality of x*. 5.2 Detecting Positive-Definiteness T heorem 5 .2.1 A real symmetric matrix A is positive definite if and only if one of the following holds: i. xTAx > 0, Vz ^ 6; ii. All the eigen values of A satisfy A; > 0 ; Hi. All the submatrices Ak have positive determinants; iv. All the pivots (without row exchanges) satisfy di > 0; v. 3 a nonsingular matrix W B A = WTW (called Cholesky Decomposition); Proof. A is positive definite. 1. (i) <& (ii) (i) => (ii): Let x, be the u nit eigen vector corresponding t o eigen value Xi. J\Xi == A J X J \^ XA J\-X{ = Xi A jXj — A j . T hen, Ai > 0 since A is positive definite. (i) <= (ii): Since symmetric matrices have a full set of o rthonormal eigen vectors (Exercise!). x — 2_,aixi ^* Ax = 2~\onAxi =>• xTAx = (2~]aixf)(2~]cti\iXi). Because of o rthonormality xTAx = ^ ctf Aj > 0. 2. (i) <£• (Hi) < > (iv) <& (v) = (i) => (Hi): d et A = \\ • X2 • • • A„, since (i) o (ii). Claim: If A is positive definite, so is every AkProof: If x Xk 0 , t hen 5.3 Semidefinite Matrices cTAx •• 75 [xk,0] Ak * ** Xk 0 = x\AkXk > 0. If we apply (i) < > (ii) for Ak (its eigen values are different, but all = are positive), then its determinant is the product of its eigen values yielding a positive result. (Hi) =$• (iv): Claim: If A = LDU, t hen the upper left corner satisfy Ak = LkDkUkUkF LkDkUk LkDkF Dk 0 Lk 0 Proof: A = 0E 0G BDkUk BDkF + CEG BC det Ak = det Lk det Dk det Uk — det Dk = d\ • di • • • dk => dk — fotA* ( Pivot=Ratio of determinants). If all determinants are positive, then all pivots are positive. (iv) => (v): In a Gaussian elimination of a symmetric matrix U = LT, t hen A = LDLT. One can take the square root of positive pivots dt > 0. Then, A = Ly/D~VDLT = WTW. (v) =* (i): xTAx = xTWTWx = \\Wx\\2 > 0. Wx = 6 => x = 9 since W is nonsingular. Therefore, xTAx > 0, Vcc ^ 6. D R emark 5.2.2 The above theorem would be exactly the same in the complex case, for Hermitian matrices A — AH. 5.3 Semidefinite Matrices T heorem 5.3.1 A real symmetric matrix A is positive semidefinite if and only if one of the following holds: i. xTAx > 0, Vx ^ 6; ii. All the eigen values of A satisfy A; > 0; Hi. All the submatrices Ak have nonnegative determinants; iv. All the pivots (without row exchanges) satisfy d, > 0; v. 3 a possibly singular matrix W 3 A = WTW; R emark 5.3.2 xTAx >0o\i>0is A = QAQT => xTAx = xTQAQTx important. = yTAy = Xxy\ + ••• + \nyl, and it is nonnegative when Ai 's are nonnegative. If A has rank r, there are r nonzero eigen values and r perfect squares. 76 5 Positive Definiteness R emark 5.3.3 (Indefinite matrices) Change of Variables: y = Cx. The quadratic form becomes yTCTACy. Then, we have congruence transformation: A H-> CT AC for some nonsingular C. The matrix CT AC has the same number of positive (negative) eigen values of A, and the same number of zero eigen values. If we let A = I, CT AC = CTC. Thus, for any symmetric matrix A, the signs of pivots agree with the signs of eigen values. A and D have the same number of positive (negative) entries, and zero entries. 5.4 Positive Definite Quadratic Forms P roposition 5.4.1 If A is symmetric positive definite, then P(x) = -xTAx assumes its minimum at the point Ax = b. Proof. Let x 9 Ax = b. T hen, Vy € R n , P(y) - P(x) = (\yTAy - yTb^ - {^xTAx -xTAx - xTb^ - xTb = -^VTAy - yTAx + = ^{V ~ xf A{y - x) >0. Hence, Vy ^ x, P(y) > P(x) => x is the minimum. D T heorem 5.4.2 (Rayleigh's principle) Without loss of generality, we may assume that Ai < A2 < • • • < An. The quotient, R(x) — x ^ , is minimized by the first eigen vector v\ and its minimum value is the smallest eigen value \\: vfAvi R(Vl) = —f v( Vi u fAi«i = -^T v[ v\ = A l- R emark 5.4.3 Va;, R(x) is an upper bound for X\. R emark 5.4.4 Rayleigh's principle is the basis for the principle component analysis, which has many engineering applications like factor analysis of the variance covariance matrix (symmetric) in multivariate data analysis. C orollary 5.4.5 If x is orthogonal to the eigen vectors vi,...,Vj-i, R(x) will be minimized by the next eigen vector Vj. then 5.5 Web material R emark 5 .4.6 Xj = m in xSR n R(x) s.t. xTv\ = 0 \j = m a x l 6 r R(x) s.t. xTvj+i =0 77 xTVj-i =0 xTvn =0 P roblems 5 .1. Prove t he following t heorem. T heorem 5 .4.7 ( Rayleigh-Ritz) Let A be symmetric, Ai < A2 < • • • < An. Ai = min x1Ax, IMI=i 5 .2. Use 1 A= " 100 t o show Theorem 5.3.1. 5 .3. Let f(x1,x2) = -x\ + -x\ + 2xxx2 + -x\ -x2 + 19. 2 10 121 0 11 An = max x1Ax. llx||=i F ind t he s tationary and b oundary p oints, t hen find t he minimizer a nd the maximizer over — 4 < x2 < 0 < x\ < 3 . W eb m aterial http://bmbiris.bmb.uga.edu/wampler/8200/using-ff/sld027.htm http://delta.cs.cinvestav.mx/~mcintosh/comun/contours/node8.html http://delta.cs.cinvestav.mx/-mcintosh/oldweb/lcau/node98.html http://dft.rutgers.edu/~etsiper/rrosc.html http://econ.lse.ac.uk/courses/ec319/M/lecture5.pdf http://employees.oneonta.edu/GoutziCJ/fall_2003/math276/maple/ Lesson_141.html http://en.wikipedia.org/wiki/Cholesky_decomposition http://en.wikipedia.org/wiki/Maxima_and_minima http://en.wikipedia.org/wiki/Positive-semidefinite_matrix http://en.wikipedia.org/wiki/Quadratic_form http://eom.springer.de/b/b016370.htm http://eom.springer.de/C/cl20160.htm 78 5 P ositive Definiteness http://eom.springer.de/N/nl30030.htm http://eom.springer.de/q/q076080.htm http://epubs.siam.org/sam-bin/dbq/article/38133 http://esperia.iesl.forth.gr/"amo/nr/bookfpdfIt 2-9.pdf http: //gaia. ecs. csus. edu/"hellerm/EEE242/chapter'/,201/pd. htm http://homepage.tinet.ie/~phabfys/maxim.htm http://iridia.ulb.ac.be/~fvandenb/mythesis/node72.html http://kr.cs.ait.ac.th/~radok/math/mat3/ml31.htm http://kr.cs.ait.ac.th/~radok/math/mat5/algebra62.htm http://kr.cs.ait.ac.th/~radok/math/mat9/03c.htm http://mat.gsia.cmu.edu/QUANT/NDTES/chapl/node8.html http://mathworld.wolfram.com/CholeskyDecomposition.html http://mathworld.wolfram.com/Maximum.html http://mathworld.wolfram.com/Pos it iveDef initeMatrix.html http://mathworld.wolfram.com/Posit iveSemidef initeMatrix.html http://mathworld.wolfram.com/Quadrat icForm.html http://mathworld.wolfram.com/topics/MaximaandMinima.html http://modular.fas.harvard.edu/docs/magma/htmlhelp/text654.htm http://ocw.mit.edu/NR/rdonlyres/Chemical-Engineering/10-34Fall-2005/ 695E79DF-llF7-4FB7-AD7E-FEDA74B9BFEF/0/lecturenotesl42.pdf http://omega.albany.edu:8008/calc3/extrema-dir/def ine-m2h.html http://oregonstate.edu/instruct/mth254h/garity/Fall2005/Notes/ 10_15_8.pdf http://people.hofstra.edu/faculty/Stefan_Waner/realworld/ Calcsumm8.html http://planetmath.org/encyclopedia/CholeskyDecomposition.html http://planetmath.org/encyclopedia/ DiagonalizationOfQuadraticForm.html http://planetmath.org/encyclopedia/PositiveDefinite.html http://planetmath.org/encyclopedia/QuadraticForm.html http://pruffle.mit.edu/3.016/collected_lectures/node39.html http://pruffie.mit.edu/3.016/Lecture_10_web/node2.html http://random.mat.sbg.ac.at/~ste/diss/node25.html http://rkb.home.cern.ch/rkb/AN16pp/node33.html http://scienceandreason.blogspot.com/2006/03/quadratic-forms.html http://sepwww.stanford.edu/sep/prof/gem/hlx/paper_html/nodell.html http://slpl.cse.nsysu.edu.tw/chiaping/la/chap6.pdf http://taylorandfrancis.metapress.com/media/59dam5dwuj 2xwl8rvvtk/ contributions/d/3/y/y/d3yy93fbcqpvu69n.pdf http: //tutorial .math. lamar. edu/AHBrowsers/1314/ ReducibleToQuadratic.asp http://web.mit.edu/18.06/www/Video/video-fall-99.html http://web.mit.edu/wwmath/vectorc/minmax/hessian.html http://www-math.mit.edu/~djk/18_022/chapter04/section02.html http://www.analyzemath.com/Equations/Quadratic_Form_Tutorial.html http://www.answers.com/topic/quadratic-form http://www.artsci.wustl.edu/~e503jn/files/math/DefiniteMatrics.pdf http://www.astro.cf.ac.uk/undergrad/module/PX3104/tpl/nodel2.html http://www.chass.utoronto.ca/"osborne/MathTutorial/QF2F.HTM 5.5 W e b material http://www.chass.utoronto.ca/~osborne/MathTutorial/QFF.HTM http://www.chass.utoronto.ca/"osborne/MathTutorial/QFS.HTM http://www.chass.utoronto.ca/~osborne/MathTutorial/QUF.HTM http://www.cs.ut.ee/~toomas_l/linalg/lin2/node25.html http://www.csie.ncu.edu.tw/~chia/Course/LinearAlgebra/sec8-2.pdf http://www.ece.mcmaster.ca/~kiruba/3sk3/lecture7.pdf http://www.ece.uwaterloo.ca/~ecel04/TheBook/04LinearAlgebra/ cholesky/ http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/property.html http://www.es.ucl.ac.uk/undergrad/geomaths/pdilink6.htm http://www.iam.ubc.ca/~norris/research/quadapp.pdf http://www.ics.mq.edu.au/"chris/mathl23/chap05.pdf http://www.imada.sdu.dk/"swann/MM02/Quadrat icForms.pdf http://www.imsc.res.in/~kapil/crypto/notes/node37.html http://www.matf.bg.ac.yu/r3nm/NumericalMethods/LAESolve/ Cholesky.html http://www.math.niu.edu/~rus in/known-math/99/posdef http://www.math.oregonstate.edu/home/programs/undergrad/ CalculusQuestStudyGuides/vcalc/min_max/min_max.html http://www.math.rutgers.edu/courses/251/s01bumby/slide011.ps2.pdf http://www.math.tamu.edu/~bollingr/Notes/bzb83.pdf http://www.math.ucla.edU/~xinweiyu/164.l.05f/1102.pdf http://www.math.uga.edu/~ chadm/quadrat ic.pdf http://www.math.uic.edu/math210/labs/lab5.html http://www.math.uic.edu/~math210/newlabs/critpts/critpts.html http://www.math.uiuc.edu/documenta/lsu/vol-lsu-eng.html http://www.math.umn.edu/~nykamp/mult ivar/Fall2003/lecture26.pdf http://www.math.vt.edu/people/javancel/Section.10.1.pdf http://www.math.wm.edu/~hugo/compl3.html http://www.mathematics.jhu.edu/matlab/8-4.html http://www.mathreference.com/ca-mv,local.html http://www.mathreference.com/la-qf,intro.html http://www.maths.abdn.ac.uk/~igc/tch/ma2001/notes/node70.html http://www.maths.abdn.ac.uk/~ igc/tch/mx3503/notes/node79.html http://www.maths.lse.ac.uk/Courses/MA207/fqmso.pdf http://www.mpri.lsu.edu/textbook/Chapter2.htm http://www.numericalmathematics.com/maxima.and_minimal.htm http://www.psi.toronto.edu/matrix/special.html http://www.quantlet.com/mdstat/scripts/mva/htmlbook/ mvahtmlnode16.html http://www.reference.com/browse/wiki/Maxima_and_minima http://www.reference.com/browse/wiki/Quadratic_form http://www.riskglossary.com/link/positive_definite.matrix.htm http://www.sciencenews.org/articles/2006031l/bob9.asp http://www.Stanford.edu/class/ee263/symm.pdf http://www.ucl.ac.uk/Mathematics/geomath/level2/pdiff/pd7.html http://www.ucl.ac.uk/Mathematics/geomath/level2/pdiff/pd8.html http://www.vision.caltech.edu/mweber/research/CNS248/node22.html 79 6 C omputational Aspects For square matrices, we can measure the sensitivity of the solution of the linear algebraic system Ax = b with respect to changes in vector 6 and in matrix A by using the notion of the condition number of matrix A. If the condition number is large, then the matrix is said to be ill-conditioned. Practically, such a m atrix is almost singular, and the computation of its inverse or solution of a linear system of equations is prone to large numerical errors. In this chapter, we will investigate computational methods for solving Ax = b, and obtaining eigen values/vectors of A. 6 .1 Solution of Ax = b Let us investigate small changes in the right hand side of Ax = b as if we are making a sensitivity analysis: 6 n> b + Ab => x i-»- x + Ax A(x + Ax) = b + Ab& A(AX) = Ab. Similarly, one can investigate the effect of perturbing the coefficient matrix A: A\-^A + AA^-x^x + Ax We will consider these cases with respect to the form of the coefficient matrix A in the following subsections. 6 .1.1 Symmetric and positive definite Let A b e symmetric. Without loss of generality, we may assume that we ordered the nonnegative eigen values: 0 < Ai < A2 < • • • < An. Since Ab is a vector itself, it could be represented in terms of the basis formed by the associated eigen vectors v\,V2,...,vn. Moreover, we can express Ab as a convex combination because its norm is sufficiently small. 82 6 Computational Aspects n A/, — 2_.aivi i=l where Vi <-> A,, 2 _ a j ~ ^' a> — 0;Vi. If A), is along i>i, i.e. At, — ev\, t hen Ax = ^- since Zix = A~lAb- T hat is, t he error of size || A>|| is amplified by the factor j - , which is just the largest eigen value of A-1. On the other hand, if b — v„, t hen x = A~xb = •£-, which makes the relative error \AX\\ A„ II A, Xi INI as much as possible. ML I An P roposition 6.1.1 For a positive definite matrix, the solution x — A lb and the error Ax — A~xAb always satisfy ||x|| > M and \\AX\\ < J !M. Therefore, the relative error is bounded by \AX < A„ \\Ah D efinition 6.1.2 The quantity c — k°- = v™*"1 is known as condition number 2 a Ai A m in of A. R emark 6.1.3 Notice that c is not affected by the size of a matrix. If A — I or A' — -k then cA = 1 = cA> = 4 ™^. However, detA = 1, det A' = 1 0~". Thus, determinant is a terrible measure of ill conditioning. E xample 6.1.4 2.00002 2 2 2.00002 => Ai = 2 x 10~°, A2 = 4.00002 => c « 2 x 10 s . In particular, 0 = Oi = 2.00001 2 Q nnm 2.00001 =$• X = X\ 0.5 0.5 and b2 = 2.00002 2 X2 Then, we have \\b\\ = 2.00001 v^, Ab = b2-b1 = 10_s 1 -1 \Ab\\ = \ / 2 x 10- 5 ; 6.1 Solution of Ax = b \\x\\ = - y> Ax = x2 -xi = 1 ana =- 83 V2 1 =* IIArll = -1 5 x 1 0~ 6 . Ml approximately ^-, 105. T/ie relative amplification in this particular instance, ",, 1}' ~ l-o i M J gwhich is a lower bound for the condition number c « 2 x R emark 6.1.5 As a rule of thumb (experimentally verified), a computer can loose l ogc decimal places to the round-off errors in Gaussian elimination. 6 .1.2 Symmetric and not positive definite Let us now drop the positivity assumption while we keep A still symmetric. T hen, nothing is changed except \"max | 6 .1.3 Asymmetric In t his case, the ratio of eigen values cannot represent the relative amplification. E xample 6.1.6 Let the parameter K ^> 0 be large enough. A= In particular, b = b\ = Then, we have \\b\\ = V l + K2, Ab = h - h = 1, Ax = X2 — X\ l |AJ V1 + K2 K 1 1K 01 <*A~X = 1 -K 0 1 Ai = A2 = 1. =£• x = x\ and b? = =>• x2 = \M = l; » \\AX\\ = Vl + K2 -1 and Ah\\ 1 \m\ \\o\\ %/i + K2 " The relative amplification in this particular instance is 1 + K2 . Hence, we should have 1 C 1 + K 2 < c(A). The condition number c(A) is not just the ratio of eigen values, which is 1; but it should have a considerably larger value in this example, since A is not symmetric. 84 6 Computational Aspects D efinition 6.1.7 The norm of A is the number defined \\A\\ = xnaxx^o |d!f R emark 6.1.8 ||.A|| bounds the "amplifying power" of the matrix. \\Ax\\ < \\A\\ \\x\\, Vz; and equality holds for at least one nonzero x. It measures the largest amount by which any vector (eigen vector or not) is amplified by matrix multiplication. P roposition 6.1.9 For a square nonsingular matrix, the solution x = A~1b and the error Ax = A~1Ab satisfy J !M< M ||. u -i||JJM. Proof. Since b=Ax=> \\b\\ < \\A\\\\x\\ and Ax = A-xAb =$> \\AX = || < | | ^ _ 1 | | ||A>||, we have H&HPHIWI and I I ^ H l ^ - 1 ! ! ! ! ^ R emark 6.1.10 When A is symmetric, D W = |A„I, i i ^ - 1 ^ J -|=> c : and the relative error satisfies \AX\\ K lAllIU-1!^ |Ai| \\Ab[ E xample 6.1.11 Let us continue the previous example, where A= Since we have « < | | ^ | | < « + 1 , and K < \\A~*\\ < K+ 1, then the relative amplification is approximately K2 « ||.A|| ||A _ 1 | R emark 6.1.12 ll2 \\A\\' = max "1 K 01 ,b = K 1 , A» = 0" -1 \\Ax\\2 xTATAx = max • M : Rayleigh quotient! 6.1 Solution of Ax = b 85 P roposition 6.1.13 The norm of A is the square root of the largest eigen value of ATA. The vector that is amplified the most is the corresponding eigen vector of ATA. Jtj 4L /lily JU ''TTIH.X**^ An U\\. 1 0 -K E xample 6.1.14 Let us further continue the previous example: 1K 01 ATA = 1 and A K l = 1 (K2 K S—1 —K K K? + 1 A2 S — K2 — 1 = 0 => s2 K2(K2+4) + 2)s + 1 = 0 = (K2 + 2)2-4(1)1 = ^ max — -(-K2-2)+y/K2(K2+4) 2(1) : K2 =$• \\A\\ = \ /A m a x « K. ] « «. 77ms, i/ie relative amplification Similarly, \\A 1 || = \/A max [(j4 1)TA is controlled by \\A\\ | |-A -1 || « « 2 - Remark 6.1.15 / / A is symmetric, then ATA = A2 and \\A\\ = m ax|A;|. Let us consider now the changes in the coefficient matrix. P roposition 6.1.16 If we perturb A, then ' IAJI \\x + Ax Proof. Ax = b (A + AA){x + Ax) AAX + AA(x + Ax) = 0&Ax |Ac||<IU-1||||ZlA||||:C + A c | | ^ E xample 6.1.17 1 10 100 A= Wffi 1 1 -i- -ix . \\AA\< where c •• \A\\ <c1 A- = -A-l{AA){x < A- H 4 =» + Ax). \*A\ I A, \x + Ax U\\ D 111 , b= in 10 in . 100 . X= Xi 10 100 . '\-JATA\- AI- 31329 = 100.5099, ||x|| = VS, 13 86 6 Computational Aspects 1_ 9 99 10 1 0989 100 1 0989 9 99 1 010 9 99 100 9 99 u A~ 1000 99 100 99 \A~X - J\max[(A-i)TA] = 28831 = 10.2021. 277 0 0 100 in => x2 = 100 1 11 10 1 11 100 - 1 - 10 AA = 1U -10 -§10 0--i- AA = i100 li 100 1 01 10 11 100 01 10 0 0 " 10 =>• Ax — x2 — x\ => 114*11 = V1020342 = 10.1012, and 100 \AA\\ = \J \ma.x\ATAAA} Jz + A J 1 0.1236 1 00.5099 14963 = 10.1236. 146 10.1012 10.1236 \\AA\ < 100.5099" = " -\\A\\ cc ^3 " sft = 57.9 < c = P | | | U _ 1 | | = 100.5099(10.2021) = 1025.412. The relative amplification in this instance is 51.9 whereas the theoretic upper bound is 1025412. R emark 6.1.18 The following are the main guidelines in practise: c and \\A\\ are never computed but estimated. c explains why AT Ax = ATb are so hard to solve in least squares problems: c(ATA) = [ C(J4)] 2 where c(.) is the condition number. The remedy is to use Gram-Schmidt or singular value decomposition, A = Qi£Q%. The entries a, in S are singular values of A, and a\ are the eigen values of ATA. Thus, \\A\\ = amax. Recall that \\Ax\\ = \\QiEQ^x\\ = \\Ex\\. 6.2 C o m p u t a t i o n of eigen values T here is no best way to compute eigen values of a matrix. But there are some terrible ways. In this section, a method recommended for large-sparse m atrices, the power method, will be introduced. Let wo be initial guess. Then, u^+i = Auk — Ak+1uo- Assume A h as full set of eigen vectors xi,x2, • • • ,xn, t hen Uk = a iAjXi H \-an\l^xn. Assume further that Ai < A2 < • • • < A n _i < An; that is, the last eigen value is not r epeated. 6.2 Computation of eigen values 87 Ai = ai IA:' Xl T he vectors uk point more and more accurately towards the direction of xn and the convergence factor is r = ' ,'^~}'. E xample 6.2.1 (Markov Process, continued) Recall Example 4-4-6: A= 'l' 0 "0.95 0.01" =>Ai = 1 <H- 5 = vi, A 2 = 0.94 0.05 0.99 1 • l • . "I = 0.95' 0.05 , "2 = 0.903 0.097 , "3 = "0.85882 0.14118 " l" 6 5 6 «4 '0 8 1'r29i" 0 1821709 1 , "210 - 0.166667] 0.83333 = av\. The convergence rate is quite low r = 0.94 = 2y^ = L21 Since the power method is designed especially for large sparse matrices, it converges after 210 iterations if the significance level is six digits after the decimal point. R emark 6.2.2 (How to increase r ) If r « 1, the convergence is slow. If | A n _i| = |Ara|, no convergence at all. There are some methods to increase the convergence rate: i. Block power method: Work with several vectors at once. Start with p orthonormal vectors, multiply by A, then apply Gram-Schmidt to orthogonalize again. Then, we have r' = ' ,^~!''. ii. Inverse power method: Operate with A^1 instead of A. vk+i — A~xvk => Avk+i = Vk (save L and U!). The convergence rate is r" = M l , provided that r" < 1. This method guarantees convergence to the smallest eigen vector. Hi. Shifted inverse power method: The best method. Let A be replaced by A — PI. All of the eigen values are shifted by (3. Consequently, r'" = K ' l a . / / we choose j3 as a good approximation to Xi, the convergence will be accelerated. (A - Pl)wk+i =wk = OL\X\ OL2X2 (Xi-P) k + ( A 2 - / 3 ) f c + ••• + (A„ - Qn^n R (3)\k- / / we know /3, then we may use A — [31 = LU and solve Ux\ ( 1,1, • • • , 1 ) T by back substitution. We can choose (3 = (3k at each step 3 (A - /3kI)wk+i = wk. If A = AT, /3k = R{uk) get the cubic convergence. M,t r "* , then we will 88 6 Computational Aspects R emark 6 .2.3 (QR A lgorithm) Start with Ao. Factor it using the GramSchmidt process into QQRQ, then reverse factors A\ = RQQO- A\ is similar to Ao: Q0 AoQo — QQ (QORO)QO — A\. So, Ak = QkRk =^ Ak+i = RkQk- Ak approaches to a triangular form in which we can read the eigen values from the main diagonal. There are some modifications to speed up this procedure as well. D efinition 6 .2.4 / / a matrix is less than a triangular form, one nonzero diagonal below the main diagonal, it is called in Hessenberg form. Furthermore, if it is symmetric then it is said to be in tridiagonal form. D efinition 6 .2.5 A Householder transformation (or an elementary reflector) is a matrix of the form T H= I-2- |2 " R emark 6 .2.6 Often v is normalized to become a unit vector u = jr^r, then H — I — 2uuT. In either case, H is symmetric and orthogonal: HTH =(I2uuT)T{I - 2uuT) = 1- AuuT + AuuTuuT = I. In the complex case, H is both Hermitian and unitary. H is sometimes called elementary reflector since P roposition 6 .2.7 Let z = e\ — ( 1,0, Then, Hx = -az = {-a, 0, • • • , 0 ) T . Proof. Hx VV , 0 ) r , and a — \\x\\, andv = x+crz. T X I HI' 2(x + az)Tx (x + az)v (x + az)T(x + az) Hx = x — (x + az) — —az. D R emark 6 .2.8 Assume that we are going to transform A into a tridiagonal or Hessenberg form U~1AU. Let a-21 031 " 1" 0 ,z= Hx = 0 an * * * * —a * * * * 0 **** 0 **** 0 **** _0.nl _ Ui 10 0 0 0 0 0H 0 0 [ /f\ andU~lAUi = 6.2 Problems 89 The second stage is similar: x consists of the last n — 2 entries in the second column, z is the first unit coordinate vector of matching length, and H2 is of order n — 2 : 100 0 010 0 00 0 0 H2 00 Uo = = U21, and U^iU^AUiM 0**** 0 0*** 00*** Following a similar approach, one may operate on the upper right corner of A simultaneously to generate a tridiagonal matrix at the end. This process is the main motivation of the QR algorithm. P roblems 6 .1. Show that for orthogonal matrices ||Q|| = c(Q) — 1. Orthogonal matrices and their multipliers (aQ) are only perfect condition matrices. 6 .2. Apply the QR algorithm for 0.5000 -1.1180 0 0 0 0 - 1.1180 91.2000 -80.0697 0 0 0 0 - 80.0697 81.0789 4.1906 0 0 0 0 4.1906 2.5913 0.2242 0 0 0 0 0.2242 0.1257 --0.0100 0 0 0 0 -0.0100 0.0041 A= 6 .3. Let A{n) 6 R n x n , A(n) = (a^, wnere aij — (a) Take A{2). 1. Let 6/ = i,j_^. 1.0 1.5 and bu = . Calculate the relative error. 0.5 1.0 2. F ind a good upper bound for the relative error obtained after perturbing t he right hand side. 3. F ind the relative error of perturbing A(2) by AA(2) . Take 1.0 as the right hand side. 0.5 4. F ind a good upper bound for the relative error obtained after perturbing .4(2). h (b) Take ^4(3)TA(3) and find its condition number and compare with the condition number of A(3). (c) Take A(4) a nd calculate its condition number after finding the eigen values using the QR a lgorithm. 90 6 C omputational Aspects W eb m aterial http://202.41.85.103/manuals/planetmath/entries/65/ MatrixConditionNumber/MatrixConditionNumber.html http://bass.gmu.edu/ececourses/ece499/notes/note4.html http://beige.ucs.indiana.edu/B673/node30.html http://beige.ucs.indiana.edu/B673/node35.html http://csdl.computer.org/comp/mags/cs/2000/01/cl038abs.htm http://csdl2.computer.org/persagen/DLAbsToc.j sp?resourcePath=/dl/ mags/cs/&toc=comp/mags/cs/2000/01/cltoc.xml&D0I=10.1109/ 5992.814656 http://efgh.com/math/invcond.htm http://en.powerwissen.com/Gl+DpIQ8h2QSmPsQTtN08Q== _QR_algorithm.html http://en.wikipedia.org/wiki/Condition_number http://en.wikipedia.org/wiki/Matrix_norm http://en.wikipedia.org/wiki/QR_algorithm http://en.wikipedia.org/wiki/Tridiagonal_matrix http://epubs.siam.org/sam-bin/dbq/article/23653 http://esperia.iesl.forth.gr/"amo/nr/bookfpdf/fll-5.pdf http://lish.cims.nyu.edu/educational/num_meth_I_2005/lectures/ lec_ll_qr.algorithm.pdf http://gosset.wharton.upenn.edu/"foster/teaching/540/ class_s_plus_l/Notes/nodel.html http://mate.dm.uba.ar/"mat iasg/papers/condi-arxiv.pdf http://math.arizona.edu/"restrepo/475A/Notes/sourcea/node53.html http://math.fullerton.edu/mathews/n2003/hessenberg/HessenbergBib/ Links/HessenbergBib_lnk_2.html http://math.fullerton.edu/mathews/n2003/qrmethod/QRMethodBib/Links/ QRMethodBib_lnk_2.html http://mathworId.wolfram.com/Condit ionNumber.html http://mpec.sc.mahidol.ac.th/numer/STEP16.HTM http://olab.is.s.u-tokyo.ac.jp/"nishida/la7/sld009.htm http://planetmath.org/encyclopedia/ConditionNumber.html http://planetmath.org/encyclopedia/MatrixConditionNumber.html http://w3.cs.huj i.ac.il/course/2005/csip/condition.pdf http://web.ics.purdue.edu/~nowack/geos657/lecture8-dir/lecture8.htm http://www-math.mit.edu/~persson/18.335/lecl4handout6pp.pdf http://www-math.mit.edu/~persson/18.335/lecl5handout6pp.pdf http://www-math.mit.edu/"persson/18.335/lecl6.pdf http://www.absoluteastronomy.com/encyclopedia/q/qr/ qr_algorithml.htm http://www.acm.caltech.edu/~mlatini/research/ presentation-qr-feb04.pdf http://www.acm.caltech.edu/"mlat ini/research/qr_alg-feb04.pdf http://www.caam.rice.edu/"timwar/MA375F03/Lecture22.ppt http://www.cas.mcmaster.ca/~qiao/publications/nm-2005.pdf http://www.cs.Colorado.edu/~mcbryan/3656.04/mail/54.htm http://www.cs.unc.edu/~krishnas/eigen/node4.html 6.3 W e b material http: //www. cs. unc. edu/~krishnas/eigen/node6 .html http://www.cs.ut.ee/"toomas_l/linalg/linl/nodel8.html http://www.cs.utk.edu/~dongarra/etemplates/node95.html http://www.esc.uvic.ca/~dolesky/csc449-540/5.5.pdf http://www.ee.ucla.edu/~vandenbe/103/lineqsb.pdf http://www.efgh.com/math/invcond.htm http://www.ims.cuhk.edu.hk/~cis/2004.4/04.pdf http: //www. krellinst. org/UCES/archive/classes/CNA/dir1.7/ uces1.7.html http://www.library.Cornell.edu/nr/bookcpdf/cl1-3.pdf http://www.library.Cornell.edu/nr/bookcpdf/cl1-6.pdf http://www.ma.man.ac.uk/~higham/pap-le.html http://www.ma.man.ac.uk/~nareport s/narep447.pdf http://www.math.vt.edu/people/renardym/class_home/nova/bifs/ node52.html http://www.math.wsu.edu/faculty/watkins/slides/qr03.pdf http://www.maths.lancs.ac.uk/~gilbert/m306c/node22.html http://www.maths.nuigalway.ie/MA385/novl4.pdf http://www.mathworks.com/company/newsletters/news_notes/pdf/ sum95cleve.pdf http://www.nasc.snu.ac.kr/sheen/nla/html/nodel3.html http://www.nasc.snu.ac.kr/sheen/nla/html/node23.html http://www.netlib.org/scalapack/tutorial/tsldl91.htm http://www.physics.arizona.edu/~restrepo/475A/Notes/sourcea/ node53.html http://www.sci.wsu.edu/math/faculty/watkins/slides/qr03.pdf http://www.ugrad.cs.ubc.ca/~cs402/handouts/handoutl2.pdf http://www.ugrad.cs.ubc.ca/~cs402/handouts/handout26.pdf http://www.ugrad.cs.ubc.ca/~cs402/handouts/handout28.pdf http://www.uwlax.edu/faculty/will/svd/condition/index.html http://www.uwlax.edu/faculty/will/svd/norm/index.html http://www2.msstate.edu/~pearson/num-anal/num-anal-notes/ qr-algorithm.pdf http://www4.ncsu.edu/eos/users/w/white/www/white/dir1.7/ seel.7.6.html 7 C onvex Sets T his chapter is compiled to present a brief summary of the most important concepts related to convex sets. Following the basic definitions, we will concentrate on supporting and separating hyperplanes, extreme points and polytopes. 7.1 P r e l i m i n a r i e s D efinition 7.1.1 A set X in K" is said to be convex if Mxi,X2 G X and Va e R + , 0 < a < 1, the point ax\ + (1 - a)a;2 € X. CONVEX NON-CONVEX F ig. 7.1. Convexity R e m a r k 7.1.2 Geometrically speaking, X is convex if for any points X\,X2 £ X, the line segment joining these two points is also in the set. This is illustrated in Figure 7.1. D efinition 7.1.3 A point x € X is an extreme point of the convex set X if and only if 94 7 Convex Sets / 3 xi, X2 (xi ^ X2) G X 3 x = (1 - a)xi + a x2, 0 < a < 1. P roposition 7 .1.4 v4nj/ extreme point is on boundary of the set. Proof. Let £0 be any interior point of X. T hen 3e > 0 B every point in t his e neighborhood of so is in t his set. Let x\ / XQ b e a point in t his e neighborhood. Consider X2 = ~X\ + 2 x 0 , \x2 ~ X0\ = \X\ - XQ\ t hen X2 is in e neighborhood. Furthermore, XQ = \{x% + X2); hence, xo is not a n extreme point. • R emark 7 .1.5 Not all boundary points of a convex set are necessarily extreme points. Some boundary points may lie between two other boundary points. P roposition 7 .1.6 Convex sets in R satisfy the following relations. ™ i. If X is a convex set and /3 £ R, the set (3X = {y : y = j3x, x G X} is convex, ii. If X and Y are convex sets, then the set X + Y = {z : z = x + y,x £ X,y EY} is convex. Hi. The intersection of any collection of convex sets is convex. o © (iii) F ig. 7.2. Proof of Proposition 7.1.6 Proof. Obvious from Figure 7.2. • A nother important concept is to form t he smallest convex set containing a given s et. D efinition 7 .1.7 Let S C i " . The convex hull of S is the set which is the intersection of all convex sets containing S. D efinition 7 .1.8 A cone C is a set such that ifxGC, then ax G C, Va € R +. A cone which is also convex is known as convex cone. See Figure 7.3. 7.2 Hyperplanes a nd P olytopes 95 NON-COtWEX F ig. 7 .3. Cones 7.2 Hyperplanes a nd P olytopes T he most important type of c onvex s et ( aside from single points) is the hyperplane. R e m a r k 7 .2.1 Hyperplanes dominate the entire theory of optimization; appearing in Lagrange multipliers, duality theory, gradient calculations, etc. The most natural definition for a hyperplane is the generalization of a plane in R 3 . D e f i n i t i o n 7 . 2 . 2 A set V in R n is said to be linear variety, x\,X2 £ V, we have ax\ + (1 — a)x2 £ V,Va £ R. if, given any R e m a r k 7 .2.3 The only difference between a linear variety and a convex set is that a linear variety is the entire line passing through any two points, rather than a simple line segment. D e f i n i t i o n 7 .2.4 A hyperplane in R™ is an ( n—1) -dimensional linear variety. It can be regarded as the largest linear variety in a space other than the entire space itself. P r o p o s i t i o n 7 .2.5 Let a £ R " , a ^ 9 and b G R . The set ff = { i £ l " : arx = b} is a hyperplane in R ". Proof. L et x\ £ H. T ranslate H b y — x\, we t hen obtain t he set M = H - Xi ={y€Rn:3xeH3y = x~ xx}, w hich is a l inear subspace of R n . M — {y G R : a? y = 0 } is a lso t he set of ™ all orthogonal vectors t o a G R n , w hich is c learly ( n — 1) d imensional. D P r o p o s i t i o n 7 .2.6 Let H be an hyperplane in R " . Then, 3a G R " 3 H = {x G R : aTx = b}. 96 7 Convex Sets Proof. Let xi 6 H, and translate by -x\ o btaining M = H — x\. Since H is a h yperplane, M is an (n — 1)-dimensional space. Let a b e any orthogonal to M, i.e. a 6 ML. T hus, M = {y e M" : a T t/ = 0}. Let 6 = aTxi we see that if x2 E H, X2 — xi G M and therefore aTX2 - aTx\ = 0 => aTX2 = & Hence, • H C {a; € K : a T x = 6}. Since i? is, by definition, of (n — 1) dimension, and { x e R : aTx = b} is of dimension (n — 1) by the above proposition, these two sets must be equal (see Figure 7.4). • a z ^ 7 ^ 7 'H e Fig. 7.4. Proof of Proposition 7.2.6 D efinition 7.2.7 Let o e l " , fo e M. Corresponding to the hyperplane H = {x : aTx = b}, there are positive and negative closed half spaces: H+ = {x : aTx > b}, H- = {x : aTx < b} and H+ = {x : aTx > b}, # _ = {x : aTa; < &}. ffa// spaces are convex sets and H+ U H- = W1. D efinition 7.2.8 A set which can be expressed as the intersection of a finite number of closed half spaces is said to be a convex polyhedron. Convex polyhedra are the sets obtained as the family of solutions to a set of linear inequalities of the form a\x < b\ a\x < b<2 a-mX < bm Since each individual entry defines a half space and the solution family is t he intersection of these half spaces. D efinition 7.2.9 A nonempty bounded polyhedron is called a polytope. 7.3 Separating and Supporting Hyperplanes 97 7.3 Separating and Supporting Hyperplanes T heorem 7.3.1 (Separating Hyperplane) Let X be a convex set and y be a point exterior to the closure of X. Then, there exists a vector a £ M.n 3 aTy < inixex aTx. (Geometrically, a given point y outside X, a separating hyperplane can be passed through the point y that does not touch X. Refer to Figure 7.5) F ig. 7.5. Separating Hyperplane Proof. Let 6 = inf^gx \x — y\ > 0 T hen, there is an XQ on the boundary of X such that |xo — y\ — S. Let z £ X. T hen, Va, 0 < a < 1, XQ + a(z — XQ) is the line segment between xo and z. T hus, by definition of xo, \x0 + a(z - x0) - y\2 > \x0 - y\2 « > (x0-y)T(x0-y)+2a(x0-y)T(z-xo)+a2(z-x0)T(z-xo) = > (x0-y)T(x0-y) < > 2a(x0 - y)T(z - x0) + a2\z - x0\2 > 0 = Let a —• 0 + , then a2 t ends to 0 more rapidly than 2a. T hus, $ (x0 - y)T(z - x0) > 0 <s> ( x 0 - y)Tz - (x0 - y)Tx0 & (x0-y)Tz > (x0-y)Tx0 = (x0-y)Ty + (x0 + y)T{x0-y) >0 + 52 = (x0-y)Ty (Since 5 > 0 ). & ( x 0 - y)Ty < (x0 - y)Tx0 < ( x 0 - y)Tz, Vz£X Let a = (XQ — y), t hen aTy < aTxo = i nf z 6 x <iTz. U 98 7 Convex Sets T heorem 7 .3.2 ( Supporting Hyperplane) Let X be a convex set, and let y be a boundary point of X. Then, there is a hyperplane containing y and containing X in one of its closed half spaces. Proof. Let{yk} be sequence of vectors, exterior t o the closure of X, converging t o y. Let {a,k} b e a sequence of corresponding vectors constructed according t o t he previous theorem, normalized so t hat \dk\ = 1, such that a^yk < i nf x € xSince {a*,} is a b oundary sequence, it converges t o a. For t his vector, we have aTy = lima^j/fe < ax. O D efinition 7 .3.3 A hyperplane containing a convex set X in one of its closed half spaces and containing a boundary point of X is said to be supporting hyperplane of X. 7.4 Extreme Points R emark 7 .4.1 We have already defined extreme points. For example, the extreme points of a square are its corners in M.2 whereas the extreme points of a circular disk are all (infinitely many!) the points on the boundary circle. Note that, a linear variety consisting of more than one point has no extreme points. L emma 7 .4.2 Let X be a convex set, H be a supporting hyperplane of X and T = X D H. Every extreme point ofT is an extreme point of X. Proof. Suppose xo 6 T is not an e xtreme point of X. T hen, xo = ax\ + (1 — a)x2 for some X\,X2 6 X, 0 < a < 1. Let H — {x : aTx = c} w ith X contained in its closed positive half space. T hen, aTX\ > c, aTx^ > c. However, since XQ £ H, c = aTxo = aaTxi + (1 — a)aTX2Thus, xi, X2 E H. Hence, x\, X2 &T a nd xo is not an e xtreme point of T. • T heorem 7 .4.3 A closed bounded convex set in R n is equal to the closed convex hull of its extreme points. Proof. T his proof is by induction on n. For n = 1, the s tatement is t rue for a line segment: [a, b] = {x £ R : x = a + (1 - a)b,0 < a < 1 }. Suppose that t he t heorem is t rue for (n — 1). Let X b e a closed bounded convex set in K n , and let K b e the convex hull of the e xtreme points of X. 7.4 Problems 99 We will show that X = K. Assume that 3y e X 3 y $ K. T hen, by Theorem 7.3.1, t here is a hyperplane s eparating y and K; 3a ^ 0 3 aTy < inf aTx xEK Let XQ = m{X£x(aTx)- xo is finite and 3xo £ X 3 aTxo = bo (because by W eierstrass' Theorem: The continuous function aTx achieve its minimum over any closed bounded set). Hence, the hyperplane H = {x : aTx = bo} is a supporting hyperplane to X. Since b0 < aTy < mixeK aTx, H is disjoint from K. Let T = H D X. T hen, T is a bounded closed convex set of H, which can be regarded as a space in R n _ 1 . T ^ 0, since XQ € T. Hence, by induction hypothesis, T contains e xtreme points; and by the previous Lemma, these are the extreme points of X. T hus, we have found extreme points of X not in K, C ontradiction. Therefore, X C K, and hence X = K (since K C X, i.e. K is closed and b ounded). • R emark 7.4.4 Let us investigate the implications of this theorem for convex polytopes. A convex polytope is a bounded polyhedron. Being the intersection of closed halfspaces, a convex polytope is closed. Thus, any convex polyhedron is the closed convex hull of its extreme points. It can be shown that any polytope has at most a finite number of extreme points, and hence a convex polytope is equal to the convex hull of a finite number of points. The converse can also be established, yielding the following two equivalent characterizations. T heorem 7.4.5 A convex polytope can be described either as a bounded intersection of a finite number of closed half spaces, or as the convex hull of a finite number of points. P roblems 7 .1. C haracterize (draw, give an example, list extreme points and halfspaces) t he following polytopes: a) zero dimensional polytopes. b) one dimensional polytopes. c) two dimensional polytopes. 7 .2. d-simplex d-simplex is the convex hull of any d+1 independent points in R (n > d). ™ S tandard d — simplex w ith d+1 vertices in R d + 1 is d+l Ad = {x£ Rd+1 : ] T Xi = 1; x{ > 0, i = 1 , . . . , d + 1 }. C haracterize A2 in R 3 . 100 7 Convex Sets 7 .3. C ube and Octahedron C haracterize cubes and octahedrons with the help of three dimensional cube C3, and octahedron C^. 7 .4. Pyramid Let P n +i=conv(C n ,:Eo) be a (n+l)-dimensional pyramid, where XQ $ Cn. Draw P3 = conv{C2 : a = 1, ( 1/2,1/2,1) T ) and write down all describing inequalities. 7 .5. Tetrahedron T he vertices of a tetrahedron of side length \[2 can be given by a particularly simple form when the vertices are taken as corners of the unit cube. Such a t etrahedron inside a cube of side length 1 has side length \/2 with vertices ( 0,0,0) T , ( 0,1,1) T , (1,0,1) T , (1,1,0) T . Draw and find a set of describing inequalities. Is it possible to express Pn+\ as a union / intersection / direct sum of a cone and a polytope? 7 .6. Dodecahedron F ind the vertices of a dodecahedron (see Figure 7.6) of side length a = \ /5 — 1. F ig. 7.6. A dodecahedron W eb material http://cepa.newschool.edu/het/essays/math/convex.htm http://cm.bell-labs.eom/who/clarkson/cis677/lecture/6/index.html http://cm.bell-labs.eom/who/clarkson/cis677/lecture/8/ 7.5 W e b material 101 http://dimax.rutgers.edu/"sj aslar/ http://dogfeathers.com/j ava/hyperslice.html http://en.wikipedia.org/wiki/Polytope http://en.wikipedia.org/wiki/Wikipedia:WikiProject.Mathematics/ PlanetMath_Exchange/52-XX_Convex_and_discrete_geometry http://eom.springer.de/c/c026340.htm http://grace.speakeasy.net/"dattorro/EDMAbstract.pdf http://grace.speakeasy.net/~dattorro/Meboo.html http://learningtheory.org/colt2004/colt04_boyd.pdf http://math.sfsu.edu/beck/teach/870/lecture5.pdf http://mathworld.wolfram.com/Convex.html http://mathworld.wolfram.com/Polytope.html http://mizar.uwb.edu.pi/JFM/pdf/convex3.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-253Spring2004/14DD65AE-0A43-4353-AE09-7B107CC4AAD7/0/ lec_ll.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-253Spring2004/81D31E98-C26B-4375-B089-FB5FAE4E99CF/0/ lec_7.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-253Spring2004/96203668-B98C-4F3C-A65D-4646F942EF71/0/ lec_3.pdf http://staff.polito.it/giuseppe.calafiore/cvx-opt/secure/ 02_cvx-sets_gc.pdf http://www-math.mit.edu/"vempala/18.433/L4.pdf http://www-personal.umich.edu/~mepelman/teaching/I0E611/Handouts/ 611Sets.pdf http://www.cas.monaster.ca/"cs4te3/notes/convexopt.pdf http://www.cas.monaster.ca/~deza/Comb0ptim_Ch7.ppt http://www.cis.upenn.edu/~cis610/polytope.pdf http://www.cs.emu.edu/afs/cs/academic/class/16741-s06/www/ Lecturel3.pdf http://www.cs.wustl.edu/"pless/506/12.html http://www.cse.unsw.edu.au/~lambert/java/3d/ConvexHull.html http://www.eecs.berkeley.edu/~wainwrig/ee227a/Scribe/ lecture12_f inal.verB.pdf http://www.eleves.ens.fr/home/trung/supporting_hyperplane.html http://www.geom.uiuc.edu/graphics/pix/Special_Topics/ Computational.Geometry/cone.html http://www.geom.uiuc.edu/graphics/pix/Special_Topics/ Computational_Geometry/half.html http://www.hss.caltech.edU/~kcb/Ecl01/index.shtml#Notes http://www.ics.uci.edu/~eppstein/junkyard/polytope.html http://www.irisa.fr/polylib/D0C/nodel6.html http://www.isye.gatech.edu/~spyros/LP/nodel5.html http://www.j stor.org/view/00029939/di970732/97p0127h/0 http://www.mafox.com/articles/Polytope http://www.math.rutgers.edu/pub/sontag/pla.txt http://www.maths.lse.ac.uk/Personal/mart in/fme9a.pdf 102 7 Convex Sets http://www.mizar.org/JFM/Vol15/convex3.html http://www.ms.uky.edu/~sills/webprelim/sec013.html http://www.mtholyoke.edu/"j s idman/wolbachPres.pdf http://www.princeton.edu/"chiangm/ele53912.pdf http://www.Stanford.edu/class/ee364/lectures/sets.pdf http://www.Stanford.edu/class/ee364/reviews/reviewl.pdf http://www.Stanford.edu/class/msande310/lecture03.pdf http://www.Stanford.edu/~dattorro/mybook.html http://www.stat.psu.edu/~jiali/course/stat597e/notes2/percept.pdf http://www.uni-bayreuth.de/departments/wirtschaftsmathematik/rambau/ Diss/diss_MASTER/node35.html http://www.wisdom.weizmann.ac.il/~feige/lp/lecture2.ps http://www2.isye.gatech.edu/~spyros/LP/nodel5.html http://www2.sj su.edu/faculty/watkins/convex.htm 8 L inear Programming A Linear Programming problem, or LP, is a problem of optimizing a given linear objective function over some polyhedron. We will present the forms of LPs in this chapter. Consequently, we will focus on the simplex method of G. B. Dantzig, which is the algorithm most commonly used to solve LPs; in practice it runs in polynomial time, but the worst-case running time is e xponential. Following the various variants of the simplex method, the duality t heory will be introduced. We will concentrate on the study of duality as a means of gaining insight into the LP solution. Finally, the series of Farkas' Lemmas, the most important theorems of alternatives, will be stated. 8.1 The Simplex Method T his section is about linear programming: optimization of a linear objective function subject to finite number (m) of linear constraints with n unknown and nonnegative decision variables. E xample 8.1.1 The following is an LP: Min z =2x + 3y s.t. 2x + y>Q x + 2y>6 x,y>0. S tandard Form: Min z —ex s.t. Ax>b x>9 104 8 Linear Programming Canonical form: Min z =c x + 6 y s.t. Ax — y = 6 x,y>0 x <S=> Min s.t. z^c'ie1} =b v E xample 8.1.2 Min z =2x + 3y s.t. 2x + y>6 x + 2y>6 x,y>0. >e. Feasible set Fig. 8 .1. The feasible solution region in Example 8.1.2 See Figure 8.1. "6' "12 - 1 0' A= 2 1 0 - 1 ,6 = 6 , c = [2] 3 0 0 8.1 The Simplex Method 105 D efinition 8 .1.3 The extreme points of the feasible set are exactly the basic feasible solutions of Ax = b. A solution is basic when n of its m+n components are zero, and is feasible when it satisfies x > 6. Phase I of the simplex method finds one basic feasible solution, and Phase II moves step by step to the optimal one. If we are already a t a basic feasible solution x, a nd for convenience we reorder its components so t hat t he n zeros correspond t o free variables. XB xN = < xB XN ,A = [B,N], (cB,cN) Min z =(cB',cJf) s.t. [B\N] XB <> = Min z —CQXB + CNXN s.t. BxB + NxN = b XB,XN >0 XB XN = 6 =b XN =' >e. « • xB - B~1[b-NxN] B~1b-B~1NxN- Let us t ake t he c onstraints BXB+NXN =bo Bxb = b-NxN - Now plug XB in the objective function z= CTBXB + CNXN = c%[B~1b - B_1NXN] + CNXN = cTBB~lb + {cTN - cBB-lN)xN. If we let XN = 0, t hen xB = B~lb > 6 => z — cB'B~1b. P roposition 8 .1.4 ( Optimality Condition) If the vector (cN - cBB~1N) is nonnegative, then no reduction in z can be achieved. The current extreme point (XB = B~1b,XN = 0) is optimal and the minimum objective function value is CBB~1b. Assume that t he o ptimality condition fails, t he usual greedy strategy is to choose t he most negative component of CJV — CBB~1N, known as D antzig's rule. T hus, we have determined which component will move from free t o basic, called as entering variable xe. We have t o decide which basic component is t o become free, called as leaving variable, x\. Let Ne b e the column of N corresponding t o xe. xB = B~lb- B~1Nexe. If we increase xe from 0, some entries of XB may begin t o decrease, and we reach a a neighboring extreme point when a component of XB reaches 0. It is the component corresponding t o x\. At t his extreme point, we have reached a new x which is b oth feasible 106 8 Linear Programming and basic: it is feasible because x > 0, it is basic since we again have n zero c omponents. xe is gone from zero to a, replaces xi which is dropped to zero. T he other components of XB might have changed their values, but remain positive. P roposition 8.1.5 (Min Ratio) Suppose u = Ne, then the value of' xe will be: . {B-lb)j 1 (B^b), a = min —r - = ,L , l Xj-.basic (B~ U)j (B U)l and the objective function will decrease to cBB~1b — aB~1u. R emark 8.1.6 (Unboundedness) The minimum is taken only over positive components of B~1u, since negative entries will increase XB and zero entries keeps XB as their previous values. If there are no positive components, then the next extreme point is infinitely far away, then the cost can be reduced forever; z — — oo/ In this case we term the optimization problem as unbounded. R emark 8.1.7 (Degeneracy) Suppose that more thann of the variables are zero or two different components if the minimum ratio formula give the same minimum ratio. We can choose either one of them to be made free, but the other will still be in the basis at zero level. Thus, the new extreme point will have (n + 1) zero components. Geometrically, there is an extra supporting plane at the extreme point. In degeneracy, there is the possibility of cycling forever around the same set of extreme points without moving toward x*, the optimal solution. In general, one may assume nondegeneracy hypothesis {xB=B~lb>6). E xample 8.1.8 Assume that we are at the extreme point V in Figure 8.1, corresponding to the following basic feasible solution: XB [xN\ zi y x z2 -1 2 1 0 0 1 2-l_ "6" 6 U 0_ rT ' Zl' y X . 2. z A = [B\N] = __ ( r T \ T \ c — \CB\CN) zi y X Z2 0 32 0 -l CN cTBB-lN = [2 0] - [0 3] - 12 01 10 2-1 -12 01 - 1 2 1 0' [1 - 2 - 1 0 ] [ 10 - 1 2 ] ^B-1^ 01 01 — 0 1 01 — 01 01 > > 8.2 Simplex Tableau cTN-cTBB~xN cTN-cTBB~lN= = [2 0] - [0 3 ] > [2 0] - [0 3] -12' 01 107 [ l Ol 2-1 ( v "3-2' 2 -1 X Z2 -4 3 Since the first component is negative, P is not optimal; x should enter the basis, i.e. x,Ne = => B^N' B-1b-B-1Nexe = Zl ,B_16 = 6 6 3 x> 2 0 0 = 2. xB = a = Min{l y = = 2, § = 3} = 2. Thus, xL = zuxe " 2' 2 U 0 = 2,y = Q-2a X xB _XN _ y Zl .Z2. 4 = [B\N] = 12 - 1 0 2 1 0-1 12 21 [J|*-l]. cT=[cT\cl] = [2 3\0 0],B 1 2 10 0 - 3 -2 1 = [ 5 | / ] = n2 1 0 1 1 2 10 - > • U1 O il 1 Ol —^ ^ 1u l3 3 ^I I 3 _3 c £ - c £ £ - 1 J V = [0 0] - [2 3] 1 _2 II I-I x -1 0 0-1 00 23 3 3 2 _1 "3 3 H ]>«b = 10 is the optimal 77ms, extreme point Q in Figure 8.1 is optimal, c^B value of the objective function. 8.2 Simplex Tableau We have achieved a transition from the geometry of the simplex method to algebra so far. In this section, we are going to analyze a simplex step which can be organized in different ways. T he Gauss-Jordan method gives rise to the simplex tableau. [A\\b] = [B\N\\b] — • [IIB^NWB-H]. 108 8 Linear Programming Adding the cost row I \B-lN\\B-lb -N B-lN B~xb 0 O c^-c^B-^ll-cSB-^ T he last result is the complete tableau. It contains the solution B~1b, t he crucial vector CNT — cBTB~1N and the current objective function value cBTB~1b w ith a superfluous minus sign indicating that our problem is minimization. The simplex tableau also contains reduced coefficient matrix B~lN t hat is used in the minimum ratio. After determining the entering variable xe, we examine the positive entries in the corresponding column of B~XN, (v = B~lu = B~1Ne) and a is determined by taking the ratio of tB-iNl\. for all positive Vj's. If the smallest ratio occurs in Ith component, then the Ith column of B should be replaced by u. T he Ith element of (B~lNe)i = V[ is distinguished as pivot element. I t is not necessary to return the starting tableau, exchange two columns and start again. Instead we can continue with the current tableau. Without loss of generality, we may assume that the first row corresponds to the leaving variable, that is the pivot element is vi. 1 : 0---0 0 B^N: V2 (B-'bh :B-XN B~lb :0:0---0 ••* :cR T CBV.* • -cTBB~lb. T he first step in the pivot operation is to divide the leaving variable's row by the pivot element to create 1 in the pivot entry. Then, we have 8.2 Simplex Tableau : J-:0---0 * • • • * : 1 109 : / B~lN\ '•.B~lN B-lb 0:0---0 • *:ce — Cnv: * • -dB^b F or all the rows except the objective function row, do the following operation. For row i, m ultiply Vi*(the updated first row) and subtract from row i. F or the objective function row, multiply the first row by (c e — CBTV) a nd s ubtract from the objective function row. W h a t we have at the end is another simplex tableau. : 0---0 a :0 + + _Vj_ :0 c ,~Ce gV.Q. . .Q *•••*: 0 :*••• * —c^B 1 b — a(ce — CgV BU) A E x a m p l e 8.2.1 The starting Ab tableau at point P is B\N\\b rT\rT\\() CC B\ N\\U =rllo " - 1 2 1 0 6" 01 2 - 1 6 = 03 2 0 0 iterations B-XN is \B~lb -cost The final tableau after Gauss-Jordan zi y Zl X Zi 1 0 3-2 y 0 1 2 -1 z 00 -4 3 RHS~ 6 6 -18 = 0\rT v\cN c ^^JVll Since the reduced cost for x is — 4 < 0 , x should enter the basis. The minimum ratio a = M m { | , | } = 2 due to z\, thus z\ should leave the basis. 110 8 Linear Programming 6" " 10 3 - 2 6 0 1 2-1 00 - 4 3 -18 77ms, x* = 2 = y* => 2* 10. 2i X y\x z2\\RHS- -III § 0|0 2 -10 y III ^ R emark 8.2.2 A Z i/ie pwo£ operation can be handled by multiplying the inZ verse of the following elementary matrix. 1 \vi :0 . .0 — V\ Vl 0 : . :. 0 E= 0. l iwjio . . 0: . :1 .: . : 1 0 .0 1_ <F> E~ Vl .: . : 0 0. • Q'-vm: 1_ Thus, the pivot operation is [^B^NWB-H} —> [E-1I\E-1B-1N\\E-1B-Ib] . New basis is BE (B except the Ith column is replaced byu = Ne) and basis inverse is (BE)"1 — E~1B~1. This is called p roduct form of the inverse. Thus, if we store E"1 's then we can implement the simplex method on a simplex tableau. 8.3 Revised Simplex M e t h o d Let us investigate what calculations are really necessary in the simplex m ethod. Each iteration exchanges a column of N w ith a column of B, a nd one has to decide which columns to choose, beginning with a basis matrix B a nd the current solution XB = B~lb. S I. C ompute row vector A = cjgB x and then cL — XN. 8.4 Duality Theory 111 5 2. If Cjy — XN > 6, stop; the current solution is optimal. Otherwise, if the most negative component is eth component, choose eth column of N to enter the basis. Denote it by u. 5 3. C ompute v = B~lu. 54. Calculate ratios of B~lb t o v = B~lu, a dmitting only positive components of v. If there are no positive components, the minimal cost is —oo; if the smallest ratio occurs at component I, t hen Ith column of current B will be replaced with u. 5 5. U pdate B (or £? _1 ) and the solution is XB = B~lb. R eturn to SI. R emark 8.3.1 We need to compute X = cB^B~l,v = B~lu, and XB = B~*b. Thus, the most popular way is to work only on B"1. With the help of previous remark, we can update B~l 's by premultiplying E_1 's. T he excessive computing (multiplying with E~lys) could be avoided by directly reinverting the current B a t a time and deleting the current E~l,s t hat contain the history. R emark 8.3.2 The alternative way of computing X,v and XB is XB = Cg,Bv — u, and BXB = b. Then, the standard decompositions (B = QR or PB = LU) lead directly to these solutions. R emark 8.3.3 How many simplex iterations do we have to take? There are at most ( ^) extreme points. In the worst case, the simplex method may travel almost all of the vertices. Thus, the complexity of the simplex method is exponential. However, experience supports the following average behavior. The simplex method travels about m extreme points, which means an operation count of about m2n, which is comparable to ordinary elimination to solve Ax = b, and that is the reason of its success. 8.4 Duality Theory T he standard primal problem is: Minimize cTx subject to Ax > b and x > 6. T he dual problem starts from the same A, b, and c and reverses everything: Maximize yTb subject to ATy < c and y > 8. T here is a complete symmetry between the two. The dual of the dual is t he primal problem. Both problems are solved at once. However, one must recognize that the feasible sets of the two problems are completely different. T he primal polyhedron is a subset of M.n, marked out by matrix A and the right hand side b. T he dual polyhedron is a subset of K m , determined by AT and the cost vector c. T he whole theory of linear programming hinges on the relation between t hem. T heorem 8.4.1 (Duality Theorem) / / either the primal problem or the dual has an optimal vector, then so does the other, and their values are the 112 8 L inear Programming same: The minimum of cTx equals the maximum ofyTb. Otherwise, if optimal vectors do not exist, either both feasible sets are empty or else one is empty and the other problem is unbounded. T heorem 8 .4.2 ( Weak Duality) If x andy are feasible vectors in the minimum and maximum problems, then yTb < cTx. Proof. Since they are feasible, Ax > b a nd ATy < c (<& yTA < cT). T hey should b e nonnegative as well: x > 8, y > 9. Therefore, we can t ake inner p roducts without ruining t he inequalities: yTAx > yTb a nd yTAx < cTx. T hus, yTb < cTx since left-hand-sides are identical. D C orollary 8 .4.3 / / the vectors x and y are feasible, and if cTx = yTb, then these vectors must be optimal. Proof. No feasible y can make yTb larger than cTx. Since our p articular y achieves this value it should b e o ptimal. Similarly, x should b e o ptimal. D T heorem 8 .4.4 ( Complementary Slackness) Suppose the feasible vectors x and y satisfy the following complementary slackness conditions: if (Ax)i > bi, then yt = 0 and if (ATy)j < Cj, then Xj = 0. Then, x and y are optimal. Conversely, optimal vectors must satisfy complementary slackness. Proof. At o ptimality we have yTb = yT(Ax) = (yTA)x = cTx. If y > 0 a nd Ax > b =4- yTb < yT(Ax). W hen yTb — yT{Ax) holds, if bi < (Ax)i, t he corresponding factor «/; should b e zero. T he same is t rue for y1Ax < cTx. If Cj > (ATy)j t hen Xj = 0 t o have yTAx = cTx. T hus, complementary slackness guarantees (and is g uaranteed by) optimality. • Proof (Strong Duality). We have t o show that yTb = cTx is really possible. Max cTx, Ax>b, x>8 = 6, «• Max [cT\9T] . M -'] XB XN X >e. [A\-I] -¥ [B\N] O ptimality condition: NT(BT) -» B~lb 0 [cH^] - • [ « ] • CB < c^. Since we have finite number of extreme points, t he o ptimality condition is eventually m et. At t hat moment, t he minimum cost is cTx = CQB~1XBMax by subject t o Ar -I y< c -> 0 B1 y< CB CN <= BTy = cB S> 8.5 Farkas' Lemma &yTB cTB^yT cTBB~x & yTb = cTBB~xb = cTx\ 113 F urthermore, this choice of y is optimal, and the strong duality theorem has been proven. This is a constructive proof, x* and y* were actually computed, which is convenient since we know that the simplex method finds the optimal values. D 8.5 Farkas' L e m m a C oll mnl H Column2 b * J •— o \J// / / >/// Column3 Column4 i i) Ax=b has s nonnegative solution rt £<i H 1 ]\ -«eJ^(// 1 i f- i Coli mnl ^/^Column2 -»•_ C olumn?" Column4 (ii) Else F ig. 8.2. Farkas' Lemma By the fundamental theorem of Linear Algebra, e ither b G U(A) or 3y e M(AT) By JLb, t hat is, there is a component of b in the left null space. Here, we immediately have the following theorem of alternatives. P roposition 8.5.1 Either Ax — b has a solution, or else there is a y 3 ATy = 9,yTb^0. If b € C one(a 1 , a 2 , a 3 , . . . ) t hen Ax — b is solvable. If 6 0 Cone(columns of A), t hen there is a separating hyperplane which goes through the origin defined by y t hat has b on the negative side. The inner product of y and b is negative (yTb < 0) since they make a wide angle (> 90°) whereas the inner p roduct of y and every column of A is positive (ATy > 0). T hus, we have the following theorem. P roposition 8.5.2 Either Ax = b,x > 9 has a solution, or else there is a y such that ATy > 6, yTb < 0. C orollary 8.5.3 Either Ax > b,x > 9 has a solution, or else there is a y such that ATy > 9, yTb < 0, y < 9. 114 8 Linear P r o g r a m m i n g Proof. Ax > b, x > » -» Ax - Iz = E ither \A-I\ X b,z>9. has a nonnegative solution or 3y 3 AT -I y > 0,yTb < 0. => ATy > 9, yTb < 0, y < 9. D R emark 8.5.4 The propositions in this section can also be shown using the primal dual pair of linear programming problems: If the dual is unbounded, the primal is infeasible. 1. Either Ax = b has a solution, or else there is a y 3 A1 y = 9, y1 b =/= 0; ( PI) : Max 6Tx s.t. Ax = b x : URE (P2) : Min 9Tx s.t. Ax = b x : URE (Dl) : Min s.t. ATy = 9 y : URE bTy ATy = 9 y : URE bTy (D2) : Max s.t. Either PI (or P2) is feasible, or Dl (or D2) is unbounded. For Dl (D2) to be unbounded, we 'must have bTy < 0 (b1 y > 0). Thus, either Ax = b or By 3 ATy = 9, yTb ^ 0. 2. Either Ax = b,x > 9 has a solution, or else there is a y such that ATy>9,yTb<0: ( P3) : Max 9Tx s.t. Ax = b x>9 (D3) : Mm s.t. ATy > 9 y. URE bTy Either PS is feasible, or D3 is unbounded. For D3 to be unbounded, we must have bTy < 0 . Thus, either Ax = b, x > 9 has a solution, or else 3y 3 Ary > 9, yTb < 0. 3. Either Ax > b, x > 9 has a solution, or else there is a y such that ATy> 9,yTb < 0, y < 9: ( P4) : Max 9rx s.t. Ax > b x>9 (D4) : Min s.t. ATy > 9 y<9 bTy 8.5 Problems 115 Either P4 is feasible, or D4 is unbounded. For D4 to be unbounded, we must have bTy < 0 . Thus, either Ax > b,x > 8 has a solution, or else 3y 3 ATy > 6, yTb < 0, y < 9. P roblems 8 .1. ( P): Max z = xi + 2x2 + 2x3 s.t. 2xi + x 2 < 8 x 3 < 10 x2 > 2 x i , x 2 , x 3 > 0. Let the slack/surplus variables be «i, S2, S3. a)Draw the polytope defined by the constraints in R 3 , identify its extreme p oints and the minimum set of supporting hyperplanes. b) Solve (P) using 1. m atrix form, 2. simplex tableau, 3. revised simplex with product form of the inverse, 4. revised simplex with B = LU decomposition, 5. revised simplex with B = QR decomposition. c) Write the dual problem, draw its polytope. 8 .2. Let P = { (xi,x 2 ,x 3 ) > 0 and 2xi — x 2 — X3 > 3 xi - x2 + x3 > 2 x i - 2 x 2 + 2 x 3 > 4 }. Let «i,S2,S3 be the corresponding slack/surplus variables. a) Find all the extreme points of P. b) Find the extreme rays of P (if any). c) Considering the extreme rays of P (if any) check whether we have a finite solution x G P if we maximize 1. x i + x 2 + x 3 , 2. - 2 x i — X2 - 3 x 3 , 3. —xi — 2x2 + 2x3. d) Let xi = 6, x 2 = 1, X3 = | . Express this solution with the convex 116 8 Linear Programming combination of extreme points plus the canonical combination of extreme rays (if any) of P. e) Let the problem be m inxi + 2x2 + 2x3 subject to (xi,X2,X3) € P. 1. Solve. 2. W hat if we reduce the right hand side of (1) by 3 and (3) by 1. 3. Consider the solution found above. What if we add a new constraint 2xi + 5x2 + X3 < 3. 8 .3. U pper bounded simplex Modify the simplex algorithm without treating the bounds as specific constraints but modifying the optimality, entering and leaving variable selection conditions to solve the following LP problem: max 2xi + 3x2 + x 3 + 4 x 4 s.t. x a + 2x2 + 3 x 3 + 5x4 < 30 xx + x2 < 13 3 x 3 + x 4 < 20 (1) (2) (3) 1 < x i < 6, 0 < x 2 < 10, 3 < x 3 < 9, 0 < x 4 < 5 a) Start with the initial basis as {si,S2,S3} where si,S2,ss a re the corresponding slack variables at their lower bounds. Use Bland's (lexicographically ordering) rule in determining the entering variables. Find the optimal solution. b) Take the dual after expressing the nonzero lower/upper bounds as specific c onstraints. Find the optimal dual values by considering only the optimal p rimal solution. 8 .4. Decomposition Let a e A be an arc of a network T = (V,A), where ||V|| = n, \\A\\ = m. V Given a node i € V, let T(i) be the set of arcs entering to i a nd H(i) b e t he set of arcs leaving from i. Let there be k — 1 , . . . , K commodities to be d istributed; Cka denotes the unit cost of sending a commodity through an arc, Uka d enotes the corresponding arc capacity, du% denotes the supply/demand a t node i, and Ua is the total carrying capacity of arc a. a) Let Xfc0 b e the decision variable representing the flow of commodity k across arc a. Give the classical Node-Arc formulation of the minimum cost multi-commodity flow problem, where commodities share capacity. Discuss t he size of the formulation. 8.6 Web material 117 Fig. 8 .3. Starting bfs solution for our multi-commodity flow instance b) Let Vk be the set of paths from source node sk t o sink node tk for commodity k. For P e V, let fp: flow on path P (decision variable), . , 1, if a is in P r lap = 0, otherwise Ckp: u nit cost of flow = ^ a Iapcka Dk- d emand for the circulation fip: u pper bound on flow = min {uka '• = 1} Give the Path-Cycle formulation, relate to the Node-Arc formulation, and discuss the size. c) Take the path cycle formulation. Let wa be the dual variable of the capacity constraint and TTk t he dual variable of the demand constraint. What will b e the reduced cost of path PI W hat will the reduced cost of path P a t the o ptimality? Write down a subproblem (column generation) that seeks a path w ith lower cost to displace the current flow. Discuss the properties. d) Solve the example instance using column generation starting from the solution given in Figure 8.3. Let us fix all capacities at 10 and all positive s upplies/demands at 10 with unit carrying costs. e) Sketch briefly the row generation, which is equivalent to the DantzigWolfe/Bender's decompositions' viewpoint. W eb material http://agecon2.tamu.edu/people/faculty/mccarl-bruce/mccspr/new04.pdf http://archives.math.utk.edu/topics/linearProg.html http://catt.bus.okstate.edu/itorms/volumes/voll/papers/murphy/ http://cepa.newschool.edu/het/es says/math/convex.htm#minkowski http://cgm.cs.mcgill.ca/~beezer/Publications/avis-kaluzny.pdf http://cis.poly.edu/rvslyke/simplex.pdf http://citeseer.ist.psu.edu/62898.html 118 8 Linear Programming http://en.wikipedia.org/wiki/Farkas's_lemma http: //en. wikipedia. org/wiki/Linear_programming http://mathworld.wolfram.com/FarkassLemma.html http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-253Spring2004/A609002F-E9DF-47F0-8DAF-7F05F16920F7/0/ lec_19.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-253Spring2004/E9B02139-0C6E-4AC6-A8B7-5BA22D281DBF/0/ lec_12.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-854JAdvanced-AlgorithmsFalll999/8C3707F7-2831-498483FB-BD7BF754A1lE/O/notes18.pdf http://ocw.mit.edu/NR/rdonlyres/Electrical-Engineering-and-ComputerScience/6-854JFall2001/FB552487-8Ell-4D14-A064-B521724DCE65/0/ notes_lp.pdf http://ocw.mit.edu/NR/rdonlyres/Mathematics/18-310Fall-2004/32478C796843-4775-B925-068489AD0774/0/liner_prog_3_dua.pdf http://ocw.mit.edu/NR/rdonlyres/Mathematics/18-310Fall-2004/ACD5267C0B38-4DDF-97AC-C4B32E20B4EE/0/linear_prog_ii.pdf http://ocw.mit.edu/0cwWeb/Sloan-School-of-Management/15-066JSystem0ptimization-and-Analysis-for-ManufacturingSummer2003/ LectureNotes/index.htm http://opim.wharton.upenn.edu/~guignard/321/handouts/duality_0K.pdf http://planetmath.org/encyclopedia/FarkasLeirana.html http://planetmath.org/encyclopedia/LinearProgrammingProblem.html http://shannon.math.gatech.edu/~bourbaki/2602/lp/lp.pdf http://web.mit.edu/15.053/www/AMP-Appendix-B.pdf http://www-math.mit.edu/18.310/28.pdf http://www-personal.umich.edu/"mepelman/teaching/I0E610/lecture5.pdf http://www.comp.leeds.ac.uk/or21/0VERHEADS/sect5.pdf http://www.cs.berkeley.edu/"vazirani/s99csl70/notes/linear3.pdf http://www.cs.helsinki.fi/u/gionis/farkas.pdf http://www.cs.nyu.edu/cs/faculty/overton/g22_lp/encyc/ article_web.html http://www.cs.toronto.edu/"avner/teaching/2411/index.html http://www.cs.toronto.edu/"avner/teaching/241l/ln/lecture6.pdf http://www.cs.uiuc.edu/class/fa05/cs473g/lectures/17-lp.pdf http://www.cs.uiuc.edu/class/fa05/cs473g/lectures/18-simplex.pdf http://www.cs.uleth.ca/~holzmann/notes/lpdual.pdf http://www.cs.wise.edu/~swright/525/handouts/dualexample.pdf http://www.cse.ucsd.edu/~dasgupta/mcgrawhill/chap7.pdf http://www.e-optimization.com/directory/trailblazers/hoffman/ linear.programming.cfm http://www.eecs.harvard.edu/"parkes/cs286r/spring02/lectures/ class8.pdf http://www.hss.caltech.edu/"kcb/Notes/LP.pdf http://www.ici.ro/camo/books/rbb.htm http://www.ie.boun.edu.tr/course_pages/ie501/Ch81.pdf http://www.imada.sdu.dk/~jbj/DM85/lec4b.pdf 8.6 W e b material 119 http://www.math.Chalmers.se/Math/Grundutb/CTH/tma947/0506/ lecture9.pdf http://www.math.kth.se/optsyst/research/5B5749/13.pdf http://www.math.mtu.edu/~msgocken/ma5630spring2003/lectures/ineq/ ineq/node8.html http://www.math.mun.ca/~sharene/cs3753_F05/BookPartII.pdf http://www.math.niu.edu/"rusin/known-math/index/90-XX.html http://www.math.Washington.edu/"burke/crs/408f/notes/lpnotes/ http://www.maths.lse.ac.uk/Courses/MA208/notes6.pdf http://www.me.utexas.edu/"j ensen/ORMM/frontpage/tours/tour_lp.html http://www.me.utexas.edu/~jensen/ORMM/supplements/methods/lpmethod/ S3_dual.pdf http://www.mosek.com/homepages/e.d.andersen/papers/linopt.ps http://www.mpi-sb.mpg.de/~mehlhom/0pt imization/Linprog.ps http://www.opt imizat ion-online.org/DB_FILE/2003/04/646.pdf http://www.optimization-online.org/DB_HTML/2004/10/969.html http://www.personal.psu.edu/tmc7/tmclinks.html http://www.princeton.edu/~rvdb/542/lectures.html http://www.princeton.edu/"rvdb/LPbook/onlinebook.pdf http://www.scs.leeds.ac.uk/or21/QVERHEADS/sect5.pdf# search=' linear'/.20programming7,20duality'/,20theory' http://www.Stanford.edu/class/msande310/lecture03.pdf http://www.Stanford.edu/class/msande310/lecture06.pdf http://www.Stanford.edu/class/msande314/lecture02.pdf http://www.stats.ox.ac.uk/"yu/ http://www.statslab.cam.ac.uk/~rrwl/opt/index98.html http://www.tutor.ms.unimelb.edu.au/duality/duality.html http://www.twocw.net/mit/NR/rdonlyres/Electrical-Engineering-andComputer-Science/6-854JAdvanced-AlgorithmsFalll999/ 8C3707F7-2831-4984-83FB-BD7BF754AllE/0/notesl8.pdf http://www.utdallas.edu/~chandra/documents/6310.htm http://www.wisdom.weizmann.ac.il/~feige/algs04.html http://www.wisdom.weizmann.ac.il/~feige/lp02.html http://www2.imm.dtu.dk/courses/02711/DualLP.pdf http://www2.maths.unsw.edu.au/applied/reports/1999/amr99_2.pdf 9 N umber Systems In this chapter, we will review the basic concepts in real analysis: order relations, ordered sets and fields, construction and properties of the real and t he complex fields, and finally the theory of countable and uncountable sets t ogether with the cardinal numbers. The known sets of numbers that we will use in this chapter are • • • • • N: Z: Q: R: C: Natural Integer Rational Real Complex 9.1 Ordered Sets D efinition 9.1.1 Let S be a set. An order on S is a relation -< such that i) If x, y are any two elements of S, then one and only one of the following is true: x < y, x = y, y < x. ii) If x,y,z G S and x <y and y -< z, then x -< z. x <y i^ y <x. x < y means x -< y or x = y without specifying one. E xample 9.1.2 S = Q has an order; define x -< y ify — x is positive. D efinition 9.1.3 An ordered set is a set S on which there is an order. D efinition 9.1.4 Let S be an ordered set and 0 ^ E C S. E is • • bounded above if3bES^VxEE,x<b where b is an upper bound of E. bounded below if 3a G S 9 Vx £ E, a -< x where a is a lower bound of E. 122 • 9 Number Systems bounded if E is both bounded above and below. E xample 9.1.5 A = {p G Q : p >- 0 ,p 2 •< 2} is • • bounded above, b — 3 / 2 , 2 , . . . are upper bounds. bounded below, a = 0, — 1/2,... are lower bounds. D efinition 9.1.6 Let S be an ordered set and 0 ^ E C S be bounded above. Suppose 3 b G S 9 : i . b is an upper bound of E. 2. if b' & S and b' -< b then b' is not an upper bound of E. Equivalently, if b" is any upper bound of E if b" >- b. Then, b is called least upper bound (lub) or supremum ( sup ) of E and denoted by b = supE = lubE. Greatest lower bound (gib) or infimum (ml) of E is defined analogously. E xample 9.1.7 S = Q, E = {p G Q : p >• 0, p2 -< 2 } inf £ = 0, but E has no supremum in S = Q . Suppose po = sup E exists in Q . Then, either po G E or po <£ E. If po 6 E, 3q € E 3 po -< Q because E has no largest element; therefore, p is not an upper bound of E. IfPo & E, thenpo >- 0 because it is an upper bound andp\ >: 2 because po £ E. Then, either pg = 2 (not true because po 6 Q) or p^ > 2 (true), then po G B = {p G Q : p y 0, p2 >- 2 }. Then, 3q0 e B 3 q0 -< p0 (*) because B has no smallest element. Vp G E, p2 -< 2 -< q^ =? qo is an upper bound of E. Moreover, po < qo because lub Contradiction to (*). D efinition 9.1.8 Let S be an ordered set. We say that S has the least upper bound property if every nonempty subset of S which is bounded above has lub in S. E xample 9.1.9 S — Q does not have lub-property. T heorem 9.1.10 Let S be an ordered set with lub-property. Then, every nonempty subset of S which is bounded below has inf in S. Proof. Let B ^ 0, B c S be bounded below, L be the set of all lower bounds of B. T hen, L ^ 0 (because B is bounded below), y G B b e arbitrary, then for any l E i w e have x <y. So, y is an upper bound of L; i.e. all elements of B a re upper bounds of L => L is bounded above, a = s up L, a G S (because S h as lub property). Claim (i): a = inf B Proof (i): Show a is lower bound of B; i.e. show V:r G B, a <x. Assume that it is not true; i.e. 3a;o G B 9 a >- XQ. T hen, XQ is not an upper bound of a (because a = s up L) => XQ <£ B (because all elements of B are upper bounds of L). C ontradiction! (XQ G B). Therefore, a is a lower bound of B. 9.2 Fields Claim (ii): a is the greatest of the lower bounds. Proof (ii): Show if a -< /?, /3 £ 5 => /? is not a lower bound of B. ft fi L (because a -< (3); i.e. /? is not a lower bound of .B. Therefore, a = inf B. • 123 9.2 Fields Let us repeat Definition 2.1.1 for the sake of completeness. D efinition 9.2.1 A field is a set F / 0 with two operations, addition(+) and multiplication(.), which satisfy the following axioms: (A) Addition Axioms: (Al) Vx, y £ F, x + y £ F (closed under +) (A2) Vx,y £ F, x + y = y + x (commutative) (A3) Vx, y,z £ F, (x + y) + z — x + (y + z) (associative) (A4) 30 e F 9 Vx G F x + 0 = x (existence of ZERO element) (A5) \fx £ F, 3 an element -x £ F B x + ( - x ) = 0 (existence of INVERSE element) (M) Multiplication Axioms: (Ml) Vx,y £ F, x • y € F (closed under •) (M2) \/x,y £ F, x • y = y • x (commutative) (MS) Vx, y,z £ F, (x • y) • z — x • (y • z) (associative) (M4) 31 7^ 0 3 Vx £ F, 1-x = x (existence of UNIT element) (M5) Vx ^ 0 3 an element ^ g F 3 j i = 1 (existence of INVERSE (D) Distributive Law: Vx, y,z £ F, x • (y + z) = xy + xz N otation : x + (-«/) =x-y; x(-) = - ; x + {y + z) = (x + y) + z \VJ V element) x • x = x2; x + x = 2x; x(yz) = xyz, • • • E xample 9.2.2 F = Q with usual + and • is a field. E xample 9.2.3 Let F = {a, b, c} where a^b, a ^ c, b ^ c. +a bc a abc b bca c cab a aa ba ca bc aa bc cb F is a field with 0 = a, 1 — b. 124 9 Number Systems P roposition 9.2.4 In a field F, the following properties hold: (a) (b) (c) (d) (e) (f) (g) x + y = x + z=>y = z (cancelation law for addition). x + y = x =>• y — 0. x + y — 0=^y= —x. —{—x) = x. x y£ 0 and xy = xz => y = z (cancelation law for multiplication). x ^ 0 and xy = x => y — 1. x ^ 0 and xy = 1 =>- y = £ . (h) x^O, j£fc = x. (i) Vx e F, Ox = 0. (j) x ^ 0 and 2 / ^ 0 , £/ien xy ^ 0 (no zero divisors). (k) Wx,ye F, (-x){-y) = xy. D efinition 9.2.5 Let F be an ordered set and a field of F is an ordered field if i) x, y, z £ F and x~<y=>x + z-<y + z, ii) xyO, y y 0 =>• xy y 0. If xy 0, call x as positive, If x -< 0, call x as negative. E xample 9.2.6 S = Q is an ordered field. P roposition 9.2.7 Let F be an ordered field. Then, (a) x y 0 « • -x -< 0. (b) x y 0 and y < z => xy -< xz. (c) x -< 0 and y < z => xyy xz. (d) x =^ 0 => x2 y 0. /n particular 1 >- 0. fej(Mx^y^0^±^±. Proof. F is an ordered field. (a) Assume x >- 0 =>• x + (-x) y 0 + ( - x ) => 0 > > x. - x - < 0 = > - x + x-<0 + x=>0^:x. (b) Let x >- 0 and y < z => 0 -< z - y => 0 -< x(z - y) = xz — xy =$ xy < xz. (c) a; -< 0 and y < z =>• - x >- 0 and z - y >- 0 =>• —x(z — y)y0=> x(z — y)< 0 => xz -< xy. (d) x =£ 0 => x y 0 => (y = x in (b)) x 2 >- 0 or $ x - < 0 = » - x ^ 0 ( j / = - x ) => ( - x ) ( - x ) = x2 y 0. (e) Let x >- 0. Show \ y 0. If not, ± ^ 0 =* (x >- 0), x± = 1 < 0, C ontradiction! Assume 0~<x-<y=>±y0, ^ y 0, therefore (by (b)) ix± > - o l1 y x -< y J > => 1 y 1 < -. x U 9.3 The Real Field 125 R emark 9.2.8 C with usual + and • is a field. But it is not an ordered field. If x — i then i2 = — 1 y 0, hence property (d) does not hold. D efinition 9.2.9 Let F (with +, •) and F' (with ©, ®) be two fields. We say F is a subfield of F' if F C F' and two operations (B and © when restricted to F are + and •, respectively. That is, ifx,y € F => x®y — x + y, xQy = x-y. Then, we have OF — OF', and IF = I F ' Moreover, if F (with -<) and F' with (with -<') are ordered fields, then we say F is an ordered subfield of F' if F is a subfield of F' and for Vx € F with 0F -< x => 0F< <' x. 9 .3 T h e Real Field T heorem 9.3.1 (Existence & U niqueness) There is an ordered field R with lub property 3 Q is an ordered subfield of R. Moreover if R' is another such ordered field, then K and M! are "isomorphic": 3 a function <f>: R i-> R' 9 i) <j) is 1-1 and onto, ii) Vx, j / 6 R , <j>(x + y) = <f>(x) + (j>{y) and cj>{xy) = 4>{x)4>(y), Hi) Vz, € K with x y 0, we have (j)(x) >- 0). T heorem 9.3.2 ( A R C H I M E D E A N P R O P E R T Y ) x, y £ R and x y 0 => 3n G N (depending on x and y) 3 nx >- y. Proof. Suppose 3x,y e R with x >- 0 for which claim is not true. Then, Vn £ N, nx •< y. Let A = {nx : n £ N}. A is bounded above (by y). a = sup A € R, since R h as lub property, x y 0 => a — x < a, so a — a: is not a n upper bound for A. Therefore, 3m £ N 3 (a — x) ~< mx => a -< (m + l)x. C ontradiction (a = s up A). D T heorem 9.3.3 ( Q is dense i n R) Vx, y £ R with x -< y, 3p£Q3x~<p<y. Proof, x , y £ R , x <y ^ y — x y 0 (By Theorem 9.3.2) 3n £ N 3 n(y - x) y 1 => ny y 1 + n x. 3mi £ N 3 m i >- nx f- (y = n x, x = 1) in Theorem 9.3.2. Let A = {m £ Z : n x -< m}. A ^ 0, because mi £ A. A is bounded below. So A h as a smallest element mo, then nx -< mo =>• (mo — 1) •< nx. If not, nx -< mo — 1, but mo is the smallest element: Contradiction. => (mo — 1) •< nx -< mo =*• nx -< mo ^ nx + 1 -< ny => x -< ^^ -< j /. Let P = = * 6 Q. • 126 9 N u m b e r Systems T heorem 9.3.4 Va; G R, x y 0, Vn G N 3 a Mrogtte y G R, y X 0 9 yn = x. Proof. [Existence]: Given x X 0, n G N. Let £ = {t G R : i X 0 and tn < x). Claim 1: E ^ 0 Let £ = ^ y x 0, £ -< 1, t -< x; 0 -< t -< 1 => i" -< t (0 -< i -< 1 => 0 -< t2 •< t < 1 => . .. => 0 -< T -< i < 1). Also we have, t < x ^> tn < x\ therefore, £ = ^ j G E, Claim 2: E1 is bounded above If 1 + x is an upper bound of E. If not, 3t€E3tyl + x. In particular, t X 1 (because a; X 0) => f X /, X 1 + x X x; therefore, t ^ E: Contradiction! • y = sup E G R because R has lub property. y X 0, because (x X 0). Claim 3: yn = x If not, then either yn -< x or x < yn. We know the following: Let () ^ a < b. Then, bn - an = (b - a){bn~l + bn~2a + ••• + a'1"1) => (*) : bn -an -< {b - a)nbn-1. 0 J/" "< * => ^ T f f - r >- 0. Find n G R 3 0 -< ^ 1 and 0 -< * = # £ £ ? . (*): (y + /i)" - (j/)" -< hn(y + h)n -< hn(y + l ) " " 1 -< x - yn Therefore, (y + h)n -<x=>y + h(zE. B ut y + h X y => y is not an upper b ound of E, C ontradiction! ii) x -< yn. Let k = v, ~Xi > 0 and x -< y [because yn — x -< ny"~l). ~ Claim: y — k is an upper bound of E. Suppose not, 3t G E 3 t X y — k X 0. T hen, tn X (y - k)n => -tn -< ~{y - k)n => yn - tn •< yn - (y ~ k)n (*): yn -{y-'k)n -< knyn~x = yn-x => yn-tn -< yn~x => tn y x => t <£ E, C ontradiction! Therefore, y — k is an upper bound of E. However, y is lub of E, Contradiction! [ Uniqueness]: Suppose y y 0, y' y 0 arc two positive roots 3 y ^ y' and yn = x — ( ?/)"• W ithout loss of generality, we may assume that , y' y y y 0, (because y ^ ?/) =^ 2/" ^ (?/)"> Contradiction! Thus, y is unique. O D efinition 9.3.5 Real numbers which are not rational are called irrational numbers. E xample 9.3.6 \/2 is an irrational number. C orollary 9.3.7 Let a. y 0, 6 x 0 and n&N. Then, ( ab) 1 /" = al/nbl/n. 9.4 The Complex Field 127 Proof. Let a = a 1 /", /? = b1'71 => an = a, (3n = b => (a/?)" = an(3n = ab y 0 a nd n" 1 root is unique => (ab)1/™ = a/3. D D efinition 9.3.8 (Extended real numbers) R U { +oo, - o o } 3 preserve the order in R and Vx £ R, —oo -< x -< oo. R U { +oo, —oo} is an ordered set and every non-empty subset has supremum/infimum in R U { +oo, —oo}. In R U { +oo, - o o } , we make the following conventions: i) For x € R, x + oo = -t-oo, x — oo = —oo, ii) If x -< 0, we have x • ( +oo) = —oo, x • (—oo) = +oo, iii) 0 • ( +oo), 0 • (—oo) are undefined. 9.4 T h e Complex Field Let C be the set of all ordered pairs (a, b) of real numbers. We say (a, b) — (c, d) if and only if a — c and b = d. Let x = (a,b), y = (c, d). Define x + y = (a + c,b + d), xy = [ac — bd, ad + be]. Under these operations C is a field with (0,0) being the zero element, and ( 1,0) being the multiplicative unit. Define 4>: R H-> C by <j>{a) = ( a,0), then (j> is 1-1. </>(a + b) = (a + 6,0) = (a, 0) + (b, 0) = 0(a) + </>(&). 0(a&) = (a&,0) = (a,0)(6,0) = 4>{a)<f>{b). Therefore, R can be identified by means of <j> with a subset of C in such a way that addition and multiplication are preserved. This identification gives us the real field as a subfield of the complex field. Let i = ( 0,1) => i2 = ( 0,1)(0,1) = ( -1,0) = < £(-l), i-e. i2 corresponds to the real — 1. Let us introduce some notation. <l>(a) = ( a,0) = a => i2 = 0 ( - l ) = - 1 , also if (a,b) € C, a + ib = (a,b). Hence, C = {a + ib: a,b£R}. If z = a + ib G C, we define 1 = a — ib (conjugate of z), Z ~\~ ~Z Z — ~Z a = Re{z) = - y - , b = Im(z) = -^r-. If z,w G C ^> z + w = J + w, Jw ='zw. If z € C => zz = a2 + b2 y 0, we define \z\ = \ / 5 i = \ / o 2 + 62. 128 9 Number Systems P roposition 9.4.1 Let z,w G C. Then, (a) z ^ 0 => \z\ > 0 and |0| = 0. W 11 = N* (c,) |^u)| = |z||i«|. (dj |ite(z)| ^ | z|, | Im(*)| < \z\. (e) \z + w\ < \z\ + \w\, [Triangle inequality]. Proof. T he first three is trivial. Then, (d) Let z = a + ib \Im(z)\ = \b\ = y/P < Va2 + b2 = \z\. (e) \z + w\2 = (z + w)(z + w) = \z\2 + zw + zw + \w\2 < (\z\ + \w\)2 ~zw + zw = 2Re(zw) < \2Re(zw)\ •< 2\zw\ => \z + w\ •< \z\ + \w\. Take positive square roots of both sides, i.e. if a >z 0, b >z 0 a nd a2 < b2 =4- a •< b. If not, b < a =$> b2 ^ ab, ba < a2 =>• b2 ^ a2. C ontradiction! • T heorem 9.4.2 (Schwartz Inequality) Let aj,bj Then, £ C, j = l,...,n. J2aibi i=i -< Proof. ByO.HB = 0 t hen bj = 0 V? => LHS = 0; therefore, 0 ^ 0 . Assume B y 0 => n n 0 r< ] T \Baj - Cbjf = Yl(Bai j=i - Cb j)(BN ~ Cb~) +y i=i = Y/B2\ai\2-JlBCaibi-Y,CBbM 3= 1 3= 1 3= 1 £\c\2\bi\ 3= 1 = B A - B\C\ 2 2 - CBC+ \C\ B = B(AB - | C| 2 ). 2 T hus, . 4 5 >: |C| 2 , since B ^ 0. D 9 .5 Euclidean Space D efinition 9.5.1 Let k £ N, we define Rk as the set of all ordered Ittuples x = (x\,... ,Xk) of real numbers x\,...,XkWe define {x + y) = (xi + 2/ii • • • )#*; + Vk)- If a £ R, ax — (ax\,... ,axk). This way Rk becomes a vector space over K. We define an inner product inRk byx-y = 5 Z i = 1 £J2/J. AndVx £ Rk, x-x X 0. We define the norm of x £Rk by \\x\\ = \Jx • x = y ^ r l = 1 x2^. 9.6 Countable and Uncountable Sets 129 D efinition 9.5.2 An equivalence relation in X is a binary relation (where ~ means equivalent) with the following properties: (a) Vx G X, x ~ x (reflexibility). (b) x ~ y => y ~ x (symmetry). (c) x ~ y, y ~ z => x ~ z (transitivity). D efinition 9.5.3 7 / ~ is an equivalence relation in X, we define the equivalence class of any x G X as the following set: [x] = {y G X : x ~ y}. R emark 9.5.4 7 / ~ is an equivalence relation in X, then the collection of all equivalence classes forms a partition of X; and conversely, given any partition of X there is an equivalence relation in X such that equivalence classes are the sets in the partition. R emark 9.5.5 Let C be any collection of nonempty sets. For X,Y G C, define X ~ Y (X and Y are numerically equivalent) if there exists a oneto one and onto function f : X i-> Y (or f~l : Y H-> X). Then, ~ is an equivalence relation in C. 9.6 Countable and Uncountable Sets D efinition 9.6.1 Let Jn = { 1,2,... ,n}, n = 1 ,2,.... Let X ^ 0. We say i) X is finite if 3 n G N, X ~ Jn. ii) X is infinite if X is not finite. Hi) X is countable if X ~ N (i.e. 3 / : N 4 l , 1-1 onto, or3g:X^N, 1-1 onto), iv) X is uncountable if X is not finite and not countable, v) X is at most countable if X is finite or countable. E xample 9.6.2 X = N is countable. Let f : N n - N be the identity function. E xample 9.6.3 X = Z is countable. Define f : N H> Z as § , if n is even; /(»)=( n-l 2 , if n is odd. E xample 9.6.4 Q + is countable. Let r G Q + , then r = — where m,n G N. List elements of Q+ in this order as in Table 9.1. If we apply the counting schema given in Figure 9.1, we get the sequence ! I2 I3 i ^ 3 '2' '3' ' 4 ' 3 ' 2 ' '"" 1 30 9 N u m b e r Systems Define f:N^Q+, / (l) = 1, /(2) = | , /(3) = 2, /(4) = | , Cantor's Counting Schema Another Counting Scheme F i g . 9 . 1 . C o u n t i n g schema for r a t i o n a l ' s T a b l e 9 . 1 . L ist of rational n u m b e r s n m 12 345 1 1/1 1/2 1/3 1/4 1/5 • • • 2 2/1 2/2 2/3 2/4 2/5 • • • 3 3/1 3/2 3/3 3/4 3/5 • • • • 4 4/1 4/2 4/3 4/4 4/5 • • • 5 5/1 5/2 5/3 5/4 5/5 • • E xample 9.6.5 Q is countable. Since < Q is countable, the elements of Q + can be listed as a sequence {2:1,0:2,2:3, • • . Then, Q~ - {q : q -< 0} can be listed as {—x±, —#2, —£3,.. •}• Q = 0 Xi —Xi X2 —X2 X3 - X3 . . . N = 1 2 3 4 5 6 7 ... / : N i-> Q can be defined in this way. XR, if n is even / ( n ) — I — Xn^, if n is odd 2 0, ifn=l. 9.6 Countable and Uncountable Sets 131 P roposition 9 .6.6 If s = {xi,i £ 1} is a countable class of countable sets, then Ui£iXi is also countable. That is, countable union of countable sets are countable. Proof. We have / : N 4 / , 1-1, onto. Let Y„ — Xf(ny T he elements of Yn can b e listed a s a sequence. Yn = {X^,X^, • • •} Vra. Use the C antor's counting scheme for the union. Another counting schema is given in Figure 9.1. • E xample 9 .6.7 X = [0,1) is not countable. Every x € [0,1) has a binary expansion x = 0.aia2«3 • • • where an = Suppose [0,1) is countable. Then, its elements can be listed as a sequence {X1, X2, X3,...}. Consider their binary expansions X1 = Q.a\a\a\ ... X2 = X= 3 ? O.ajalal... O.afalal... Tpfn Letai _/0,i/ai = l _ f 0, */ al = 1 -\l,ifal=0'a2-\l,ifal=0'a3-\l,ifa3 _/0,t/oi = l = 0'--- Let x = 0 .aia 2 a 3 . . . e [0,1). But this number is not contained in the list {X1, X2, x is different from X1 by the first digit after 0; x is different from X2 by the second digit after 0; x is different from X3 by the third digit after 0; X3,...} Therefore, x ^ Xn, Vn; since x and Xn differ in the nth digit after zero. So, X = [0,1) is not countable. E xample 9 .6.8 X = ( 0,1) is not countable. Since X — [0,1) is not countable, excluding a countable number of elements (just zero) does not change uncountability. Thus, X = ( 0,1) is uncountable. E xample 9 .6.9 For any open interval (a, b) we have M ) ~ ( 0 , 1 ) f : (a,b) ^ Refer to Figure 9.2. E xample 9.6.10 X — K is not countable. Since R ~ (—1,1), by projection R is not countable [because (0,1) is not countable]. One way of showing 1-1 correspondence between any open interval and (0,1) is illustrated in Figure 9.3. (0,1). 132 9 Number Systems 1 !• ^ %- -%f y >y y ^ F ig. 9 .2. Uncountability equivalence of (a,b) and (0,1) -1 +1 F ig. 9 .3. The correspondence between (-1,1) a nd '. E x a m p l e 9 . 6 . 1 1 / : R i-» ( — f , f ) , / ( x ) = arctan(rr) is a i - i dence, i.e. fix) is 1-1 and onto. Refer to Figure 9.4- correspon- * = arctan(x) ^^-^ 7l<2 j, *arctan(x) x ~~rJ2 F ig. 9 .4. The correspondence between (-f , f ) and P r o p o s i t i o n 9 . 6 . 1 2 / / (a, b) is any open interval, (0,l)~(a,&)~R~[0,l)- then Proof. 3 / : ( 0 , l ) ^ [ 0 , l ) i s l - l ( / ( a : ) = a ;). 3g : [ 0,1) ^ R is 1-1 {g(x) = x). 1:R4 ( 0,1) is 1-1 and onto (f(x) [ 0,1) 4 M 4 (0,1) is 1-1. • = x). B y Cantor-Schruder-Bernstein Theorem [0,1) ~ (0,1). 9.6 Problems 133 D efinition 9.6.13 Roughly speaking, the cardinality of a set (or cardinal number of a set) is the number of elements in this set. IfX = 9, Card(X) = 0, ffX~Jn = {l,2,...,n}, Card(X) = n, If X ~ N (i.e. countable), Card(X) = Ko (aleph zero), IfX~R, Card(X) = Nx (aleph one). D efinition 9.6.14 Let m and n be two cardinal numbers We say m -< n if there are two sets X and Y 3 Card(X) = m, Card(Y) = n. R emark 9.6.15 The list of cardinal numbers: 0-<l-<2^----<n-<----<N0^:tt1=c. R emark 9 .6.16 Question: 3? a cardinal number between Ko and Hi ? The answer is still not known. Conjecture: The answer is no! Question: Is there a cardinal number bigger than Hi ? The answer is yes. Consider P(R) : the set of all subsets of R (power set of R). Hi = Card(R) -< Card(P(R)). We know if Card(X) = n, then Card(P(X)) = 2n. Analogously Card(P(N)) = 2*° = Hx. Then, we can say that Card(P(R)) = 2*1 = H2. P roblems 9 .1. Let A b e a non-empty subset of R which is bounded below. Define — A — {-x : x G A}. Show that inf A = - sup(-A). 9 .2. Let b y 1. Prove the following: a) Vm,n G Z with nyQ, (bm)lln = (bl'n)m. b) Vm,n G Z with n y 0, (bm)n = bmn = (bn)m. c) Vn G Z with n y 0, ll'n = 1. d) V n , 3 £ Z with n,qy0, b1/^ = ( fe 1 /") 1 ^ = (blli)lln. q e) \tp,q€l bP+ = &>&. 9 .3. Do the following: 11 a) Let m,n,p,q G Z with n -< 0, q y 0 and r = ^ = E. Show that ( 6" ) /" = (ypy/i using the above properties. b) Prove that bT+s = brbs if r and s are rational. r c) Let x G R. Define B(x) = {bl : t G Q, t < x} Show that if r G Q, 6 = sup B(r). d) Show that fox+y = bxW Vx, y G R. 9 .4. Fix by 1 and y y 0. Show the following: a) VnGN, bn -1 yn(b-l). b) (b - 1) h n{bl'n - 1). Hint: Vn G N, ft1/" y 1 holds. So replace (b y 1) 134 9 N u m b e r Systems above by bl//'n >- 1. c) If t >• 1 and n >- f ^ j , then 6 1 /" x i. d) If w 3 6"' X y, then bw+l/n -< y for sufficiently large n. e) If fe"J x ?y, then fr'""1/" > . ^ for sufficiently large n. _ f) Let 71 = {u; e K : 61U X t /}. Show that x = sup/1 satisfies 6* = y. g) Prove that x above is unique. 9 .5. Let F be an ordered field. Prove the following: a) x, y & F and x2 + y2 = 0 =^- x = 0 and y — 0. b) xi, x'2, • • •, .x'n G F and .xf + • • • + x^ = 0 => xi = x-i = • • • = x „ = 0. 9 .6. Let m be a fixed integer. For a, b 6 Z, define a ~ 6 if a — 6 is divisible by m, i.e. there is an integer fc such that a — b = mk. a) Show that ~ is an equivalence relation in Z. b) Describe the equivalence classes and state the number of distinct equivalence classes. 9 .7. Do the following: a) Let X = R, and x ~ y if x £ [0,1] and y £ [0,1]. Show that ~ is symmetric and transitive, but not reflexive. b) Let. X / I and ~ is a relation in X. T he following seems to be a proof of the statement that if this relation is symmetric and transitive, then it is necessarily reflexive: x ~ y => y ~ x, x ~ y and y ~ x ==> x ~ x; therefore, x ~ x, Vx £ X. In view of part a), this cannot be a valid proof. W hat is the flaw in the reasoning? 9 .8. Prove the following: a) If Xy, X2, • • •, Xn are countable sets, then X — II]l=:1Xi is also countable. b) Every countable set is numerically equivalent to a proper subset of itself. c) Let X and Y be non-empty sets and / : X 1 > Y be an onto function. Prove — t hat if X is countable then Y is at most countable. W eb material http://129.118.33.l/~pearce/courses/5364/notes_2003-03-31.pdf http://alpha.fdu.edu/~mayans/core/real_numbers.html http://comet.lehman.cuny.edu/keenl/realnosnotes.pdf http://en.wikipedia.org/wiki/Complex_number http://en.wikipedia.org/wiki/Countable http://en.wikipedia.org/wiki/Field_(mathematics) http://en.wikipedia.org/wiki/Numeral_system http://en.wikipedia.org/wiki/Real_number 9.7 W e b material 135 http://en.wikipedia.org/wiki/Real_numbers http://en.wikipedia.org/wiki/Uncountable_set http://eom.springer.de/f/f040090.htm http://eom.springer.de/U/u095130.htm http://kr.cs.ait.ac.th/~radok/math/mat5/algebra21.htm http://math.berkeley.edu/~benjamin/741ecture38s05.pdf http://mathforum.org/alejandre/numerals.html http://mathworld.wolfram.com/CountablyInfinite.html http://numbersorg.com/Algebra/ http://pirate.shu.edu/proj ects/reals/infinity/uncntble.html http://planetmath.org/encyclopedia/MathbbR.html http://planetmath.org/encyclopedia/Real.html http://planetmath.org/encyclopedia/Uncountable.html http://plato.Stanford.edu/entries/set-theory/ http://www-db.Stanford.edu/pub/cstr/reports/cs/tr/67/75/ CS-TR-67-75.pdf http://www.absoluteastronomy.com/c/countable_set http://www.answers.com/topic/complex-number http://www.cse.cuhk.edu.hk/~csc3640/tutonotes/tuto3.ppt http://www.csie.nctu.edu.tw/~myuhsieh/dmath/Module-4.5Countability.ppt http://www.cut-the-knot.org/do_you_know/few_words.shtml http://www.dpmms.cam.ac.uk/~wtglO/countability.html http://www.eecs.umich.edu/~aey/eecs501/lectures/count.pdf http://www.faqs.org/docs/sp/sp-121.html http://www.faqs.org/docs/sp/sp-122.html http://www.introducingmathematics.com/settheoryone/01.html http://www.jcu.edu/math/vignettes/infinity.htm http://www.math.brown.edu/~sjmiller/l/CountableAlgTran.pdf http://www.math.niu.edu/~beachy/aaol/contents.html http://www.math.niu.edu/~rusin/known-math/index/11-XX.html http://www.math.niu.edu/~rusin/known-math/index/12-XX.html http://www.math.toronto.edu/murnaghan/courses/mat240/field.pdf http://www.math.ucdavis.edu/~emsilvia/mathl27/chapter1.pdf http://www.math.ucdavis.edu/"emsilvia/mathl27/chapter2.pdf http://www.math.uic.edu/"lewis/laslOO/uncount.html http://www.math.uiuc.edu/~r-ash/Algebra/Chapter3.pdf http://www.math.umn.edu/~garrett/m/intro_algebra/notes.pdf http://www.math.unl.edu/~webnotes/classes/classAppA/classAppA.htm http://www.math.uvic.ca/faculty/gmacgill/guide/cardinality.pdf http://www.math.uvic.ca/faculty/gmacgill/M222F03/Countable.pdf http://www.math.vanderbilt.edu/"schectex/courses/thereals/ http://www.math.wise.edu/~ram/math541/ http://www.mathreference.com/set-card,cable.html http://www.mes.vuw.ac.nz/courses/MATH114/2006FY/Notes/11.pdf http://www.msc.uky.edu/ken/mal09/lectures/real.htm http://www.swarthmore.edu/NatSci/wstromql/stat53/CountableSets.doc http://www.topology.org/tex/conc/dgchaps.html http://www.trillia.com/zakon-analysisl-index.html 10 B asic Topology In this chapter, basic notions in general topology will be defined and the related theorems will be stated. This includes the following: metric spaces, open and closed sets, interior and closure, neighborhood and closeness, compactness and connectedness. 1 0.1 M e t r i c Spaces In Rfc, we have the notion of distance: I f p = (xi,x2,...,xk)T,q = (yi,V2,---,yk)T, P,Q € Rfc, then h (xk - Vk)2 dz(p,q) = y/{xi - yi)2 + ( x 2 - y 2 ) 2 + D efinition Let 10.1.1 X 7^ 0 be a set. Suppose there is a function d : X X X => E_)_ = [0,00) with the following properties: i) d(p, q) = 0 <=> p = q; ii) d(p,q) = d(q,p), \/p,q; Hi) d(p,q) < d(p,r) + d(r,q), \/p,q,r [triangle inequality]. Then, d is called a metric (or distance function) and the pair (X, d) is called a metric space. E xample 10.1.2 Let X ^ 0 be any set. For p,q e X define d(va) =\ 0 1' i, d(P q) ' is called the discrete metric. ifp¥:q ifp = q D efinition 10.1.3 Let S be any fixed nonempty set. A function f : S •->• R is called bounded if f(S) is a bounded subset o /R. 138 10 Basic Topology E xample 10.1.4 Is / : R H-» R, f(s) = s2 bounded? (Exercise!). / : R i-> R, f(s) = a rctan(s) — t an _ 1 (s) is bounded. See Figure 9.4Definition 10.1.5 Let X = B{S) = all bounded functions f : S i-> R. For f,g £ B(S), we define the distance asd(f,g) — s up{|/(s) — g(s)\ : s £ S}. P roposition 10.1.6 d(f,g) > 0 is a metric, Vf,g £X = B(S). Proof, by proving axioms of a metric: (i) (=») if d(f,g) = 0 =» \f(s)-g(s)\ = 0 , V s e 5 ^ f(s) = g(s), Vs £ S => f = g. iff = g^d(f,g)=0. (ii) t rivial, (iii) Proposition 10.1.7 Let A ^ 0, B ^ 0 be subsets ofR. Define A + B = {a + b: a £ A, b £ B} . If A and B are bounded above then A + B is bounded above and sup(A + B) < sup A + sup B. Proof. Let x = sup A, y = sup B. Given c e A + B, t hen 3a £ A, b £ B 3 c = a + b. T hen, c = a + b < x + y. Moreover, sup(A + B) < x + y. • P roposition 10.1.8 Let C, D be nonempty subsets o /R, let D be bounded above. Suppose Vc £ C, 3d £ D 3 c < d. Then, C is also bounded above and s up C < sup D. Proof. Given c £ C, 3d £ D 3 c < d. So, Vc £ C, c < y = sup£>. Hence, y is an upper bound for C. Therefore, sup C < sup D. D Triangular Inequality: Let f,g,h £ B(S). C = { |/(s) - g(s)\ :s£S}, t hen d(f,g) = supC. A = {\f{s) - h(s)\ :s£S}, t hen d(f, h) = s up A B = {\h\s) - g(s)\ :s£S}, t hen d(h,g) = supB. Given x £ C , then 3s £ S 3 x — | /(s) - g(s)\ x=\f(s)-g(s)\ = \f(s)-h(s) + h(s)-g(s)\ < \f(s)-h(S)\ + • € \h(s)-g(s)\ => sup C < sup(A + B) < sup A + sup B. E xample 10.1.9 Let X = Rk, P = (Xl,...,xk)T and Q = {yi,...,yk)T fe R. di(p,q) = \xi-yi\-\ h \xk -yk\ : h metric. 2 <kip, q) = [(xi - 2/i) H + (xfe - yk)2}1/2 • h metric. doo(p,q) =max{\x1 -yx\,...,\xk - yk\) : l^ metric. See Figure 10.1. 10.1 Metric Spaces 139 6^ -d,— 'Oinfty • P <... F ig. 10.1. Example 10.1.9 D efinition 10.1.10 Let (X,d) be a metric space, p 6 X, r > 0. Br(p) = {q S X : d(p, q) < r} open ball centered at p of radius r. Br\p] — {q £ X : d(p, q) < r} closed ball centered at p of radius r. E xample 10.1.11 X = M2, d = d2. See Figure 10.2. CLOSED BALL OPEN BALL Br[P] / jy " P Br(P) s ^, „' F ig. 10.2. Example 10.1.11 E xample 10.1.12 Let us have X ^ 0, and the discrete metric. MP) {p} , if r < 1 f {p}, if r < 1 ={{p},if r = l Br\p] ={ X, if r = 1 X, if r > 1 { X, if r>l E xample 1 0.1.13 X = B c (a, b) = {/ : (a, b) >-> R : / is bounded} f,9EX^ d(f,g) = s up{|/(s) - g(s)\ : a € (a, 6)} Let f € X,r > 0, Br(f) is the set of all functions g whose graph lie within the dashed envelope in Figure 10.3. E xample 10.1.14 X = W2 with d\ metric: d\{p,q) = 12/i -xi\ See Figure 10.4+ |s/2 -x2\. 140 10 Basic Topology F ig. 10.3. Example 10.1.13 E xample 10.1.15 X = K2 with rfoo metric: doo(p,q) = m ax{|t/i -xi\,\y2 - x2\} • D efinition 10.1.16 A subset E 7^ 0 of a vector space V is convex if tp + (1 - t)q G E whenever p,q G E and t € [0,1]. P roposition 10.1.17 X — M.k withd2, d\ or d^ metric. Then, every (open) ball Br(p) is convex. Proof. Using c ^ metric: F ix Br{p). Let u,v £ Br(p),0 < t < 1. Show that tu + (1 - t)v G Br(p) : L e t p = (pi,...,Pk), u = (ui,...,uk), v = (vi,...,vk). T hen, doo(tu+ (1 - t)v,p) = d00(tu + (1 - t)v,tp+ (1 - t)p) = max{|<Mi + (1 - t)vi - tpi - (1 - t)pi\}i=1 = \tuj + (1 - t)vj - tpj - (1 - *)pj-| = \t{uj - pj) + (1 - i)(u,- - p7-)l < | <||uj-pj| + | l-t||u J --pj| = td00(u,p) + (l-t)d00(u,p) D < tr+(l-t)r = r. D efinition 10.1.18 Let (X,d) be a metric space, E C X. A point p G E is called an interior point of E if 3r > 0 3 Br(p) C E. The set of all interior points of E is denoted by intE or E° and is called the interior of E (intEcE). 1 / 1/ 1/ / X S X X ^ N I / \ > Y K ivs %. X X r Xl X \' P r \ I 1 \ Rectilinear Euclidean Tchebycheff's F ig. 10.4. Example 10.1.14 10.1 Metric Spaces E xample 10.1.19 See Figure 10.5. q G intE but p g" intE. 141 F ig. 10.5. Example 10.1.19 E xample 10.1.20 Let X be any set with at least two elements, with the discrete metric: Letpe X, E = {p}. Then, intE = E, r < 1 => Br(p) = p c E => p £ intE. E xample 10.1.21 Let X = R2 with d2 metric. See Figure 10.6. T F ig. 10.6. Example 10.1.21 E = {p = (x,y) £ R2 : 1 < x2 + y2 < 4} => intE = {p = (x,y) e M2 : 1 < x2 + y2 < 4 } . D efinition 10.1.22 E is said to be open set if intE = E, i.e. \fpeE,3r>0 3 Br(p) C E. E xample 10.1.23 In K2, E = {p = (x,y) £ E2 : 1 < x2 + y2 < 4 } is open. R emark 10.1.24 By convention, E = 0, E = X are open sets. D efinition 10.1.25 Let p £ X. A subset N of X is called a neighborhood of p if p G intN. 142 10 Basic Topology \ ,--/ i i P i i i i . / Fig. 10.7. Example 10.1.26 E xample 10.1.26 N is a neighborhood of P but it is not neighborhood of Q. See Figure 10.7. D efinition 10.1.27 A point p S X is called a limit point (or accumulation point) (or cluster point) of the set E C X if every neighborhood N of p contains q of E 3 q ^ p. i.e. V neighborhood N of p, 3q € E f\ N, q ^ p. Equivalent^, Vr > 0, 3<7 G E D Br(p) 9 q ^ p. E xample 10.1.28 £ = { p = ( i , j / ) € R 2 : K i 2 + t / 2 < 4 } u { ( 3 , 0 ) } . Limit points of E are all points p = (x,y) 3 1 < x2 + y2 < 4. See Figure 10.8. Fig. 10.8. Example 10.1.28 D efinition 10.1.29 A point p € E is called an isolated point of E if p is not a limit point of E; i.e. 3 r > 0 3 Br{p) n E = p. E xample 10.1.30 X = R, d = dx: E= i { \W~)> 0 is the only limit point of E. \fp £ E are all isolated points. D efinition 10.1.31 E is closed if every limit point of E belongs to E. E xample 10.1.32 See Figure 10.9. 10.1 Metric Spaces CLOSED OPEN Not CLOSED Not OPEN 143 (o) (d) (o) F ig. 1 0.9. Example 10.1.32 D efinition 10.1.33 E is perfect if it is closed and every point of E is a limit point of E; i.e. if E is closed and has no isolated points. E is bounded if 3M > 0 9 Vp, q G E d\p, q] < M. E is dense in X if every point of X is either a point of E or a limit point of E. E xample 10.1.34 X = K, E — N is unbounded. Suppose it is bounded. Then, 3M > 0 B Vx, y G N, \x - y\ < M. Let n G N be B n > M + 1 => | l - n | = r a - l < M - » n < M + l . Contradiction! E xample 10.1.35 X = M, E = Q (Q is dense in R ; i.e. given x G K either x G Q or x is a limit point ofQ). Let i £ l , if x G Q , we are done. If x ^ Q, we will show that x is a limit point of Q: Given r > 0, Br{x) = (x — r, x + r). Then, 3y € Q 3 x — r<y<x + r=> y G Br(x) n Q and y ^x => x €R, y G Q. Let us i ntroduce t he following notation: E': set of all limit points of E. E — E U E', E is called t he closure of E. pE E < * W > 0, Br(p) n £ / 0. = P roposition 10.1.36 Every open ball Br(p) is an open set. Proof. Let q G Br(p), we will show that 3s > 0 3 Bs(q) C Br(p): q G Br(p) => d(q,p) < r , let s = r - d(q,p) > 0. Let z G Bs(q), d(z,p) < d{z,q) + d{q,p) < s + d(q,p) =r =>• z G Br(p). • T heorem 10.1.37 p is a limit point of E if and only if every neighborhood N of p contains infinitely many points of E. Proof. (<=): trivial. (=>): Let p be the limit point of E. Let N be an a rbitrary neighborhood of p. T hen, 3r > 0 B Br(p) C N. Since Br(p) is a neighborhood of p 144 10 Basic Topology 3qx G Br(p) r\E3qi^p^ 3q2 eBr(p)DE 9 d(q,p) = n > 0. q2^p. T hen, q2^qi- Since q2 ^ p, r2 = d(q2,p) > 0. 3q3 e Br2(p) D E 9 q3^p^q2^qi;--.D C orollary 10.1.38 7 / F is a /mite set, E' = 0. T heorem 10.1.39 E is open if and only if Ec is closed. Proof. (=>): Let E b e open, Let p be a limit point of Ec. Show p G Ec. Suppose not: p G E =>• 3 r > 0 9 -Br(p) C E [because E is open] (*) Since p is a limit point of Ec, for every neighborhood N of p, N D Ec ^ 0. In p articular (by t aking N = Br(p)), Br(p) OEc ^%, Contradiction to (*). (<=): Assume Ec is closed. Show E is open; i.e. Vp G E, 3r > 0 9 F r (p) C E. Let p G E => p £ F c => p is not a limit point of F c . So 3r > 0 9 Br(p)f)Ec 4 does n ot contain any q / p (p e ither). => _Br(p) ( l Ec = 0 =>• B r (p) C E. D * ~ T heorem 10.1.40 Let E C X, then (a) E is closed. (b) E-E-^Eis closed. (c) E is the smallest closed set which contains E; i.e. if F is closed and E C F => E CF. Proof. EcX. ( a): ( F ) c is open. Let p G ( F ) c => p <£ E_=> 3r > 0 B Br{p) n F = 0 => Br(p) C ( F ) c . Show that Br{p) C {E)c: If it is not t rue 3q G Br(p) and q $ (E)c =$• q€ Ec. F ind s > 0 3 £ s (g) C £ r (p). Then B s (g) n E ^ 0 => B r (g) n F ^ 0. Contradiction, ( b): (=>): Immediate from ( a). («=): F is closed. Show E = E, i.e. F C F . Let p G F = F U F ' , if p G F , we are done. If p G F ' =>• p G F (because F is closed), (c): Let F be closed, E C F. Show that E C F . Let p G E = E U E', if p G F => p G F . If p G F ' we have t o show that p G F ' : Given r > 0, show B r (p) fl F contains a point q ^ p. Since p G F ' , S r (p) n F contains a point < ^ p . T hen, q G F r (p) n F (because F c F ) . / So, p G F ' => p G F (because F is closed). • * 10.1 Metric Spaces L et (X,d) b e a m etric space, then 145 1. T h e u nion of a f inite collection of o pen sets is o pen. 2. T h e i ntersection of a f inite collection of o pen sets is o pen ( not t rue for i nfinite). 3. T h e i ntersection of any c ollection of c losed sets is c losed. 4. T h e u nion of a finite collection of c losed sets is c losed ( not n ecessarily true for infinite). 5. £ is o pen < > Ec is c losed. = 6. E is c losed < > E = E. = 7. £ is the s mallest closed s et c ontaining E. 8. intE is the l argest open s et c ontained in E ( i.e. if A C E a nd A is o pen t hen A C intE). E x a m p l e 1 0 . 1 . 4 1 Intersection of infinitely open, X = R , d(x,y) = \x - y\: Let An = f T = i A-. = [ 0,1]. / / 0 < .x < 1 ( A m i G ( - £ , Let x G 0 ^ = 1 ^ n - s / t o w * * o i 0 < a; < 1: If not, x < 0 or x > 1. If x > 1, 3n E N x < 0 is similar. many open sets needs not to be ( - ^ , n ± i ) , n = 1 , 2 , . . . . Then, ^ ) = An, Vn => a; G n ~ = 1 A « 3 1< ^ < x , j ; ^ An. Case P r o p o s i t i o n 1 0 . 1 . 4 2 L ei 0 ^ £ C R 6e bounded above. Then, sup E G £ . Proof, y = s up £ , s how t h a t Vr > 0, Br(y) n L / 0: S ince y — r<y=>y —r is n ot u pper bound of E. Bx G £ 9 y >_ x > y — r =$• x € (y — r,y -\-r)C\E => Br{y)C\E^%. 0 L et ( X, rf) be a m etric space a nd I / 7 C X , t hen 7 i s a m etric space in i ts own r ight with t he s ame distance function d. I n t his case, (Y, d) is a s ubspace of (X, d). If £ C Y, E m ay be o pen in (Y, d) b ut not o pen in (X, d). E x a m p l e 1 0 . 1 . 4 3 X = R 2 , Y = R , £ = (a, 6 ): WTierc considered in R , £ is o pen whereas E is not open in R 2 , as seen in Figure 10.10. •** ~ •% Hr^m a N\ c -^// b F ig. 10.10. E xample 10.1.43 D e f i n i t i o n 1 0 . 1 . 4 4 Let E C Y C X. We say E is open (respectively closed) relative to Y if E is open (respectively closed) as a subset of the metric space (Y,d). 146 10 Basic Topology E is open relative to Y •» Vp e E 3r > 0 3 Br(p) r\Y CE. E is closed relative to Y •£> Y \ E = Y n Ec is open relative to y . T heorem 10.1.45 LetX CY C E. Then, (a) E is open relative to Y ^3 an open set F in X 3 E = F f]Y. (b) E is closed relative to Y <&3 a closed set F in X 3 E — F <~)Y. Proof. X C Y C E. (a) (=»): Let E b e open relative to Y. T hen, Vp G E 3rp > 0 3 Brp{p) n Y C E. Let F = {JpeE Brp(p). F is open in X. \J[Brp(p)nY}cE p€E FHYCE Conversely, q G E, t hen q e Brq(q) C F, (<=)• qeEcY=^qeF(lY^EcFC]Y E = FDY where F is open in X. Given p G E =>• p G F. Since F is open, 3r > 0 3 Br{p) C F. Br(p)f)Y CFDY = E. (b) (=»): £? is closed relative to Y =>• Y \ £ is open relative to Y. T hen, 3F G X open in X 9 Y\E = FC\Y. E = Y\(Y\E) = Y\(FnY) = Yn(Fr\Y)c = YnFcU<D = YDFc. c F closed in X. (<=) = £ = F n 7 where F is closed in X. Y\E = Yn{Fr\Y)c = YHFC (Fc open in X) = > Y \ £ is open relative toy. => E is closed relative to Y. D 1 0.2 Compact Sets D efinition 10.2.1 Let (X, d) be a metric space, E C X be a nonempty subset of X. An open cover of E is a collection of open sets {d : i G / } in X 3 E C UiGi. 10.2 Compact Sets E xample 10.2.2 X = Rk with d2 metric: E = B1(0), for n £ N, Gn = 3^.(0) = > £ c U??=i Gn. E xample 10.2.3 X = R, E = ( 0,1): V.x 6 (0, 1),G X = ( -1,1) => E C Ux€(o,i) G 147 - Definition 10.2.4 i? is said to be compact if for every open cover {Gi : i £ 1} of E, we can find Gi,,..., Gin 3 EC [Gu U Gh u • • • U G,;J. E xample 10.2.5 / n X = R, J3 = (0,1) is not compact: Consider {Gx : x £ ( 0,1)} where Gx = ( — 1,2;). Suppose 3xi,..., xn 6 (0,1) 3 (0,1) C U r = i ( - 1 ' a ; i ) - Let Y = m ax{.x 1 ,...,x„} =* 0 < y < 1 =* (0,1) C ( -1,y). Lei ,i = ^ => 0 < K 1, a; £ ( -1,2/) Contradiction! Thus, (0,1) is no/, compact. R emark 10.2.6 / « i/ie Euclidean space, open sets are not compact. T heorem 10.2.7 Let K C Y C X. Then, K is compact relative to Y if and only if K is compact relative to X. Proof. (=>): Suppose K is compact relative to Y. Let {Gi,i £ 1} be an open cover of K in X. T hen, K c U i e / ft, so X = / C n Y C (U, :e/ G^tlY = \Ji(:]{GiC\Y): open relative to Y. Since K is open relative to Y, 3i\,. . ., in 3 K <z {Gix n Y) u (Gi2 n y ) u • • • u {Gin n y ) => ir c \J"=1 Gz. {<=): Suppose K is compact relative to X. Let {Ei,i £ 7} be any open cover of K in Y. T hen, V?; e / 3 an open set Gt £ X 3El = GlnY.K c (\JieT Et) c ( (J i e / G<). So, {Gi, i G 7} is an open cover in X. T hen, 3i\,... ,in 3 K c Gu U Gi2 U • • • U Gin => K = K n r C ( G^ n y ) U • • • U ( G ln n Y) = EuU..UEln. 0 T heorem 10.2.8 Let {X, d) be a metric space and K C X be compact. Then, K is closed. Proof We will show that K° is open. Let p £ Kc be an arbitrary fixed point. Vg £ K => d(p, g) > 0. Let ?*9 = \d{p,q) > 0. Vr, = Br{p), Wq = #r(</)- # C U 9 e/c ^f/ (because if is compact) => 3 gi,... ,<7„ e i f S / f C ^ U - U W,„ = W. Let K = Vqx n V92 n • • • n Vqn Air = Min {rqi,..., r,ln } > 0, then V = Br{p). Let us show that W n V = 0: If not, 3 z e V K n V = > z e V K = > z € Wg, for some i — 1 ,... ,n. Hence, d(z,qi) < r,h — ^d{p,q{). z £ V => z £ Vq% for the same i. Thus, d{z,p) < r(h = ^d{p,qi). 148 10 Basic Topology => d{p, qi) < d{p, z) + d(z, qi) < d(p, qi). C ontradiction! Therefore, W D V = 0. T hus, V = Br(p) C Xc C Kc => Kc is open => K is closed. D T heorem 10.2.9 Closed subsets of compact sets are compact. C orollary 10.2.10 If F is closed and K is compact, then FC\K is compact. T heorem 10.2.11 Let {Ki\i G / } be a collection of compact subsets of a metric space such that the intersection of every finite subcollection of Ki is nonempty. Then, Proof. Assume f]ieI Ki — 0. F ix a member of {Ki, i G 1} a nd call it K.. T hen, / cn[ f l Ki] = 9^JCc[\J K?]. Since K is c ompact, 3KX, ...,Kn 3 l C c [K[ U • • • U K%\ => K. D Kx n • • • D Kn = 0, since we intersect a finite subcollection, we have a c ontraposition ( Contradiction). D C orollary 10.2.12 If (Kn) is a sequence of nonempty compact sets 3 K\ D K2D---, then OZi Kn ? 0- Theorem 10.2.13 (Nested Intervals) Let(In) be a sequence of non-empty, closed and bounded intervals in R 3 I\ C I2 C • • •, then n=l Proof. Let In = [an,bn] 3 an <bn. T hen, h C h C • • • => ai < a2 < • • • < a „ < • • • < bn < • • • b2 < 61. Moreover, if k < n => Ik C In a nd a^ < an <bn <bkLet E = 0 1,02,... is bounded above by 61. Let x — sup-E, then Vn, an < x. Let us show that Vn, x < bn: If n ot, 3 n 3 bn < x => 3a,k £ E 3 bn < a/.. case 1: k < n =>• a^ < an < bn < a^, C ontradiction! case 2: k > n => an < ak < bk < bn < a^, C ontradiction! T hus, Vn, a: < bn =* a; G / „ , Vn =>• x e fX°=i A. => l T = i J » ? 0• 10.2 Compact Sets R e m a r k 1 0 . 2 . 1 4 Here are some remarks: 149 1. 7 /lim„_ l o o (6 n - an) = l i m ^ ^ length(In) = 0, => f£°=i In consists of one point. 2. If In 's are not closed, conclusion is false, e.g. In = ( 0, - ) . 3. If In's are not bounded, conclusion is false, e.g. In = [ n,oo]. D e f i n i t i o n 1 0 . 2 . 1 5 Let ax < b\,..., ak < bk be real numbers, then the set of all points p G Rk 3 p = {x\,... ,xk), a^ < xt < l>i, i = 1 , . . . , k is called a k-cell. So a k-cell is [ai,bi] x • • • x [ak,bk]. T h e o r e m 1 0 . 2 . 1 6 Let k G N be fixed. Let In be a sequence of k-cells in Rfc 3 IiDl2D---.Then, ' fl~i W 0. T h e o r e m 1 0 . 2 . 1 7 Every k-cells is compact (with d2 metric). Proof. L et / = [ai,bi] x • • • x \ak,bk} C Rfc b e a k-cell. If a\ = b\,... ,ak = bk, t hen / consists of one point. Then, / is compact. So assume for at least one j , dj < bj, j G { 1 , . . . k}. L et 5 = [Yll=i(°i — di)2}1 > 0. Suppose / is not c ompact. So, there is an open cover {Ga,a G A} of / 3 {Ga} d oes not have a ny finite subcollection the union of whose elements covers / . L et c% = ^ | ^ . Then, \aubi\ = [ a^c,] n [ci:bi\. T his way / can be divided into 2fc k -cells Qj 3 | J L i Qc = IAlso, Vj we have p,q G Qj, d(p,q) < ^S. S ince / cannot be covered by a finite number of Ga's, a t least one of the Qj's, s ay /] cannot be covered by a finite number of Ga's. S ubdivide I\ i nto 2 cells b y halving each side. Continue this way . . . We eventually get a sequence {/„} of k-cells such t h a t a ) / , C h C • • •; b ) / „ cannot be covered by any finite subcollection of {Ga,x c) p,q £ In => d(p, q) < ^-S, Vn. G A} , Vn; B y a) | X L i In + $• L e t P* e 0 ^ = 1 In C I, t hen 3 a 0 G A 3 p* Gan is open, 3r > 0 3 Br{p*) c G Q() . Find n0 G N 3 % < 2n" S how /,,,„ c G„„ : p*Gfir=i Ai c ^n„- Let p G J n o , by c) d{p,p*) => p G Br(p*) C G „„ => /„,„ C G a o a nd this contradicts to c ompact. • T h e o r e m 1 0 . 2 . 1 8 Consider Rk with d2 metric, lowing are equivalent: G Gao. S ince [i.e. ^ < r]. < ^ 5 < r. b). Thus, / is let E C Rfc. Then, the fol- (a) E is closed and bounded. (b) E is compact. (c) Every infinite subset of E has a limit point which is contained in E. 150 10 Basic Topology R emark 10.2.19 Consider the following remarks on Theorem 10.2.18: 1. The equivalence of (a) and (b) is known as Heine-Ba,rel Theorem,: A subset E of Rk is compact if and only if it is closed and bounded. 2. (b)<&(c) holds in every metric space. 3. (c)=> (a) , (b)=> (a) hold in every metric space. 4- (a)=>(c) , (a)=>(b) are not true in general. T heorem 10.2.20 (Balzano-Weierstrass) Every bounded infinite subset of Rfc has a limit point in Rk. Proof. Let E C Kfc be infinite and bounded. Since E is bounded 3 a k-cell / 9 E C I. Since I is compact, E has a limit point p E I c Rk. • T heorem 10.2.21 Let P ^ 0 be a perfect set in Rk. Then, P is countable. 10.3 The Cantor Set D efinition 10.3.1 Let E0 = [0,l], E, = [ 0 , i ] U [ | , l ] , E2 = [0, £ ] U [ £, £ ] U [ $, £ ] U [ JM], continue this way. Then, Cantor set C is defined as oo C= f]En. n=l Some properties are listed below: 1. 2. 3. 4. 5. C is compact. C^0. C contains no segment (a, (3). C is perfect. C is countable. Proof (Property 3). In the first step, ( | , | ) has been removed; in the second s tep (Jy, -p-), ( T^, ^ ) have been removed; and so on. C contains no open interval of the form ( 3 k ^ 1 , 3kJ,~2), since all such intervals have been removed in t he I s * , . . . , ( n - l) s ' ; steps. Now, suppose C contains an interval (a,/?) where a < j3. Let a > 0 be a c onstant which will be determined later. Choose n £ N 3 3 ~" < —^-- Let k be the smallest integer B a < ^ pr^, i.e. ^K~ < k, t hen k - 1 < a3'3~l- Show ^ < 0, i.e. fc < ^ ^ k < 1 + a 31=i; so show 1 + 2 ^LI < S^Lzl] i.e. 10.4 Connected Sets 1 < \{(3T - 2 - aT + 1] = ^ 1 a 3~"3 n - 1 a ")3" — > ^ >— 1 151 > 1, is what we want. So, a > 4. T hen, (^f^1, ^ | ^ ) C ( a,/?) C C , Contradiction! • Proof (Property 4). Let x £ C be an a rbitrary point of C. Let Br(x) = (x — r,x + r) be any open ball centered a t x. F ind n 6 N 3 4r < r, x G C = CC=1Em => x e £ „ = / f U ••• U / £ , , (disjoint intervals). So x £ / " for some j = 1 ,2,... ,2™. Then, x £ (x - r,x + r ) D / j 1 and length(/™) = -L < r => / " C (x - r, x + r). Let y be the end point of 7 3 y ^ x. T hen, y £ C n(x — r,x + r) => x is ™ a limit point of C. D 1 0.4 Connected Sets D efinition 10.4.1 Let (X,d) be a metric space and A,B C X. We say A and B are separated if A n B = 0 and .4 n B = 0. /I subset E of X is said to be disconnected if 3 two nonempty separated sets A,B3E = AUB. E C X is called, connected if it is not a union of two nonempty separated sets, i.e. 3 no nonempty separated subsets A, B B E — AL) B (V A,B pairs). E xample 10.4.2 X =• K2, with d2,dx or Let E = {(x,y) : x2 < y2} = {{x,y) : \x\ < \y\}. See Figure 10.11. CONNECTED DISCONNECTED Fig. 10.11. Example 10.4.2 T heorem 10.4.3 A subset E ^ 0 of R is connected if and only if E is an interval (E is an interval if and only if z,x £ E and x < z => Vy with x < y < z => y £ E). 152 10 Basic Topology (*). Proof. Let us mark the statement 2, x £ E and x < z => Vy with x<y<z=$-y€E Let E ^ 0 be connected. If £" is not an interval => (*) does not hold. i.e. 3x,z G E 3 x < z and 3y 3 x < y < z and y ^ E. Let yly — (—oo,y) n U, i?w = (y, +oo)niJ. i4y 7^ 0 (because x G J4W) and By^% (because 2 G 5 y ) . Ay U By = [ (-oo, y) U (?y, oo)] (IE = E. Ay C ( -oo, y) => .4 y C ( -oo, y] and ^ 5 ,; C (y, oo) => Ay n Sy C (-oo, y] n (;</, + oo) = 0 => Ay n 5 y = 0. Similarly, Ay II By = Q => E is disconnected, Contradiction! Suppose ZJ is disconnected. Then 3 nonempty separated sets A, B 3 A U B = E1. Let i e A,t/ e B. Assume without loss of generality x < y (because A n B = 0,_z ^_y). Let 2 = sup(A n [x, y}), t hen z E An [x, y\ C A (because Ac B => A C B), z ^ B. Since x £ An[x, y], we have x < 2. 2 G .4 n [x, y] C [x, y] = [x,y] =$• z < y =$• x < z < y. z = i] G A 1 If ' > => jy G J4 n B = 0, Contradiction; hence, 2 < iy. So, x < z < y, and 2 £ A. If z £ A =$• x < z < y. So x,y €. E 3 x < y and Z 3 I < 2 < J , z ^ E because z <£ B,z g A. So (*) does not hold. lfz€zA=>ztf:B (because sets are separated). Claim: (z,y) <£ B. If not, (z,y) C B 4 (z,i/) C B 4 [z,i/] C B =* z e B , C ontradiction. Therefore, 3z\ G (z,y) 3 z\ & B => x < z < z\ < y => z\ € [x,y]. If 2] G A, then z\ < z => z\ ^ A, z\ tf: E. =$• x,y €• E 3 x < y and 2i 3 x < z\ < y, C ontradiction to (*)! • P roblems 1 0.1. Let X y^ 0 be any set. Let d, g be two metrics on X. We say the metrics d and y are equivalent if there are two constants: A, B > 0 3 Ag(p, q) < d{p, q) < Bg(p, q), Vp, Show that the metrics di,d2,d00 qeX. A,B. for Rk are all equivalent, i.e. find 1 0.2. Let (X,d) be a metric space, p G X, r > 0. One is inclined to believe t hat£? r (p) = Br\p\\ i.e. the closure of the open ball is the closed ball. Give an example to show that this is not necessarily true. 1 0.3. Show that a metric space (X, d) is disconnected if and only if X has a n onempty proper subset which is both open and closed. 1 0.4. Consider the Printed Circuit Board (PCB) given in Figure 10.12 having 36 legs separated uniformly along the sides of the wafer. Suppose that a CNC 10.4 P r o b l e m s 153 F ig. 10.12. The PCB example m achine with a robot arm makes vias (a kind of drill operation) at points A, B,..., L. A h igh volume of P C B ' s are processed one after another. a ) Suppose t h a t the robot arm moves in horizontal as well as vertical direction u sing a single motor. It switches its direction in an infinitesimal time unit. T he CNC programmer uses the following logic to find the sequence of vias t o be processed: Start from A, g o to the closest neighbor if it has not been p rocessed yet. Break the ties in terms of ascending lexicographical order of l ocations. Once the initial sequence (Hamiltoncan tour) is obtained, examine t he nonconsecutive pair of edges of the tour if it is possible to delete these e dges and construct another tour (which is uniquely determined by the four l ocations) t h a t yields smaller tour in length. In order to check whether there e xist such an opportunity, the programmer calculates the gains associated w ith all possible pairs once. Suppose t h a t the connections between ( a , /3) and ( 7, S) is broken in the current tour. Then, new connections (a, 7 ) and (/3, 5) is c onstructed in such a way t h a t some portion of the tour is reversed and a new t our spanning all locations is obtained. Once all the gains are calculated, all t he independent switches is made. This improvement procedure is executed o nly once. 1. F ind the initial tour after deciding on the appropriate metric. 2. I mprove the tour. b ) W h a t if the robot arm moves in any direction using its motor? c) W h a t if the robot arm moves in horizontal as well as vertical direction u sing two independent but identical motors? d ) Suppose t h a t we have N P C B s to process. All the operation times are i dentical, each taking p t ime units. T h e robot arm moves at a speed of one leg distance per unit time along each direction. Let C\ b e the cost of making t he robot arm to move along any direction using the single motor and Ci b e t he cost of adding a second motor. Using the improved solutions found, which 154 10 Basic Topology r obot configuration is to be selected when t he o pportunity cost of keeping t he system busy is C0 per unit time? W eb m aterial http://br.endernet.org/~loner/settheory/reals2/reals2.html http://community.middlebury.edu/~schar/Courses/fs023.F02/paper1/ bahls.txt http://en.wikibooks.org/wiki/Metric_Spaces http://en.wikipedia.org/wiki/Closure_(topology) http://en.wikipedia.org/wiki/Compact_set http://en.wikipedia.org/wiki/Compact_space http://en.wikipedia.org/wiki/Discrete_space http://en.wikipedia.org/wiki/Limit_point http://en.wikipedia.org/wiki/Metric_space http: //eom . springer . de/c/c023470 . htm http://eom.springer.de/c/c023530 .htm http://eom.springer.de/C/c025350.htm http://eom.springer.de/m/m063680.htm http://homepages.cwi.nl/~bens/lmetric.htm http://homepages.nyu.edu/~eol/Book-PDF/chapterC.pdf http://kr.cs.ait.ac.th/~radok/math/mat6/calcl3.htm http://math.bu.edu/DYSYS/FRACGE0M/node5.html http://mathstat.carleton.ca/~ckfong/ba4.pdf http://mathworld.wolfram.com/ClosedSet.html http://mathworld.wolfram.com/CompactSet.html http://mathworld.wolfram.com/CompleteMetricSpace.html http://mathworld.wolfram.com/MetricSpace.html http://mathworld.wolfram.com/Topology.html http://msl.cs.uiuc.edu/planning/node196.html http://msl.cs.uiuc.edu/planning/node200.html http://oregonstate.edu/~peterseb/mth614/docs/40-metric-spaces.pdf http://pirate.shu.edu/projects/reals/topo/open.html http://pirate.shu.edu/"wachsmut/ira/topo http://planetmath.org/encyclopedia/ ANonemptyPerfectSubsetOfMathbbRThatContainsNoRationalNumber.html http://planetmath.org/encyclopedia/ ClosedSubsetsOfACompactSetAreCompact.html http://planetmath.org/encyclopedia/Compact.html http://planetmath.org/encyclopedia/MetricSpace.html http://planetmath.org/encyclopedia/NormedVectorSpace.html http://planning.cs.uiuc.edu/node184.html http://staff.um.edu.mt/jmusl/metrics.pdf http://uob-community.ballarat.edu.au/~smorris/topbookchap92001.pdf http://webOl.shu.edu/projects/reals/topo/compact.html http://webOl.shu.edu/projects/reals/topo/connect.html http://webOl.shu.edu/projects/reals/topo/open.html 10.5 W e b material http://www-db.stanford.edu/~sergey/near.html http://www-history.mcs.st-andrews.ac.uk/Extras/Kuratowski_ Topology.html http://www.absoluteastronomy.com/c/compact_space http://www.absoluteastronomy.com/c/connected_space http://www.absoluteastronomy.com/m/metric_space http://www.all-science-fair-projects.com/science_fair_projects_ encyclopedia/Limit_point http://www.answers.com/topic/limit-point-1 http://www.bbc.co.uk/dna/h2g2/A1061353 http://www.cs.colorado.edu/~lizb/topology-defs.html http://www.cs.colorado.edu/~lizb/topology.html http://www.cs.mcgill.ca/~chundt/354review.pdf http://www.di.ens.fr/side/slides/vermorel04metricspace.pdf http://www.dpmms.cam.ac.uk/~tkc/Further_Analysis/Notes.pdf http://www.fact-index.com/t/to/topology_glossary.html http://www.hss.caltech.edu/~kcb/Notes/MetricSpaces.pdf http://www.mast.queensu.ca/~speicher/Section8.pdf http://www.math.buffalo.edu/~sww/Opapers/COMPACT.pdf http://www.math.ksu.edu/~nagy/real-an/ 155 http://www.math.louisville.edu/~lee/05Spring501/chapter4.pdf http://www.math.miami.edu/~larsa/MTH551/Notes/notes.pdf http://www.math.niu.edu/~rusin/known-math/index/54EXX.html http://www.math.nus.edu.sg/~matwy1/d.pdf http://www.math.ohio-state.edu/~gerlach/math/BVtypset/node7.html http://www.math.okstate.edu/mathdept/dynamics/lecnotes/node33.html http://www.math.sc.edu/~sharpley/math555/Lectures/ MetricSpaceTopol.html http://www.math.ucdavis.edu/~emsilvia/mathl27/chapter3.pdf http://www.math.unl.edu/~webnotes/classes/class34/class34.htm http://www.mathacademy.com/pr/prime/articles/cantset/ http://www.mathreference.com/top-ms,intro.html http://www.maths.mq.edu.au/~wchen/lnlfafolder/lfa02-ccc.pdf http://www.maths.nott.ac.uk/personal/jff/G13MTS/ http://www.ms.unimelb,edu.au/~rubin/mathl27/summary2.pdf http://www.msc.uky.edu/ken/ma570/lectures/lecture2/html/compact.htm http://www.unomaha.edu/wwwmath/MAM/2002/Poster02/Fractals.pdf http://www22.pair.com/csdc/car/carfre64.htm http://wwwrsphysse.anu.edu.au/~vbrll0/thesis/ch2-connected.pdf 11 C ontinuity In this chapter, we will define the fundamental notions of limits and continuity of functions and study the properties of continuous functions. We will discuss these properties in more general context of a metric space. The concept of compactness will be introduced. Next, we will focus on connectedness and investigate the relationships between continuity and connectedness. Finally, we will introduce concepts of monotone and inverse functions and prove a set of Intermediate Value Theorems. 11.1 Introduction D efinition 11.1.1 Let (X, dx), (Y, dy) be two metric spaces; E ^ 0, E C X. Let f : E i-> Y,p £ E,q € Y. We say limn^+p f(x) = q or f(x) —> q as x — p » if Me > 0 ,3£ > 0 9 Vx G E with dx(x,p) < 5 we have dY(f{x), q) < e (Le. Ve > 0, 38 > 0 3 f{E D Bf(p)) C B »(g);. F ig. 1 1.1. Limit and continuity D efinition 11.1.2 Let (X,dx), (Y,dY) be metric spaces; 0 ^ E C X, and f : X t-¥ Y,p E E. f is said to be continuous at p if 158 11 C ontinuity Ve > 0,36 > 0 9 Vx 6 E with dx(x,p) < S we have du(f(x), f(p)) < s. R emark 11.1.3 The following characteristics are noted: • • / has to be defined at p, but p does not need to be a limit point of E. Ifp is an isolated point ofE, then f is continuous at p. That is, given e > 0 (no matter what £ is), find S 3 E fl Bf{p) = {p}. Then, x & E, d{p, x) < 6 =>• x = p. Hence, dy(f(x), f(p)) = 0 < e. Ifp is a limit point of E, then f is continuous atp-^f- limx_>p / ( x ) = / ( p ) . • D efinition 11.1.4 If f is continuous at every point of E, we say f is continuous on E. P roposition 11.1.5 Let (X,dx),(Y,dY),(Z,dz) be metric spaces and 0 ^ E C X, f : E i-» Y, g : f(E) i-» Z. If f is continuous at p G E and g is continuous at f(p), then g o f is continuous at p. Proof. Let q = / ( p ) . Let e > 0 be given. Since g is continuous a t q, 3n > 0 9 Vy G f(E) w ith dy(y,q) < n we have d-2(g{y),g(q)) < e. Since / is continuous a t p, 35 > 0 9 Vx e E w ith dx{x,p) < 6 => we have d y ( / ( x ) , / ( p ) ) < n. $ Let x G E b e 9 djr(:r,p) < < . T hen, y = f(x) G / ( E ) and dY(y,q) = 5 dy(f(x),f(p)) < n. Hence, dz{g{f{x)),g{f{p))) = dz(g(y),g(q)) < s. • T heorem 11.1.6 Lei ( X, dx-),(Y, d y) 6e metric spaces, and let f : X t-> Y. T/ien, / is continuous on X if and only i /V open se£ V in Y, / - 1 ( V ) = { p € •^ : / (p) S y } is open in X. Proof. (=>•): Let V be open in Y. If f~l(V) ^ 0, let p G / _ 1 ( ^ ) be a rbitrary. Show -1 -1 3r > 0 9 B *(p) C / ( ^ ) : p G / ( V ) implies / (p) G V. Since V is open, 3 s > 0 9 BY(f(p)) C V. Since / is continuous a t p, for e = s, 3r > 0 9 Vx G X w ith 4 ( x , p ) < r =• < W ( x ) , / (p)) < s =* / ( x ) G B s y (/(p)) =» x G / " ^ ( V ) . Let p G X b e a rbitrary. Given e > 0, let V = J 3^(/(p)) b e open. Then, / - 1 ( V ) is open a nd p G f~l{V). Hence, 3<J 9 B«(p) C / _ 1 ( V ) . If < 4(x,p) < 5 => x G B f (p) C / " H V ) , then / ( x ) G V => rf,(/(x),/(p)) < e. D C orollary 11.1.7 / : X —> Y is continuous on X if and only i /V closed set C in Y, f~l{C) is closed in X. Proof. f-l{Ec) = {f-\E))c. U D efinition 11.1.8 Let (X,d) be a metric space and / : , . . . , / * : X i-» R. De/me / : X M- Rfc 6j/ / ( x ) = ( / i ( x ) , . . . , fk{x))T, then / i , . . . , /fc are ca/ted components of f. 11.2 Continuity and Compactness 159 P roposition 11.1.9 / is continuous if and only if every component is continuous. Proof. (=>•): F ix j . Show that fj is continuous: Fix p £ X. Show that fj is continuous a t p. Given e > 0 35 > 0 3 Vx w ith d2{x,p) < 5, t hen 1/iW - / » l = *(/>(*),/;(p)) < d2(f(x),f(p)) < e. (<=): Assume that Vj, fj is continuous a t p S l . Show that / is continuous a t p. Let e > 0 be given. / i is continuous a t p => 35\ > 0 3 d2(x,p) < S\ =>• |/i(x) — / i(p)| < -4?. / 2 is continuous a t p => 3<52 > 0 9 d2(x,p) < 52 => 1/2(2;) — / 2(f)! < TTJ/it is continuous a t p => 3 4 > 0 3 d2(x,p) < 5k => |/fc(a;) - /fe(p)| < -j%Let 5 = m in{#i, ...,5k} > 0. Let X be 9 d(x,p) < 5. T hen, k 1 1 /2 d2(f(x),f(p)) = [J2\fi^)-fi(p)\2]1/2< £ (TE). = 6. 1 1.2 Continuity a n d C ompactness T heorem 11.2.1 The continuous image of a compact space is compact, i.e. if f : X i-t Y is continuous and (X,d) is compact, then f(X) is a compact subspace of (Y, dy). Proof. Let {Va : a 6 A} be any open cover of f(X). Since / is continuous, f-l(Va) is open in X. f(x) C \Ja&A Va =» X C / " H/fr)) C \JaeA f~HVa). Since X is c ompact, 3au...,an 9 X C [ / _ 1 ( V Q l ) | J - ' ' U / " 1 ^ ™ ) ] => / (*) C / [ / - 1 ( V a l ) U - - - U / _ 1 ( ^ a J ] = ^ U - ' - U ^ , since for A C /-'/W.rV^CBwehave /(IK) = U/(^)and /_1(IK) = U/"1^)- D C orollary 11.2.2 yl continuous real valued function on a compact metric space attains its maximum and minimum. Proof. f(X) is a compact subset of K => f(X) is b ounded. Let m = i nf/(a;), M = sup f(x). T hen, m,M e R; since f(X) is b ounded. Also, m,M e f(X). F urthermore, f(x) = f(x), since / ( X ) is c ompact. Thus, 3p e X 3 m = / (p) and 3? 6 I 3 M = / (g). Finally, m = / (p) < f{x) < f{q) = M,VxeX. D T heorem 11.2.3 Let (X,dx) be a compact metric space, (Y, dy) be a metric space, f : X H-» Y be continuous, one-to-one and onto. Then, f~x :Y>-¥Xis continuous. 160 11 Continuity Proof. Let g = f'1 : Y -> X. Show that V closed set C in X, g~l{C) is a closed set in Y: 5 _ 1 (C) = ( /~ 1 )~ 1 (C) — / ( C ) , since X is compact. Hence, / ( C ) is closed, thus g~l{C) is closed. D R emark 11.2.4 If compactness is relaxed, the theorem is not true. For example, take X — [0,2n), with d\ metric. Y — {(x, y) £ R2 : x2 + y2 = 1} with d% metric. f :X<->Y, f(t) = (cos t, sin t). f is one-to-one, onto, continuous. However / _ 1 is not continuous at P = ( 0,1) = /(0). If we let e = 7r, suppose there is a 8 > 0 9 V(x,y) £ Y wiift d2{{x,y), ( 1,0)) < 8, then we have \r1(x,y)-r1(l,0)\<e. However, for {x,y) 9 ^ < f~1(x,y) < 2ir (S = V%), we have \f-\x,v)-r1M\>\>*Thus, we do not have \f~\x,y) - r\l,0)\ < e = 7 V(x,y) €Y B d2[(x,y), (1,0)] < J. T 1 1.3 U n i f o r m Continuity D efinition 11.3.1 Let (X, dx), (Y, dy) be two metric spaces, f : X M- Y. We say f is uniformly continuous on X if \/e > 0, 38 > 0 3 Vp, < € X with dx(p, q) < 8, we have dY(f(p), ? f(q)) < e. R emark 11.3.2 Uniform continuity is a property of a function on a set, whereas continuity can be defined at a single point. If f is uniformly continuous on X, it is possible for each e > 0 to find one number 5 > 0 which will do for all points p of X. Clearly, every uniform continuous function is continuous. E xample 11.3.3 f{x,y) = 2x+\, E= {(x,y)eR2:l<y<2}. Let us show that f is uniformly continuous on E. Let e > 0 be given. Suppose we have found 8 > 0 whose value will be determined later. Let p = (x,y),q = (u,v) G E be such that d2(p,q) < 8, Show \f(x,y) — f(u,v)\ < e: d2(p,q) < 8 =>\x-u\ <8 and \y - v\ < 8 => \f(x,y) - f(u,v)\ = \2x+-^ -2u* -^\ < 2\x-u\ + {^-^)<28 we have \f(x,y) P + {y-y)(v+y) = 28+ l«-»jl»+»l. Since I"-»JI +"I < 48, (vy)2 f(u,v) < 65 = s. Hence, one can safely choose 8 = | > 0. 11.4 Continuity and Connectedness 161 E xample 11.3.4 f(x) = -,E = ( 0,1) C R. Let us show that f is not uniformly continuous on E but continuous on E: given e > 0, let S > 0 be chosen. Let x € E and \x — xo\ < 6. If x0 — 6 > 0, then \x — x0\ < 5 <> x0 - 8 < x < x0 + 5. £ 1 X 1 XQ < jx 0 -x| <J ^ < 5 (XQ <£ ^ s< exl 1 + EXQ XXQ XXQ — 5)XQ Hence, f is continuous at XQ and 6 depends on e and Xo . However, dependence on XQ does not imply that f is not uniformly continuous, because some other calculation may yield another 6 which is independent of XQ. So, we must show that the negation of uniform continuity to hold: 3e > 0 3 V 5 > 0 3 xi,x 2 e E 3 | xj - x2\ < S but | /(xi) - f(x2)\ < > e. Let e = 1. Let S be given. If 5 <\, one can find k 3 5 < -A-^ i.e. k = [ | — 1 ]. Thus, k>2.Letxi=5, x2 = 6 + £ => 0 < xi < | , 0 < x2 < 25 < § < 1 => Xl,x2 Sik+D E E. \xi - x2\ = f < f < 6, I /On) - f{x2)\ 3 > 1. //(5 > | => Let 5' = | . Find x\,x2 i -rii \x% — x \ 2 (S/k) < 5' < 5 and W+T) Ifix^-fix^lKe. T heorem 11.3.5 Let (X,dx) be a compact metric space, (Y, dy) be a metric space, and f : X H-> Y be continuous on X. Then, f is uniformly continuous. R emark 11.3.6 Let 0 ^ E C R be non-compact. Then, (a) 3 a continuous f : E — R which is not bounded. If E is noncompact then > either E is not closed or not bounded. If E is bounded and not closed, then E has a limit point x$ 3 XQ £ E. Let f(x) = J- , Vx G E. If E is unbounded then let f{x) = x, Vx G E. (b) 3 a continuous bounded function f : E — R which has no maximum. » If E is bounded let x§ be as in (a). Then, f(x) — 1+/x1_x ^ , Vx G E. sup f{x)=l but 3 no x G E 3 f{x) = 1. (c) If E is bounded, 3 a continuous function f : E — R which is not • uniformly continuous. Let XQ be as in (a). Let f{x) = —^—, Vx G E which is not uniformly continuous. 11.4 Continuity and Connectedness T heorem 11.4.1 Let (X,dx),(Y,dy) be metric spaces, 0 ^ E G X be connected and let f : X >-> y be continuous on X. Then, f(E) is connected. 162 11 Continuity Proof. Assume that f{E) is not connected, i.e. 3 n o n e m p t y A , B c y 9 A f l f l = f), A n B = 0, f{E) = A U B. LetG = EDf~1(A), H = Enf~1(B), A ± 0 =* 3q G A C / (£?) =• q = f(p) for some_p E E =$ p e f~l{A)_^ p £ G =$• G J± $. Assume G n H ^ 0. Let p G G n J? =*• p G i f = £7 n / _ 1 ( ^ ) =* / (p) G B , p G G = £ n / - 1 ( y l ) c /-'(A) Aci4 / _1 (*) _1 ( A ) : closed =» / - ( > 0 C / x _1 (3) =^p6 / ( 3 ) =» / (P) G A (**) (*)+(**)=> / (p) e i n f l / l , Contradiction. Thus, G n ff = 0. Similarly, G O # = 0. X 1 £ c r'iHE)) = / - ^ UB) = r (A) u r ^ ) • £ = £ n t /- 1 ^) u rHB)] = [En f~\A)] u [£ n r\B)) = GUH, meaning that E is not connected. Contradiction! C orollary 11.4.2 (Intermediate Value Theorem) Let f : [a, b] — K be » continuous and assume f(a) < f(b). Let c G R be such that f{a) <c< f(b) =>ce f([a,b]), i.e. 3x G (a,b) 3 f(x) = c. Proof. [a,b] is connected, so / ([a, 6]) is connected; thus /([a, 6]) is an interval [a,0].f(a),f(b)e[a,l3]=*cef([a,b]), 3x G [a, b] 3 f(x) = c, / ( a ) < c =$> x ^ a a nd /(ft) > c =>• x ^ b. T hus, x G (a, 6). • E x a m p l e 11.4.3 Let I=[0,l],f:I—>Ibe continuous. Let us show that 3x G I 3 f{x) = x. Let g{x) = f{x) —x be continuous. Show 3x £ I 3 g{x) = O.If$ such x => Vx G I we have g(x) > 0 or g(x) < 0. (i) g(x) > 0, V i € / ^ f(x) > x, Vx G / . Then, / ( l ) > 1; a Contradiction, (ii) g(x) < 0, Vx G 7 => / ( x ) < x,Vx G J . TTien, / (0) < 0; a Contradiction. D efinition 11.4.4 (Discontinuities) Let f : (a,b) - » X w/iere (-X", d) is a metric space. Let xbe3a<x<b and q € X. We say, f(x+) — q or limj-yj,.). f(t) = q if Ve > 0 35 > 0 9 V£ wii/i x < £ < x + (5we /lave d(f(t),f(x)) < e. f(x+) = q <$ V subsequence {tn} with x < tn < 6,Vn and l i m ^ o o bn = x we have limn_>oo f(t) - q. f(x-) = l i m ^ z - f(t) is defined analogously. Let x G (a, b) => limt_,.x f(t) exists •& f(x+) = f(x-) = l im*-^ f(t). Suppose f is discontinuous at some x G (a, b). 11.4 Continuity a nd C onnectedness 163 F ig. 1 1.2. E xample 11.4.3 (i) If f(x+) or f(x—) does not exist, we say the discontinuity at x is of the second kind, (ii) If f(x+) and f(x—) both exist, we say the discontinuity at x is of the first kind or simple discontinuity. (Hi) If f(x+) = f(x-), but f is discontinuous at x, then the discontinuity at x is said to be removable. y =l-x F ig. 1 1.3. E xample 11.4.5 E x a m p l e 11.4.5 / : K - > K , f(x) = / is continuous (only) at x = | : Let e > 0 be given. Let 5 = e. Let teRB\t t £ Q = > \f(t) - f(x)\ = \t-x\ = \t-±\<8 x, xeQ l - x, x e M \ i — x\<6 = e. 11 where x—\. ten\Q=>\f(t)-f(x)\ |1 — x\ < 6 = e. Hence, f is continuous at x — 2 ' CLAIM:f is discontinuous every other point than x —\ (without loss of generality, we may assume that x > ^): Let x ^ \. Show f(x+) does not exist. Let e = ' x2~~ ' . Assume f(x+) exists, = \i-t-x. _ 164 11 Continuity then for this specific e > 0, 35 > 0 9 W with x < t < x + 5, we have \f(t)-f(x)\<e. CASE 1: X G Q . FindteR\Q3x<t<x + S \f(t)-f(x)\ <e= &=^. But 4 |/(£) - f(x)\ = |1 - t - x\ = |2x - 1 + t - x | = |(2x - 1) - (x - t)\. Since \a — b\ > | | a| — |6| |, | /(t) - / ( x ) | > | |2x - 1| - |x - 1| | > \2x -l\-\x-t\> Then, we have \2x-l\~8<\f(t)-f(x)\<1-^^. =>• (5 > 2~ ' , Contradiction since 5 > 0 confeeiafcen as small as we want. CASE 2: X G R \ < > Proceed in similar way, but choose t as rational. Q. \2x -l\-6. 1 1.5 Monotonic Functions D efinition 11.5.1 Let f : (a,b) t-» R. / is said to 6e monotonically increasing (decreasing) on (a, 6) if and only if a < xi < x2 < b => f(Xl) < f{x2) (/(aJi) > f(x2)). P roposition 11.5.2 Let f : (a, b) i-> R 6e monotonically increasing on (a,b). Then, Vx 6 (a,fr),f(x+) and f{x—) exist and sup f(t) = f(x-)<f(x)<f(x+)= a<t<x inf / ( * ) . x<t<b Furthermore, a < x\ < x2 < b =$• f(xi+) < f{x2—). T heorem 11.5.3 Let f : (a,b) H-> R be monotonically decreasing on (a,b), then Vx G (a, 6), f(x+) and f(x—) exisi and inf /(*) = / ( x - ) > / (x) > f(x+) = s up / ( f ) . a<t<x x<t<b Furthermore, a < x\ < x2 < b =>• / ( x i + ) > f(x2 — ) . Proof. Let x G (a, 6) be arbitrary. Vf with 0 < £ < x , we have /(£) > / ( x ) . So, { /(i) : a < £ < x} is bounded below by / ( x ) . Let ^4 = ini{f(t) : a < t < x}. We will show A — f(x\—): Let e > 0 be given. Then, A + e is no longer lower bound of {f(t) : a < t < x}. Hence, 3t0 G (a, x) 9 / ( i 0 ) < A + e. Let < = x - t0 Vt3x-5 5 = t0<t< x => f(x) < f(t0) < A + e and f(t) > A > A - e. Hence, Vr G (x - £, x) we have A - s < f(t) < A + e =>• |/(r) - ^ | < e. T hus, 4 = f(x-). Therefore, info<t<x f(t) = A = / ( x - ) > f(x). Similarly, sup a ; < t < b / (x) = / ( x + ) < f(x). Let a < x i < x2 < b, apply first part b <- x2 a nd x <r- x\. / ( x i + ) = SUPar^^xa /(*) ^ i ^ i K K i ) ! /(<) = f{*2 + )• 11.5 Monotonic Functions f f(x-) f(x) ^0 165 f(x+) F ig. 1 1.4. P roof of Theorem 11.5.3 C o r o l l a r y 1 1 . 5 . 4 Monotonic type. functions have no discontinuities of the second T h e o r e m 1 1 . 5 . 5 Let f : (a, b) H » K be monotonic. Let A be the set of discontinuous points of f, then A is at most countable. Proof. A ssume / is d ecreasing, then A = {x G {a,b} : f(x+) < f(x—)}. Vx G A, find f(x) G Q B f(x+) < r(x) < f(x-) a nd fix r(x). D efine g : A i-» < > b y Q g(x) = r{x). W e will show t h a t g is o ne-to-one: Let x\ ^ X2 G A,X\ < X2 =>r(xi) > f(xi+) > f{x2—) > r(x2) => r(x\) / r (x2). Thus, < i s o ne-to-one, ? a nd A is n umerically equivalent t o Q b y g(x) = r ( x ) . Therefore, ^4 i s a t m ost c ountable. • R e m a r k 1 1 . 5 . 6 The points in A may not be isolated. In fact, given any countable subset E of (a, b) (E may even be dense), there is a monotonic function f : (a,b) H-> R B f is discontinuous at every x G E and continuous at every other point. The elements of E as a sequence { x i , £ 2 , . . . } . Let cn > 0 3 Ylcn is convergent. Then, every rearrangement Yl c4>(n) a^so converges and has the same sum. Given x G (a, b) let Nx = {n : xn < x}. This set may be empty or not. Define f(x) as follows 1 luneNx This function is called saltus function c »> otherwise function. or pure jump (a) f is monotonically increasing on (a,b): Leta<x<y<b.IfNx=0, f(x) = 0 and f(y) > 0 . / / Nx jL 0, x < y =» f(x) = E n e j v . c » < T.neNy ^ = f(y). (b) f is discontinuous at every xm G E: Letxm G E befixed. f{xm+) = MXm<t<bf(t), f{xm-) = s u p a < s < X m f(s). Let xm < t < b, a < s < xm be arbitrary =>• a < s < xm <t<b. Then, NscNt, m<E;Nt, m<£Ns=>meNt\Ns. J\t) ~ j(S) — Z^nENt C " — 2-m€NB C « ~ Z^nENa\Nt C » — Cm => 166 11 Continuity f(t) > Cm + / («)• Fix f(s) => cm + f(s) is a bound for all f(t)'s. So, take the infimum over t's. f(xm+) > f(s) + cm •& f(xm+) — cm > f(s). If we take supremum over s's, we will have f(xm+) — cm > f{xm-) => f(xm+) - f{xm-) > cm. Therefore, f(xm+) ^ f(xm-) (In fact, f(xm+) - f(xm-) = cm). (c) f is continuous at every x £ (a,b) \ E: Let x £ (a,b)\E be fixed. We will show that f is continuous at x. Let e > 0 be given, since ^ c„ converges, 3N 3 X I^LJV+I c„ < oo CJV+I +SJV = s => r N+i — s — sjv- Let 5' = Min{\x — a ^l,..., |x — XN\,X — a,b — x}. Let u 2 . Claim (i) If x < xn < x + 5 then n > N + 1. If n < N + 1, then \x — XN\ > 5' = 25, Contradiction. Claim (ii) If x — 5 < xn < x, then n > N + 1. f(x) — e < f(x — 5), f(x + 5)<f(X)+S, Cn c f{x)-f(x-S) c £ For = Y/„eNxCn-Tln€N:c_sCn the C C £ = 12neNx\Nx-S f( ) x - Sr=w+i " < n-2^neNx C second claim, f(x + 5) « - z2n=N+l «< - — J2n£Nx+s " ~ 12n&Nx+s\Nx Lett be 3 \t-x\ < 6, i.e. x-5 < t < x+6 => f(x-6) < f{t) < F(x+5). Hence, f{x) - e < f(t) < f(x) + e, \f(t) - f(x)\ < e. Problems 1 1.1. Let (X, d) be a metric space. A function / : X t-t R is semi-continuous (Isc) if V6 £ R t he set {x £ X : f{x) > b} is u pper semi-continuous (use) if V6 £ R t he set {x £ X : f(x)<b} X. Show that a) / is Isc < > Ve > 0, Vx0; 35 > 0 3 x £ Bs(x0) => f(x) > f(x0) £ b) / is use < > Ve > 0, Vx0; 35 > 0 3 x £ B5(x0) => f(x) < f(x0) £ called lower open in X; is open in - e. + s. 1 1.2. Let (X, dx) b e a compact metric space, (Y, dy) be a metric space and let / : X t-> Y be continuous and one-to-one. Assume for some sequence {pn} in X a nd for some q £Y, linXr^oo f(pn) = q. Show that 3p £ X 3 lim pn = P and f(p) = q. x—>oo 1 1.3. Give a mathematical argument to show that a heated wire in the shape of a circle (see Figure 11.5) must always have two diametrically opposite points w ith the same temperature. W eb material http://archives.math.utk.edu/visual.calculus/1/continuous.7/ 11.6 Web m aterial 167 Fig. 11.5. A heated wire index.html http://at.yorku.ca/course/atlas2/node7. html http://at.yorku.ca/i/a/a/b/23.dir/ch2.htm http: //bvio. ngic. re. kr/Bvio/index. php/Honotonic_f unction http://cepa.newschool.edu/het/essays/math/contin.htm http://clem.mscd.edu/~talmanl/TeachCalculus/Chapter020.pdf http://documents.kenyon.edu/math/neilsenj.pdf http://en.wikipedia.org/wiki/Intermediate_value_theorem http://en.wikipedia.org/wiki/List_of_general_topology_topics http://en.wikipedia.org/wiki/Monotonic_function http://en.wikipedia.org/wiki/Uniformly_continuous http://eom.springer.de/T/t093150.htm http://homepages.nyu.edu/~eol/Book-PDF/chapterD.pdf http://intermediate_value_theorem.iqexpand.com/ http://math.berkeley.edu/~aclayton/mathl04/8-10_Final_review.pdf http://math.furman.edu/~dcs/book/c2pdf/sec25.pdf http://math.furman.edu/~dcs/book/c3pdf/sec37.pdf http://math.stanford.edu/~aschultz/w06/mathl9/ coursenotes_and_handouts/ http://mathworld.wolfram.com/IntermediateValueTheorem.html http://mcraefamily.com/MathHelp/CalculusTheoremllntermediateValue.htm http://nostalgia.wikipedia.org/wiki/Connectedness http://ocw.mit.edu/0cwWeb/Mathematics/18-100BAnalysis-IFall2002/ LectureNotes/index.htm http://oregonstate.edu/instruct/mth251/cq/Stage4/Lesson/IVT.html http://personal.Stevens.edu/~nstrigul/Lecture4.pdf http://personal.Stevens.edu/~nstrigul/Lecture5.pdf http://personal.Stevens.edu/~nstrigul/Lecture8.pdf http://personal.Stevens.edu/~nstrigul/Lecture9.pdf http://pirate.shu.edu/projects/reals/cont/proofs/ctunifct.html http://planetmath.org/encyclopedia/IntermediateValueTheorem.html http://planetmath.org/encyclopedia/UniformlyContinuous.html http://poncelet.math.nthu.edu.tw/chuan/cal98/uniform.html http://toshare.info/en/Monotonic_function.htm http: //tutorial .math. lamar. edu/AHBrowsers/2413/Continuity. asp http://web01.shu.edu/projects/reals/cont/contin.html http://webalt.com/Calculus-2006/HowTo/Functions/Intermediate_Value_ 168 11 C ontinuity Theorem.ppt http://whyslopes.com/Calculus-Introduction/TheoremOne_Sided_Range.html http://www-history.mcs.st-and.ac.uk/~john/analysis/Lectures/L20.html http://www.absoluteastronomy.com/i/intermediate_value_theorem http://www.answers.com/topic/continuous-function-topology http://www.bostoncoop.net/~tpryor/wiki/index.php?title=Monotonic http://www.calculus-help.com/funstuff/tutorials/limits/limit06.html http://www.cut-the-knot.org/Generalization/ivt.shtml http://www.danceage.com/biography/sdmc_Monotonic http://www.econ.umn.edu/~mclennan/Classes/Ec5113/ ec5113-lec05-1.16.99.pdf http://www.f astload.org/mo/Monotonic.html http://www.geocities.com/Athens/Delphi/5136/Continuity/ continuity.html http://www.karlscalcuius.org/ivtproof.html http://www.math.ksu.edu/~mkb9154/chapter3/ivt.html http://www.math.ku.dk/~moller/e03/3gt/3gt.html http://www.math.louisville.edu/~lee/RealAnalysis/realanalysis.html http://www.math.mcgill.ca/drury/rootm.pdf http://www.math.sc.edu/~sharpley/math554/Lectures/math554_ Lectures.html http://www.math.ucdavis.edu/~emsilvia/mathl27/chapter5.pdf http://www.math.ucsb.edu/~gizem/teaching/Sl17/S117.html http://www.math.unl.edu/~webnotes/classes/class28/class28.htm http://www.math.unl.edu/~webnotes/contents/chapters.htm http://www.math.uu.se/~oleg/topoman.ps http://www.mathreference.com/top-ms,ivt.html http://www.mathreference.com/top-ms,unif.html http://www.maths.abdn.ac.uk/~igc/tch/ma200l/notes/node38.html http://www.maths.mq.edu.au/~wchen/lnlfafolder/lfa02-ccc.pdf http://www.maths.nott.ac.uk/personal/j ff/G12RAN/pdf/Uniform.pdf http://www.maths.ox.ac.uk/current-students/undergraduates/handbookssynopses/2001/html/mods-01/nodel0.html http://www.maths.ox.ac.uk/current-students/undergraduates/lecturematerial/Hods/analysis2/pdf/analysis2-notes.pdf http://www.maths.qmul.ac.uk/~reza/MAS101/HV-WEB.pdf http://www.maths.ted.ie/pub/coursework/424/GpReps-II.pdf http://www.nuprl.org/documents/real-analysis/node6.html http://www.people.vcu.edu/~mikuleck/courses/limits/tsld028.htm http://www.recipeland.com/facts/Monotonic http://www.sees.swarthmore.edu/users/02/rebecca/pdf/Math47.pdf http: //www. sosmath. com/calculus/limcon/limcon06/limcon06. html http://www.termsdefined.net/mo/monotone-decreasing.html http://www.thebestlinks.com/Connected_space.html http://zeus.uwindsor.ca/math/traynor/analysis/analbook.pdf www.isid.ac.in/~arup/courses/topology.ps 12 D ifferentiation In physical terms, differentiation expresses t he r ate a t which a q uantity, y, changes with respect t o the change in a nother quantity, x, on which it has a functional relationship. This small chapter will start with t he discussion of t he derivative, which is one of the two c entral concepts of calculus ( the o ther is the i ntegral). We will discuss t he Mean Value Theorem a nd look a t some applications that include t he relationship of the derivative of a function w ith whether t he function is increasing or decreasing. We will expose Taylor's t heorem as a generalization of t he Mean Value Theorem. In calculus, Taylor's t heorem gives t he a pproximation of a differentiable function near a point by a polynomial whose coefficients depend only on the derivatives of the function a t that point. There are many OR applications of Taylor's approximation, especially in linear a nd non-linear optimization. 1 2.1 Derivatives D efinition 12.1.1 Let f : [a,b] K-> K. VX G [a, b], let (j>(t) = / ( t ) l f ( x ) , a < t < b, t ^ x. f'{x) = limt_>.x <j>(t) provided that the limit exists. f is called the derivative of f. If f is defined at x, we say f is differentiable at x. If f is defined at\fx € E C [a,b], we say f is differentiable on E. Moreover, lefthand (right-hand) limits give rise to the definition of left-hand (right-hand) derivatives. R emark 12.1.2 If f is defined on (a,b) and if a < x < b, then f can be defined as above. However, f'(a) and f'(b) are not defined in general. T heorem 12.1.3 Let f be defined on [a,b], f is differentiable at x G [a, b] then f is continuous at x. Proof. Ast-^x, f(t) - f{x) = f(t) f {x) tZ x (t - x) -> f'{x) - 0 = 0. • 170 12 Differentiation continuous R emark 12.1.4 The converse is not true. One can construct functions which fail to be differentiate at isolated points. Let us s tate some properties: Suppose / a nd g are denned on [a, b] a nd are differentiable a t x G [a,b]. T hen, f + g, f • g and f/g are differentiable a t x, and (a) (f + gy(x) = f'(x)+g'(x). (b) (f-gy(x) = f'(x)g(x) + f(x)g>(x). (c) (f/g)'(x) = /'(*)g(*)-/(«)g'(«), g{x) + o. (d) Chain Rule: If h(t) = (g°f)(t) = g(f{t)), a < t < b, a nd if/ is continuous a t [a, b], f exists a t x £ [a,b], g is defined over range of / and g is differentiable a t f(x). T hen, h is differentiable a t x a nd h'(x) = g'(f(x))f(x). E xample 12.1.5 (Property ( c)) The derivative of a constant is zero. If f(x) — x then f'(x) — 1. / / f(x) — x • x = x2 then f'(x) — x + x = 2x by property (b). In general, if f(x) = xn then f'(x) = nxn~1, n € N. If f{x) = i = x~l then f'(x) = ^ = -x~2. In this case, x ^ 0. if f(x) = x ~ n , n E N then f'(x) = — nx~- ("+ 1 ). Thus, every polynomial is differentiable, and every rational function is differentiable except at the points where denominator is zero. E xample 12.1.6 (Property ( d)) Let / (*){ X sin -0,, x ^ 00 x= Then, f'(x) = sin \ - \ cos A, x ^ 0. At x = 0, A is no* de/med / ( t | l ^ 0 ) = sin j . yls £ —• 0, £/ie Kraii does not exist, thus / '(0) does not exist. > 1 2.2 M e a n Value Theorems D efinition 12.2.1 Let f : [a,b] H-» K. We say f has a local maximum at peXif35>0 B f{q) < f(p), V</ G X with d{p,q) < 5. Local minimum is defined similarly. T heorem 12.2.2 Let f : [a,b] i-> K. / / / has a local maximum at x £ (a,b) and if f'(x) exists, then f'(x) = 0. Proof. We will prove t he maximum case: Choose S as in the definition: a < x — 5<x<x + 5<b. lix-8<t<x, t hen mtZfJx) > 0. Let t -> x => /'(a:) > 0. If x < t < x + (5, t hen T hus, / ' ( x ) = 0. D f{t) f {x) tZ x (minimum) < 0. Let t -> oo => /'(a;) < 0. 12.2 Mean Value Theorems 171 T heorem 12.2.3 Suppose f : [a,b] H->- R is differentiable and f'(a) < A < f'(b) [ /'(a) > A > / '(&)]. Then, 3x e (a, 6) 3 /'(a;) = A. Proof. Let g(<) = /(£) - Xt. T hen, #'(a) < 0 [g'(a) > 0] so that g(h) < g(a) [g(ti) > g(a)] for some *i e ( a,6), so that g(f2) < g{b) [g{h) > # ( a )l for some £2 G (a, b). Hence, g a ttains its minimum [maximum] on [a,b] a t some points x 6 (a,b). By the first mean value theorem, g'(x) = 0. Hence, f'{x) = X. D C orollary 12.2.4 / / / is differentiable on [a,b], then f simple discontinuities on [a,b]. cannot have any R emark 12.2.5 But f may have discontinuities of the second kind. T heorem 12.2.6 (L'Hospital's Rule) Suppose f and g are real and differentiable in (a,b) and g'(x) ^ 0, Vx € (a,b) where oo < a < b < + oo. Suppose ff'(x) , ; . -> A as x -> a (o). / / /(x) — 0 and g(x) — 0 as x — a or i/ /(x) -> +oo and g(x) — + oo as > > > > x — a, i/ien > #(») 4 OS I - > O. Proof. Let us consider the case —oo < ^4 < +oo: Choose g 6 R 3 A < g, and choose r 9 A < r < q. By (o), 3c E (a, b) 3 a < x < c => :L-y-~ < r. ( £) g(x) If a < x < y < c, t hen by the second mean value theorem, fix) * 6 ( M ) ,&ia.m< r .(», g(x)-g{y) g'(y) Suppose /(x) -> 0 and g(x) - » 0 as x -> a. Then, (4) | M <r<q,a<y<c. Suppose g(x) -* +oo as x — a. Keeping y fixed, we can choose c\ £ (a,y) 3 > g{x) > g{y) and g(x) > 0 if a < x < cx. Multiplying (4) by \g(x)-g(y)]/g(x), we have $ g < r - r f ( g + $ g , o < z < c x . If a: - • a 3c2 € ( a, C l ) 9 $ g < q, a < x < C2. Summing with (4») Vo 3 A < q yields 3c2 3 -j-r f(x\ < q if a < x < c2. Similarly, if —oo < A < + oo and p 3 p < A, 3c^ 3 p < :4fy, a < x < C3. D 172 12 Differentiation 1 2.3 Higher Order Derivatives D efinition 12.3.1 / / / has a derivative / ' on an interval and if f is itself differentiate, we denote derivative of f as / " , and call the second derivative of f. Higher order derivatives are denoted by / ' , / " , / ^ 3 ' , . . . ,f^n\ each of which is the derivative of the previous one. T heorem 12.3.2 (Taylor's Theorem) Let f : [a,b] i-> K, n G N, f(n~l) be continuous on [a,b], and f^n\t) exists Vi G [a,b]. Let a / (3 G [a,b] and define fc=0 Then, 3x G (a, (3) 3 / (/3) = p(/3) + ^ f s i (/3 - a ) " . R emark 12.3.3 Forn = 1, the above theorem is just the mean value theorem. Proof. Let M 3 / (/3) = p(f3) + M(0 - a)n. Let g(t) = f(t)-p(t)-M(t-a)n, a < t < b, t he error function. We will show t hat n\M = f{n){x) for some x G (a, 6). We have g^(t) = fn\t)-n\ M, a < t < b. If 3x G (a, 6) 9 ^*n^(x) = 0, we are done. p(k\a) = f{k)(a), k = 0,...,n-l => g{a) = g'(a) = g»(a) = • • • = ^ ""^(a) = 0. Our choice of M yields g{(i) = 0, t hus g'(x\) = 0 for some x\ G ( a,/3) by t he Mean Value Theorem. This is for <?"(•), one may continue in t his manner. T hus, g(n\xn) = 0, for some xn G ( a, x „_i) C (a,fi). • D efinition 12.3.4 A function is said to be of class Cr if the first r derivatives exist and continuous. A function is said to be smooth or of class C°° if it is of class Cr, V r G N . T heorem 12.3.5 (Taylor's Theorem) Let f : A H-> E , be of class Cr for A C M", an open set. Let x,y G A and suppose that the segment joining x and y lies in A. Then, 3c in that segment 3 r_1 1 1 f{y)-f(x) = ^-fW(y-x,...,y-x) fc=i ' + r -(c)fV(y-x,...,y-x) ' where fW(y-x,...,y-x) = Y;iu...,ik (e,f•*•&,,, ) fan ~ xh) •'' (!/<„ ~ xiJ Setting y = x + h, we can write Taylor's formula as f(X + h) = f{x) + f'(x) • h + • • • + ^J—^f^)ix) .(h,...,h) • + Rr-^X, h), 12.4 Web material where Rr-\(x,h) is the remainder. Furthermore, 173 mr1 P roblems 1 2.1. Suppose / : [0, oo) H-» K is continuous, /(0) = 0, / is differentiable on (0, oo) and / ' is nondecreasing. Prove that g(x) — ^p- is nondecreasing for x > 0. 1 2.2. Let A C R " be an open convex set and / : A H-J- IRm be differentiable. If f'(t) = 0, Vi then show that / is constant. 1 2.3. C ompute the second order Taylor's formula for f(x,y) a round the origin. = sin(x + 2y) 1 2.4. Let feC2 and x* e _." be local minimizer. a) Prove the first order necessary condition (x* is a local minimizer then V /(x*) = 6) using Taylor's approximation. b) Prove the second order necessary condition (x* is a local minimizer then V 2 /(x*) is positive semi-definite) using Taylor's approximation. c) Design an iterative procedure to find V/(x) = 8 in such a way that it s tarts from an initial point and updates as x& = Xk-\ + Pk- T he problem at each iteration is to find a direction Pk t hat makes V/(xfc_i) closer to the null vector. Use the second order Taylor's approximation to find the best pk a t any iteration. d) Use the above results to find a local solution to m i n / ( x 1 , x 2 ) = x\ + 2x\ + 2Ax\ + x\ + Y2x\. S tart from [1, if. W eb material http://archives.math.utk.edu/visual.calculus/3/index.html http://calclab.math.tamu.edU/~belmonte/ml51/L/c5/L53.pdf http://ccrma-www.stanford.edu/"jos/mdft/Formal_Statement_Taylor_s_ Theorem.html http://courses.math.nus.edu.sg/mall04/lecture_notes/Notes_l.pdf http://d.faculty.umkc.edu/delawarer/RDvsiCalcList.htm http://en.wikipedia.org/wiki/Derivative http://en.wikipedia.org/wiki/L'Hopital's_rule 174 12 Differentiation http://en.wikipedia.org/wiki/Mean_value_theorem http://en.wikipedia.org/wiki/Taylor's_theorem http://grus.berkeley.edu/"j rg/ay202/node191.html http://hilltop.bradley.edu/"jhahn/Note3.pdf http://home.uchicago.edu/~lfmedina/MathRev3.pdf http://kr.cs.ait.ac.th/~radok/math/mat1l/chap7.htm http://kr.cs.ait.ac.th/~radok/math/mat6/calc2.htm http://mathworld.wolfram.com/Derivative.html http://mathworld.wolfram.com/LHospitalsRule.html http://mathworld.wolfram.com/Mean-ValueTheorem.html http://ocw.mit.edu/ans7870/textbooks/Strang/strangtext.htm http://ocw.mit.edu/0cwWeb/Mathematics/18-100BAnalysis-IFall2002/ LectureNotes/ http://people.hofstra.edu/faculty/stefan_waner/RealWorld/ math19index.html http://pirate.shu.edu/projects/reals/cont/derivat.html http://saxonhomeschool.harcourtachieve.com/en-US/Products/ shcalculustoc.htm http //web.mit.edu/wwmath/calculus/differentiation/ http //www-math.mit.edu/"djk/18_01/chapter26/section01.html http //www. absoluteastronomy. com/l/lh'/,C3'/,B4pitals_rulel http //www.analyzemath.com/calculus.html http //www.jtaylorll42001.net/ http //www.ma.utexas.edu/cgi-pub/kawasaki/plain/derivatives/1.html http //www.math.dartmouth.edu/"m3cod/textbooksections.htm http //www.math.harvard.edu/comput ing/math/tutorial/taylor.html http //www.math.hmc.edu/calculus/tutorials/ http //www.math.scar.utoronto.ca/calculus/Redbook/goldch7.pdf http //www.math.tamu.edu/-fulling/coalweb/lhop.htm http //www.math.tamu.edu/-fulling/coalweb/taylor.htm http //www.math.tamu.edu/"tom.vogel/gallery/node12.html http //www.math.uconn.edu/-corluy/calculus/lecturenotes/nodel5.html http //www.mathdaily.com/lessons/Category:Calculus http //www.mathreference.com/ca,tfn.html http //www.maths.abdn.ac.uk/"igc/tch/egl006/notes/nodel36.html http //www.maths.abdn.ac.uk/~igc/tch/mal002/appl/node54.html http //www.maths.abdn.ac.uk/-igc/tch/mal002/diff/node39.html http //www.maths.abdn.ac.uk/"igc/tch/ma2001/notes/node46.html http //www.maths.lse.ac.uk/Courses/MA203/sec4a.pdf http //www.maths.manchester.ac.uk/-mdc/old/211/notes4.pdf http //www.mathwords.com/index_calculus.htm http //www.npac.syr.edu/REU/reu94/williams/ch3/chap3.html http //www.physics.nau.edu/~hart/matlab/node52.html http //www.sosmath.com/calculus/diff/derll/derll.html http //www.sosmath.com/tables/derivative/derivative.html http //www.toshare.info/en/Mean_value_theorem.htm http //www.univie.ac.at/future.media/moe/galerie/diff1/diff1.html http //www.wellington.org/nandor/Calculus/notes/notes.html http //www.wikipedia.org/wiki/Mean_value_theorem 13 P ower Series and Special Functions In mathematics, power series are devices that make it possible to employ much of the analytical machinery in settings that do not have natural notions of "convergence". They are also useful, especially in combinatorics, for providing compact representations of sequences and for finding closed formulas for recursively defined sequences, known as the method of generating functions. We will discuss first the notion of series, succeeded by operations on series and tests for convergence/divergence. After power series is formally defined, we will generate exponential, logarithmic and trigonometric functions in this c hapter. Fourier series, gamma and beta functions will be discussed as well. 1 3.1 Series 1 3.1.1 Notion of Series D efinition 13.1.1 An expression oo X/ fc=0 Uk = X/Uk ~ u°+Ui+"2 ^ — 0 oo where the numbers Uk (terms of the series) depend on the index k — 0 , 1 , 2 , . . . is called a (number) series. The number Sn — u0 + ui -i \-un, n = 0 , 1 , . . . is called the nth partial sum of the above series. We say that the series is convergent if the limit, l i m n - ^ Sn = S, exists. In this case, we write oo S = Up + Ui + U2 + • • • = } j Uk k=0 and call S the sum of the series; we also say that the series converges to S. 176 13 Power Series and Special Functions P roposition 13.1.2 (Cauchy's criterion) The series oo k-0 is convergent if and only if Ve > 0, SN 3 \fn,p e N ,n > N, \un+1 + ••• + un+p\ = \Sn+p - Sn\ < e. R emark 13.1.3 In particular, putting p — 1 we see that ifJ2T=oUk *s con~ vergent its general term Uk tends to zero. This condition is necessary but not sufficient! D efinition 13.1.4 The series are called the remainder series of the series E oo fe=0 W fc; U „+l + W„+2 H = 2_, un+kfc = l oo Since the conditions of Cauchy 's criterion are the same for the series and its remainder series, they are simultaneously convergent or divergent. If they are convergent, the remainder series is m n—+oo *—* fc=l lim S~] un+k = lim ( S „ + m - S „) = S - S„. n—>oo / / the series are real and nonnegative, its partial sums form a nondecreasing sequence Si < S2 < S3 < • • • and if this sequence is bounded (i. e. Sn < M, n = 1,2,...), then the series is convergent and its sum satisfies the inequality lim Sn = S < M. n—>oo / / this sequence is unbounded the series is divergent linin-^oo Sn = 00. In this case, we write J^J*Lo uk — 00 and say that the series with nonnegative terms is divergent to 00 or properly divergent. E xample 13.1.5 The nth partial sum of the series 1 + z + z2 + • • • is S„(z)= i _ z forz^l. If \z\ < 1 then zn+l = \z\n+ - > 0, that is zn+1 - > 0 as n - > 00. If\z\>l then -> 00. Finally, if \z\ = 1 then zn+1 = cos(n + 1)9 + i sin(n + 1)6, where 6 is the argument of z, and we see that the variable zn+1 has no limit as n — 00 > because its real or imaginary part (or both) has no limit as n — 00. For • z — 1, the divergence of the series is quite obvious. We see that the series is convergent and has a sum equal to (1 — z)"1 in the open circle \z\ < 1 of the complex plane and is divergent all other points z. 13.1 Series 1 3.1.2 Operations o n S eries 177 P roposition 13.1.6 IfJ2T=o uk and SfcLo Vk are convergent series and a € C, then the series Yl'kLo aUk and 12<kLo(Uk -'- Vk) are a^so convergent and we have oo oo oo oo oo ^ c r a f e = a ^ M f c and Y^(uk±vk) k=0 k=0 fc=0 = fc=0 y^uk±y^Vkfc=0 Proof. Indeed, J2T auk = linin^oc ^ o auk = «lim„->oo J2ouk=a E o ° ufc> a n d X)~(«fc ± vk) = linin^oo J ]g (wfc ± We = l im nr+00 ^ )o ufc ± lim.n_>oo S o wfc f) D J2™uk±J2'o'vk- = R emark 13.1.7 .ft should be stressed that, generally speaking, the convergence of ^ ^ ° Uk ± X^o° Wfc does not imply the convergence of each of the series £ £1:0 wfc and SfcLo^*" w ' l * c ' 1 c a n be confirmed by the example below: (a-a) + (a-a)-\ , Va e C. 1 3.1.3 Tests for p ositive series T heorem 13.1.8 (Comparison Tests) Let there be given two series oo oo (i) ^Uk o with nonnegative terms. and (ii) ^ « f c o (a) If Uk < Vk, VA;, the convergence of series (ii) implies the convergence of series (i) and the divergence of series (i) implies the divergence of series (ii). (b) If linifc-Kx, ^ = A > 0, then series (i) and (ii) are simultaneously convergent and divergent. Proof. Exercise! D T heorem 13.1.9 (D'Alembert's Test) Let there be a positive series oo Y^Uk 3 uk > 0, Vk = o 0,1,... (a) If ^^ < q < 1, VA;, then the series J2™ uk is convergent. If ^ ^ > 1, then the series ]Po° uk *s divergent. (b) If linifc-^oo ^^ = q then the series J^'o' Uk *s convergent for q < 1 and divergent for q > 1. Proof. We t reat t he cases individually. 178 13 Power Series and Special Functions "1 U2 u-2 U0 U\ «n Un U n _! nil (a) We have un = "o U\ a nd therefore Mfc + l Uk 1 w Vra = 0 , 1 , 2 , . . . < q < 1 =>• un < u0qn, q < l ,Vn. Since the series ] T ^ «o<Zn is convergent, the series ^ ^ ° Uk is convergent. M fc+l Uk > 1 => M„ > wo, Vn. Since the series «o + «o + • • • is divergent, so is 53^° Uk(b) linife^oo ^ = ? < 1 ^ V £ > 0 3 ? + £ < 1 ; T O have 2±±i < q + e < 1, k > N, where iV is sufficiently large. Then, the series Y^N uk ' s convergent and hence so is Y^f wfc. On the other hand, l im fc-+oo Uk ^±l=q>l^^±l>1^k>N Uk for sufficiently large N, a nd therefore 53 o° u^ is divergent. • T heorem 13.1.10 (Cauchy's Test) Let 53o° Uk be a series with positive terms, (a) oo (uk)* < q < 1,VA; =£• the series 2_]uk *s convergent. o oo (uk)J > 1,Vfc =>• £/ie series ]>Uk is divergent. o (b) If'limfc_>oo(wfc)E = 9; £/&en *^e series £Zo°ufc «s convergent for q < 1 and divergent for q > 1. R emark 13.1.11 Lei a series be convergent to a sum S. Then, the series obtained from this series by rearranging and renumbering its terms in an arbitrary way is also convergent and has the same sum S. 1 3.2 Sequence of Functions D efinition 13.2.1 A sequence of functions ( /„}, n = 1 ,2,3,... converges uniformly on E to a function f if Ve > 0, 3N e N 3 n > N =» \fn(x) - f{x)\ < e, Vx e E. Similarly, we say that the series Y fn(x) converges uniformly on E if the sequence (Sn) of partial sums converges uniformly on E. 13.3 Power Series 179 R emark 13.2.2 Every uniformly convergent sequence is pointwise convergent. If ( /„) converges pointwise on E, then there exist a function f such that, for every e > 0 and for every x £ E, there is an integer N, depending on e and x, such that | /„(x) — f{x)\ < e holds if n > N; if ( /„) converges uniformly on E, it is possible, for each e > 0, to find one integer N which will do for all x £ E. P roposition 13.2.3 (Cauchy's uniform convergence) A sequence of functions, (fn), defined on E, converges uniformly on E if and only if Ve > 0 , 3N eN3m>N,n>N,x e E=> \fm{x) - fn{x)\ < e. C orollary 13.2.4 Suppose linin-yoo fn{x) = / ( x ) , x G E. Put M„ = s u p | / n ( x ) - / ( x ) | . xeE Then, fn — / uniformly on E if and only if Mn —• 0 as n —• oo. > > > P roposition 13.2.5 (Weierstrass) Suppose ( /„) is a sequence of functions defined on E, and | /(x)| < M „, x € E, n = 1, 2, 3 , . . . Then, ^ fn converges uniformly on E if ^2 Mn converges. P roposition 13.2.6 lim lim fn{t) = lim lim t~¥x n—+oo n—>oo t—>x fn(t). R emark 13.2.7 The above assertion means the following: Suppose / „—>•/ uniformly on a set E in a metric space. Let x be a limit point of E, and suppose that l imt_> x / n (i) — An, n = 1 ,2,3... Then, (An) converges, and > l i m ^ a f(t) = limn_>oo An. C orollary 13.2.8 / / (/„) is a sequence of continuous functions on E, and if fn —> f uniformly on E, then f is continuous on E. R emark 13.2.9 The converse is not true. A sequence of continuous functions may converge to a continuous function, although the convergence is not uniform. 1 3.3 Power Series D efinition 13.3.1 The functions of the form oo f( ) = YlCnxU n =0 x or more generally, f( ) = Y2cn(x-a)r ra=0 x oo are called analytic functions. 180 13 Power Series and Special Functions converges for \x\ < R, and T heorem 13.3.2 Suppose the series J2^=ocnxn define oo f(X) = YlcnXn, \X\<R n =0 which converges uniformly on [—R+e,R — e], no matter which e > 0 is chosen. The function f is continuous and differentiable in (—R,R), and oo f'{x) = Yjncn{x-a)n-\ n=l \x\<R C orollary 13.3.3 / has derivatives of all orders in (—R,R), which are given by f(k)(x) In particular, f(k\0) = k\ck, k = 0,1,2,... R emark 13.3.4 The above formula is very interesting. On one hand, it shows how we can determine the coefficients of the power series representation of f. On the other hand, if the coefficients are given, the values of derivatives of f at the center of the interval (—R,R) can be read off immediately. A function f may have derivatives of all order, but the power series need not to converge to f(x) for any x ^ 0. In this case, f cannot be expressed as a power series about the origin. T heorem 13.3.5 (Taylor's) Suppose, f(x) = Y^=ocnXn, the series converging in \x\ < R. If —R < a < R, then f can be expanded in a power series about the point x = a which converges in \x — a\ < R— \a\, and = J2 n{n - 1) • • • (n - k + l)cn(x n=k a)n~k. /(„-i;ffl2>(,-.,-. n =0 R emark 13.3.6 If two power series converge to the same function in (-R, then the two series must be identical. R), 1 3.4 Exponential and Logarithmic Functions We can define oo n n =0 It is one of the exercise questions to show that this series is convergent Vz G C. If we have an absolutely convergent (if |uo| + |ui| + • • • is convergent) series, we can multiply the series element by element. We can safely do it for E(z): 13.4 Exponential a nd L ogarithmic Functions 00 v; v 181 E(z)E(w)^ = n\ ^^ TO! ^ ^ =^E kUn -7k\) EE E ^-Tn ' n =0 °°i n =0 n yn °° ,.,m m =0 °° / °° n yk..,n-k n =0fc=0 , \rj V ' /\ V fc=0 ' n=0 T his yields • • E(z)E(-z) = E(z -z) = E(0) = 1, V* € C . J 5(«) ^ 0, V2 G C . E(x) > 0, Vx G ffi. -B(a;) — + 0 0 a s x - 4 + 0 0 . > 0 < x < y =» £?(ar) < £ ( y ) , £ ( - ? / ) < £ ( - x ) . H ence, £?(ir) is s trictly increasing o n t h e r eal axis. l i m ^ 0 * <«+*0-*('> = E(z). E{z\ + • • • + zn) = E{z\) • • • E(zn). L et us t ake z\ = • • • = zn = 1. S ince E(l) = e, we o btain E(n) — e ", n = 1 , 2 , 3 , . . . Furthermore, if p — n\m, w here n , m e N, t hen [ £ ( p ) ] m = £ ( m p ) = £ ( n ) = e n s o t h a t E(p) = ep, p 6 Q + . S ince E(—p) = e~p, p e Q + , t h e a bove equality holds for all r ational p . S ince xy — snpp€QBp<y xp, Vx, j / £ l , a; > 1, we define ex — s up p 6 Q 9 p < ; 2 . e p . T h e continuity a n d m onotonicity properties of E s how t h a t E(x) = ex = e xp(x). T hus, a s a s ummary, w e h ave t h e f ollowing proposition: P r o p o s i t i o n 1 3 . 4 . 1 The following (a) (b) (c) (d) (e) (f) are true: . • • ex is continuous and differentiable for all x, (e*)' = e*, ex is a strictly increasing function of x, and ex > 0, ex+y = exey, ex - > + 0 0 as x -) + 0 0 , ex — 0 as x — —00, > > l i m x ^ + 0 0 xne~x = 0, Vn. Proof. W e h ave already proved ( a) to (e). S ince ex > ?n+1y, for x > 0, t hen xne-x < {"±1)1 a n d ( f ) follows. D S ince E i s s trictly increasing a n d d ifferentiable o n R, it has a n i nverse f unction L w hich i s a lso strictly increasing a n d d ifferentiable whose domain i s E(R) = K +. E(L(y)) = y, y > 0 < > L(E(x)) £ =I,I£R. D ifferentiation yields £ ' ( £ ( * ) ) • E[x) = 1 = L'(y) • y « • L '(y) = - , j , > 0. y 182 13 Power Series and Special Functions x = 0 => L{1) — 0. Thus, we have L (y) = rv dx — = logyJi x Let u — E(x), v = E(y); L{uv) = L{E{x)E(y)) = L{E{x + y))=x + y = L(u) + L(v). We also have logx —> + oo as x — + oo and logs —• —oo as x —> 0. Moreover, > > x n = E(nL(x)), x G R +; n, m G N, Qlogx e x™ = £ ( — L (x) J x a = £ (aL(x)) = a , Va € Q. One can define x , for any real a and any x > 0 by using continuity and monotonicity of E and L. (xa)' = E{aL{x))One more property of log x is lim x~a l ogx = 0, Va > 0. x—»-f oo = ax"-1 1 3.5 Trigonometric Functions Let us define C(x) = \{E(ix) + E(-ix)}, z S(x) = ^[E(ix) zz E(-ix)}. By the definition of £(z), we know E(z) = £ (2). Then, C(x), S(x) G R, x G R. Furthermore, £ (ix) = C(x)+iS(x). T hus, C (x), 5(x) are real and imaginary parts of E(ix) if x G R. We have also | £(ix)| 2 = £ (ix)£(ix) = E(ix)E{-ix) = E(0) = 1. so that \E(ix)\ = 1, x G R. Moreover, C (0) = 1, 5(0) = 0; and C"(x) = - S ( x ) , S'(x) = C (x) 13.5 Trigonometric Functions 183 We assert that there exists positive numbers x such that C(x) = 0. Let XQ b e t he smallest among them. We define number 7r by 7T = 2XQ. T hen, C (§) = 0, and S(§) = ± 1 . Since C(x) > 0 in (0, § ), S is increasing in (0, § ); hence 5 ( f ) = 1. Therefore, and t he a ddition formula gives E(iri) = - 1 , E(2m) = 1; hence £ ( z + 27™) = £ :(z),V2eC. T heorem 13.5.1 The following are true: (a) The function E is periodic, with period 2-Ki. (b) The functions C and S are periodic, with period 2n. (c) If0<t< 2ir, then E{it) ^ 1. (d) IfzeCB \z\ = 1, 3 unique t e [0, 2TT) 3 E{it) = z. R emark 13.5.2 The curve 7 defined by 7(f) = E(it), 0 < t < 2w is a simple closed curve whose range is the unit circle in the plane. Since -f'(t) — iE(it), the length of 7 is JQ \j'(t)\ dt = 2w. This is the expected result for the circumference of a circle with radius 1. The point ^(t) describes a circular arc of length to as t increases from 0 to to. Consideration of the triangle whose vertices are z\ — 0, Z2 = j(to)> and Z3 = C(to) shows that C(t) and S(t) are indeed identical with cos(t) and sin(t) respectively, the latter are defined as ratios of sides of a right triangle. T he saying the complex field is algebraically complete means that every n onconstant polynomial with complex coefficients h as a complex root. T heorem 13.5.3 Suppose ao,..., an G C, n G N, an ^ 0, n 0 Then, P(z) = 0 for some z € C . Proof. W ithout loss of generality, we may assume that an = 1. P ut n = inf26C \P(z)\- If \A = R t hen \P{z)\ > Rn(l - {a^R-1 \a0\ R~n). 184 13 Power Series and Special Functions T he right hand side of the above inequality tends to oo as R —)• oo. Hence, 3i?o B \P(z)\ > n if \z\ > Ro- Since \P\ is continuous on the closed disc with center at the origin and radius RQ, it attains its minimum; i.e. 3ZQ 9 | P(^o)| — We claim that [i = 0. If not, put Q(z) = pZ)T hen, Q is nonconstant polynomial, Q(0) = 1, and |<5(z)| > 1, Vz. There is a smallest integer k, 1 < k < n such that Q{z) = 1 + bkzk + ••• + bnzn, bk ^ 0. By Theorem 13.5.1 (d), 9 e l 3 eik0bk = - \bk\. If r > 0 and rk \bk\ < 1, we have |l + bkrkeike\ = 1 - rk \bk\, so that | Q(re i 0 )| < 1 - rfc[|6fc| - r \bk+1\ rn~k \bn\\. For sufficiently small r, the expression in squared braces is positive; hence | <9(re iS )| < 1, Contradiction. Thus, \i = 0 = P(z0). D 1 3.6 Fourier Series D efinition 13.6.1 A trigonometric polynomial is a finite sum of the form N f(x) = ao + 2_j ian n=l c o s nx + bn sin nx), x 6 M, where ao,a\,..., a^, b\,..., 6 A € C One can rewrite T N -iV which is more convenient. It is clear that, every trigonometric polynomial is periodic, with period 2ir. R emark 13.6.2 J /rt € N, emx is the derivative of —.— which also has period 2n. Hence, n = 0, — / einx dr = l *' 2TT J ~ 1 0, n = ± l , ± 2 , . . . i /iye multiply f(x) by e~lmx where m £ Z, then if we integrate, we have 1 re mx dx for \m\ < N. Otherwise, \m\ > N, the integral above is zero. Therefore, the trigonometric polynomial is real if and only if C—n — Cxi, 7i — U , . . . , i V . 13.7 Gamma Function D efinition 13.6.3 A trigonometric series is a series of the form oo 185 f(x) = — oo Y,c"einx>xeR- If f is an integrable function on [—IT, IT], the numbers cm are called the Fourier coefficients of f, and the series formed with these coefficients is called the Fourier series of f. 1 3.7 G a m m a F unction D efinition 1 3.7.1 For 0 < x < oo, /»oo r(x)= is known as the gamma function. / Jo tx-le-ldt. P roposition 13.7.2 Let F(x) be defined above. (a) r(x + 1) = xr{x), 0 < x < oo. (b) r(n + 1) = n\, n < N. T(l) = 1. = (c) log-T is convex on ( 0,oo). P r o p o s i t i o n 1 3.7.3 If f is a positive function on (0, oo) such that (a) f(x+l) = xf(x), (b) / ( I ) = 1, (c) l o g / is convex. then f{x) = r{x). P r o p o s i t i o n 13.7.4 If x,y G R+, [\^{i-t)y^dt Jo This integral is so-called beta function R e m a r k 13.7.5 Let t = sin9, then = r}fr^. r(x + y) (3(x,y). r 2 f~2 ( sing) 2 - 1 (cosefy^de = Jo ' The special case x = y = | gives r(x + y) };f)r{y}. 186 13 Power Series and Special Functions R emark 13.7.6 Let t = s2 in the definition of T. / >oo r(x) = 2 Jo The special case x — \ gives s2x-le-a2 ds, 0 < x < oo. f This yields / »oo e~s ds — \Zn. ^>^(1H^) R emark 13.7.7 (Stirling's Formula) This provides a simple approximate expression for T(x + 1) when x is large. The formula is ^yx— 1 lim F^+S X *°° (f = 1. ) V2KX Problems 1 3.1. Prove Theorem 13.1.8, the comparison tests for nonnegative series. 1 3.2. Discuss the convergence and divergence of the following series: a) Eo°° £ b) ET f £ > w h e r e « > o c) E~(ei - 1) • OSTMi + i ) e) E r 9fc+V^> w h e r e V > ° 1 3.3. One can model every combinatorial problem (instance r) as ^PXJ = r, Xi e St C Z +. Let A^- = s Q j&Si 0, j # Si T hen, the power series i j-0 k=0 is known as the generating function, where the number of distinct solutions to ^2t Xi = r is the coefficient ar. We know that, one can write down a generating function for every combinatorial problem in such a way that ar is the number 13.7 Problems of solutions in a general instance r. Use generating functions to a) Prove the binomial theorem 187 u«>-£ (I) i=0 x ' and extend to the multinomial (you may not use the generating functions) t heorem < *+•••*>•= . i\,.. E .,ik € £+ (,„.".„») \-ik = n i\-\ b) Prove that {l + x + x2+x3 + ...)n = J2 »=0 n — 1 + ^> c) Find the probability of having a sum of 13 if we roll four distinct dice. d) Solve the following difference equation: an — 5 a„_i—6a n _2, Vn = 2, 3, 4 , . . . w ith ao — 2 and a\ = 5 as boundary conditions. 1 3.4. Consider the following air defense situation. There are i = 1 , . . . , / enemy air threats each to be engaged to one of the allied z — 1 , . . . , Z high value zones with a value of wz. T he probability that a threat (i) will destroy its target (z) is qiz. More than one threats can engage to a single zone. On the other h and, there are j = 1 , . . . , J allied air defense systems that can engage the incoming air threats. The single shot kill probability of an air defense missile fired by system j t o a threat i is Pji. Let the main integer decision variable be Xji indicating the number of missiles fired from system j t o threat i. a) Write down the nonlinear constraint if there is a threshold value di, t he minimum desired probability for destroying target i. Try to linearize it using one of the functions defined in this chapter. b) Let our objective function that maximizes the expected total weighted survival of the zones be max E * w*az (0), where az = Y\i 1 - qiz [U.j{l - Pji)Xji) = Hi Pizan Then, 7Z = l og(a z ) = J2% l °g(Az) = S i $iz d we have the second objective function: m&xJ2z Wzlz (0')- Isn't this equivalent to max^T^ wz J2i $iz (0")> where Siz = log 1 - qiz ( F T / 1 ~ Pji)Xji) ? S i n c e P i z = 1~9« \Ilj(l ~ Pji)Xsi) and we have m&x5iz — maxlog(/3iz) = m ax/3 i2 = m in(l - @iz) = m inlog(l - j3iz), 188 13 Power Series and Special Functions our fourth objective function (linear!) is min _3z wz _3i @iz (0"')> where 0iz = l og(l - piz) = log(qiz) + \J2j[\og(l -Pji)]xjij. Since we can d rop t he c onstants, \og(qiz), in the objective function, we will have t he fifth objective function as r n i n ^ wz ^ f 53 -[log(l - Pji)]xji) (0tv), which is not (clearly) equivalent t o the initial objective function in catching t he same optimum solution! W here is the flaw? (0)? = (0')? = (0")? = (0'")? = ( <H? W eb m aterial http://archives.math.utk.edu/visual.calculus/6/power.1/index.html http://archives.math.utk.edu/visual.calculus/6/series.4/index.html http://arxiv.org/PS_cache/math-ph/pdf/0402/0402037.pdf http://calclab.math.tamu.edU/~belmonte/ml52/L/ca/LA4.pdf http://calclab.math.tamu.edu/~belmonte/ml52/L/ca/LA5.pdf http://cr.yp.to/2005-261/bender1/IS.pdf http://education.nebrwesleyan.edu/Research/StudentTeachers/ secfall2001/Serinaldi/Chap*/.209/tsld009.htm http://en.wikipedia.org/wiki/Power_series http://en.wikipedia.org/wiki/Trigonometric_function* Series_definitions http://en.wikipedia.org/wiki/Wikipedia:WikiProject.Mathematics/ PlanetMath_Exchange/40-XX_Sequences,.series,.summability http://eom.springer.de/cZc026150.htm http://eom.springer.de/T/t094210.htm http://faculty.eicc.edu/bwood/mal55supplemental/ supplementalmal55.html http://home.att.net/"numericana/answer/analysis.htm http://kr.cs.ait.ac.th/~radok/math/matll/chap8.htm http://kr.cs.ait.ac.th/~radok/math/mat6/calc8.htm http://kr.cs.ait.ac.th/~radok/math/mat6/calc81.htm http://math.fullerton.edu/mathews/c2003/ ComplexGeometricSeriesMod.html http://math.fullerton.edu/mathews/n2003/ComplexFunTrigMod.html http://math.furman.edu/-dcs/book/c5pdf/sec57.pdf http://math.furman.edu/~dcs/book/c8pdf/sec87.pdf http://mathworld.wolfram.com/ConvergentSeries.html http://mathworld.wolfram.com/HarmonicSeries.html http://mathworld.wolfram.com/PowerSeries.html http://media.pearsoncmg.com/aw/aw_thomas_calculus_ll/topics/ sequences.htm http://motherhen.eng.buffalo.edu/MTH142/spring03/lec11.html http://oregonstate.edu/~peterseb/mth306/docs/306w2005_prob_l.pdf http://persweb.wabash.edu/facstaff/footer/Courses/Mlll-112/Handouts/ 13.8 Web m aterial 189 http://planetmath.org/encyclopedia/PowerSeries.html http://planetmath.org/encyclopedia/SlowerDivergentSeries.html http://shekel.j ct.ac.il/"math/tutorials/complex/node48.html http://sosmath.com/calculus/series/poseries/poseries.html http://syssci.atu.edu/math/faculty/finan/2924/cal92.pdf http: //tutorial. math. lamar. edu/AHBrowser s/2414/ ConvergenceOfSeries.asp http://web.mat.bham.ac.uk/R.W.Kaye/seqser/intro2series http://www.cs.unc.edu/~dorianm/academics/comp235/fourier http://www.du.edu/"etuttle/math/logs.htm http://www.ercangurvit.com/series/series.htm http://www.math.cmu.edu/~bobpego/21132/seriestools.pdf http://www.math.columbia.edu/~kimball/CalcII/w9.pdf http://www.math.Columbia.edu/~rf/precalc/narrative.pdf http://www.math.harvard.edu/~jay/writings/p-adicsl.pdf http://www.math.hmc.edu/calculus/tutorials/convergence/ http://www.math.mcgill.ca/labute/courses/255w03/L18.pdf http://www.math.niu.edu/~rus in/known-math/index/40-XX.html http://www.math.princeton.edu/"nelson/104/SequencesSeries.pdf http://www.math.ucla.edu/~elion/ta/33b.1.041/midterm2.pdf http://www.math.unh.edu/~j j p/radius/radius.html http://www.math.unl.edu/~webnotes/classes/class38/class38.htm http://www.math.uwo.ca/courses/0nline_calc_notes/081/unit6/Unit6.pdf http://www.math.wpi.edu/Course_Materials/MA1023B04/seq_ser/ nodel.html http://www.math2.org/math/expansion/tests.htm http: //www.math2. org/math/oddsends/complexity/e'/,5Eitheta.htm http://www.mathreference.com/lc-ser,intro.html http://www.maths.abdn.ac.uk/~igc/tch/ma2001/notes/node53.html http://www.maths.mq.edu.au/~wchen/lnfycfolder/fycl9-ps.pdf http://www.mecca.org/~halfacre/MATH/series.htm http://www.ms.uky.edu/~carl/ma330/sin/sinl.html http://www.pa.msu.edu/~stump/champ/10.pdf http://www.richland.edu/staff/amoshgi/m230/Fourier.pdf http://www.sosmath.com/calculus/improper/gamma/gamma.html http://www.sosmath.com/calculus/powser/powser01.html http://www.sosmath.com/calculus/series/poseries/poseries.html http: //www. stewartcal cuius. com/data/CALCULUS'/.20Early'/, 20Transcendentals/upfiles/FourierSeries5ET.pdf http://www4.ncsu.edu/~acherto/NCSU/MA241/sections81-5.pdf http://www42.homepage.villanova.edu/frederick.hartmann/Boundaries/ Boundaries.pdf www.cwru.edu/artsci/math/butler/notes/compar.pdf 14 S pecial Transformations In functional analysis, the Laplace transform is a powerful technique for analyzing linear time-invariant systems. In actual, physical systems, the Laplace transform is often interpreted as a transformation from the time-domain point of view, in which inputs and outputs are understood as functions of time, to t he frequency-domain point of view, where the same inputs and outputs are seen as functions of complex angular frequency, or radians per unit time. T his transformation not only provides a fundamentally different way to understand the behavior of the system, but it also drastically reduces the complexity of the mathematical calculations required to analyze the system. The Laplace transform has many important Operations Research applications as well as applications in control engineering, physics, optics, signal processing and probability theory. The Laplace transform is used to analyze continuoustime systems whereas its discrete-time counterpart is the Z transform. The Z transform among other applications is used frequently in discrete probability theory and stochastic processes, combinatorics and optimization. In this c hapter, we will present an overview of these transformations from differential/difference equation systems' viewpoint. 1 4.1 Differential Equations D efinition 14.1.1 An (ordinary) differential equation is an equation that can be written as: # ( t , j / , y ' , . . . ) j / ( n ) ) = 0. A solution of above is a continuous function y : I H> R where I is a real interval such that $(t, y, y',..., j / " ) ) = 0, W E I. A differential equation is a linear differential equation of order n if j /"> + an^(t)y^-^ + ••• + ai(t)y' + a0(t)y = b(t) where a n _ i , • • • ,a\, ao,b are continuous functions on I to K. Ificti — Ci, the above has constant coefficients. If b(t) — 0, Wt 6 / , then the above is called 192 14 Special Transformations homogeneous, otherwise it is non-homogeneous. If we assume 0 6 / , and 2/(0) = 2/o, y'(0) = y'o,...,y(-n~1)(0) = y{^'X) where y0,y'0,... ,y0n~1] are n (*) specified real numbers, this is called initial value problems where y$ 's are the prescribed initial values. E xample 14.1.2 (The 1*' and 2nd o rder linear initial value problems) y'(t) = a(t)y(t) + f(t), y(0) = y0; and for n — 2, the constant coefficient problem is y"(t) + aiy'(t) R emark 14.1.3 Let y(t) = yi(t) y'(t) = y2(t) ,(«-!) (t) = y„(t) 0 0 <> A = = 0 Ct0 + a0y(t) = b(t); y(0) = y0, y'(0) = y'0. y'1(t) = y2(t) y'2(t) = y3(t) y'n(t) = 1 0 0 —Oil -an-iyn{t) 0 •• atiy2{t) - a0yi(t) + b(t) 2/1 (*) 1 •• 0 •• -012 •• 0 0 2/2(0 . v(f) = 1 -«n-l. yn-i(t) yn(t) 0' 0 2/o 2/o 2/o = , / (*) = L2/0 ( n-2) 2/0 ( n-1) J 0 b(t). We have linear differential systems problem: y'(t) = Ay(t) + f{t); j/(0) = y0. 1 4.2 Laplace Transforms D efinition 14.2.1 The basic formula for the Laplace transformation y to rj is n(s Jo *»(*) dt. We call the function, n, the Laplace transform of y if Eteo € R 9 r](s) exists, Vs > XQ. We call y as the inverse-Laplace transform of n. V(s) = C{y(t)}, y(t) = C~l {V(s)} . 14.2 Laplace Transforms P roposition 14.2.2 //J/:RHR 193 satisfies (i) y{t) = 0fort<0, (ii) y(t) is piecewise continuous, (Hi) y(t) = 0{eXot) for some x0 6 M, then y(t) has a Laplace transform. Tables 14.1 and 14.2 contain Laplace transforms and its properties. Table 14.1. A Brief Table for Laplace Transforms Inverse Laplace Transform Valid s > xo z/W (1) (2) (3) 1 eat v(s) -J-,aeC s~a ' xp Ka t , m = 1 ,2,. m at ( 4 ) . t e , m = l ,2 sin bt (5) cos bt (6) ect sin dt (7) ect cos dt (8) m ,I$r b 0 s a +6 2 a (s-c)2+d2 ( 3 -c)" 2 +d 2 Table 14.2. Properties of Laplace Transforms (1) (2) (3) (4) (5)2/,e (6) (7) (8) (9) ft\ () Inverse y(t) ay(t) + bz(t) y'(t) y{n)(t) J 0, i < c where c > 0 ~\l, t>c se""»(s).<»>0 tmy(t), m = l , 2 , . . . L aplace Transform 7?(S) otj(a) + b((s) sri{s) - j/(0) a"»j(a) - a — y o ) e-c8 s 7 j(0S + &) (-l)m7?(m)(s) r ^Ct) /n J/(* - w)«(w) d" /s°° n(u) du v(s)C(s) R emark 14.2.3 If a = c+id is non-real, £{eat} = C{ect cos dt}+iC{edt then obtain Laplace transform using (2) in Table HA. sin dt} R emark 14.2.4 Proceed the following steps to solve an initial value problem: 194 14 Special Transformations 51. y(t)^V(s). 52. Solve the resulting linear algebraic equation, call the solution n(s) the formal Laplace transform of y(t). 53. Find the inverse-Laplace transform y(t). S4- Verify that y(t) is a solution. E xample 14.2.5 Find the solution to y'(t) = -4y(t) + f(t); where f(t) is the unit step function 2/(0) = 0, and I = [0,oo). Transforming both sides, we have sr)(s) - j/(0) = -4n(s) + sn(s) — -4n(s) H At the end of S2, we have n{s) = Sfs+A\ • 1 s(s + 4) Therefore, V(s) = -.e s s+ 4 e~s s e~ s . 1/1 4 \s 1 s + 4/ ' Thus, ^_/0, E xample 14.2.6 Let us solve y'(t) = ay(t) + f(t); 2/(0) = 0 t<l; such that y'(t) = f(t). Let us take y'(t) = f(t) then sn(s) — y0 — (f)(s), where 4>(s) = £ {/(<)}. Thus, V(s) = 2 / o - + - <A(s). s s We use formula (9) in Table 14-2. y(t) = 2/o + / f(u) du. Jo 14.2 Laplace Transforms / / we relax y'{t) = f(t), then we have 1 195 1 V(s) = Vos — a + and y(t) = eaty0+ (s) f ea^f(u)du; Jo where </>(s) is the Laplace transform of f(t). R emark 14.2.7 In order to solve the matrix equation, y'(t) = Ay(t) + f(t); we will take the Laplace transform as r](s)(sI-A) = yo+4>(s). 2/(0) = 2/o where n(s) = [r)i(s),--- ,rjn(s)]T is the vector of Laplace transforms of the components of y. If s is not an eigenvalue of A, then the coefficient matrix is nonsingular. Thus, for sufficiently large s V(s) = (si - Ay'yo + (si - A)-1^) where the matrix (si — A)-1 is called the resolvent matrix of A and for f{t) = 0. C(etA) = (si - A)'1 E xample 14.2.8 Let us take an example problem as Matrix exponentials. The problem of finding etA for an arbitrary square matrix A of order n can be solved by finding the Jordan form. For n > 3 , one should use a computer. However, we will show that how Cayley-Hamilton Theorem leads to another method for finding etA when n = 2. Let us take the following system of equations y[(t) = y2(t) + 1 t/i(0) = 3, J/2(*) = !/i(*) + < 2/2(0) = 1. Then, 01 10 S~1AS = , / (*) = 11 1 -1 etA01 10 2/o 11 1 -1 0 0 e' 10 0-1 s-1 _ 1" ~2 ec + e" 196 14 Special Transformations Then, the unique solution is y(t) = etAyo +p(t), where etAyo 2e* + e" 2e< - e~ and p(t) = / 0 V (e~ u + u e"") + e-'(e~" - ue - ")] du /o [e'(e-" + we"") + e ^ e " " + we" du Then, after integration we have p(t) = -2 v(t) 3et-t 3e*-2 One can sotoe i/ie akwe differential equation system using Laplace transforms: y'(t) = Ay(t) + f(t)^s s -1 -1 s »i(s)~ 01 10 l ^ i(s) ii + i *7i(«) <£> (*) Then, the resolvent matrix is (sI-A)-^ 1 (s-l)(s + l s1 1s (**) If we multiply both sides of (*) 6y (**), we /wroe T](S) = 1 ( s - i ) ( s + i) I s-l 3 s+ 1 Ls + 3 1 +s 2 ( s - l ) ( s + l) 1 s2 + l 2s 0 -2 T)(s) + y(*) = 3e(-£ 3 e' - 2 I n order to find e , we expand right hand side of (•*) as ["1 l l n(s) = 11 1 L2 2i J + s + 1 i 22 •\ 2- If we invert it, we will have the following •" = £ e* + e * el — e * 14.3 Difference Equations 197 1 4.3 Difference Equations Let us start with first-order difference equations: »<* + '> = » « + > « } Ay{k) = m k = h%^ T he initial value problem of the above equation can be solved by the following recurrence relation: y(k)=y(k+l)-f(k), Therefore, we find A = -1,-2,... : ( 2/o + £ *lS/(«)>* = 1,2,3,...; V(k) = < 2/o, k = 0; U - £ : = * / ( « ) , fc = - i , - 2 , . . . For second-order equations, we will consider first the homogeneous case: y(fc + 2) + axy{k + 1) + a0y(fc) = 0; j/(0) = yQ, 2/(1) = 2/1 • We seek constants Ai, A2 3 z(k + 1) = A2z(fc); z(0) = yi - Ait/0 which are the roots of A2 + a iA + ao = 0. If Ai 7^ A2, then y(k) = ciAf + C2A2 where Ci, c2 are the unique solutions of ci + c2 = yo, C1A1+ c2A2 = 2/1. If Ai = A2 = A, then y(k) = ci\h + c2Afc where ci,c 2 are the unique solutions of c\ — 2/o, ciA + c2A = yx. W hen the roots are non-real, A = pel6 and A = pe~l6, then y(k) — c\pk cos kO + c2/9fc sin k6, where c\ and c2 are the unique solutions of ci = j/oi cicos0 +C2sin# = 2/1. If we have systems of equations, y{k + 1) = ^ ( f c ) , fc = 0 , 1 , 2 , . . . ; 2/(0) = j / 0 , we, t hen, have as a recurrence relation 198 14 Special Transformations y(k) = Aky0 and A0 = I. W hen A is singular, there does not exist a unique solution y(—1) satisfying Ay( — 1) = 2/o- When yl is non-singular, j,(fc) = ^ - 1 J / ( A : + l ) . T hen, j / ( - l ) = A^yo, y (-2) = ^~22/o, ••• where ^- f c = A ^ A - ^ 1 = ( ^- 1 ) f c , A = 2 , 3 , . . . Recall that, if A = SJS'1 t hen Ak = S^S'1. T hen, y(k) = SJS-1y0, For the non-homogeneous case, y(fc + 1) = Aj/(fc) + /(fc). If >1 is nonsingular, k = 0,1,... 2/(fc) = ^fc!/o+p(*0, where p(fc + 1) = Ap(k) + f(k); p(0) = 0. This yields -l p(k) = E xample 14.3.1 For k = 0 , 1 , . . . , -YlAk~1-vf{.v). v—k Vi(k + l) = y2(k) + l, yi(0) = 3, y2{k + 1) = yi(k) + 1, 3/2(0) = 1. Ak = l 2+(-l)fe l + ( -l) f c l - ( - l ) f c , Aky0 = fc fc 2-(-l)fc 1 - ( -l) l + ( -l) fc-i p(*) = « £ «=0 A;-M + ( - l ) " ( 2 - f c + w) fc-u-(-l)u(2-fc + u) W^e know, > £<*_,,, >*G+I> M=0 and fc-i fc-i |(2 - *) ^C- )" + | S "(-1)" = | - §(-!) u =0 u=0 1 1 r 2fc2-3(-l)fc + 3 P(k) = _2k2 + 4k + 3{-l)k - 3 5 y(k) = k2 +k + (-!)* + 14.4 Z Transforms 199 1 4.4 Z Transforms D efinition 14.4.1 The Z Transformation y to rj is /^ V ^ V(U) oo , . , ~, rj(z) = 2 ^ ^-~, u=o 2 where z e C. We call the function rj the Z transform of y if 3 r 6 R 9 r](z) converges whenever \z\ > r, in such cases y is the inverse Z transform of n. n(z) = Z {y(t)} , y(t) = Z"1 {V(z)} . P roposition 14.4.2 If y satisfies (i) y(k) = 0fork = - 1 , - 2 , . . . , (ii) y(k) = 0(kn), neZ+, then y has a Z transform. If 7]{z) is the Z transform for some function \z\ > r, t hen that function is v(k). yW i^Jczk-lv(z)dz,k "\0, fc = 0,1,2,... = -l,-2,. where C is positively oriented cycle of radius r' > r and center at z — 0. For Z transform related information, please refer to Tables 14.3 and 14.4. R emark 14.4.3 Z {y(k + 1)} = j/(l) +'& The Laplace transform ofy'(t) + y^. + ... = Zri(z) - zy(Q). z zl is s'ij(s) —y(Q). R emark 14.4.4 The procedure to follow for using Z transforms to solve an initial value problem, is as follows: 51. 52. 53. S4y(k) ^ r,(z). Solve the resulting linear algebraic equation 'q(z) = Z {y(k)}. Find the inverse Z transform y(k) = Z~x {t]{z)}. Verify that y(k) is a solution. E xample 14.4.5 y(k + 1) = ay{k) + f(k), k = 0 , 1 , . . . ; y(0) = y0, a ^ 0 z y0 H 1 4>{z) = 7/1(2) + rj2(z). zrj(z) - zyQ = arj{z) + <j){z) => r/{z) = 200 14 Special Transformations T able 14.3. A Brief Table for Z t ransforms Valid Z transform r](z) -l (2) (3) y{k) 1 k k2 3 \z\ > r r (4) fc (5) k{m , m = 0 , 1, ak (6) kak (7) (8) (9) (10) ( 11) k\ e-ak u -1) 4 ( *-«) 2 it z z- e- ° z s in b 2 2 —2z c os fc+e"2ft 2 (2 —cos 6) 2 2 - 2 z c o s b + fi~2" ze""sinb z^ — 2ze~a c os 6 + e — 2 " z(z — ti'~" c o s fc) z 2 — 2 ^ e ~ a c os b-f-e~ 2 f l sin bk cos bk e-ak sm bk e~akcosbk (12) (13) T able 14.4. P roperties of Z t ransforms (1) (2) (3) (4) (5) (6) (7) (8) Inverse y(k) ayi(k) + by2(k) y(k + \) y(k + n) Z transform V(z) arii(z) + brj2(z) zri(z) - zy(0) znn{z) - zny(0) ~zn~]y(l) zy(n-l) «~c77(a) y(k-c), c> 0 k a y(k) ^ 7)( z ) ,d j ky{k) 4 rfz 2 k y(k) -^[-^'W] m 0) k y(k), m = 0 , 1 , 2 , . . . (-*£)%(*) (10) Y.ku=oV^k ~ u)y2{u) 7 71(2)772(2) (11) ^JrP'11l(/')')2(P^2)<i/' yi(fc)ya(fe) (12) £ « = 0 2/(U) ^(*) We know Then, by - 0(2) z—a ' superposition, 2— a -, and rj2{z) 2 V2(k) = f(k y(k) = aky0 + J2 f(k u=0 1 - ")«"• 14.4 Problems R emark 14.4.6 In order to solve the linear difference system y(k + l) = Ay(k) + f(k); y(0) = y0, 201 we will take the Z transform of the components of y(k), then we have (zI-A)rj(z) = zy0 + (t>{z), where n(z) = [771(2),-•• ,r]n(z)}T is the vector of Z transforms of the components of y. If z is not an eigenvalue of A, then the coefficient matrix is nonsingular. Thus, for sufficiently large \z\, the unique solution is V(z) = z(zl - A)-lVo + (zl - i 4)" V («), where we have Z{Ak}=z(zI-A)-1. In order to find a particular solution, we solve (zl — A) and find its inverse Z transform. 1 p(z) = 4>(z) forp(z) E xample 14.4.7 Let us take our previous example problem: yi(k + 1) = y2(k) + 1, 2/i(0) = 3, y2(k + l) = yi(k) + l, 2/2(0) = 1. V(z) = (z-l)(z + l) Z{Aky0} z1 [lz_ z-\ z + + ( *-l)(z + l) [ i f 2 2 z z+1 z 1 -1 1 zl C ^IFJ n(z) = z (z-ir (z-iy 1 ' 1' 4 3 .4. + (z-1) + ( -l) f c + (*+l) + 19 8 13 => y(k) = k2 +k " 0" 1 .2_ L4 J Problems 1 4.1. Solve y"(t) - y(t) = e2t; y(0) = 2, j ,'(0) = 0. 1 4.2. Solve y(k + 1) = y(k) + 2ek; y(0) = 1. 202 14 Special Transformations 1 4.3. Consider a combat situation between Blue (x) and Red (y) forces in which Blue is under a directed fire from Red at a rate of 0.2 Blue-units/unittime/Red-firer and Red is subjected to directed fire at a rate of 0.3 Redunits/unit-time/Blue-firer plus a non-combat loss (to be treated as self directed fire) at a rate of 0.1 Red-units/unit-time/Red-unit. Suppose that there are 50 Blue and 100 Red units initially. Find the surviving Red units at times t = 0 ,1,2,3,4 using the Laplace transformation. 1 4.4. F ind the closed form solution for the Fibonacci sequence Fk+2 — -Ffe+i + Ffc, F\ = 1, i 7 . = 1 using the ^-transformation and calculate -Fioo- Web material http://ccrma.stanford.edu/~jos/filters/Laplace_Transform_ Analysis.html http://claymore.engineer.gvsu.edu/~j ackh/books/model/chapters/ laplace.pdf http://cnx.org/content/ml0110/latest/ http://cnx.org/content/ml0549/latest/ http://dea.brunel.ac.uk/cmsp/Home_Saeed_Vaseghi/Chapter04-ZTransform.pdf http://dspcan.homestead.com/files/Ztran/zdiff1.htm http://dspcan.homestead.com/files/Ztran/zlap.htm http://en.wikipedia.org/wiki/Laplace_Transform http://en.wikipedia.org/wiki/Z-transform http://eom.springer.de/1/1057540.htm http://eom.springer.de/ZZz130010.htm http://fourier.eng.hmc.edu/el02/lectures/Z_Transform/ http://home.case.edu/"pjh4/MATH234/zTransform.pdf http://homepage.newschool.edu/~foleyd/GEC06289/laplace.pdf http://kwon3d.com/theory/filtering/ztrans.html http://lanoswww.epf1.ch/studinfo/courses/cours_dynsys/extras/ Smith(2002)_Introduction_to_Laplace_Transform_Analysis.pdf http://lorien.ncl.ac.uk/ming/dynamics/laplace.pdf http://math.fullerton.edu/mathews/c2003/ztransform/ZTransformBib/ Links/ZTransformBib_lnk_3.html http://math.fullerton.edu/mathews/c2003/ZTransformBib.html http://math.ut.ee/~toomas_l/harmonic_analysis/Fourier/node35.html http://mathworld.wolfram.com/LaplaceTransform.html http://mathworld.wolfram.com/Z-Transform.htm http://mywebpages.comcast.net/pgoodmann/EET357/Lectures/Lecture8.ppt http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-ComputerScience/6-003Fall-2003/LectureNotes/ http://phyastweb.la.asu.edu/phy501-shumway/notes/lec20.pdf http://planetmath.org/encyclopedia/LaplaceTransform.html http://umech.mit.edu/weiss/PDFfiles/lectures/lecl2wm.pdf http://umech.mit.edu/weiss/PDFfiles/lectures/lec5wm.pdf 14.5 W e b material 203 http://web.mit.edu/2.161/www/Handouts/ZLaplace.pdf http://www.absoluteastronomy.com/z/z-transform http://www.atp.ruhr-uni-bochum.de/rt1/syscontrol/node11.html http://www.atp.ruhr-uni-bochum.de/rtl/syscontrol/node6.html http://www.cbu.edu/~rprice/lectures/laplace.html http://www.cs.huj i.ac.il/"control/handouts/laplace_Boyd.pdf http://www.dspguide.com/ch33.htm http://www.ece.nmsu.edu/ctrlsys/help/lxprops.pdf http: //www. ece. rochester. edu/courses/ECE446/The'/,20z-transf orm. pdf http://www.ece.utexas.edu/~bevans/courses/ee313/lectures/ 15_Z_Transform/index.html http://www.ece.utexas.edu/~bevans/courses/ee313/lectures/ 18_Z_Laplace/index.html http://www.ee.Columbia.edu/"dpwe/e4810/lectures/L04-ztrans.pdf http://www.efunda.com/math/laplace_transform/index.cfm http://www.facstaff.bucknell.edu/mastascu/eControlHTML/Sampled/ Sampledl.html http://www.faqs.org/docs/sp/sp-142.html http://www.geo.Cornell.edu/geology/classes/brown/eas434/Notes/ Fourier%20family.doc http://www.intmath.com/Laplace/Laplace.php http://www.just.edu.j o/~hazem-ot/signal1.pdf http://www.ling.upenn.edu/courses/ling525/z.html http://www.ma.umist.ac.uk/kd/ma2ml/laplace.pdf http://www.maths.abdn.ac.uk/~igc/tch/engbook/node59.html http://www.maths.manchester.ac.uk/~kd/ma2ml/laplace.pdf http://www.plmsc.psu.edu/~www/matsc597/fourier/laplace/laplace.html http://www.realtime.net/~drwolf/papers/dissertation/nodell7.html http://www.roymech.co.uk/Related/Control/Laplace_Transforms.html http://www.sosmath.com/diffeq/laplace/basic/basic.html http://www.swarthmore.edu/NatSci/echeevel/Ref/Laplace/Table.html http://www.u-aizu.ac.jp/~qf-zhao/TEACHING/DSP/lec04.pdf http://www.u-aizu.ac.jp/"qf-zhao/TEACHING/DSP/lec05.pdf www.brunel.ac.uk/depts/ee/Research_Programme/COM/Home_Saeed_Vaseghi/ Chapter04-Z-Transform.pdf www.ee.ucr.edu/"yhua/eel41/lecture4.pdf S olutions 206 Solutions P roblems of Chapter 1 l.l (a) Since, / is continuous at x: Vei > 0 3Ji > 0 9 Vy 9 \x - y\ < ^ =» |/(x) - f(y)\ < ex. g is continuous at x: Ve2 > 0 382 > 0 3 \/y 9 |x - y\ < S2 => \g(x) - g(y)\ < e2. F ix ei and e2 a t | . 3tfi > 0 9 My 3 \x - y\ < 8, =• |/(x) - /(j/)| < | 35i > 0 3 \/y 3 |x - y\ < S2 => \g(x) - g(y)\ < | Let 8 = min{<5i,£2} > 0. My 9 |x - 2/| < J =• |/(x) - /(j/)| < | , | 5 (x) - g(j/)| < 1 l(/ + </)(*) - (/ + </)(*) I = I/O*) + <K*) - f(v) - g(v)\ < \f(x)-f(y)\ + \g(x)-g(y)\<^ + ^e T hus, Ve > 0 3d > 0 9 Vy 9 |x - 2/| < 8 =» | ( / + <?)(x) - (/ + fl)(j/)| < e. Therefore, / + g is continuous at x. (b) / is continuous at x: Vei > 0 3<5 > 0 9 My 3 |x - j / | < 8 =» |/(x) - /(j/)| < e. F ix e = e. Then, 3 J > 0 (say 8) 3 My (can fix at y) 9 |x — y\ < 8 =>• | /(x) — / (j/)| < e. We have |x — f/| < 5 =>• | /(x) — / (j/)| < £• Vy 9 |x - y\ < 8, | /(x) - f(y)\ <c\x-y\. Choose y 3 \x — y\ < 8, \f(x) — f(y)\ < c\x — y\ < c6. If < . c - >, we will reach the desired condition. One can choose 0 < 8 < min {£,§}. My 9 \x - y\ < 5 < 6, | /(x) - f(y)\ < c\x - y\ < cS < e. S olutions 207 1.2 Observation: Every time we break a piece, the total number of pieces is increased by one. When there is no pieces to break, each piece is a small s quare. At the beginning when we had the whole chocolate with n squares after 6=0 breaks, we had p—1 piece. After one break (6=1), we got p—2 pieces. Therefore, p is always greater by one than b, i.e. p = b + 1. In the end, p = b+ 1 = n. T he above argument constitutes a direct proof. Let us use induction to prove t hat the above observation b = n — 1 is correct. 1. n = 2 => b — 1, i.e. if there are only two squares, we clearly need one break. 2. Assume that for 2 < k < n — 1 squares it takes only k — 1 b reaks. In order t o break the chocolate bar with n squares, we first split into two with k\ and &2 squares {k\ + k2 = n). By the induction hypothesis, it will take ki — 1 b reaks to split the first bar and k^ — 1 t o split the second. Thus, t he total is b = 1 + (fci - 1) + (k2 - 1) = h + k2 - 1 = n - 1. 1.3 Full Forward Method: n\ ) n.! \ {n — r)\r\ n\ r\(n — r)\ 77.! in \n — r Combinatorial Method: (") denotes the number of different ways of selecting r objects out of n objects in an urn. If we look at the same phenomenon from the viewpoint of the objects left in the urn, the number of different ways of selecting n — r objects out of n is ( " ) • These two must be equal since we derive them from two viewpoints of the same phenomenon. C C) = C71) + O Full Backward Method: n-l\ rJ (n-l\ _ \r — lj » (n-1)! (n — 1 — r)\r{r — 1)! (n — 1)! [n — r + r] (n — r)\r\ (n - 1) (n — r)(n — r — 1)! (r — 1)! fn^ \r/ Combinatorial Method: (") denotes the number of different ways of selecting r balls out of n objects in 208 Solutions a n urn. Let us fix a ball, call it super ball. Two mutually exclusive alternatives exist; we either select the super ball or it stays in the urn. Given that the s uper ball is selected, the number of different ways of choosing r — 1 balls out of n— 1 is (™l|). In the case that the super ball is not selected, (n~1) denotes t he number of ways of choosing r balls out of n — 1. By the rule of sum, the right hand side is equal to the left hand side. («0(S) + G) + - + O = 2 » : We will use the corollary to the following theorem. T heorem S . l . l (Binomial Theorem) ( i + .)-=cy + cy + ... + (^ C orollary S.1.2 Let x = 1 in the Binomial Theorem. Then, Combinatorial Method: 2 " is the number of subsets of a set of size n. (JJ) = 1 is for the empty set, (^) = 1 is for the set itself, and ("), r = 2 , . . . , n — 1 is the number of proper subsets of size r. w a y (?) = (") (r_;)= Forward - Backward Method: n\ fm\ mj \r J n\fn — r\ r) \m — T) n\ m\ (n — m)\ ml (m — r)\r\ n\ (n — r)\ r\ (n — r )! (n — m )! (m — r )! n\ (n — m)\ (m — r)\r\ n\ r! (n — m)\ (m — r)\ Combinatorial Method: ( ^) denotes the number of different ways of selecting m I ndustrial Engineering s tudents out of n M .E.T.U. students and (™) denotes the number of different ways of selecting r Industrial Engineering students taking the Mathematics for O.R. course out of m I.E. students. On the other hand, (™) denotes the number of ways of selecting r Industrial Engineering students taking Mathematics for O.R. from among n M .E.T.U. students and (^1^.) denotes the number of different ways of selecting in — r I ndustrial Engineering students Solutions 209 not taking Mathematics for O.R. out of n — r M .E.T.U. students not taking M athematics for O.R. These two are equivalent. ( e)(s)+rr l )+-+( B r) = (B+rr+1): Trivial: Apply item (b) r-times to the right hand side. Combinatorial Method: T he right hand side, ( n + ^ + 1 ), denotes the number of different ways of selecting r balls out of m — n + 2 balls with repetition, known as the multi-set problem. Let | be the column separator if we reserve a column for each of m objects, let \J be used as the tally mark if the object in the associated column is selected. T hen, we have a string of size r + (m — 1) in which there are r tally marks and m — 1 column separators. For instance, if we have three objects {x, y, z}, and we sample four times, "\/l\A/lv / " means x and z are selected once and y is selected twice. Then, the problem is equivalent to selecting the places of r tally marks in the string of size r + (m — 1), which is ( r + ™ _ 1 ). Let us fix the super ball again. The left hand side is the list of the number of times that the super ball is selected in the above multi-set problem instance. T hat is, (Q) refers to the case in which the super ball is not selected, (n~J~ ) refers to the case in which the super ball is selected once, and ( n + r ) refers to t he case in which the super ball is always selected. These two are equivalent. 210 Solutions P roblems of C h a p t e r 2 2 .1 (a) 2 3 4 5 6 7 8 9 10 11 12 13 [A\\h] = 1 0000000 0 000000010 1 1 000000 0 011000010 001100001 000110000 000011000 000000101 000001100 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 1 0 a + b —> b; a + b —)• a; b + c —> c c + d —)• d\ ; d + e -» e; e + /->-/; f + g -^ g; g + i -> i; h + i -> i 0D 1 0000000 1 0000 0 10000000 0 1 000000 1 0000 1 10000000 0 0100000 1 0000 1 11000000 0 0010000 0 0000 1 11100000 0 0001000 0 1000 1 11110000 0 0000 100 0 10 11 1 11111000 0 00000 10 0 1 111 1 11111100 0 0000001 0 110 1 0 000000 10 0 0 0 0 0 0 0 0 0 0 0 0 0 1 11111111 [A\\h h\N c F ig. S.l. The tree T in Problem2.1 Each basis corresponds to a spanning tree T in G = (V,E), where T C E connects every vertex and ||T|| = ||V|| — 1. Here, we have T = Solutions { 1,2,3,4,5,6,7,8}. See Figure S.l. 211 (b) Each row represents a fundamental cocycle (cut) in the graph. In the tree, we term one node as root (node i), and we can associate an edge of the tree w ith every node like 1 -> b, 2 -¥ a, 3 - » c, 4 -4 d, • • • , 8 ->• h as if we hanged the tree to the wall by its root. Then, if the associated edge (say edge 6) in the tree for the node (say / ) in the identity part of Zi is removed, we p artition the nodes into two sets as V\ = {a,b,c,d,e,f} and V2 = {g,h,i}. T he nonzero entries in Zf correspond to edges 10,12,13, defining the set of edges connecting nodes in different parts of this partition or the cut. The set of such edges are termed as fundamental cocycle. See Figure S.2. F ig. S.2. The cocycle denned by cutting edge 6 —• / in Problem2.1 > (c) Each column represents a fundamental cycle. If we add the edge identified by 1$ p art into T, we will create a cycle defined by the nonzero elements of yi. See Figure S.3. (d) The first 8 columns of A form a basis for column space 11(A). T he columns of matrix Y is a basis for the null space Af(A). T he rows of C c onstitute a basis for the row space 1Z(AT). Finally, the row(s) of matrix D is (are) the basis vectors for the left-null space Af(AT). R emark S.2.1 If our graph G = (V,E) is bipartite, i.e. V = V1IJV2 9 Vif]V2 = 0, Vi ± 0 + V2 and Me = (vx,v2) G E, vi G Vu v2 G V2, and we solve m&x.cTx s.t. Ax = b, x > 0 using standard simplex algorithm over GF(2), we will have exactly what we know as the transportation simplex method. Furthermore, for general graphs G = (V, E), if we solve m axc T x s.t. 212 Solutions F ig. S.3. The fundamental cycle defined by edge 10 in Problem2.1 Ax = b, x > 0 using a standard simplex algorithm over GF(2), we will get the network simplex method. 2 .2 (a) 4 (5,2) = 00 2 0 00 0 6 00 0 0 00 0 0 00 00 12 0 0 20 [N\B] [B\N] = \UB\UN] -)• [h\VN] where 20 06 00 00 0 0' 00 12 0 0 20 "0 0" 00 00 00 UB = , uN = = 0 4x2, VN = 00 00 00 00 = 0 4x2- Then, TZ(A) = Span {2ei,6e2,12e3,20e4} T he rank of A(n, k) is r = 4. 1Z(AT) = S pan {2e3,6e4,12e5,20ee} = Span{e3,e4,e$,eQ} = R4. = Span {e±, e2, e%, e^} = R 4 . if1) Af(A) = Span < 0 0 0 0 1 (o\) 1 0 0 0 > — Span { ei,e2} I Voy1 T hus, R 6 = Tl(AT)®Af(A) J V(4 r ) = {0} , dimAf (A T ) = 0. = R 4 ©R 2 and R 4 = 1l{A)®N{AT) = R 4 0 0 = R4. WJ Solutions (b) Differentiator: 0 • • 0 n;=i» 0 A(n,k) = 0• • 0 0- • 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = l 213 o IlEj-'i o [N(n,k)\B(n,k)] III=n-fc+l [/„--*+i|0] J| . [B( n,k)\N(n,h)]-> T hen, K{A) = Span{ | Y[i J e i,--- , I V*=l i J e „_ fc+ i = Span{ei,.--,en_fc+1}=R"-fc+1. 7 e(A r ) = Sp«n<M J J i J e f c + x , - - , I I \i=l / JJ i\e„ / \j=n-fc+l = Span{ek+1,--0 A/"(i4) = Span < , e n } = R "- f c + 1 . /o\l > = Span{ei,--- ,ek} = '. IW T \ 0/J = 0. M{A ) = {0} , dimM(AT) 214 Solutions (c) Integrator: B(n,k) = Utii 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 I mir" 0 0 0 n" r After permuting some rows, we have n?=i< 0 0 0 o PB(n,k) = 0 0 0 o 0 0 n5 0 0 ^o H j = n~fc+1 ^ [PB(n,k)} where I\LI< \UB] -» 0 In-k+1 0 UE 0 0 0 0 0 0 0 0 rtfi'- * 0 0 1 0 0 0 0 0 1 0r T hus, K(B) = M.n-k+1 F urthermore, M(BT) = Rk and M{B) = {6} . =K(BT). Solutions 2 .3 215 1. Let n = 4 and characterize bases for the four fundamental subspaces r elated to A = [yi\y2\ • • • \yn]-10 0 1 1-10 0 0 1-10 00 1-1 1000 0 100 0 0 10 0001 10 0 - 1 - 1 00 0 0 1 0-1 -1-10 0 0 1 - 1 0 0 010 00 1-1 0 00 1 [A\\h -> 10 0-1 0 1 0-1 00-1 1 00 1-1 ' 1 0 0 - 1 - 1 0 00" 1 000] 00 0 10 - 1 - 1 - 1 1-10 0 -> 001 -1 -1 -1 -10 = 1 110 0 001 000 0 1 1 1 1 -1 -1 -1 Si = - 1 0 00 -1-1 00 -1 -1 -10 Af(A) = Span{t}, , Sn= where r/3|v^ 0 Si] SII\ where Vjv = [1111]. T hus, TZ(A) = Span{yi,y2,y3}. t= -vN h-3 1 1 1 1 Moreover, -1 0 0 1 1 -1 0 0 0 1 -1 0 K(A ) T = Span < 5 J Span{-y4,-y1,-y2} And finally, N(AT) = Span {Sn} = Span j [ 1 1 1 1 ] T \ . T he case for n = 3 is illustrated in Figure S.4. y\ is on the plane defined by Span{ei,e2}, y2 is on the plane defined by Span {e2,e^} a nd 2/3 is on the Span{ei,es}. Let us take {2/1,2/2} in the basis for 11(A), which defines the red plane on the right hand side of the figure. The normal to t he plane is defined by the basis vector of Af(A) = Span { [1,1,1] T }. We have M(A) = {K{A))L since N{A) = tf(AT) (therefore, K(AT) = H(A) by the Fundamental Theorem of Linear Algebra-Part 2) in this particular exercise. Let us discuss the general case. Let e = ( !,-•• , 1 ) T [A\\In] -» ITI-I\VN 0 Si] Sn 216 Solutions A t ,r ^ w 0' •**>* /, wis .ttT* R(A)=Span(yi.y2) F ig. S.4. The range and null spaces of A — [j/i |y2J2/3] -1 where Vjv -e, S , = -10 0 0" ) &II : '•• 0 _ 1 . . . _ i 0_ [ l , . . . , l ] = eT. T hus, 11(A) = Span {j/i, • • • , yn-i}- A/-(A) = Span {£}, where i—i *= r -vwi L Ji J i Moreover, 1l(AT) = Span {-yn, T -yi, ••• , -yn-2} • And finally, Af(A ) = S pan {5//} = Span {[1, • • • , 1 ] T } = Span {e} . We have Af(A) = ( ftM)- 1 - since A/"U) = -^(A- ) (therefore, 1Z(AT) = 11(A) by the Fundamental Theorem of Linear Algebra-part 2) in this particular exercise. Solutions 217 P roblems of Chapter 3 3 .1 A= 12 0-1 1-13 2 1-13 2 -1 1 - 3 1 [ a 1 a2 a3 a4] . Vi = a 1 1 1 -1 => vfvi = 4 , v~[a2 = - 1 , v\a3 = 9, vfa4 = 2. i>2 = a -vi T V2V2 — , t ^ c T = - 9 , v2a = --. 27 v3 = a ° •27-W2 - - J - « i - This result is acceptable since a3 = 2a1 — a2; h ence it is dependent on a1 a nd a2. W4 = a 4 - ^2 -jf-^ -2 4 —Wi T T hus, 2 V2 Qi 12 6 6 6J Qi o V4 _^ _^ 6 6 3J \\vi\ \v21 Kl a 1 = 2q\Vi a2 = -\qi 3 + h&q2 = -\vi+v2 % = 2 W l - «2 + =£v2 + vA a = 2 ( 2 f t ) - (-§(& + ^ ) < ? 2 = | g i + ^ a 4 = qi - A /392 - \ / 6 5 4 = \vx I 2 1 2 1 2 1 2 ^ 2 \/3 6 y /3 6 y/3 6 Q u <£> Q= . \/6 6 ,R= y/6 6 V6 3. r2 - i L 0 ^ 0 2 3 ^ 3 _ 3 ^ 32 X n _^ 0 0 \/6 218 3.2 Solutions V = A) + P\x + e =» E[y) =/30+ 0ix. D ata: yi= Po + P\x\ x 1 X\ "j/i" 1 x2 2/2 = Po + P\ 1 & _ J- X m _ '/So' 2/2 = _ 2/m_ « • A/3 = y. T he problem is to minimize SSE = ||y - Apf = E H i f a i - Po - Pixi)2 \Po' such that Ap is as close as possible t o y. T he solution is to choose P = Pi 1 xi 1 xi Vm = Po + P\Xm =* A J A = m Ex» , d et(,4 T A) = m ^ x 2 - ( ^ x* I > i E^ 2 (ATAy1 = Ex? -£*< m £ i f - ( E ^ ) 2 L -E^i m 0 = (^)-MTj, 2/1 P= 1 E^f -E^ I I ••• I *^1 **-*2 ' ' * •&m 2/2 0 = (ATA)-1ATy /?: = m Ex -(E i) TO X 2 x E 2/i E x i2/i x 2 A) (A'M)-1^ = E ?- (E *) E x1 E 2/J - E xi E xi2/i ~ E a* E 2« + "i E x,2/i / We know from statistics that Pi = ^PL,Po = y-Pix, bbx bbx where —^1, Since bbxx = / \Xj — x) = y ^ xi — 2x y ^ X•i m x 2 -f ; V= -, SSxy = ^2{xi-x)(yi-y), SSXX = ] P ( x ; - x ) ( x , - x ) . Solutions SSXX = ] P x2 - 2mx2 + mx2 = y ^ x ;2 Pi — -2 219 mx , TO -mSSxy __ - E x» E 2/i + ExiVi which is dictated by the matrix equation above. 0o = •v S S ^ - x SSxy _ y E A ~ myx2 - x ]T xtyi + myx2 A> = y E xl - x E ^2/i _ E ?/; E ^ - E ^ E ^2/* bbxx X mbbxx E ^ i E 2/i - E i E a^iJ/i A) " »!>?-Q><) 2 which is dictated by the matrix equation above. We may use calculus to solve min SSE: SSE = Hi/ - A/3||2 = £ ( W - [A, + /3iXi])2 S S£ = J2 Vf - 2 5 Z Vifa - 2/3i X) ^ * + m /3 ° + 2^>^ I ] Xi + # 1 3 x?dSSE d(3o -2 ^ i/j + 2m/30 + 2/3! ^ Xi =0 ^3 E E — - ax. «• /„0 = — 2/i - /?i —^t = y - f„ dSSE dpi = - 2 J2 xiVi + 2/3o J2 Xi + 2 & Yl x2i = ° mx mx «• 5 3 aiij/i - (?/ - /3ix) ^ Xj - ft ^ x2 = 0 a _ E x iVi -yYuxi 2-jXi ~~ x 2-jXi _ Ea:»yi /_jXi ~ v = ssxv bbxx As it can be observed above, the matrix system and the calculus minimization yield the same solution! Let the example data be (1,1), (2,4), (3,4), (4,4), (5,7). Then, "r 1 A/3 = y& 12 13 14 15 ra 1 PO " 1" 4 =4 4 7 = 10. A1 A = 5 15 15 55 13 , det{ATA) 3 11 220 Solutions $ • 00 = 0i 0o = 0i 1 (A<A)-Wy=^ 11 -3 11111 12345 0= {A1A)^A*y 10 ' 0.4' " 11 - 3 ' "20" '0o 72 = 1.2 = 01 -3 1 5 = 3, y = 4, SSxy = ( l - 3 ) ( l - 4 ) + ( 2 - 3 ) ( 4 - 4 ) + ( 3 - 3 ) ( 4 - 4 ) + ( 4 - 3 ) ( 4 - 4 ) + ( 5 - 3 ) ( 7 - 4 ) = 12, SSXX = (1 - 3) 2 + (2 - 3) 2 + (3 - 3) 2 + (4 - 3) 2 + (5 - 3) 2 = 10. 12 ft = - , # > = 4 - 1 . 2 ( 3 ) = 0.4. 3 .3 (a) Let us interchange the first two equations to get A[ = LU: "2 1 3" •2 A[ = 1 32 32 1 = 3 1 3 "Ml 10 00 "3 2 1" 0Ii u 1 0 Of_ 3 3 Here, the form of L is a bit different, but serves for the purpose. We solve LUx = 6'j = [19,8,3] T in two stages: Lc = b', t hen Ux = c. •2 Lc =b[ & 3 1 § 3 - Ml 10 0 0_ "ci" C2 __ 1 .C3. ' 19" 8 <> ^ 3 " 3" =*• c3 = 18. c2 = 7 ci = 3 => xi = 0. 32 1 nil Ux = c<=> U 33 00f_ F inal check: 'xi X2 .X3. _ 7 _18_ <£> => x 2 = - 2 x3 = 7 "132" A ix = 2 1 3 3 2 1_ " 8" 0' - 2 = 19 / 7 3_ Solutions (b) Let us take the first three columns of A2 as the basis: "2 1 3 " " 10" 1 32 , N = 01 32 1 _10_ Xi X4 , XB X2 Xi , XN 221 B = X5 Let XN — 0. T hen, BXB — b2 is solved by LU decomposition as above: " r22 - 1 i i Cl" C2 " 8" _ = 19 19 3 3" &< ci = 3 Lc = b2& k 3 1 3 7 L =*> C3 — y - 10 10 0 0_ Xi «2 c2 3. => c2 = 18 . C "321" £/x = c<£> 0 1 s U => X l = -- 00 XS X JV 3 T_ 3 = 18 60 .7. <> = ^3 Xz_ 10 3 X2 = f -B^NXN. "-23" = 1 1 16 1 0 ] T 3 ' 3' 3J • T If XJV ^ 6, t hen x# = [• -j±,f,f]T • 11 • 3 16 3 10 3. Let " = [1,1] T hen, Xl xB . X2 X = - 3. _5_ 18 _7_ 18 _i_ 18 • 11 • 3 16 3 10 3 1 L 1J _ 1 6 31 19_ F inal check: A2x = 2 13 10 1 3201 32 1 10 8" = 19 3 / 0 0_ -23 A2x = 2 1 310 1 320 1 32 1 10 31 19 6 6 19 3 / (c) " 1 2" A* = 45 78 . 10 11. A^A^ is clearly invertible, and (A3A3) l J A 1 4 7 10 3 2 5 8 11 AJA3 = 1 07 90 _47 45 166 188 188 214 = 47 " 45 83 90 222 Solutions A {AIM \ - i A1T = 107 90 47 45 83 90 _ iZ 45 1 4 7 10 2 5 8 11 J 10 7 15 1_ 30 2 5 4 5 13 30 J 15 3_ 10 x = (AUsr1ATb3 Xi X2 _9 10 7 15 L 30 2 5 4 5 13 30 J 15 _3_ 10 [21 5 6 34 " 30 53 30 ( 9) LsJ T he A3 — QR decomposition is given below: -0.07762 -0.83305 -0.39205 -0.38249 -0.31046 -0.45124 0.23763 0.80220 -0.54331 -0.06942 0.70087 -0.45693 -0.77615 0.31239 -0.54646 0.03722 - 12.8840 -14.5920 0.0000 -1.0413 0.0000 0.0000 0.0000 0.0000 Q= R= T he equivalent system Rx — QTb$ is solved below: - 0.07762 -0.31046 -0.54331 -0.77615 - 0.83305 -0.45124 -0.06942 0.31239 - 0.39205 0.23763 0.70087 -0.54646 - 0.38249 0.80220 -0.45693 0.03722 12.8840 --14.5920 0.0000 - 1.0413 0.0000 0.0000 0.0000 0.0000 Xi Qh T = - 11.1770 - 1.8397 0.2376 0.8022 - 11.1770 - 1.8397 0.2376 0.8022 = - 1.1333 Rx = Xi X2 -11.177-1.7667(-14.592) -12.884 *2 = E$S = 1-7667 (0) T he two solutions, (9) and (<C>), are equivalent. " 1 2" A3x = 45 78 .10 11. - 1.1333 1.7667 = 2.4201 4.3503 6.2805 8.2107 ^ "2" 5 6 .8. = 63 Solutions 223 "2" [2.4201 5 4.3503 P 3 a:-6|| = 6 6.2805 . 8. [8.2107 error. = 0.4201 -0.6497 0.2805 0.8495 = 0.8695 is the minimum (d) -10 01 1-1 00 0 1-10 0 0-11 -110 0 0-110 0 0-1-1 10 01 2-1 0-1 -1 2-1 0 0-1 2-1 -1 0-1 2 A4 = Ai = AiAt = Clearly, AJA4 is not invertible. Then, we resort to the singular value decomposition A4 = Q1SQ2 , where I '2 \/2 2 o| i 2 2 "4 0 0 0" 0200 ,£ = 0020 .0000. I Q2 _I 2 v^ 2 of i 2 2 0 -& Qi u 0- ^ u ^2 2 of V2 2 1 2 2 of \/2 1 22 T hen, x = Q2^Qjbi finds the solution: _1 2 1 _1 2 2 1 2 -^ IE = 0^0 1000 0100 00i 0 0 0 00 0-^ 1 2 2 0^ I 2 I I 2J 1 "2 /2 2 1 2 1 2 1" 2 0& ^2 2 1 2 U 0 n ^ /2 2 1 1 2 2J 0 1 2 2 4 3 3 Xi ^2 => X3 X4 0.22855 - 0.42678 - 0.25000 0.12500 224 Solutions P roblems of Chapter 4 4 . 1 In order to prove t h a t d et A = an An + a^A^ H h ainAin, ( property 11) where Aij's a re cofactors (Aij = ( — l ) , + J ' d e t M j j , where the m inor Mij is formed from A b y deleting row i a nd column j); w ithout loss of generality, we may assume t h a t i = 1. L et us apply some row operations, On «31 «22 «13 a Oln «2n «3n O i l «22 «13 0 0 --- 0,\n a2„ «21 ^22 «23 2 2 «33 022 «23 ' • • ->• a 3 2 Q33 • • • Ctzn O n l « n 2 On3 0 a „2 «n3 " • « r n. I n particular, a 2 2 = - a '* a » o »+ 0 «°n . w here atj = " a i j a ; 1 + a ^ a i 1 , i,j = 2,..., F urthermore, an 0 ^22 ^13 ' Q 22 C*23 • • 0,in a2n a3n an 0 —• «22 «13 • • C*22 Q23 • ' din OLln >!->• 0 • «22 «33 • • 0 0 0 / ? 3 3 • • fan 0 / 3„ 3 • • • Pn 0 an2 an3 • w here /3y = - " ' J 0 Otnn _ " ^ " " , M = 2 , . . . , n. I n particular, /333 = - 0:230:32 + «33Q22 «22 ( ai2«3i ~ fl32Qii)(a23aii - Q13Q21) + (033011 ~ Qi303i)(o220ii - 012O21) « ii(022011 - a i 2 a 2 i ) If we open u p the parentheses in the numerator, the terms without an c ancel e ach other, and if we factor an o ut and cancel with the same t e r m in the d enominator, we will have A»3 = Q 12Q23Q31 + Q13Q32Q21 — O l l Q 2 3 0 3 2 ~ 0 1 3 0 3 1 0 2 2 ~ 0 1 2 0 2 1 0 3 3 + Q11Q22Q33 ~a12a1l + 0 22&11 If we further continue the row operations to reach the upper triangular form, w e will have A-+ • an 0 0 0 022 «13 " Q 22 a • o i„ • 23 • ain 0 /333 • 0 0• Snn Solutions Let Cnn — Yz- T hus, det A = an- a22 • fe • • • (nn = au -Q12Q21 + 0 2 2 0 1 1 225 an 0 12023031 + 013Q32Q21 ~ Q11023032 ~ Q13Q31022 ~ 012021033 + Q11022033 -012021 +022011 "Z' zz Since the denominator of one term cancels the numerator of the previous term, d et A = Z = y , aipia2P2 peP *npt det [ e pi , eP2,..., ePn J, (*) where P h as all n\ p ermutations (pi,... ,pn) of the numbers { 1, 2 , . . . , n}, ePi is the pf1 canonical unit vector and det Pp = d et[e pi , eP2,..., ePn] = ± 1 such t hat the sign depends on whether the number of exchanges in the permutation m atrix Pp is even or odd. Consider the terms in the above formula for det A involving a n . They occur when the choice of the first column is p\ — 1 yielding some permutation P = (P2J • • • >Pn) °f the remaining numbers {2, 3 , . . . , n}. We collect all these t erms as An where the cofactor for a n is -An = y , pep E' a 2P2"-a'npn O-ei i p . Hence, det A should depend linearly on the row (an, 012,. • det A = a n An + 012^12 H h alnA\n. )Oi„): Let us prove Property 11 using the induction approach. The base condition was already be shown to be true by the example in the main text. We may use (*) as the induction hypothesis for n = k. Claim: J2PeP a2p2 " ' anpn d et Pp = (—1)1+1 d e t M n . We will use induction for proving the claim. Base(n = 3): An = 022033 - a 2 3a 32 = ( -l) 2 ** 2 2 ° 2 3 . 0 32 033 I nduction(n = k + 1): Y.^pa2,P2 • • • ak+i,Pk+1 det Pp- = ( - 1 ) 1 + 1 d e t M n Using the induction hypothesis for n — k in (*) we have: det M n = a 2 2A 22 H h a2n^2n, in which we may use the induction hypothesis of the claim for the cofactor A2j- T he rest is almost trivial. 226 Solutions 4 .2 Let "1 1 21 A= 0 1 1 -1 2 -2 -1 -1 -1 121 1 0 -1 131 224 d(s) 2 ) 5 , fc = 1, Ax = 2 , m = 5 . A i = A - 11 = -1 1 -1 -1 -1 2-1121 0 1-1 0-1 1-1111 2-2222 => dimN{A{) = 5 - rank^) A\ = 0 => dimN{A\) Choose v2 e J V(-AI) = 5 - 3 = 2. = 5 => mj = 2, m(s) = (s - 2) 2 . 9 Axv2 ^ 0. = (-l,2,0,l,2)T. « 2 = e ? = ( l , 0 , 0 , 0 , 0 ) r => v1=A1v2 Choose v4 ^ OT2 3 a ^ 0, v4 e Af(Al) 3 Axv2 / 6. v4 = e\ = ( 0,1,0,0,0) 2 V3 AlV4 = (l,-l,l,-l,2)T. Choose W € Af(Ai) i ndependent from v\ and vz5 V5 = ( 1 , 0 , 0 ) - 1 , 0 ) T . T hus, - 1 1 10 1 2 0-11 0 00 10 0 10-10-1 2 0-20 0 "2 1 2 S-lAS = 21 2 2 5= 4.3 -A- -i- 0 " 10 lo r A= 0 —— v =* d(s) = ( a - — J ,fc= 1,A = 1 10 10 To'n =3 0 0 *>. "o ^ o" A1 = A-l/ dimM(Ai) 0 0i 000 = 3 - ranfc(A1) = 3 - 2 = 1. Solutions 227 A2 = Al = (A--lf => dimM(Ai) = 00^ 00 0 00 0 = 3 - 1 = 2. = 3 - rank(Ai) A\ = 0 => rfim7V(yl?) = 3 =>• m = 3, m(s) = f s - — J . Choose v3 e Af(Al) 9 » 2 = AiV3 ^6 ^ Afv3 = uit;3 = e^ = ( 0,0,l) J => v2 = A1v3= T hus, 1 00 0 ,— ,0 1 0' Vi = A\V2 Vioo , 0,0 5= 0^0 0 01 ool => S~lAS in 10 -1- 10 A 10 = 10 10 1 10 45 0 1 10 = 001 • 1 100 00" 0 TO 0 0 0 1_ 10 100 10 1 10 J10 J = S yl 1 0 ^- 1 . Note that t he calculation of A10 is as h ard as t hat of A10 since A is not diagonal. However, because (easy t o prove by induction) [A 1 1 n A1 = A we have A 10 A" (i)A"- 1 Q )A"- 2 A" (^)A"- 1 An (^)°io(^)9«(^)8 = ( TO) 1 0 IO(^) (TO)10. 10 10 1 100 4500 0 1 100 00 1 Hence, it is still useful t o have Jordan decomposition. 4 .4 (a) —— = - 0.03Yi - 0.02F2 — - = - 0.04Yi - 0 .0iy 2 at at ^ = - 0.05.Yi - 0.02X2 ~ = - 0.03Xi - 0.00X2 at at = [Xi,X2,Yi,Y2}. T hen, t he above equation is r ewritten as dW = AW, dt Let WT 228 where Solutions ' A= L 0 0 1 20 3 100 0 0 1 50 3 100 1 25 1 -1 50 1 100 0 0 0 0 0 and the initial condition is W0 = [ 100,60,40,30] T . (b) A = SAS-\ where 0.46791 -0.46791 -0.20890 -0.20890 0.54010 -0.54010 0.69374 0.69374 0.64713 0.64713 0.33092 -0.33092 0.26563 0.26563 -0.60464 0.60464 0.79296 0.23878 0.63090 0.34529 -0.79296 -0.23878 0.63090 0.34529 -0.61736 0.53484 0.27717 -0.67525 -0.61736 0.53484 -0.27717 0.67525 S= -0.052845 0.000000 0.000000 0.000000 0.000000 0.052845 0.000000 0.000000 and A — 0.000000 0.000000 -0.010365 0.000000 0.000000 0.000000 0.000000 0.010365 AtC-U T he solution is W = SeMS~lW0 X1(t) Yi(t) Y2(t) 0.46791 -0.46791 -0.20890 -0.20890 0.54010 -0.54010 0.69374 0.69374 0.64713 0.64713 0.33092 -0.33092 0.26563 0.26563 -0.60464 0.60464 - 0.052845 t „ 0.052845 t -0.010365t „ 0.010365« 0.79296 0.23878 0.63090 0.34529 -0.79296 -0.23878 0.63090 0.34529 -0.61736 0.53484 0.27717 -0.67525 -0.61736 0.53484 -0.27717 0.67525 129.220 -58.028 -38.816 -20.475 100 60 40 30 Since 5 _ 1 W 0 = w e have Solutions ' *!(*)* X2(t) Yi(t) .46791 -.46791 -.20890 -.20890 .54010 -.54010 .69374 .69374 .64713 .64713 .33092 -.33092 .26563 .26563 -.60464 .60464 ( 129.22)e- 0052845 * (-58.028)e0052845t (-38.816)e-°-010365' (-20.475)e0010365t 229 Mt) X2(Q) Yi(0) y2(o) ' *i(3)" "100.0000' 60.0000 40.0000 30.0000 'Xi(lY J xa(i) Yi(l) Y*(l) "98.3222' 58.2381 33.8610 27.0258 "94.7227" 54.5719 16.0369 18.3547 i -X1(2)' X2(2) Yi(2) Y2(2) "96.8859" 56.7490 27.8324 24.0983 "93.9900" 53.8772 10.2360 15.5246 X 2 (3) Yi(3) y 2 (3) "95.6871' ' X 1 (4)' 55.5282 X 2 (4) 21.8967 ; ^ ( 4 ) 21.2102 y 2 (4) 'XiW ? X2(5) Yi(5) r2(5) 230 Solutions P roblems of Chapter 5 5 .1 Proof. Let Q~lAQ = A and Q~l = QT, x = Qy => R(x) = yTAy yTy Xiyf + • • • + XnVl vi + '-' + vl 2/1 = 1,2/2 = • • • = yn = 0 => Ai < i?(x) since Ai(2/f + ' •' + Vl) < Aiy? + •' • + A„2/£ «= Ai = min { AJf = 1 . Similarly, A„(A) — m ax|| x || = i xTAx. 5 .2 i. xTAx > 0, Vx ^ 0; "2 1 0' x r .4x = [xix 2 x 3 ] — — 12 1 100 0 1 1 Xi ^2 . 3. X D — [2x1 + xix2 + xxx2 + 2x2 + ^2X3 + x 2 x 3 + X3] 100 - ^ [(xi + x2f + ( x 2 + x 3 ) 2 + xi] > 0, Vx ^ 6\ ii. All the eigen values of A satisfy A, > 0; det(,I-A) = — 0 100s - 2 -1 = 0<£> -1 100s-2 -1 0 -1 100s - 1 s 3 -0.05s 2 +0.0006s-0.000001 = (s-0.002)(s-0.01552)(s-0.03248) = 0 => Ai = 0.002 > 0, A2 = 0.01552 > 0, A3 = 0.03248 > 0! iii. All the submatrices Ak have nonnegative determinants; Since each entry of A is nonnegative, all 1 x 1 minors are OK. 21 20 = 2 >0, 10 = 3 >0, = 1>0 11 12 21 21 10 20 = 2 >0, = 2 >0, = 1 >0 01 01 11 12 21 11 = 1 > 0,1 1= 2 > 0 01 = 1 >0, 01 All 2 x 2 minors are OK. Solutions 210 6 1 2 1 = 1 = 1 0 det(^) > 0! 0 11 T he 3 x 3 minor, itself, is OK as well, iv. All the pivots (without row exchanges) satisfy di > 0; "2 1 0 ' "2 1 0" 1 2 1 <-* 0 | 1 01 1 0 1 1_ di 231 <-)• L °°|J "2 1 0" 0*1 T |>0,d^A>o,rf3 =^>0! v. 3 a possibly singular matrix W B A = WTW; A 1 "2 1 0" 121 01 1 100 1 10 01 1 loo i -{i 1L dxi dx2 "1 1 0" 0 11 001 }(i "100" 1 10 01 1 = WTW and W = 5.3 is nonsingular! V /(x) = x\ + Xi + 2X2 2xx + x2 - 1 (an - l)(ari - 2) = 0 xi = 1 — 2 xi Therefore, xA — 1 - 1 , xB = 2 -3 are stationary points inside t he region defined by — 4 < x 2 < 0 < x i < 3. Moreover, we have t he following boundaries xi = "0 " L X2 , XII = '3" and xin = Xi \ 1^2 J [ ol -4 , Xiv Xl 0 defined by xc = [ol 0 , xD = - 4_ , xE = 0 [3] , XF = [ 3] -4 Let t he Hessian matrix be 2 V f(x) = 2 dx\dx\ dx\dx^ J_ af dxzdxi Qx^dx^ 2 2xi + 1 2 2 1 232 Solutions T hen, we have V2/(*A) 32 21 and V2f(xB) = 52 21 Let us check the positive definiteness of V2f(xA) vTV2f{xA)v = [vi,vi] 32 21 using the definition: Zv{ + 4«iv2 + v\ If Vl = - 0.5 and v2 = 1.0, we will have v T V 2 /(x A )w < 0. On the other hand, if vi = 1.5 and v2 = 1.0, we will have vTV2f(xA)v > 0. Thus, S72f(xA) is indefinite. Let us check V 2 / ( X B ) : v T V 2 /(x B )w = [vi,i>2] V2/(XB) 52 21 = 5v2 + 4viv2 + v\ = v\ + (2wi + v 2 ) 2 > 0. is a local minimizer with T hus, f{xB) is positive definite and xB = 19.166667. F ig. S.5. Plot of f(xi,x2) = \x\ + \x\ + 2xix2 + \x\ - x2 + 19 Solutions Let us check the boundary denned by xj\ / (0, X2) = l-x\ - X2 + 19 =» ^ ^ ^ = a* - 1 = 0 => Z2 = 1. 233 Since — £.? = 1 > 0, x2 = 1 > 0 is the local minimizer outside the feasible region. As the first derivative is negative for — 4 < x2 < 0, we will check x2 = 0 for minimizer and x2 = —4 for maximizer (see Figure S.5). Let us check the boundary denned by xn'- Since —{> f2' = 1 > 0, x2 = — 5 < —4 is the local minimizer outside the feasible region. As the first derivative is positive for — 4 < x2 < 0, we will check x2 = —4 for minimizer and £2 = 0 for maximizer (see Figure S.5). Let us check the boundary defined by xju: f(Xl,0) Since d2f = ±x* + \x\ + 19 =* ^ - f t =x21+x1=0=>x1=0, -1. £i'0) = 2xx + 1, a;i = 0 is the local minimizer ( d2 ^° s ' 0) = 1 > 0) on the boundary, and x\ = —1 is the local maximizer ( —^~ 2 ' ' = — 1 < 0) outside the feasible region. As the first derivative is positive for 0 < x2 < 3, we will check x2 — 3 for maximizer (see Figure S.5). Let us check the boundary denned by x/y: / (an, - 4 ) = \x\ + \x\ - 8Xl + 31 => ^ rf/( 2'~4) = ^ + ^1-8 = 0 Xl = _ l ± v / l + 32 . _ Since c'2/(jcxV~4) = 2xx + 1 again, the positive root xi = =i±sM = 2.3723 is the local minimizer (— dxi— > 0), and the negative root is the local maximizer but it is outside the feasible region. As the first derivative is positive for 0 < x2 < 3, we will check x2 = 3 for maximizer again (see Figure S.5). To sum up, we have to consider ( 2 , - 3 ) , (0,0) and (2.3723,-4) for the minimizer; (3,0) and (0, —4) for the maximizer: / ( 2 , - 3 ) = 19.16667, /(0,0) = 19, /(2.3723,-4) = 19.28529 => ( 0,0) is the minimizer! / (3,0) = 32.5, /(0, - 4 ) = 31 => (3,0) is the maximizer! 234 Solutions P roblems of Chapter 6 6 .1 The norm of a matrix A is denned as ||.4|| = -y/largest eigen value of AT A. If Q is orthogonal then QT = Q~l •£> QTQ = I and the unique eigen value of QTQ is 1. Hence \\Q\\ = \\QT\\ = i. F urthermore, c=\\Q\\\\Q-1\\ Hence for orthogonal matrices, c = | | Q | | = l. Let Q = aQ. T hen QT T hus, = \\Q\\2 = l. IIS|| = | |Q T || = i, and C=||Q||||Q1 | | = a | | Q | | i | | Q | | = | |Q|| 2 = l . For orthogonal matrices, ||Q|| = c(Q) = 1. Orthogonal matrices and their multipliers (aQ) are only perfect condition matrices. It is left as an exercise t o prove the only p art. 6 .2 A = QQRQ, where ' -0.4083 -0.3762 --0.5443 0.5452 -0.3020 0.0843' 0.9129 -0.1882 --0.2434 0.2438 -0.1351 0.0377 0 0.9111 --0.2696 0.2701 -0.1496 0.0418 Qo = 0 0 --0.7562 -0.5672 0.3142 -0.0877 0 - 0.4986 -0.8349 0.2331 0 0 0 0 0.2689 0.9632 0 0 - 1.2247 83.7098 - 73.0929 0 0 0 0 - 87.8778 87.3454 3.8183 0 0 0 0 - 5.5417 -3.0895 -0.1695 0 0 0 0 - 0.4497 -0.1898 0.0050 0 0 0 0 - 0.0372 0.0095 0 0 0 0 0 0.0016 - 76.9159 80.2207 0 0 0 0 80.2207 94.3687 -5.0493 0 0 0 0 - 5.0493 3.8305 0.3400 0 0 0 0 0.3400 0.3497 0.0185 0 0 0 0 0.0185 0.0336 0.0004 0 0 0 0 0.0004 0.0016 i?0 = Ax = R0Qo = Ai = QiRi, where Solutions - 0.6921 -0.5964 -0.3911 0.1109 -0.0079 0.7218 -0.5718 -0.3750 0.1063 -0.0076 0 0.5633 -0.7948 0.2253 -0.0161 0 0 - 0.2734 -0.9595 0.0685 0 0 0 -0.0712 -0.9974 0 - 0.0135 0 0 0 -0.0001 -0.0001 -0.0002 0.0009 -0.2331 0.9999 235 Qx 0 - 111.1369 123.6364 -3.6447 0 0 0 - 8.9636 5.0452 0.1915 0 0 0 - 1.2438 -3.0895 -0.0051 0 0 Ri = 0 0 - 0.4497 -0.0202 0 0 0 0 0 -0.0322 -0.0005 0 0 0 0.0016 0 0 0 0" 166.1589 -6.4701 0 0 0 0 - 6.4701 7.9677 -0.7006 0 0 0 0 - 0.7006 1.0885 0.0711 0 A2 = RiQi 0 0 0 0.0711 0.2511 0.0023 0 0 0 0 0.0023 0.0322 0 0.0016 0 0 0 0 R5Q5 0 0 0 0 166.4231 0 7.7768 -0.0002 0 0 0 0 - 0.0002 1.0218 0.0002 0 0 0 0 0.0002 0.2447 0 0 0 0 0 0 0.0321 0 0 0 0 0 0 0.0016 ^ 6 = QeR&, where -1.0000 0 0 0 0 -1.0000 0 0 0 0 - 1.0000 0.0002 0 0 - 0.0002 -1.0000 0 0 0 - 1.0000 0 0 0 0 1.0000 -166.4231 0 0 0 0 - 7.7768 0.0002 0 0 0 - 1.0218 -0.0003 0 0 0 - 0.2447 0 0 0 0 0 0 0 0 0 Qe -Rfi -0.0321 0 0.0016 236 Solutions 0 0 166.4231 0 0 0 0 0 0 7.7768 0 0 0 0 0 - 0.0002 1.0218 0.0001 0 0 0.0001 0.2447 0 0 1 0 0 0 0.0321 0 0.0016 0 0 A-j = ReQe = A7 = Q7R7, where 0 0 0 - 1.0000 0 0 0 0 0 0 0 - 1.0000 0 0 0 0 - 1.0000 0.0001 0 0 0 0 - 0.0001 -1.0000 0 - 1.0000 0 0 0 0 0 0 0 0 0 1.0000 0 " -166.4231 0 0 0 0 0.0002 0 0 0 0 - 7.7768 0 0 - 1.0218 -0.0001 0 0 0 0 0 - 0.2447 0 0 0 - 0.0321 0 0 0 0 0 0 0 0 0 0.0016 0 0 "166.4231 0 0 0 0 0 7.7768 0 0 0 0 1.0218 0 0 0 0 1jQr = 0 0 0 0.2447 0 0 0 0 0.0321 0 0 0 0 0 0 0 0 0.0016 T he diagonal entries are the eigen values of A. 6 .3 (a) Take .4(2). 1. A(2) = 1 11 .2 3_ Qr = R7 [A(2)\h] = .2 ill10 3T 1 " l | | 1 0" <-> 0 ^ l - 1 1 y 12I 21. <-)• " 10 4-6" 0 lj—6 12 i = A(2) " 1 6 / = = [h\A(2)-\ "1.0" 0.5 = 1" 0 4 -6' - 6 12 Solutions xn = A(2)-%I Ab = bi - bu = - 0.5 - 0.5 = 4 -6 - 6 12 1.5 1.0 237 =HIA,II = \H VlO, ||a;/|| = VT=» \\xi\\ xi - xu f -3 is T hen, the relative error for this case is s-1 l ^ 2 = 5.0. 2. T he maximum error is the condition number d et(sJ - 4(2)) = = (-D(-J)-^o 13 " *2 = 6 -v Therefore, c[4(2)] = % = ^ § = £ f f f § = 19.2815 is the upper 4-v/l3 b ound. 3. = >Ai = 4 + \/T3 ^ • 4 (2) + AA(2) hxin Ax = xui -xi - = 1| 1 I 0 23 + 2" I1 1.0 0.5 = bi => xiu = bi 2.0 0.5 =$• Ax\\ _ y / 425 = 1.84391 \xi + Ax\\ ~ y/h25 4+ | |4(2)|| = A2 = | AA(2)|| / 13 = 1.26759 , which is 0.9343. Then, is the largest eigenvalue of 1 '2 2 3 M(2)| P(2)| 0.9343 1.2676 0.7371 \\xr + Ax i M2)\ 1.84391 = 2.5017 0.7371 l l^(2)|| 4. T he maximum error is ||4(2)|| ||4(2) _ 1 ||, where ||4(2) _ 1 ||is the largest eigenvalue of 4 ( 2 ) _ 1 as calculated below: det(s/-4(2)-1) s-4 6 6 = (s-4)(s-12)-36 = 0 s-12 238 Solutions => Hi = 16 - V208 M2 = 16 + V208 T hen, ||yl(2)|| | | ^ ( 2 ) _ 1 | | = 1.2676(15.2111) = 19.2815 = c[A(2)}. We know, fi\ = j - and ji2 — j ~ - Consequently, c[A(2)-1} = ^ /^i = 19.2815 = AL = ^ IT A = c [^( 2 )] i (b) Take .4(3). ri 4 (3) 1 1I •2 3 i11 234 111 L3 4 5 :3)rA(3) = J s '49 3 21 36 4 40 3 61 3 4 144 10 21 3 769 . 40 10 3600 3 4 61 144 21 40 3 10 769 3600 _ * d et(aJ - A{3)TA{3)) = 49 36 3„ 4a 21 40 =0 •3- s 10 * (s - 3/415409)(s - 255/17041)(s - 1192/601) = 0 ; v\ 415409' vi 255 1192 17041 vz = 60T 1 192 6 01 4 15409 v • c[A(3)M(3)] = ^z = = 274635.3 1AA A d et(s/ - A(3)) = I I I 23 =0 745/529) = 0 =*• ^ i = J{2£- = 524.0566 \ 1 234 ill (sAi = 26/9675) (s Aa 34 3 89/3180)(s 5 389 26 A2 = 3 180' 9 675' T 745 529 2 c[A(3)] 2b 9 675 Clearly, c[A(3) A(3)} = ( c[4(3)]) . (c) Take A(A). m= Il 45 I1 56 Solutions - 0.83812 0.52265 -0.15397 -0.02631 -0.41906 -0.44171 0.72775 0.31568 -0.27937 -0.52882 -0.13951 -0.78920 -0.20953 -0.50207 -0.65361 0.52613 239 -4(4) = Q0Ro = -1.19320 -0.67049 -0.47493 -0.36984 0.00000 -0.11853 -0.12566 -0.11754 0.00000 0.00000 -0.00622 -0.00957 0.00000 0.00000 0.00000 0.00019 4(4)i = RoQo 1.49110 0.10941 0.0037426 -3.9372 x 1 0~ 5 0.10941 0.17782 0.0080931 -9.4342 x 1 0~ 5 0.00374 0.00809 0.0071205 -0.00012282 -3.9372 x 1 0~ 5 -9.4342 x 1 0~ 5 -0.00012282 9.8863 x 10 5 - 0.997320 A(4)i = QiRi = 0.073211 -0.000868 2.1324 x 10~ 6 0.046000 -0.0002696 - 0.073173 -0.996260 - 0.002503 -0.045938 -0.998790 2.6333 x 1 0~ 5 0.000538 0.017545 0.0175510 0.9998500 -1.49520 -0.12214 -0.0043425 4.6479 x 10~ 5 0.00000 -0.16952 -0.0081160 9.6801 x 10~ 5 0.00000 0.00000 -0.0067449 0.00012010 0.00000 0.00000 0.0000000 9.6718 x 10~ 5 1.500100 0.012424 1.6887 x 1 0~ 5 0.012424 0.169260 0.0003099 1.6887 x 1 0" 5 0.000310 0.0067389 2.5468 x 10~9 5.1991 x 10" 8 1.6969 x 10~6 2.5468 5.1991 1.6969 9.6703 x x x x 10~ 9 KT 8 1 0" 6 10~5 A(A)2 = RiQi A(4)2 = Q2R2 0.008282 -3.9108 x 10" 6 - 1.3792 x 10" 1 0 - 0.008282 - 0.999960 0.0018313 1.5392 x 1 0~ 7 5 -1.1257 x 10~ - 0.001831 -1.0000000 -0.00025182 -1.6977 x 10- 9 -3.0723 x 10- 7 - 0.0002518 1.00000000 - 0.999970 - 1.50010 -0.01383 -1.9529 x 10" 5 - 2.9966 x 10 - 9 0.00000 -0.16915 -0.0003221 -5.5105 x 1 0~ 8 0.00000 0.00000 -0.0067383 -1.7212 x 1 0~ 6 0.00000 0.00000 0.0000000 9.6702 x 1 0~ 5 ^4(4)3 = R2Q2 1.500200 0.001401 7.5850 x 1 0- 8 -1.6405 x 1 0" 0.001401 0.169140 1.2340 x 1 0~ 5 -2.9710 x 1 0" 8 7.5850 x 1 01.2340 x 1 0- 5 0.0067383 -2.4351 x 10 13 -1.6417 x 1 0~ -2.9710 x 1 0- 1 1 -2.4351 x 1 0~ 8 9.6702 x 10 240 Solutions 4 (4) 3 = Q3-R3 - 1.0000000 0.0009338 -1.7566 x 10~ 8 8.8905 x 10~ 15 - 0.0009338 -1.0000000 7.2955 x 10~5 - 8.7996 x KT 1 1 8 - 5.0559 x 10- - 7.2955 x 10" 5 - 1.0000 3.6138 x lO" 6 13 10 6 1.0943 x 1 0" 1.7565 x K T 3.6138 x 10" 1.0000 -1.50020 -0.00156 -8.7713 x 10~8 0.00000 -0.16914 -1.2831 x 10~5 0.00000 0.00000 -0.006738 0.00000 0.00000 0.000000 A(4)4 = R3Q3 1.500200 0.000158 3.4068 x lO" 1 0 - 1.0796 x 10~ 16 0.000158 0.169140 4.9159 x 10~ 7 1.7074 x 1 0" 1 4 10 3.4068 x 104.9159 x 10~ 7 0.0067383 3.4947 x lO" 1 0 17 14 1.0582 x K T 1.6986 x 1 0~ 3.4947 x 10~ 10 9.6702 x 10~ 5 4 (4)4 = Q4R4 1.9304 x 1 0~ 13 3.1503 x lO" 1 1 2.4701 x 1 0 - 8 9.6702 x 10~5 0.00010528 -7.8899 x 10" 1 1 -5.7307 x 1 0~ 1 9 ' - 1.0000 - 1.0000 2.9064 x 10~ 6 5.0310 x 10" 1 4 - 0.00010528 -6 10 -2.9064 x 1 0 - 1.0000 - 5.1863 x 10" 8 - 2.2709 x 10~ -18 13 - 7.0539 x 1 0 - 1.0042 x 10~ - 5.1863 x 10~8 1.0000 - 1.50020 -0.00018 -3.9397 x 10~ 10 1.0608 x K T 1 6 0.00000 -0.16914 - 5.1117 x 10- 7 - 1.8100 x 10- 1 4 0.00000 0.00000 - 0.0067383 -3.5448 x lO" 1 0 0.00000 0.00000 0.0000000 9.6702 x 10~5 , 4(4)5 = R4Q4 1.5002 1.7808 x 10~5 1.5304 x 1 0~ 12 1.1853 x 1 0~ 1 6 ' 5 1.7808 x 1 0~ 0.16914 1.9584 x 10~8 - 9.8322 x lO" 1 7 12 8 1.5302 x 1 0~ 1.9584 x 1 0~ 0.0067383 -5.0152 x 10~ 12 22 18 - 6.8213 x 10~ -9.7112 x 10~ - 5.0153 x 10~ 12 9.6702 x 10" 5 1.5002 T hus, A = 0.16914 0.0067383 0.0000967 rAfA„ and ^ ^ 1.5002 = 0 1)000967 = 15514 Solutions 241 P roblems of Chapter 7 7 .1 a) A zero dimensional polytope is a point. b) One dimensional polytopes are line segments. c) Two dimensional polytopes are n-gons: t riangle (3), rectangle (4), trapezoid (4), pentagon (5), 7.2 Zi 2 =conv(ei,e 2 ,e3). See Figure S.6. (1,0,0) x i +1- / F ig. S.6. A2 in 7 .3 C 3 =conv((0,0,0) T , (a, 0 ,0) T , (0, a, 0 ) T , (0,0, a)T, (a, a, 0 ) T , (a, 0, a)T, (0, a, a)T, (a, a, a)T) Cn = {xeWl A -4- : 0 <Xi < a , i = l , . . . , n ; a k €R+}. f ~ CUBE OCTAHEDRON F ig. S.7. Cube and octahedron C 3 4 =conv((a, 0,0) T , (0, a, 0) T , (0,0, a)T, ( - a , 0,0) T , (0, - a , 0) T , (0,0, C* = J x e Kn : ^ | xi| < a , a E R + . -a)T) Solutions X (0.1.0) ts F ig. S.8. 3-dimensional pyramid 7 .4 See Figure S.8 for a drawing of Pn+\. L et a1 b e the normal to face Fj, i — 0 , 1 , 2 , 3 , 4 . Let alx < bi b e the r espective defining inequalities. W e know Fo is the X\-X2 p lane. Then, Fo = {x * K 3 : x 3 = 0 } . E W e know t h a t a2 a nd a 4 a re perpendicular to x 2 --axis. Similarly, a 1 a nd aA a re perpendicular to xj axis. Thus, a} = ( 0 , * , * ) T , a2 = ( * , 0 , * ) T , a 3 = ( 0 , * , * ) 7 ' , a 4 = ( * , 0 , * ) r . S ince F] contains ( 1 / 2 , 1 / 2 , 1 ) , (1,0,0), (0,0,0), what we have is F i = {x e R 3 : Oxi - 2x 2 + l z 3 = ° } • S ince F 2 c ontains ( 1 / 2 , 1 / 2 , 1 ) , (1,0,0), (1,1,0), we have F 2 = { x e R 3 : 2 xi + 0x 2 + l x 3 = 2 } . S ince F 3 c ontains ( 1 / 2 , 1 / 2 , 1 ) , (1,1,0), ( 0 , 1 , 0 ) , it is F 3 = { x e R 3 : Oxi + 2x 2 + I 13 = 2} . A nd finally, ( 1 / 2 , 1 / 2 , 1 ) , (0,1,0), (0,0,0) are in F 4 , F 4 = { x £ R 3 : - 2 x i + 0x 2 + l x 3 = 0 } . T herefore, F 3 = {x G R 3 : x 3 > 0, - 2 x 2 + x 3 < 0, 2xi + x 3 < 2 , 2 x 2 + x 3 < 2, - 2 x i +x3 <0}. S olutions 243 Pn+\ is not a union of a cone at XQ a nd a poly t ope. P n +i is a direct sum of a cone at xo and Cn. P „ + i is an intersection of a cone at x 0 and C n +i provided that XQ £ 7 .5 See Figure S.9. Cn+i\Cn. (1,1,0) ^r , (0,1,1 '• • %! (1,0,1) F ig. S.9. A tetrahedron T he diagonal ray (1,1,1) T of the cube is orthogonal to facet F4. T hus, F4 = {x e R 3 : xi + X2+ x3 = a}. Since this facet contains (0,1,1) T , (1,0, l ) r , ( 1,1,0) T , t he value of a is 2. Therefore, F4 = {x 6 K3 : x i + x2 + x3 = 2} . Since (0,0,0) T is on the tetrahedron, the following halfspace is valid and facet defining ff4={i6i3: xi+x2 + x3< 2 } Similarly, F i = {x £ R 3 : x i - x 2 - x 3 = 0 } , F2 = {x e R 3 : - X ! + x 2 - x 3 = 0 } , F 3 = { i e R 3 : - x i - x 2 + x 3 = 0} . T he following set describes the tetrahedron: xi +x2 + x 3 < 2, x i - x2 - x 3 < 0, —x\ + x2 - x 3 < 0, —xi — x 2 + x 3 < 0. 244 Solutions 7 i / ~%i.i.-i) / j/v,-',-D j f(0,f',-« RS^—™~™™™™™-T«™™| ^ / r (-•,0,-f') (t,0,-f'> It/ 1* / ** #7 ' * I/ * f\*7^ ***'*, ,/^\. V L* / / rl« *ff ?~ZZZJL~~««~^^ F ig. S.10. The dodecahedron, 0: golden ratio 7 .6 See Figure S.10. T he polyhedron vertices of a dodecahedron can be given in a simple form for a dodecahedron of side length a = \/b - 1 by ( 0 , ± r \ ± < / > f , ( ±^,0,±<A- 1 ) T , ( ± < T \ ± ^ 0 ) T a n d ( ± l , ± l , ± l ) T ; where <j> — ^^Figure S.ll. is the golden ratio. We know (f> - 1 = ^ a nd 0 = 2 cos f. See S olutions (•"',•.0) 245 F ig. S . l l . The extreme points of the dodecahedron, cj>: golden ratio 246 Solutions P roblems of Chapter 8 8 .1 a) We have six variables and three constraints, therefore we have (3) = 20 c andidate bases. X\ X2 X3 Si S2 S3 A= 2 10 10 0 0 0 10 1 0 0 10 0 0-1 = { x 1 , x 2 , x 3 } , B2 = {xi,x2,si}, B3 = {xi,x2,s2}, B4 = {xi,x2,s3}, = {xi,x3,si}, B6 = {xi,x3,s2}, B7 = {21,2:3,53}, J58 = {x1,s1,s2}, = {xi,si,s3}, Bw = {xi,s2,s3}, Bn - {x2,x3,si}, BX2 = {x2,x3,s2}, = {x2,x3,s3}, B14 = { x 2 , s i , s 2 } , B15 = { x 2 , s i , s 3 } , B16 = {x2,s2,s3}, = {x3,s1,s2}, B1S = {x3,si,s3}, B19 = {x3,s2,s3}, B20 = {si,s2,s3}. B2, B4, B$, BQ, Bg, BQ, B\2, S15, Bn, Big are not bases since they form singular matrices. Br,Bio,Bis,B20 are infeasible since they do not satisfy nonnegativity constraints. Thus, what we have is (xi, x2, x3, si, s2, s3)T = ( 3,2,10,0,0,0) T from Bx •-> point F, {xi,x2,x3, si,s2, s3)T = ( 3,2,0,0,10,0) T from B3 <->• point C, (xx,x2, x3, si,s2, s3)T = (0, 2,10, 6, 0, 0) T from Blx <-> point E, (x1,x2,x3,s1,s2,s3)T = ( 0,8,10,0,0, 6) T from B13 '-> p oint D, T (xi,x2,x3,S!,s2,s3) = ( 0,2,0,6,10,0) r from Bu <-» point B, T ( x i , x 2 , x 3 , s i , s 2 , s 3 ) = ( 0,8,0,0,10,6) T from Bi6 <->• point A. See Figure S.12. b) 1. matrix form: Let xB = (si,x3,x2)T, xN B= (x1,s2,s3)T. 101 010 001 =>B~ "10-1" 01 0 00 1 T hen, " 1 0 -- 1 " 01 0 00 1 ' 8" " 6" 10 = 10 2 2 Bi B5 BQ B\3 Bn xB = B-Xb We are on point E. z = cTBxB = [Q,2, 2] 6 10 2 = 24. cTN-cTBB-lN= "10-1" 01 0 [1,0,0] -[0,2,2] 0 0 1 "2 0 0" 0 1 0 = [ 1,-2,2]. 0 0-1 Solutions 247 D=(0,8,10) x3 A ,V,Y, V 72> Fig. S.12. Exercise 8.1: Primal and dual polyhedra T hus, S3 enters. B-iNss = 10-1 01 0 00 1 0 0 -1 = 1 0 -1 •Si X3 x2 T hen, T hus, «i leaves. New partition is XB — (sz,X3,X2)T, OOll 0 10 -1 0 1 x?j = (xi,S2,s\)T. J 5- 1 "10 - 1 " 01 0 10 0 £= 248 Solutions xB = B~lb = We are on point D . z = cTBxB = [0,2,2] 6 10 = 36. " 20 1" 0 1 0 = [ -3,-2,-2]. 000 [10-1] 01 0 10 0 [ 10" [ 6" 8 = 10 2 8 cl-clB^N = [1,0,0] -[0,2,2] " 10 - 1 " 01 0 10 0 T hus, D is the o ptimal point. 2. simplex tableau: X\ X2 X3 Si S2 S3 Xz 2 0 x2 0 z -1 «1 0 0 1 0 0 1 0 0 1 0 0 0 01 10 0-1 2 -2 RHS~ 6 10 2 24 Xi X2 X3 Si S2 S 3 S3 2^3 x2 z 2 0 2 3 0 0 1 0 0 1 0 0 1 0 0 2 01 10 00 20 RHS' 6 10 8 3 6. 3. revised simplex with product form of the inverse: Let xB = ( s 1 ,x 3 ,a;2) T , xN = ( a;i,s 2 ,s 3 ) T - Then, B~l = cTBB~l = [ 0,2,2]. =1-[0,2,2] 0 1 0 0 0 -1 l 10-1 01 0 00 1 w rXl =cXl ~wNXl = 1>0. cS2 - wNS2 = 0 - [0,2,2] = -2<0. uNS3 = 0 - [ 0 , 2 , 2 ] = 2>0. f00 S3 is the entering variable a nd Si leaves. Ex = -flO -^01 xB = £ 1 - 1 6 = (6,8,10) T . Solutions w=[0,2,2]Ei1B-1 T l 249 = [ 2,2,0]. w = c BB~ = [0,2,2] rXl = cXl - wNxi = 1 - [2,2,0] = - 3 < 0. rS2 = cS2 - wNS2 = 0 - [2,2,0] = - 2 < 0. rSl =cSl -wNs> O ptimal. =0-[0,2,2] = -2<0. 4. revised simplex with B = LU decomposition: Let XB = (si,X3,X2)T, XN = (x\,S2,sz)T. T hen, B = 101 010 001 is upper t riangular, L = I3. Solve BXB — bhy back substitution. X2 = 2, x3 = 10, si = 8 — X2 = 6. Solve wB = CB by back substitution. i y i = 0 , W2 = 2, W3 = 2 — tui = 2. T he rest is the same, S3 enters and s± leaves. 001" 010 New basis is B = - 10 1 PB = LU & "00 r 010 1 00 001" "-10 r 010 = 0 1 0 = hU. -101 001 Solve BxB - Pb= ( 2,10, 8) T by substitution. x2 = 2, £3 = 10, s 3 = x2 - 2 = 6. Solve wB = PcB = ( 2,2,0) r by substitution. wi = 0, w2 = 2, w3 = 2 - wi = 2. T he rest is the same. 250 Solutions 101 0 1 0 is upper 001 t riangular, Q = I3. T he rest is the same as above, s 3 enters and si leaves. Let XB = (si,x3,x2)T, XN = (xi,s2,s3)T. T hen, B B= " 0 0 1' 001' 010 = 0 10 -10 1 -100 "10-1" 0 1 0 = QR. 00 1 = Qb', 5. revised simplex with B = QR decomposition: In order to solve BxB = QRxB = b = ( 8,10,2) r = Q(RxB) b'3 = 8, b'2 = 10, &i = - 2 =*• x 2 = 8, x 3 = 0, s 3 = x 2 - 2 = 6. In order to solve wB = c ^, first solve wQR — cB = w'R. w[ - 0, w'2 = 2, w3 = 2 + wi = 2. T hen, solve wQ = w' W3 = 0 , w2 = 2, wi = 2. T he rest is the same. c) (D): Min w =8yi + I0y2 - 2y3 s.t. 2?/i > 1 2/1 - 2/3 > 2 2/2 > 2 2/1,2/2,2/3 > 0. See Figure S.12. 8.2 The second constraint is redundant whose twice is exactly the last constraint plus the nonnegativity of x\. T hen, Xi X2 X3 Si S3 A= 2 -1 -1 -1 0 1-2 2 0-1 a) The bases are B i = {xi,x2}, B2 = {xi,x3}, B3 = { xi,«i}, B4 = { xi,s 3 }, B5 = {x2,x3}, B6 = {x2, S i}, B7 = {x2, s 3 }, B8 = {x3, s i}, B9 = {x3, s3}, Bw = {si, s3}. All bases except B2,B3 yield infeasible solutions since they do not satisfy t he nonnegativity constraints. Thus, (xi,x2,x3, Si,s3)T = ( 2,0,1,0,0) r from B2, and (xi,x2,x3,S!, s3)T = ( 4,0,0, 5, 0) T from B3. Solutions b) Method 1: At (2,0,1,0,0) T , we have x2 si 251 .S3 B~]N = _ i —2 _ i 5 _3 5 5 5 1 _2 5 5 , B-^: X\ - 5 ^2 = 2 ,T2 = (9 => r = (2 + §0,0,1 + f 0,0,0) T is If x 2 enters 2 3 - | x 2 = 1 x2 > 0 feasible for 9 > 0. Thus, r 1 = ( | , l , f , 0 , 0 ) T is an unboundedness direction and hence an extreme ray. xi - ± s 3 = 2 \ If s 3 enters X3 - |,S;3 = i \ - » s 3 = 0 =» r = (2 + ±0,0,1 + | 0 , O , 0 ) r is feasible for 0 > 0. Thus, r 2 = ( ±,0, f, 0,1) T is another unboundedness direction and hence an extreme ray. At (4,0,0,5,0) 7 \ we have B-lN x2 x3 s 3 -3 2 -1 -3 5 -2 B~lb- s3 > 0 J x'l - 2x2 = 4 I If x 2 enters .si - 3x2 --= 5 > <-+ x 2 = 0 => r = (4 + 2(9,0, 0,5 + 30, O)7' is x2 > 0 I feasible for 0 > 0. Thus. and hence an extreme ray. Xi S3 = 4 ~) ( 2,1,0,3,0) is an unboundedness direction If .s3 enters sx - 2 s 3 = 5 \ <-> s3 = 0 => r = (4 + 0,0,0, 2(2, 0 ) r is feasible •S3 > 0 J for 0 > 0. Thus, r 4 = ( 1,0,0,2,1) T is another unboundedness direction and hence an extreme ray. Method 2: Try to find some nonnegative vectors in N{A). r 0<rl =(l,l,l,0,o\ eAf(A). 0 <r2 Q,0,^,0,1) eAf(A). 6 ><r 3 = ( 2,l,0,3,0) r GAA(yl). 0<r4 = ( 1,0,0,2, l ) T e / / ( > * ) • 252 Solutions S o, t hey are rays. Since every pair of the above vectors have zeros in different p laces, we cannot express one ray as a linear combination of the others, they a re extreme rays. c) 1. Xi + X2 + £ 3 : c1 = ( l , l , l , 0 , 0 ) T = > ( (cl)Trl 1 \T„2 1 = f + l + f + 0 + 0 = ^ > 0 < - ) - unbounded (c ) (ci)Tr3 = _ (ci)Tr4 = f + o + f + o2 0 I > 0 "-> unbounded + l + 0 + 0 + 0 = 3 > 0 - - » unbounded 1 + 0 + o + 0 + 0 = l > 0 < - > unbounded T hus, t here is no finite solution. 2 . —2xi — X2 — 3 x3: c^ = ( - 2 , - l , - 3 , 0 , 0 ) J ( (c2)Tri :=: =• bounded _|_ 1 _9 + o + 0=-f^O-> 2\T_2 _ (c,2) - l - o - i + o- 0 = - | ^ 0 a b o u n d e d ( c2)T r 3 = _ 4 _ 1 + 0 + o + 0 -5 i> 0 °-> bounded (c2)Tr4 = - 2 + 0 + 0 + 0 + 0 = - 2 ^ 0 -->• bounded T hus, t here is finite solution. 3 . -x\ - 2^2 + 2a;3: c3 = ( - l , - 2 , 2 , 0 , 0 ) T ^ ( {cz)Trl 3T2 = - § - 2 + § + 0 + 0 = - § ^ 0 < - + bounded (c ) r = - | + 0 + | + 0 + 0 = | > 0 ^ unbounded ( c 3JT r 3 = _ 2 _ 2 + 0 + 0 + 0 = - 4 ^ 0 ' - > bounded [ ( c 3 ) T r 4 = - 1 + 0 + 0 + 0 + 0 = - 1 ^ 0 ^ unbounded T hus, t here is no finite solution, d ) xi = 6, x2 = 1, x3 = \ Si = ±5 «3 = 1 *i — 2 ' r 6n ~2 1 0 1 =a 1 J 0 2 i_ 0 4 0 (l-«) 4 5 1 + M2 i 5 0 2 5 2 1 + /^3 1 0 + M4 0 5 0 + A«i 3 5 0 0 0 1 0 3 0 0 2 1 a,Mi)M2.M3,A«4 > 0 W e have 5 unknowns and 5 equations. The solution is Solutions 253 6' 1 } 4i i ~2 2 0 1 1 +2 0 0 4 0 0 5 0 4 5 1 5 1 0 +0 3 +0 0 0 2 5 0 1 2 1 +1 0 3 0 1 0 +1 0 2 1 convex combination of e xtreme points canonical combination of extreme rays 1. \xi X2 X3 X\ 1 X3J0 -z|0 -1 0 -I 1 4 0 «1 2 S 1 5 «3 1 5 2 5 RHS1 2 1 -4 7"" 5 1 " 3" 5 6 0 1 S _1 (6-zl6) (3 4 L 5 5J \ 2 1" 55 12 3 )1 2 1 L 5 J IbJ T he values of basic variables will change but not the optimal basis. , 0,1) which satisfies the new constraint, no 3. T he solution above is problem! 8 .3 a) 1. B = { si,52,33} - > B = I,cB bounds and cjj = ( 2,3,1,4). 6, J\f — {xi,X2,xs,X4} a t their lower " 1" 0 3 0 [30] xB =B~1b-B-lNx N 13 20 20 12 11 [1235] - 1 100 0034 Sl 30 13 20 - 10 1 9 = = «2 S3 Z = Cg'xB + cJjXN = 2 • 0 + 3 + 0 = 5. cN — CgB~ N — ( 2,3,1,4). Then, Bland's rule (lexicographical order) m arks the first variable. Since the reduced cost of #1 is positive and Xi is a t its lower bound; as x\ is increased, so is z. Hence, x± enters. 254 Solutions 0 0 0 Si < «2 = S3 20 12 11 - 1 1 0 a < 6 — 1 = 5 (bounds of x\) = >a = min{20,12,5} = 5. xi leaves immediately at its upper bound, xi = 6. T ( 2,3,1,4), xTB = ( 15,7,11) z = 12 + 0 + 3 + 0 = 15, 2. B = I,cB = C N ~ CgB N = (2, 3,1,4). Then, Bland's rule marks the second variable. Since the reduced cost of x2 is positive and x2 is at its lower bound; as x2 is increased, so is z. Hence, x2 enters. 0 0 0 Si < Si S3 = 15 7 11 - 2 1 0 a < 10 - 0 = 10(bounds of x2) => a = m i n i —-,7,10 > = 7. T hus, S2 leaves. 3. B= {si,x2,s3} 120 => B - 0 1 0 4 001 "301 13 20 CO O B' "1 - 2 0" 0 10 0 01 "1035" 1 100 0034 1 7 11 "6" 0 3 0 Si XB = X2 .S3. "1 - 2 0" = 0 10 0 01 - 1 - 2 3 5' 1 100 0 034 "1 - 2 0 " 0 10 0 01 4 13 20 4 13 20 3 —6 9 = 2 = ( 0,3,0) 1 7 + ( 2,0,1,4) 11 CO O = 21 + (12 + 3) = 36. cTN-cTBB~lN = ( 2,0,1,4) - ( 0 , 3 , 0 ) -2 3 5 100 034 = (2,0,1,4)-(3,3,0,0) = (-1,-3,1,4), Solutions 255 where Af = { xi,si,X3,X4}. Then, Bland's rule (lexicographical order) m arks the first variable. Since the reduced cost of xi is negative and xi is a t its upper bound; as X\ is decreased, z is increased. Hence, X\ enters. 0 0 0 Si < x2 «3 = 1 7 11 - -1 1 a< 0 00 10 CO a < 6 — 1 = 5(bounds of x\) = >a = m i n { l , 1 0 - 7 , 5 } = 1. T hus, s\ leaves. 4. 1 20 B = {__,__, s 3 } => B = 1 1 0 =*B~l = 001 Xi - 1 20 1 -10 0 01 O xB = x2 S3 - 1 2 0" 1 -10 0 01 ' 30' 13 20 "0" 0 — 3 0 - 1 2 0" 1 -1 0 0 01 -4 17 20 "1035] 0100 0034 -9 9 9 O CO O -4 17 20 1 2-3-5" 1-13 5 0034 5 8 11 - = 5 8 11 z = (2,3,0) ( 0,0,1,4) (10 + 24) + 3 = 37. c;-4r1iV=(0,0,l,4)-(2,3,0) -1 2 - 3 - 5 1-13 5 0034 = ( 0,0,1,4) - (1,1,3,5) = ( - 1 , - 1 , - 2 , - 1 ) , where Af = {s\, s2,xz,Xi). All of the reduced costs are negative for all t he nonbasic variables that all are at their lower bounds. Hence, x* = ( xi,X2,X3,X4) T = ( 5,8,3,0) T is the optimum solution, where z* = 37. b )(P): max 2xi + 3x2 + x$ + 4x4 s.t. x i + 2x2 + 3x 3 + 5 x 4 < 30 xi+x2 < 13 (y2) (j/i) 256 Solutions 3x3 + x 4 < 2 0 -xi < -1 (2/3) (2/4) (1/5) (l/e) (2/T) x\ < 6 £2 < 10 -x3 < -3 X3 < 9 x4 < 5 (j/8) (2/9) Xi,x2,x3,x4 > 0 ( D): m in302/i + 13t/2 + 2O2/3 - 2/4 + 62/5 + Kfye - 32/7 + 92/8 + 5?/9 s .t. 2/i + 2/2 - 2/4 + 2/5 > 2 22/i + 2/2 + 2/6 > 3 ( xi) (x 2 ) (x3) 3i/i + 3y 2 - 2/7 + 2/8 > 1 52/i + 2/3 + 2/9 > 4 (2:4) 2/1,2/2,2/3,2/4,2/5,2/6,2/7,2/8,2/9 > 0 T h e optimal primal solution, x* = (xi,X2,X3,X4) T = ( 5 , 8 , 3 , 0 ) T , satisfies c onstraints (2/1,2/2,2/7) a s b inding, i.e. the corresponding slacks are zero. By c omplementary slackness, the dual variables 2/1,2/2,2/7 might be in the optimal d ual basis. T h e other primal constraints have positive surplus values at the o ptimality, therefore 2/3 = 2/4 = Vt — 2/6 = V& = Vl = 0- Moreover, the r educed costs of the surplus variables at the optimal primal solution are b o t h 1 for s\ a nd S2, which are the optimal values of y\ = 2/2 = 1- Since the o ptimal primal basis contains the nonzero valued x\ a nd X2, the corresponding d ual constraints are binding: 1 + 1 — 0 + 0 = 2-*/ and 2 + 1 + 0 = 3A/Furthermore, the optimal primal solution has nonbasic variables X3 and X4, t hen the corresponding dual surplus variables may be in the dual basis: 3 + 3 — 2/7 + 0 > 1, and 5 + 0 + 0 > 4 = > the corresponding surplus, say t\ = 1 i n the d ual optimal basis. T h e optimal primal objective function value is z* = 3 7, w hich is equal to the optimal dual objective function value by the strong d uality theorem. Then, 37 = 30(1) + 1 3 ( 1 ) + 20(0) - (0) + 6(0) + 1 0 ( 0 ) - 3j/J + 9 (0) + 5(0) + 0t\ + 0 ^ 0 ^ + 0tl, y ielding 2/7 = 2. Solutions 8.4 257 Fig. S.13. A multi-commodity flow instance Let us t ake the instance given in Figure S.13, where K = 3 and V= {l,2,3,4,5,6,A,C,I,K,OP}, A - {a, b, c, d, e, f, g, h, i, j , k,l,m, n, o, p} . Let u s fix all capacities a t 10 and all positive supplies/demands a t 10 w ith u nit carrying costs. a) Cka'Eka k a S.t. / J %ka / J %ka — U*ki aET(i) a£H(i) k •Kka — l^ka Xka > 0 (integer) In general, we have mK variables, m + nK c onstraints a nd mK simple b ounds other than the nonnegativity constraints. In our example instance, we have 258 Solutions Min (xia + x2a + x3a) H s.t. 1 {xip + x2p + x3p) - (xla + xig) - (xlb + xlf) = 0\ (x2a + X29) - (x2b + x2f) = 0 > node 1 (X3a + XZg) - (x3b + X3f) = 0 J ( zip) - (xio) = 10 (x2p) - {x2o) = —10 ^ node OP (x3p) - (x3o) = 0 Xla + X2a + X3a < 10 xip + x2p + x3p < 10 xia,--- ,x3p>0 (integer) b) Min ] T Y k s.t. pevk Ck rfp Y Pf=V k fp = D<< Y Y ^fr ^ U* k P£Vk fp < V-P fP>0 (integer) We have (huge number of) K2m variables, m + K c onstraints and K1m simple bounds other than the nonnegativity constraints. The following sets t he relation between the decision variables of the two formulations whose c onstraints are isomorphic: xak = Y taPfP' fp P£Vk =m ^YXak k ( a PP l i e d recursively). S olutions 259 I n our example instance, «i is node A a nd £3 is node / . If we enumerate p a t h s (some of t h e m is given in Figure S.14), we have C omm. P a t h # P a t h 1 ai->fh^hh^mi->p 5 6 ai-»&i-»di-»/i->ji->/jh->mH-»p ai->/i-»<7h->-6h->-dh->/i-».7'i->-/i>->-mi->p 7 8 9 10 11 12 13 14 o Mo H-> o o t-4 o i-> k A; jfe n n > -»Zi->-ji->/ii->e>-->-c n n i ->ei->'(ii->Zh->-jh->-5i->6i->c 15 16 17 18 19 k 4J4li4ra^pi->o^n4i i ->-j^/ii->ei->c!H->TOH->ni-»z k fc 1 ^ y 1 > ^ 1 > 6 1 > rf i—>- m ——• —• —• 1—> n *-> i 20 A; F ig. S.14. Some paths in our multi-commodity flow instance 2 60 Solutions a nd the formulation will be Min 5 /i + • • • + 1 0/ 20 s.t. A + • • • + / e = 10 h + • • • + hi = 10 / 12 + • • • + / 20 = 10 fl+f2 + f3 + U + h + f6< 10 / l + /2 + /3 + h + h + k + / l6 + /l9 + /20 < 10 / l , - - - ,/20 > 0 (integer) T he first three constraints make the capacity constraints for arcs a, c, i and k r edundant. c) Wa -B- ] P ^ i a p / p < t/ a Dk T f «-» ]T] fp= T c P<EVk T hen, the reduced cost of a path P will be ^ ( C i a + Wa) -TTfc, a nd the current solution is optimal when min I Y Vc f c a + wa) } > ^k, Vfc. P6V k °^ [ftp J T he above problem is equivalent to find the shortest path between s^ and t^ using arc costs Cfc0 + wa for each commodity k. T he problem is decomposed into K single commodity shortest path problems with a dynamic objective function that favors paths with arcs that have not appeared many times in c urrent paths. d) [fli f2i f3, f4, fei f6i f7, f8, f9, flQ, fll, fl2, fl3, fl4, flbi fl6i fl7i flS, fl9, f2o\ l^bi Sd-> ^ e i &f 1 Sg-> ^ / i ; $j1 &fo $mi Sni ^01 Sp\* S olutions 261 c = [ 5 5 7 7 8 9 4 7 7 9 9 4 6 6 6 8 8 8 10 10|000000000000] 1 1 1 1 1 00 0 00 0 00 0 0000 0 00 0 000 0 0000 10 0 00 0 00 1 1 1 10 0 00 0 0000 0 00 0 000 0 0000 1 0 00 0 00 0 00 001 1 1 1 m i l 0 00 0 000 0 0000 0 10 01 1001 0 10 100 0 0110 100 0 000 0 0000 Oil 1 1 1 0 00 0 10 110 0 1111 0 10 0 000 0 0000 001 0 00 1 10 110 0 10 0 100 1 001 0 000 0 0000 101 101 0 00 100 0 00 0 0000 0 00 1 000 0 0000 0 00 101 001 110 100 0 0110 0 00 0 100 0 0000 101 01 10 10 101 Oil 1 1001 0 00 0 010 0 0000 0 00 01 101 11 1 1 1 1 1 1 1 1 1 1 0 00 0 001 0 0000 0 00 01 101 11 10 0 00 0 0000 0 00 0 000 1 0000 1 1 1 1 1 1 0 00 0 00 0 00 1 1 1 1 1 0 00 0 000 0 1000 0 00 0 00 1 1 1 1 10 001 1 1 1 1 1 0 00 0 000 0 0100 0 00 0 00 1 1 1 1 10 0 00 100 11 0 00 0 000 0 0010 100 001 0 00 0 00 0 00 1 0011 0 00 0 000 0 0001 F ig. S.15. S tarting bfs s olution for our m ulti-commodity flow i nstance: r epeated 262 Solutions B, = 1 00 0 00 0 00 0 00 0 00 0 10 0 00 0 00 0 00 0 00 0 01 0 00 0 00 0 00 0 00 0 01 100 0 00 0 00 0 00 0 01 0 10 0 00 0 00 0 00 0 10 0 01 0 00 0 00 0 00 100 0 00 1 00 0 00 0 00 0 01 0 00 0 10 0 00 0 00 1 00 0 00 00 10 00 0 00 00 10 00 0 00 1 00 0 00 0 00 0 00 0 00 0 10 0 00 100 0 00 0 00 0 01 0 00 0 10 0 00 0 00 0 00 1 00 0 10 0 00 0 00 0 00 0 10 1 00 0 00 0 00 0 00 00 1 c^ = [5 4 6 0 0 0 0 0 0 0 0 0 0 0 0] 2/i =CTB1BI l = [Aw] [5,4,6|0]. T hen, the lengths of arcs are c&a + wa = 1 + 0 = 1, V arcs. For commodity one, the minimum shortest path (P2 : a464d4 m\-$ p) o ther than Pi has length 5 which is equal to the corresponding dual variable 7Ti = 5. For commodity two, the minimum shortest path has length 6 which is strictly greater than the corresponding dual variable n2 = 4. However, P12 : k i->- j i-t h i-+ i has length 4 < 6 = ^3! Thus, /12 enters to the basis w ith the updated column {B-lA12)T and the updated RHS is X =[001-1-100-11000000] B l = (Bllb)T XTBY = (B^bf =[fl h / l 3 H Sd Se Sf Sg Sh Sj Si Sm Sn S0 Sp] = [10 10 10 0 0 0 0 0 0 0 10 0 0 0 0] , therefore the slack variable corresponding to arc h, s/,, leaves. Solutions 263 Bn 1 00 0 00 0 00 0 00 0 00 0 10 0 00 0 00 0 00 0 00 00 10 00 0 01 0 00 0 00 0 01 1 00 0 00 0 00 0 00 0 01 0 10 0 00 0 00 0 00 0 10 00 10 00 0 00 0 00 1 00 0 00 1 00 0 00 0 00 00 10 00 0 10 0 00 0 00 1 00 0 00 0 01 0 00 0 00 0 01 0 00 00 1 100 0 00 0 00 0 00 0 00 0 10 0 00 1 00 0 00 0 00 0 01 0 00 0 10 0 00 0 00 0 00 1 00 0 10 0 00 0 00 0 00 0 10 1 00 0 00 0 00 0 00 0 01 cl2 = [5 4 6 0 0 0 0 0 4 0 0 0 0 0 0] 2/2 = [7 4 6|0 0 0 0 0 - 2 0 0 0 0 0 0] T hen, the lengths of arcs are Cka + u>0 = 1 + 0 = 1, V arcs except arc h, whose length is 1 - 2 = - 1 . For commodity one, the minimum shortest path P2 : a^b^d^m^-p has length 5, which is strictly less than the corresponding dual variable wi = 7. T hus, fi enters to the basis with the updated column {B^lA2)T and the updated RHS is X =[101000-1-1-100000-1] B2 = (B2lb)T = [fl h / l3 Sb Sd Se Sf Sg /12 Sj Si Sm Sn S0 Sp] xTB2 = (B^bf = [10 10 10 0 0 0 0 0 0 0 10 0 0 0 0] ; therefore, either / x or fa leaves. We choose / 1 ! 264 Solutions Ba 1 00 0 00 0 000 0 0000 0 10 0 00 0 000 0 0000 00 10 00 0 010 0 0000 101 100 0 000 0 0000 101 0 10 0 000 0 0000 0 10 0 01 0 000 0 0000 0 00 0 00 1 000 0 0000 0 01 0 00 0 100 0 0000 0 00 0 00 0 010 0 0000 0 01 0 00 00 11 0 0000 0 00 0 00 0 000 1 0000 1 00 0 00 0 000 0 1000 0 10 0 00 0 000 0 0100 0 10 0 00 0 000 0 0010 0 00 0 00 0 000 0 0001 cl3 = [5 4 6 0 0 0 0 0 4 0 0 0 0 0 0] 2/3 = [5 4 6J0 0 0 0 0 - 2 0 0 0 0 0 0 ] T hen, t he lengths of arcs are Cka + wa = 1 + 0 = 1, V arcs except arc h, whose length is 1 — 2 = — 1. For all the t hree commodities, t he m inimum shortest distances between t he source a nd the sink nodes are g reater a nd equal t o the corresponding dual variables. Therefore, t he current solution given below is o ptimal. X ^3 = (B3lb)T = [h h / l 3 Sb Sd Se Sf Sg / l 2 Sj Si Sm Sn S0 Sp] xl3 = (B^bf = [ 10 10 0 0 0 0 10 10 10 0 10 0 0 0 10] T he optimum solution is depicted in Figure S.16. e) When t he number of variables (columns of A) is huge, t he following question is asked: C an one generate column A* by some oracle that can answer t he question, Does there exist a column with with reduced cost < 0? If so, t he oracle returns one. So, the sketch of so called "A Column Generation Algorithm" is given below: S I. Solve LP(J): mm for some J C I — { 1 , . . . , n). S2. Using dual variables 7r that are o ptimal for LP(J), ask the oracle if t here exists j 0 J such that CjirA^ < 0. If so, a dd it to J a nd perform pivot(s) t o solve new LP(J); Go back S I. If n ot, we have t he o ptimal solution t o L P over all columns. Solutions 265 Fig. S.16. The optimum solution for our multi-commodity flow instance In a sense, we partition the optimization problem into two levels: Main / Subproblem, or Master / Slave, or Superior / Inferior; where the subproblem h as a structure that can be exploited easily. The main problem generates dual variables and the subproblem generates new primal variables; and the loop stops when primal-dual conditions are satisfied. T he dual to the above column generation approach gives rise to the separation problem, where we are about to solve LP with large number of rows ( equations). We first solve over restricted subset of rows (analogous to solving over subset of columns) and ask oracle if other rows are satisfied. If so, we are done; if not, we ask the oracle to return a separating hyperplane that h as current rows satisfied in one half space and a violation in the other. This approach leads to Bender's decomposition. 266 Solutions P roblems of Chapter 9 9 .1 Let a = inf A Then, Vx e A, a < x « > -x < -a. Hence, (-A) in = bounded above. Also, —a is an upper bound of (—A). So, s up(-.A) < -a <=> - s up(--A) > a = inf A. Conversely, let /? = sup(—.4). Then, Vx G A, — x < (3 •£> x > —/?. Hence, —/? is a lower bound of A. So, inf A > - /J — - s u p ( - . A ) . T hus, inf A = - s u p ( - A ) . 9.2 a) If m = 0, (bm)l'n = (60)1/™ = l 1 /" = 1 (see (c)). (b---b)1/n b1/n---b1/n If m > 0, (ft"1)1/" = < ^-—' = ' » '= m times m times If m < 0, let m' = - m > 0. Then, (vm\l/n \ U (b^n)m. _ — _ / i.-m'\l/n _ / 1 \ l / n _ —\ U 1 _ ~ 1 1 (hl/n)-m _ ~ (ul/n\m (" ) • I I — Vhm' ) — (hm'\l/n /U/ti\m' b) IfTO— 0, all terms are 1. n If m > 0, (bm)n = v bm •••bm , " . ' = b...b...b...b m m =6mn. Similarly, (bm)n = bmn. IfTO< 0, letTO'= - TO > 0. Then, (i.m\n \ U (u—m'\n —\ U / 1 \n — \~g^7) 1 — (fcm')n 1 — bm'n ~ 1 (,-(mn) umn ~ " ) ) c) Let l1/™ = x where x >- 0. Then, xn — 1. Also, l1/™ = 1. Since the positive nth r oot of 1 is unique, we get x = 1. d) Let 61/™? = a , and (6 1 /")!/? = p where a,/3 X 0. T hen, b = anq and 6 1 /" = /?« => 6 = (/39)n = / 3" n = £ " 9 = anq. Since the positive nqth root of 6 is unique, we get a = /3, i.e. bl/nq = (61/™)1/9. Similarly, frl/ng _ /bl/q\l/n_ e) If p = 0, then W+q = b0+q = W = b°W = WW. Similarly, if q = 0, 6?+9 = bPW. So assume p ^ 0, g ^ 0. Solutions b---b b---bb---b Case 1 : p > 0, q > 0, W+q = ^~" = ^ - v - ' ^ - ' = ^69; P+ < ? p q Case 2 : p < 0, q > 0, Let p' = - p > 0. So, lf+q = b-P'+q; ' Case 2a: p' = q ^ &-"'+« = 6° = l = j£ = ^ = fcPfc*. ,, 6•••b b- • -b W ,„ Case 26 : p' < q =• b~" +q = s ~ ^ - ' = ^ ^ -r^-r = &= WW. 267 q-p q-p & "3 P' _ C ase 2c : p' > a =» b-r>'+q = 6 "CP'-«) = ^ =^ = £ = *W. Case 3 : p > 0, q < 0, similar to Case 2; I C ase 4 : p < 0, a < 0, then, p + a < 0 => &»+« = p ^ , = ^ ^ = WW. 9 .3 a) Let a = (bm)lln, (3 = (&P)1/9 where a,/3 > 0. ft™ =-, ft = ( a n ) £ . ' C ase 1 : m — 0, => p = 0. So, a = /3 = 1. C ase 2 : m > 0, =» p > 0. a = ft™)1/" =>. a » = Similarly, 6 = (/3«)* =>• 6mP = a"? = /?«m. T hus, np = qm=> anp = / 3 n p . Since the positive (np)th root is unique, a = 0. Case 3 : m < 0, =>• p < 0. Let m' = —m, p' = —p => m',p' > O.Case 2! (um\l/n = _ /L-m'\l/n _ / 1 = (bp)1/g ^> 1 U/n _ 1 _ (b-p'y/i r So, 6 , r £ Q are well defined. b) Let r = ^ , s = | where n,g > 0. br+s _ (pmq+np^i-q _ ^mg^p^ _^ mq ^^ np ^ _ = ((bm)")^ ((W)n)^ = (((bm)q)1^)^n(((bp)n)^n)1^q m1n p l/q = {b ) l {b ) = bb. rs c) Let 6* e B{r). T hen, < e Q , t < r ^ r - t > 0 , r - * e Q . Since 6 > 1 and r — t is a nonnegative rational number, we get br~l > 1. Claim: Let & > 1, s e Q+. 6s > 1. Proof: If s = 0 =>• 6s = b° = 1. Assume s > 0. Then, s = E where p, 0 > 0. ftS = ( &P)l/«. 6>1=i . a = 6 P >1=> . 6 » = al/« > L Hence, 1 > 6 r - ' = Wb'1 = ^ =• 6* < br. T hat is V6' £ B(r), b* < br; i.e. r b is an upper bound for B(r). T hen, sup(£(r)) < br. If r € Q, br £ B(r). So, br < s up(5(r)). Thus, br = s up(j5(r)). Now, we can safely define bx = s up(B(x)), Vx £ R. d) Fix 6 r a rbitrary in -B(a;) and fix 6s a rbitrary in B(y): r, s £ Q, r < x, s < y. 268 Solutions T hen, r + s G Q, r + s < x + y =$• br+s = bsbr G B(x + y) => brbs < bx+y. ^ r r Keep s fixed. 6 < ^ - , V6 € -B(x). Thus, ^ j - is an upper bound for B(x). Hence, 6* = sup(£(x)) < ^ < > bs < ^ . Similarly, 6r < ^ . £ Now vary s. W B(y). € -B(y), &s < ^js~- T hus, ^ j - is an upper bound for h by = sup{B(y)) < ^- => bxby < bx+y. bx Claim: bxby > bx+y. Proof: Suppose not. bxby < bx+y for some x, y 6 K. 3a G Q C K 3 6x6y < a < 6a!+!/, by Archimedean property, b * , ^ > 0 =>• a > 0. Since a < 6 x+!/ , a is NOT an upper bound of B(x + y). So, 3br G B(x + y) 3 a > br. Let t = £ > 1. If n > f f i (see problem 9.4-c)) 6 1 / n < i = £ =>• a < fc^-1/" = ^ r-i/n ^rue for r ationals). Also r — 1/n < r < x + y. So, r — A — x < y. 3v€<Q>3r-^-x<v<y. T hen, v < y and r - ^ -1> < x. T hus, 6W € -B(y) r and 6 - i - " 6 B (x). That is, bv < by and br-*-v < bx « • 6 r - " = br~"-vbv < bxby <a< br~". We have a contradiction from the first and the last terms of the above relation. 9 .4 a) bn-l = (b-l)(bn-1+bn-2 + --- + b+l) > ( 6 - 1)(1H hi) > (b-l)n. b) Let t = b1/". Apply part a) for t: tn - 1 > n(t - 1) => b - 1 > n{bxln - 1). c)£fA<n=>^<t-l=>^± bl'n < t. + l < i . We have ^=i > blln - 1. Thus, d) Let t = $ • > 1. Use part c), 6 1 /" < t = $ • => fc^+V" = fc^1/" < 4 y if e) y > 0 => t = ^ lAV — l/n > 1. If n > bjE$, use c), 6 1 /" < t = % => y < •& = f) Claim: A is bounded above. Proof: If not, V/3 > 0, 3w G A 3 w > j3. In p articular, Vn G N, 3w G A 9 w > n . Hence, Vn G N, 3w G A 3 6" < 6W < y, i.e. Vn G N, 6" < y. If 0 < y < 1, we have a Contradiction since bn > 1. Assume y > 1, use (c) \fn 3 n > fEj, V1/n < b => y < bn. Hence, Vn 3 n > | 5 j we have bn <y <bn, C ontradiction. Let x = sup(yl) = sup {w G R : bw -< y). S olutions 269 Claim: bx = y. Proof: If not, bx < y or bx > y. If bx < y, by (d) Vn G N, bx+lln < y, x + 1/n G A. C ontradiction to the upper bound x > x + 1/n. If bx > y, t hen Claim: if u < x, u G A. Proof: u < x => u is nor an upper bound of A. 3weA3u<w=>w-u>0^> bw~u > 1 =>• | ^ > 1 =*- bw > bu. So, u G A. x y < b => use (e) Vn G N 9 y < bx~1/n; so x + 1/n g" A. T hus, x < x - 1/n (u < x =$> u 6 A), C ontradiction. Hence, bx = y. g) Let b > 1, y > 0 be fixed. Suppose x ^ x' B bx — y = bx . W ithout loss of generality, we may assume that , x < x' =$• x' — x > 0 => frx-x _^ fox > fjx ^ C ontradiction. 9 .5 a) Vz G F, z2 y 0 (if z = 0 => z2 = 0. If z y 0 => z2 y 0). Assume that x ^ 0 => x2 y 0. If y2 >: 0 => a;2 + y2 y 0, Contradiction. So x = 0, then $ 4 x 2 + y 2 = 0 + y2 = 0 => y = 0. b) Trivial by induction. 9 .6 Note that "a ~ 6 if a — 6 is divisible by TO" is different from saying ' < 2 ^ is an integer", since the above one is defined for all fixed m G Z including m = 0, but the latter one is defined for allTOjt 0. a) a ~ a, Va G Z ( take k = 0 ). Then, ~ is reflexive. a ~ 6 = 4 > 3 A : G Z 9 a — 6 = mk. T hen, b — a = m(—k) where — k G Z. Thus, b ~ a, yielding that ~ is symmetric. a ~ & and 6 ~ c => 3fci,/c2 € Z 9 a — b = k\m, b — c = k^m. T hen, a — c — (fci + k2)m where &i + fo G Z. Hence, a ~ c, meaning that ~ is t ransitive. T hus, ~ is an equivalence relation. b) Case 1:TO= 0. Then, a ~ b o - a = b. So, [a] = {o}, and the number of equivalence classes is 00. Case 2:TO^ 0. Then, a ~ 6 o 3 f c G Z 3 a = 6 +TO/C.Hence, [a] = {a, a + m, a — m, a + 2m, a — 2m, • • • } , and the number of distinct equivalence classes is \m\. 9 .7 a) x ~ y =>• a; G [0,1] and ?/ G [0,1] =>• y ~ x (i.e. symmetric). a; ~ y and y ~ z =$• x G [0,1] and y G [0,1] and z G [0,1] => x ~ 2: (i.e. t ransitive). If x ^ [0,1], t hen x ~ x does not hold. For reflexibility we want x ~ x t o hold 270 Solutions Vx € R. Hence, ~ is not reflexive. b) T he s tatement x ~ y =>• y ~ x, x ~ y a nd y ~ x => x ~ x; therefore, x ~ x, Vx € X s tarts with t he following assumption: Vx € X, 3y 6 X 3 x ~ y. If ~ is s ymmetric a nd t ransitive a nd also has t his additional property, then it is necessarily reflexive. B ut if it does not have this property, then it is not reflexive. 9 .8 a) We will make t he proof by induction on n. If n = 1, X = X\ is countable by h ypothesis. Assume that t he proposition is t rue for n = k, i.e. X\ x • • • x Xk is countable. We will prove t he proposition for n = k + 1, i.e. prove that X = Xi x • • • x Xk x Xk+i is c ountable. Let Y = X\ x • • • x Xk- T hen, X = Y x Xk+i a nd Y is countable by the induction hypothesis. Then, t he elements of Y a nd Xk+i can be listed as sequences Y = {y\,y2, • • •}> ^Oc+i = { xi,X2,...}. Now, for X = Y x Xk+i, we use C antor's counting scheme a nd see that X is countable. b) Let X be c ountable. Then, X = { xi,X2,...}. Let A = { x2,X3,...}. T hen, A is a p roper subset of X a nd / : X \-> A defined by f(xn) = xn+i, n = 1 ,2,... is one-to-one a nd onto. Thus, every countable set is numerically equivalent t o a p roper subset of itself. c) If / : X H-> Y is o nto, then 3g : Y H4 X 3 f o g = idy. Moreover, g is one-to-one. Let A — g(Y), t hen Ac X and g : Y H-> A is one-to-one a nd o nto. So, A ~ Y. Since A C X a nd X is countable, A is either finite or c ountable. To see t hat A c annot be u ncountable, we express X — { xi, X 2,...}. If A is not finite, then A = {xi1,Xi2,...}, where in's are positive integers a nd in ^ im for n ^ m. Now, we define / : N »-» A by / ( n ) = Xj n . T hen, / is one-to-one a nd onto. If A is finite, A ~ y => Y" is finite; if A is c ountable, >1 ~ Y => Y ^ * is countable. Thus, Y is at most countable. Solutions 271 P roblems of Chapter 10 1 0.1 F ix x,y eM.k a rbitrary. k k d2(x,y) = \^2{xi - Vi)2]1/2, di(x,y) = ^ » =i i |a?i - j / j | , i=i doo{x,y) = max{|a;i - j/<|} = \XJ - % | . di ~ doo-. doo(x,y) = \XJ -yj\ < ^2\xi » =i -yi\ = di(x,y) => A = 1. doo(x,y) = |£j — 3/j | > \xi -yi\, =>• kdocix^) = k\xj - yj\ > ^ Vi = 1,2, ...,/c |xi - i/j| => B = k. [ ^ ( ^ y ) ] 2 = (XJ - yjf < ^2(xi » =i - y{f => d^x.y) < d2{x,y) =>A = 1. [doo(x,y)]2 = (a:, -1/,)\ 2 > \xt - yt\, Vi = 1,2, ...,fc =>•fc[doo(a;,2 / ) ] 2 > M 2 ( x , 2 / ) ] 2 ^ S = Vfc. d i ~ c^: c?i ~ ^oo and d2 ~ d x =>• <^i ~ ^2<> 10.2 Consider the discrete metric dtp, a) = < ' . ' on X. Br(p) = {p} ,Br\p} = X, Br(P) = {p} ? X. 1 0.3 Let 0 ^ A C X . A is both open and closed. Let B = Ac, B is also both open and closed. A U B = X. If A is closed then B is open, we have A (1B = A n B = 0. If B is closed then .4 is open, we have B n i = 4 n B = 0. Thus X is disconnected. ( =*) X is disconnected. 3 4 ^ 0,3B ^ 0 B X = AU B and (An B) n (Af) B) = 0 =» A n B = 0. Thus Ac = B ^ 0 =» A g X . i U B = I = > v 4 U B = I , . 4 n B = 0=^vl = (B) c , i.e. ^ is open. (<=0 272 Solutions AUB = X =>AUB = X, AC\B = %=>B = (A)c i.e. B is open. A and B are separated and A\J B = X => A = Bc so A is closed. Similarly, B is closed. 1 0.4 Let us place the origin at the lower left corner of the PCB. Then, I I |Ei i i i i ,, A= hi l ,B= "1] 4 " Si t\ [ 91 ,c= ,F= 2 l> = \°~ b ( T' = 4J [6" ("fi "7] ,H = ,/ = 2 Lu [b " 8l 3 ,K = " ql 1 ,E = Ml ,/ = .L= " ql 7 Use /i norm: k A B CD E F G H I J K L A032 B303 C230 D634 E856 F634 G696 H967 I785 J987 K 8 11 8 L 14 11 12 6 86 6 9 7 9 3 53 9688 4 64 6 7 5 7 0 2 4 10 5 9 9 2 0 4 10 5 9 9 4406355 10 10 6 0 5 3 5 5535044 9953402 9955420 12 12 8 4 7 3 3 8 6 8 10 5 7 5 8 14 11 11 8 12 12 8 12 6 88 4 10 75 37 35 06 60 Nearest neighbor (in l\ m etric): i n t C K ) B 4 D(D or F) H-> E i-» F H> H M- / ( / or J) H> J H->- i f H-S- G »> L h-> A -• I nitial tour length is 54. See Figure S.17. S olutions 273 F ig. S.17. N earest neighbor (in h m etric): initial solution -6- Delete (E, F) k, (L, A): g ain=18-12 Improved Tour: Length is 48 F ig. S.18. N earest neighbor (in h m etric): first improvement T he gain values are tabulated below. See Figure S.18. GAIN (B,D) (D,E) (E,F) (F,H) (H,I) (I, J) (J,K) (K,G) (G,L) (L,A) -2 -8 -6 -8 - 8 -10 -12 -8 -6 (A,C) -4 -2 -4 (C,B) -8 -8 -12 -14 -4 2 -2 -2 - 8 -12 -14 -14 (B,D) -4 0 -4 (D,E) - 8 -14 -16 -16 -4 0 (E,F) -2 -8 -10 -10 -4 6 (F,H) -4 -6 -6 2 0 -2 0 2 6 (H,I) -2 4 0 (I, J) 2 4 (J,K) (K,G) 6 T he maximum gain is 6, due to the deletion of (E, F) a nd (L, A). T h e situation a fter this step is illustrated in Figure S.19. 274 Solutions ^ i s- —§*•# ! * te L •*- "j , j»M1 r M_. ^ f X Delete {I, J) & ( G,L): gain=12~8 \ ' Improved Tour: Length is 44 Fig. S.19. Nearest neighbor (in h metric): second improvement b) Use I2 norm: hA A B C D E F G H I J K L BCDEFGHI JK L 0 3 \/2 \/26 \/40 VT8 \/26 \/4l A/37 \/53 8 10 3 0^ V5 \/i3 3 y/41 ^ ViO V50 V73 V73 V2 VE 0 4 \/26 \/8 V^O 5 5 \/37 \/50 \/74 \/26 \/5 4 0 \/2 \/8 \/40 vT7 \/4T V45 \/74 \/50 \ /40vT3\/26 \/2 0 Vl0 \/58 \/l3 \/4l V i l \/72 6 \ /l8 3 v/Sv/Sv/lOOv/^Ov/syHv/rrv/slv/Si V^6 \/4l V^O \/40 \/58 v 7 ^ 0 5 \/5 \/l3 v ^ v/58 \/41\/26 5 VTf Vl3 VE 5 0 vTO >/8 5 \/l3 \/37\/40 5 x /ilv'ilx/is VE Vw 0 V2 VE v ^ >/53 v^O \/37 V^45 v^41 V^7 Vl3 v ^ "v^ 0 VE Vl7 8 ^ 73 \/50 V74 \/72 \/34 VTO 5 \/E VE 0 6 10 V73V74VE0 6 v/34 v ^ \/T3 %/29 VT7 6 \/0 Nearest neighbor (in I2 m etric): <-+I>-+ G{G or I nitial tour length is 38.3399. See Figure S.20. K)^K^L^-A S olutions 275 F ig. S.20. N earest neighbor (in £2 m etric): initial solution T h e gain values are tabulated below. See Figure S.21 for the improvement. GAIN (B,D)(D,E)(E,F)(F,H)(H,I) (A,C) - 3.35 - 7.37 - 4.58 -5.59 (C,B) - 3.96 - 2.70 -3.46 (B,D) - 1.04 -2.65 (D,E) - 2.78 (E,F) (F,H) (H,I) (I, J) (J,K) (K,G) (I, J) - 9.45 - 8.76 - 9.82 - 10.28 - 5.43 - 3.64 (J,K) - 6.90 - 6.93 - 9.06 - 10.37 - 5.48 - 4.13 - 1.70 (K,G)(G,L)(L,A) -7.59 -9.19 -7.62 -7.38 0.63 -9.61 -7.38 -1.41 -11.12 -7.19 -1.98 -7.12 -5.15 2.92 -4.07 -1.20 0.00 -1.25 -0.29 1.94 - 1.27 -0.21 1.21 - 1.62 1.75 - 2.45 -8.24 -7.01 -6.74 -6.28 - 1.74 J. jE Delete (E,F) k (L,A): g ain=2.9196 Improved Tour: Length is 35.42026 F ig. S.21. N earest neighbor (in h m etric): improvement 276 Solutions c) Use loo norm: UABCDEFGHIJKL 0315635 3022335 1204524 5240126 6351037 3322304 5546740 5544325 6655532 7766543 8877653 8877657 Nearest neighbor (in loo m etric): 5 5 4 4 3 2 5 0 3 2 4 3 678 678 56 7 56 7 55 6 34 5 23 3 324 0 12 10 2 22 0 54 6 8 8 7 7 6 5 7 3 5 4 6 0 >-»/>-> G(G or K) K+ K .-> L H-> A Initial tour length is 33. See Figure S.22. Fig. S.22. Nearest neighbor (in lx metric): initial solution Solutions 277 T he gain values are tabulated below. See Figure S.23 for the improvement. GAIN (B,D) (D,E) (E,F) (F,H) (H,I) (I, J) (J,K) (K,G) (G,L) (L,A) (AC) (C,B) (B,D) (D,E) (E,F) (F,H) (H,I) (I, J) (J,K) (K,G) -4 -8 -4 -4 -3 0 -4 -3 -3 -2 -8 -7 -7 -6 -2 -10 -9 -9 -9 -4 -4 -7 -6 -8 -9 -4 -6 -2 -8 -7 -7 -8 -6 -6 -2 -1 -8 -7 -7 -6 -2 2 0 0 -1 0 -3 -4 2 0 0 -1 0 -4 Delete (E,F) & (L,A): gain=2 Improved Tour: Length is 31 Fig. S.23. Nearest neighbor (in /oo metric): improvement d) Case 1: we need to complete the tour for the consecutive PCB's: C urrent situation (/i norm): Tour duration is 44 time units. P roposition 1 (l2 n orm): Tour duration is 35.42026 time units. P roposition 2 (/<*, norm): Tour duration is 31 time units. P roposition 1 is economically feasible if (44 - 35.42026)./VCo > C\. Similarly, proposition 2 is economically feasible if (44 - 31)NC0 > C2. Case 2: we may delete the most costly connection: For the odd numbered PCBs among 1 , . . . , N; 278 Solutions w ith with with li norm: Li-tJi-tKi-tGi->Ii->Hi->Fi-+Ai-*Cy-*B*->Di-*E length 38; l2 norm: f H C ^ / n J ^ i / ^ F ^ i ^ C ^ B i - y D ^ B ^ L length 29.42026; loo norm: K^G^I^J^H^Ft-^A^C^B^D^fE^L l ength 25. For t he even numbered PCBs, we reverse t he order as li norm: l2 norm: loo norm: E*-*Di-+Bi->Ci->Ai-tFi->Hi->Ii->Gi-tKi-tJi-tL; L^tE\-^D^B^C^tA^F^H^J^I^G^K\ Li-^Ei-^D^Bi-^Ci-^Ai-^Fi-^Hi-^J^Ih-^G^K. C urrent situation (li n orm): Path duration is 38 t ime units. P roposition 1 (l2 n orm): Path duration is 29.42026 time units. P roposition 2 (loo n orm): Path duration is 25 t ime units. P roposition 1 is economically feasible if (38 - 29.42026)./VCo > C\. Similarly, proposition 2 is economically feasible if (38 — 25)NC0 > C2. If 8.57974 < jfe- a nd 13 < jf^r, t hen we keep t he existing robot a rm configuration. Otherwise, we select proposition 1 if 0.65998 > Q-\ select proposition 2 if 0.65998 < £ L . Solutions 279 P r o b l e m s of C h a p t e r 11 11.1 a) (=>): Let e > 0 and XQ be given. Let b — / (xo) — e. T hen, by assumption, t he set B = {x £ X : f(x) > f(xo) — e } is open. Moreover, xo £ B since / (x 0 ) > f(x0) - e. So, 36 > 0 3 Bs(x0) C B; t hat is, x £ Bs(x0) => x £ B => f(x) > f(x0) - e. («=): Let b £ K be given. We will show that the set A = {x £ X : / (x) > b} is open. If A = 0, then A is open. Assume A ^ 0, show that every point of A is an interior point. Let XQ G A. T hen, /(xo) > b. Let e = f(xo) — b. T hen, by our assumption, 35 > 0 3 x € Bs(x0) =>• /(x) > / ( x 0 ) - e = J ^ i € i Hence, BS(XQ) C A, t hat is a?o S inM. b) Similar as above. 1 1.2 / is continuous and X is compact =>• / ( X ) = B is compact in y . q G / ( X ) = B — B since .B is compact, therefore closed. So, by q £ B — f{X), we have 3p € X B q = / (p). Next, we will show that pn -* p. f : X <-+ B is continuous, one-to-one and onto. Since X is compact, f"1 : B H-> X is continuous. Moreover, f(pn),q £ J3 and /(p„) — g. Then, > f-Hf(Pn)\ , / -H9) 1 1.3 Let the wire be the circle Cr = { (x, y) : x 2 + 2/2 = r - 2 }. For a = (x, y) G C r , let T(a) be the temperature at a and let / : Cr t-> K be such that / ( a ) = T(a) — T (—a). Note that a and —a are diametrically opposite points. T hen, T, and hence, / are continuous. Claim: 3a e Cr 3 f(a) = 0. Proof: Assume not, Ma G Cr, T(a) ^ T(-a). Define A = {a £ Cr : f(a) > 0 }, B — {a £ Cr : f(a) < 0 }. Then, A and B are both open in Cr. Why? (since t hey are the inverse images of the open sets (0, +oo) and (-co, 0) under the continuous function /.) A n B ^ 0, because of the heated wire; AL)B = Cr, since we assumed Va G Cr, T(a) ^ T(—a); moreover, A / 0, there is at least one point (the point where heat is applied). Suppose not, then Cr = B, \/a G Cr, / ( a ) < 0 < > T (a) < T(-a). B ut, then T ( - a ) < T(-(-a)) £ = T(a), C ontradiction. Hence A ^ 0. Similarly, with the same argument, 5 ^ 0 , think of the opposite point to where heat is applied. So, A is nonempty, proper (Ac = B =fi 0) subset of Cr which is both open and closed (Ac is open). Thus, Cr is disconnected. Contradiction. A nother way of proving the statement is the following: Let x £ A and y £ B, a nd we know that / is continuous as well as / ( x ) > 0 > f(y). Apply t he intermediate value theorem (Corollary 11.4.2) to conclude that 3a £ Cr 3 / ( a ) = 0. 2 80 Solutions P r o b l e m s of C h a p t e r 1 2 1 2.1 Use the Mean Value Theorem: h : R > » R is nondecreasing if h'(x) > 0. — y<x=> h(x) - h{y) = ti{c)(x - y) > 0 => /i(z) > /i(y). < xf'(x), 0<c<x. .9%T) = xf'(x)x;fix\ f(x) = f(x) - / (0) = f'(c)x So g'(x) > 0, V.T =4- g is nondecreasing. 1 2.2 Use the Mean Value Theorem: fi(y) - fi(x) = / /(cj)(y - x). / ' = 0 =» /.; = 0, Vi; thus /,(y) = /',;(x') which means / is constant. 1 2.3 | | ( 0 , 0 ) = cos(0 + 2-0) = l, §£(0,0) = 2cos(0 + 2-0) = 2 ; 0 ( 0 , 0 ) = 0, 0 ( 0 , 0 ) = 0, 0 ( 0 , 0 ) = 0 and j g £ ( 0 , 0 ) = 0. f(x,y)=x + 2y + R2(x,y)(0,Q), where j g f # ( 0 , 0 ) ^ 0 as {x,y) -* ( 0,0). 1 2.4 a) Let us take the first order Taylor's approximation for any nonzero direction h, f(x* + h) = f(x*) Since , I Vf(x*)Th + R,(x*, h), R ^f'h) - 0 as h - 6. 'y | | i t} w | / i 7 V 2 /(£)/2, where £ = ,x* + a/j,, 0 < a < 1, we say that /(.x* + / ' 0 « / ( x * ) + V/(:r*) T ft. Since x* is a local minimize!', fix*) < f(x* + h), V/i small. Therefore, for all feasible directions Vfix*)Th > 0, where the left hand side is known as t he directional derivative of the function. Since we have an unconstrained minimization problem, all directions h (and so are inverse directions —h) are feasible, Vfix*)Th > 0 > -Vfix*)Th = Vfix*)ri~h) > 0,V/i ^ 0. T hus, we must have V/(.x*) = 9. b) Let us take the second order Taylor's approximation for any nonzero (but small in magnitude) direction h, fix* + h)^ fix*) + Vfix*)Th -f l -hTV2fix*)h. Since Vfix*) = 0, we have Solutions 281 f(x*+h)^f(x*)+1-hTV2f(x*)h. Suppose that V 2 /(x*) is not positive semi-definite. Then, 3 « e l n 9 vTV2f(x*)v < 0; even for the remainder term, vTV2f(£)v < 0 if \\x* — £|| is small enough. If we take h as being along v, we should have f(x*) > f(x* + h), C ontradiction t o the local minimality of f(x*). T hus, V 2 /(x*) is positive semi-definite. If we combine the first order necessary condition and the second order necessary condition after deleting the term -semi-, we will arrive at the sufficiency condition for x* being the strict local minimizer. c) At every iteration, we will approximate f(x) by a quadratic function Q(p) using the first three terms of its Taylor series about the point xk-\: f(xk-i + p) « / (x f e -i) + V/(x f c _ 1 ) r p + -pTV2f(xk^)p = Q(p); and we will minimize Q as a function of p, t hen we will finally set xk = Xk-l +PkLet us take the derivative of Q: dQ dp V /(x f c _i) + V 2 / ( a * - i ) r P * V/(x f c _! + p). Since we expect 6 = V /(x f c _i + pk) « V /(x f c _i) + V 2 /(x f c _i) r p f c , V 2 /(fc-i) r Pfc = -V/(x f c _x) &pk = - [V 2 /(x f t _ 1 )]- 1 V/(x f c _ 1 ). T his method of finding a root of a function is known as Newton's method, which has a quadratic rate of convergence except in some degenerate cases. N ewton's method for finding V/(x) = 8 is simply to iterate as xk = xk-i — [vvock-or'v/fo-i). v /( v 2 /( Let X(o) = 1 1 Xi X2 d) See Figure S.24 for the plot of the bivariate function, / ( x i , x 2 ) = x\ + 2x\ + 24x2 + x\ + 12x2, m t n e question. Xi X2 ) "[ 1 1 )" 4xf + ®xi + 4 8 x i 4x1 + 2 4 x 2 )" 12xf + 12xi + 48 0 1 2x2 + 24 0 11 ™ 58 , V2/ 28 72 0 . T hen, 0 36 =^v/ 282 Solutions - 0.5 - 0.5 -1 -1 F ig. S.24. Plot of f(xux2) 0.194444 0.222222 0.194444 0.222222 = xi + 2x\ + 2Ax\ + x% + 12x| 9.589592 5.377229 50.78704 0 0 24.59259 V/ V2/ T hen, x{2) = 0.194444 0.222222 V/ V2/ T hen, Z (3) = 5 0.78704 2 4.59259 9.589592 5.377229 0.270179 0.085676 0.005625 0.003570 0.005625 0.003570 0.005625 0.003570 48.06788 0 0 24.00015 0.005625 0.003570 l 48.06788 1 24.00015 0.270179 0.085676 0.000191000 0.000000364 0.000003981 0.00000002 V/ 0.00000398 0.00000002 Solutions V2/ T hen, x 283 0.00000398 0.00000002 48.00005 0 0 24 w= 0.00000398" 0.00000002 • 1 4 8.00005 1 24. 0.0001910000] 0.000000364 J ~ , which is r^ 40 U 1.98 x 10" 1 2 0 Finally, V / ( T hus, x* = 1.98 x 1 0" 1 2 ' n )r^n" J/ ' 9.5 x 1 0" 1 1 ' 0 L . Since V 2 /(x*) is 0 0 24 diagonal with positive entries, it is positive definite. Therefore, x* = 6 is a local minimizer with f(x*) = 0. =• v/(x*) = e, v2/(x*) = " J «n 284 Solutions P roblems of Chapter 13 13.1 Let there be given two series A = 2~] Uk and B — \ with nonnegative terms. (a) / / W < Vk, yk, the convergence of series B implies the convergence of f c series A and the divergence of series A implies the divergence of series B. Suppose that B is convergent. Let S = Yl™ vk be finite. n n vk ^2 uk < ] P vk < S, n = 0 ,1,... o o t hus partial sum of A is bounded, hence it is convergent. Suppose that A is divergent. Thus its nth p artial sum increases indefinitely t ogether with n. n 0 th n 0 T hus, n p artial sum of B increases indefinitely together with n, too. T hat is, B is divergent. (b) / / lrnifc-^oo ^ = a > 0, then series A and B are simultaneously convergent and divergent. limfc-»oo % = a > 0, vk > 0, V*. Then, Ve > 0 3N 9 a - e < — < a + e, Vfc > N. Vk =>• Vk{a — e) < Uk < Vk(a + e). If B is convergent, so is ^ ^ ° Vk(a — e). T hus, A is convergent by (a). If B is divergent, so is ^ ^ ° vk{a — e). Thus, A is divergent by (a). 1 3.2 a) Eo°° X~k\^0 I t is convergent for x = 0. Let us assume that x > 0. I*±! uk J*+i)i * * _• 0 as A - • oo. k+1 = = S olutions 285 T hus, it is convergent. b) £ r f^> where a > 0: I t is convergent for x = 0. Let us assume that 0 < x < 1. *"+xa f = -—r— = x I k \a 1 — x as k — oo. > > «fc f£ \k + lj T hus, it is convergent. If x > 1, it is divergent since ^ ^ —• x as fc — oo. If > > x = 1, we have E i ° fc~a, a > 0. Then, Un+l l im —, , n->oo « „ \k + 1 T he series is convergent when a > 1 and divergent when a < 1. In the special case where a = 1, it is (f), the harmonic series which is divergent. c) J2T(ei - 1): e* - l s a ^ a s A : — • oo. Thus, it is divergent (see part f) below). d )£rm(l + £): In (l + jr) ss jr as k — oo. Thus, it is divergent (see part f) below). > e) E r qk+^, where q- > 0: t /u^ = ^1+fc -> 5 as k -> oo. Thus, it is convergent for 0 < q < 1 and divergent for q > 1. If q = 1, then w^ = 1 and E 1 is divergent. oEra >oo 1 , ^=*±I " A: £ = - * - - > l a s *->«>. fc + 1 T he Harmonic series is divergent! 1 3.3 a) For each object i = 1 , . . . , n, either it is selected or not; that is Xi € Si = { 0,1}. T hen, n 5 (x) = J J( a:0 + ; cl ) = (1 + x ) n ' W ithout loss of generality, we may assume that r — E xi objects are selected. We know from Problem 1.3.a) t hat the number of distinct ways of selecting 286 Solutions r <n objects out of n objects is (™). Thus, ar = (™). We cannot choose more t han n objects; that is ar = 0, r > n. Therefore, X , ,_n V / r =0 \rI Let us prove the power expansion as a corollary to the Binomial theorem. T he Binomial theorem states that (l + z)n = £ " = 0 (").?*. Let z = | . T hen i + *V = y ^ fnNi f £Y ^ (x + y \ n = (x + n)n = \ p (n\ (^ v y) i^\i)\y) \ v) v f jwW i =0 ^ ' Let us prove the multinomial theorem as a corollary to the Binomial theorem by induction on k. z i,.. . , 4 S Z+ h-\ \-ik=n Let Z = 2 a nd a;i = x, X2 = y. We use the power expansion to state that t he induction base (k = / = 2) is true. Let use assume as induction hypothesis t hat ( *' + '••*>"- , £ (, n ,/ \ "„.".,«i)• holds. (Xi + • • • + Xi + Xi + 1)n = i i , . . . ,h,ii+i € Z + «i H M ; + ij+i = n needs to be shown. Let x = xi + • • • + X[ and y = xi+i in the power expansion. Solutions 287 (x + y)n = Yf(^)(x1 i=0 + -.. + xl)ixti . „ ,t/ *-^ \«i,••-,«/ i\-\ Yi\ — i Y^ I• 71 ". • * »i «*i™ ix^-x^x*«+' +i i i ; x x b) For each object i = 1 , . . . , n, either it is not selected or selected once, twice, t hrice, and so on; that is x, £ Si = Z +. Then, n </(x) = J J ^ 0 + x 1 + x 2 + • • •) = (1 + x + x 2 + • • • ) " , » =i W ithout loss of generality, we may assume that r = J3 Xj objects are selected. We know from 14.4 that the number of distinct ways of selecting r objects out of n objects with replacement is ( " j ^ 1 ) = ( n ~"* +r ). Thus, ar = (n~l+r). Therefore, 5(x) = ( i+x+x 2 + --.)n = ] r ( n ~ 1 + r V . r=0 ^ ' x i + x 2 + x 3 + x 4 = 13, Xi = 1 ,2,3,4,5,6 Vi => 5 (x) = (x + x 2 + x 3 + x 4 + x5 + x 6 ) 4 = x 4 (l + x + x 2 + x 3 + x 4 + x 5 ) 4 We are interested in the coefficient of x 13 of g(x), which is the coefficient of x 9 of h(x) = (1 + x + x 2 + x 3 + x 4 + x 5 ) 4 . p(x) = 1+ x + x 2 + x 3 + x 4 H xp(x) = x + x2 + x 3 + x 4 + • • • (1 - x)p(x) — 1 =^> p(x) = 1-x 288 Solutions p(x) = 1 + x + x2 + x3 + x4 + x 5 + x6 + x7 + Similarly, x6p(x) = (1 - x6)p(x) = 1 + x + x2 + x3 + x4 + x5 T hen, x 6 + x5 + = 1 — x6 1-x h(x) = (1 - x6)4\p(x)}4 = (1 - x6)4(l +x + x2+x3 + x4 + ---)4 = k(x)l(x); by the Binomial theorem and by the multiset problem «*> - W w* ur w Therefore, the probability is 3\ . + M \ +/ 5 \ + /6\ 1 +/7\ + 2 3 4 U r • • •+1 • r + . / 12 T he ninth convolution of k(x)l(x) is the answer: 'J) C . a )-(0(0-»-"»'— 140 P (having a sum of 13) = -TJ- = 0.1080247 d) an - 5 a n _i + 6a n _ 2 = 0, Vn — 2 , 3 , 4 , . . . <=$ anxn - 5 a„_ix n + 6 a n _ 2 z n = 0, Vn = 2 , 3 , 4 , . . . Summing the above equation for all n, we get oo oo oo ^2 anx n=2 n - 5 ^2 an-ix n=2 n + 6^ n—2 a „_ 2 a;" = 0 \g{x) - axx - a0] - 5x[g(x) - a0] + 6x2[g(x)} = 0 Using the boundary conditions (do = 2 and ai — 5) we have ,. ao + a\x — 5a,QX 2 — 5x 6x2-5x + l (3x-l)(2x-l) 1 l-2x 1 1 - 3x g(x) = (1 + 2x + 4x 2 + • • • + 2 V + •••) + (1 + 3x + 9x2 + ••• + 3 V + •••) Solutions 289 =>an = 2n+3n. 1 3.4 a) The left hand side of the following constraint represents the complementary survival probability of a threat, l-\\{l-Pji)Xii>du\/i. 3 T hen, 1 - dk > H(l-Pji)Xji 3 <* Clog(l - dj) > X )[Clog(l -Pji)}xji,VC 3 > 0- W ith a suitable choice of £, and let —6; = (,log(l — di), —Oji = £ log(l —Pj%), we will have 2_Ja0iX3i - ^»> ^*' Let ajj = [fljij and /3j = [6«J (with a suitable choice of £ > 0), yielding ^a,^ 3 > /3;, Vi. b) The first three objective functions are equivalent to each other, so are the last two. The flaw lies in the equivalence of the third and the fourth objective functions: m ax/3; 2 ^k m in(l — /3,z). In particular, m ax y x + y2 + 2/3 = m in(l - yx) + (1 - y2) + (1 - 2/3) is true. However, maxj/ij/22/3 = min(l - 2/i)(l - y2)(l - y3) = 1 is false because of the cross terms. 2/12/22/3 290 Solutions P roblems of Chapter 14 1 4.1 y"(t) - y(t) = e2t & s2r](s) - 2s - r)(s) = s-2 <& r?(s)(s2 - 1) = s-2 2s r + 2s. 1 V(s) = ( s - 2 ) ( s 2 - l ) 1. If V(s) = ^ + 2-l' s fOT ABC + ^ r + ^ r = ((,-vU-iy - l ) - S o l v e = »-2)(^ A+B+C=0 W + C =0 -A + 2 5 - 2C = 1 ''- *H-*4C=T= ^ j . Solve for tf,F: = » E = 1, F = 1. Thus, ??(s) = ^ T V T ^ -(3+1) 6(8-2) + 6 2. If V(a) = ^ + ^ = I ^ I = 6(s-l)r) £ + F=2 E-F =0 T hus, i;(a) = Then, we have - +• -+1 ~ « +!• ^ 1 4.2 = 3 (^2) + 6 ^TI) + 2 ( ^ 1 ) ^*<*> = = r)(z) + 2 +2 r2t+h~*+?*• = z+2 z—e 2 /0 + 1) = t/(fc) + 2efe <=* zt](z)-z V{z) = z r <=> rj(z)(z-l) z — e* 2-1 (2-l)(,z-e)' If»?(«) = rz —l)(z —e)v = z —1 + z —e t hen A = ^ - and B = e-- 1V -A -s( -TTi 'V / (2-lllz-ei z— 1 z —e 1 —e -^ Therefore, !/(*) = 2-1 +2 1 A 2 1-e u - 1 ; +e— 1 \ z— e «.j,(fc) = l + (l-efc). 1—e 1 4.3 ^ = - 0.2y, ^ = - 0 . 3 s - O.ly, x(0) = 50, y(0) = 100. § -o*-a l .*g~a4-<u4^- I u», +I u2 = o<*) Solutions dt2 = s2r](s) - sy(0) - y (0) = s2rj(s) - 100s + 25, 291 since %\t=0 = - 0.3(50) - 0.1(100) = - 2 5 . Moreover, V dt &y_ sr)(s) - j/(0) = srj(s) - 100 ( *) : [s2rj{s) - 100s + 25] - 0.06??(s) + 0.1[s?y(s) - 100] = 0 <£> v(s)(s2 - 0.06 + 0.1s) = 100s - 25 + 10 &• ri(s) = 100s - 15 (s + 0 .3)(s-0.2) 90 s + 0.3 10 s-0.2 A s + 0.3 B + s-0.2 A + B = 100, - 0 . 2 4 + 0 .3S = - 1 5 => 4 = 90, i5 = 10 => ,., ,, y(t) = 9 0e-°- 3t + 10e°-2t r 2/(o)i 2/(1) 2/(2) 2/(3) L 2/(4) J 1 4.4 "100.0000" 78.88767 64.31129 = 54.81246 49.36289 Let x(n) — Fn+i, a nd the initial conditions are x(0) = 1, x(l) = 1. x(n + 1) = x(n) + x(n - 1), n - 2, 3 , . . . ( ¥) r][x(n + 1)] = zrj(z) — zx(0) — zr)(z) - z a nd rj[x(n - 1)] = \r){z). ( ¥) : x(n + 1) = x (n) + x (n - 1) o- zrj(z) - z - rj(z) <*ri{z) 7]{z) = •«• r)(z) = 0 z-l-± z ^ z z z{l-±-\) z^ z/ J 1 l_ A ) ^ ^ l (l - ^ ) (l - ^ + £ 1 1 +75 22 4 +5 =1 , M* ) + B f i2zi ^ ] = 0 ^ =0 ^ 5 = ^ ^ 2 1 10 ( ¥) : V(z) Since Z 5 + Vb , 10 I ! _ i ^ i ; + 5-\/5 / io ^ i 1_ _ ^ ( yrs-) = any(n), we have 292 Solutions T hus, Finally, Fioo = x(99) = 354 224 848179 261915 075 I ndex affine c ombination, 15 hull, 16 a leph, 133 A rchimedean P roperty, 125 a xiom, 4, 123, 138 B alzano-Weierstrass, 150 b asis, 16, 29, 211, 212, 246 c hange, 19, 55 o rthonormal, 39 r epresentation, 17 B inomial t heorem, 187, 208, 286, 288 c anonical c ombination, 15, 116, 252 C antor, 1 31, 132, 150-151 c ardinal n umbers, 132-133 Cauchy, 176, 178, 179 C ayley-Hamilton, 195 c ombat modelling, 6 5, 202, 228, 290 c ombination, 207-209 affine, 15 canonical, 15, 116, 252 convex, 15, 116, 252 c ombination, 15, 116, 252 cone, 94 hull, 16, 94, 98 p olyhedron, 96 p olytope, 96, 99-100, 115, 241-244, 246 s et, 9 3-102 corollary, 4 C ramer's rule, 51 c ube, 100, 241 D 'Alembert, 177 d-simplex, 99, 241 d ecomposition Cholesky, 74, 75, 231 J ordan, 65, 195, 226 L DU, 22, 23, 53 LU, 21, 24, 25, 44, 47, 115, 220, 221, 249 Q R, 41, 44, 47, 88, 89, 115, 217, 221, 250 singular value, 43, 44, 47, 75, 86, 223 S pectral T heorem, 64 definiteness indefinite, 76 n egative, 73-77, 2 30-233 p ositive, 73-77, 81, 82, 230-233, 281 semi, 75, 281 definition, 3 d eterminants, 5 1-53, 74, 75, 230 cofactor, 5 3, 65, 224 difference e quations, 60-62, 187, 1 97-202, 288, 290, 291 linear, 15 complex, 6 3-65, 125, 127-128, 176, 180-186 c onjugate, 63 h ermitian, 64, 75 u nitary, 64 cone, 16, 94, 100, 243 c onjecture, 4 convex, 140 294 Index g raph, 28, 116, 210, 257 Heine-Barel, 150 hull affine, 16 convex, 16, 94 linear, 16 h yperplane, 95-98, 113 inner product, 35, 40 integer, 129, 133, 266-269 i rrational, 126 i somorphism, 17, 125 kernel, 24-28, 44, 54, 57, 211-215 Laplace transforms, 192-196, 199, 201, 202, 290 law of cosines, 37 L east Squares Approximation, 38-47, 218 l emma, 4 linear c ombination, 15, 95 d ependence, 15 e quation systems, 20, 24, 38, 42, 4 4-47, 8 1-86, 105, 220, 234-240 hull, 16 i ndependence, 15, 55 t ransformation, 18 variety, 95 Linear Programming, 103-119, 246-265 bfs, 105, 246, 250 b ounded simplex method, 116, 253-255 canonical form, 103 column generation, 116, 262 c omplementary slackness, 112, 256 d ecomposition, 116, 257-265 degeneracy, 106 d uality, 111-119, 250, 256 F arkas Lemma, 113-115 m inimum ratio, 106, 116, 254 o ptimality, 105, 116, 253, 255 revised simplex method, 110, 115, 248-250 simplex method, 105, 115, 246 simplex tableau, 107, 115, 248 s tandard form, 103 differential equations, 60, 6 2-63, 65, 1 91-196, 201, 202, 228, 290 d odecahedron, 100, 244 eigen value, 54-65, 74-76, 81, 82, 84-89, 195, 230, 234-240 eigen vector, 54-65, 84-89, 234-240 e xample, 3 F ibonacci, 60, 61, 202, 292 field, 13, 28, 123-129 complex, 125, 127-128, 176, 180-186 d ense, 125, 143 o rdered, 124-129, 133, 134, 266-269 r ational, 122, 124, 130, 133, 143, 267 r eal, 125-128, 131, 133, 134, 143, 268 function b eta, 185 b ounded, 137-139, 159, 164 class C r , 172 c ontinuous, 157-166, 169, 170, 279 d erivative, 169-173, 280-283 differentiable, 169-170, 173 d irectional derivative, 280 d iscontinuities, 162-164, 171 e xponential, 180-183, 195 g amma, 183, 185-186 g enerating, 187, 285-289 I ntermediate Value Theorem, 162, 279 L 'Hospital's Rule, 171 local maximum, 170 local minimum, 170, 173, 281, 283 l ogarithm, 181-182, 187, 289 M ean Value Theorem, 170-173, 280 m etric, 138 m onotonic, 164-166, 173 o ptima, 159, 173, 281 s altus, 165 s emi-continuous, 166, 279 s equence, 178-179 T aylor's approximation, 172, 173, 180, 2 80-281 t rigonometric, 182-184 uniform continuity, 160-161 F undamental Theorem of Linear A lgebra, 27, 35, 215 G aussian elimination, 20, 52, 107 I ndex u nboundedness, 106, 251, 252 Markov process, 61, 87 m atrix a symmetric, 83 b asis, 39 block diagonal, 58 c olumn space, 26, 28, 44, 211-215 c ondition number, 82-86, 234-240 d eterminant, 51 d iagonal, 21, 22, 43, 55, 56, 65, 226, 227 e lementary, 21, 110 h ermitian, 64 Hessenberg, 88 Hessian, 73, 231, 281 i dempotent, 38 i dentity, 52 incidence, 28 J ordan form, 65, 226 left null space, 27, 28, 44, 211-215 m inor, 53 m ultiplication, 17 n onsingular, 74 n orm, 84-86, 234-240 n ormal, 65 null space, 26, 28, 44, 54, 57, 211-215 o rthogonal, 39, 89, 234 p ermutation, 24 p ivot, 21, 22, 53, 74, 75, 231 powers, 60, 227 p rojection, 38 p seudo inverse, 42, 47 r ank, 25, 26, 38, 44, 75 resolvent, 195 row space, 25, 28, 44, 211-215 s ingular, 75 s quare, 51, 81 s ymmetric, 37, 38, 73-75, 81-84, 230 t race, 54 t rapezoidal, 24 t riangular, 21, 22, 41, 53, 54 t ridiagonal, 88 u nitary, 64 m etric du 138, 139, 152, 153, 271-274, 277 d 2 , 138, 141, 149, 152, 153, 271, 2 74-275, 277 doo, 138, 140, 152, 153, 271, 276-277 295 closed ball, 139, 152, 271 d iscrete, 137, 139, 141, 271 o pen, 149 open ball, 139, 143, 152, 271 space, 137, 157-166, 279 m etricspaces, 137-154 m ulti-commodity network flow problem, 116, 257-264 m ulti-set problem, 209, 287, 288 m ultinomial theorem, 286 n atural, 129 n eighborhood, 94 N ewton's method, 281 n orm, 33-35, 40, 82-89, 137 h, 34, 138, 153, 272-274, 277 h, 34, 138, 153, 274-275, 277 loo, 34, 138, 153, 276-277 m atrix, 84-86, 234-240 n umber systems, 121-134, 266-270 o ctahedron, 100, 241 o rthogonality, 35-47 c omplement, 35 G ram-Schmidt, 40, 87 o rthonormality, 39 vector space, 35 p ivot, 21, 22, 53, 108, 110 p oint basic, 105, 246, 250 bfs, 105, 246, 250 b oundary, 71, 77, 94, 231 e xtreme, 93, 94, 98-100, 105, 115, 246, 250 i nterior, 94, 140 isolated, 142, 158, 165, 170 limit, 142, 143, 150, 158, 179 m aximum, 7 1-73, 77, 233 m inimum, 7 1-73, 76, 77, 233 n eighborhood, 142, 143 s addle, 71, 73 s tationary, 73, 77, 231 p olyhedron, 96 p olynomial, 18, 20, 29, 170, 212 c haracteristic, 51 d erivative, 20, 29, 213 i ntegral, 29, 214 m inimal, 58 296 Index closed, 142, 144-148, 150, 153, 158, 161, 271 closed ball, 139, 152, 271 closure, 143, 144, 152, 271 c ompact, 147-150, 1 59-161, 166, 279 c onnected, 151-153, 161-164, 166, 2 71, 279 c ontinuity, 157-166, 279 convex, 93-102, 173 c ountable, 129-134, 150, 270 dense, 143 finite, 129-133 gib, 122, 133, 266 infimum, 122, 133, 159, 266 i nterior, 141, 145 k-cell, 149 l ub, 122, 133, 266 n eighborhood, 142, 143 n ested intervals, 148 o pen, 141, 143-146, 153, 158, 173, 271 o pen ball, 139, 143, 152, 271 o rdered, 121-123 perfect, 143, 150 s eperated, 151, 153, 271 s upremum, 122, 133, 145, 159, 266 u ncountable, 129-133 s pan, 16, 24, 95 S tirling, 186 Taylor a pproximation, 74, 172, 173, 180, 2 80-281 t heorem, 172 t etrahedron, 100, 243 t heorem, 4 T SP, 3 3, 153, 272-278 v ariable basic, 25, 44, 105 d ependent, 25, 105 e ntering, 105, 116, 253 B land's rule, 116, 253 D antzig's rule, 105 free, 25 i ndependent, 25, 105 leaving, 105, 116, 254 n onbasic, 25, 44, 105, 253 vector space, 13 t rigonometric, 184 p olytope, 96, 99-100, 115, 241-244, 246 p rojection, 37-43 proof, 4 proof making, 5-9, 206-209 c ombinatorial method, 207-209 c onstruction, 6 c ontradiction, 8 c ontraposition, 8 forward-backward method, 5, 207-208 i nduction, 7, 98, 225, 227, 269, 270, 286 selection, 6 s pecialization, 6 t heorem of alternatives, 9, 113-115 u niqueness, 7 p roposition, 3 p yramid, 100, 242 Q R algorithm, 88, 89, 234-236, 238 q uadratic form, 76, 281 quantifiers, 4 r ational, 122, 130, 133, 267 ray, 115, 251, 252 R ayleigh, 76-77, 84, 230 regression, 47, 218 r elation equivalence, 129, 134, 269 o rder, 121, 134, 269 r emark, 4 Schwartz Inequality, 35, 128 scientific inquiry, 1 series, 1 75-188, 284-289 convergent, 165, 175-180, 186, 2 84-285 d ivergent, 176-180, 186, 284-285 Fourier, 184 p artial sum, 175 power, 179-188, 285-289 r emainder, 176 t ests, 177-179, 186, 284-285 t rigonometric, 185 set a t most countable, 129-133, 165 b ounded, 121, 143, 150, 161 C antor, 150-151 I ndex c olumn space, 26, 28, 44, 211-215 d irect sum, 59 E uclidean, 128, 147 half space, 96 left null space, 27, 28, 44, 211-215 null space, 24-26, 28, 44, 57, 211-215 o rthogonal, 35 row space, 25, 28, 44, 211-215 s ubspace, 14 W eierstrass, 99, 179 Z -transforms, 199-202, 290, 291 297 Early Titles in the INTERNATIONAL SERIES IN OPERATIONS RESEARCH & MANAGEMENT SCIENCE Frederick S. Hillier, Series Editor, Stanford University S aigal/ A MODERN APPROACH TO LINEAR PROGRAMMING N agurney/ PROJECTED DYNAMICAL SYSTEMS & VARIATIONAL INEQUALITIES WITH APPLICATIONS P adberg & Rijal/ LOCATION, SCHEDULING, DESIGN AND INTEGER PROGRAMMING V anderbei/ LINEAR PROGRAMMING J aiswal/ MILITARY OPERATIONS RESEARCH Gal & Greenberg/ ADVANCES IN SENSITIVITY ANALYSIS & PARAMETRIC PROGRAMMING P rabhu/ FOUNDATIONS OF QUEUEING THEORY F ang, Rajasekera & Tsao/ ENTROPY OPTIMIZATION & MATHEMATICAL PROGRAMMING Y u/ OR IN THE AIRLINE INDUSTRY H o & Tang/ PRODUCT VARIETY MANAGEMENT E l-Taha & Stidham/ SAMPLE-PATH ANALYSIS OF QUEUEING SYSTEMS M iettinen/ NONLINEAR MULTIOBJECTIVE OPTIMIZATION C hao & Huntington/ DESIGNING COMPETITIVE ELECTRICITY MARKETS W eglarz/ PROJECT SCHEDULING: RECENT TRENDS & RESULTS S ahin & Polatoglu/ QUALITY, WARRANTY AND PREVENTIVE MAINTENANCE T avares/ ADVANCES MODELS FOR PROJECT MANAGEMENT T ayur, Ganeshan & Magazine/ QUANTITATIVE MODELS FOR SUPPLY CHAIN MANAGEMENT W eyant, J./ ENERGY AND ENVIRONMENTAL POLICY MODELING S hanthikumar, J.G. & Sumita, U./APPLIED PROBABILHYAND STOCHASTIC PROCESSES L iu, B. & Esogbue, A.O./ DECISION CRITERIA AND OPTIMAL INVENTORY PROCESSES G al, T., Stewart, T.J., Hanne, T. / MULTICRITERIA DECISION MAKING: Advances in MCDM Models, Algorithms, Theory, and Applications F ox, B.L. / STRATEGIES FOR QUASI-MONTE CARLO H all, R.W. / HANDBOOK OF TRANSPORTATION SCIENCE G rassman, W.K. / COMPUTATIONAL PROBABILHY P omerol, J-C. & Barba-Romero, S. /MULTICRITERION DECISION IN MANAGEMENT A xsater, S. /INVENTORY CONTROL W olkowicz, H., Saigal, R., & Vandenberghe, L. / HANDBOOK OF SEMI-DEFINITE PROGRAMMING: Theory, Algorithms, and Applications H obbs, B.F. & Meier, P. / ENERGY DECISIONS AND THE ENVIRONMENT: A Guide to the Use of Multicriteria Methods D ar-El, E. / HUMAN LEARNING: From Learning Curves to Learning Organizations A rmstrong, J.S. / PRINCIPLES OF FORECASTING: A Handbook for Researchers and Practitioners B alsamo, S., Persone, V., & Onvural, R./ ANALYSIS OF QUEUEING NETWORKS WITH BLOCKING B ouyssou, D. et al. / EVALUATION AND DECISION MODELS: A Critical Perspective H anne, T. / INTELLIGENT STRATEGIES FOR MET A MULTIPLE CRITERIA DECISION MAKING S aaty, T. & Vargas, L. / MODELS, METHODS, CONCEPTS and APPLICATIONS OF THE ANALYTIC HIERARCHY PROCESS C hatterjee, K. & Samuelson, W. / GAME THEORY AND BUSINESS APPLICATIONS H obbs, B. et al. / THE NEXT GENERATION OF ELECTRIC POWER UNIT COMMITMENT MODELS V anderbei, R.J. / LINEAR PROGRAMMING: Foundations and Extensions, 2nd Ed. K imms, A. / MATHEMATICAL PROGRAMMING AND FINANCIAL OBJECTIVES FOR SCHEDULING PROJECTS B aptiste, P., Le Pape, C. & Nuijten, W. / CONSTRAINT-BASED SCHEDULING F einberg, E. & Shwartz, A. / HANDBOOK OF MARKOV DECISION PROCESSES: Methods and Applications R amik, J. & Vlach, M. / GENERALIZED CONCAVITY IN FUZZY OPTIMIZATION AND DECISION ANALYSIS S ong, J. & Yao, D. / SUPPLY CHAIN STRUCTURES: Coordination, Information and Optimization K ozan, E. & Ohuchi, A. / OPERATIONS RESEARCH/MANAGEMENT SCIENCE AT WORK B ouyssou et al. / AIDING DECISIONS WITH MULTIPLE CRITERIA: Essays in Honor of Bernard Roy Early Titles in the INTERNATIONAL SERIES IN OPERATIONS RESEARCH & MANAGEMENT SCIENCE (Continued) C ox, Louis Anthony, Jr. / RISK ANALYSIS: Foundations, Models and Methods D ror, M., L 'Ecuyer, P . & Szidarovszky, F. / MODELING UNCERTAINTY: An Examination of Stochastic Theory, Methods, and Applications D okuchaev, N. / DYNAMIC PORTFOLIO STRATEGIES: Quantitative Methods and Empirical Rules for Incomplete Information S arker, R., Mohammadian, M. & Yao, X. / EVOLUTIONARY OPTIMIZATION D emeulemeester, R. & Herroelen, W, / PROJECT SCHEDULING: A Research Handbook G azis, D .C. / TRAFFIC THEORY Z hu/ QUANTITATIVE MODELS FOR PERFORMANCE EVALUATION AND BENCHMARKING E hrgott & Gandibleux/M(/L77M,E CRITERIA OPTIMIZATION: State of the Art Annotated Bibliographical Surveys B ienstock/ Potential Function Methods for Approx. Solving Linear Programming Problems M atsatsinis & Siskos/ INTELLIGENT SUPPORT SYSTEMS FOR MARKETING DECISIONS A lpern & Gal/ THE THEORY OF SEARCH GAMES AND RENDEZVOUS Hntt/HANDBOOK OF TRANSPORTATION SCIENCE - 2"J Ed. G lover & Kochenberger/ZMiVftBOOA" OF METAHEURISTICS G raves & Ringuest/ MODELS AND METHODS FOR PROJECT SELECTION: Concepts from Management Science, Finance and Information Technology H assin & Haviv/ TO QUEUE OR NOT TO QUEUE: Equilibrium Behavior in Queueing Systems G ershwin et a l/ ANALYSIS & MODELING OF MANUFACTURING SYSTEMS M aros/ COMPUTATIONAL TECHNIQUES OF THE SIMPLEX METHOD H arrison, Lee & Neale/ THE PRACTICE OF SUPPLY CHAIN MANAGEMENT: Where Theory and Application Converge S hanthikumar, Yao & ZijnV STOCHASTIC MODELING AND OPTIMIZATION OF MANUFACTURING SYSTEMS AND SUPPLY CHAINS N abrzyski, Schopf & W§glarz/ GRID RESOURCE MANAGEMENT: State of the Art and Future Trends T hissen & Herder/ CRITICAL INFRASTRUCTURES: State of the Art in Research and Application C arlsson, Fedrizzi, & Fuller/ FUZZY LOGIC IN MANAGEMENT S oyer, Mazzuchi & Singpurwalla/ MATHEMATICAL RELIABILITY: An Expository Perspective C hakravarty & Eliashberg/ MANAGING BUSINESS INTERFACES: Marketing, Engineering, and Manufacturing Perspectives * A list of the more recent publications in the series is at the front of the book * ... View Full Document