This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 5.3 LEAST SQUARES PROBLEMS KIAM HEONG KWA pp. 223224 In practice, the coefficient matrix A and the righthand side b of an m n system A x = b are obtained emprically. In consequence, the system A x = b is usually inconsistent. In this case, we look for a vector b x R n for which A b x is closest to b in the following sense. (Clearly, if x is a solution of A x = b , then it satisfies the desired property.) Define the residual r : R n R m of the system A x = b by setting r ( x ) = b A x and consider the norm of the residual k r k : R n R given by k r ( x ) k = k b A x k for each x R n , where k k is the Euclidean norm. Then for a vector b x R n , A b x is closest to b if and only if k r ( b x ) k is a (global) minimum, i.e., k r ( b x ) k k r ( x ) k for all x R n . Such a vector b x that minimizes k r k is called a least squares solution of the system A x = b . Note that minimizing k r k is equivalent to minimizing k r k 2 . Let b x be a least squares solution of A x = b . Then p = A b x R ( A ) , the column space of A . By the definition of a least squares solution, p is an element of R ( A ) that is closest to b . Since R ( A ) R m is a subspace, such a closest vector p always exists and is unique. Moreover, it is characterized by the inclusion b p R ( A ) = N ( A T ) 1 . In other words, b x is a least squares solution of A x = b if and only if r ( b x ) = b A b x R ( A ) = N ( A T ) . This is a consequence of the following general statement. Let S R m be a subspace and let b R m but otherwise arbi trary. There is a unique element p S , called the projection of b Therem 5.3.1 Date : July 15, 2011. 1 The fundamental subspaces theorem from the last section says that N ( A ) = R ( A T ) and N ( A T ) = R ( A ) for any matrix A . 1 2 KIAM HEONG KWA onto S , such that k b p k < k b y k for any y S , y 6 = p . In addition, such an element p of S is char acterized by the condition b p S . Proof. To begin with, recall that R m = S S , so that b has the unique representation b = p + q , where p S and q S . Hence if y S , so that p y S , we have k b y k 2 = k ( b p ) + ( p y ) k 2 = k b p k 2 + k p y k 2 because b p = q p y . Thus we see that k b y k k b p k and the equality holds if and only if k p y k = 0 or p = y . This shows the existence and the uniqueness of such an element p of S in the statement of the theorem. In fact, the above calculation also shows that if p S is such that b p S , then k b y k k b p k for any y S . Conversely, let z S be such that b z 6 S . Then, in particular, z 6 = p and, as in the above calculation, we have k b z k 2 = k ( b p ) + ( p z ) k 2 = k b p k 2 + k p z k 2 , so that k b z k > k b p k ....
View
Full
Document
This note was uploaded on 11/11/2011 for the course MATH 571 taught by Professor Kim during the Summer '08 term at Ohio State.
 Summer '08
 KIM
 Linear Algebra, Algebra, Least Squares

Click to edit the document details