La11-arips - LINEAR ALGEBRA W W L CHEN c W W L Chen 1997 2005 This chapter is available free to all individuals on the understanding that it is not

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2005. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied, with or without permission from the author. However, this document may not be kept on any information storage and retrieval system without permission from the author, unless such system is not accessible to any individuals other than its owners. Chapter 11 APPLICATIONS OF REAL INNER PRODUCT SPACES 11.1. Least Squares Approximation Given a continuous function f : [a, b] → R, we wish to approximate f by a polynomial g : [a, b] → R of degree at most k , such that the error b |f (x) − g (x)|2 dx a is minimized. The purpose of this section is to study this problem using the theory of real inner product spaces. Our argument is underpinned by the following simple result in the theory. PROPOSITION 11A. Suppose that V is a real inner product space, and that W is a finite-dimensional subspace of V . Given any u ∈ V , the inequality u − projW u ≤ u − w holds for every w ∈ W . In other words, the distance from u to any w ∈ W is minimized by the choice w = projW u, the orthogonal projection of u on the subspace W . Alternatively, projW u can be thought of as the vector in W closest to u. Proof of Proposition 11A. Note that u − projW u ∈ W ⊥ Chapter 11 : Applications of Real Inner Product Spaces and projW u − w ∈ W. page 1 of 11 c Linear Algebra W W L Chen, 1997, 2005 It follows from Pythagoras’s theorem that u−w 2 = (u − projW u) + (projW u − w) 2 = u − projW u 2 + projW u − w 2 , so that u−w 2 − u − projW u 2 = projW u − w ≥ 0. 2 The result follows immediately. Let V denote the vector space C [a, b] of all continuous real valued functions on the closed interval [a, b], with inner product b f, g = f (x)g (x) dx. a Then b |f (x) − g (x)|2 dx = f − g, f − g = f − g 2 . a It follows that the least squares approximation problem is reduced to one of finding a suitable polynomial g to minimize the norm f − g . Now let W = Pk [a, b] be the collection of all polynomials g : [a, b] → R with real coefficients and of degree at most k . Note that W is essentially Pk , although the variable is restricted to the closed interval [a, b]. It is easy to show that W is a subspace of V . In view of Proposition 11A, we conclude that g = projW f gives the best least squares approximation among polynomials in W = Pk [a, b]. This subspace is of dimension k + 1. Suppose that {v0 , v1 , . . . , vk } is an orthogonal basis of W = Pk [a, b]. Then by Proposition 9L, we have g= f , v0 f , v1 f , vk v0 + v1 + . . . + vk . v0 2 v1 2 vk 2 Example 11.1.1. Consider the function f (x) = x2 in the interval [0, 2]. Suppose that we wish to find a least squares approximation by a polynomial of degree at most 1. In this case, we can take V = C [0, 2], with inner product 2 f, g = f (x)g (x) dx, 0 and W = P1 [0, 2], with basis {1, x}. We now apply the Gram-Schmidt orthogonalization process to this basis to obtain an orthogonal basis {1, x − 1} of W , and take g= x2 , 1 x2 , x − 1 1+ (x − 1). 2 1 x−1 2 Now 2 x2 , 1 = x2 dx = 0 8 3 Chapter 11 : Applications of Real Inner Product Spaces 2 and 1 2 = 1, 1 = dx = 2, 0 page 2 of 11 c Linear Algebra W W L Chen, 1997, 2005 while 2 x2 , x − 1 = x2 (x − 1) dx = 0 2 4 3 x−1 and 2 = x − 1, x − 1 = (x − 1)2 dx = 0 2 . 3 It follows that g= 4 2 + 2(x − 1) = 2x − . 3 3 Example 11.1.2. Consider the function f (x) = ex in the interval [0, 1]. Suppose that we wish to find a least squares approximation by a polynomial of degree at most 1. In this case, we can take V = C [0, 1], with inner product 1 f, g = f (x)g (x) dx, 0 and W = P1 [0, 1], with basis {1, x}. We now apply the Gram-Schmidt orthogonalization process to this basis to obtain an orthogonal basis {1, x − 1/2} of W , and take g= ex , 1 ex , x − 1/2 1+ 12 x − 1/2 2 x− 1 2 . Now 1 1 ex dx = e − 1 ex , 1 = ex , x = and 0 ex x dx = 1, 0 so that ex , x − 1 2 = ex , x − 1x 3e e ,1 = − . 2 22 Also 1 1 2 = 1, 1 = dx = 1 and x− 0 1 2 2 = 1 1 x − ,x − 2 2 1 x− = 0 1 2 2 dx = 1 . 12 It follows that g = (e − 1) + (18 − 6e) x − 1 2 = (18 − 6e)x + (4e − 10). Remark. From the proof of Proposition 11A, it is clear that u − w is minimized by the unique choice w = projW u. It follows that the least squares approximation problem posed here has a unique solution. 11.2. Quadratic Forms A real quadratic form in n variables x1 , . . . , xn is an expression of the form n n (1) cij xi xj , i=1 j =1 i≤j where cij ∈ R for every i, j = 1, . . . , n satisfying i ≤ j . Chapter 11 : Applications of Real Inner Product Spaces page 3 of 11 c Linear Algebra W W L Chen, 1997, 2005 Example 11.2.1. The expression 5x2 + 6x1 x2 + 7x2 is a quadratic form in two variables x1 and x2 . It 1 2 can be written in the form 5x2 + 6x1 x2 + 7x2 = ( x1 1 2 x2 ) 5 3 3 7 x1 x2 . Example 11.2.2. The expression 4x2 + 5x2 + 3x2 + 2x1 x2 + 4x1 x3 + 6x2 x3 1 2 3 variables x1 , x2 and x3 . It can be written in the form 4 4x2 + 5x2 + 3x2 + 2x1 x2 + 4x1 x3 + 6x2 x3 = ( x1 x2 x3 ) 1 1 2 3 2 is a quadratic form in three 1 5 3 2 x1 3 x2 . 3 x3 Note that in both examples, the quadratic form can be described in terms of a real symmetric matrix. In fact, this is always possible. To see this, note that given any quadratic form (1), we can write, for every i, j = 1, . . . , n, c if i = j , ij (2) aij = 1 cij if i < j , 2 1 2 cji if i > j . Then n n n cij xi xj = i=1 j =1 i≤j n aij xi xj = ( x1 ... i=1 j =1 a11 . xn ) . . an1 ... a1n x1 . . . . . . . . . . ann xn The matrix a11 . . A= . an1 ... a1n . . . . . . ann is clearly symmetric, in view of (2). We are interested in the case when x1 , . . . , xn take real values. In this case, we can write x1 . x = . . . xn It follows that a quadratic form can be written as xt Ax, where A is an n × n real symmetric matrix and x takes values in Rn . Many problems in mathematics can be studied using quadratic forms. Here we shall restrict our attention to two fundamental problems which are in fact related. The first is the question of what conditions the matrix A must satisfy in order that the inequality xt Ax > 0 holds for every non-zero x ∈ Rn . The second is the question of whether it is possible to have a change of variables of the type x = P y, where P is an invertible matrix, such that the quadratic form xt Ax can be represented in the alternative form yt Dy, where D is a diagonal matrix with real entries. Chapter 11 : Applications of Real Inner Product Spaces page 4 of 11 c Linear Algebra W W L Chen, 1997, 2005 Definition. A quadratic form xt Ax is said to be positive definite if xt Ax > 0 for every non-zero x ∈ Rn . In this case, we say that the symmetric matrix A is a positive definite matrix. To answer our first question, we shall prove the following result. PROPOSITION 11B. A quadratic form xt Ax is positive definite if and only if all the eigenvalues of the symmetric matrix A are positive. Our strategy here is to prove Proposition 11B by first studying our second question. Since the matrix A is real and symmetric, it follows from Proposition 10E that it is orthogonally diagonalizable. In other words, there exists an orthogonal matrix P and a diagonal matrix D such that P t AP = D, and so A = P DP t . It follows that xt Ax = xt P DP t x, and so, writing y = P t x, we have xt Ax = yt Dy. Also, since P is an orthogonal matrix, we also have x = P y. This answers our second question. Furthermore, in view of the Orthogonal diagonalization process, the diagonal entries in the matrix D can be taken to be the eigenvalues of A, so that λ1 .. , D= . λn where λ1 , . . . , λn ∈ R are the eigenvalues of A. Writing y1 . y = . , . yn we have (3) 2 2 xt Ax = yt Dy = λ1 y1 + . . . + λn yn . Note now that x = 0 if and only if y = 0, since P is an invertible matrix. Proposition 11B now follows immediately from (3). Example 11.2.3. Consider the quadratic written in the form xt Ax, where 22 A = 2 5 12 form 2x2 + 5x2 + 2x2 + 4x1 x2 + 2x1 x3 + 4x2 x3 . This can be 1 2 3 1 2 2 x1 x = x2 . x3 and The matrix A has eigenvalues λ1 = 7 and (double root) λ2 = λ3 = 1; see Example 10.3.1. Furthermore, we have P t AP = D, where √ √ √ 1/ √ 3 1/√6 1/ 2 700 P = 2/√6 0√ −1/ 3 and D = 0 1 0. √ 1/ 6 −1/ 2 1/ 3 001 2 2 2 Writing y = P t x, the quadratic form becomes 7y1 + y2 + y3 which is clearly positive definite. Chapter 11 : Applications of Real Inner Product Spaces page 5 of 11 c Linear Algebra W W L Chen, 1997, 2005 Example 11.2.4. Consider the quadratic form 5x2 + 6x2 + 7x2 − 4x1 x2 + 4x2 x3 . This can be written 1 2 3 in the form xt Ax, where 5 −2 0 x1 A = −2 6 2 and x = x2 . 0 27 x3 The matrix A has eigenvalues λ1 = 3, λ2 P t AP = D, where 2/3 2/3 P = 2/3 −1/3 −1/3 2/3 = 6 and λ3 = 9; see Example 10.3.3. Furthermore, we have −1/3 2/3 2/3 3 D = 0 0 and 0 0. 9 0 6 0 2 2 2 Writing y = P t x, the quadratic form becomes 3y1 + 6y2 + 9y3 which is clearly positive definite. Example 11.2.5. Consider the quadratic form x2 + x2 + 2x1 x2 . Clearly this is equal to (x1 + x2 )2 and 1 2 is therefore not positive definite. The quadratic form can be written in the form xt Ax, where A= 1 1 1 1 and x1 x2 x= . It follows from Proposition 11B that the eigenvalues of A are not all positive. Indeed, the matrix A has eigenvalues λ1 = 2 and λ2 = 0, with corresponding eigenvectors 1 1 1 −1 and . Hence we may take P= √ 1/√2 1/ 2 √ 1/ √ 2 −1/ 2 and D= 2 0 0 0 . 2 Writing y = P t x, the quadratic form becomes 2y1 which is not positive definite. 11.3. Real Fourier Series Let E denote the collection of all functions f : [−π, π ] → R which are piecewise continuous on the interval [−π, π ]. This means that any f ∈ E has at most a finite number of points of discontinuity, at each of which f need not be defined but must have one sided limits which are finite. We further adopt the convention that any two functions f, g ∈ E are considered equal, denoted by f = g , if f (x) = g (x) for every x ∈ [−π, π ] with at most a finite number of exceptions. It is easy to check that E forms a real vector space. More precisely, let λ ∈ E denote the function λ : [−π, π ] → R, where λ(x) = 0 for every x ∈ [−π, π ]. Then the following conditions hold: • • • • • • • • • • For For For For For For For For For For every every every every every every every every every every f, g ∈ E , we have f + g ∈ E . f, g, h ∈ E , we have f + (g + h) = (f + g ) + h. f ∈ E , we have f + λ = λ + f = f . f ∈ E , we have f + (−f ) = λ. f, g ∈ E , we have f + g = g + f . c ∈ R and f ∈ E , we have cf ∈ E . c ∈ R and f, g ∈ E , we have c(f + g ) = cf + cg . a, b ∈ R and f ∈ E , we have (a + b)f = af + bf . a, b ∈ R and f ∈ E , we have (ab)f = a(bf ). f ∈ E , we have 1f = f . Chapter 11 : Applications of Real Inner Product Spaces page 6 of 11 c Linear Algebra W W L Chen, 1997, 2005 We now give this vector space E more structure by introducing an inner product. For every f, g ∈ E , write f, g = 1 π π f (x)g (x) dx. −π The integral exists since the function f (x)g (x) is clearly piecewise continuous on [−π, π ]. It is easy to check that the following conditions hold: • • • • For For For For every every every every f, g ∈ E , we have f , g = g , f . f, g, h ∈ E , we have f , g + h = f , g + f , h . f, g ∈ E and c ∈ R, we have c f , g = cf, g . f ∈ E , we have f , f ≥ 0, and f , f = 0 if and only if f = λ. Hence E is a real inner product space. The difficulty here is that the inner product space E is not finite-dimensional. It is not straightforward to show that the set 1 √ , sin x, cos x, sin 2x, cos 2x, sin 3x, cos 3x, . . . 2 (4) in E forms an orthonormal “basis” for E . The difficulty is to show that the set spans E . Remark. It is easy to check that the elements in (4) form an orthonormal “system”. For every k, m ∈ N, we have 1 1 √ ,√ 2 2 1 √ , sin kx 2 1 √ , cos kx 2 π 1 π 1 = π 1 = π = −π π −π π −π 1 dx = 1; 2 1 √ sin kx = 0; 2 1 √ cos kx = 0; 2 as well as 1 π 1 cos kx, cos mx = π sin kx, sin mx = π 1 π −π π 1 cos kx cos mx dx = π −π sin kx sin mx dx = π 1 1 if k = m, (cos(k − m)x − cos(k + m)x) dx = 0 if k = m; −π 2 π 1 1 if k = m, (cos(k − m)x + cos(k + m)x) dx = 0 if k = m; −π 2 and sin kx, cos mx = 1 π π sin kx cos mx dx = −π 1 π π −π 1 (sin(k − m)x + sin(k + m)x) dx = 0. 2 Let us assume that we have established that the set (4) forms an orthonormal basis for E . Then a natural extension of Proposition 9H gives rise to the following: Every function f ∈ E can be written uniquely in the form ∞ (5) a0 (an cos nx + bn sin nx), + 2 n=1 known usually as the (trigonometric) Fourier series of the function f , with Fourier coefficients a √0 = 2 1 f, √ 2 Chapter 11 : Applications of Real Inner Product Spaces = 1 π π f (x) dx, −π page 7 of 11 c Linear Algebra W W L Chen, 1997, 2005 and, for every n ∈ N, an = f , cos nx = 1 π π f (x) cos nx dx and bn = f , sin nx = −π π 1 π f (x) sin nx dx. −π Note that the constant term in the Fourier series (5) is given by 1 f, √ 2 1 a √ = 0. 2 2 Example 11.3.1. Consider the function f : [−π, π ] → R, given by f (x) = x for every x ∈ [−π, π ]. For every n ∈ N ∪ {0}, we have an = 1 π π x cos nx dx = 0, −π since the integrand is an odd function. On the other hand, for every n ∈ N, we have bn = 1 π π x sin nx dx = −π π 2 π x sin nx dx, 0 since the integrand is an even function. On integrating by parts, we have bn = 2 π − x cos nx n π π + 0 0 cos nx dx n = 2 π x cos nx n − π + 0 sin nx n2 π = 0 2(−1)n+1 . n We therefore have the (trigonometric) Fourier series ∞ 2(−1)n+1 sin nx. n n=1 Note that the function f is odd, and this plays a crucial role in eschewing the Fourier coefficients an corresponding to the even part of the Fourier series. Example 11.3.2. Consider the function f : [−π, π ] → R, given by f (x) = |x| for every x ∈ [−π, π ]. For every n ∈ N ∪ {0}, we have an = 1 π π −π |x| cos nx dx = π 2 π x cos nx dx, 0 since the integrand is an even function. Clearly a0 = 2 π π x dx = π. 0 Furthermore, for every n ∈ N, on integrating by parts, we have 2 an = π x sin nx n π π − 0 0 sin nx dx n 2 = π x sin nx n π 0 cos nx + n2 0 π = 0 if n is even, − 4 πn2 if n is odd. On the other hand, for every n ∈ N, we have bn = 1 π Chapter 11 : Applications of Real Inner Product Spaces π −π |x| sin nx dx = 0, page 8 of 11 c Linear Algebra W W L Chen, 1997, 2005 since the integrand is an odd function. We therefore have the (trigonometric) Fourier series π − 2 ∞ n=1 n odd 4 π cos nx = − 2 πn 2 ∞ k=1 4 cos(2k − 1)x. π (2k − 1)2 Note that the function f is even, and this plays a crucial role in eschewing the Fourier coefficients bn corresponding to the odd part of the Fourier series. Example 11.3.3. Consider the function f : [−π, π ] → R, given for every x ∈ [−π, π ] by +1 if 0 < x ≤ π , 0 if x = 0, −1 if −π ≤ x < 0. f (x) = sgn(x) = For every n ∈ N ∪ {0}, we have 1 π an = π sgn(x) cos nx dx = 0, −π since the integrand is an odd function. On the other hand, for every n ∈ N, we have bn = 1 π π sgn(x) sin nx dx = −π π 2 π sin nx dx, 0 since the integrand is an even function. It is easy to see that if n is even, 0 π 2 cos nx bn = − = 4 if n is odd. π n 0 πn We therefore have the (trigonometric) Fourier series ∞ n=1 n odd 4 sin nx = πn ∞ k=1 4 sin(2k − 1)x. π (2k − 1) Example 11.3.4. Consider the function f : [−π, π ] → R, given by f (x) = x2 for every x ∈ [−π, π ]. For every n ∈ N ∪ {0}, we have an = 1 π π x2 cos nx dx = −π π 2 π x2 cos nx dx, 0 since the integrand is an even function. Clearly a0 = 2 π π x2 dx = 0 2π 2 . 3 Furthermore, for every n ∈ N, on integrating by parts, we have 2 π x2 sin nx n 2 = π x2 sin nx n an = π π − 0 π 0 0 2x sin nx dx n 2x cos nx + n2 π 0 = 2 π 2 sin nx − n3 Chapter 11 : Applications of Real Inner Product Spaces x2 sin nx n π 0 π + 0 2x cos nx n2 π π − 0 0 2 cos nx dx n2 4(−1)n . = n2 page 9 of 11 c Linear Algebra W W L Chen, 1997, 2005 On the other hand, for every n ∈ N, we have bn = π 1 π x2 sin nx dx = 0, −π since the integrand is an odd function. We therefore have the (trigonometric) Fourier series ∞ π2 4(−1)n cos nx. + 3 n2 n=1 Problems for Chapter 11 1. Consider the function f : [−1, 1] → R : x → x3 . We wish to find a polynomial g (x) = ax + b which minimizes the error 1 −1 |f (x) − g (x)|2 dx. Follow the steps below to find this polynomial g : a) Consider the real vector space C [−1, 1]. Write down a suitable real inner product on C [−1, 1] for this problem, explaining carefully the steps that you take. b) Consider now the subspace P1 [−1, 1] of all polynomials of degree at most 1. Describe the polynomial g in terms of f and orthogonal projection with respect to the inner product in part (a). Give a brief explanation for your choice. c) Write down a basis of P1 [−1, 1]. d) Apply the Gram-Schmidt process to your basis in part (c) to obtain an orthogonal basis of P1 [−1, 1]. e) Describe your polynomial in part (b) as a linear combination of the elements of your basis in part (d), and find the precise values of the coefficients. 2. For each of the following functions, find the best least squares approximation by linear polynomials of the form ax + b, where a, b ∈ R: a) f : [0, π/2] → R : x → sin x b) f : [0, 1] → R : x → x3 x c) f : [0, 2] → R : x → e 3. Consider the quadratic form 2x2 + x2 + x2 + 2x1 x2 + 2x1 x3 in three variables x1 , x2 , x3 . 1 2 3 a) Write the quadratic form in the form xt Ax, where x1 x = x2 x3 and where A is a symmetric matrix with real entries. b) Apply the Orthogonal diagonalization process to the matrix A. c) Find a transformation of the type x = P y, where P is an invertible matrix, so that the quadratic form can be written as yt Dy, where y1 y = y2 y3 and where D is a diagonal matrix with real entries. You should give the matrices P and D explicitly. d) Is the quadratic form positive definite? Justify your assertion both in terms of the eigenvalues of A and in terms of your solution to part (c). Chapter 11 : Applications of Real Inner Product Spaces page 10 of 11 c Linear Algebra W W L Chen, 1997, 2005 4. For each of the following quadratic forms in three variables, write it in the form xt Ax, find a substitution x = P y so that it can be written as a diagonal form in the variables y1 , y2 , y3 , and determine whether the quadratic form is positive definite: a) x2 + x2 + 2x2 − 2x1 x2 + 4x1 x3 + 4x2 x3 b) 3x2 + 2x2 + 3x2 + 2x1 x3 1 2 3 1 2 3 2 2 2 c) 3x1 + 5x2 + 4x3 + 4x1 x3 − 4x2 x3 d) 5x2 + 2x2 + 5x2 + 4x1 x2 − 8x1 x3 − 4x2 x3 1 2 3 e) x2 − 5x2 − x2 + 4x1 x2 + 6x2 x3 1 2 3 5. Determine which of the following matrices are positive definite: 011 311 a) 1 0 1 b) 1 1 2 110 121 6 −2 −1 3 −2 4 d) −2 6 −1 e) −2 6 2 −1 −1 5 4 23 617 c) 1 1 2 729 2000 0 1 0 1 f) 0020 0101 6. Find the trigonometric Fourier series for each of the following functions f : [−π, π ] → C: a) f (x) = x|x| for every x ∈ [−π, π ] b) f (x) = | sin x| for every x ∈ [−π, π ] c) f (x) = | cos x| for every x ∈ [−π, π ] d) f (x) = 0 for every x ∈ [−π, 0] and f (x) = x for every x ∈ (0, π ] e) f (x) = sin x for every x ∈ [−π, 0] and f (x) = cos x for every x ∈ (0, π ] f) f (x) = cos x for every x ∈ [−π, 0] and f (x) = sin x for every x ∈ (0, π ] g) f (x) = cos(x/2) for every x ∈ [−π, π ] h) f (x) = sin(x/2) for every x ∈ [−π, π ] Chapter 11 : Applications of Real Inner Product Spaces page 11 of 11 ...
View Full Document

This note was uploaded on 06/13/2009 for the course TAM 455 taught by Professor Petrina during the Fall '08 term at Cornell University (Engineering School).

Ask a homework question - tutors are online