This preview shows page 1. Sign up to view the full content.
Unformatted text preview: LINEAR ALGEBRA
W W L CHEN
c W W L Chen, 1997, 2005. This chapter is available free to all individuals, on the understanding that it is not to be used for ﬁnancial gain,
and may be downloaded and/or photocopied, with or without permission from the author.
However, this document may not be kept on any information storage and retrieval system without permission
from the author, unless such system is not accessible to any individuals other than its owners. Chapter 11
APPLICATIONS OF
REAL INNER PRODUCT SPACES 11.1. Least Squares Approximation
Given a continuous function f : [a, b] → R, we wish to approximate f by a polynomial g : [a, b] → R of
degree at most k , such that the error
b f (x) − g (x)2 dx
a is minimized. The purpose of this section is to study this problem using the theory of real inner product
spaces. Our argument is underpinned by the following simple result in the theory.
PROPOSITION 11A. Suppose that V is a real inner product space, and that W is a ﬁnitedimensional
subspace of V . Given any u ∈ V , the inequality
u − projW u ≤ u − w
holds for every w ∈ W .
In other words, the distance from u to any w ∈ W is minimized by the choice w = projW u, the
orthogonal projection of u on the subspace W . Alternatively, projW u can be thought of as the vector
in W closest to u.
Proof of Proposition 11A. Note that
u − projW u ∈ W ⊥
Chapter 11 : Applications of Real Inner Product Spaces and projW u − w ∈ W.
page 1 of 11 c Linear Algebra W W L Chen, 1997, 2005 It follows from Pythagoras’s theorem that
u−w 2 = (u − projW u) + (projW u − w) 2 = u − projW u 2 + projW u − w 2 , so that
u−w 2 − u − projW u 2 = projW u − w ≥ 0. 2 The result follows immediately.
Let V denote the vector space C [a, b] of all continuous real valued functions on the closed interval
[a, b], with inner product
b f, g = f (x)g (x) dx.
a Then
b f (x) − g (x)2 dx = f − g, f − g = f − g 2 .
a It follows that the least squares approximation problem is reduced to one of ﬁnding a suitable polynomial
g to minimize the norm f − g .
Now let W = Pk [a, b] be the collection of all polynomials g : [a, b] → R with real coeﬃcients and of
degree at most k . Note that W is essentially Pk , although the variable is restricted to the closed interval
[a, b]. It is easy to show that W is a subspace of V . In view of Proposition 11A, we conclude that
g = projW f
gives the best least squares approximation among polynomials in W = Pk [a, b]. This subspace is of dimension k + 1. Suppose that {v0 , v1 , . . . , vk } is an orthogonal basis of W = Pk [a, b]. Then by Proposition
9L, we have
g= f , v0
f , v1
f , vk
v0 +
v1 + . . . +
vk .
v0 2
v1 2
vk 2 Example 11.1.1. Consider the function f (x) = x2 in the interval [0, 2]. Suppose that we wish to ﬁnd a
least squares approximation by a polynomial of degree at most 1. In this case, we can take V = C [0, 2],
with inner product
2 f, g = f (x)g (x) dx,
0 and W = P1 [0, 2], with basis {1, x}. We now apply the GramSchmidt orthogonalization process to this
basis to obtain an orthogonal basis {1, x − 1} of W , and take
g= x2 , 1
x2 , x − 1
1+
(x − 1).
2
1
x−1 2 Now
2 x2 , 1 = x2 dx =
0 8
3 Chapter 11 : Applications of Real Inner Product Spaces 2 and 1 2 = 1, 1 = dx = 2,
0
page 2 of 11 c Linear Algebra W W L Chen, 1997, 2005 while
2 x2 , x − 1 = x2 (x − 1) dx =
0 2 4
3 x−1 and 2 = x − 1, x − 1 = (x − 1)2 dx =
0 2
.
3 It follows that
g= 4
2
+ 2(x − 1) = 2x − .
3
3 Example 11.1.2. Consider the function f (x) = ex in the interval [0, 1]. Suppose that we wish to ﬁnd a
least squares approximation by a polynomial of degree at most 1. In this case, we can take V = C [0, 1],
with inner product
1 f, g = f (x)g (x) dx,
0 and W = P1 [0, 1], with basis {1, x}. We now apply the GramSchmidt orthogonalization process to this
basis to obtain an orthogonal basis {1, x − 1/2} of W , and take
g= ex , 1
ex , x − 1/2
1+
12
x − 1/2 2 x− 1
2 . Now
1 1 ex dx = e − 1 ex , 1 = ex , x = and 0 ex x dx = 1,
0 so that
ex , x − 1
2 = ex , x − 1x
3e
e ,1 = − .
2
22 Also
1 1 2 = 1, 1 = dx = 1 and x− 0 1
2 2 = 1
1
x − ,x −
2
2 1 x− =
0 1
2 2 dx = 1
.
12 It follows that
g = (e − 1) + (18 − 6e) x − 1
2 = (18 − 6e)x + (4e − 10). Remark. From the proof of Proposition 11A, it is clear that u − w is minimized by the unique choice
w = projW u. It follows that the least squares approximation problem posed here has a unique solution. 11.2. Quadratic Forms
A real quadratic form in n variables x1 , . . . , xn is an expression of the form
n n (1) cij xi xj ,
i=1 j =1
i≤j where cij ∈ R for every i, j = 1, . . . , n satisfying i ≤ j .
Chapter 11 : Applications of Real Inner Product Spaces page 3 of 11 c Linear Algebra W W L Chen, 1997, 2005 Example 11.2.1. The expression 5x2 + 6x1 x2 + 7x2 is a quadratic form in two variables x1 and x2 . It
1
2
can be written in the form
5x2 + 6x1 x2 + 7x2 = ( x1
1
2 x2 ) 5
3 3
7 x1
x2 . Example 11.2.2. The expression 4x2 + 5x2 + 3x2 + 2x1 x2 + 4x1 x3 + 6x2 x3
1
2
3
variables x1 , x2 and x3 . It can be written in the form 4
4x2 + 5x2 + 3x2 + 2x1 x2 + 4x1 x3 + 6x2 x3 = ( x1 x2 x3 ) 1
1
2
3
2 is a quadratic form in three
1
5
3 2
x1
3 x2 .
3
x3 Note that in both examples, the quadratic form can be described in terms of a real symmetric matrix.
In fact, this is always possible. To see this, note that given any quadratic form (1), we can write, for
every i, j = 1, . . . , n,
c
if i = j , ij (2)
aij = 1 cij if i < j ,
2 1
2 cji if i > j .
Then
n n n cij xi xj =
i=1 j =1
i≤j n aij xi xj = ( x1 ... i=1 j =1 a11
.
xn ) .
.
an1 ... a1n
x1
. . .
.
.
.
. . . . ann xn The matrix a11
.
.
A=
.
an1 ... a1n
.
.
. . . . ann is clearly symmetric, in view of (2).
We are interested in the case when x1 , . . . , xn take real values. In this case, we can write x1
.
x = . .
.
xn
It follows that a quadratic form can be written as
xt Ax,
where A is an n × n real symmetric matrix and x takes values in Rn .
Many problems in mathematics can be studied using quadratic forms. Here we shall restrict our
attention to two fundamental problems which are in fact related. The ﬁrst is the question of what
conditions the matrix A must satisfy in order that the inequality
xt Ax > 0
holds for every nonzero x ∈ Rn . The second is the question of whether it is possible to have a change of
variables of the type x = P y, where P is an invertible matrix, such that the quadratic form xt Ax can
be represented in the alternative form yt Dy, where D is a diagonal matrix with real entries.
Chapter 11 : Applications of Real Inner Product Spaces page 4 of 11 c Linear Algebra W W L Chen, 1997, 2005 Definition. A quadratic form xt Ax is said to be positive deﬁnite if xt Ax > 0 for every nonzero x ∈ Rn .
In this case, we say that the symmetric matrix A is a positive deﬁnite matrix.
To answer our ﬁrst question, we shall prove the following result.
PROPOSITION 11B. A quadratic form xt Ax is positive deﬁnite if and only if all the eigenvalues of
the symmetric matrix A are positive.
Our strategy here is to prove Proposition 11B by ﬁrst studying our second question. Since the matrix
A is real and symmetric, it follows from Proposition 10E that it is orthogonally diagonalizable. In other
words, there exists an orthogonal matrix P and a diagonal matrix D such that P t AP = D, and so
A = P DP t . It follows that
xt Ax = xt P DP t x,
and so, writing
y = P t x,
we have
xt Ax = yt Dy.
Also, since P is an orthogonal matrix, we also have x = P y. This answers our second question.
Furthermore, in view of the Orthogonal diagonalization process, the diagonal entries in the matrix D
can be taken to be the eigenvalues of A, so that λ1
..
,
D=
.
λn
where λ1 , . . . , λn ∈ R are the eigenvalues of A. Writing y1
.
y = . ,
.
yn
we have
(3) 2
2
xt Ax = yt Dy = λ1 y1 + . . . + λn yn . Note now that x = 0 if and only if y = 0, since P is an invertible matrix. Proposition 11B now follows
immediately from (3).
Example 11.2.3. Consider the quadratic
written in the form xt Ax, where 22
A = 2 5
12 form 2x2 + 5x2 + 2x2 + 4x1 x2 + 2x1 x3 + 4x2 x3 . This can be
1
2
3 1
2
2 x1
x = x2 .
x3 and The matrix A has eigenvalues λ1 = 7 and (double root) λ2 = λ3 = 1; see Example 10.3.1. Furthermore,
we have P t AP = D, where
√
√
√ 1/ √
3
1/√6 1/ 2
700
P = 2/√6
0√
−1/ 3 and
D = 0 1 0.
√
1/ 6 −1/ 2 1/ 3
001
2
2
2
Writing y = P t x, the quadratic form becomes 7y1 + y2 + y3 which is clearly positive deﬁnite.
Chapter 11 : Applications of Real Inner Product Spaces page 5 of 11 c Linear Algebra W W L Chen, 1997, 2005 Example 11.2.4. Consider the quadratic form 5x2 + 6x2 + 7x2 − 4x1 x2 + 4x2 x3 . This can be written
1
2
3
in the form xt Ax, where 5 −2 0
x1
A = −2 6 2 and
x = x2 .
0
27
x3
The matrix A has eigenvalues λ1 = 3, λ2
P t AP = D, where 2/3
2/3
P = 2/3 −1/3
−1/3 2/3 = 6 and λ3 = 9; see Example 10.3.3. Furthermore, we have −1/3
2/3 2/3 3
D = 0
0 and 0
0.
9 0
6
0 2
2
2
Writing y = P t x, the quadratic form becomes 3y1 + 6y2 + 9y3 which is clearly positive deﬁnite. Example 11.2.5. Consider the quadratic form x2 + x2 + 2x1 x2 . Clearly this is equal to (x1 + x2 )2 and
1
2
is therefore not positive deﬁnite. The quadratic form can be written in the form xt Ax, where
A= 1
1 1
1 and x1
x2 x= . It follows from Proposition 11B that the eigenvalues of A are not all positive. Indeed, the matrix A has
eigenvalues λ1 = 2 and λ2 = 0, with corresponding eigenvectors
1
1 1
−1 and . Hence we may take
P= √
1/√2
1/ 2 √
1/ √
2
−1/ 2 and D= 2
0 0
0 . 2
Writing y = P t x, the quadratic form becomes 2y1 which is not positive deﬁnite. 11.3. Real Fourier Series
Let E denote the collection of all functions f : [−π, π ] → R which are piecewise continuous on the
interval [−π, π ]. This means that any f ∈ E has at most a ﬁnite number of points of discontinuity, at
each of which f need not be deﬁned but must have one sided limits which are ﬁnite. We further adopt
the convention that any two functions f, g ∈ E are considered equal, denoted by f = g , if f (x) = g (x)
for every x ∈ [−π, π ] with at most a ﬁnite number of exceptions.
It is easy to check that E forms a real vector space. More precisely, let λ ∈ E denote the function
λ : [−π, π ] → R, where λ(x) = 0 for every x ∈ [−π, π ]. Then the following conditions hold:
•
•
•
•
•
•
•
•
•
• For
For
For
For
For
For
For
For
For
For every
every
every
every
every
every
every
every
every
every f, g ∈ E , we have f + g ∈ E .
f, g, h ∈ E , we have f + (g + h) = (f + g ) + h.
f ∈ E , we have f + λ = λ + f = f .
f ∈ E , we have f + (−f ) = λ.
f, g ∈ E , we have f + g = g + f .
c ∈ R and f ∈ E , we have cf ∈ E .
c ∈ R and f, g ∈ E , we have c(f + g ) = cf + cg .
a, b ∈ R and f ∈ E , we have (a + b)f = af + bf .
a, b ∈ R and f ∈ E , we have (ab)f = a(bf ).
f ∈ E , we have 1f = f . Chapter 11 : Applications of Real Inner Product Spaces page 6 of 11 c Linear Algebra W W L Chen, 1997, 2005 We now give this vector space E more structure by introducing an inner product. For every f, g ∈ E ,
write
f, g = 1
π π f (x)g (x) dx.
−π The integral exists since the function f (x)g (x) is clearly piecewise continuous on [−π, π ]. It is easy to
check that the following conditions hold:
•
•
•
• For
For
For
For every
every
every
every f, g ∈ E , we have f , g = g , f .
f, g, h ∈ E , we have f , g + h = f , g + f , h .
f, g ∈ E and c ∈ R, we have c f , g = cf, g .
f ∈ E , we have f , f ≥ 0, and f , f = 0 if and only if f = λ. Hence E is a real inner product space.
The diﬃculty here is that the inner product space E is not ﬁnitedimensional. It is not straightforward
to show that the set
1
√ , sin x, cos x, sin 2x, cos 2x, sin 3x, cos 3x, . . .
2 (4) in E forms an orthonormal “basis” for E . The diﬃculty is to show that the set spans E .
Remark. It is easy to check that the elements in (4) form an orthonormal “system”. For every k, m ∈ N,
we have
1
1
√ ,√
2
2
1
√ , sin kx
2
1
√ , cos kx
2 π 1
π
1
=
π
1
=
π
= −π
π
−π
π
−π 1
dx = 1;
2
1
√ sin kx = 0;
2
1
√ cos kx = 0;
2 as well as
1
π
1
cos kx, cos mx =
π
sin kx, sin mx = π 1
π
−π
π
1
cos kx cos mx dx =
π
−π
sin kx sin mx dx = π 1
1 if k = m,
(cos(k − m)x − cos(k + m)x) dx =
0 if k = m;
−π 2
π
1
1 if k = m,
(cos(k − m)x + cos(k + m)x) dx =
0 if k = m;
−π 2 and
sin kx, cos mx = 1
π π sin kx cos mx dx =
−π 1
π π
−π 1
(sin(k − m)x + sin(k + m)x) dx = 0.
2 Let us assume that we have established that the set (4) forms an orthonormal basis for E . Then a
natural extension of Proposition 9H gives rise to the following: Every function f ∈ E can be written
uniquely in the form
∞ (5) a0
(an cos nx + bn sin nx),
+
2
n=1 known usually as the (trigonometric) Fourier series of the function f , with Fourier coeﬃcients
a
√0 =
2 1
f, √
2 Chapter 11 : Applications of Real Inner Product Spaces = 1
π π f (x) dx,
−π
page 7 of 11 c Linear Algebra W W L Chen, 1997, 2005 and, for every n ∈ N,
an = f , cos nx = 1
π π f (x) cos nx dx and bn = f , sin nx = −π π 1
π f (x) sin nx dx.
−π Note that the constant term in the Fourier series (5) is given by
1
f, √
2 1
a
√ = 0.
2
2 Example 11.3.1. Consider the function f : [−π, π ] → R, given by f (x) = x for every x ∈ [−π, π ]. For
every n ∈ N ∪ {0}, we have
an = 1
π π x cos nx dx = 0,
−π since the integrand is an odd function. On the other hand, for every n ∈ N, we have
bn = 1
π π x sin nx dx =
−π π 2
π x sin nx dx,
0 since the integrand is an even function. On integrating by parts, we have
bn = 2
π − x cos nx
n π π +
0 0 cos nx
dx
n = 2
π x cos nx
n − π +
0 sin nx
n2 π =
0 2(−1)n+1
.
n We therefore have the (trigonometric) Fourier series
∞ 2(−1)n+1
sin nx.
n
n=1
Note that the function f is odd, and this plays a crucial role in eschewing the Fourier coeﬃcients an
corresponding to the even part of the Fourier series.
Example 11.3.2. Consider the function f : [−π, π ] → R, given by f (x) = x for every x ∈ [−π, π ]. For
every n ∈ N ∪ {0}, we have
an = 1
π π
−π x cos nx dx = π 2
π x cos nx dx,
0 since the integrand is an even function. Clearly
a0 = 2
π π x dx = π.
0 Furthermore, for every n ∈ N, on integrating by parts, we have
2
an =
π x sin nx
n π π −
0 0 sin nx
dx
n 2
=
π x sin nx
n π
0 cos nx
+
n2 0 π =
0 if n is even, − 4 πn2 if n is odd. On the other hand, for every n ∈ N, we have
bn = 1
π Chapter 11 : Applications of Real Inner Product Spaces π
−π x sin nx dx = 0,
page 8 of 11 c Linear Algebra W W L Chen, 1997, 2005 since the integrand is an odd function. We therefore have the (trigonometric) Fourier series
π
−
2 ∞
n=1
n odd 4
π
cos nx = −
2
πn
2 ∞
k=1 4
cos(2k − 1)x.
π (2k − 1)2 Note that the function f is even, and this plays a crucial role in eschewing the Fourier coeﬃcients bn
corresponding to the odd part of the Fourier series.
Example 11.3.3. Consider the function f : [−π, π ] → R, given for every x ∈ [−π, π ] by
+1 if 0 < x ≤ π ,
0
if x = 0,
−1 if −π ≤ x < 0. f (x) = sgn(x) =
For every n ∈ N ∪ {0}, we have
1
π an = π sgn(x) cos nx dx = 0,
−π since the integrand is an odd function. On the other hand, for every n ∈ N, we have
bn = 1
π π sgn(x) sin nx dx =
−π π 2
π sin nx dx,
0 since the integrand is an even function. It is easy to see that if n is even,
0
π 2 cos nx
bn = −
= 4 if n is odd.
π
n 0
πn
We therefore have the (trigonometric) Fourier series
∞
n=1
n odd 4
sin nx =
πn ∞
k=1 4
sin(2k − 1)x.
π (2k − 1) Example 11.3.4. Consider the function f : [−π, π ] → R, given by f (x) = x2 for every x ∈ [−π, π ]. For
every n ∈ N ∪ {0}, we have
an = 1
π π x2 cos nx dx =
−π π 2
π x2 cos nx dx,
0 since the integrand is an even function. Clearly
a0 = 2
π π x2 dx =
0 2π 2
.
3 Furthermore, for every n ∈ N, on integrating by parts, we have
2
π x2 sin nx
n 2
=
π x2 sin nx
n an = π π −
0
π
0 0 2x sin nx
dx
n 2x cos nx
+
n2 π
0 = 2
π 2 sin nx
−
n3 Chapter 11 : Applications of Real Inner Product Spaces x2 sin nx
n
π
0 π +
0 2x cos nx
n2 π π −
0 0 2 cos nx
dx
n2 4(−1)n
.
=
n2
page 9 of 11 c Linear Algebra W W L Chen, 1997, 2005 On the other hand, for every n ∈ N, we have
bn = π 1
π x2 sin nx dx = 0,
−π since the integrand is an odd function. We therefore have the (trigonometric) Fourier series
∞ π2
4(−1)n
cos nx.
+
3
n2
n=1 Problems for Chapter 11
1. Consider the function f : [−1, 1] → R : x → x3 . We wish to ﬁnd a polynomial g (x) = ax + b which
minimizes the error
1
−1 f (x) − g (x)2 dx. Follow the steps below to ﬁnd this polynomial g :
a) Consider the real vector space C [−1, 1]. Write down a suitable real inner product on C [−1, 1] for
this problem, explaining carefully the steps that you take.
b) Consider now the subspace P1 [−1, 1] of all polynomials of degree at most 1. Describe the polynomial g in terms of f and orthogonal projection with respect to the inner product in part (a).
Give a brief explanation for your choice.
c) Write down a basis of P1 [−1, 1].
d) Apply the GramSchmidt process to your basis in part (c) to obtain an orthogonal basis of
P1 [−1, 1].
e) Describe your polynomial in part (b) as a linear combination of the elements of your basis in part
(d), and ﬁnd the precise values of the coeﬃcients.
2. For each of the following functions, ﬁnd the best least squares approximation by linear polynomials
of the form ax + b, where a, b ∈ R:
a) f : [0, π/2] → R : x → sin x
b) f : [0, 1] → R : x → x3
x
c) f : [0, 2] → R : x → e
3. Consider the quadratic form 2x2 + x2 + x2 + 2x1 x2 + 2x1 x3 in three variables x1 , x2 , x3 .
1
2
3
a) Write the quadratic form in the form xt Ax, where x1
x = x2 x3
and where A is a symmetric matrix with real entries.
b) Apply the Orthogonal diagonalization process to the matrix A.
c) Find a transformation of the type x = P y, where P is an invertible matrix, so that the quadratic
form can be written as yt Dy, where y1
y = y2 y3
and where D is a diagonal matrix with real entries. You should give the matrices P and D
explicitly.
d) Is the quadratic form positive deﬁnite? Justify your assertion both in terms of the eigenvalues of
A and in terms of your solution to part (c).
Chapter 11 : Applications of Real Inner Product Spaces page 10 of 11 c Linear Algebra W W L Chen, 1997, 2005 4. For each of the following quadratic forms in three variables, write it in the form xt Ax, ﬁnd a
substitution x = P y so that it can be written as a diagonal form in the variables y1 , y2 , y3 , and
determine whether the quadratic form is positive deﬁnite:
a) x2 + x2 + 2x2 − 2x1 x2 + 4x1 x3 + 4x2 x3
b) 3x2 + 2x2 + 3x2 + 2x1 x3
1
2
3
1
2
3
2
2
2
c) 3x1 + 5x2 + 4x3 + 4x1 x3 − 4x2 x3
d) 5x2 + 2x2 + 5x2 + 4x1 x2 − 8x1 x3 − 4x2 x3
1
2
3
e) x2 − 5x2 − x2 + 4x1 x2 + 6x2 x3
1
2
3
5. Determine which of the following matrices are positive deﬁnite: 011
311
a) 1 0 1 b) 1 1 2 110
121 6 −2 −1
3 −2 4
d) −2 6 −1 e) −2 6 2 −1 −1 5
4
23 617
c) 1 1 2 729 2000
0 1 0 1
f) 0020
0101 6. Find the trigonometric Fourier series for each of the following functions f : [−π, π ] → C:
a) f (x) = xx for every x ∈ [−π, π ]
b) f (x) =  sin x for every x ∈ [−π, π ]
c) f (x) =  cos x for every x ∈ [−π, π ]
d) f (x) = 0 for every x ∈ [−π, 0] and f (x) = x for every x ∈ (0, π ]
e) f (x) = sin x for every x ∈ [−π, 0] and f (x) = cos x for every x ∈ (0, π ]
f) f (x) = cos x for every x ∈ [−π, 0] and f (x) = sin x for every x ∈ (0, π ]
g) f (x) = cos(x/2) for every x ∈ [−π, π ]
h) f (x) = sin(x/2) for every x ∈ [−π, π ] Chapter 11 : Applications of Real Inner Product Spaces page 11 of 11 ...
View
Full
Document
This note was uploaded on 06/13/2009 for the course TAM 455 taught by Professor Petrina during the Fall '08 term at Cornell University (Engineering School).
 Fall '08
 PETRINA

Click to edit the document details