This preview shows pages 1–10. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: ’E
CS
UR,
E
S .
. d, ‘
' I) ﬁt, a. mag! Z i 7 O . PHM
Ht Least 8
3mm; ouﬂmu‘o 94
(L m PE
FS‘
. COMM/x ;.D
A, meal/L
FFQYIOUX A C
9A7) GM.“ (HJF,<u>H)
Quad Wat W0 VMUVS ’X, FVL a. Hilbvﬁi 8 am
7 P H am Said 7% be, OYYngmaue (x)\/>=O ms is o hm whHm as 9<.L>1.
x ‘13 MW ml +0 \Hru; set of Vecfovs 8 If 7(ij the 71/55 is o 7‘04 Wnﬁ’en
as Xi g.
A Se 0 vecfovs S is cal/{Ml orkﬂwgﬂd ij V x¢y,x%68.
and is (31(in WWI/LoermaX }f in adoOVm/n
//’X // = / V 9< e S, Gem/(akin I‘VI‘EUWVQ‘I'OALIm/Lﬂf [mummodmd LQJC \/ be an Mme/1 [wool/«wt 37mm. and 1/6 1/
(at 5% V be, a um‘f Veafav (HM/=1) and
CGVISXOUA \MNL, SUbSdQQUL 8: S/baw%b:{. Fpefme g=<’l/,b7b
We Mud/1% jaw/IA an 0/95th Wmimﬂbm or 7/ mW Sum/mu; More, raw
5% MWH: 801% W aﬂ/Zowmj Wobfm: b3) mm //1/—s// 365 Carma/La We, vac/for §= <zu)[,> b '15 W
epﬁmol aﬂmeMM {f ‘U in 5: S/Domélpf. WE.) Z’u)b> MUUj be rajaI/OULJ as WLL &nj%
of Wm Wafﬂe/n 07 1/ M 5.
EXOM’LEUL (Zr/OWL” Schmidt ON’l/Lonmymhzmhm Gyix/UL 4L co/bzcﬁmoj vedars?= §L,,bz, Inn?
ohm/um \j’yarm am, [mayhem/Let gram V) we,
VoiS/IIL +0 Co'vwhuct a setof vectors
C = gavel Cr? Wat form/w an OWOVzmmoue
bags fm Spam {E}. \ﬂ/M'S mwj be accormdzﬂpbuaé
by VIM Gram gchrmdf oMonmmaﬂzaLkﬁwL
Ouvocedwm: (a) Mimhu k=2> g =5.) and $675
CI ’ L
WI be {VUL kl
ijz 19Kéz.=’ <CL bk>CL
(C) 9K=Jée gﬂt
Ck = Lj" H K”
else increwnlgj k and Wadi 09) UVL‘}*'1\M\.€,
Set ’E is @xbwuﬁeck Afuchd MampLLOfﬂwl—vﬁiw‘ff
CW‘oLmW/ufo/(Mmj 505 in R3:
@5H21H31EJ5 V09, ML W G/mm— Sclxxm‘dt ocLoUA/w 'fo %v.n0(.
an car/Wono’imd bag/{5 69cm maﬁa}: , ; ’ =l ’
.tjl'b, [Ejami (1] ‘ 0
'Y3‘ 105 ‘ <C.I)L3>CI ’ (CZ;ES>CL ;[O]
0 7M C= éc, ,czé’ fem m WM OIWOYmeaQ bags. +———— E. The Projection Theorem 1 Let V be an innerproduct space and let S be a subspace of V.
Fix v E V and consider the problem: inf llv r 3H (#) 565 In other words, we wish to ﬁnd an optimal approximation s in S to 1). It may happen that the infimum above is not achieved. In this case, we will say that
the optimization problem (#) does not admit a solution. It may happen that the infimum above is achieved. Any vector s that achieves the
inti mum will be called optimal (there may be many such vectors). Theorem (a) Suppose the optimization problem {#) is solvable. Then, s is optimal if and only
2'f
(v ~ s)J_S
Further, the optimal vector g is unique.
(b) Suppose S is complete. Then is solvable.
(c) Suppose S is ﬁnitedimensional. Let
B = {b1,b2,,bn}
be a basis for S and deﬁne the matrix M 6 cm“ by
<b1,b1> <b1,bn> M: . . ,
<bmb1> <bmbn> Then M is nonsingular. Moreover, is solvable and the unique solution is given. by n
S: E crib.
{:1 where
01 <b1,’U>
. = Ar‘ .
0,, <1)", u> —Rolla Proof: (a) Let é be optimal. We will show that (v — §)lS. Suppose not, i.e., there
exists 51 E S such that <sl,v—§>=a7é0 Deﬁne B = cr/llsﬂl2 and the vector snew = s + 551 E 5. Observe that [Iv—5mm“2 = llvSABSIH2
= v—§2+HB.91]2—— <v—§,le> — <le,v—§>
: llv—§H2+lﬁlzllslllzBa—ﬁa = “v  éllz lﬁ2512 which shows that s is not optimal. This contradicts our hypothesis, proving our claim
that (v — §)LS. To prove the reverse, let s be any vector that satisﬁes (1) — §)LS. We will show that
:3 be optimal. For this, examine llv—Sll2 = llU—é‘Jr§—6‘ll2
2+§—s2+ <v—§,§—s> + <§—s,'u—§> = llv — $1
The ﬁnal two inner products above are zero, since (1) — §)LS. This gives us
llv  3H2 = llv — éll2 + “3  5:“2 2 H12 — §2 for all s which establishes optimality of s. Finally, we will show uniqueness of the optimal vector. For this, let both be
optimal. Then, (we have just proved this) (1) — §)LS and (v — .§)LS. Now examine 2 —.e; 0:» l 
40:)
0:») I
Ch)
V 7—] which implies that — s = 0. This completes the proof. (b) Let S be complete. Deﬁne 'y 2 infses H1) — We have to establish that the
imfemum is achieved, or that there exists a vector 50 E S such that — so“ = 7. It is clear from the deﬁnition of 7 that for any 8 E 5, He — SH 2 7. Also, there exists a
sequence of vectors {8);} E S such that lim H1) — 8k“ 2 'y
k—ooo Now, using the parallelogram law, we get
“(Si — U) + (U — W“2 + “(32‘ * U) — (U  81)”2 = 2Si — UH2 + 2llsj — 11H2 This can be rearranged to give ”+%w ma—MF+Nn—UW*MW‘ 2 “Si — Sjllz S ZHSi — UHZ + lesj — 11H2 — 472
Then, using the fact that limk_.00 Ilv — 8);” = 'y, we get lim “3, — 3,“? g 272 + 272 — 472 : 0 k—ooo This forces limk_.00 “s, — sj = 0. Thus, the sequence of vectors {3k} 6 S is Cauchy.
Since 8 is complete, we are assured that this sequence converges to a vector 8" E 8. Finally, using continuity of norms, “11—80” = lim H'U—Skll =7 k—voo completing the proof. DD (c) We ﬁrst show that Al is nonsingular. Suppose not. Then, there exists a vector
0 79 a E C" such that Ma 2 0. This implies that n n
0:0‘N10222 a: <b¢,bj> aj = <q,q> = Iqu2
i=1j=l ' n
where q z aibi).
12:] As a consequence, q = Z ajbj = 0 (1)
1:1
Honrever, because [3 = {b1,b2, ~   ,b,,} is a basis, it is linearly independent. This contradicts ( l I, prOviug that M is invertible. Next, since 8 is ﬁnite dimensional, it is complete. Then, it is clear from part (b) of
the Projection Theorem that the optimization problem (#) is solvable. Since 8 is a
basis for S, we can write the optimal vector 5 E S as the linear combination n
5' = Z ajbj
jZI From part (a) the unique optimal vector s above must satisfy (v~—§)J_S, or equivalently,
<b,¢, (v — > = 0 for i = 1,  '  , n. Rewriting this gives 71 <b.,i,U> = <bj,.§> =2: <bi,bj> aj j=l
Stacking up these equations for i = 1,   ' , 71 yields
<b1,’U> <b1,‘U>
. 2 Ma or a = M‘1 _
<bn, v > <bn, v>
proving the claim. DD. 2 Example (Least Squares.)
We can employ the Projection theorem above to establish the following central result.
Theorem Let A E R’”"" have rank(/i) : n. Fin: 3; E R'" and consider the problem ﬁg}, “AI ‘ yll2 Then, the unique solution of this problem is
i = (A’A)_1A’y 3 Example (Fourier cosine series.) The classical Fourier series approximation may be viewed as an instance of the pro—
jection theorem. Consider the inner product space L2 [0, 2n]. Let
S : Span {1,cos(t),    ,cos(kt)} Given a function f(t) E L2[0,27r] we seek the optimal approximation in S of f.
Using the Projection theorem we obtain f(t) : a0 + a1 cos(t) +    + ak. c0s(lct) where 1 277 1 271’
‘10 _ “Wit, 01! = — f(t)cos(Zt)dt, Z = 1,2, . . . 7 k 27f 0 7T 0
These familiar formulae are the expressions for the Fourier cosine expansion of f. Deﬁnition Let 5 be a subspace of a Hilbert space V. The orthogonal complement
ofS is the set Si deﬁned by 5i={v€V: ILLS} Lemma {(1) Si is a subspace. (b) Si is complete. (C) (3i)i 2 5 (d) US is complete, then V = 8 EB Si (e) [f5 is complete, then (Si)i = 5
Proof: (a) Ly E 5i implies that < a:,q > = < y,q > = 0 for all q E S. Then,
<$+y,q>=0and <am,q>=0forallq€5. E] (b) Let 173° be a Cauchy sequence in Si. Since this is also a Cauchy sequence in V
which is a Hilbert space, we know that I], —> .r" E V. In order to show that Si is
complete, we have to show that the limit :5” above is in Si. For this, note that by continuity of the inner product, for any 3 E 5, <$°,s> = lim <mk,s> = lim 0:0
k—‘oo k—‘oo because 1']; E 5i. This proves that 1;” 6 Si. [:1 (e) Let s E S. Then, for all q E 5i, we have < q,s > = 0 which implies that
s E (SiH. D
(d) Fix 1) E V. Let s be the optimal approximation of v in 5. This exists by the
Projection Theorem on observing that 5 is complete. Then, we can write v=.§+(v—s) Note that from the Projection Theorem, (v — .§)_J_S or (v — E 5i. We have thus
shown that any vector in V can be expressed as the sum of a vector in 5 with one in SJ‘i All that remains is to show that this decomposition is unique. For this, suppose
we can write 11 E V as v=s+q251+q1 where 5,51 68 q,q1€SL
Then, 5 — $1 = q q1. But 8 and SJ are subspaces. Thus,
s—slzq—ql €808i
Then, we have
0: <s—51,q—q1> = <s—sl,s—sl> = “3—51”2 forcing s = .91 and q = q1, establishing uniqueness of the decomposition. D (e) All that remains is to show that (Si)i Q S. For this, let q E (SHi. Using the
directsum decomposition in (d)7 we can write q = 5 +7‘ with s E S and r 6 Si. Since
q E (SiH, we have that <q,7‘> = 0. Then, 0: <q,7‘> = <s,7‘> + <7‘,r> = <r,7‘> where the ﬁnal equality above follows because sir. It is then clear that 7' = O, forcing
q = s E S, proving the claim. (3 ...
View
Full
Document
This note was uploaded on 04/11/2011 for the course EE 221A taught by Professor Clairetomlin during the Fall '10 term at Berkeley.
 Fall '10
 ClaireTomlin

Click to edit the document details