lecture5

lecture5 - ’E CS UR, E S- . . d, ‘ ' I) fit, a. mag! Z...

Info iconThis preview shows pages 1–10. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 6
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 8
Background image of page 9

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 10
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: ’E CS UR, E S- . . d, ‘ ' I) fit, a. mag! Z i 7 O . PHM Ht Least 8 3mm; ouflmu‘o 94 (L m- PE FS‘ . COMM/x ;.D A, meal/L FFQYIOUX A C 9A7) GM.“ (HJF,<u->H) Quad Wat W0 VMUVS ’X, FVL a. Hilbvfii 8 am 7 P H am Said 7% be, OYYngmaue (x)\/>=O ms is o hm whHm as 9<.L>1. x ‘13 MW ml +0 \Hru; set of Vecfovs 8 If 7(ij the 71/55 is o 7‘04 Wnfi’en as Xi g. A Se 0 vecfovs S is cal/{Ml orkflwgfld ij V x¢y,x%68. and is (31(in WWI/LoermaX }f in adoOVm/n //’X // = / V 9< e S, Gem/(akin I‘VI‘EUWVQ‘I'OALIm/Lflf [mummodmd LQJC \/ be an Mme/1 [wool/«wt 37mm. and 1/6 1/ (at 5% V be, a um‘f Veafav (HM/=1) and CGVISXOUA \MNL, SUbSdQQUL 8: S/baw%b:{. Fpefme g=<’l/,b7b We Mud/1% jaw/IA an 0/95th Wmimflbm or 7/ mW Sum/mu; More, raw 5% MWH: 801% W afl/Zowmj Wobfm: b3) mm //1/—s// 365 Carma/La We, vac/for §= <zu)[,>- b '15 W epfimol aflmeMM {f ‘U in 5: S/Domélpf. WE.) Z’u)b> MUUj be rajaI/OULJ as WLL &nj% of Wm Waffle/n 07 1/ M 5. EXOM’LEUL (Zr/OWL” Schmidt ON’l/Lonmymhzmhm Gyix/UL 4L co/bzcfimoj vedars-?= §L,,bz,-- Inn? ohm/um \j’yarm am, [mayhem/Let gram V) we, VoiS/IIL +0 Co'vwhuct a setof vectors C = gavel Cr? Wat form/w an OWOVzmmoue bags fm Spam {E}. \fl/M'S mwj be accormdzflpbuaé by VIM Gram- gchrmdf oMonmmaflzaLkfiwL Ouvocedwm: (a) Mimhu k=2> g =5.) and $675 CI ’ L WI be {VUL kl ijz 19K-éz.=’ <CL bk>CL (C) 9K=Jée gflt Ck = Lj" H K” else increwnlgj k and Wadi 09) UVL‘}*'1\M\.€, Set ’E is @xbwufieck- Afuchd MampLLOfflwl—vfiiw‘ff CW‘oLmW/ufo/(Mmj 505 in R3: @5H21H31EJ5 V09, ML W G/mm— Sclxxm‘dt ocLoUA/w 'fo %v.n0(. an car/Wono’imd bag/{5 69cm mafia}: , ; ’ =l ’ .tjl'b, [Ejami (1] ‘ 0 'Y3‘ 105 ‘ <C.I)L3>CI ’ (CZ;ES>CL ;[O] 0 7M C= éc, ,czé’ fem m WM OIWOYmeaQ bags. +———— E. The Projection Theorem 1 Let V be an inner-product space and let S be a subspace of V. Fix v E V and consider the problem: inf llv r 3H (#) 565 In other words, we wish to find an optimal approximation s in S to 1). It may happen that the infimum above is not achieved. In this case, we will say that the optimization problem (#) does not admit a solution. It may happen that the infimum above is achieved. Any vector s that achieves the inti mum will be called optimal (there may be many such vectors). Theorem (a) Suppose the optimization problem {#) is solvable. Then, s is optimal if and only 2'f (v ~ s)J_S Further, the optimal vector g is unique. (b) Suppose S is complete. Then is solvable. (c) Suppose S is finite-dimensional. Let B = {b1,b2,---,bn} be a basis for S and define the matrix M 6 cm“ by <b1,b1> <b1,bn> M: . . , <bmb1> <bmbn> Then M is nonsingular. Moreover, is solvable and the unique solution is given. by n S: E crib.- {:1 where 01 <b1,’U> . = Ar‘ . 0,, <1)", u> —Rolla Proof: (a) Let é be optimal. We will show that (v — §)lS. Suppose not, i.e., there exists 51 E S such that <sl,v—§>=a7é0 Define B = cr/llsfll2 and the vector snew = s + 551 E 5. Observe that [Iv—5mm“2 = llv-SA-BSIH2 = ||v—§||2+HB.91|]2—— <v—§,le> — <le,v—§> : llv—§H2+lfilzllslllz-Ba—fia = “v - éllz -lfi|2||51||2 which shows that s is not optimal. This contradicts our hypothesis, proving our claim that (v — §)LS. To prove the reverse, let s be any vector that satisfies (1) — §)LS. We will show that :3 be optimal. For this, examine llv—Sll2 = llU—é‘Jr§—6‘ll2 2+||§—s||2+ <v—§,§—s> + <§—s,'u—§> = llv — $1 The final two inner products above are zero, since (1) — §)LS. This gives us llv - 3H2 = llv — éll2 + “3 - 5:“2 2 H12 — §||2 for all s which establishes optimality of s. Finally, we will show uniqueness of the optimal vector. For this, let both be optimal. Then, (we have just proved this) (1) — §)LS and (v — .§)LS. Now examine 2 —.e; 0:» l | 40:) 0:») I Ch) V 7—] which implies that — s = 0. This completes the proof. (b) Let S be complete. Define 'y 2 infses H1) — We have to establish that the imfemum is achieved, or that there exists a vector 50 E S such that — so“ = 7. It is clear from the definition of 7 that for any 8 E 5, He — SH 2 7. Also, there exists a sequence of vectors {8);} E S such that lim H1) — 8k“ 2 'y k—ooo Now, using the parallelogram law, we get “(Si — U) + (U — W“2 + “(32‘ * U) — (U - 81)”2 = 2||Si — UH2 + 2llsj — 11H2 This can be rearranged to give ”+%w ma—MF+Nn—UW*MW‘ 2 “Si — Sjllz S ZHSi — UHZ + lesj — 11H2 — 472 Then, using the fact that limk_.00 Ilv — 8);” = 'y, we get lim “3, — 3,“? g 272 + 272 — 472 : 0 k—ooo This forces limk_.00 “s, — sj|| = 0. Thus, the sequence of vectors {3k} 6 S is Cauchy. Since 8 is complete, we are assured that this sequence converges to a vector 8" E 8. Finally, using continuity of norms, “11—80” = lim H'U—Skll =7 k—voo completing the proof. DD (c) We first show that Al is nonsingular. Suppose not. Then, there exists a vector 0 79 a E C" such that Ma 2 0. This implies that n n 0:0‘N10222 a: <b¢,bj> aj = <q,q> = Iqu2 i=1j=l ' n where q z aibi). 12:] As a consequence, q = Z ajbj = 0 (1) 1:1 Honrever, because [3 = {b1,b2, ~ - - ,b,,} is a basis, it is linearly independent. This contradicts ( l I, prOviug that M is invertible. Next, since 8 is finite dimensional, it is complete. Then, it is clear from part (b) of the Projection Theorem that the optimization problem (#) is solvable. Since 8 is a basis for S, we can write the optimal vector 5- E S as the linear combination n 5' = Z ajbj jZI From part (a) the unique optimal vector s above must satisfy (v~—§)J_S, or equivalently, <b,¢, (v — > = 0 for i = 1, - ' - , n. Rewriting this gives 71 <b.,i,U> = <bj,.§> =2: <bi,bj> aj j=l Stacking up these equations for i = 1, - - ' , 71 yields <b1,’U> <b1,‘U> . 2 Ma or a = M‘1 _ <bn, v > <bn, v> proving the claim. DD. 2 Example (Least Squares.) We can employ the Projection theorem above to establish the following central result. Theorem Let A E R’”"" have rank(/i) : n. Fin: 3; E R'" and consider the problem fig}, “AI ‘ yll2 Then, the unique solution of this problem is i = (A’A)_1A’y 3 Example (Fourier cosine series.) The classical Fourier series approximation may be viewed as an instance of the pro— jection theorem. Consider the inner product space L2 [0, 2n]. Let S : Span {1,cos(t), - - - ,cos(kt)} Given a function f(t) E L2[0,27r] we seek the optimal approximation in S of f. Using the Projection theorem we obtain f(t) : a0 + a1 cos(t) + - - - + ak. c0s(lct) where 1 277 1 271’ ‘10 _ “Wit, 01! = — f(t)cos(Zt)dt, Z = 1,2, . . . 7 k 27f 0 7T 0 These familiar formulae are the expressions for the Fourier cosine expansion of f. Definition Let 5 be a subspace of a Hilbert space V. The orthogonal complement ofS is the set Si defined by 5i={v€V: ILLS} Lemma {(1) Si is a subspace. (b) Si is complete. (C) (3i)i 2 5 (d) US is complete, then V = 8 EB Si (e) [f5 is complete, then (Si)i = 5 Proof: (a) Ly E 5i implies that < a:,q > = < y,q > = 0 for all q E S. Then, <$+y,q>=0and <am,q>=0forallq€5. E] (b) Let 173° be a Cauchy sequence in Si. Since this is also a Cauchy sequence in V which is a Hilbert space, we know that I], —> .r" E V. In order to show that Si is complete, we have to show that the limit :5” above is in Si. For this, note that by continuity of the inner product, for any 3 E 5, <$°,s> = lim <mk,s> = lim 0:0 k—‘oo k—‘oo because 1']; E 5i. This proves that 1;” 6 Si. [:1 (e) Let s E S. Then, for all q E 5i, we have < q,s > = 0 which implies that s E (SiH. D (d) Fix 1) E V. Let s be the optimal approximation of v in 5. This exists by the Projection Theorem on observing that 5 is complete. Then, we can write v=.§+(v—s) Note that from the Projection Theorem, (v — .§)_J_S or (v — E 5i. We have thus shown that any vector in V can be expressed as the sum of a vector in 5 with one in SJ‘i All that remains is to show that this decomposition is unique. For this, suppose we can write 11 E V as v=s+q251+q1 where 5,51 68 q,q1€SL Then, 5 — $1 = q -q1. But 8 and SJ- are subspaces. Thus, s—slzq—ql €808i Then, we have 0: <s—51,q—q1> = <s—sl,s—sl> = “3—51”2 forcing s = .91 and q = q1, establishing uniqueness of the decomposition. D (e) All that remains is to show that (Si)i Q S. For this, let q E (SHi. Using the direct-sum decomposition in (d)7 we can write q = 5 +7‘ with s E S and r 6 Si. Since q E (SiH, we have that <q,7‘> = 0. Then, 0: <q,7‘> = <s,7‘> + <7‘,r> = <r,7‘> where the final equality above follows because sir. It is then clear that 7' = O, forcing q = s E S, proving the claim. (3 ...
View Full Document

This note was uploaded on 04/11/2011 for the course EE 221A taught by Professor Clairetomlin during the Fall '10 term at Berkeley.

Page1 / 10

lecture5 - ’E CS UR, E S- . . d, ‘ ' I) fit, a. mag! Z...

This preview shows document pages 1 - 10. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online