<a href="/keyword/least-squares-estimation/" >least squares estimation</a> Namrata Vaswani, [e-mail] <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 1 Recall: Geometric Intuition for Least Squares Minimize J(x) = ||y - Hx||2 Solution satisfies: H T H x = H T y, i.e. x = (H T H)-1 H T y ^ ^ So H T (y - H x) = 0 ^ The least error (y - H x) is to column space of H ^ Think 3D: minimum error is always to plane of projection <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 2 Weighted Least Squares y = Hx + e Minimize J(x) = (y - Hx)T W (y - Hx) Solution: x = (H T W H)-1 H T W y ^ (2) ||y - Hx||2 W (1) Given that E[e] = 0 and E[eeT ] = V , Min. Variance Unbiased Linear Estimator of x: choose W = V -1 in (2) Min. Variance of a vector: variance in any direction is minimized <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 3 Proof (skip if you want to) Given x = Ly, find L, s.t. E[Ly] = E[LHx] = E[x], so LH = I ^ Let L0 = (H T V -1 H)-1 H T V -1 Error variance E[(x - x)(x - x)T ] ^ ^ E[(x - x)(x - x)T ] ^ ^ = = E[(x - LHx - Le)(x - LHX - Le)T ] E[LeeT LT ] = LV LT Say L = L - L0 + L0 . Here LH = I, L0 H = I, so (L - L0 )H = 0 LV LT = = = L0 V LT + (L - L0 )V (L - L0 )T + 2L0 V (L - L0 )T 0 L0 V LT + (L - L0 )V (L - L0 )T + (H T V -1 H)-1 H T (L - L0 )T 0 L0 V LT + (L - L0 )V (L - L0 )T L0 V LT 0 0 Thus L0 is the optimal estimator (Note: for matrices) <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 4 Regularized Least Squares Minimize J(x) = (x - x0 )T -1 (x - x0 ) + (y - Hx)T W (y - Hx) 0 x J(x) = = z x - x0 , y y - Hx0 (3) x T -1 x + y T W y 0 z M -1 z I 0 x - H y -1 0 0 0 W M <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 5 Solution: Use weighted least squares formula with y = ~ ~ H= Get: x = x0 + (-1 + H T W H)-1 H T W (y - Hx0 ) ^ 0 I H ~ , W = M 0 y , Advantage: improves condition number of H T H, incorporate prior knowledge about distance from x0 <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 6 Recursive Least Squares Use in one of following situations: When number of equations much larger than number of variables: Storage problem Getting data sequentially, do not want to re-solve the full problem again The dimension of x is large, want to avoid inverting matrices Goal: At step i - 1, have xi-1 : minimizer of ^ (x - x0 )T -1 (x - x0 ) + ||Hi-1 x - Yi-1 ||2 i-1 , Yi-1 = [y1 , ...yi-1 ]T 0 W Find xi : minimizer of (x - x0 )T -1 (x - x0 ) + ||Hi x - Yi ||2 i , ^ 0 W <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 7 Hi = Hi-1 hi (hi is a row vector), Yi = [y1 , ...yi ]T (column vector) For simplicity of notation, assume x0 = 0 and Wi = I. T Hi Hi T = Hi-1 Hi-1 + hT hi i T T = (-1 + Hi Hi )-1 Hi Yi 0 T T = (-1 + Hi-1 Hi-1 + hT hi )-1 (Hi-1 Yi-1 + hT yi ) i i 0 xi ^ Define Pi So Pi T = (-1 + Hi Hi )-1 , P0 = 0 0 -1 = [Pi-1 + hT hi ]-1 i Use Matrix Inversion identity: <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 8 (A + BCD)-1 = A-1 + A-1 B(C -1 + DA-1 B)-1 DA-1 Pi = Pi-1 - Ki hi Pi-1 where Ki = Pi-1 hT (1 + hi Pi-1 hT )-1 i i Thus x0 ^ xi ^ = 0 = = T Pi Hi Yi T = [Pi-1 - Ki hi Pi-1 ][Hi-1 Yi-1 + hT yi ] i (4) xi-1 + Ki (yi - hi xi-1 ) ^ ^ T The last equality uses the facts that (i) xi-1 = Pi-1 Hi-1 Yi-1 , (ii) ^ [Pi-1 - Ki hi Pi-1 ]hT yi = Ki yi (expand Ki , obtain this after a few i manipulations). <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 9 Here we considered the weight Wi = I. If Wi = I, the equation for Ki 1/2 1/2 modifies to (replace yi by wi yi &amp; hi by wi hi , where wi = (Wi )i,i ) Ki = Pi-1 hT (wi -1 + hi Pi-1 hT )-1 i i (5) Also, here we considered yi to be a scalar and hi to be a row vector. In general: yi can be a k-dim vector, hi will be a matrix with k rows, and the above formulae still apply, replace 1 by I everywhere RLS with Forgetting factor Weight older data with smaller weight J(x) = Exponential forgetting: (i, j) = i-j , &lt;1 i j=1 (yj - hj x)2 (i, j) Moving average: (i, j) = 0 if |i - j| &gt; and (i, j) = 1 otherwise <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 10 Summarizing Recursive LS In general can assume that yi is k dimensional and so hi has k rows. Weight matrix (Wi )i,i = wi . Solution is: x0 ^ Ki Pi xi ^ = = = = x0 , P0 = 0 Pi-1 hT (wi -1 + hi Pi-1 hT )-1 i i (I - Ki hi )Pi-1 xi-1 + Ki (yi - hi xi ) ^ (6) This is a recursive way to get the Regularized LS solution T xi = (-1 + Hi Wi Hi )-1 Yi ^ 0 T T T with Hi = [hT , hT , ...hT ]T , Yi = [y1 , y2 , ...yi ]T 1 2 i (7) <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 11 Connection with Kalman Filtering The above is also the Kalman filter estimate of the state for the following system model: xi yi = = xi-1 -1 hi xi + vi , vi N (0, Ri ), wi = Ri (8) <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 12 Kalman Filter Motivation RLS was for static data: estimate the signal x better and better as more and more data comes in, e.g. estimating the mean intensity of an object from a video sequence RLS with forgetting factor assumes slowly time varying x Kalman filter: if the signal is time varying, and we know (statistically) the dynamical model followed by the signal: e.g. tracking a moving object x0 xi = N (0, 0 ) Fi xi-1 + vx,i , vx,i N (0, Qi ) The observation model is as before: yi = hi xi + vi , vi N (0, Ri ) <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 13 Goal: get the best (minimum mean square error) estimate of xi from Yi Cost: J(^i ) = E[(xi - xi )2 |Yi ] x ^ Minimizer: conditional mean xi = E[xi |Yi ] ^ This is also the MAP estimate, i.e. xi also maximizes p(xi |Yi ) ^ <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 14 Example Applications Recursive LS: Adaptive noise cancelation Channel equalization using a training sequence Object intensity estimation: x = intensity, yi = vector of intensities of object region in frame i, hi = 1m (column vector of m ones) Keep updating estimate of location of an object that is static, using a sequence of location observations coming in sequentially Recursive LS with forgetting factor: object not static but drifts very slowly (e.g. floating object) or object intensity changes very slowly Kalman filter: Track a moving object (estimate its location, velocity at each time), when acceleration is assumed i.i.d. Gaussian <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 15 Material adapted from Chapters 2, 3 of Linear Estimation, by Kailath, Sayed, Hassibi <a href="/keyword/least-squares-estimation/" >least squares estimation</a> 16