There are actually two equivalent ways to think about PCA The first is

There are actually two equivalent ways to think about

• Notes
• 18

This preview shows page 10 - 13 out of 18 pages.

There are actually two equivalent ways to think about PCA. The first is statistical: we are trying to find a transform that is carefully tuned to the (second-order) statistics of the data. The second perspective, which is what we will adopt in this course, is more geometrical: given a set of vectors, we are trying to find a subspace of a certain dimension that comes closest to containing this set. Specifically, suppose that we have data points x 1 , . . . , x N R D , and want to find the K -dimensional affine space (subspace plus offset) that comes closest to containing them. Here is a picture 4 Example From Chapter 14 of Hastie, Tibshirani, and Friedman 16 4 From Ch. 14 of Tibshirani and Hastie’s Elements of Statistical Learning . 76 Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 23:01, November 5, 2019

Subscribe to view the full document.

Our goal is to find an offset μ R D and a matrix Q with orthonormal columns such that x n μ + n for all n = 1 , . . . , N, for some θ n R K . We cast this as the following optimization prob- lem. Given x 1 , . . . , x N , solve minimize μ , Q , { θ n } N X n =1 k x n - μ - n k 2 2 subject to Q T Q = I . Note that if we fix μ and define e x n = x n - μ , then we can recast the optimization with respect to Q and the θ n as minimize Q , { θ n } N X n =1 k e x n - n k 2 2 subject to Q T Q = I . If f X and Θ denote the matrices whose columns are given by e x 1 , . . . , e x n and θ 1 , . . . , θ n respectively, then we can also write this as minimize Q : D × K Θ : K × N k f X - Q Θ k 2 F subject to Q T Q = I . This is exactly the optimization problem that we looked at previ- ously in our Subspace Approximation Lemma! Thus the solution is given by computing the SVD of f X = U Σ V and then taking as our solution b Q = U K , b Θ = U T K f X , where U K = u 1 u 2 · · · u K contains the first K columns of U . 77 Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 23:01, November 5, 2019
Finally, let us return to the question of how to set μ . For any given μ , the solution for Q and Θ is given by the Subspace Approximation Lemma. This results in setting θ n = Q T ( x n - μ ) . Plugging this in for θ n in our objective function, we have that x n - μ - n = x n - μ - QQ T ( x n - μ ) = ( I - QQ T )( x n - μ ) . Hence, the problem of selecting μ reduces to the optimization prob- lem minimize μ N X n =1 k ( I - QQ T )( x n - μ ) k 2 2 The vector μ is unconstrained; we can solve for the optimal μ by taking a gradient and setting it equal to zero. To make this easier, note that k ( I - QQ T )( x n - μ ) k 2 2 = ( x n - μ ) T ( I - QQ T )( x n - μ ) by simply expanding out the norm squared as an inner product and then using the fact that I - QQ T is a projector, i.e., it is symmetric and ( I - QQ T ) 2 = I - QQ T . Thus, by taking a gradient and setting it equal to zero we have 0 = - 2 N X n =1 ( I - QQ T )( x n - μ ) = - 2( I - QQ T ) N X n =1 x n !

Subscribe to view the full document.

• Fall '08
• Staff

What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern