15419lec6.pdf - Future reality Statistics 154 Spring 2019...

This preview shows 1 out of 11 pages.

Statistics 154, Spring 2019 Modern Statistical Prediction and Machine Learning Lecture 6: CV for PCA, NMF, and hierarchical Instructor: Bin Yu ( [email protected] ); office hours: Tu: 9:30-10:30 am; Wed : 1:30-2:30 pm (change) office: 409 Evans GSIs: Yuansi Chen (Mon: 10-12; 4-6); Raaz Dwivedi (Mon: 12-2; 2-4) [email protected] ; [email protected] (Yuansi: Tuesday 1-3; Raaz: Monday 10:30-11:30, Thurs. 9:30-10:30) 1 Future reality
Image of page 1

Subscribe to view the full document.

of Machine Learning/Statistics Taxonomy Reinforcement & Bandit Learning Indirect (reward) Unlabeled Data Unsupervised Learning Dimensionality Reduction Clustering Labeled Data Supervised Learning Regression Classification Thanks to J. Gonzalez
Image of page 2
How do we visualize beyond 4 dimensions? Ames data has 80 features We can at most “look” at 3-dim data (or 4-dim with movies of data) Visual cortex takes up about 30% of our brain so we want to see projections of data into low dim spaces (2d or 3d, for example) How to project? Random projection? Or?
Image of page 3

Subscribe to view the full document.

2/8/19 4 Dimensionality reduction is needed for Visualization Fast computation Smaller storage and faster communication One form for regularization: Simpler models (to reduce variance of estimation) Dimension reduction for more interpretable results of high-dim data
Image of page 4
The central idea of PCA is : to reduce the dimensionality of a data set that has a large number of interrelated variables, while retaining as much as possible of the variation present in the data set. This is achieved by transforming to a new set of variables (PCs) which are uncorrelated (orthogonal) , and which are ordered so that (hopefully) the first few retain most of the variation present in all of the original variables. Jolliffe, Principal Component Analysis, 2nd edition PCA: Core Idea
Image of page 5

Subscribe to view the full document.

PCA recap, illustrated in p=2 dim 2/8/19 6 p d 1 u 1 p d 2 u 2 G = X 0 X = UDU T is the data matrix in the new coordinate system spanned by and u 1 u 2 Data matrix U = ( u 1 , ..., u p ) X n p The ith row of Z is called the “principal components” of the ith data point is the jth PC direction, unit vector in u j Z n p = XU R p (eigen decomposition) R p D = diag ( d 1 , . . . , d p )
Image of page 6
2/8/19 7 As we have shown, an equivalent way to define principal components is to first find the linear combination (loading) with norm 1 of the columns of X such that its variance is maximized. Such a loading forms the first principal component vector. Similarly, we can define the second principal component vector that is the orthogonal to the first and has norm 1 and maximize the variance of the resulted linear combination of the columns of X. PCA: maximizing variance
Image of page 7

Subscribe to view the full document.

PCA in 3-dim to 2-dim: 1-dim lost when is it not a bad idea? 2/8/19 8
Image of page 8
Eigen values matter 2/8/19 9
Image of page 9

Subscribe to view the full document.

2/8/19 10 An important property of the eigen values is that they add up to the trace of the sample covariance matrix or the variance of the original column vectors of X (with columns centered).
Image of page 10
Image of page 11
You've reached the end of this preview.
  • Fall '17
  • Singular value decomposition, eigenface

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern