This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: STA 4107/5107 Statistical Learning: Principle Components and Partial Least Squares Regression March 28, 2007 1 Introduction Principal components analysis is traditionally presented as an interpretive multivariate technique, where the loadings are chosen to maximally explain the variance in the variable. However, we will consider it here mainly as a statistical learning tool, by using the derived components in a least squares regression to predict unobserved response variables using the principal components. Principal components aims to explain as much of the variation in the data as possible by finding linear combinations that are independent of each other and in the direction of the greatest variation. Each principal component is a linear combination of all variables. The first principal component explains the most variation, the second PC the second most, and so on. There are as many principal components as there are variables, but we usually choose only the first few for both exploratory and regression analysis. Partial least squares is a method of data dimension reduction, similar to principal components, to find the most relevant factors for both prediction and interpretation, and is derived from Herman Wold’s development of iterative fitting of bilinear models (Wold, 1981, 1983). Partial least squares regression (PLSR) improves upon principal components analysis by actively using the response variables during the bilinear decomposition of the predictors. Principal components focuses on the variance in the predictors, while partial least squares focuses on the covariance between the response and the predictors. By balancing the information in both the predictors and the response, PLS reduces the impact of large, but irrelevant predictor variations. Estimation of prediction error is achieved using cross-validation. 1.1 Statistical Learning There are many statistical data analysis techniques that fall into the category of statistical learning . The two regression techniques considered here, partial least squares regression (PLSR) and principal components regression, are among them. The most commonly used statistical learning method is normal linear regression. We refer to it as ‘learning’ because we have data that we want to use to discover the relationship between a quantity that we would like to predict and one or several other quantities, called predictors. That is, we would like to ‘learn from examples.’ The data we have 1 consists of measurements of both the response (the quantity we would like to predict), along with corresponding measurements of the predictors, and is called the ‘training set’. We then use our data to ‘train’ a statistical algorithm, which produces a mathematical relationship via parameter estimation, from the known data, which we can then use to predict the quantity of interest in a test data set where all we have measured are the predictors. In this sense we have ‘learned’ from our training data to predict future observations.our training data to predict future observations....
View Full Document
This note was uploaded on 07/14/2011 for the course STA 4702 taught by Professor Staff during the Spring '08 term at University of Florida.
- Spring '08
- Least Squares