A Tutorial on Principal Component Analysis
Jonathon Shlens
*
Systems
Neurobiology
Laboratory,
Salk
Insitute
for
Biological
Studies
La Jolla, CA 92037 and
Institute
for
Nonlinear
Science,
University
of
California,
San
Diego
La Jolla, CA 920930402
(Dated: December 10, 2005; Version 2)
Principal component analysis (PCA) is a mainstay of modern data analysis  a black box that
is widely used but poorly understood. The goal of this paper is to dispel the magic behind this
black box. This tutorial focuses on building a solid intuition for how and why principal component
analysis works; furthermore, it crystallizes this knowledge by deriving from simple intuitions, the
mathematics behind
PCA
. This tutorial does not shy away from explaining the ideas informally,
nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers
of all levels will be able to gain a better understanding of
PCA
as well as the when, the how and
the why of applying this technique.
I. INTRODUCTION
Principal component analysis (
PCA
) has been called
one of the most valuable results from applied linear al
gebra.
PCA
is used abundantly in all forms of analysis 
from neuroscience to computer graphics  because it is a
simple, nonparametric method of extracting relevant in
formation from confusing data sets. With minimal addi
tional effort
PCA
provides a roadmap for how to reduce
a complex data set to a lower dimension to reveal the
sometimes hidden, simplified structure that often under
lie it.
The goal of this tutorial is to provide both an intuitive
feel for
PCA
, and a thorough discussion of this topic.
We will begin with a simple example and provide an intu
itive explanation of the goal of
PCA
. We will continue by
adding mathematical rigor to place it within the frame
work of linear algebra to provide an explicit solution. We
will see how and why
PCA
is intimately related to the
mathematical technique of singular value decomposition
(
SVD
). This understanding will lead us to a prescription
for how to apply
PCA
in the real world. We will discuss
both the assumptions behind this technique as well as
possible extensions to overcome these limitations.
The discussion and explanations in this paper are infor
mal in the spirit of a tutorial. The goal of this paper is to
educate
. Occasionally, rigorous mathematical proofs are
necessary although relegated to the Appendix. Although
not as vital to the tutorial, the proofs are presented for
the adventurous reader who desires a more complete un
derstanding of the math. The only assumption is that the
reader has a working knowledge of linear algebra. Please
feel free to contact me with any suggestions, corrections
or comments.
*
Electronic address:
[email protected]
II. MOTIVATION: A TOY EXAMPLE
Here is the perspective: we are an experimenter. We
are trying to understand some phenomenon by measur
ing various quantities (e.g. spectra, voltages, velocities,
etc.) in our system. Unfortunately, we can not figure out
what is happening because the data appears clouded, un
clear and even redundant. This is not a trivial problem,
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '08
 Wilczynski
 Linear Algebra, Singular value decomposition

Click to edit the document details