Introduction
Why dimensionality reduction is important: -
To store data particular space is used, by using we can reduce the space required. Also less
dimensions lead to less computation/training time. Some algorithms do not perform well when
we have a large dimension. It helps in visualizing data. Complexity depends on no. of inputs (d)
& size of data sample (N) Decreasing input decrease the complexity of algorithm during testing.
Simpler models are more robust on small datasets i.e. less variance
More interpretable is simpler explanation Example To classify whether the email is spam or not
Dimensionality reduction can be done in two different ways:
1.
By only keeping the most relevant variables from the original dataset
2.
By finding a smaller set of new variables
Two component of dimensionality reduction
Feature selection:
To find a subset of original set of variables of features
It usually involves three ways:
1.
Filter 2. Wrapper 3. Embedded
we are interested to finding k of the d dimensions that give us most information,
ignoring the remaining
d
–
k

2.
Feature extraction:
This reduces the data in a high dimensional space to a lower
dimension space. finding new set of k dimensions that are combinations of the original d
dimension. These methods maybe supervised or unsupervised. Extraction method are
LDA and PCA
Method
1.
Principal Component Analysis (PCA)
2.
Linear Discriminant Analysis (LDA)
3.
Generalized Discriminant Analysis (GDA)
Important term in Dimensionality reduction
A
covariance
refers to the measure of how two random variables will change together and
is
used to calculate the correlation
between
variables.
The
variance
refers to the spread of the data set
—
how far apart the numbers are in relation to
the mean, for instance. (Also explain overfitting underfitting)
Factor Analysis
Factor analysis is a linear statistical model. It is used to explain the variance among the observed
variable and condense a set of the observed variable into the unobserved variable called factors.
Observed variables are modeled as a linear combination of factors and error terms. Factor or
latent variable is associated with multiple observed variables, who have common patterns of
responses. Each factor explains a particular amount of variance in the observed variables. It helps
in data interpretations by reducing the number of variables.
Factor analysis is a method for investigating whether a number of variables of interest X1,
X2,……., Xl, are linearly related to a smaller number of unobservable factors F1, F2,..……, Fk.