Chapter 12
Logistic Regression
12.1
Modeling Conditional Probabilities
So far, we either looked at estimating the conditional expectations of continuous
variables (as in regression), or at estimating distributions. There are many situations
where however

Classication
Qualitative variables take values in an unordered set C,
such as:
eye color2 cfw_brown, blue, green
email2 cfw_spam, ham.
Given a feature vector X and a qualitative response Y
taking values in the set C, the classication task is to build
a

Lecture Notes on Linear Regression
Mrio A. T. Figueiredo,
a
Departamento de Engenharia Electrotcnica e de Computadores,
e
Instituto Superior Tcnico, Lisboa, Portugal
e
Latest update: March 2010
1
Least Squares Linear Regression
We are given a set of input

Linear regression
Chapter 3
Linear regression is a simple approach to supervised
learning. It assumes that the dependence of Y on
X1 , X2 , . . . Xp is linear.
1 / 48
Linear regression
(x) = a simple + 2 x2 + to p xp
Linear regression is 0 + 1 x1 approa

Moving Beyond Linearity
The truth is never linear!
1 / 23
Moving Beyond Linearity
The truth is never linear!
Or almost never!
1 / 23
Moving Beyond Linearity
The truth is never linear!
Or almost never!
But often the linearity assumption is good enough.
Giv

MULTICOLLINEARITY
In multiple regression models, we have ignored something that we
probably shouldn't have and that's what is called multicollinearity.
Multicollinearity exists when two or more of the predictors in a
regression model are moderately or hig

25
20
0
50
100
200
TV
300
linear
regression
line
5
10
15
Sales
25
20
5
10
15
Sales
25
15
5
10
Sales
20
What is Statistical Learning?
0
10
20
30
40
50
0
20
Radio
40
60
80
100
Newspaper
Shown are Sales vs TV, Radio and Newspaper, with a blue
linear-regressi

Linear Model Selection and Regularization
Recall the linear model
Y =
0
+
1 X1
+ +
p Xp
+ .
In the lectures that follow, we consider some approaches for
extending the linear model framework. In the lectures
covering Chapter 7 of the text, we generalize

Ch4 LDA, QDA, ROC curves
Supplemental Notes
Discriminant Analysis
We want to be able to use knowledge of labeled
data (i.e., those whose group membership is
know) in order to classify the group membership
of unlabeled data
We previously consi

Cross-validation and the Bootstrap
In the section we discuss two resampling methods:
cross-validation and the bootstrap.
1 / 44
Cross-validation and the Bootstrap
In the section we discuss two resampling methods:
cross-validation and the bootstrap.
The