This preview shows pages 1–3. Sign up to view the full content.
1
: CHAPTER
Elementary
Regression Analysis
In this chapter, we shall study three methods which are capable of gener
ating estimates of statistical parameters in a wide variety of contexts. These
are the method of moments, the method of least squares and the principle of
maximum likelihood.
We shall study the methods only in relation to the simple linear regression
model; and we shall show that each entails assumptions which may be more or
less appropriate to the context in which we wish to apply the model.
In the case of the regression model, the three methods generate estimating
equations which are formally identical; but this does not justify us in taking a
casual approach to the statistical assumptions which sustain the model. To be
casual in making our assumptions is to invite the danger of misinterpretation
when the results of the estimation are in hand.
We begin with the method of moments, we shall proceed to the method
of least squares, and we shall conclude with a brief treatment of the method of
maximum likelihood.
Conditional Expectations
Let
y
be a continuously distributed random variable whose probability
density function is
f
(
y
). If we wish to predict the value of
y
without the help
of any other information, then we might take its expected value which is deﬁned
by
E
(
y
)=
Z
y
yf
(
y
)
dy.
The expected value is a socalled minimummeansquareerror (m.m.s.e.)
predictor. If
π
is the value of a prediction, then the meansquare error is given
(1)
M
=
Z
y
(
y

π
)
2
f
(
y
)
dy
=
E
'
(
y

π
)
2
“
=
E
(
y
2
)

2
πE
(
y
)+
π
2
;
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document D.S.G. POLLOCK : ECONOMETRIC THEORY
and, using the methods of calculus, it is easy to show that this quantity is
minimised by taking
π
=
E
(
y
).
Now let us imagine that
y
is statistically related to another random variable
x
whose value we have already observed. For the sake of argument, let us
assume that we know the form of the joint distribution of
x
and
y
which is
f
(
x, y
). Then the minimummeansquareerror prediction of
y
is given by the
conditional expectation
(2)
E
(
y

x
)=
Z
y
y
f
(
x, y
)
f
(
x
)
dy
wherein
(3)
f
(
x
Z
y
f
(
x, y
)
dy
is the socalled marginal distribution of
x
. We may state this proposition
formally in a way which will assist us in proving it:
(4)
Let ˆ
y
=ˆ
y
(
x
) be the conditional expectation of
y
given
x
which
is also expressed as ˆ
y
=
E
(
y

x
). Then we have
E
{
(
y

ˆ
y
)
2
}≤
E
{
(
y

π
)
2
}
, where
π
=
π
(
x
) is any other function of
x
.
Proof.
Consider
(5)
E
'
(
y

π
)
2
“
=
E
h
'
(
y

ˆ
y
)+(ˆ
y

π
)
“
2
i
=
E
'
(
y

ˆ
y
)
2
“
+2
E
'
(
y

ˆ
y
)(ˆ
y

π
)
“
+
E
'
(ˆ
y

π
)
2
“
.
In the second term, there is
(6)
E
'
(
y

ˆ
y
y

π
)
“
=
Z
x
Z
y
(
y

ˆ
y
y

π
)
f
(
x, y
)
∂y∂x
=
Z
x
‰Z
y
(
y

ˆ
y
)
f
(
y

x
)
∂y
±
y

π
)
f
(
x
)
∂x
=0
.
Here the second equality depends upon the factorisation
f
(
x, y
f
(
y

x
)
f
(
x
)
which expresses the joint probability density function of
x
and
y
as the product
of the conditional density function of
y
given
x
and the marginal density func
tion of
x
. The ﬁnal equality depends upon the fact that
R
(
y

ˆ
y
)
f
(
y

x
)
=
E
(
y

x
)

E
(
y

x
) = 0. Therefore
E
{
(
y

π
)
2
}
=
E
{
(
y

ˆ
y
)
2
}
+
E
{
y

π
)
2
}≥
E
{
(
y

ˆ
y
)
2
}
, and the assertion is proved.
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 10/27/2009 for the course MBA 1918272 taught by Professor Peter during the Fall '09 term at Aberystwyth University.
 Fall '09
 Peter

Click to edit the document details