2.160 System Identification, Estimation, and Learning
Lecture Notes No. 2
0
May 3, 2006
15. Maximum Likelihood
15.1 Principle
Consider an unknown stochastic process
Unknown
Stochastic
Proces
Observed data
()
N
N
y
y
y
y
,
,
,
2
1
"
=
Assume that each observed datum
is generated by an assumed stochastic process
having a PDF:
i
y
λ
πλ
2
2
1
;
,
m
x
e
x
m
f
−
−
=
(
1
)
where
is mean,
m
is variance, and
is the random variable associated with
.
x
i
y
We know that mean
and variance
m
are determined by
∑
=
=
N
i
i
y
N
m
1
1
(
∑
=
−
=
N
i
i
m
y
N
1
2
1
)
(
2
)
Let us now obtain the same parameter values,
and
m
, based on a different
principle: Maximum Likelihood.
Assuming that the
observations
are stochastically independent,
consider the joint probability associated with the
N
observations:
N
N
y
y
y
,
,
,
2
1
"
∏
=
−
−
=
N
i
m
x
N
i
e
x
x
x
m
f
1
2
1
2
2
1
,
,
;
,
"
(3)
Now, once
have been observed (have taken specific values), what
parameter values,
and
N
y
y
y
,
,
,
2
1
"
m
, provide the highest probability in
( )
N
x
x
x
m
f
"
,
,
;
,
2
1
?
In other words, what values of
and
m
are most likely the case? This means that
we maximize the following functional with respect to
m
and
:
( )
N
m
y
y
y
m
f
Max
"
,
,
;
,
2
1
,
(
4
)
1