{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

# ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

This preview shows pages 1–6. Sign up to view the full content.

1 Chapter 4 Eﬃcient Likelihood Estimation and Related Tests 1. Maximum likelihood and eﬃcient likelihood estimation 2. Likelihood ratio, Wald, and Rao (or score) tests 3. Examples 4. Consistency of Maximum Likelihood Estimates 5. The EM algorithm and related methods 6. Nonparametric MLE 7. Limit theory for the statistical agnostic: P / ∈ P

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2
Chapter 4 Eﬃcient Likelihood Estimation and Related Tests 1 Maximum likelihood and eﬃcient likelihood estimation We begin with a brief discussion of Kullback - Leibler information . Definition 1.1 Let P be a probability measure, and let Q be a sub-probability measure on ( X , A ) with densities p and q with respect to a sigma-finite measure µ ( µ = P + Q always works). Thus P ( X ) = 1 and Q ( X ) 1. Then the Kullback - Leibler information K ( P, Q ) is K ( P, Q ) E P log p ( X ) q ( X ) . (1) Lemma 1.1 For a probability measure Q and a (sub-)probability measure Q , the Kullback-Leibler information K ( P, Q ) is always well-defined, and K ( P, Q ) [0 , ] always = 0 if and only if Q = P . Proof. Now K ( P, Q ) = log1 = 0 if P = Q , log M > 0 if P = MQ, M > 1 . If P = MQ , then Jensen’s inequality is strict and yields K ( P, Q ) = E P log q ( X ) p ( X ) > log E P q ( X ) p ( X ) = log E Q 1 [ p ( X ) > 0] log1 = 0 . Now we need some assumptions and notation. Suppose that the model P is given by P = { P θ : θ Θ } . 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4 CHAPTER 4. EFFICIENT LIKELIHOOD ESTIMATION AND RELATED TESTS We will impose the following hypotheses about P : Assumptions: A0. θ = θ implies P θ = P θ . A1. A ≡ { x : p θ ( x ) > 0 } does not depend on θ . A2. P θ has density p θ with respect to the σ finite measure µ and X 1 , . . . , X n are i.i.d. P θ 0 P 0 . Notation: L ( θ ) L n ( θ ) L ( θ | X ) n i =1 p θ ( X i ) , l ( θ ) = l ( θ | X ) l n ( θ ) log L n ( θ ) = n i =1 log p θ ( X i ) , l ( B ) l ( B | X ) l n ( B ) = sup θ B l ( θ | X ) . Here is a preliminary result which motivates our definition of the maximum likelihood estimator. Theorem 1.1 If A0 - A2 hold, then for θ = θ 0 1 n log L n ( θ 0 ) L n ( θ ) = 1 n n i =1 log p θ 0 ( X i ) p θ ( X i ) a.s. K ( P θ 0 , P θ ) > 0 , and hence P θ 0 ( L n ( θ 0 | X ) > L n ( θ | X )) 1 as n → ∞ . Proof. The first assertion is just the strong law of large numbers; note that E θ 0 log p θ 0 ( X ) p θ ( X ) = K ( P θ 0 , P θ ) > 0 by lemma 1.1 and A0. The second assertion is an immediate consequence of the first. Theorem 1.1 motivates the following definition. Definition 1.2 The value θ = θ n of θ which maximizes the likelihood L ( θ | X ), if it exists and is unique, is the maximum likelihood estimator (MLE) of θ . Thus L ( θ ) = L (Θ) or l ( θ n ) = l (Θ). Cautions: θ n may not exist. θ n may exist, but may not be unique. Note that the definition depends on the version of the density p θ which is selected; since this is not unique, different versions of p θ lead to different MLE’s
1. MAXIMUM LIKELIHOOD AND EFFICIENT LIKELIHOOD ESTIMATION 5 When Θ R d , the usual approach to finding θ n is to solve the likelihood (or score ) equations ˙ l ( θ | X ) ˙ l n ( θ ) = 0 ; (2) i.e.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 40

ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online