ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

This preview shows pages 1–6. Sign up to view the full content.

1 Chapter 4 Eﬃcient Likelihood Estimation and Related Tests 1. Maximum likelihood and eﬃcient likelihood estimation 2. Likelihood ratio, Wald, and Rao (or score) tests 3. Examples 4. Consistency of Maximum Likelihood Estimates 5. The EM algorithm and related methods 6. Nonparametric MLE 7. Limit theory for the statistical agnostic: P/ ∈P

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2
Chapter 4 Eﬃcient Likelihood Estimation and Related Tests 1 Maximum likelihood and eﬃcient likelihood estimation We begin with a brief discussion of Kullback - Leibler information . Defnition 1.1 Let P bea probability measure, and let Q bea sub-probability measure on ( X , A ) with densities p and q with respect to a sigma-Fnite measure µ ( µ = P + Q always works). Thus P ( X )=1 and Q ( X ) 1. Then the Kullback - Leibler information K ( P, Q )is K ( P, Q ) E P ½ log p ( X ) q ( X ) ¾ . (1) Lemma 1.1 ±ora probability measure Q and a (sub-)probability measure Q , the Kullback-Leibler information K ( P, Q )is always well-deFned, and K ( P, Q ) ½ [0 , ] always =0 if and only if Q = P. ProoF. Now K ( P, Q )= ½ log1 = 0 if P = Q, log M> 0i f P = MQ, M > 1 . If P 6 = MQ , then Jensen’s inequality is strict and yields K ( P, Q E P µ log q ( X ) p ( X ) > log E P µ q ( X ) p ( X ) = log E Q 1 [ p ( X ) > 0] ≥− . 2 Now we need some assumptions and notation. Suppose that the model P is given by P = { P θ : θ Θ } . 3

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4 CHAPTER 4. EFFICIENT LIKELIHOOD ESTIMATION AND RELATED TESTS We will impose the following hypotheses about P : Assumptions: A0. θ 6 = θ implies P θ 6 = P θ . A1. A ≡{ x : p θ ( x ) > 0 } does not depend on θ . A2. P θ has density p θ with respect to the σ Fnite measure µ and X 1 ,...,X n are i.i.d. P θ 0 P 0 . Notation: L ( θ ) L n ( θ ) L ( θ | X ) n Y i =1 p θ ( X i ) , l ( θ )= l ( θ | X ) l n ( θ ) log L n ( θ n X i =1 log p θ ( X i ) , l ( B ) l ( B | X ) l n ( B )= sup θ B l ( θ | X ) . Here is a preliminary result which motivates our deFnition of the maximum likelihood estimator. Theorem 1.1 If A0 - A2 hold, then for θ 6 = θ 0 1 n log µ L n ( θ 0 ) L n ( θ ) = 1 n n X i =1 log p θ 0 ( X i ) p θ ( X i ) a.s. K ( P θ 0 ,P θ ) > 0 , and hence P θ 0 ( L n ( θ 0 | X ) >L n ( θ | X )) 1a s n →∞ . Proof. The Frst assertion is just the strong law of large numbers; note that E θ 0 log p θ 0 ( X ) p θ ( X ) = K ( P θ 0 θ ) > 0 by lemma 1.1 and A0. The second assertion is an immediate consequence of the Frst. 2 Theorem 1.1 motivates the following deFnition. DeFnition 1.2 The value b θ = b θ n of θ which maximizes the likelihood L ( θ | X ), if it exists and is unique, is the maximum likelihood estimator (MLE) of θ .Thu s L ( b θ L (Θ) or l ( b θ n l (Θ). Cautions: b θ n may not exist. b θ n may exist, but may not be unique. Note that the deFnition depends on the version of the density p θ which is selected; since this is not unique, di±erent versions of p θ lead to di±erent MLE’s
1. MAXIMUM LIKELIHOOD AND EFFICIENT LIKELIHOOD ESTIMATION 5 When Θ R d , the usual approach to fnding b θ n is to solve the likelihood (or score ) equations ˙ l ( θ | X ) ˙ l n ( θ )=0 ; (2) i.e.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 04/14/2010 for the course STATS 610 taught by Professor Moulib during the Fall '09 term at University of Michigan.

Page1 / 40

ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online