ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Chapter 4 Efficient Likelihood Estimation and Related Tests 1. Maximum likelihood and efficient likelihood estimation 2. Likelihood ratio, Wald, and Rao (or score) tests 3. Examples 4. Consistency of Maximum Likelihood Estimates 5. The EM algorithm and related methods 6. Nonparametric MLE 7. Limit theory for the statistical agnostic: P/ ∈P
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2
Background image of page 2
Chapter 4 Efficient Likelihood Estimation and Related Tests 1 Maximum likelihood and efficient likelihood estimation We begin with a brief discussion of Kullback - Leibler information . Defnition 1.1 Let P bea probability measure, and let Q bea sub-probability measure on ( X , A ) with densities p and q with respect to a sigma-Fnite measure µ ( µ = P + Q always works). Thus P ( X )=1 and Q ( X ) 1. Then the Kullback - Leibler information K ( P, Q )is K ( P, Q ) E P ½ log p ( X ) q ( X ) ¾ . (1) Lemma 1.1 ±ora probability measure Q and a (sub-)probability measure Q , the Kullback-Leibler information K ( P, Q )is always well-deFned, and K ( P, Q ) ½ [0 , ] always =0 if and only if Q = P. ProoF. Now K ( P, Q )= ½ log1 = 0 if P = Q, log M> 0i f P = MQ, M > 1 . If P 6 = MQ , then Jensen’s inequality is strict and yields K ( P, Q E P µ log q ( X ) p ( X ) > log E P µ q ( X ) p ( X ) = log E Q 1 [ p ( X ) > 0] ≥− . 2 Now we need some assumptions and notation. Suppose that the model P is given by P = { P θ : θ Θ } . 3
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
4 CHAPTER 4. EFFICIENT LIKELIHOOD ESTIMATION AND RELATED TESTS We will impose the following hypotheses about P : Assumptions: A0. θ 6 = θ implies P θ 6 = P θ . A1. A ≡{ x : p θ ( x ) > 0 } does not depend on θ . A2. P θ has density p θ with respect to the σ Fnite measure µ and X 1 ,...,X n are i.i.d. P θ 0 P 0 . Notation: L ( θ ) L n ( θ ) L ( θ | X ) n Y i =1 p θ ( X i ) , l ( θ )= l ( θ | X ) l n ( θ ) log L n ( θ n X i =1 log p θ ( X i ) , l ( B ) l ( B | X ) l n ( B )= sup θ B l ( θ | X ) . Here is a preliminary result which motivates our deFnition of the maximum likelihood estimator. Theorem 1.1 If A0 - A2 hold, then for θ 6 = θ 0 1 n log µ L n ( θ 0 ) L n ( θ ) = 1 n n X i =1 log p θ 0 ( X i ) p θ ( X i ) a.s. K ( P θ 0 ,P θ ) > 0 , and hence P θ 0 ( L n ( θ 0 | X ) >L n ( θ | X )) 1a s n →∞ . Proof. The Frst assertion is just the strong law of large numbers; note that E θ 0 log p θ 0 ( X ) p θ ( X ) = K ( P θ 0 θ ) > 0 by lemma 1.1 and A0. The second assertion is an immediate consequence of the Frst. 2 Theorem 1.1 motivates the following deFnition. DeFnition 1.2 The value b θ = b θ n of θ which maximizes the likelihood L ( θ | X ), if it exists and is unique, is the maximum likelihood estimator (MLE) of θ .Thu s L ( b θ L (Θ) or l ( b θ n l (Θ). Cautions: b θ n may not exist. b θ n may exist, but may not be unique. Note that the deFnition depends on the version of the density p θ which is selected; since this is not unique, di±erent versions of p θ lead to di±erent MLE’s
Background image of page 4
1. MAXIMUM LIKELIHOOD AND EFFICIENT LIKELIHOOD ESTIMATION 5 When Θ R d , the usual approach to fnding b θ n is to solve the likelihood (or score ) equations ˙ l ( θ | X ) ˙ l n ( θ )=0 ; (2) i.e.
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 04/14/2010 for the course STATS 610 taught by Professor Moulib during the Fall '09 term at University of Michigan.

Page1 / 40

ch4 - 1 Chapter 4 Ecient Likelihood Estimation and Related...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online