Unformatted text preview: OR360 Section 10 Notes Parameter Estimation 1 Estimation Principles Suppose a sample X1 , . . . , Xn comes from a distribution f (x, ) that depends on the unknown parameter (or vector of parameters) . We would like to estimate the true value of based on the observed data. A function of the data used to estimate is called a point estimator ^ of and is denoted by .
1 Example: If X1 , . . . , Xn comes from a distribution with mean , then = n (X1 + . . . + Xn ) ^ is an estimator for . ^ ^ Definition: An estimator is unbiased if E = . Example: above is unbiased, because: ^ E [^] = E 1 1 1 (X1 + . . . + Xn ) = (E [X1 ] + . . . + E [Xn ]) = (nE [X]) = E [X] = n n n 2 Method of Moments Recall E X k is the kth moment of X. If we know X has density f (x, ), then we can compute the moments of X as a function of , as E X k  = xk f (x, )dx . Based on a sample X1 , . . . , Xn , we can estimate the moments of X as:
n m1 = ^
i=1 Xi , n n m2 = ^
i=1 Xi2 mk = ^ n n i=1 Xik , n Solving the equations E [X] = m1 , E X 2  = m2 , . . . gives an estimate for . Example: X1 , . . . , Xn are binomial with n = 20 (known) and p unknown. Then E [X] = 20p, so the equation is: n Xi 20p = n
i=1 and solving for p gives the estimator p = ^ 1 20 n Xi i=1 n = X 20 . Example: Suppose X1 , . . . , Xn come from a gamma distribution with unknown parameters 2 (, ). Recall E [X] = and E X 2 = Var (X) + E [X]2 = 2 + 2 . So the equations are: + 2 2 2
n =
i=1 n Xi n Xi2 n =
i=1 1 OR360 Solving this for and gives:
1 Xi n i=1 n 2 Xi n i=1 n 1 Xi n i=1 n 2 Xi n i=1 n 2 2 ^ =  n Xi i=1 n 2 , = ^  n Xi i=1 n 3 Maximum Likelihood Define the likelihood function of an independent sample X1 , . . . , Xn drawn from f (x, ) by: L() = fX1 ,...,Xn (X1 , . . . , Xn ; ) = fX1 (X1 , ) fXn (Xn , ) . If the Xi are iid, then L() = n f (Xi , ). i=1 Intuitively, the likelihood represents the probability of observing the values X1 , . . . , Xn if the ^ parameter is ; we wish to find the value of that gives the highest probability. This value is then called the Maximum Likelihood Estimator or MLE.
d Find the maximum value of L() by solving d L() = 0 (and checking that it is a maximum!) In many cases, it is easier to find the maximum of log L() instead this is equivalent because log x is an increasing function of x. Example: Suppose X1 , . . . , Xn are Bernoulli with parameter , that is P {Xi = x} = x (1  )1x for x {0, 1} . To estimate , write the likelihood and take its logarithm, then set it to zero: L() = = P Xi (1  )1Xi
Xi (1  )n P Xi log L() = log Xi + log(1  ) n  Xi Taking the derivative and setting it equal to zero gives us: d Xi n  Xi log L() =  d 1 0 = (1  ) Xi  n + ^ = Xi =X. n Xi ^ Check the second order condition to verify it is a maximum at : d Xi n  Xi L() =  2  <0, d2 (1  )2 because the denominators are both positive, Xi > 0 and n > Xi so the numerators are positive as well, thus both summands are negative. Note also that X is an unbiased estimator, as noted earlier.
2 OR360 Example: Suppose we wish to estimate the gravitational constant g. Write = g. We drop an object from the Empire State Building and measure the distance it has fallen at times ti , i = 1, . . . , n. Call these distances x1 , . . . , xn . If our measurement error is normal, then 1 xi = 2 t2 + i , where i N (0, 2 ). Suppose is known what is the MLE of g? i Given the above, Xi N set the derivative to zero: L() =
i=1 1 2 2 2 ti , n , so we write down the likelihood, take its logarithm and 1 e
(xi  1 t2 )2 2 i 2 2 2 2 1 n log L() = n log = n log d log L() = d 2 2 1 2 2 
i=1 (xi  1 t2 )2 2 i 2 2 x2  i xi t2 + i 1 2 4 t 4 i  1 2 2 1 4 1  xi t2 + t i 2 2 2 i 1 1 1 xi t2  2 t4 0 =  2 i i 2 2 2 2 xi t2 i = . t4 i Example: Suppose X1 , . . . , Xn is a sample from the uniform distribution on [0, ] for unknown , so f (xi , ) = 1 for xi [0, ]. The likelihood is: L() =
1 n 0 if 0 x1 , . . . , xn otherwise ^ The graph below shows that the MLE for is = max(x1 , . . . , xn ).
0.12 0.1 0.08 L() 0.06 0.04 0.02 0 0 1 2 max(x )
i 3 4 5 6 7 8 9 10 ^ Let's check if it is unbiased. As computed in the past, P x = 1  P {max(xi ) x} = 1  P {X1 x}n = 1 
x n , so ^ E =
0 1 x n dx =  n+1 n . = (n + 1)n n+1
n+1 n ^ Thus, it is not unbiased! However, the alternative estimator = unbiased.
3 max(x1 , . . . , xn ) is ...
View
Full Document
 Summer '08
 WEBER
 Maximum likelihood, Estimation theory, µ, Xn

Click to edit the document details