Section10Notes - OR360 Section 10 Notes Parameter...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: OR360 Section 10 Notes Parameter Estimation 1 Estimation Principles Suppose a sample X1 , . . . , Xn comes from a distribution f (x, ) that depends on the unknown parameter (or vector of parameters) . We would like to estimate the true value of based on the observed data. A function of the data used to estimate is called a point estimator ^ of and is denoted by . 1 Example: If X1 , . . . , Xn comes from a distribution with mean , then = n (X1 + . . . + Xn ) ^ is an estimator for . ^ ^ Definition: An estimator is unbiased if E = . Example: above is unbiased, because: ^ E [^] = E 1 1 1 (X1 + . . . + Xn ) = (E [X1 ] + . . . + E [Xn ]) = (nE [X]) = E [X] = n n n 2 Method of Moments Recall E X k is the k-th moment of X. If we know X has density f (x, ), then we can compute the moments of X as a function of , as E X k | = xk f (x, )dx . Based on a sample X1 , . . . , Xn , we can estimate the moments of X as: n m1 = ^ i=1 Xi , n n m2 = ^ i=1 Xi2 mk = ^ n n i=1 Xik , n Solving the equations E [X|] = m1 , E X 2 | = m2 , . . . gives an estimate for . Example: X1 , . . . , Xn are binomial with n = 20 (known) and p unknown. Then E [X] = 20p, so the equation is: n Xi 20p = n i=1 and solving for p gives the estimator p = ^ 1 20 n Xi i=1 n = X 20 . Example: Suppose X1 , . . . , Xn come from a gamma distribution with unknown parameters 2 (, ). Recall E [X] = and E X 2 = Var (X) + E [X]2 = 2 + 2 . So the equations are: + 2 2 2 n = i=1 n Xi n Xi2 n = i=1 1 OR360 Solving this for and gives: 1 Xi n i=1 n 2 Xi n i=1 n 1 Xi n i=1 n 2 Xi n i=1 n 2 2 ^ = - n Xi i=1 n 2 , = ^ - n Xi i=1 n 3 Maximum Likelihood Define the likelihood function of an independent sample X1 , . . . , Xn drawn from f (x, ) by: L() = fX1 ,...,Xn (X1 , . . . , Xn ; ) = fX1 (X1 , ) fXn (Xn , ) . If the Xi are iid, then L() = n f (Xi , ). i=1 Intuitively, the likelihood represents the probability of observing the values X1 , . . . , Xn if the ^ parameter is ; we wish to find the value of that gives the highest probability. This value is then called the Maximum Likelihood Estimator or MLE. d Find the maximum value of L() by solving d L() = 0 (and checking that it is a maximum!) In many cases, it is easier to find the maximum of log L() instead this is equivalent because log x is an increasing function of x. Example: Suppose X1 , . . . , Xn are Bernoulli with parameter , that is P {Xi = x} = x (1 - )1-x for x {0, 1} . To estimate , write the likelihood and take its logarithm, then set it to zero: L() = = P Xi (1 - )1-Xi Xi (1 - )n- P Xi log L() = log Xi + log(1 - ) n - Xi Taking the derivative and setting it equal to zero gives us: d Xi n - Xi log L() = - d 1- 0 = (1 - ) Xi - n + ^ = Xi =X. n Xi ^ Check the second order condition to verify it is a maximum at : d Xi n - Xi L() = - 2 - <0, d2 (1 - )2 because the denominators are both positive, Xi > 0 and n > Xi so the numerators are positive as well, thus both summands are negative. Note also that X is an unbiased estimator, as noted earlier. 2 OR360 Example: Suppose we wish to estimate the gravitational constant g. Write = g. We drop an object from the Empire State Building and measure the distance it has fallen at times ti , i = 1, . . . , n. Call these distances x1 , . . . , xn . If our measurement error is normal, then 1 xi = 2 t2 + i , where i N (0, 2 ). Suppose is known what is the MLE of g? i Given the above, Xi N set the derivative to zero: L() = i=1 1 2 2 2 ti , n , so we write down the likelihood, take its logarithm and 1 e- (xi - 1 t2 )2 2 i 2 2 2 2 1 n log L() = n log = n log d log L() = d 2 2 1 2 2 - i=1 (xi - 1 t2 )2 2 i 2 2 x2 - i xi t2 + i 1 2 4 t 4 i - 1 2 2 1 4 1 - xi t2 + t i 2 2 2 i 1 1 1 xi t2 - 2 t4 0 = - 2 i i 2 2 2 2 xi t2 i = . t4 i Example: Suppose X1 , . . . , Xn is a sample from the uniform distribution on [0, ] for unknown , so f (xi , ) = 1 for xi [0, ]. The likelihood is: L() = 1 n 0 if 0 x1 , . . . , xn otherwise ^ The graph below shows that the MLE for is = max(x1 , . . . , xn ). 0.12 0.1 0.08 L() 0.06 0.04 0.02 0 0 1 2 max(x ) i 3 4 5 6 7 8 9 10 ^ Let's check if it is unbiased. As computed in the past, P x = 1 - P {max(xi ) x} = 1 - P {X1 x}n = 1 - x n , so ^ E = 0 1- x n dx = - n+1 n . = (n + 1)n n+1 n+1 n ^ Thus, it is not unbiased! However, the alternative estimator = unbiased. 3 max(x1 , . . . , xn ) is ...
View Full Document

Ask a homework question - tutors are online