Stat841f09 - Wiki Course Notes

# 2 besides step size halving will solve this problem 3

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: vergence of the method, which means the iteration would converge only if the initial point is closed enough to the exact solution. However, in practice, choosing an appropriate initial value is really trivial, namely, it is not often to find a initial too far from the exact solution to make the iteration invalid. [2] Besides, step- size halving will solve this problem. [3] For multiclass cases: the Newton algorithm can also be expressed as an iteratively reweighted least squares algorithm, but with a vector of nondiagonal weight matrix per observation. And we can use coordinate- descent method to maximize the log- likelihood efficiently. response and a Ps e udo Code 1. 2. Set , the label associated with each observation 3. Compute . according to the equation for all 4. Compute the diagonal matrix 5. by setting . 6. . 7. If the new to . for all . value is sufficiently close to the old value, stop; otherwise go back to step 3. Comparis on with Linear Regres s ion Similaritie s 1. They are both to attempt to estimate 2. They are both have linear boundaris. (For logistic regression, we just mentioned about the case that note :For linear regression, we assume the model is linear. The boundary is or now). (linear) For logistic regression, the boundary is (linear) Diffe re nce s 1. Linear regression: 2. Logistic regression: is linear function of , is not guaranteed to fall between 0 and 1 and to sum up to 1. is a nonlinear function of , and it is guaranteed to range from 0 to 1 and to sum up to 1. Comparis on with LDA 1. 2. 3. 4. 5. 6. The linear logistic model only consider the conditional distribution . No assumption is made about . The LDA model specifies the joint distribution of and . Logistic regression maximizes the conditional likelihood of given : LDA maximizes the joint likelihood of and : . If is d- dimensional,the number of adjustable parameter in logistic regression is . The number of parameters grows linearly w.r.t dimension. If is d- dimensional,the number of adjustable parameter in LDA is...
View Full Document

Ask a homework question - tutors are online