10_RegularizationAndRidgeRegression.pdf - IAML Regularization and Ridge Regression Nigel Goddard School of Informatics Semester 1 1 12 Regularization I

# 10_RegularizationAndRidgeRegression.pdf - IAML...

• 12

This preview shows page 1 - 4 out of 12 pages.

IAML: Regularization and Ridge Regression Nigel Goddard School of Informatics Semester 1 1 / 12

Subscribe to view the full document.

Regularization I Regularization is a general approach to add a “complexity parameter” to a learning algorithm. Requires that the model parameters be continuous. (i.e., Regression OK, Decision trees not.) I If we penalize polynomials that have large values for their coefficients we will get less wiggly solutions ˜ E ( w ) = | y - Φ w | 2 + λ | w | 2 I Solution is ˆ w = (Φ T Φ + λ I ) - 1 Φ T y I This is known as ridge regression I Rather than using a discrete control parameter like M (model order) we can use a continuous parameter λ I Caution: Don’t shrink the bias term! (The one that corresponds to the all 1 feature.) 2 / 12
Regularized Loss Function I The overall cost function is the sum of two parabolic bowls. The sum is also a parabolic bowl. I The combined minimum lies on the line between the minimum of the squared error and the origin.

Subscribe to view the full document.

• Fall '12
• CharlesSutton

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern