lec3-svm - Machine Learning Lecture 3 Yang Yang Department...

Info icon This preview shows pages 1–26. Sign up to view the full content.

View Full Document Right Arrow Icon
Machine Learning Lecture 3 Yang Yang Department of Computer Science & Engineering Shanghai Jiao Tong University
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Support Vector Machine
Image of page 2
Linear classifiers–which line is better?
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Pick the one with the largest margin!
Image of page 4
A Good Separator X X O O O O O O X X X X X X O O
Image of page 5

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Noise in the Observations X X O O O O O O X X X X X X O O
Image of page 6
Ruling Out Some Separators X X O O O O O O X X X X X X O O
Image of page 7

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Lots of Noise X X O O O O O O X X X X X X O O
Image of page 8
Maximizing the Margin X X O O O O O O X X X X X X O O
Image of page 9

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Parameterizing the decision boundary m features
Image of page 10
Parameterizing the decision boundary “confidence” for j th data point
Image of page 11

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Distance between examples closest to the line/hyperplane margin = 2 ࠵? = 2a/ ǁ w ǁ Note: ‘a’ is arbitrary (can normalize equations by a) Maximizing the margin
Image of page 12
Support Vector Machine Solve efficiently by quadratic programming (QP) well-studied solution algorithms
Image of page 13

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Support Vector Machine Linear hyperplane defined by “support vectors” i: ( w . x i + b ) y i = 1 Moving other points a little doesn’t effect the decision boundary Only need to store the support vectors to predict labels of new points How many support vectors in linearly separable case? m+1
Image of page 14
What if data is not linearly separable?
Image of page 15

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
What if data is still not linearly separable? Allow “error” in classification Maximize margin and minimize # mistakes on training data C - tradeoff parameter Not convex 0/1 loss (doesn’t distinguish between near miss and bad mistake)
Image of page 16
What if data is still not linearly separable? Allow “error” in classification ξ j - “slack” variables (>1 if x j misclassifed) pay linear penalty if mistake C - tradeoff parameter(chosen by cross- validation) convex! Soft margin approach
Image of page 17

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Soft-margin SVM Soften the constraints: ( w . x j + b ) y j 1- ξ j j ξ j 0 j Penalty for misclassifying: C ξ j How do we recover hard margin SVM? Set C =
Image of page 18
Slack variables as Hinge loss Regularized loss Hinge loss
Image of page 19

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Hinge Loss regularization Loss: hinge loss
Image of page 20
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern