linearSVM

linearSVM - Lecture 20: Support Vector Machines Outline ¯...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture 20: Support Vector Machines Outline ¯ Discriminative learning of classifiers. Learning a decision boundary. Issue: generalization. ¯ Linear Support Vector Machine (SVM) classifier. Margin and generalization. Training of linear SVM. Linear Classification ¯ Binary classification problem: we assign labels Ý ¾ ½ ½ to input data Ü. × Ò´Û ¡ Ü · Û¼µ and its decision surface is a ¼. hyperplane defined by Û ¡ Ü · Û¼ ¯ Linear classifier: Ý ¯ Linearly separable: we can find a linear classifier so that all the training examples are classified correctly. Ý Û ¡ Ü · Û¼ ¼ ½ Ò Perceptrons ¯ Find line that separates input patterns so that output Ó side, Ó ½ on other, and these match target values Ý Ó´Üµ × Ò´Û ¡ Ü · Û¼µ ·½ on one Ý ´Üµ rewrite – for every training example : Ý ´Û ¡ Ü · Û¼µ ¼ ¯ We can adjust weights Û Û¼ by Perceptron learning rule, which guarantees to converge to the correct solution in the linear separable case. ¯ Problem: which solution will has the best generalization? Geometrical View of Linear Classifiers ¯ Margin: minimal gap between classes and decision boundary. ¯ Answer: The linear decision surface with the maximal margin. Geometric Margin ¯ Some Vector Algebra: x0 x L w * Û £ Any two points ܽ and ܾ lying in Ä, we have Û ¡ ´Ü½ ܾµ Û Û is the unit vector normal to the surface of Ä. Any point ܼ in Ä, Û ¡ ܼ Û¼. The signed distance of Ü to Ä is given by ½ Û £ ¡ ´Ü ܼ µ Û ¯ Geometric margin of ´Ü Ý µ w.r.t Ä: ­ ¼, which implies ´Û ¡ Ü · Û¼µ ½ Ý Û ´Û ¡ Ü · Û¼µ. ¯ Geometric margin of ´Ü Ý µÒ ½ w.r.t Ä: Ñ Ò ­ . Linear SVM Classifier ¯ Linear SVM maximizes the geometric margin of training dataset: Ñ ÜÛ Û¼ ½ × Ø Ý Û ´Û ¡ Ü · Û¼ µ (1) ½ Ò ¯ For any solution satisfying the constraints, any positively scaled multiple satisfies them too. So arbitrarily setting Û linear SVM as: (Ñ Ò Ü ¸ Ñ Ò ½ ¾ Ü ¾) ½ Ñ ÒÛ Û¼ ¾ Û ¾ × Ø Ý ´Û ¡ Ü · Û¼µ ½ ½ , we can formulate (2) ½ Ò ¯ With this setting, we define a margin around the linear decision boundary with thickness ½ Û. Solution to Linear SVM ¯ We can convert the contrained minimization to an unconstrained optimization problem by representing the constraints as penality terms: ÑÒ ÛÛ ¼ ½ Û ¾ · Ô Ò Ð ØÝ Ø ÖÑ ¾ ¯ For data ´Ü Ý µ, use the following penality term: ¼ Ý ´Û ¡ Ü · Û¼µ ½ ÓØ ÖÛ × ½ Ñ Ü « ´½ Ý Û¼ · Û ¡ Ü µ « ¼ ¯ Rewrite the minimization problem Ò ½ ÑÛ Ò Û ¾· Ñ Ü « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼¾ ½« ¼ Ò ½ ÑÛ Ñ Ü Ò Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼« ¼ ¾ ½ ¯ « ’s are called the Lagrange multipliers. (3) Solution to Linear SVM (cont’d) ¯ We can swap ’max’ and ’min’: Ò ½ ÑÛ Ñ Ü Ò Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼« ¼ ¾ ½ Ò ½ Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Ñ Ü ÛÛ ÑÒ ¼¾ « ¼ ½  ßÞ ´Û (4) µ Û¼ « ¯ We first minimize  ´Û Û¼ «µ w.r.t Û Û¼ for any fixed setting of the Lagrange multipliers: Û Û¼  ´Û Û¼ «µ  ´Û Û¼ «µ Ò Û ½ Ò ½ «ÝÜ «Ý ¼ ¼ (5) (6) Solution to Linear SVM (cont’d) ¯ Substitute (5) and (6) back to  ´Û Û¼ «µ: ½ Ñ Ü Û ÛÒ Ñ Û ¾· ¼¾ « ¼ È Ñ¼ Ü « «Ý Ò ¼ ½ « ¾ ½ Ò ½ « ´½ Ý Û¼ · Û ¡ Ü µ Â Ò ½ ßÞ ´Û (7) µ Û¼ « Ý Ý « « ´Ü ¡ Ü µ ¯ Finally, we transform the original linear SVM training to a quadratic programming problem (7), which has the unique optimal solution. ¯ We can find the optimal setting of the Lagrange multipliers « , then solve the optimal weights Û Û¼ . ¯ Essentially, we transform the primal problem to its dual form. Why should we do this? Summary of Linear SVM ¯ Binary and linear separable classfication. ¯ Linear classifier with maximal margin. ¯ Training SVM by maximizing Ò subject to « ¼ and ¯ Weights Û Ò È ½ « ¾ ½ Ò È «Ý ½ ¼. Ý Ý « « ´Ü ¡ Ü µ ½« Ý Ü . ¯ Only a small subset of « ’s will be nonzero and the corresponding data Ü ’s are called support vectors. ¯ Prediction on a new example Ü is the sign of Û¼ · Ü ¡ Û Û¼ · Ü ¡ ´ Ò ½ «Ýܵ Û¼ · ¾ËÎ « Ý ´Ü ¡ Ü µ ...
View Full Document

This note was uploaded on 06/16/2011 for the course CS 5141 taught by Professor Chenenhong during the Spring '10 term at USTC.

Ask a homework question - tutors are online