{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

linearSVM

# linearSVM - Lecture 20 Support Vector Machines Outline ¯...

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Lecture 20: Support Vector Machines Outline ¯ Discriminative learning of classiﬁers. Learning a decision boundary. Issue: generalization. ¯ Linear Support Vector Machine (SVM) classiﬁer. Margin and generalization. Training of linear SVM. Linear Classiﬁcation ¯ Binary classiﬁcation problem: we assign labels Ý ¾ ½ ½ to input data Ü. × Ò´Û ¡ Ü · Û¼µ and its decision surface is a ¼. hyperplane deﬁned by Û ¡ Ü · Û¼ ¯ Linear classiﬁer: Ý ¯ Linearly separable: we can ﬁnd a linear classiﬁer so that all the training examples are classiﬁed correctly. Ý Û ¡ Ü · Û¼ ¼ ½ Ò Perceptrons ¯ Find line that separates input patterns so that output Ó side, Ó ½ on other, and these match target values Ý Ó´Üµ × Ò´Û ¡ Ü · Û¼µ ·½ on one Ý ´Üµ rewrite – for every training example : Ý ´Û ¡ Ü · Û¼µ ¼ ¯ We can adjust weights Û Û¼ by Perceptron learning rule, which guarantees to converge to the correct solution in the linear separable case. ¯ Problem: which solution will has the best generalization? Geometrical View of Linear Classiﬁers ¯ Margin: minimal gap between classes and decision boundary. ¯ Answer: The linear decision surface with the maximal margin. Geometric Margin ¯ Some Vector Algebra: x0 x L w * Û £ Any two points Ü½ and Ü¾ lying in Ä, we have Û ¡ ´Ü½ Ü¾µ Û Û is the unit vector normal to the surface of Ä. Any point Ü¼ in Ä, Û ¡ Ü¼ Û¼. The signed distance of Ü to Ä is given by ½ Û £ ¡ ´Ü Ü¼ µ Û ¯ Geometric margin of ´Ü Ý µ w.r.t Ä: ­ ¼, which implies ´Û ¡ Ü · Û¼µ ½ Ý Û ´Û ¡ Ü · Û¼µ. ¯ Geometric margin of ´Ü Ý µÒ ½ w.r.t Ä: Ñ Ò ­ . Linear SVM Classiﬁer ¯ Linear SVM maximizes the geometric margin of training dataset: Ñ ÜÛ Û¼ ½ × Ø Ý Û ´Û ¡ Ü · Û¼ µ (1) ½ Ò ¯ For any solution satisfying the constraints, any positively scaled multiple satisﬁes them too. So arbitrarily setting Û linear SVM as: (Ñ Ò Ü ¸ Ñ Ò ½ ¾ Ü ¾) ½ Ñ ÒÛ Û¼ ¾ Û ¾ × Ø Ý ´Û ¡ Ü · Û¼µ ½ ½ , we can formulate (2) ½ Ò ¯ With this setting, we deﬁne a margin around the linear decision boundary with thickness ½ Û. Solution to Linear SVM ¯ We can convert the contrained minimization to an unconstrained optimization problem by representing the constraints as penality terms: ÑÒ ÛÛ ¼ ½ Û ¾ · Ô Ò Ð ØÝ Ø ÖÑ ¾ ¯ For data ´Ü Ý µ, use the following penality term: ¼ Ý ´Û ¡ Ü · Û¼µ ½ ÓØ ÖÛ × ½ Ñ Ü « ´½ Ý Û¼ · Û ¡ Ü µ « ¼ ¯ Rewrite the minimization problem Ò ½ ÑÛ Ò Û ¾· Ñ Ü « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼¾ ½« ¼ Ò ½ ÑÛ Ñ Ü Ò Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼« ¼ ¾ ½ ¯ « ’s are called the Lagrange multipliers. (3) Solution to Linear SVM (cont’d) ¯ We can swap ’max’ and ’min’: Ò ½ ÑÛ Ñ Ü Ò Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Û ¼« ¼ ¾ ½ Ò ½ Û ¾· « ´½ Ý Û¼ · Û ¡ Ü µ Ñ Ü ÛÛ ÑÒ ¼¾ « ¼ ½ Â ßÞ ´Û (4) µ Û¼ « ¯ We ﬁrst minimize Â ´Û Û¼ «µ w.r.t Û Û¼ for any ﬁxed setting of the Lagrange multipliers: Û Û¼ Â ´Û Û¼ «µ Â ´Û Û¼ «µ Ò Û ½ Ò ½ «ÝÜ «Ý ¼ ¼ (5) (6) Solution to Linear SVM (cont’d) ¯ Substitute (5) and (6) back to Â ´Û Û¼ «µ: ½ Ñ Ü Û ÛÒ Ñ Û ¾· ¼¾ « ¼ È Ñ¼ Ü « «Ý Ò ¼ ½ « ¾ ½ Ò ½ « ´½ Ý Û¼ · Û ¡ Ü µ Â Ò ½ ßÞ ´Û (7) µ Û¼ « Ý Ý « « ´Ü ¡ Ü µ ¯ Finally, we transform the original linear SVM training to a quadratic programming problem (7), which has the unique optimal solution. ¯ We can ﬁnd the optimal setting of the Lagrange multipliers « , then solve the optimal weights Û Û¼ . ¯ Essentially, we transform the primal problem to its dual form. Why should we do this? Summary of Linear SVM ¯ Binary and linear separable classﬁcation. ¯ Linear classiﬁer with maximal margin. ¯ Training SVM by maximizing Ò subject to « ¼ and ¯ Weights Û Ò È ½ « ¾ ½ Ò È «Ý ½ ¼. Ý Ý « « ´Ü ¡ Ü µ ½« Ý Ü . ¯ Only a small subset of « ’s will be nonzero and the corresponding data Ü ’s are called support vectors. ¯ Prediction on a new example Ü is the sign of Û¼ · Ü ¡ Û Û¼ · Ü ¡ ´ Ò ½ «ÝÜµ Û¼ · ¾ËÎ « Ý ´Ü ¡ Ü µ ...
View Full Document

{[ snackBarMessage ]}

Ask a homework question - tutors are online