SVM - Support Vector Machines Elegant combination of...

Info iconThis preview shows pages 1–10. Sign up to view the full content.

View Full Document Right Arrow Icon
Support Vector Machines Elegant combination of statistical learning theory and machine learning – Vapnik Good empirical results Non-trivial implementation Can be slow and memory intensive Binary classifier Much current work
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
SVM Comparisons In order to have a natural way to compare and gain intuition on SVMs, we will first review Quadric/Higher Order Machines Radial Basis Function Networks
Background image of page 2
SVM Overview Non-linear mapping from input space into a higher dimensional feature space Linear decision surface (hyper-plane) sufficient in the high dimensional feature space Note that this is the same as we do with standard MLP/BP Avoid complexity of high dimensional feature space with kernel functions which allow computations to take place in the input space, while giving the power of being in the feature space Get improved generalization by placing hyper- plane at the maximum margin
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
Standard (Primal) Perceptron Algorithm Target minus Output not used. Just add (or subtract) a portion (multiplied by learning rate) of the current pattern to the weight vector If weight vector starts at 0 then learning rate can just be 1 R could also be 1 for this discussion
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Dual and Primal Equivalence Note that the final weight vector is a linear combination of the training patterns The basic decision function (primal and dual) is How do we obtain the coefficients αi
Background image of page 6
Dual Perceptron Training Algorithm Assume initial 0 weight vector
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Dual vs. Primal Form Gram Matrix: all ( xi · xj ) pairs – Done once and stored (can be large) αi : One for each pattern in the training set. Incremented each time it is misclassified, which would have led to a weight change in primal form Magnitude of αi is an indicator of effect of pattern on weights ( embedding strength ) Note that patterns on borders have large αi while easy patterns never effect the weights Could have trained with just the subset of patterns with αi > 0 (support vectors) and ignored the others Can train in dual. How about execution? Either way (dual can be efficient if support vectors are few) Would if not linearly separable. αi would keep growing. Could do early stopping or bound the αi with some maximum C, thus allowing outliers.
Background image of page 8
Feature Space and Kernel Functions Since most problems require a non-linear decision surface, we do a non-linear map Φ ( x ) = ( Φ1 ( x ), Φ2 ( x ), …, ΦN ( x )) from input space to feature space Feature space can be of very high (even infinite) dimensionality By choosing a proper kernel function/feature space, the high dimensionality can be avoided in computation but effectively used for the decision surface to solve complex problems - "Kernel Trick" A Kernel is appropriate if the matrix of all K ( x i , x j ) is positive semi- definite (has non-negative eigenvalues). Even when this is not satisfied many kernels still work in practice (sigmoid).
Background image of page 9

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 10
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 31

SVM - Support Vector Machines Elegant combination of...

This preview shows document pages 1 - 10. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online