svmtutorialPSB2002

svmtutorialPSB2002 - Support Vector and Kernel Methods for...

Info iconThis preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Support Vector and Kernel Methods for Pattern Recognition Nello Cristianini BIOwulf Technologies nello@support-vector.net http:// www.support-vector.net/tutorial.html PSB 2002 www.support-vector.net A Little History ! Support Vector Machines (SVM) introduced in COLT- 92 (conference on learning theory) greatly developed since then. ! Result: a class of algorithms for Pattern Recognition (Kernel Machines) ! Now: a large and diverse community, from machine learning, optimization, statistics, neural networks, functional analysis, etc. etc ! Centralized website: www.kernel-machines.org ! Textbook (2000): see www.support-vector.net
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 www.support-vector.net Basic Idea ! Kernel Methods work by embedding the data into a vector space, and by detecting linear relations in that space ! Convex Optimization, Statistical Learning Theory, Functional Analysis are the main tools www.support-vector.net Basic Idea ! “Linear relations”: can be regressions, classifications, correlations, principal components, etc. ! If the feature space chosen suitably, pattern recognition can be easy
Background image of page 2
3 www.support-vector.net General Structure of Kernel-Based Algorithms ! Two Separate Modules: Learning Module Kernel Function A learning algorithm: performs the learning In the embedding space A kernel function: takes care of the embedding www.support-vector.net Overview of the Tutorial ! Introduce basic concepts with extended example of Kernel Perceptron ! Derive Support Vector Machines ! Other kernel based algorithms (PCA;regression; clustering;…) ! Bioinformatics Applications
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
4 www.support-vector.net Just in case … ! Inner product between vectors ! Hyperplane: xz ii i , = x x o o o x x o o o x x x w b wx b , += 0 www.support-vector.net Preview ! Kernel methods exploit information about the inner products between data items ! Many standard algorithms can be rewritten so that they only require inner products between data (inputs) ! Kernel functions = inner products in some feature space (potentially very complex) ! If kernel given, no need to specify what features of the data are being used
Background image of page 4
5 www.support-vector.net Basic Notation ! Input space ! Output space ! Hypothesis ! Real-valued: ! Training Set ! Test error ! Dot product x X yY hH fX Sx y x y xz ii ∈=−+ = {,} :R {( , ),. ..,( , ),. ...} , 11 ε www.support-vector.net Basic Example: the Kernel-Perceptron ! We will introduce the main ideas of this approach by using an example: the simplest algorithm with the simplest kernel ! Then we will generalize to general algorithms and general kernels
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
6 www.support-vector.net Perceptron ! Simplest case: classification. Decision function is a hyperplane in input space ! The Perceptron Algorithm (Rosenblatt, 57) ! Useful to analyze the Perceptron algorithm, before looking at SVMs and Kernel Methods in general www.support-vector.net Perceptron ! Linear Separation of the input space x x o o o x x o o o x x x w b fx wx b hx s ignf x () , (() ) =+ =
Background image of page 6
7 www.support-vector.net Perceptron Algorithm Update rule (ignoring threshold): ! if then ywx ik i (, ) 0 ww y x kk i i + ←+ 1 1 η www.support-vector.net Observations ! Solution is a linear combination of training points !
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 69

svmtutorialPSB2002 - Support Vector and Kernel Methods for...

This preview shows document pages 1 - 8. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online