Vector Support and Kernel Machines
Nello Cristianini BIOwulf Technologies [e-mail] http://www.support-vector.net/tutorial.html
ICML 2001
A Little History
z
z
z
z
SVMs introduced in COLT-92 by Boser, Guyon, Vapnik. Greatly developed ever since. Initially popularized in the NIPS community, now an important and active field of all Machine Learning research. Special issues of Machine Learning Journal, and Journal of Machine Learning Research. Kernel Machines: large class of learning algorithms, SVMs a particular instance.
www.support-vector.net
A Little History
z z z z
z
z
Annual workshop at NIPS Centralized website: www.kernel-machines.org Textbook (2000): see www.support-vector.net Now: a large and diverse community: from machine learning, optimization, statistics, neural networks, functional analysis, etc. etc Successful applications in many fields (bioinformatics, text, handwriting recognition, etc) Fast expanding field, EVERYBODY WELCOME ! -
www.support-vector.net
Preliminaries
z
z
z
Task of this class of algorithms: detect and exploit complex patterns in data (eg: by clustering, classifying, ranking, cleaning, etc. the data) Typical problems: how to represent complex patterns; and how to exclude spurious (unstable) patterns (= overfitting) The first is a computational problem; the second a statistical problem.
www.support-vector.net
Very Informal Reasoning
z
z
The class of kernel methods implicitly defines the class of possible patterns by introducing a notion of similarity between data Example: similarity between documents
z z z
By length By topic By language
z
Choice of similarity Choice of relevant features
www.support-vector.net
More formal reasoning
z
z
z
z
Kernel methods exploit information about the inner products between data items Many standard algorithms can be rewritten so that they only require inner products between data (inputs) Kernel functions = inner products in some feature space (potentially very complex) If kernel given, no need to specify what features of the data are being used
www.support-vector.net
Just in case
z
Inner product between vectors
i x, z = xzi
z
Hyperplane:
i
x
w, x + b = 0
x o x
x
w
o x
x
b
o o o o
x
www.support-vector.net
Overview of the Tutorial
z
z z z z z
Introduce basic concepts with extended example of Kernel Perceptron Derive Support Vector Machines Other kernel based algorithms Properties and Limitations of Kernels On Kernel Alignment On Optimizing Kernel Alignment
www.support-vector.net
Parts I and II: overview
z z z z z
Linear Learning Machines (LLM) Kernel Induced Feature Spaces Generalization Theory Optimization Theory Support Vector Machines (SVM)
www.support-vector.net
Modularity
z
IMPORTANT CONCEPT
Any kernel-based learning algorithm composed of two modules:
A general purpose learning machine A problem specific kernel function
z z
z
Any K-B algorithm can be fitted with any kernel Kernels themselves can be constructed in a modular way Great for software engineering (and for analysis)
www.support-vector.net
1-Linear Learning Machines
z
z
Simplest case: classification. Decision function is a hyperplane in input space The Perceptron Algorithm (Rosenblatt, 57) Useful to analyze the Perceptron algorithm, before looking at SVMs and Kernel Methods in general
z
www.support-vector.net
Basic Notation
z z z z z z z
Input space Output space Hypothesis Real-valued: Training Set Test error Dot product
x X y Y = {1,+1} h H f:X R S = {( x1, y1),...,( xi, yi ),....} x, z
www.support-vector.net
Perceptron
z
Linear Separation of the input space
x x o x x
f ( x ) = w, x + b h( x ) = sign( f ( x ))
w
o x
x
b
o o o o
x
www.support-vector.net
Perceptron Algorithm
Update rule (ignoring threshold): if yi( wk, xi ) 0 then
z
x x o x x
wk + 1 wk + yixi k k +1
w
o x
x
b
o o o o
x
www.support-vector.net
Observations
z
Solution is a linear combination of training points w = i y ix i
i 0
z z
Only used informative points (mistake driven) The coefficient of a point in combination reflects its difficulty
www.support-vector.net
Observations - 2
z
Mistake bound:
2
o
x
R M
z z
g
x x x
x
x o x
o o o o
coefficients are non-negative possible to rewrite the algorithm using this alternative representation
www.support-vector.net
Dual Representation
IMPORTANT CONCEPT
The decision function can be re-written as follows:
w = iyixi
f (x) = w, x + b = iyi xi,x + b
www.support-vector.net
Dual Representation
z
And also the update rule can be rewritten as follows: if
z
yi
3 y x , x + b8 0
jj j i
then
i i +
z
Note: in dual representation, data appears only inside dot products
www.support-vector.net
Duality: First Property of SVMs
z
z
DUALITY is the first feature of Support Vector Machines SVMs are Linear Learning Machines represented in a dual fashion
f (x) = w, x + b = iyi xi,x + b
z
Data appear only within dot products (in decision function and in training algorithm)
www.support-vector.net
Limitations of LLMs
Linear classifiers cannot deal with
z z
Non-linearly separable data Noisy data + this formulation only deals with vectorial data
z
www.support-vector.net
Non-Linear Classifiers
z
One solution: creating a net of simple linear classifiers (neurons): a Neural Network (problems: local minima; many parameters; heuristics needed to train; etc) Other solution: map data into a richer feature space including non-linear features, then use a linear classifier
www.support-vector.net
z
Learning in the Feature Space
z
Map data into a feature space where they are linearly separable x ( x )
f
o
x x o x o ox X f (o) f (o)
f (x) f (x) f (x) f (o) f (o)
www.support-vector.net
f (x) F
Problems with Feature Space
Working in high dimensional feature spaces solves the problem of expressing complex functions BUT: There is a computational problem (working with very large vectors) And a generalization theory problem (curse of dimensionality)
z z z
www.support-vector.net
Implicit Mapping to Feature Space
We will introduce Kernels:
z
z
z
Solve the computational problem of working with many dimensions Can make it possible to use infinite dimensions efficiently in time / space Other advantages, both practical and conceptual
www.support-vector.net
Kernel-Induced Feature Spaces
z
In the dual representation, the data points only appear inside dot products:
f (x) = iyi (xi),(x) + b
z
The dimensionality of space F not necessarily important. May not even know the map
www.support-vector.net
Kernels
z
IMPORTANT CONCEPT
A function that returns the value of the dot product between the images of the two arguments
K ( x1, x 2 ) = ( x1), ( x 2 )
z
Given a function K, it is possible to verify that it is a kernel
www.support-vector.net
Kernels
z
One can use LLMs in a feature space by simply rewriting it in dual representation and replacing dot products with kernels:
x1, x 2 K ( x1, x 2 ) = ( x1), ( x 2 )
www.support-vector.net
The Kernel Matrix
z
IMPORTANT CONCEPT
(aka the Gram matrix):
K(1,1) K(2,1) K(1,2) K(2,2) K(1,3) K(2,3) K(1,m) K(2,m)
K=
K(m,3) K(m,m) K(m,1) K(m,2)
www.support-vector.net
The Kernel Matrix
z z
z
z
The central structure in kernel machines Information bottleneck: contains all necessary information for the learning algorithm Fuses information about the data AND the kernel Many interesting properties:
www.support-vector.net
Mercers Theorem
z
z
The kernel matrix is Symmetric Positive Definite Any symmetric positive definite matrix can be regarded as a kernel matrix, that is as an inner product matrix in some space
www.support-vector.net
More Formally: Mercers Theorem
z
Every (semi) positive definite, symmetric function is a kernel: i.e. there exists a mapping
such that it is possible to write:
K ( x1, x 2 ) = ( x1), ( x 2 )
Pos. Def.
f L2
www.support-vector.net
I
K ( x , z ) f ( x ) f (z )dxdz 0
Mercers Theorem
z
Eigenvalues expansion of Mercers Kernels:
K ( x1, x 2 ) = i( x1)i( x 2 ) i
i
z
That is: the eigenfunctions act as features !
www.support-vector.net
Examples of Kernels
z
Simple examples of kernels are:
K ( x, z ) = x, z K ( x, z ) = e
d
x z 2 / 2
www.support-vector.net
Example: Polynomial Kernels
x = ( x1, x 2 ); z = ( z1, z 2 ); x, z
2
= ( x 1 z1 + x 2 z 2 ) 2 =
22 = x12 z12 + x2 z2 + 2 x1z1 x 2 z 2 = 2 2 = ( x12 , x2 , 2 x1 x 2 ),( z12 , z2 , 2 z1z 2 ) =
= ( x ), ( z )
www.support-vector.net
Example: Polynomial Kernels
www.support-vector.net
Example: the two spirals
z
Separated by a hyperplane in feature space (gaussian kernels)
www.support-vector.net
Making Kernels
z
IMPORTANT CONCEPT
z z z z z
The set of kernels is closed under some operations. If K, K are kernels, then: K+K is a kernel cK is a kernel, if c>0 aK+bK is a kernel, for a,b >0 Etc etc etc can make complex kernels from simple ones: modularity !
www.support-vector.net
Second Property of SVMs:
SVMs are Linear Learning Machines, that Use a dual representation AND Operate in a kernel induced feature space f (x) = iyi (xi),(x) + b (that is: is a linear function in the feature space implicitely defined by K)
z z
www.support-vector.net
Kernels over General Structures
z
z
Haussler, Watkins, etc: kernels over sets, over sequences, over trees, etc. in Applied text categorization, bioinformatics, etc
www.support-vector.net
A bad kernel
z
would be a kernel whose kernel matrix is mostly diagonal: all points orthogonal to each other, no clusters, no structure 1 0 0 0 1 0 0 0 1 0 1 0 0
www.support-vector.net
No Free Kernel
z
IMPORTANT CONCEPT
If mapping in a space with too many irrelevant features, kernel matrix becomes diagonal Need some prior knowledge of target so choose a good kernel
z
www.support-vector.net
Other Kernel-based algorithms
z
Note: other algorithms can use kernels, not just LLMs (e.g. clustering; PCA; etc). Dual representation often possible (in optimization problems, by Representers theorem).
www.support-vector.net
%5($.
www.support-vector.net
NEW TOPIC
The Generalization Problem
z
The curse of dimensionality: easy to overfit in high dimensional spaces
(=regularities could be found in the training set that are accidental, that is that would not be found again in a test set)
z
The SVM problem is ill posed (finding one hyperplane that separates the data: many such hyperplanes exist) Need principled way to choose the best possible hyperplane
z
www.support-vector.net
The Generalization Problem
z
z
z
Many methods exist to choose a good hyperplane (inductive principles) Bayes, statistical learning theory / pac, MDL, Each can be used, we will focus on a simple case motivated by statistical learning theory (will give the basic SVM)
www.support-vector.net
Statistical (Computational) Learning Theory
z
z
z
Generalization bounds on the risk of overfitting (in a p.a.c. setting: assumption of I.I.d. data; etc) Standard bounds from VC theory give upper and lower bound proportional to VC dimension VC dimension of LLMs proportional to dimension of space (can be huge)
www.support-vector.net
Assumptions and Definitions
z z z
z
z
distribution D over input space X train and test points drawn randomly (I.I.d.) from D training error of h: fraction of points in S misclassifed by h test error of h: probability under D to misclassify a point x VC dimension: size of largest subset of X shattered by H (every dichotomy implemented)
www.support-vector.net
VC Bounds
~ VC = O m
VC = (number of dimensions of X) +1 Typically VC >> m, so not useful Does not tell us which hyperplane to choose
www.support-vector.net
Margin Based Bounds
= = ~ (R / )2 O m y if ( x min i f i)
Note: also compression bounds exist; and online bounds.
www.support-vector.net
Margin Based Bounds
z
IMPORTANT CONCEPT
(The worst case bound still holds, but if lucky (margin is large)) the other bound can be applied and better generalization can be achieved:
~ ( R / ) = O m
z z
2
Best hyperplane: the maximal margin one Margin is large is kernel chosen well
www.support-vector.net
Maximal Margin Classifier
z
z z
Minimize the risk of overfitting by choosing the maximal margin hyperplane in feature space Third feature of SVMs: maximize the margin SVMs control capacity by increasing the margin, not by reducing the number of degrees of freedom (dimension free capacity control).
www.support-vector.net
Two kinds of margin
z
Functional and geometric margin:
funct = min yif ( xi )
x
g
o
x x x
x
x o x
yif ( xi ) geom = min f
o o o o
www.support-vector.net
Two kinds of margin
www.support-vector.net
Max Margin = Minimal Norm
z
z
If we fix the functional margin to 1, the geometric margin equal 1/||w|| Hence, maximize the margin by minimizing the norm
www.support-vector.net
Max Margin = Minimal Norm
Distance between The two convex hulls
x
w, x w, x
+
+ b = +1 + b = 1
g
o
x x x
x
x o x
w,( x + x ) = 2 w 2 + ,( x x ) = w w
o o o o
www.support-vector.net
The primal problem
z
IMPORTANT STEP
Minimize: subject to:
w, w yi 4 w, xi + b 9 1
www.support-vector.net
Optimization Theory
z
z z
The problem of finding the maximal margin hyperplane: constrained optimization (quadratic programming) Use Lagrange theory (or Kuhn-Tucker Theory) Lagrangian: 1 w, w iyi w, xi + b 1" #$ ! 2
1
6
0
www.support-vector.net
From Primal to Dual
1 L( w ) = w, w iyi w, xi + b 1" #$ ! 2
1
6
i 0
Differentiate and substitute:
L =0 b L =0 w
www.support-vector.net
The Dual Problem
z
IMPORTANT STEP
z
1 W ( ) = i ijyiyj xi, xj i 2 ,j i Subject to: i 0
Maximize:
iyi = 0
i
z
The duality again ! Can use kernels !
www.support-vector.net
Convexity
z
IMPORTANT CONCEPT
z z
This is a Quadratic Optimization problem: convex, no local minima (second effect of Mercers conditions) Solvable in polynomial time (convexity is another fundamental property of SVMs)
www.support-vector.net
Kuhn-Tucker Theorem
Properties of the solution: Duality: can use kernels KKT conditions: i yi w, xi + b 1 = 0
z z
1
6
z
Sparseness: only the points nearest to the hyperplane (margin = 1) have positive weight iii
i
w = y x
z
They are called support vectors
www.support-vector.net
KKT Conditions Imply Sparseness
Sparseness: another fundamental property of SVMs
x
g
o
x x x
x
x o x
o o o o
www.support-vector.net
Properties of SVMs - Summary
9 9 9 9 9
Duality Kernels Margin Convexity Sparseness
www.support-vector.net
Dealing with noise
In the case of non-separable data in feature space, the margin distribution 2 can be optimized 2
1 R + 2 m
(
)
yi 4 w, xi + b 9 1 i
www.support-vector.net
The Soft-Margin Classifier
Minimize:
Or:
1 w, w + C i 2 i 2 1 w, w + C i 2 i yi 4 w, xi + b 9 1 i
www.support-vector.net
Subject to:
Slack Variables
1 R + 2 m
(
2
)
2
yi 4 w, xi +...