class03

class03 - Reproducing Kernel Hilbert Spaces 9.520 February...

Info iconThis preview shows pages 1–14. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypoth- esis spaces called Reproducing Kernel Hilbert Spaces (RKHS) and to derive the general solution of Tikhonov regularization in RKHS. Here is a graphical example for generalization: given a certain number of samples... f(x) x suppose this is the “true” solution... f(x) x ... but suppose ERM gives this solution! f(x) x Regularization The basic idea of regularization (originally introduced in- dependently of the learning problem) is to restore well- posedness of ERM by constraining the hypothesis space H . The direct way – minimize the empirical error subject to f in a ball in an appropriate normed functional space H – is called Ivanov regularization. The indirect way is Tikhonov regularization (which is not ERM). Ivanov regularization over normed spaces ERM finds the function in H which minimizes n 1 V ( f ( x i ) , y i ) n i =1 which in general – for arbitrary hypothesis space H – is ill-posed . Ivanov regularizes by finding the function that minimizes n 1 V ( f ( x i ) , y i ) n i =1 while satisfying 2 f H ≤ A, with · , the norm in the normed function space H Function space A function space is a space made of functions. Each function in the space can be thought of as a point. Ex- amples: 1. C [ a, b ], the set of all real-valued continuous functions in the interval [ a, b ]; 2. L 1 [ a, b ], the set of all real-valued functions whose ab- solute value is integrable in the interval [ a, b ]; 3. L 2 [ a, b ], the set of all real-valued functions square inte- grable in the interval [ a, b ] Normed space A normed space is a linear (vector) space N in which a norm is defined. A nonnegative function · is a norm iff ∀ f,g ∈ N and α ∈ IR 1. f ≥ 0 and f = 0 iff f = 0; 2. f + g ≤ f + g ; 3. αf = | α | f . Note, if all conditions are satisfied except f = 0 iff f = 0 then the space has a seminorm instead of a norm. Examples 1. A norm in C [ a, b ] can be established by defining f = max | f ( t ) | . a ≤ t ≤ b 2. A norm in L 1 [ a, b ] can be established by defining b f = | f ( t ) | dt. a 3. A norm in L 2 [ a, b ] can be established by defining b 1 / 2 f = f 2 ( t ) dt . a From Ivanov to Tikhonov regularization Alternatively, by the Lagrange multipler’s technique , Tikhonov regularization minimizes over the whole normed function space H , for a fixed positive parameter λ , the regularized functional n 1 V ( f ( x i ) , y i ) + λ f 2 H . (1) n i =1 In practice, the normed function space H that we will con- sider, is a Reproducing Kernel Hilbert Space (RKHS). Lagrange multiplier’s technique Lagrange multiplier’s technique allows the reduction of the constrained minimization problem Minimize I ( x ) subject to Φ( x ) ≤ A (for some A ) to the unconstrained minimization problem Minimize J ( x ) = I ( x ) + λ Φ( x ) (for some λ ≥ 0) Hilbert space A Hilbert space...
View Full Document

Page1 / 40

class03 - Reproducing Kernel Hilbert Spaces 9.520 February...

This preview shows document pages 1 - 14. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online