This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypoth esis spaces called Reproducing Kernel Hilbert Spaces (RKHS) and to derive the general solution of Tikhonov regularization in RKHS. Here is a graphical example for generalization: given a certain number of samples... f(x) x suppose this is the “true” solution... f(x) x ... but suppose ERM gives this solution! f(x) x Regularization The basic idea of regularization (originally introduced in dependently of the learning problem) is to restore well posedness of ERM by constraining the hypothesis space H . The direct way – minimize the empirical error subject to f in a ball in an appropriate normed functional space H – is called Ivanov regularization. The indirect way is Tikhonov regularization (which is not ERM). Ivanov regularization over normed spaces ERM finds the function in H which minimizes n 1 V ( f ( x i ) , y i ) n i =1 which in general – for arbitrary hypothesis space H – is illposed . Ivanov regularizes by finding the function that minimizes n 1 V ( f ( x i ) , y i ) n i =1 while satisfying 2 f H ≤ A, with · , the norm in the normed function space H Function space A function space is a space made of functions. Each function in the space can be thought of as a point. Ex amples: 1. C [ a, b ], the set of all realvalued continuous functions in the interval [ a, b ]; 2. L 1 [ a, b ], the set of all realvalued functions whose ab solute value is integrable in the interval [ a, b ]; 3. L 2 [ a, b ], the set of all realvalued functions square inte grable in the interval [ a, b ] Normed space A normed space is a linear (vector) space N in which a norm is defined. A nonnegative function · is a norm iff ∀ f,g ∈ N and α ∈ IR 1. f ≥ 0 and f = 0 iff f = 0; 2. f + g ≤ f + g ; 3. αf =  α  f . Note, if all conditions are satisfied except f = 0 iff f = 0 then the space has a seminorm instead of a norm. Examples 1. A norm in C [ a, b ] can be established by defining f = max  f ( t )  . a ≤ t ≤ b 2. A norm in L 1 [ a, b ] can be established by defining b f =  f ( t )  dt. a 3. A norm in L 2 [ a, b ] can be established by defining b 1 / 2 f = f 2 ( t ) dt . a From Ivanov to Tikhonov regularization Alternatively, by the Lagrange multipler’s technique , Tikhonov regularization minimizes over the whole normed function space H , for a fixed positive parameter λ , the regularized functional n 1 V ( f ( x i ) , y i ) + λ f 2 H . (1) n i =1 In practice, the normed function space H that we will con sider, is a Reproducing Kernel Hilbert Space (RKHS). Lagrange multiplier’s technique Lagrange multiplier’s technique allows the reduction of the constrained minimization problem Minimize I ( x ) subject to Φ( x ) ≤ A (for some A ) to the unconstrained minimization problem Minimize J ( x ) = I ( x ) + λ Φ( x ) (for some λ ≥ 0) Hilbert space A Hilbert space...
View
Full
Document
 Spring '04
 RuthRosenholtz

Click to edit the document details