This preview shows pages 1–9. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Regularization Networks 9.520 Class 17, 2003 Tomaso Poggio Plan Radial Basis Functions and their extensions Additive Models Regularization Networks Dual Kernels Conclusions About this class We describe a family of regularization techniques based on radial kernels K and called RBFs. We introduce RBF extensions such as Hyper Basis Functions and characterize their relation with other techniques includ ing MLPs and splines. Radial Basis Functions Radial Basis Functions, as MLPs, have the universal ap proximation property. Theorem: Let K be a Radial Basis Function function and I i the ndimensional cube [0 , 1] n . Then finite sums of the form f ( x ) = N X i =1 c i K ( x x i ) are dense in C [ I i ]. In other words, given a function h C [ I i ] and &gt; 0, there is a sum, f ( x ), of the above form, for which:  f ( x ) h ( x )  &lt; for all x I n . Notice that RBF correspond to RKHS defined on an infi nite domain. Notice also that RKHS do not in general have the same approximation property: RKHS generated by a K with an infinite countable number of strictly positive eigenvalues are dense in L 2 but not necessarily in C ( X ), though they can be embedded in C ( X ). Density of a RKHS on a bounded domain (the nonRBF case) We first ask under which condition is a RKHS dense in L 2 ( X, ). 1. when L K is strictly positive the RKHS is infinite dimensional and dense in L 2 ( X, ). 2. in the degenerate case the RKHS is finite dimensional and not dense in L 2 ( X, ). 3. in the conditionally strictly positive case the RKHS is not dense in L 2 ( X, ) but when completed with a finite number of polynomials of appropriate degree can be made to be dense in L 2 ( X, ). Density of a RKHS on a bounded domain (cont) Density of RKHS defined on a compact domain X in C ( X ) (in the sup norm) is a trickier issue that has been answered very recently by Zhou (in preparation). It is however guaranteed for radial kernels K for K continuous and integrable, if density in L 2 ( X, ) holds (with X the infinite domain). These are facts for radial kernels and unrelated to RKHS properties span K ( x y ) : y R n is dense in L 2 ( R n ) iff the Fourier transform of K goes not vanish on set of positive Lebesque measure (N. Wiener). span K ( x y ) : y R n is dense in C ( R n ) (topology of uniform convergence) if K C ( R n ), K L 1 ( R n ). Some good properties of RBF Well motivated in the framework of regularization theory; The solution is unique and equivalent to solving a linear system; Degree of smoothness is tunable (with ); Universal approximation property; Large body of applied math literature on the subject; Interpretation in terms of neural networks (?!); Biologically plausible; Simple interpretation in terms of smooth lookup table ; Similar to other nonparametric techniques, such as nearest neigh bor and kernel regression (see end of this class). Some notsogood properties of RBF...
View
Full
Document
This note was uploaded on 11/11/2011 for the course BIO 9.07 taught by Professor Ruthrosenholtz during the Spring '04 term at MIT.
 Spring '04
 RuthRosenholtz

Click to edit the document details