Now the set of features for one sample is one row of the kernel matrix κ x x 1

# Now the set of features for one sample is one row of

• Notes
• 49

This preview shows page 31 - 40 out of 49 pages.

Now the set of features for one sample is one row of the kernel matrix: [ κ ( x , x 1 ) , . . . , κ ( x , x n )] . COS 424/SML 302 Features and Kernels February 18, 2019 31 / 49

Subscribe to view the full document.

Kernel function properties A kernel function may or may not satisfy these two properties: (symmetric) x , x 0 ∈ X , κ ( x , x 0 ) = κ ( x 0 , x ), (non-negative) x , x 0 ∈ X , κ ( x , x 0 ) 0 . A kernel with these properties will loosely have the interpretation as a similarity quantification between the two samples. COS 424/SML 302 Features and Kernels February 18, 2019 32 / 49
Example: Kernels for classification Let’s say we have n scalar samples (i.e., a one dimensional input space) and a linear classifier. X Can we separate the two classes? COS 424/SML 302 Features and Kernels February 18, 2019 33 / 49

Subscribe to view the full document.

Example: Kernels for classification Kernel solution : project features to a higher dimension, and use a linear classifier in projected feature space . Let φ ( x i ) = [ x i , x 2 i ]; then κ ( x i , x j ) = φ ( x i ) T φ ( x j ) = x i x j + x 2 i x 2 j . X X 2 COS 424/SML 302 Features and Kernels February 18, 2019 34 / 49
Example: Kernels for classification X X 2 If we project to x 2 , will these samples be linearly separable? COS 424/SML 302 Features and Kernels February 18, 2019 35 / 49

Subscribe to view the full document.

Useful kernels Let’s talk about five useful types of kernels: Linear kernel Gaussian kernel Mercer kernels String kernel Fisher kernels COS 424/SML 302 Features and Kernels February 18, 2019 36 / 49
Linear kernels Letting φ ( x ) = x , we get the linear kernel , defined by the inner product between the two feature vectors: κ ( x , x 0 ) = x T x 0 The dimension of the feature space D of a linear kernel is the dimension of the input space X , or the number of features of each sample A linear kernel is useful when it is not necessary to perform an analysis in an alternative feature space (e.g., bag-of-words representations) COS 424/SML 302 Features and Kernels February 18, 2019 37 / 49

Subscribe to view the full document.

Gaussian Kernels The Gaussian kernel , also known as the squared exponential kernel (SE) or radial basis function (RBF), is defined by κ ( x , x 0 ) = σ 2 exp - 1 2 p X j = 1 1 2 j ( x j - x 0 j ) 2 This kernel has parameters σ 2 (output variance), and j (characteristic length scale), for feature j . Ex: height ( x 1 ); shoe size ( x 2 ) X 1 X 2 COS 424/SML 302 Features and Kernels February 18, 2019 38 / 49
Gaussian kernels: the kernel trick The dimension of the feature space D of a Gaussian kernel is | D | = : we can write this kernel as an infinite arithmetic series κ ( x , x 0 ) = φ ( x ) T φ ( x 0 ) = X j =1 φ j ( x ) φ j ( x 0 ) = exp - 1 2 p X j =1 1 σ 2 j ( x j - x 0 j ) 2 We never explicitly represent each sample in infinite feature space . Instead, we compute the kernel in this infinite dimensional feature space with the kernel function; the projection is implicit. Kernel trick Computations are O ( n × n ), but feature space has arbitrarily high dimension. Is every data set linearly separable in | D | = ?

Subscribe to view the full document.

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes