Now the set of features for one sample is one row of the kernel matrix κ x x 1

Now the set of features for one sample is one row of

This preview shows page 31 - 40 out of 49 pages.

Now the set of features for one sample is one row of the kernel matrix: [ κ ( x , x 1 ) , . . . , κ ( x , x n )] . COS 424/SML 302 Features and Kernels February 18, 2019 31 / 49
Image of page 31

Subscribe to view the full document.

Kernel function properties A kernel function may or may not satisfy these two properties: (symmetric) x , x 0 ∈ X , κ ( x , x 0 ) = κ ( x 0 , x ), (non-negative) x , x 0 ∈ X , κ ( x , x 0 ) 0 . A kernel with these properties will loosely have the interpretation as a similarity quantification between the two samples. COS 424/SML 302 Features and Kernels February 18, 2019 32 / 49
Image of page 32
Example: Kernels for classification Let’s say we have n scalar samples (i.e., a one dimensional input space) and a linear classifier. X Can we separate the two classes? COS 424/SML 302 Features and Kernels February 18, 2019 33 / 49
Image of page 33

Subscribe to view the full document.

Example: Kernels for classification Kernel solution : project features to a higher dimension, and use a linear classifier in projected feature space . Let φ ( x i ) = [ x i , x 2 i ]; then κ ( x i , x j ) = φ ( x i ) T φ ( x j ) = x i x j + x 2 i x 2 j . X X 2 COS 424/SML 302 Features and Kernels February 18, 2019 34 / 49
Image of page 34
Example: Kernels for classification X X 2 If we project to x 2 , will these samples be linearly separable? COS 424/SML 302 Features and Kernels February 18, 2019 35 / 49
Image of page 35

Subscribe to view the full document.

Useful kernels Let’s talk about five useful types of kernels: Linear kernel Gaussian kernel Mercer kernels String kernel Fisher kernels COS 424/SML 302 Features and Kernels February 18, 2019 36 / 49
Image of page 36
Linear kernels Letting φ ( x ) = x , we get the linear kernel , defined by the inner product between the two feature vectors: κ ( x , x 0 ) = x T x 0 The dimension of the feature space D of a linear kernel is the dimension of the input space X , or the number of features of each sample A linear kernel is useful when it is not necessary to perform an analysis in an alternative feature space (e.g., bag-of-words representations) COS 424/SML 302 Features and Kernels February 18, 2019 37 / 49
Image of page 37

Subscribe to view the full document.

Gaussian Kernels The Gaussian kernel , also known as the squared exponential kernel (SE) or radial basis function (RBF), is defined by κ ( x , x 0 ) = σ 2 exp - 1 2 p X j = 1 1 2 j ( x j - x 0 j ) 2 This kernel has parameters σ 2 (output variance), and j (characteristic length scale), for feature j . Ex: height ( x 1 ); shoe size ( x 2 ) X 1 X 2 COS 424/SML 302 Features and Kernels February 18, 2019 38 / 49
Image of page 38
Gaussian kernels: the kernel trick The dimension of the feature space D of a Gaussian kernel is | D | = : we can write this kernel as an infinite arithmetic series κ ( x , x 0 ) = φ ( x ) T φ ( x 0 ) = X j =1 φ j ( x ) φ j ( x 0 ) = exp - 1 2 p X j =1 1 σ 2 j ( x j - x 0 j ) 2 We never explicitly represent each sample in infinite feature space . Instead, we compute the kernel in this infinite dimensional feature space with the kernel function; the projection is implicit. Kernel trick Computations are O ( n × n ), but feature space has arbitrarily high dimension. Is every data set linearly separable in | D | = ?
Image of page 39

Subscribe to view the full document.

Image of page 40

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes