COS 424SML 302 Features and Kernels 39 49 Mercer Kernels A

Cos 424sml 302 features and kernels 39 49 mercer

This preview shows page 39 - 48 out of 49 pages.

COS 424/SML 302 Features and Kernels February 18, 2019 39 / 49
Image of page 39

Subscribe to view the full document.

Mercer Kernels A Mercer kernel is defined by the inner product between two p -vectors: κ ( x , x 0 ) = φ ( x ) T φ ( x 0 ) = p X j =1 φ j ( x ) φ j ( x 0 ) . For { x 1 , . . . , x n } , we can define a Gram matrix , which is the n × n matrix K with entries K i , j = κ ( x i , x j ). The Gram matrix from a Mercer kernel is: positive semi-definite (all eigenvalues are nonnegative) symmetric (i.e., K = K T ). COS 424/SML 302 Features and Kernels February 18, 2019 40 / 49
Image of page 40
Mercer Kernels: examples Mercer kernel examples Linear kernel: φ ( x ) = x Polynomial kernels, e.g., φ ( x ) = [ x , x 2 , x 3 ] Gaussian kernel [Schoelkopf & Smola 2002] Sums of Mercer kernels, e.g., κ ( x , x 0 ) = κ 1 ( x , x 0 ) + κ 2 ( x , x 0 ) COS 424/SML 302 Features and Kernels February 18, 2019 41 / 49
Image of page 41

Subscribe to view the full document.

String Kernels As an alternative to bag-of-words representation, we might be interested in finding all substrings of any pair of strings , or sequences of characters. We can use a string kernel : x : The quick brown fox jumped ove r the lazy dog. x 0 : Yesterday I went to town and saw a dove . κ 3 ( x , x 0 ) = 2 COS 424/SML 302 Features and Kernels February 18, 2019 42 / 49
Image of page 42
String kernels: definition Let A denote an alphabet, e.g., A = { a , ..., z } Define A * = [ A 1 , A 2 , . . . , A m ], where m is the length of the longest substring to match Notation: superscripts are regular expression operators. A i is the set of all possible strings of length i with any character from alphabet A * is known as the Kleene star operator. COS 424/SML 302 Features and Kernels February 18, 2019 43 / 49
Image of page 43

Subscribe to view the full document.

String kernels: definition Basis function φ ( x ) maps a string x to a vector of length | A * | Each element is the number of times we observe substring A * j in x The string kernel is the weighted sum over all substrings in A * : κ ( x , x 0 ) = X s A * w s φ s ( x ) φ s ( x 0 ) How can we avoid computations in the dimension of the feature space? COS 424/SML 302 Features and Kernels February 18, 2019 44 / 49
Image of page 44
In practice, string kernels use a suffix tree String kernel κ ( x , x 0 ) can be computed in O ( | x | + | x 0 | ), or linear time, for weight function w , by only considering existing substrings. A suffix tree contains all possible suffixes in a string, condensed To compute κ m ( x , x 0 ), build a m level suffix tree for string x , and find all suffix matches with string x 0 This is a linear time process. Suffix Tr s = abaaba$ a b $ $ a $ b a $ a b a $ a b a $ a b a $ SufTrie(s) Edges of t letters fro Every path represent Every suffi path from Why are a How man Carl Kingsford suffix trie slides COS 424/SML 302 Features and Kernels February 18, 2019 45 / 49
Image of page 45

Subscribe to view the full document.

Specialized string kernels Customize a string kernel by setting weights w s to specific values. Specialized string kernels w s = 0 for | s | > 1: comparing the alphabet between strings (i.e., substrings of length one) w s = | s | : weight matches by the length of the matched substring w s = 0 for s outside of a dictionary: equivalent to (weighted) bag-of-words kernel COS 424/SML 302 Features and Kernels February 18, 2019 46 / 49
Image of page 46
Fisher Kernels We can construct a kernel based on an arbitrary generative model using the concept of a Fisher kernel .
Image of page 47

Subscribe to view the full document.

Image of page 48

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes