K nearest neighbor classification K nearest neighbors KNN is a nonparametric

K nearest neighbor classification k nearest neighbors

This preview shows page 9 - 20 out of 57 pages.

K nearest neighbor classification K nearest neighbors (KNN) is a nonparametric method for classification. Classify unlabeled sample x * given training data D = { ( x 1 , z 1 ) , . . . , ( x n , z n ) } , and similarity function f ( x , x * ): Find the most similar K samples to x * from D according to f ( · , · ): these are the K nearest neighbors to x * Set ˆ z * to the most common class label from the K nearest neighbors COS424/SML 302 Classification methods February 20, 2019 9 / 57
Image of page 9

Subscribe to view the full document.

K nearest neighbors: example Classify sample as male or female based on height and number of siblings. Let the training set be our class survey data D ; let the distance metric be Euclidean distance : f ( x , x 0 ) = v u u t p X j =1 ( x j - x 0 j ) 2 . We can start by choosing number of nearest neighbors K = 1. COS424/SML 302 Classification methods February 20, 2019 10 / 57
Image of page 10
Example: predicting gender from height, siblings 0 2 4 6 60 65 70 75 80 85 Height (inches) Siblings as.factor(sex) 0 1 COS424/SML 302 Classification methods February 20, 2019 11 / 57
Image of page 11

Subscribe to view the full document.

K nearest neighbors: example with K=1 0 1 2 3 4 5 7 57 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 77 84 Sampled height (inches) Siblings 0.00 0.25 0.50 0.75 1.00 round(sex) factor(roun 0 1 accuracy = 0 . 80 How should you choose the parameter K ? COS424/SML 302 Classification methods February 20, 2019 12 / 57
Image of page 12
K nearest neighbors: example with K=3 0 1 2 3 4 5 7 57 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 77 84 Sampled height (inches) Siblings 0.00 0.25 0.50 0.75 1.00 round(sex) factor(roun 0 1 accuracy = 0 . 86 COS424/SML 302 Classification methods February 20, 2019 13 / 57
Image of page 13

Subscribe to view the full document.

Kernelized classifiers In the last lecture, we used kernels to project our data to a high dimensional feature space . We can use a simple classifier in this space. Consider the relationship between a high-dimension linear classifier and a two-dimension linear classifier: We first map the input two-dimensional space x up to a high-dimensional feature space; we fit a hyperplane in feature space (linear classifier); the hyperplane projected back onto the two-dimensional input space is not necessarily linear Ex: polynomial kernel COS424/SML 302 Classification methods February 20, 2019 14 / 57
Image of page 14
Recall: Kernels for classification One dimensional feature space and linear kernel: X COS424/SML 302 Classification methods February 20, 2019 15 / 57
Image of page 15

Subscribe to view the full document.

Recall: Kernels for classification Use φ ( x ) = [ x , x 2 ] to project up to 2D feature space: X X 2 COS424/SML 302 Classification methods February 20, 2019 16 / 57
Image of page 16
Recall: Kernels for classification Linear classifier is successful in this space: X X 2 COS424/SML 302 Classification methods February 20, 2019 17 / 57
Image of page 17

Subscribe to view the full document.

Recall: Kernels for classification The linear classifier projected down to the one-dimensional space: X COS424/SML 302 Classification methods February 20, 2019 18 / 57
Image of page 18
Kernelized Machine What classifiers can be kernelized? All feature vectors can be kernelized ; not all classifiers are able to exploit the kernel trick . If μ d ∈ X is a set of d = 1 : D centroids , or points in the feature space, and κ ( x , x 0 ) is a kernel, we can write: φ ( x ) = [ κ ( x , μ 1 ) , κ ( x , μ 2 ) , . . . , κ ( x , μ D )] Here, φ ( x ) is called a kernelized feature vector .
Image of page 19

Subscribe to view the full document.

Image of page 20
  • Spring '09
  • Machine Learning, K-nearest neighbor algorithm, Support vector machine, Statistical classification

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes