K nearest neighbor classification K nearest neighbors KNN is a nonparametric

# K nearest neighbor classification k nearest neighbors

• Notes
• 57

This preview shows page 9 - 20 out of 57 pages.

K nearest neighbor classification K nearest neighbors (KNN) is a nonparametric method for classification. Classify unlabeled sample x * given training data D = { ( x 1 , z 1 ) , . . . , ( x n , z n ) } , and similarity function f ( x , x * ): Find the most similar K samples to x * from D according to f ( · , · ): these are the K nearest neighbors to x * Set ˆ z * to the most common class label from the K nearest neighbors COS424/SML 302 Classification methods February 20, 2019 9 / 57

Subscribe to view the full document.

K nearest neighbors: example Classify sample as male or female based on height and number of siblings. Let the training set be our class survey data D ; let the distance metric be Euclidean distance : f ( x , x 0 ) = v u u t p X j =1 ( x j - x 0 j ) 2 . We can start by choosing number of nearest neighbors K = 1. COS424/SML 302 Classification methods February 20, 2019 10 / 57
Example: predicting gender from height, siblings 0 2 4 6 60 65 70 75 80 85 Height (inches) Siblings as.factor(sex) 0 1 COS424/SML 302 Classification methods February 20, 2019 11 / 57

Subscribe to view the full document.

K nearest neighbors: example with K=1 0 1 2 3 4 5 7 57 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 77 84 Sampled height (inches) Siblings 0.00 0.25 0.50 0.75 1.00 round(sex) factor(roun 0 1 accuracy = 0 . 80 How should you choose the parameter K ? COS424/SML 302 Classification methods February 20, 2019 12 / 57
K nearest neighbors: example with K=3 0 1 2 3 4 5 7 57 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 77 84 Sampled height (inches) Siblings 0.00 0.25 0.50 0.75 1.00 round(sex) factor(roun 0 1 accuracy = 0 . 86 COS424/SML 302 Classification methods February 20, 2019 13 / 57

Subscribe to view the full document.

Kernelized classifiers In the last lecture, we used kernels to project our data to a high dimensional feature space . We can use a simple classifier in this space. Consider the relationship between a high-dimension linear classifier and a two-dimension linear classifier: We first map the input two-dimensional space x up to a high-dimensional feature space; we fit a hyperplane in feature space (linear classifier); the hyperplane projected back onto the two-dimensional input space is not necessarily linear Ex: polynomial kernel COS424/SML 302 Classification methods February 20, 2019 14 / 57
Recall: Kernels for classification One dimensional feature space and linear kernel: X COS424/SML 302 Classification methods February 20, 2019 15 / 57

Subscribe to view the full document.

Recall: Kernels for classification Use φ ( x ) = [ x , x 2 ] to project up to 2D feature space: X X 2 COS424/SML 302 Classification methods February 20, 2019 16 / 57
Recall: Kernels for classification Linear classifier is successful in this space: X X 2 COS424/SML 302 Classification methods February 20, 2019 17 / 57

Subscribe to view the full document.

Recall: Kernels for classification The linear classifier projected down to the one-dimensional space: X COS424/SML 302 Classification methods February 20, 2019 18 / 57
Kernelized Machine What classifiers can be kernelized? All feature vectors can be kernelized ; not all classifiers are able to exploit the kernel trick . If μ d ∈ X is a set of d = 1 : D centroids , or points in the feature space, and κ ( x , x 0 ) is a kernel, we can write: φ ( x ) = [ κ ( x , μ 1 ) , κ ( x , μ 2 ) , . . . , κ ( x , μ D )] Here, φ ( x ) is called a kernelized feature vector .

Subscribe to view the full document.

• Spring '09
• Machine Learning, K-nearest neighbor algorithm, Support vector machine, Statistical classification

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes