Coursehero >> Pennsylvania >> Pittsburgh >> CS 3750

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

3750 CS Machine Learning Lecture 24 Clustering Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square CS 2750 Machine Learning Clustering Groups together similar instances in the data sample Basic clustering problem: distribute data into k different groups such that data points similar to each other are in the same group Similarity between data points is defined in terms of some distance metric (can be chosen) Clustering is useful for: Similarity/Dissimilarity analysis Analyze what data points in the sample are close to each other Dimensionality reduction High dimensional data replaced with a group (cluster) label CS 2750 Machine Learning 1 Clustering example We see data points and want to partition them into groups Which data points belong together? 3 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 3 CS 2750 Machine Learning Clustering example We see data points and want to partition them into the groups Which data points belong together? 3 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 3 CS 2750 Machine Learning 2 Clustering example We see data points and want to partition them into the groups Requires a distance measure to tell us what points are close to each other and are in the same group 3 Euclidean distance 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 3 CS 2750 Machine Learning Clustering example A set of patient cases We want to partition them into the groups based on similarities Patient # Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Age Sex 55 62 67 65 70 M M F F M Heart Rate 85 87 80 90 84 Blood pressure 125/80 130/85 126/86 130/90 135/85 CS 2750 Machine Learning 3 Clustering example A set of patient cases We want to partition them into the groups based on similarities Patient # Patient 1 Patient 2 Patient 3 Patient 4 Patient 5 Age Sex 55 62 67 65 70 M M F F M Heart Rate 85 87 80 90 84 Blood pressure 125/80 130/85 126/86 130/90 135/85 How to design a distance measure to quantify similarities? CS 2750 Machine Learning Clustering example. Distance measures In general, one can choose an arbitrary distance measure. Properties of the distance measures: Assume 2 data entries a,b Positiveness: Symmetry: Identity: d (a, b) 0 d ( a , b ) = d (b , a ) d (a, a ) = 0 CS 2750 Machine Learning 4 Distance metrics Assume pure real-valued data-points: 12 23.5 33.6 17.2 34.5 78.5 89.2 19.2 41.4 66.3 78.8 8.9 36.7 78.3 90.3 21.4 30.1 71.6 88.5 12.5 What distance metric to use? CS 2750 Machine Learning Distance metrics Assume pure real-valued data-points: 12 23.5 33.6 17.2 34.5 78.5 89.2 19.2 41.4 66.3 78.8 8.9 36.7 78.3 90.3 21.4 30.1 71.6 88.5 12.5 What distance metric to use? Euclidian: works for an arbitrary k-dimensional space d (a, b) = (a i =1 k i bi ) 2 CS 2750 Machine Learning 5 Distance measures. Assume pure real-valued data-points: 12 23.5 33.6 17.2 34.5 78.5 89.2 19.2 41.4 66.3 78.8 8.9 36.7 78.3 90.3 21.4 30.1 71.6 88.5 12.5 What distance metric to use? Squared Euclidian: works for an arbitrary k-dimensional space d 2 (a, b) = (a i =1 k i bi ) 2 CS 2750 Machine Learning Distance metrics Assume pure real-valued data-points: 12 23.5 33.6 17.2 34.5 78.5 89.2 19.2 41.4 66.3 78.8 8.9 36.7 78.3 90.3 21.4 30.1 71.6 88.5 12.5 Yet another distance metric L1 : Manhattan distance d (a, b) = Etc. .. CS 2750 Machine Learning |a i =1 k i bi | 6 Distance metrics Assume pure real-valued data-points: 12 23.5 33.6 17.2 34.5 78.5 89.2 19.2 41.4 66.3 78.8 8.9 36.7 78.3 90.3 21.4 30.1 71.6 88.5 12.5 1/ n n-norm distance k d ( a , b ) = ( a i bi ) n i =1 max-norm distance d ( a , b ) = max i ( a i b i ) CS 2750 Machine Learning 1 Distance metrics Generalized distance metric: d 2 ( a, b ) = ( a b ) 1 ( a b ) T 1 is a matrix that weights attributes proportionally to their importance. Different weights lead to a different distance. If = I we get squared Euclidean Limitation: un-normalized distances If = is a covariance matrix we get the Mahalanobis distance Advantage: takes into account correlations among attributes If = ' is a covariance matrix restricted to diagonal elements we obtain a normalized Euclidean metric CS 2750 Machine Learning 7 Distance metrics Assume pure binary values data: 0 1 1 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 What distance metric to use? CS 2750 Machine Learning Distance metrics Assume pure binary values data: 0 1 1 0 1 1 0 1 0 1 0 1 1 0 1 1 1 1 1 1 What distance metric to use? Hamming or Edit distance: The number of attributes that need to be changed to make the entries the same The same metric can be used for pure categorical values CS 2750 Machine Learning 8 Distance metrics Combination of real-valued and categorical attributes Patient # Age Sex Heart Rate Blood pressure Patient Patient Patient Patient Patient 1 2 3 4 5 55 62 67 65 70 M M F F M 85 87 80 90 84 125/80 130/85 126/86 130/90 135/85 What distance metric to use? CS 2750 Machine Learning Distance metrics Combination of real-valued and categorical attributes Patient # Age Sex Heart Rate Blood pressure Patient Patient Patient Patient Patient 1 2 3 4 5 55 62 67 65 70 M M F F M 85 87 80 90 84 125/80 130/85 126/86 130/90 135/85 What distance metric to use? A weighted sum approach: e.g. a mix of Euclidian and Edit distances for subsets of attributes CS 2750 Machine Learning 9 Clustering Clustering is useful for: Similarity/Dissimilarity analysis Analyze what data points in the sample are close to each other Dimensionality reduction High dimensional data replaced with a group (cluster) label Problems: Pick a correct similarity measure (problem specific) Choose the correct number of groups Many clustering algorithms require us to provide the number of groups ahead of time CS 2750 Machine Learning Clustering algorithms Partitioning algorithms: K-means algorithm suitable only when data points have continuous values; groups are defined in terms of cluster centers (also called means). refinement of the method to categorical values: K-medoids Probabilistic methods (with EM) Latent variable models: class (cluster) is represented by the value of latent (hidden) variable Examples: mixture of Gaussians, a Na ve Bayes with a hidden class Hierarchical methods Agglomerative Divisive CS 2750 Machine Learning 10 K-means K-Means algorithm: Initialize randomly k values of means (centers) Repeat two steps until no change in the means: Partition the data according to the current set of means (using the similarity measure) Move the means to the center of the data in the current partition Stop when no change in the means Properties: Minimizes the sum of squared center-point distances for all clusters The algorithm always (local converges optima). CS 2750 Machine Learning K-Means example 3 3 2 2 1 1 0 0 -1 -1 -2 -2 -3 -3 -2 -1 0 1 2 3 -3 -3 -2 -1 0 1 2 3 3 2 1 0 -1 -2 -3 -3 -2 -1 0 1 2 3 CS 2750 Machine Learning 11 K-means algorithm Properties: converges to centers minimizing the sum of squared centerpoint distances (still local optima) The result is sensitive to the initial means values Advantages: Simplicity Generality can work for more than one distance measure Drawbacks: Can perform poorly with overlapping regions Lack of robustness to outliers Good for attributes (features) with continuous values Allows us to compute cluster means k-medoid algorithm used for discrete data CS 2750 Machine Learning K-means algorithm Properties: converges to centers minimizing the sum of squared centerpoint distances (still local optima) The result is sensitive to the initial means values Advantages: Simplicity Generality can work for more than one distance measure Drawbacks: Can perform poorly with overlapping regions Lack of robustness to outliers Good for attributes (features) with continuous values Allows us to compute cluster means k-medoid algorithm used for discrete data CS 2750 Machine Learning 12 K-medoids write CS 2750 Machine Learning Probabilistic clustering Soft version of clustering Each point belongs to a cluster with some probability Based on a latent variable model One latent variable (class, cluster variable) Values of the latent variable denote clusters Each cluster defines a distribution of points that belong to the cluster (class-conditional distributions) Algorithm: Learn the ML parameters of the latent variable model EM algorithm CS 2750 Machine Learning 13 Example: a Gaussian mixture model Probability of occurrence of a data point x is modeled as p ( x ) = p (C = i ) p (x | C = i ) where i =1 k P(C) C p(X | C = i) X p (C = i ) = probability of a data point coming from cluster (class) C=i = cluster conditional density (modeled as a Gaussian) for class i Remember: C is hidden !!!! CS 2750 Machine Learning p ( x | C = i ) N ( i , i ) Gaussian mixture model In the Gaussian mixture Gaussians are not labeled We can apply EM algorithm: re-estimation based on the class posterior (a responsibility of cluster for a point) hil = p (C l = i | x l , ' ) = p (C l = i | ' ) p ( xl | C l = i , ' ) l Ni = h l p (C u =1 m = u | ' ) p ( xl | C l = u , ' ) il N ~ i = i N 1 ~ i = Ni Count replaced with the expected count h l il xj 1 ~ i = Ni h l il ( x j i )( x j i ) T CS 2750 Machine Learning 14 Gaussian mixture algorithm Special case: fixed covariance matrix for all hidden groups (classes) and uniform prior on classes Algorithm: Initialize means i for all classes i Repeat two steps until no change in the means: 1. Compute the class posterior for each Gaussian and each point (a kind of responsibility for a Gaussian for a point) Responsibility: h il = p (C l = i | ' ) p ( xl | C l = i, ' ) m 2. Move the means of the Gaussians to the center of the data, N weighted by the responsibilities New mean: i = u =1 p (C l = u | ' ) p ( xl | C l = u , ' ) l =1 N h il x l h il l =1 CS 2750 Machine Learning K-means approximation to EM Expectation-Maximization: posterior measures the responsibility of a Gaussian for every point h il = p (C l = i | ' ) p ( x l | C l = i, ' ) u =1 K- Means Only the closest Gaussian is made responsible for a point m p (C l = u | ' ) p ( xl | C l = u , ' ) hil = 1 hil = 0 If i is the closest Gaussian Otherwise i Re-estimation of means = N h l =1 N il x il l h l =1 Results in moving the means of Gaussians to the center of the data points it covered in the previous step CS 2750 Machine Learning 15 Probabilistic (EM-based) algorithms Latent variable models Examples: Na ve Bayes with hidden class Mixture of Gaussians Partitioning: the data point belongs to the class with the highest posterior Advantages: Good performance on overlapping regions Robustness to outliers Data attributes can have different types of values Drawbacks: EM is computationally expensive and can take time to converge Density model should be given in advance CS 2750 Machine Learning Hierarchical clustering Uses an arbitrary similarity/dissimilarity measure. Typical similarity measures d(a,b) : Pure real-valued data-points: Euclidean, Manhattan, Minkowski distances Pure binary values data: Number of matching values Pure categorical data: Number of matching values Combination of real-valued and categorical attributes Weighted approaches CS 2750 Machine Learning 16 Hierarchical clustering Approach: Compute dissimilarity matrix for all pairs of points uses distance measures Construct clusters greedily: Agglomerative approach Merge pair of clusters in a bottom-up fashion, starting from singleton clusters Divisive approach: Splits clusters in top-down fashion, starting from one complete cluster Stop the greedy construction when some criterion is satisfied E.g. fixed number of clusters CS 2750 Machine Learning Cluster merging Construction of clusters through greedy agglomerative approach Merge pair of clusters in a bottom-up fashion, starting from singleton clusters Merge clusters based on cluster (or linkage) distances. Defined in terms of point distances. Examples: Min distance Max distance d min ( C i , C j ) = p C i , q C min d ( p, q) j d max ( C i , C j ) = p C i , q C max d ( p, q) j Mean distance d mean ( C i , C j ) = 1 | C i || C j | p i C i q j C d (p ,q i j j ) CS 2750 Machine Learning 17 Hierarchical clustering example 10 8 6 4 2 0 -2 -1 0 1 2 3 4 5 6 7 8 9 CS 2750 Machine Learning Hierarchical clustering example dendogram 6 10 8 6 4 5 2 0 4 -2 -1 0 1 2 3 4 5 6 7 8 9 3 2 1 0 3 6 23 1 2 19 4 8 14 5 17 25 27 26 28 29 30 7 11 9 12 10 13 15 18 20 21 16 22 24 CS 2750 Machine Learning 18 Hierarchical clustering Advantage: Smaller computational cost; avoids scanning all possible clusterings Disadvantage: Greedy choice fixes the order in which clusters are merged; cannot be repaired Partial solution: combine hierarchical clustering with iterative algorithms like k-means CS 2750 Machine Learning Other clustering methods Spectral clustering Uses similarity matrix Multidimensional scaling techniques often used in data visualization for exploring similarities or dissimilarities in data. CS 2750 Machine Learning 19

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: