Dragoljub Clustering
Pokrajac 2003
Clustering and Unsupervised Learning
Clustering is one of algorithms for unsupervised learning
Class labels are not known in advance Unlike in classification, clustering models are learned purely based on attribute values
What is Clustering ?
Given: Set of unlabeled patterns
Each pattern contain one or more groups
Goal: Group patterns such that patterns "similar" to each other are in one group, and "dissimilar" in distinct groups Such distinguished groups are called clusters
Example
Group these figures according some criteria
Attributes: Number of edges Color
Clustering by color
Clustering by number of edges
Some Issues in Clustering
The actual number of clusters is not known Potential lack of apriori knowledge about data and cluster shapes Clustering could be performed on-line Time complexity, when working with large amounts of data
Types of Clustering Algorithms
Partitioning Hierarchical Clusters for large data sets
Partitioning Methods
Only one set of clusters is created at the output of the algorithm Number of clusters is usually specified Dataset is being partitioned into several groups and groups are updated through iterations K-means, EM, PAM, CLARA, CLARANS...
K Means Algorithm
Repeat:
Randomly initialize K cluster centers Each point assign to the nearest cluster center Re-estimate cluster centers by averaging coordinates of points assigned to each cluster
TO DO: Find better pictures (with better initialization)
Points To Be Clustered Into Three Clusters
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Randomly Choose 3 cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n1 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n2 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n3 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n4 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n5 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n6 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n7 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n8 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio ra n9 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Recompute Cluster Centers
4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Assign Each Point to the Closest Center
ite tio 1 ra n 0 4
3
2
1
0
-1
-2
-3 -3
-2
-1
0
1
2
3
4
5
Problems With K-means
Number of Clusters Must Be prespecified Algorithm Sensitive on Initialization
May not converge to proper clusters TO DO: show this! Algorithm does not care about the shape of the clusters Algorithm does not care about densities
"Blue" cluster much denser than the other clusters
EM Algorithm
Idea:
Each point came from one of Gaussian distributions
Goal: estimate parameters of Gaussian distributions
Mixture of Gaussian Distributions
With probability p1 data came from the distribution D1 determined by:
Mean 1, covariance matrix 1, conditional density function p(x|D1)
With probability p2 data came from the distribution D2 determined by:
Mean 2, covariance matrix 2, conditional density function p(x|D2)
... With probability pK data came from the distribution DK determined by:
Mean K, covariance matrix K, conditional density
Gaussian Mixture - Formula
Similar to well-known formula of total probability, we have formula of "total" probability density... p(x)= p1*p(x|D1)+ p2*p(x|D2)+... pK*p(x|DK) As we know, since conditional distributions are here Gaussians, we have:
p x | Dj =
(
)
( 2 ) d det ( j )
1
e
( x - j ) j -1 ( x - j )T -
2
, j = 1,2,..., K
EM Algorithm Details
We need to set number of clusters K in advance Consists of two phases Expectation
Compute for every pattern probability that is came from each of K clusters, conditioned by observed attributes
Maximization
Update estimated values for
Means of the clusters Covariance matrices of the clusters Cluster priors (probability that a random point belongs to given cluster)
EM in Matlab
for j=1:n_iterations % "Expectation" phase: Compute for patterns probabilities that they came from % each of K clusters for h=1:K Pmat(:,h)=p(h)*multinorm_distr_value(mu(h,:),sigma{h},X); end Pmat=Pmat./repmat(sum(Pmat,2),1,K); % % "Maximization" phase %Compute new means for h=1:K mu(h,:)=Pmat(:,h)'*X/p(h)/N; %N patterns end %Compute new sigmas for h=1:K XM=(X-repmat(mu(h,:),N,1)).*repmat(Pmat(:,h)).^0.5,1,2); sigma{h}=XM'*XM/p(h)/N; end % Compute new prior probabilities for h=1:K p(h)=w'*Pmat(:,h)/N; end
end
EM Algorithm- Example
3500 points from mixture of three twodimensional Gaussian Distributions EM algorithm initialized with
Distribution means close to true means Covariance matrices equal to covariance matrix on all data Equal priors
Each point colored by mixture of three primary colors More clear color: more certain in the cluster membership
True Cluster Membership
Estimated and Correct Values of Distribution Parameters -Sigmas and priors correctly estimated -Means not correctly estimated
- 0.0025353 0.31173 1.0116 ^ ^ 1 = 1 = - 0.0025353 0.29621 - 1.023813 0.060267 - 0.040135 1.0084 ^ ^ 2 = 2 = - 0.040135 0.059549 3.0116 1.0227 ^ = 0.93642 0.77717 ^3 = 3 0.77717 1.0524 1.0034 0.3 0 1 = 0 0.3 0.06 - 0.04 2 = - 0.04 0.06 1 0.8 3 = 0.8 1 3 1 = - 1 1 2 = - 2 1 3 = 1 p1 = 0.57143 p 2 = 0.28571 p 3 = 0.14286 ^ p1 = 0.5701 ^ p 2 = 0.28581 ^ p 3 = 0.14409
Problems with EM Algorithm
Slow Convergence depends on the initialization Assumes Gaussian clusters
Hierarchical Clustering
Set of clusters is created The way clusters are created is depicted by dendograms
Agglomerative Divisive
Agglomerative Clustering
Put each pattern into one separate cluster While there are more than c clusters
Merge two clusters closest according to some distance criterion
Output c clusters
Distance Criterion
SINGLE LINK
Minimal distance between points in two clusters
COMPLETE LINK
Maximal distance between points in two clusters
AVERAGE LINK
Average distance between points in two clusters
Single Link Distance
Complete Link Distance
Average Link
Average Distances between all pairs
Properties of Single Link Distance
Favors elongated clusters
These two clusters Are closer than the Others in Single Link Sense
Properties of Complete Link Distance
Favors Compact clusters
These two clusters Are closer than the Others in complete link Sense
Single Link - Example
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
Each orange square is initially a separate cluster
Single Link distance=0.25
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
We start to merge the closest points
Single Link distance=0.5
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
Adding the next point
Single Link distance=0.5
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
Constructing new cluster
Single Link distance=1
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
Constructing new cluster
Single Link distance=3.5355
3 2 5 .7 2 .5 2 5 .2 2 1 5 .7 1 .5 1 5 .2 1 0 5 .7 0 .5 0 5 .2 0 -0 5 .2 -0 .5 -0 5 .7 -1 -1 5 .2 -1 .5 -1 5 .7 -2 -2 5 .2 -2 .5 -2 5 .7 -3 -3 -2 5 .5 .2 -2 -1 5 .5 .2 -1 -0 5 .5 .2 0 0 50 .7 1 1 51.5 1 5 2 2.2 .5 2 5 3 .7 -2 -2 5 .7 -1 -1 5 .7 -0 -0 5 .2 .50 5 .2 .7 52 .7
Merging
Dendogram: For each two merged clusters shows Their distance at the moment of merging
5
Distance
4
3
2
1 0 5 .7 0 .5 0 5 .2 0 9 1 0 1 1 8 4 6 5 7 1 2 3
Main Problems
Slow
Distances be should recomputed O(n2) time complexity
Divisive Clustering
We start from only one cluster and successively split the clusters into smaller E.g. using Minimal Spanning Tree (MST)
Minimal Spanning Tree is tree connecting all edges of the graph such that the sum of vertices is minimal
Note: MST can also be used to do single link...
Divisive Clustering using MST
Consider patterns as vertices of fully connected graph Consider each pair of vertices as connected with edge length equal to the distance between points Compute MST Sort edges of MST in decreasing order While there are remaining edges
Form new cluster by deleting the longest remaining edge
Divisive Clustering Using MST
Original Data
Divisive Clustering Using MST
Fully Connected Graph
Divisive Clustering Using MST
Minimum Spanning Tree
Divisive Clustering Using MST
Make first Two clusters splitting longest edge in MST
Divisive Clustering Using MST
Make third clusters splitting second largest edge and proceed till 11 clusters are formed
Clustering Large Datasets
Issues
Time complexity vs.
Number of patterns Number of attributes
Spatial complexity
What if not whole dataset can fit in main memory?
DBSCAN
Density based clusters Time complexity, using special data structure R* trees is O(NlogN) where N is number of patterns
Couple of Definitions
Core point: pattern in which neighborhood there are more than Nmin patterns
Nmin minimal number of patterns in the neighborhood Example: Nmin =9
Core point
Non-core point
Density Reachable Points
Some points density reachable from p1 (via p2 and p3)
Core point p3: Neighbor of p2
Neighborhood of p1
Core point p2: Neighbor of p1
Core point p1
Neighborhood of p2
Density Reachable Points Formally...
Point q is density reachable from p1 if:
p1 is a core point There are some core points p2, p3,...,pM such that p2 is in neighborhood of p1 p3 is in neighborhood of p2 p4 is in neighborhood of p3
...
pM is in neighborhood of pM-1 q is in neighborhood of pM
NOTE: q does not need to be a core point!
Density- Based Cluster
A density-based cluster contains all the points density reachable from an arbitrary core point in the cluster!
Idea of DBSCAN
Initially, all patterns of the database are unlabelled. BUT: For each pattern, we check whether it is labeled (it can be labeled if it was in some previously detected cluster) If the pattern is not labeled, we will check whether it is a core point, so that it may initiate a new cluster If the pattern has a cluster label, we do not do nothing but instead process next pattern
Idea of DBSCAN -Cont
If the examined point is a core point, it seeds a new cluster. We observe the neighbors
If the neighbor is already labeled, it means it is already examined so we do not need to assign label or to reexamine that Otherwise (neighbor is unlabeled)
Each neighbor is assigned a label of a new cluster We recursively examine all core points in the neighborhood
DBSCAN - Algorithm
DBSCAN: FOR each pattern in dataset IF the pattern is not already assigned to a cluster
IF CORE_POINT(pattern)==Yes
ASSIGN new cluster label to the pattern EXAMINE (pattern.neighbors) EXAMINE (pattern.neighbors): FOR each neighbor IN pattern.neighbors IF neighbor is not already assigned to a cluster ASSIGN new cluster label to the neighbor IF CORE_POINT(neighbor)==Yes EXAMINE (neighbor.neighbors) ELSE return;
Important Note
In practical realization, we can avoid having recursive calls
Maintain and update the list of all nodes from various neighborhoods that need to be examined
NOTE: Instead of list, we could improve performance by using sets (sets do not contain duplicates...)
This leads to the following practical, nonrecursive version of DBSCAN
Non-Recursive DBSCAN
FOR each pattern in dataset IF the pattern is not already assigned to a cluster IF CORE_POINT(pattern)==Yes ASSIGN new cluster label to the pattern ADD pattern.neighbors to the list WHILE list is not empty TAKE neighbor from the beginning of the list (and remove it from the list) IF neighbor is not already assigned to a cluster ASSIGN new cluster label to the neighbor IF CORE_POINT(neighbor)==Yes ADD neighbor.neighbors to the list; END WHILE
Remark
In addition to the functionality provided in the described algorithm DBSCAN may assign a NOISE label to a pattern Pattern is NOISE if it is not a core point and if it is not density reachable from some core point NOISE patterns do not belong to any class
DBSCAN - Example
1 2 1 0
8
6
4
2
0 0
2
4
6
8
1 0
1 2
1 2
1 0
Unclassified Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
Search neighborhood epsilon,=2.5, Nmin=7
4
2
0
-2
0
2
4
6
8
1 0
1 2
First Iteration
1 2
Unclassified
1 0
Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
4
2
0 -2 0 2 4 6 8 1 0 1 2
1 2 1 1
Unclassified
1 0 9 8 7 6 5 4 3 2 1 -2 0 2 4 6 8 1 0 12
Unclassified neighborhood Currently searched Assigned cluster label Noise
1 2
1 0
Unclassified Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
4
2
0
-2
0
2
4
6
8
1 0
1 2
1 4
1 2
Unclassified
1 0
Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
4
2
0 -2 0 2 4 6 8 1 0 1 2
1 2 1 1
Unclassified
1 0 9 8 7 6 5 4 3 2 1 0 2 4 6 8 1 0 1 2
Unclassified neighborhood Currently searched Assigned cluster label Noise
1 2
1 0
Unclassified Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
4
2
0
-2
0
2
4
6
8
1 0
1 2
1 4
1 2
Unclassified
1 0
Unclassified neighborhood Currently searched Assigned cluster label
8
Noise
6
4
2
0 0 2 4 6 8 1 0 1 2 1 4
1 2 1 1
Unclassified
1 0 9 8 7 6 5 4 3 2 1 0 2 4 6 8 1 0 1 2
Unclassified neighborhood Currently searched Assigned cluster label Noise...