This preview shows page 1. Sign up to view the full content.
Unformatted text preview: More Clustering
CURE Algorithm
NonEuclidean Approaches 1 The CURE Algorithm
x Problem with BFR/k means: Assumes clusters are normally distributed in each dimension.
And axes are fixed ellipses at an angle are not OK. x CURE: Assumes a Euclidean distance.
Allows clusters to assume any shape.
2 Example: Stanford Faculty Salaries
h e e
salary e e e
h hh h h h e e h
h
h eh h e e
e h age
3 Starting CURE
1. Pick a random sample of points that fit in main memory.
2. Cluster these points hierarchically group nearest points/clusters.
3. For each cluster, pick a sample of points, as dispersed as possible.
4. From the sample, pick representatives by moving them (say) 20% toward the centroid of the cluster.
4 Example: Initial Clusters
h e e
salary e e e
h hh h h h e e h
h
h eh h e e
e h age
5 Example: Pick Dispersed Points
h e e
salary e e e
h hh h h h e e h
h
h eh h h e e
e
Pick (say) 4
remote points
for each
cluster. age
6 Example: Pick Dispersed Points
h e e
salary e e e
h hh h h h e e h
h
h eh h h e e
e
Move points
(say) 20%
toward the
centroid. age
7 Finishing CURE
x Now, visit each point p in the data set.
x Place it in the “closest cluster.”
Normal definition of “closest”: that cluster with the closest (to p ) among all the sample points of all the clusters. 8 Curse of Dimensionality
x One way to look at it: in large
dimension spaces, random vectors are perpendicular. Why?
x Argument #1: Lots of 2dim subspaces. There must be one where the vectors’ projections are almost perpendicular.
x Argument #2: Expected value of cosine of angle is 0.
9 Cosine of Angle Between Random Vectors
x Assume vectors emanate from the origin (0,0,…,0).
x Components are random in range [1,1].
x (a1,a2,…,an).(b1,b2,…,bn) has expected value 0 and a standard deviation that grows as √n.
x But lengths of both vectors grow as √n.
x So dot product around √n/ (√n * √n) = 1/√n.
10 Random Vectors Continued
x Thus, a typical pair of vectors has an angle whose cosine is on the order of 1/√n.
x As n > ∞ , that’s 0; i.e., the angle is about 90°. 11 Interesting Consequence
x Suppose “random vectors are perpendicular,” even in nonEuclidean spaces.
x Suppose we know the distance from A to B, say d (A,B ), and we also know d (B,C ), but we don’t know d (A,C ).
x Suppose B and C are fairly close, say in the same cluster.
x What is d (A,C )?
12 Diagram of Situation
Approximately
perpendicular B
A
C
Assuming points lie in a plane:
d (A,B )2 + d (B,C )2 = d (A,C )2 13 Important Point
x Why do we assume AB is perpendicular to AC, and not that either of the other two angles are right
angles?
1. AB and AC are not “random vectors”; they each go to points that are far away from A and close to each other.
2. If AB is longer than AC, then it is angle ACB that is right, but both ACB and ABC 14
are approximately rightangles. Dealing With a NonEuclidean Space
x Problem: clusters cannot be represented by centroids.
x Why? Because the “average” of “points” might not be a point in the space.
x Best substitute: the clustroid = point in the cluster that minimizes the sum of the squares of distances to the points in the cluster.
15 Representing Clusters in Non
Euclidean Spaces
x Recall BFR represents a Euclidean cluster by N, SUM, and SUMSQ.
x A nonEuclidean cluster is represented by:
N.
The clustroid.
Sum of the squares of the distances from clustroid to all points in the cluster. 16 Example of CoD Use
x Problem: in nonEuclidean space, we want to decide whether to merge two clusters. Each cluster represented by N, clustroid, and “SUMSQ.”
Also, SUMSQ for each point in the cluster, even if it is not the clustroid. x Merge if SUMSQ for new cluster is “low.”
17 Estimating SUMSQ
p ’s clustroid, c other clust
roid, b p 18 Suppose p Were the Clustroid of Combined Cluster
x It’s SUMSQ would be the sum of: 1. Old SUMSQ(p) [for old cluster containing p].
2. SUMSQ(b) plus d (p,b)2 times number of points in b ’s cluster. 1. Critical point: vector p >b assumed perpendicular to vectors from b to all other points in its cluster justifies (2).
19 Combining Clusters Continued
x We can thus estimate SUMSQ for each point in the combined cluster. Take the point with the least SUMSQ as the clustroid of the new cluster provided that SUMSQ is small enough. 20 The GRGPF Algorithm
x From Ganti et al. see reading list.
x Works for nonEuclidean distances.
x Works for massive (diskresident) data.
x Hierarchical clustering.
x Clusters are grouped into a tree of disk blocks (like a Btree or Rtree).
21 Information Retained About a Cluster
1. N, clustroid, SUMSQ.
2. The p points closest to the clustroid, and their values of SUMSQ.
3. The p points of the cluster that are furthest away from the clustroid, and their SUMSQ’s. 22 At Interior Nodes of the Tree
x Interior nodes have samples of the clustroids of the clusters found at descendant leaves of this node.
x Try to keep clusters on one leaf block close, descendants of a level1 node close, etc.
x Interior part of tree kept in main memory.
23 Picture of the Tree
main
memory samples cluster data cluster data on disk
24 Initialization
x Take a mainmemory sample of points.
x Organize them into clusters hierarchically.
x Build the initial tree, with level1 interior nodes representing clusters of clusters, and so on.
x All other points are inserted into this tree.
25 Inserting Points
x Start at the root.
x At each interior node, visit one or more children that have sample clustroids near the inserted point.
x At the leaves, insert the point into the cluster with the nearest clustroid. 26 Updating Cluster Data
x Suppose we add point X to a cluster.
x Increase count N by 1.
x For each of the 2p + 1 points Y whose SUMSQ is stored, add d (X,Y )2.
x Estimate SUMSQ for X. 27 Estimating SUMSQ(X )
x If C is the clustroid, SUMSQ(X ) is, by the CoD assumption: Nd (X,C )2 + SUMSQ(C ) Based on assumption that vector from X to C is perpendicular to vectors from C to all the other nodes of the cluster. x This value may allow X to replace one of the closest or furthest nodes. 28 Possible Modification to Cluster Data
x There may be a new clustroid one of the p closest points because of the addition of X.
x Eventually, the clustroid may migrate out of the p closest points, and the entire representation of the cluster needs to be recomputed.
29 Splitting and Merging Clusters
x Maintain a threshold for the radius of a cluster = √(SUMSQ/N ).
x Split a cluster whose radius is too large.
x Adding clusters may overflow leaf blocks, and require splits of blocks up the tree.
Splitting is similar to a Btree.
But try to keep locality of clusters. 30 Splitting and Merging (2)
x The problem case is when we have split so much that thetree no longer fits in main memory.
x Raise the threshold on radius and merge clusters that are sufficiently close. 31 Merging Clusters
x Suppose there are nearby clusters with clustroids C and D, and we want to consider merging them.
x Assume that the clustroid of the combined cluster will be one of the p furthest points from the clustroid of one of those clusters.
32 Merging (2)
x Compute SUMSQ(X ) [from the cluster of C ] for the combined cluster by summing:
1. SUMSQ(X ) from its own cluster.
2. SUMSQ(D ) + N [d (X,C )2 + d (C,D )2]. x Uses the CoD to reason that the distance from X to each point in the other cluster goes to C, makes a right angle to D, and another right angle to the point.
33 Merging Concluded
x Pick as the clustroid for the combined cluster that point with the least SUMSQ.
x But if this SUMSQ is too large, do not merge clusters.
x Hope you get enough mergers to fit the tree in main memory. 34 Fastmap
x Not a clustering algorithm rather, a method for applying multidimensional scaling.
That is, mapping the points onto a small
dimension space, so the CoD does not apply. 35 Fastmap (2)
x Assumes nonEuclidean space. But like GRGFP pretends it is working in 2
dimensional Euclidean space when it is convenient to do so. x Goal: map n points in much less than O(n 2) time.
I.e., you cannot compute distances between each pair of points and place points in kdim. space to minimize error. 36 Fastmap Key Idea
x Create a “dimension” in nonEuclidean space by:
1. Pick a pair of points A and B that are far apart.
x Start with random A; pick most distant B. 1. Treat AB as an “axis” and project all points onto AB, using the law of cosines. 37 Projecting Point C Onto AB
C d(A,C) A d(B,C) d(A,B) B x x = [d 2(A,C) + d 2(A,B) – d 2(B,C)]/2d (A,B)
38 Revising Distances
x Having computed the position of every point along the pseudoaxis AB, we need to lower the distances between points in the “other dimensions.” 39 Picture
D
dold(C,D)
C A dnew(C,D) =
√dold(C,D)2 – (xy)2 B x
y 40 But …
x We can’t afford to compute new distances for each pseudodimension.
It would take O(n 2) time. x Rather, for each pseudodimension, store the position along the pseudoaxis for each point, and adjust the distance between points by squaresubtractsqrt only when needed.
I.e., one of the points is an axisend. 41 Fastmap Summary
x Pick a number of dimensions k.
FOR i = 1 TO k DO BEGIN
Pick a pseudoaxis AiBi;
Compute projection of each
point onto this pseudoaxis;
END;
x Each step is O(ni ); total O(nk 2).
42 ...
View
Full
Document
This document was uploaded on 01/25/2012.
 Spring '09

Click to edit the document details