cs345-cl2 - More Clustering CURE Algorithm Non­Euclidean...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: More Clustering CURE Algorithm Non­Euclidean Approaches 1 The CURE Algorithm x Problem with BFR/k ­means:  Assumes clusters are normally distributed in each dimension.  And axes are fixed ­­­ ellipses at an angle are not OK. x CURE:  Assumes a Euclidean distance.  Allows clusters to assume any shape. 2 Example: Stanford Faculty Salaries h e e salary e e e h hh h h h e e h h h eh h e e e h age 3 Starting CURE 1. Pick a random sample of points that fit in main memory. 2. Cluster these points hierarchically ­­­ group nearest points/clusters. 3. For each cluster, pick a sample of points, as dispersed as possible. 4. From the sample, pick representatives by moving them (say) 20% toward the centroid of the cluster. 4 Example: Initial Clusters h e e salary e e e h hh h h h e e h h h eh h e e e h age 5 Example: Pick Dispersed Points h e e salary e e e h hh h h h e e h h h eh h h e e e Pick (say) 4 remote points for each cluster. age 6 Example: Pick Dispersed Points h e e salary e e e h hh h h h e e h h h eh h h e e e Move points (say) 20% toward the centroid. age 7 Finishing CURE x Now, visit each point p in the data set. x Place it in the “closest cluster.”  Normal definition of “closest”: that cluster with the closest (to p ) among all the sample points of all the clusters. 8 Curse of Dimensionality x One way to look at it: in large­ dimension spaces, random vectors are perpendicular. Why? x Argument #1: Lots of 2­dim subspaces. There must be one where the vectors’ projections are almost perpendicular. x Argument #2: Expected value of cosine of angle is 0. 9 Cosine of Angle Between Random Vectors x Assume vectors emanate from the origin (0,0,…,0). x Components are random in range [­1,1]. x (a1,a2,…,an).(b1,b2,…,bn) has expected value 0 and a standard deviation that grows as √n. x But lengths of both vectors grow as √n. x So dot product around √n/ (√n * √n) = 1/√n. 10 Random Vectors ­­­ Continued x Thus, a typical pair of vectors has an angle whose cosine is on the order of 1/√n. x As n ­> ∞ , that’s 0; i.e., the angle is about 90°. 11 Interesting Consequence x Suppose “random vectors are perpendicular,” even in non­Euclidean spaces. x Suppose we know the distance from A to B, say d (A,B ), and we also know d (B,C ), but we don’t know d (A,C ). x Suppose B and C are fairly close, say in the same cluster. x What is d (A,C )? 12 Diagram of Situation Approximately perpendicular B A C Assuming points lie in a plane: d (A,B )2 + d (B,C )2 = d (A,C )2 13 Important Point x Why do we assume AB is perpendicular to AC, and not that either of the other two angles are right­ angles? 1. AB and AC are not “random vectors”; they each go to points that are far away from A and close to each other. 2. If AB is longer than AC, then it is angle ACB that is right, but both ACB and ABC 14 are approximately right­angles. Dealing With a Non­Euclidean Space x Problem: clusters cannot be represented by centroids. x Why? Because the “average” of “points” might not be a point in the space. x Best substitute: the clustroid = point in the cluster that minimizes the sum of the squares of distances to the points in the cluster. 15 Representing Clusters in Non­ Euclidean Spaces x Recall BFR represents a Euclidean cluster by N, SUM, and SUMSQ. x A non­Euclidean cluster is represented by:  N.  The clustroid.  Sum of the squares of the distances from clustroid to all points in the cluster. 16 Example of CoD Use x Problem: in non­Euclidean space, we want to decide whether to merge two clusters.  Each cluster represented by N, clustroid, and “SUMSQ.”  Also, SUMSQ for each point in the cluster, even if it is not the clustroid. x Merge if SUMSQ for new cluster is “low.” 17 Estimating SUMSQ p ’s clustroid, c other clust­ roid, b p 18 Suppose p Were the Clustroid of Combined Cluster x It’s SUMSQ would be the sum of: 1. Old SUMSQ(p) [for old cluster containing p]. 2. SUMSQ(b) plus d (p,b)2 times number of points in b ’s cluster. 1. Critical point: vector p ­>b assumed perpendicular to vectors from b to all other points in its cluster ­­­ justifies (2). 19 Combining Clusters ­­­ Continued x We can thus estimate SUMSQ for each point in the combined cluster. Take the point with the least SUMSQ as the clustroid of the new cluster ­­­ provided that SUMSQ is small enough. 20 The GRGPF Algorithm x From Ganti et al. ­­­ see reading list. x Works for non­Euclidean distances. x Works for massive (disk­resident) data. x Hierarchical clustering. x Clusters are grouped into a tree of disk blocks (like a B­tree or R­tree). 21 Information Retained About a Cluster 1. N, clustroid, SUMSQ. 2. The p points closest to the clustroid, and their values of SUMSQ. 3. The p points of the cluster that are furthest away from the clustroid, and their SUMSQ’s. 22 At Interior Nodes of the Tree x Interior nodes have samples of the clustroids of the clusters found at descendant leaves of this node. x Try to keep clusters on one leaf block close, descendants of a level­1 node close, etc. x Interior part of tree kept in main memory. 23 Picture of the Tree main memory samples cluster data cluster data on disk 24 Initialization x Take a main­memory sample of points. x Organize them into clusters hierarchically. x Build the initial tree, with level­1 interior nodes representing clusters of clusters, and so on. x All other points are inserted into this tree. 25 Inserting Points x Start at the root. x At each interior node, visit one or more children that have sample clustroids near the inserted point. x At the leaves, insert the point into the cluster with the nearest clustroid. 26 Updating Cluster Data x Suppose we add point X to a cluster. x Increase count N by 1. x For each of the 2p + 1 points Y whose SUMSQ is stored, add d (X,Y )2. x Estimate SUMSQ for X. 27 Estimating SUMSQ(X ) x If C is the clustroid, SUMSQ(X ) is, by the CoD assumption: Nd (X,C )2 + SUMSQ(C )  Based on assumption that vector from X to C is perpendicular to vectors from C to all the other nodes of the cluster. x This value may allow X to replace one of the closest or furthest nodes. 28 Possible Modification to Cluster Data x There may be a new clustroid ­­­ one of the p closest points ­­­ because of the addition of X. x Eventually, the clustroid may migrate out of the p closest points, and the entire representation of the cluster needs to be recomputed. 29 Splitting and Merging Clusters x Maintain a threshold for the radius of a cluster = √(SUMSQ/N ). x Split a cluster whose radius is too large. x Adding clusters may overflow leaf blocks, and require splits of blocks up the tree.  Splitting is similar to a B­tree.  But try to keep locality of clusters. 30 Splitting and Merging ­­­ (2) x The problem case is when we have split so much that thetree no longer fits in main memory. x Raise the threshold on radius and merge clusters that are sufficiently close. 31 Merging Clusters x Suppose there are nearby clusters with clustroids C and D, and we want to consider merging them. x Assume that the clustroid of the combined cluster will be one of the p furthest points from the clustroid of one of those clusters. 32 Merging ­­­ (2) x Compute SUMSQ(X ) [from the cluster of C ] for the combined cluster by summing: 1. SUMSQ(X ) from its own cluster. 2. SUMSQ(D ) + N [d (X,C )2 + d (C,D )2]. x Uses the CoD to reason that the distance from X to each point in the other cluster goes to C, makes a right angle to D, and another right angle to the point. 33 Merging ­­­ Concluded x Pick as the clustroid for the combined cluster that point with the least SUMSQ. x But if this SUMSQ is too large, do not merge clusters. x Hope you get enough mergers to fit the tree in main memory. 34 Fastmap x Not a clustering algorithm ­­­ rather, a method for applying multidimensional scaling.  That is, mapping the points onto a small­ dimension space, so the CoD does not apply. 35 Fastmap ­­­ (2) x Assumes non­Euclidean space.  But like GRGFP pretends it is working in 2­ dimensional Euclidean space when it is convenient to do so. x Goal: map n points in much less than O(n 2) time.  I.e., you cannot compute distances between each pair of points and place points in k­dim. space to minimize error. 36 Fastmap ­­­ Key Idea x Create a “dimension” in non­Euclidean space by: 1. Pick a pair of points A and B that are far apart. x Start with random A; pick most distant B. 1. Treat AB as an “axis” and project all points onto AB, using the law of cosines. 37 Projecting Point C Onto AB C d(A,C) A d(B,C) d(A,B) B x x = [d 2(A,C) + d 2(A,B) – d 2(B,C)]/2d (A,B) 38 Revising Distances x Having computed the position of every point along the pseudo­axis AB, we need to lower the distances between points in the “other dimensions.” 39 Picture D dold(C,D) C A dnew(C,D) = √dold(C,D)2 – (x­y)2 B x y 40 But … x We can’t afford to compute new distances for each pseudo­dimension.  It would take O(n 2) time. x Rather, for each pseudo­dimension, store the position along the pseudo­axis for each point, and adjust the distance between points by square­subtract­sqrt only when needed.  I.e., one of the points is an axis­end. 41 Fastmap ­­­ Summary x Pick a number of dimensions k. FOR i = 1 TO k DO BEGIN Pick a pseudo-axis AiBi; Compute projection of each point onto this pseudo-axis; END; x Each step is O(ni ); total O(nk 2). 42 ...
View Full Document

This document was uploaded on 01/25/2012.

Ask a homework question - tutors are online