1

Clustering

Genoveva Vargas-Solar
http://www.vargas-solar.com/big-data-analytics
French Council of Scientific Research, LIG & LAFMIA Labs

Montevideo, 22nd November – 4th December, 2015

I N F O R M A T I Q U E
+ High Dimensional Data 2

High dim. Graph Infinite Machine

Apps
data data data learning

Locality sensitive PageRank, Filtering data Recommender

SVM
hashing SimRank streams systems

Community Web Decision Association

Clustering
Detection advertising Trees Rules

Dimensionality Duplicate
Spam Queries on Perceptron,
reduction document
Detection streams kNN
detection

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ High Dimensional Data 3

n Given a cloud of data points we want to understand its

structure

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Clustering 4

Discover a set of classes given a data collection

? ? ? ?
? ?? ? ?
? ? ? ? ??
?
+ Approaches 5

n Asa branch of statistics, clustering analysis extensively

studied focused on distance-based clustering analysis
n AutoClass with Bayesian networks
n P. Cheeseman, J. Stutz, Bayesian classification (AutoClass): Theory and results, In U.M. Fayyad, G.
Piatetsky-Shapiro, P. Smyth, R. Uthurusamy, eds. Advances in Knowledge Discovery and Data mining,
AAAI/MIT Press, 1996

n Assume that all data point are given in advance and can be
scanned frequently
n In machine learning clustering analysis
n Clustering analysis
+ Approaches 6

n As a branch of statistics

n In machine learning clustering analysis

n Refers to unsupervised learning
n Classes to which an object belongs to are not pre-specified
n Conceptual clustering
n Distance measurement may not be based on geometric distance but on that a group of objects represents a certain
conceptual class
n One needs to define a similarity between the objects and then apply it to determine the classes
n Classes are collections of objects low interclass similarity and high intra class similarity

n Clustering analysis
+ Approaches 7

n As a branch of statistics

n In machine learning clustering analysis

n Clustering analysis
n Probability analysis
n Assumption that probability distributions on separate attributes are statistically independent one another (not always true)
n The probability distribution representation of clusters à expensive clusters’ updates and storage
n Probability-based tree built to identify clusters is not height balanced
n Increase of time and space complexity

D. Fisher, Improving inference through conceptual clustering, In Proceedings of the AAAI Conference, July, 1987

D. Fisher, Optimization and simplification of hierarchical clusterings, In Proceedings of the 1st Conference on Knowledge Discovery and Data mining, August,
1985
 8

Clustering analysis
+ The Problem of Clustering 9

n Given a set of points, with a notion of distance between points,

group the points into some number of clusters, so that
n Members of a cluster are close/similar to each other
n Members of different clusters are dissimilar

n Usually:
n Points are in a high-dimensional space
n Similarity is defined using a distance measure
n Euclidean, Cosine, Jaccard, edit distance, …

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Clusters & Outliers 10
Beegles
Heigth x
x
xx x
x x
Chihuahua x x x x x
Chihuahua x x x x x
x xx x xx x
x x x x
x x

x x x Dachshuds
x x x x
x x x
x
Outlier Cluster
Weight
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Points and spaces 11

n A dataset suitable for clustering is a collection of points, which are objects

belonging to some space.

n In its most general sense, a space is just a universal set of points, from which the
points in the dataset are drawn.

n However, we should be mindful of the common case of a Euclidean space

n Points are vectors of real numbers.
n The length of the vector is the number of dimensions of the space.
n The components of the vector are commonly called coordinates of the represented points

n All spaces for which we can perform a clustering have a distance measure,
giving a distance between any two points in the space
 12

Measuring distance
+ Distance measure 13

Suppose we have a set of points, called a space.

n A distance measure on this space is a function d(x, y) that

n takes two points in the space as arguments and
n produces a real number

n Satisfies the following axioms:

1. d(x, y) ≥ 0 (no negative distances)
2. d(x,y) = 0 if and only if x = y (distances are positive, except for the distance from a point
to itself)
3. d(x, y) = d(y, x) (distance is symmetric)
4. d(x, y) ≤ d(x, z) + d(z, y) (the triangle inequality)

Axion 4: intuitively say that to travel from x to y, we cannot obtain any benefit if we are forced to travel via some particular
third point z. It makes all distance measures behave as if distance describes the length of a shortest path from one point to
another
+ Euclidean distance 14

n An n-dimensional Euclidean space is one where points are

vectors of n real numbers.

n The conventional distance measure in this space, which we shall

refer to as the L2-norm, is defined:
+ Euclidean distance is a d-measure 15

n The Euclidean distance between two points cannot be negative (axiom 1)

n The positive square root is intended (axiom 2)

n all squares of real numbers are nonnegative, any i such that xi ̸= yi forces the
distance to be strictly positive
n if xi = yi for all i, then the distance is clearly 0.

n Symmetry follows because (xi − yi)2 = (yi − xi)2 (axiom 3)

n The triangle inequality requires a good deal of algebra to verify

n It is well understood to be a property of Euclidean space
n The sum of the lengths of any two sides of a triangle is no less than the length of the
third side
+ Other distance measures in 16
Euclidean spaces
n For any constant r, the Lr-norm is the distance measure d
defined by:

n The case r = 2 is the usual L2-norm

+ L1-norm – Manhattan distance 17

n The distance between two points is the

n sum of the magnitudes of the differences in each dimension`

n Manhattan distance
n The distance one would have to travel between points if one were
constrained to travel along grid lines
n As on the streets of a city such as Manhattan
+ L - norm distance 18
∞

n The limit as r approaches infinity of the Lr-norm.

n As r gets larger, only the dimension with the largest difference

matters

n L∞-norm is defined as the maximum of |xi − yi| over all

dimensions i
+ L - examples 19
x

n Consider the two-dimensional Euclidean space and the points

(2,7) and (6,4)

n L2-norm

n L1-norm

n L∞-norm
+ Jaccard distance 20

n The Jaccard similarity is a measure of how close sets are, although it is not
really a distance measure
n the closer sets are, the higher the Jaccard similarity

n Jaccard distance: 1 minus the Jaccard similarity is a distance measure

n where SIM(x, y) is the probability a random minhash function maps x and y

to the same value

n 1 minus the ratio of the sizes of the intersection and union of sets x and y
+ Cosine distance 21

n Cosine distance between two points is the angle that the vectors to those
points make.
n The angle will be in the range 0 to 180 degrees, regardless of how many dimensions
the space has
n First compute the cosine of the angle, and then
n Apply the arc-cosine function to translate to an angle in the 0-180 degree range.

n Given two vectors x and y, the cosine of the angle between them is the
n dot product x.y

n divided by the L2-norms of x and y (i.e., their Euclidean distances from the origin)
+ Cosine distance example 22

n Let our two vectors be

n The dot product x.y

n L2-norm of x

n The cosine of the angle between x and y

n The angle whose cosine is 1/2 is 60 degrees, so that is the cosine

distance between x and y.
+ Cosine distance: final remarks 23

Makes sense in spaces that have dimensions, including

n Euclidean spaces

n Discrete versions of Euclidean spaces,

n spaces where points are vectors with integer components or Boolean
(0 or 1) components
n points may be thought of as directions.
n There is no distinction between a vector and a multiple of that vector
+ Edit distance (1) 24

n The distance between two strings x = x1x2 ···xn and y = y1y2

···ym is the smallest number of insertions and deletions of single
characters that will convert x to y

n The edit distance between the strings x = abcde and y = acfdeg

is 3. To convert x to y:
n Delete b
n Insert f after c
n Insert g after e

n No sequence of fewer than three insertions and/or deletions will

convert x to y. Thus, d(x, y) = 3
+ Edit distance (2) 25

n edit distance d(x, y): can be calculated as the length of x

plus the length of y minus twice the length of their LCS

n Compute a longest common subsequence (LCS) of x and y

n a string that is constructed by deleting positions from x and y, and
n that is as long as any string that can be constructed that way
+ Edit distance example 26

n The strings x = abcde and y = acfdeg

n Have a unique LCS, which is acde
n We can be sure it is the longest possible, because it contains every symbol
appearing in both x and y

n These common symbols appear in the same order in both strings, so we are
able to use them all in an LCS

n Given the length of x is 5, the length of y is 6, and the length of their LCS is 4

n The edit distance length(X)+length(y) – 2(LCS): 5+6−2×4 = 3

+ Hamming distance 27

n Given a space of vectors

n Hamming distance between two vectors is

n the number of components in which they differ

n Most commonly, Hamming distance is used when the vectors are

Boolean

n The Hamming distance between the vectors 10101 and 11110 is 3.

n vectors differ in the second, fourth, and fifth components, while they
agree in the first and third components.
+ Non Euclidean space 28

n Euclidean space property: the average of points in a Euclidean

space always exists and is a point in the space.

n There are spaces where the notion of average makes no sense (e.g.,
average of strings)

n Vector spaces, may or may not be Euclidean

n If the components of the vectors can be any real numbers, then the space
is Euclidean
n if we restrict components to be integers, then the space is not Euclidean
(cannot find an average of the vectors [1, 2] and [3, 1])
 29

A hard problem by examples

+ Clustering is a hard problem! 30

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Why is it hard? 31

n Clustering in two dimensions looks easy

n Clustering small amounts of data looks easy

n And in most cases, looks are not deceiving

n Many applications involve not 2, but 10 or 10,000 dimensions

n High-dimensional spaces look different: Almost all pairs of points

are at about the same distance

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Clustering Problem: Galaxies 32

n A catalog of 2 billion “sky objects” represents objects by

their radiation in 7 dimensions (frequency bands)

n Problem: Cluster into similar objects, e.g., galaxies, nearby

stars, quasars, etc.

n Sloan Digital Sky Survey

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Clustering Problem: Music CDs 33

n Intuitively: Music divided into categories, and customers

prefer a few categories
n But what are categories really?

n Represent a CD by a set of customers who bought it:

n Similar CDs have similar sets of customers, and vice-versa

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Clustering Problem: Music CDs 34

Space of all CDs:

n Think of a space with one dim. for each customer

n Values in a dimension may be 0 or 1 only
n A CD is a point in this space (x1, x2,…, xk), where xi = 1 iff the i th customer
bought the CD

n For Amazon, the dimension is tens of millions

n Task: Find clusters of similar CDs

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Clustering Problem: Documents 35

Finding topics:

n Represent a document by a vector

(x1, x2,…, xk), where xi = 1 iff the i th word
(in some order) appears in the document
n It actually doesn’t matter if k is infinite; i.e., we don’t limit the set of
words

n Documents with similar sets of words may be about the

same topic

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Cosine, Jaccard, and Euclidean 36

n As with CDs we have a choice when we think of documents

as sets of words or shingles:
n Sets as vectors: Measure similarity by the cosine distance
n Sets as sets: Measure similarity by the Jaccard distance
n Sets as points: Measure similarity by Euclidean distance

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 37

Clustering methods
 Overview (1) 38

n Agglomerative (bottom up):

n Initially, each point is a cluster
n Repeatedly combine the two
“nearest” clusters into one
n Divisive (top down):
n Start with one cluster and
recursively split it

n Point assignment:
n Maintain a set of clusters
n Points belong to “nearest” cluster

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Overview (2) 39

n Euclidean space, or whether with arbitrary distance measure.

n In a Euclidean space it is possible to summarize a collection of points by their
centroid – the average of the points
n In a non-Euclidean space, there is no notion of a centroid, and we are forced to
develop another way to summarize clusters

n Data is small enough to fit in main memory, or data must reside in secondary
memory
n Algorithms for large amounts of data often must take shortcuts, since it is infeasible
to look at all pairs of points, for example
n It is also necessary to summarize clusters in main memory, since we cannot hold all
the points of all the clusters in main memory at the same time
 40

Hierarchical clustering
+ Hierarchical Clustering 41

n Key operation:
Repeatedly combine
two nearest clusters

n Three important questions:

1) How do you represent a cluster of more than one point?
2) How do you determine the “nearness” of clusters?
3) When to stop combining clusters?

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Hierarchical Clustering 42

Key operation: Repeatedly combine two nearest clusters

(1) How to represent a cluster of many points?

n Key problem: As you merge clusters, how do you represent the “location” of each
cluster, to tell which pair of clusters is closest?
n Euclidean case: each cluster has a centroid = average of its (data)points

(2) How to determine “nearness” of clusters?

n Measure cluster distances by distances of centroids

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Hierarchical clustering 43

(5,3)
o
(1,2)
o
x (1.5,1.5) x (4.7,1.3)
o (2,1)
x (1,1) o (4,1)
x (4.5,0.5)
o (0,0) o (5,0)

Data:
o … data point
x … centroid Dendrogram

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Stopping strategies 44

n Knowledge or belief, about how many clusters there are in the data
n For example, if we are told that the data about dogs is taken from Chihuahuas,
Dachshunds, and Beagles
n Then we know to stop when there are three clusters left

n Stop combining when at some point the best combination of existing clusters
produces a cluster that is “inadequate”

n Continue clustering until there is only one cluster.

n meaningless to return a single cluster consisting of all the points.
n Rather, return the tree representing the way in which all the points were combined.
n Good sense in applications, in which the points are genomes of different species, and the
distance measure reflects the difference in the genome.
n The tree represents the evolution of these species, that is, the likely order in which two
species branched from a common ancestor
+ And in the Non-Euclidean Case? 45

What about the Non-Euclidean case?

n The only “locations” we can talk about are the points themselves
n i.e., there is no “average” of two points
n Distances cannot be based on “location” of points

n Approach 1:
(1) How to represent a cluster of many points?
clustroid = (data)point “closest” to other points
(2) How do you determine the “nearness” of clusters? Treat clustroid as if
it were centroid, when computing inter-cluster distances

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 “Closest” Point? 46

n (1) How to represent a cluster of Centroid

many points? Datapoint
clustroid = point “closest” to other X
points
Clustroid
n Possible meanings of “closest”: Cluster on
n Smallest maximum distance to
3 datapoints
other points
n Smallest average distance to other
points min ∑ d ( x, c)2
Centroid is the avg. of all c
n Smallest sum of squares of x∈C
distances to other points (data)points in the cluster. This
means centroid is an “artificial” point.
n For distance metric d clustroid c Clustroid is an existing (data)point
of cluster C is: that is “closest” to all other points in
the cluster.

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Defining “Nearness” of Clusters 47

(2) How do you determine the “nearness” of clusters?

n Approach 2:
Intercluster distance = minimum of the distances between any two
points, one from each cluster
n Approach 3:
Pick a notion of “cohesion” of clusters, e.g., maximum distance from
the clustroid
n Merge clusters whose union is most cohesive

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Cohesion 48

n Approach 3.1: Use the diameter of the merged cluster =

maximum distance between points in the cluster

n Approach 3.2: Use the average distance between points in the

cluster

n Approach 3.3: Use a density-based approach

n Take the diameter or avg. distance, e.g., and divide by the number of
points in the cluster

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Implementation 49

n Naïve implementation of hierarchical clustering:

n At each step, compute pairwise distances between all pairs of
clusters, then merge
n O(N3)

n Careful implementation using priority queue can reduce time to

O(N2 log N)
n Still too expensive for really big datasets that do not fit in memory

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 50

K-means clustering
51
+ k–means Algorithm(s) 52

n Assumes Euclidean space/distance

n Start by picking k, the number of clusters

n Initialize clusters by picking one point per cluster

n Example: Pick one point at random, then k-1 other points, each as
far away as possible from the previous points

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Populating Clusters 53

1) For each point, place it in the cluster whose current centroid it is nearest

2) After all points are assigned, update the locations of centroids of the k
clusters

3) Reassign all points to their closest centroid

n Sometimes moves points between clusters

n Repeat 2 and 3 until convergence

n Convergence: Points don’t move between clusters and centroids stabilize

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Assigning Clusters 54

x
x
x
x
x

x x x x x x

x … data point
… centroid
Clusters after round 1
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Example: Assigning Clusters 55

x
x
x
x
x

x x x x x x

x … data point
… centroid
Clusters after round 2
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Example: Assigning Clusters 56

x
x
x
x
x

x x x x x x

x … data point
… centroid
Clusters at the end
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
 Getting the k right 57

How to select k?

n Try different k, looking at the

change in the average distance Best value
to centroid as k increases of k
Average
distance to
n Average falls rapidly until right centroid k
k, then changes little

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Picking k 58

Too few; x
x
many long
xx x
distances x x
to centroid. x x x x x
x x x x x
x xx x xx x
x x x x
x x

x x x
x x x x
x x x
x

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Picking k 59

x
Just right; x
distances xx x
rather short. x x
x x x x x
x x x x x
x xx x xx x
x x x x
x x

x x x
x x x x
x x x
x

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Picking k 60

Too many; x
little improvement x
in average xx x
distance. x x
x x x x x
x x x x x
x xx x xx x
x x x x
x x

x x x
x x x x
x x x
x

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 61

The BFR algorithm

Extension of k-means to large data
+ BFR Algorithm 62

n BFR [Bradley-Fayyad-Reina] is a
variant of k-means designed to
handle very large (disk-resident) data sets

n Assumes that clusters are normally distributed around a centroid in a

Euclidean space
n Standard deviations in different
dimensions may vary
n Clusters are axis-aligned ellipses

n Efficient way to summarize clusters

(want memory required O(clusters) and not O(data))

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ BFR Algorithm 63

n Points are read from disk one main-memory-full at a time

n Most points from previous memory loads are summarized by

simple statistics

n To begin, from the initial load we select the initial k centroids by

some sensible approach:
n Take k random points
n Take a small random sample and cluster optimally
n Take a sample; pick a random point, and then
k–1 more points, each as far from the previously selected points as
possible

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Three Classes of Points 64

3 sets of points which we keep track of:

n Discard set (DS):

n Points close enough to a centroid to be summarized

n Compression set (CS):

n Groups of points that are close together but not close to any existing centroid
n These points are summarized, but not assigned to a cluster

n Retained set (RS):

n Isolated points waiting to be assigned to a compression set

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ BFR: “Galaxies” Picture 65
Points in
the RS

Compressed sets.
Their points are in
the CS.

A cluster. Its points The centroid

are in the DS.
Discard set (DS): Close enough to a centroid to be summarized
Compression set (CS): Summarized, but not assigned to a cluster
Retained set (RS): Isolated points
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Summarizing Sets of Points 66

For each cluster, the discard set (DS) is summarized by:

n The number of points, N

n The vector SUM, whose ith component is the sum of the

coordinates of the points in the ith dimension

n The vector SUMSQ: ith component = sum of squares of

coordinates in ith dimension

A cluster.
All its points are in the DS. The centroid
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Summarizing Points: Comments 67

n 2d + 1 values represent any size cluster

n d = number of dimensions

n Average in each dimension (the centroid) can be calculated as SUMi / N

n SUMi = ith component of SUM

n Variance of a cluster’s discard set in dimension i is: (SUMSQi / N) – (SUMi / N)2

n And standard deviation is the square root of that

n Next step: Actual clustering

Note: Dropping the “axis-aligned” clusters assumption would require

storing full covariance matrix to summarize the cluster. So, instead of
SUMSQ being a d-dim vector, it would be a d x d matrix, which is too
big!
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ The “Memory-Load” of Points 68

Processing the “Memory-Load” of points (1):

1) Find those points that are “sufficiently close” to a cluster centroid and add
those points to that cluster and the DS
n These points are so close to the centroid that they can be summarized and then
discarded

2) Use any main-memory clustering algorithm to cluster the remaining points

and the old RS
n Clusters go to the CS; outlying points to the RS

Discard set (DS): Close enough to a centroid to be summarized.

Compression set (CS): Summarized, but not assigned to a cluster
Retained set (RS): Isolated points

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ The “Memory-Load” of Points 69

Processing the “Memory-Load” of points (2):

3) DS set: Adjust statistics of the clusters to account for the new points
n Add Ns, SUMs, SUMSQs

4) Consider merging compressed sets in the CS

5) If this is the last round, merge all compressed sets in the CS and all RS
points into their nearest cluster

Discard set (DS): Close enough to a centroid to be summarized.

Compression set (CS): Summarized, but not assigned to a cluster
Retained set (RS): Isolated points

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ BFR: “Galaxies” Picture 70
Points in
the RS

Compressed sets.
Their points are in
the CS.

A cluster. Its points The centroid

are in the DS.
Discard set (DS): Close enough to a centroid to be summarized
Compression set (CS): Summarized, but not assigned to a cluster
Retained set (RS): Isolated points

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ A Few Details… 71

n Q1) How do we decide if a point is “close enough” to a

cluster that we will add the point to that cluster?

n Q2) How do we decide whether two compressed sets (CS)

deserve to be combined into one?

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ How Close is Close Enough? 72

n Q1) We need a way to decide whether to put a new point into

a cluster (and discard)

n BFR suggests two ways:

n The Mahalanobis distance is less than a threshold
n High likelihood of the point belonging to currently nearest
centroid

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Mahalanobis Distance 73

n Normalized Euclidean distance from centroid

n For point (x1, …, xd) and centroid (c1, …, cd)

1. Normalize in each dimension: yi = (xi - ci) / σi
2. Take sum of the squares of the yi
3. Take the square root
* -
𝑥' − 𝑐'
𝑑 𝑥, 𝑐 = &
𝜎'
'+, σ i … standard deviation of points in
the cluster in the ith dimension

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Mahalanobis Distance 74

n If clusters are normally distributed in d dimensions, then after

transformation, one standard deviation = 𝒅
n i.e., 68% of the points of the cluster will
have a Mahalanobis distance < 𝒅

n Accept a point for a cluster if

its M.D. is < some threshold,
e.g. 2 standard deviations

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Picture: Equal M.D. Regions 75

n Euclidean vs. Mahalanobis distance

Contours of equidistant points from the origin

Uniformly distributed points, Normally distributed points, Normally distributed points,

Euclidean distance Euclidean distance Mahalanobis distance
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
+ Should 2 CS clusters be combined? 76

Q2) Should 2 CS subclusters be combined?

n Compute the variance of the combined subcluster

n N, SUM, and SUMSQ allow us to make that calculation quickly

n Combine if the combined variance is below some threshold

n Many alternatives: Treat dimensions differently, consider density

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 77

The CURE algorithm

Extension of k-means to clusters of arbitrary shapes
 The CURE Algorithm 78

n Problem with BFR/k-means: Vs.

n Assumes clusters are normally
distributed in each dimension
n And axes are fixed – ellipses at
an angle are not OK

n CURE (Clustering Using REpresentatives):

n Assumes a Euclidean distance
n Allows clusters to assume any shape
n Uses a collection of representative
points to represent clusters

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Stanford Salaries 79

h h

h
e e
e
h e
e e h
e e e e

salary h
e
h
h
h h
h h h

age

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Starting CURE 80

2 Pass algorithm. Pass 1:

0) Pick a random sample of points that fit in main memory

1) Initial clusters:
n Cluster these points hierarchically – group nearest points/clusters

2) Pick representative points:

n For each cluster, pick a sample of points, as dispersed as possible
n From the sample, pick representatives by moving them (say) 20% toward
the centroid of the cluster

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Initial Clusters 81

h h

h
e e
e
h e
e e h
e e e e
h
salary e
h
h
h h
h h h

age

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Pick Dispersed Points 82

h h

h
e e
e
h e
e e h
e e e e
h
salary e Pick (say) 4
h
h remote points
h h for each
h h h cluster.

age

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+ Example: Pick Dispersed Points 83

h h

h
e e
e
h e
e e h
e e e e
h
salary e Move points
h
h (say) 20%
h h toward the
h h h centroid.

age

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 Finishing CURE 84

Pass 2:

n Now, rescan the whole dataset

and visit each point p in the data
set

n Place it in the “closest cluster”

n Normal definition of “closest”: p
Find the closest representative
to p and assign it to
representative’s cluster

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

 85

Final remarks
+ Summary 86

n Clustering: Given a set of points, with a notion of distance

between points, group the points into some number of clusters

n Algorithms:
n Agglomerative hierarchical clustering:
n Centroid and clustroid
n k-means:
n Initialization, picking k

n BFR
n CURE

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

87