Clustering Techniques

• Clustering: introduction
• Clustering approaches
• Exclusive clustering: K-means algorithm
• Agglomerative clustering: Hierarchical algorithm
• Overlapping clustering: Fuzzy C-means algorithm
• Cluster validity problem
• Cluster quality criteria: Davies-Bouldin index
 Clustering (introduction)
• Clustering is a type of unsupervised machine learning
• It is distinguished from supervised learning by the fact that there is not a
priori output (i.e. no labels)
– The task is to learn the classification/grouping from the data
• A cluster is a collection of objects which are similar in some way
Eg: a group of people clustered based on their height and weight
• Normally, clusters are created using distance measures
– Two or more objects belong to the same cluster if they are “close” according to a given
distance (in this case geometrical distance like Euclidean or Manhattan)
• Another measure is conceptual
– Two or more objects belong to the same cluster if this one defines a concept common to
all that objects
– In other words, objects are grouped according to their fit to descriptive concepts, not
according to simple similarity measures
 Clustering (introduction)
• Example: using distance based clustering

• This was easy but how if you had to create 4 clusters?

• Some possibilities are shown below but which is correct?
 Clustering (introduction – ctd)
• So, the goal of clustering is to determine the intrinsic grouping in a set of
unlabeled data
• But how to decide what constitutes a good clustering?
• It can be shown that there is no absolute “best” criterion which would be
independent of the final aim of the clustering
• Consequently, it is the user which must supply this criterion, to suit the
application
• Some possible applications of clustering
– data reduction – reduce data that are homogeneous (similar)
– find “natural clusters” and describe their unknown properties
– find useful and suitable groupings
– find unusual data objects (i.e. outlier detection)
 Clustering – an early application example
• Hertzsprung-Russell diagram clustering stars by temperature
and luminosity
– Two astonomers in the early 20th
century clustered stars into three
groups using scatter plots
– Main sequence: 80% of stars
spending active life converting
hydrogen to helium through
nuclear fusion
– Giants: Helium fusion or fusion
stops: generates great deal of light
– White dwarf: Core cools off

Diagram from Google Images

 Clustering – Major approaches
• Exclusive (partitioning)
– Data are grouped in an exclusive way, one data can only belong to one cluster
– Eg: K-means
• Agglomerative
– Every data is a cluster initially and iterative unions between the two nearest clusters
reduces the number of clusters
– Eg: Hierarchical clustering
• Overlapping
– Uses fuzzy sets to cluster data, so that each point may belong to two or more clusters
with different degrees of membership
– In this case, data will be associated to an appropriate membership value
– Eg: Fuzzy C-Means
• Probabilistic
– Uses probability distribution measures to create the clusters
– Eg: Gaussian mixture model clustering, which is a variant of K-means
– Will not be discussed in this course
 Exclusive (partitioning) clustering
• Aim: Construct a partition of a database D of N objects into a set of K
clusters
• Method: Given a K, find a partition of K clusters that optimises the chosen
partitioning criterion

• K-means (MacQueen’67) is one of the commonly used clustering

algorithm
• It is a heuristic method where each cluster is represented by the centre of
the cluster (i.e. the centroid)

• Note: One and two dimensional (i.e. with one and two features) data are
used in this lecture for simplicity of explanation
• In general, clustering algorithms are used with much higher dimensions
 K-means clustering algorithm

Given K, the K-means algorithm is implemented in four steps:

1. Choose K points at random as cluster centres (centroids)
2. Assign each instance to its closest cluster centre using
certain distance measure (usually Euclidean or Manhattan)
3. Calculate the centroid of each cluster, use it as the new
cluster centre (one measure of centroid is mean)
4. Go back to Step 2, stop when cluster centres do not change
any more
 K-means – an example

• Say, we have the data: {20, 3, 9, 10, 9, 3, 1, 8, 5, 3, 24, 2, 14, 7, 8, 23, 6, 12,
18} and we are asked to use K-means to cluster these data into 3 groups
• Assume we use Manhattan distance*
• Step one: Choose K points at random to be cluster centres
• Say 6, 12, 18 are chosen

*note for one dimensional data, Manhattan

distance=Euclidean distance
 K-means – an example (ctd)

• Step two: Assign each instance to its closest cluster centre using
Manhattan distance
• For instance:
– 20 is assigned to cluster 3
– 3 is assigned to cluster 1
 K-means – Example (ctd)

• Step two continued: 9 can be assigned to cluster 1, 2 but let

us say that it is arbitrarily assigned to cluster 2
• Repeat for all the rest of the instances
 K-Means – Example (ctd)

• And after exhausting all instances…

• Step three: Calculate the centroid (i.e. mean) of each cluster, use it as the new
cluster centre

• End of iteration 1
• Step four: Iterate (repeat steps 2 and 3) until the cluster centres do not change any
more
Exercise
 Distance from Distance from Distance from
Point belongs
Given Points center (2, 10) of center (6, 6) of center (1.5, 3.5) of
to Cluster
Cluster-01 Cluster-02 Cluster-03

A1(2, 10) 0 8 7 C1

A2(2, 5) 5 5 2 C3

A3(8, 4) 12 4 7 C2

A4(5, 8) 5 3 8 C2

A5(7, 5) 10 2 7 C2

A6(6, 4) 10 2 5 C2

A7(1, 2) 9 9 2 C3

A8(4, 9) 3 5 8 C1
 K - means

• Strengths
– Relatively efficient: where N is no. objects, K is no. clusters, and T is
no. iterations. Normally, K, T << N.
– Procedure always terminates successfully (but see below)

• Weaknesses
– Does not necessarily find the most optimal configuration
– Significantly sensitive to the initial randomly selected cluster centres
– Applicable only when mean is defined (i.e. can be computed)
– Need to specify K, the number of clusters, in advance
 K-means in MATLAB
• Use the built in ‘kmeans’
function
• Example, for the data that we
saw earlier
• The ‘ind’ is the output that
gives the cluster index of the
data, while ‘c’ is the final
cluster centres
• For Manhanttan distance, use
…‘distance’, ‘cityblock’…
• For Euclidean (default), no
need to specify distance
measure
Nearest Neighbour Clustering
 Agglomerative clustering
• K-means approach starts out with a fixed number of clusters
and allocates all data into the exactly number of clusters
• But agglomeration does not require the number of clusters K
as an input
• Agglomeration starts out by forming each data as one cluster
– So, data of N object will have N clusters
• Next by using some distance (or similarity) measure, reduces
the number so clusters (one in each iteration) by merging
process
• Finally, we have one big cluster than contains all the objects
• But then what is the point of having one big cluster in the
end?
 Dendrogram (ctd)
• While merging cluster one by one, we can draw a tree diagram known as
dendrogram
• Dendrograms are used to represent agglomerative clustering
• From dendrograms, we can get any number of clusters
• Eg: say we wish to have 2 clusters, then cut the top one link
– Cluster 1: q, r
– Cluster 2: x, y, z, p
• Similarly for 3 clusters, cut 2 top links
– Cluster 1: q, r
– Cluster 2: x, y, z
– Cluster 3: p

A dendrogram example
 Hierarchical clustering - algorithm
• Hierarchical clustering algorithm is a type of agglomerative clustering

• Given a set of N items to be clustered, hierarchical clustering algorithm:

1. Start by assigning each item to its own cluster, so that if you have N items, you
now have N clusters, each containing just one item
2. Find the closest distance (most similar) pair of clusters and merge them into a
single cluster, so that now you have one less cluster
3. Compute pairwise distances between the new cluster and each of the old clusters
4. Repeat steps 2 and 3 until all items are clustered into a single cluster of size N
5. Draw the dendogram, and with the complete hierarchical tree, if you want K
clusters you just have to cut the K-1 top links

Note any distance measure can be used: Euclidean, Manhattan, etc

 Hierarchical clustering algorithm– step 3
• Computing distances between clusters for Step 3 can be implemented in
different ways:
– Single-linkage clustering
– The distance between one cluster and another cluster is computed as the shortest
distance from any member of one cluster to any member of the other cluster
– Complete-linkage clustering
– The distance between one cluster and another cluster is computed as the greatest
distance from any member of one cluster to any member of the other cluster
– Centroid clustering
– The distance between one cluster and another cluster is computed as the distance from
one cluster centroid to the other cluster centroid
Hierarchical clustering algorithm– step 3
 Hierarchical clustering – an example
• Assume X=[3 7 10 17 18 20]
1. There are 6 items, so create 6 clusters initially

2. Compute pairwise distances of clusters (assume Manhattan distance)

The closest clusters are 17 and 18 (with distance=1), so merge these two clusters
together

3. Repeat step 2 (assume single-linkage):

The closest clusters are cluster17,18 to cluster20 (with distance |18-20|=2),
so merge these two clusters together
 Hierarchical clustering – an example (ctd)
• Go on repeat cluster mergers until one big cluster remains

• Draw dendrogram (draw it in reverse of the cluster mergers) – remember

the height of each cluster corresponds to the manner of cluster
agglomeration
Complete Link: Distance between two clusters is the longest distance
between a pair of elements from the two clusters
 p1 0
p2 0.24 0
Distance matrix
p3 0.22 0.15 0
p4 0.37 0.20 0.15 0
p5 0.34 0.14 0.28 0.29 0
p6 0.23 0.25 0.11 0.22 0.39 0
p1 p2 p3 p4 p5 p6
p1 0
p2 0.24 0
p3 0.22 0.15 0
p4 0.37 0.20 0.15 0
p5 0.34 0.14 0.28 0.29 0
p6 0.23 0.25 0.11 0.22 0.39 0
p1 p2 p3 p4 p5 p6
p1 0
p2 0.24 0
p3 0.22 0.15 0
p4 0.37 0.20 0.15 0
p5 0.34 0.14 0.28 0.29 0
p6 0.23 0.25 0.11 0.22 0.39 0
p1 p2 p3 p4 p5 p6
p1 0
p2 0.24 0
p3 0.22 0.15 0
p4 0.37 0.20 0.15 0
p5 0.34 0.14 0.28 0.29 0
p6 0.23 0.25 0.11 0.22 0.39 0
p1 p2 p3 p4 p5 p6
Dendrogram
 Hierarchical clustering – an example (ctd) –using MATLAB
• %Hierarchical clustering example
• X=[3 7 10 17 18 20]; %data
• Y=pdist(X', 'cityblock'); %compute pairwise Manhattan distances
• Z=linkage(Y, 'single'); %do clustering using single-linkage method
• dendrogram(Z); %draw dendrogram –note only indices are shown

4 5 6 1 2 3
 Comparing agglomerative vs exclusive clustering

• Agglomerative - advantages
– Preferable for detailed data analysis
– Provides more information than exclusive clustering
– We can decide on any number of clusters without the need to redo
the algorithm –in exclusive clustering, K has to be decided first, if a
different K is used, then need to redo the whole exclusive clustering
algorithm
– One unique answer
• Disadvantages
– Less efficient than exclusive clustering
– No backtracking, i.e. can never undo previous steps
 Overlapping clustering –Fuzzy C-means algorithm
• Both agglomerative and exclusive clustering allows one data to be in one
cluster only
• Fuzzy C-means (FCM) is a method of clustering which allows one piece of
data to belong to more than one cluster
• In other words, each data is a member of every cluster but with a certain
degree known as membership value
• This method (developed by Dunn in 1973 and improved by Bezdek in 1981)
is frequently used in pattern recognition
• Fuzzy partitioning is carried out through an iterative procedure that updates
membership uij and the cluster centroids cj by
N
1
uij  2 u m
x
ij i
C
 || xi  c j ||  m 1 cj  i 1

 
k 1  || x  c ||

u
N
m
i k  ij
i 1

where m > 1, and represents the degree of fuzziness (typically, m=2)

 Overlapping clusters?
• Using both agglomerative and exclusive clustering methods, data X1 will be member of
cluster 1 only while X2 will be member of cluster 2 only
• However, using FCM, data X can be member of both clusters
• FCM uses distance measure too, so the further data is from that cluster centroid, the
smaller the membership value will be
• For example, membership value for X1 from cluster 1, u11=0.73 and membership value for
X1 from cluster 2, u12=0.27
• Similarly, membership value for X2 from cluster 1, u21=0.2 and membership value for X2
from cluster 2, u22=0.8

• Note: membership values are in the range 0 to 1 and membership values for each data
from all the clusters will add to 1
 Fuzzy C-means algorithm
• Choose the number of clusters, C and m, typically 2
1. Initialise all uij, membership values randomly – matrix U0
N
2. At step k: Compute centroids, cj using u m
x
ij i
cj  i 1
N

u
i 1
m
ij

1
3. Compute new membership values, uij using uij  2
C
 || xi  c j ||  m 1
 
k 1  || x  c ||

i k 
4. Update Uk+1  Uk
5. Repeat steps 2-4 until change of membership values is very small, Uk+1-Uk
< where  is some small value, typically 0.01

Note: || means Euclidean distance, | means Manhattan

However, if the data is one dimensional (like the examples here), Euclidean distance =
Manhattan distance
 Fuzzy C-means algorithm – an example
• X=[3 7 10 17 18 20] and assume C=2
• Initially, set U randomly U  0.1 0.2 0.6 0.3 0.1 0.5
0.9 0.8 0.4 0.7 0.9 0.5

u m
x
ij i
• Compute centroids, cj using cj  i 1
N
, assume m=2
u
i 1
m
ij

• c1=13.16; c2=11.81
1
uij 
• Compute new membership values, uij using 2
C
 || xi  c j ||  m 1

• New U:  
k 1  || xi  ck || 

0.43 0.38 0.24 0.65 0.62 0.59
U
0.57 0.62 0.76 0.35 0.38 0.41

• Repeat centroid and membership computation until changes in

membership values are smaller than say 0.01
 Fuzzy C-means algorithm –using MATLAB
• Using ‘fcm’ function in
MATLAB

• The final membership

values, U gives an indication
on similarity of each item to
the clusters

• For eg: item 3 (no. 10) is

more similar to cluster 1
compared to cluster 2 but
item 2 (no. 7) is even more
similar to cluster 1
 Fuzzy C-means algorithm –using MATLAB
• ‘fcm’ function requires Fuzzy
Logic toolbox
• So, using MATLAB but without
‘fcm’ function:

u m
x
ij i
cj  i 1
N

u
i 1
m
ij

1
uij  2
C
 || xi  c j ||  m 1
 
k 1  || xi  ck

|| 
 Clustering validity problem
• Problem 1
• A problem we face in clustering is to decide the optimal number
of clusters that fits a data set
• Problem 2
• The various clustering algorithms behave in a different way
depending on
– the features of the data set (geometry and density distribution of clusters)
– the input parameters values (eg: for K-means, initial cluster choices
influence the result)
• So, how do we know which clustering method is better/suitable?
• We need a clustering quality criteria
 Clustering validity problem
• In general, good clusters, should have
– High intra–cluster similarity, i.e. low variance among intra-cluster members
where variance for x is defined by
1 N
var( x)  
N  1 i 1
( xi  x ) 2
with x as the mean of x
• For eg: if x=[2 4 6 8], then so var(x)=6.67
x 5

• Computing intra-cluster similarity is simple

• For eg: for the clusters shown

• var(cluster1)=2.33 while var(cluster2)=12.33

• So, cluster 1 is better than cluster 2
• Note: use ‘var’ function in MATLAB to compute variance
 Clustering Quality Criteria
• But this does not tell us anything about how good is the overall clustering
or on the suitable number of clusters needed!
• To solve this, we also need to compute inter-cluster variance
• Good clusters will also have low inter–cluster similarity, i.e. high variance
among inter-cluster members in addition to high intra–cluster similarity, i.e.
low variance among intra-cluster members
• One good measure of clustering quality is Davies-Bouldin index
• The others are
– Dunn’s Validity Index
– Silhouette method
– C–index
– Goodman–Kruskal index
• So, we compute DB index for different number of clusters, K and the best
value of DB index tells us on the appropriate K value or on how good is the
clustering method
 Davies-Bouldin index
• It is a function of the ratio of the sum of within-cluster (i.e. intra-cluster) scatter
to between cluster (i.e. inter-cluster) separation
• Because a low scatter and a high distance between clusters lead to low values of
Rij , a minimization of DB index is desired
• Let C={C1,….., Ck} be a clustering of a set of N objects:

1 k
DB  . Ri
k i 1

with Ri  maxand
Rij var(Ci )  var(C j )
Rij 
j 1,..k ,i  j i j || ci  c j ||

where Ci is the ith cluster and ci is the centroid for cluster i

Numerator of Rij is a measure of intra-cluster similarity while the denominator is a

measure of inter-cluster separation
Note, Rij=Rji
 Davies-Bouldin index example
For eg: for the clusters shown
var(Ci )  var(C j )
•Compute Rij 
i j || ci  c j ||
Note, variance of one element is
zero and centroid is simply the
•var(C1)=0, var(C2)=4.5, var(C3)=2.33 element itself

Centroid is simply the mean here, so c1=3, c2=8.5, c3=18.33

•So, R12=1, R13=0.152, R23=0.797

Ri  max Rij
•Now, compute j 1,..k ,i  j

•R1=1 (max of R12 and R13); R2=1 (max of R21 and R23); R3=0.797 (max of R31 and
R32) k
1
Finally, compute DB  . Ri
k i 1
•DB=0.932
 Davies-Bouldin index example (ctd)
• For eg: for the clusters shown

var(Ci )  var(C j )
• Compute Rij 
i j || ci  c j ||

• Only 2 clusters here

• var(C1)=12.33 while var(C2)=2.33; c1=6.67 while c2=18.33
• R12=1.26
Ri  max Rij
• Now compute j 1,.. k ,i  j

• Since we have only 2 clusters here, R1=R12=1.26; R2=R21=1.26

• Finally, compute
• DB=1.26
 Davies-Bouldin index example (ctd)
• DB with 2 clusters=1.26, with 3 clusters=0.932
• So, K=3 is better than K=2 (since DB smaller, better clusters)
• In general, we will repeat DB index computation for all cluster
sizes from 2 to N-1
• So, if we have 10 data items, we will do clustering with K=2,
…..9 and then compute DB for each value of K
– K=10 is not done since each item is its own cluster
• Then, we decide the best clustering size (and the best set of
clusters) would be the one with minimum values of DB index
 The k-Medoids algorithm
Now, we shall study a variant of partitioning algorithm called k-Medoids
algorithm.

Motivation: We have learnt that the k-Means algorithm is sensitive to outliers

because an object with an “extremely large value” may substantially distort the
distribution. The effect is particularly exacerbated due to the use of the SSE (sum-
of-squared error) objective function. The k-Medoids algorithm aims to diminish
the effect of outliers.

Basic concepts:
•The basic concepts of this algorithm is to select an object as a cluster center (one
representative object per cluster) instead of taking the mean value of the objects
in a cluster (as in k-Means algorithm).
•We call this cluster representative as a cluster medoid or simply medoid.
1.Initially, it selects a random set of k objects as the set of medoids.
2.Then at each step, all objects from the set of objects, which are not currently
medoids are examined one by one to see if they should be medoids.

CS 40003: Data Analytics 65

The k-Medoids algorithm

CS 40003: Data Analytics 66

PAM (Partitioning around Medoids)

CS 40003: Data Analytics 67

PAM (Partitioning around Medoids)

Fig 16.7: Illustration of PAM

CS 40003: Data Analytics 68

PAM (Partitioning around Medoids)

CS 40003: Data Analytics 69

PAM (Partitioning around Medoids)

CS 40003: Data Analytics 70

Comments on PAM

CS 40003: Data Analytics 71

• Step1: Initialize number of clusters k = 2. Let the randomly
selected two medoids be M1 (4,6) and M2 (6,7).
• Step2: Calculate Cost. The dissimilarity of each non-medoid
object with the medoids is calculated and tabulated:
• For simplicity, we will take here Manhattan distance as
our cost measure:

• Each object is assigned to the cluster of that medoid

whose dissimilarity is less.
The objects 0,1 go to cluster M1 and object 4 go to
cluster M2.
• Each object is assigned to the cluster of that medoid whose
dissimilarity is less.
The objects 0,1 go to cluster M1 and object 4 go to cluster M2.
• The total cost:
(dj) = (1+1) + (2) = 4
• Step3: Now randomly select one non-medoid object and recalculate
the Cost. Let the randomly selected non-medoid object be (4,5) and
take it as a candidate for medoid. The dissimilarity of each non-
medoid object with the medoids M1(4,5) and M2(6,7) is calculated
and tabulated:

• Each object is assigned to the cluster of that medoid whose

dissimilarity is less.
The objects 0,2 go to cluster M1 and object 4 go to cluster M2.
The total cost:
(di) = (2+1) + (2) = 5
• Since, di > dj which is not satisfying the condition di < dj. So,
We undo the swap. Hence (4,6) and (6,7) are the final
medoids. The Clustering would be in the following way:
Complexity
•Total possibilities for a node: k*(n-k)
•Cost to compute E: (n-k)
•Total cost for each node: k*(n-k)^2
•Since time complexity for each iteration:; where n is the number
of objects in data, k is the number of clusters. Therefore, PAM
works efficiently for small data sets but does not scale well for
large data sets.