Clustering : Unsupervised

Learning
By Robert – just started learning!
Notes adopted from Jiawei Han:- concepts and techniques of data
mining

January 28, 2024 Data Mining: Concepts and Techniques 1

 . Cluster Analysis
1. What is Cluster Analysis?

2. Types of Data in Cluster Analysis

3. Other Clustering Methods

4. Summary

5. References

January 28, 2024 Data Mining: Concepts and Techniques 2

 What is Cluster Analysis?
• Cluster: a collection of data objects
• Similar to one another within the same cluster
• Dissimilar to the objects in other clusters
• Cluster analysis
• Finding similarities between data according to the characteristics
found in the data and grouping similar data objects into clusters
• Unsupervised learning: no predefined classes
• Typical applications
• As a stand-alone tool to get insight into data distribution
• As a preprocessing step for other algorithms

January 28, 2024 Data Mining: Concepts and Techniques 3

 Clustering: Rich Applications and Multidisciplinary Efforts

• Pattern Recognition
• Spatial Data Analysis
• Create thematic maps in GIS by clustering feature spaces
• Detect spatial clusters or for other spatial mining tasks
• Image Processing
• Economic Science (especially market research)
• WWW
• Document classification
• Cluster Weblog data to discover groups of similar access
patterns
January 28, 2024 Data Mining: Concepts and Techniques 4
 Examples of Clustering Applications

• Marketing: Help marketers discover distinct groups in their customer bases,

and then use this knowledge to develop targeted marketing programs

• Land use: Identification of areas of similar land use in an earth observation

database

• Insurance: Identifying groups of motor insurance policy holders with a high

average claim cost

• City-planning: Identifying groups of houses according to their house type,

value, and geographical location

• Earth-quake studies: Observed earth quake epicenters should be clustered

along continent faults
January 28, 2024 Data Mining: Concepts and Techniques 5
 Quality: What Is Good Clustering?

• A good clustering method will produce high quality clusters with

• high intra-class similarity
• low inter-class similarity
• The quality of a clustering result depends on both the similarity
measure used by the method and its implementation
• The quality of a clustering method is also measured by its ability
to discover some or all of the hidden patterns

January 28, 2024 Data Mining: Concepts and Techniques 6

 Measure the Quality of Clustering

• Dissimilarity/Similarity metric: Similarity is expressed in terms of

a distance function, typically metric: d(i, j)
• There is a separate “quality” function that measures the
“goodness” of a cluster.
• The definitions of distance functions are usually very different for
interval-scaled, boolean, categorical, ordinal ratio, and vector
variables.
• Weights should be associated with different variables based on
applications and data semantics.
• It is hard to define “similar enough” or “good enough”
• the answer is typically highly subjective.
January 28, 2024 Data Mining: Concepts and Techniques 7
 Requirements of Clustering in Data Mining

• Scalability
• Ability to deal with different types of attributes
• Ability to handle dynamic data
• Discovery of clusters with arbitrary shape
• Minimal requirements for domain knowledge to determine input
parameters
• Able to deal with noise and outliers
• Insensitive to order of input records
• High dimensionality
• Incorporation of user-specified constraints
• Interpretability and usability
January 28, 2024 Data Mining: Concepts and Techniques 8
 . Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10. Constraint-Based Clustering

11. Outlier Analysis

January 28, 2024 12. Summary Data Mining: Concepts and Techniques 9
 Data Structures

 x11 ... x1f ... x1p 

• Data matrix  
• (two modes)  ... ... ... ... ... 
x ... x if ... x ip 
 i1 
 ... ... ... ... ... 
x ... x nf ... x np 
 n1 

• Dissimilarity matrix  0 
 d(2,1) 0 
• (one mode)  
 d(3,1) d ( 3,2) 0 
 
 : : : 
d ( n,1) d ( n,2) ... ... 0

January 28, 2024 Data Mining: Concepts and Techniques 10

 Type of data in clustering analysis

• Interval-scaled variables

• Binary variables

• Nominal, ordinal, and ratio variables

• Variables of mixed types

January 28, 2024 Data Mining: Concepts and Techniques 11

 Interval-valued variables

• Standardize data
• Calculate the mean absolute deviation:
s f  1n (| x1 f  m f |  | x2 f  m f | ... | xnf  m f |)

where m f  1n (x1 f  x2 f  ...  xnf )

.

• Calculate the standardized measurement (z-score)

xif  m f
zif  sf

• Using mean absolute deviation is more robust than using

standard deviation
January 28, 2024 Data Mining: Concepts and Techniques 12
 Similarity and Dissimilarity Between
Objects

• Distances are normally used to measure the similarity or

dissimilarity between two data objects
• Some popular ones include: Minkowski distance:
d (i, j)  q (| x  x |q  | x  x |q ... | x  x |q )
i1 j1 i2 j2 ip jp
where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two p-
dimensional data objects, and q is a positive integer
• If q = 1, d is Manhattan distance

d (i, j) | x  x |  | x  x | ... | x  x |
i1 j1 i2 j 2 ip jp
January 28, 2024 Data Mining: Concepts and Techniques 13
 Similarity and Dissimilarity Between
Objects (Cont.)

• If q = 2, d is Euclidean distance:
d (i, j)  (| x  x |2  | x  x |2 ... | x  x |2 )
i1 j1 i2 j2 ip jp
• Properties
• d(i,j)  0
• d(i,j) = 0
• d(i,j) = d(j,i)
• d(i,j)  d(i,k) + d(k,j)

• Also, one can use weighted distance, parametric Pearson

product moment correlation, or other disimilarity measures

January 28, 2024 Data Mining: Concepts and Techniques 14

Binary Variables
Object j

1 0 sum
• A contingency table for binary
1 a b a b
data Object i
0 c d cd
sum a  c b  d p

• Distance measure for symmetric

d (i, j)  bc
binary variables: a bc  d
• Distance measure for asymmetric d (i, j)  bc
binary variables: a bc
• Jaccard coefficient (similarity
simJaccard (i, j)  a
measure for asymmetric binary a bc
January 28, 2024
variables): Data Mining: Concepts and Techniques 15
 Dissimilarity between Binary Variables

• Example
Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4
Jack M Y N P N N N
Mary F Y N P N P N
Jim M Y P N N N N
• gender is a symmetric attribute
• the remaining attributes are asymmetric binary
• let the values Y and P be set to 1, and the value N be set to 0
01
d ( jack , mary )   0.33
2 01
11
d ( jack , jim )   0.67
111
1 2
d ( jim , mary )   0.75
11 2
January 28, 2024 Data Mining: Concepts and Techniques 16
 Nominal Variables

• A generalization of the binary variable in that it can take more than

2 states, e.g., red, yellow, blue, green
• Method 1: Simple matching
• m: # of matches, p: total # of variables
d (i, j)  p  p
m

• Method 2: use a large number of binary variables

• creating a new binary variable for each of the M nominal states

January 28, 2024 Data Mining: Concepts and Techniques 17

 Ordinal Variables

• An ordinal variable can be discrete or continuous

• Order is important, e.g., rank
• Can be treated like interval-scaled
rif {1,...,M f }
• replace xif by their rank
• map the range of each variable onto [0, 1] by replacing i-th
object in the f-th variable by
rif 1
zif 
M f 1

• compute the dissimilarity using methods for interval-scaled

variables
January 28, 2024 Data Mining: Concepts and Techniques 18
 Ratio-Scaled Variables

• Ratio-scaled variable: a positive measurement on a nonlinear

scale, approximately at exponential scale, such as AeBt or
Ae-Bt
• Methods:
• treat them like interval-scaled variables—not a good choice!
(why?—the scale can be distorted)
• apply logarithmic transformation
yif = log(xif)
• treat them as continuous ordinal data treat their rank as
interval-scaled
January 28, 2024 Data Mining: Concepts and Techniques 19
 Variables of Mixed Types
• A database may contain all the six types of variables
• symmetric binary, asymmetric binary, nominal, ordinal,
interval and ratio
• One may use a weighted formula to combine their effects
 pf  1 ij( f ) d ij( f )
d (i, j ) 
• f is binary or nominal:  p

f  1 ij
(f)

dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise

• f is interval-based: use the normalized distance
• f is ordinal or ratio-scaled
• compute ranks rif and
• and treat zif as interval-scaled r 1
zif 
if

M f 1
January 28, 2024 Data Mining: Concepts and Techniques 20
 Vector Objects

• Vector objects: keywords in documents, gene features

in micro-arrays, etc.
• Broad applications: information retrieval, biologic
taxonomy, etc.
• Cosine measure

• A variant: Tanimoto coefficient

January 28, 2024 Data Mining: Concepts and Techniques 21

 . Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10. Constraint-Based Clustering

11. Outlier Analysis

January 28, 2024 12. Summary Data Mining: Concepts and Techniques 22
 Major Clustering Approaches (I)

• Partitioning approach:
• Construct various partitions and then evaluate them by some criterion, e.g.,
minimizing the sum of square errors
• Typical methods: k-means, k-medoids, CLARANS

• Hierarchical approach:
• Create a hierarchical decomposition of the set of data (or objects) using some
criterion
• Typical methods: Diana, Agnes, BIRCH, ROCK, CAMELEON

• Density-based approach:
• Based on connectivity and density functions
• Typical methods: DBSACN, OPTICS, DenClue

January 28, 2024 Data Mining: Concepts and Techniques 23

 Major Clustering Approaches (II)
• Grid-based approach:
• based on a multiple-level granularity structure
• Typical methods: STING, WaveCluster, CLIQUE

• Model-based:
• A model is hypothesized for each of the clusters and tries to find the best fit of that
model to each other
• Typical methods: EM, SOM, COBWEB

• Frequent pattern-based:
• Based on the analysis of frequent patterns
• Typical methods: pCluster

• User-guided or constraint-based:
• Clustering by considering user-specified or application-specific constraints
January 28, 2024 • Typical methods: COD (obstacles),
Data Mining:constrained clustering
Concepts and Techniques 24
 Typical Alternatives to Calculate the Distance between
Clusters
• Single link: smallest distance between an element in one cluster and an element
in the other, i.e., dis(Ki, Kj) = min(tip, tjq)

• Complete link: largest distance between an element in one cluster and an

element in the other, i.e., dis(K i, Kj) = max(tip, tjq)

• Average: avg distance between an element in one cluster and an element in the
other, i.e., dis(Ki, Kj) = avg(tip, tjq)

• Centroid: distance between the centroids of two clusters, i.e., dis(K i, Kj) = dis(Ci,
Cj)

• Medoid: distance between the medoids of two clusters, i.e., dis(K i, Kj) = dis(Mi,
Mj)

January 28, 2024

• Medoid: one chosen, centrally located
Data Mining: Conceptsobject in the cluster
and Techniques 25
Centroid, Radius and Diameter of a Cluster (for numerical data sets)

• Centroid: the “middle” of a cluster iN 1(t )

Cm  N
ip

• Radius: square root of average distance from any point of the cluster to its
centroid
 N (t  cm ) 2
Rm  i 1 ip
N

• Diameter: square root of average mean squared distance between all pairs
of points in the cluster
 N  N (t  t ) 2
Dm  i 1 i 1 ip iq
N ( N 1)

January 28, 2024 Data Mining: Concepts and Techniques 26

 . Cluster Analysis
1. What is Cluster Analysis?
2. Types of Data in Cluster Analysis
3. A Categorization of Major Clustering Methods
4. Partitioning Methods
5. Hierarchical Methods
6. Density-Based Methods
7. Grid-Based Methods
8. Model-Based Methods
9. Clustering High-Dimensional Data
10. Constraint-Based Clustering

11. Outlier Analysis

January 28, 2024 12. Summary Data Mining: Concepts and Techniques 27
Partitioning Algorithms: Basic Concept

• Partitioning method: Construct a partition of a database D of n objects into a

set of k clusters, s.t., min sum of squared distance

 km1tmiKm (Cm  tmi ) 2

• Given a k, find a partition of k clusters that optimizes the chosen partitioning

criterion
• Global optimal: exhaustively enumerate all partitions
• Heuristic methods: k-means and k-medoids algorithms
• k-means (MacQueen’67): Each cluster is represented by the center of the
cluster
• k-medoids or PAM (Partition around medoids) (Kaufman &
Rousseeuw’87): Each cluster is represented by one of the objects in the
cluster
January 28, 2024 Data Mining: Concepts and Techniques 28
 The K-Means Clustering Method

• Given k, the k-means algorithm is implemented in four

steps:
• Partition objects into k nonempty subsets
• Compute seed points as the centroids of the clusters of
the current partition (the centroid is the center, i.e.,
mean point, of the cluster)
• Assign each object to the cluster with the nearest seed
point
• Go back to Step 2, stop when no more new assignment

January 28, 2024 Data Mining: Concepts and Techniques 29

 The K-Means Clustering Method

• Example
10 10
10
9 9
9
8 8
8
7 7
7
6 6
6
5 5
5
4 4
4
Assign 3 Update 3
3

2 each
2 the 2

1
objects
1

0
cluster 1

0
0
0 1 2 3 4 5 6 7 8 9 10 to most
0 1 2 3 4 5 6 7 8 9 10 means 0 1 2 3 4 5 6 7 8 9 10

similar
center reassign reassign
10 10

K=2 9 9

8 8

Arbitrarily choose K 7 7

object as initial cluster

6 6

5 5

center 4 Update 4

2
the 3

1 cluster 1

0
0 1 2 3 4 5 6 7 8 9 10
means 0
0 1 2 3 4 5 6 7 8 9 10

January 28, 2024 Data Mining: Concepts and Techniques 30

 Comments on the K-Means Method

• Strength: Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t

is # iterations. Normally, k, t << n.
• Comparing: PAM: O(k(n-k)2 ), CLARA: O(ks2 + k(n-k))
• Comment: Often terminates at a local optimum. The global optimum may be
found using techniques such as: deterministic annealing and genetic
algorithms
• Weakness
• Applicable only when mean is defined, then what about categorical data?
• Need to specify k, the number of clusters, in advance
• Unable to handle noisy data and outliers
• Not suitable to discover clusters with non-convex shapes

January 28, 2024 Data Mining: Concepts and Techniques 31

 Variations of the K-Means Method

• A few variants of the k-means which differ in

• Selection of the initial k means

• Dissimilarity calculations

• Strategies to calculate cluster means

• Handling categorical data: k-modes (Huang’98)

• Replacing means of clusters with modes

• Using new dissimilarity measures to deal with categorical objects

• Using a frequency-based method to update modes of clusters

• A mixture of categorical and numerical data: k-prototype method

January 28, 2024 Data Mining: Concepts and Techniques 32
 What Is the Problem of the K-Means Method?

• The k-means algorithm is sensitive to outliers !

• Since an object with an extremely large value may substantially distort the
distribution of the data.

• K-Medoids: Instead of taking the mean value of the object in a cluster as a

reference point, medoids can be used, which is the most centrally located
object in a cluster.
10 10
9 9
8 8
7 7
6 6
5 5
4 4
3 3
2 2
1 1
0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10

January 28, 2024 Data Mining: Concepts and Techniques 33

Other Clustering based methods
Partitioning Methods such as PAM
Hierarchical Methods
Density-Based Methods
Grid-Based Methods
Model-Based Methods
Clustering High-Dimensional Data
Constraint-Based Clustering
Outlier Analysis
 Summary
• Cluster analysis groups objects based on their similarity and has
wide applications
• Measure of similarity can be computed for various types of data
• Clustering algorithms can be categorized into partitioning
methods, hierarchical methods, density-based methods, grid-
based methods, and model-based methods
• Outlier detection and analysis are very useful for fraud
detection, etc. and can be performed by statistical, distance-
based or deviation-based approaches
• There are still lots of research issues on cluster analysis

January 28, 2024 Data Mining: Concepts and Techniques 35

 Problems and Challenges

• Considerable progress has been made in scalable clustering

methods
• Partitioning: k-means, k-medoids, CLARANS
• Hierarchical: BIRCH, ROCK, CHAMELEON
• Density-based: DBSCAN, OPTICS, DenClue
• Grid-based: STING, WaveCluster, CLIQUE
• Model-based: EM, Cobweb, SOM
• Frequent pattern-based: pCluster
• Constraint-based: COD, constrained-clustering
• Current clustering techniques do not address all the
requirements adequately, still an active area of research
January 28, 2024 Data Mining: Concepts and Techniques 36
 References (1)
• R. Agrawal, J. Gehrke, D. Gunopulos, and P. Raghavan. Automatic subspace clustering of high dimensional data
for data mining applications. SIGMOD'98

• M. R. Anderberg. Cluster Analysis for Applications. Academic Press, 1973.

• M. Ankerst, M. Breunig, H.-P. Kriegel, and J. Sander. Optics: Ordering points to identify the clustering
structure, SIGMOD’99.

• P. Arabie, L. J. Hubert, and G. De Soete. Clustering and Classification. World Scientific, 1996

• Beil F., Ester M., Xu X.: "Frequent Term-Based Text Clustering", KDD'02

• M. M. Breunig, H.-P. Kriegel, R. Ng, J. Sander. LOF: Identifying Density-Based Local Outliers. SIGMOD 2000.

• M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial
databases. KDD'96.

• M. Ester, H.-P. Kriegel, and X. Xu. Knowledge discovery in large spatial databases: Focusing techniques for
efficient class identification. SSD'95.

• D. Fisher. Knowledge acquisition via incremental conceptual clustering. Machine Learning, 2:139-172, 1987.

• D. Gibson, J. Kleinberg, and P. Raghavan. Clustering categorical data: An approach based on dynamic systems.
January 28, 2024 Data Mining: Concepts and Techniques 37
VLDB’98.
 References (2)
• V. Ganti, J. Gehrke, R. Ramakrishan. CACTUS Clustering Categorical Data Using Summaries. KDD'99.

• D. Gibson, J. Kleinberg, and P. Raghavan. Clustering categorical data: An approach based on dynamic systems. In
Proc. VLDB’98.

• S. Guha, R. Rastogi, and K. Shim. Cure: An efficient clustering algorithm for large databases. SIGMOD'98.

• S. Guha, R. Rastogi, and K. Shim. ROCK: A robust clustering algorithm for categorical attributes. In ICDE'99, pp. 512-
521, Sydney, Australia, March 1999.

• A. Hinneburg, D.l A. Keim: An Efficient Approach to Clustering in Large Multimedia Databases with Noise. KDD’98.

• A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Printice Hall, 1988.
• G. Karypis, E.-H. Han, and V. Kumar. CHAMELEON: A Hierarchical Clustering Algorithm Using Dynamic Modeling.
COMPUTER, 32(8): 68-75, 1999.
• L. Kaufman and P. J. Rousseeuw. Finding Groups in Data: an Introduction to Cluster Analysis. John Wiley & Sons,
1990.
• E. Knorr and R. Ng. Algorithms for mining distance-based outliers in large datasets. VLDB’98.
• G. J. McLachlan and K.E. Bkasford. Mixture Models: Inference and Applications to Clustering. John Wiley and Sons,
1988.
• P. Michaud. Clustering techniques. Future Generation Computer systems, 13, 1997.

January 28, 2024• R. Ng and J. Han. Efficient and effective Data

clustering
Mining:method forand
Concepts spatial data mining. VLDB'94.
Techniques 38
 References (3)

• L. Parsons, E. Haque and H. Liu, Subspace Clustering for High Dimensional Data: A Review , SIGKDD
Explorations, 6(1), June 2004

• E. Schikuta. Grid clustering: An efficient hierarchical clustering method for very large data sets. Proc. 1996
Int. Conf. on Pattern Recognition,.

• G. Sheikholeslami, S. Chatterjee, and A. Zhang. WaveCluster: A multi-resolution clustering approach for

very large spatial databases. VLDB’98.

• A. K. H. Tung, J. Han, L. V. S. Lakshmanan, and R. T. Ng. Constraint-Based Clustering in Large Databases,

ICDT'01.

• A. K. H. Tung, J. Hou, and J. Han. Spatial Clustering in the Presence of Obstacles , ICDE'01

• H. Wang, W. Wang, J. Yang, and P.S. Yu. Clustering by pattern similarity in large data sets, SIGMOD’ 02.

• W. Wang, Yang, R. Muntz, STING: A Statistical Information grid Approach to Spatial Data Mining, VLDB’97.

• T. Zhang, R. Ramakrishnan, and M. Livny. BIRCH : an efficient data clustering method for very large
databases. SIGMOD'96.

January 28, 2024 Data Mining: Concepts and Techniques 39