Introduction to Information Retrieval

Clustering: Definition
(Document) clustering is the process of grouping a set of
documents into clusters of similar documents.
Documents within a cluster should be similar.
Documents from different clusters should be dissimilar.
Clustering is the most common form of unsupervised
learning.
Unsupervised = there are no labeled or annotated data.

Introduction to Information Retrieval

Data set with clear cluster structure

Introduction to Information Retrieval

Classification vs. Clustering

Classification: supervised learning
Clustering: unsupervised learning
Classification: Classes are human-defined and part of the
input to the learning algorithm.
Clustering: Clusters are inferred from the data without
human input.
However, there are many ways of influencing the outcome of
clustering: number of clusters, similarity measure,
representation of documents, . . .

Introduction to Information Retrieval

The cluster hypothesis

Cluster hypothesis. Documents in the same cluster behave
similarly with respect to relevance to information needs. All
applications of clustering in IR are based (directly or indirectly) on
the cluster hypothesis.
Van Rijsbergens original wording: closely associated documents
tend to be relevant to the same requests.

Introduction to Information Retrieval

Yahoo! Hierarchy isnt clustering but is the kind

of output you want from clustering
www.yahoo.com/Science
(30)
agriculture
...

biology

physics
...

CS
...

space
...

...

dairy

botany
cell
AI
courses
crops
magnetism
HCI
agronomy
evolution
forestry
relativity

craft
missions

Introduction to Information Retrieval

Google News: automatic clustering gives an

effective news presentation metaphor

Introduction to Information Retrieval

Scatter/Gather: Cutting, Karger, and Pedersen

Sec. 16.1

Introduction to Information Retrieval

For visualizing a document collection and its

themes
Wise et al, Visualizing the non-visual PNNL
ThemeScapes, Cartia
[Mountain height = cluster size]

Introduction to Information Retrieval

Sec. 16.1

For improving search recall

Cluster hypothesis - Documents in the same cluster behave similarly
with respect to relevance to information needs
Therefore, to improve search recall:
Cluster docs in corpus a priori
When a query matches a doc D, also return other docs in the
cluster containing D
Hope if we do this: The query car will also return docs containing
automobile
Because clustering grouped together docs containing car with
those containing automobile.

Why might this happen?

Introduction to Information Retrieval

Desiderata for clustering

General goal: put related docs in the same cluster, put
unrelated docs in different clusters.
How do we formalize this?

The number of clusters should be appropriate for the data set

we are clustering.
Initially, we will assume the number of clusters K is given.
Later: Semiautomatic methods for determining K

Secondary goals in clustering

Avoid very small and very large clusters
Define clusters that are easy to explain to the user
Many others . . .
10

Introduction to Information Retrieval

Flat vs. Hierarchical clustering

Flat algorithms
Usually start with a random (partial) partitioning of docs into
groups
Refine iteratively
Main algorithm: K-means

Hierarchical algorithms
Create a hierarchy
Bottom-up, agglomerative
Top-down, divisive

11

Introduction to Information Retrieval

Hard vs. Soft clustering

Hard clustering: Each document belongs to exactly one
cluster.
More common and easier to do

Soft clustering: A document can belong to more than one

cluster.
Makes more sense for applications like creating browsable
hierarchies
You may want to put sneakers in two clusters:
sports apparel
shoes

You can only do that with a soft clustering approach.

12

Introduction to Information Retrieval

Flat algorithms
Flat algorithms compute a partition of N documents into a
set of K clusters.
Given: a set of documents and the number K
Find: a partition into K clusters that optimizes the chosen
partitioning criterion
Global optimization: exhaustively enumerate partitions,
pick optimal one
Not tractable

Effective heuristic method: K-means algorithm

13

Introduction to Information Retrieval

K-means
Perhaps the best known clustering algorithm
Simple, works well in many cases
Use as default / baseline for clustering documents

14

Introduction to Information Retrieval

Document representations in clustering

Vector space model

As in vector space classification, we measure relatedness
between vectors by Euclidean distance . . .
. . .which is almost equivalent to cosine similarity.

15

Introduction to Information Retrieval

K-means
Each cluster in K-means is defined by a centroid.
Objective/partitioning criterion: minimize the average
squared difference from the centroid
definition of centroid:

where we use to denote a cluster.

We try to find the minimum average squared difference by
iterating two steps:
reassignment: assign each vector to its closest centroid
recomputation: recompute each centroid as the average of
the vectors that were assigned to it in reassignment

16

Introduction to Information Retrieval

K-means algorithm

17

Introduction to Information Retrieval

Worked Example: Set of to be clustered

18

Introduction to Information Retrieval

Worked Example: Random selection of initial

centroids

Exercise: (i) Guess what the

optimal clustering into two clusters is in this case; (ii) compute the
centroids of the clusters
19

Introduction to Information Retrieval

Worked Example: Assign points to closest center

20

Introduction to Information Retrieval

Worked Example: Assignment

21

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

22

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

23

Introduction to Information Retrieval

Worked Example: Assignment

24

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

25

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

26

Introduction to Information Retrieval

Worked Example: Assignment

27

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

28

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

29

Introduction to Information Retrieval

Worked Example: Assignment

30

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

31

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

32

Introduction to Information Retrieval

Worked Example: Assignment

33

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

34

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

35

Introduction to Information Retrieval

Worked Example: Assignment

36

Introduction to Information Retrieval

Worked Example: Recompute cluster centroids

37

Introduction to Information Retrieval

Worked Example: Assign points to closest centroid

38

Introduction to Information Retrieval

Worked Example: Assignment

39

Introduction to Information Retrieval

Worked Example: Recompute cluster caentroids

40

Introduction to Information Retrieval

Worked Ex.: Centroids and assignments after

convergence

41

Introduction to Information Retrieval

K-means is guaranteed to converge

RSS = sum of all squared distances between document vector
and closest centroid
RSS decreases during each reassignment step.
because each vector is moved to a closer centroid

RSS decreases during each recomputation step.

There is only a finite number of clusterings.
Thus: We must reach a fixed point.
Assumption: Ties are broken consistently.

42

Introduction to Information Retrieval

Optimality of K-means
Convergence does not mean that we converge to the optimal
clustering!
This is the great weakness of K-means.
If we start with a bad set of seeds, the resulting clustering
can be horrible.

43

Introduction to Information Retrieval

Convergence Exercise: Suboptimal clustering

What is the optimal clustering for K = 2?

Do we converge on this clustering for arbitrary seeds di , dj?
44

Introduction to Information Retrieval

Initialization of K-means
Random seed selection is just one of many ways K-means
can be initialized.
Random seed selection is not very robust: Its easy to get a
suboptimal clustering.
Better ways of computing initial centroids:
Select seeds not randomly, but using some heuristic (e.g., filter
out outliers or find a set of seeds that has good coverage of
the document space)
Use hierarchical clustering to find good seeds
Select i (e.g., i = 10) different random sets of seeds, do a Kmeans clustering for each, select the clustering with lowest RSS
45

Introduction to Information Retrieval

Time complexity of K-means

Computing one distance of two vectors is O(M).
Reassignment step: O(KNM) (we need to compute KN
document-centroid distances)
Recomputation step: O(NM) (we need to add each of the
documents < M values to one of the centroids)
Assume number of iterations bounded by I
Overall complexity: O(IKNM) linear in all important
dimensions

46

Introduction to Information Retrieval

What is a good clustering?

Internal criteria
Example of an internal criterion: RSS in K-means

But an internal criterion often does not evaluate the actual

utility of a clustering in the application.
Alternative: External criteria
Evaluate with respect to a human-defined classification

47

Introduction to Information Retrieval

External criteria for clustering quality

Based on a gold standard data set, e.g., the Reuters
collection we also used for the evaluation of classification
Goal: Clustering should reproduce the classes in the gold
standard
(But we only want to reproduce how documents are divided
into groups, not the class labels.)
First measure for how well we were able to reproduce the
classes: purity

48

Introduction to Information Retrieval

External criterion: Purity

= {1, 2, . . . , K} is the set of clusters and

C = {c1, c2, . . . , cJ} is the set of classes.
For each cluster k : find class cj with most members nkj in k
Sum all nkj and divide by total number of points

49

Introduction to Information Retrieval

Sec. 16.3

Purity example

Cluster I

Cluster II

Cluster I: Purity = 1/6 (max(5, 1, 0)) = 5/6

Cluster II: Purity = 1/6 (max(1, 4, 1)) = 4/6
Cluster III: Purity = 1/5 (max(2, 0, 3)) = 3/5

Cluster III

Purity is (1/17)
(5 + 4 + 3) 0.71.

Introduction to Information Retrieval

Example for computing purity

Purity is (1/17) (5 + 4 + 3) 0.71.

51

Introduction to Information Retrieval

Rand index
Definition:
Based on 2x2 contingency table of all pairs of documents:

TP+FN+FP+TN is the total number of pairs.

There are
pairs for N documents.
Example:
= 136 in o//x example
Each pair is either positive or negative (the clustering puts
the two documents in the same or in different clusters) . . .
. . . and either true (correct) or false (incorrect): the
clustering decision is correct or incorrect.

52

Introduction to Information Retrieval

Rand Index: Example

As an example, we compute RI for the o//x example. We first
compute TP + FP. The three clusters contain 6, 6, and 5 points,
respectively, so the total number of positives or pairs of
documents that are in the same cluster is:

Of these, the x pairs in cluster 1, the o pairs in cluster 2, the

pairs in cluster 3, and the x pair in cluster 3 are true positives:

Thus, FP = 40 20 = 20. FN and TN are computed similarly.

53

Introduction to Information Retrieval

Rand measure for the o//x example

(20 + 72)/(20 + 20 + 24 + 72) 0.68.

54

Introduction to Information Retrieval

Two other external evaluation measures

Two other measures
Normalized mutual information (NMI)
How much information does the clustering contain about the
classification?
Singleton clusters (number of clusters = number of docs) have
maximum MI
Therefore: normalize by entropy of clusters and classes

F measure
Like Rand, but precision and recall can be weighted
55

Introduction to Information Retrieval

Normalized Mutual Information (NMI)

56

Introduction to Information Retrieval

Evaluation results for the o//x example

All four measures range from 0 (really bad clustering) to 1 (perfect

clustering).

57

Introduction to Information Retrieval

How many clusters?

Number of clusters K is given in many applications.
E.g., there may be an external constraint on K. Example: In the
case of Scatter-Gather, it was hard to show more than 1020
clusters on a monitor in the 90s.

What if there is no external constraint? Is there a right

number of clusters?
One way to go: define an optimization criterion
Given docs, find K for which the optimum is reached.
What optimiation criterion can we use?
We cant use RSS or average squared distance from centroid as
criterion: always chooses K = N clusters.
58

Introduction to Information Retrieval

Exercise

Your job is to develop the clustering algorithms for a

competitor to news.google.com
You want to use K-means clustering.
How would you determine K?

59

Introduction to Information Retrieval

Simple objective function for K (1)

Basic idea:
Start with 1 cluster (K = 1)
Keep adding clusters (= keep increasing K)
Add a penalty for each new cluster

Trade off cluster penalties against average squared distance

from centroid
Choose the value of K with the best tradeoff

60

Introduction to Information Retrieval

Simple objective function for K (2)

Given a clustering, define the cost for a document as
(squared) distance to centroid
Define total distortion RSS(K) as sum of all individual
document costs (corresponds to average distance)
Then: penalize each cluster with a cost
Thus for a clustering with K clusters, total cluster penalty is
K
Define the total cost of a clustering as distortion plus total
cluster penalty: RSS(K) + K
Select K that minimizes (RSS(K) + K)
Still need to determine good value for . . .
61

Introduction to Information Retrieval

Finding the knee in the curve

Pick the number of

clusters where curve
flattens. Here: 4 or 9.

62

Introduction to Information Retrieval

Resources
Chapter 16 of IIR
Resources at http://ifnlp.org/ir
K-means example
Keith van Rijsbergen on the cluster hypothesis (he was one of
the originators)
Bing/Carrot2/Clusty: search result clustering

63