Evolutionary Clustering

Deepayan Chakrabarti Ravi Kumar Andrew Tomkins

Yahoo! Research, 701 First Ave, Sunnyvale, CA 94089.
{deepay, ravikumar, atomkins}@yahoo-inc.com

ABSTRACT must trade off the benefit of maintaining a consistent clus-

We consider the problem of clustering data over time. An tering over time with the cost of deviating from an accurate
evolutionary clustering should simultaneously optimize two representation of the current data.
potentially conflicting criteria: first, the clustering at any The benefits of evolutionary clustering compared to tra-
point in time should remain faithful to the current data as ditional clustering appear in situations in which the current
much as possible; and second, the clustering should not shift (say, daily) clustering is being consumed regularly by a user
dramatically from one timestep to the next. We present a or system. In such a setting, evolutionary clustering is useful
generic framework for this problem, and discuss evolutionary for the following reasons:
versions of two widely-used clustering algorithms within this (1) Consistency: A user will find each day’s clustering
framework: k-means and agglomerative hierarchical cluster- familiar, and so will not be required to learn a completely
ing. We extensively evaluate these algorithms on real data new way of segmenting data. Similarly, any insights derived
sets and show that our algorithms can simultaneously at- from a study of previous clusters are more likely to apply to
tain both high accuracy in capturing today’s data, and high future clusters.
fidelity in reflecting yesterday’s clustering. (2) Noise removal: Providing a high-quality and histori-
cally consistent clustering provides greater robustness against
Categories and Subject Descriptors: H.3.3 [Informa- noise by taking previous data points into effect. As we de-
tion Storage and Retrieval]: Information Search and Re- scribe later, our method subsumes standard approaches to
trieval windowing and moving averages.
General Terms: Algorithms, Experimentation, Measure- (3) Smoothing: If the true clusters shift over time, evo-
ments lutionary clustering will naturally present the user with a
Keywords: Clustering, Temporal Evolution, Agglomera- smooth view of the transition.
tive, k-means (4) Cluster correspondence: As a side effect of our frame-
work, it is generally possible to place today’s clusters in
correspondence with yesterday’s clusters. Thus, even if the
1. INTRODUCTION clustering has shifted, the user will still be situated within
Evolutionary clustering is the problem of processing times- the historical context.
tamped data to produce a sequence of clusterings; that is, a
clustering for each timestep of the system. Each clustering Overview of framework. Formally, let Ct be the cluster-
in the sequence should be similar to the clustering at the ing produced by the algorithm for data arriving at timestep
previous timestep, and should accurately reflect the data i. The snapshot quality of Ct measures how well Ct repre-
arriving during that timestep. sents the data at timestep t. The history cost of the clus-
The primary setting for this problem is the following. Ev- tering is a measure of the distance between Ct and Ct−1 ,
ery day, new data arrives for the day, and must be incorpo- the clustering used during the previous timestep. Typically,
rated into a clustering. If the data does not deviate from the snapshot quality is defined in terms of the data points
historical expectations, the clustering should be “close” to themselves, while the history cost is a function of the clus-
that from the previous day, providing the user with a famil- ter models. The overall cost of the clustering sequence is a
iar view of the new data. However, if the structure of the combination of the snapshot quality and the history cost at
data changes significantly, the clustering must be modified each timestep.
to reflect the new structure. Thus, the clustering algorithm For intuition, we consider an extreme example to show
why the introduction of history cost may have a signifi-
cant impact on the clustering sequence. Consider a data
set in which either of two features may be used to split the
Permission to make digital or hard copies of all or part of this work for data into two clusters: feature A and feature B. Each fea-
personal or classroom use is granted without fee provided that copies are ture induces an orthogonal split of the data, and each split
not made or distributed for profit or commercial advantage and that copies is equally good. However, on odd-numbered days, feature
bear this notice and the full citation on the first page. To copy otherwise, to A provides a slightly better split, while on even-numbered
republish, to post on servers or to redistribute to lists, requires prior specific days, feature B is better. The optimal clustering on each day
permission and/or a fee.
KDD’06, August 20–23, 2006, Philadelphia, Pennsylvania, USA. will shift radically from the previous day, while a consistent
Copyright 2006 ACM 1-59593-339-5/06/0008 ...$5.00. clustering using either feature will perform arbitrarily close
to optimal. In this case, a clustering algorithm that does of outliers [4]. Zhang et al. [18] proposed a probabilistic
not take history into account will produce a poor clustering model for online document clustering, where the emphasis
sequence. is again on detecting novelty. Clustering was also used in
This approach may be contrasted to incremental cluster- automatic techniques for discovering and retrieving topically
ing, in which a model is incrementally updated as new data related material in data streams, and to detect novel events
arrive, primarily to avoid the cost of storing all historical from a temporally ordered collection of news stories [17, 2].
similarities [9]. In evolutionary clustering, however, the fo- However, the main goal of topic detection and tracking is
cus is upon optimizing a new quality measure which incor- to detect new events in a time-line using methods such as
porates deviation from history. clustering, and not to produce clusterings that incorporate
history into the clustering objective.
Summary of Contributions. We consider two classical In some data stream applications, time has played a dif-
clustering algorithms and extend them to the evolutionary ferent role with respect to clustering. Aggarwal et al. [1]
setting: (1) the traditional k-means algorithm that provides study the problem of clustering data streams at different
a flat clustering of points in a vector space, and (2) a bottom- time horizons using an online statistical aggregation stage
up agglomerative hierarchical clustering algorithm. These and then an offline clustering stage. For more details on
represent two major categories of clustering methods and clustering from a data stream analysis perspective, see [10].
we chose to use them to demonstrate the generality of our The notion of clustering time-series has been considered
framework. These algorithms are applied to a large evolv- in statistics, data mining, and machine learning. Tempo-
ing dataset of user-tags placed on images from flickr.com, ral correlation is perhaps the best-known approach to time-
tracked over 68 weeks. Our experiments demonstrate all series similarity [5]. Smyth [14] considers general cluster-
the advantages of evolutionary clustering mentioned previ- ing of sequence data using a probabilistic approach based
ously, namely, stable and consistent clusterings are obtained on hidden Markov models; see also [12]. Immorlica and
even when the data exhibits noisy behavior, and a smooth Chien [6] propose a low-dimensional representation of time-
sequence of clusters with very low history cost can be ob- series for clustering. They use a variety of basis functions in-
tained for only a small reduction in total snapshot quality. cluding piecewise-constant, piecewise-linear, and piecewise-
Our framework for evolutionary clustering is in fact quite aggregate approximations. Vlachos et al. propose a Fourier
general. With suitable definitions of history cost and snap- approach to this problem [15]. Our work, however, has much
shot quality, many other static clustering algorithms can broader scope; we must consider object feature similarity in
now be extended to perform evolutionary clustering under addition to the similarities in their time series, as will be
our framework. Furthermore, we focused on a setting in explained in the next section.
which today’s data must be clustered before tomorrow’s
data is available. However, there are other natural settings.
For example, the entire sequence may be available to the
3. FRAMEWORK
algorithm at once, but the algorithm must again produce a In this section, we present our framework for evolutionary
sequence of clusterings that accurately reflect the data at clustering. We begin by differentiating between algorithms
each timestep while shifting as gently as possible over time. that must cluster data as it arrives, versus algorithms that
This formulation applies when the data should be clustered have access to the entire time series before beginning work.
retroactively but interpretability across time is an important An evolutionary clustering algorithm is online if it must
consideration. Similarly, there are settings in which the nu- provide a clustering of the data during timestep t before
merical variable over which we track evolution is not time; seeing any data for timestep t+1. If the algorithm has access
for example, we may cluster products by dollar value, and to all data beforehand, it is offline. Offline algorithms may
ask that the clusterings for similar price buckets be as simi- “see the future,” and should perform at least as well as their
lar as possible. Our formulation can capture such directions online counterparts. However, the online setting is arguably
as well, but we will not explore them in this paper. more important for real-world applications, and there are
no natural, efficient offline algorithms that may be easily
employed as lower bounds. We therefore leave the offline
2. RELATED WORK problem for future work, and focus on the online version.
Clustering is a well-studied problem; see, for instance [16,
8]. However, to the best of our knowledge, evolutionary 3.1 Overview of the framework
clustering has not been considered before. Recall that an evolutionary clustering algorithm must pro-
In the following, we briefly review the interplay of time duce a sequence of clusterings, one for each timestep. The
and notions related to clustering, including classification and clustering produced during a particular timestep should mea-
online topic detection and tracking. sure well along two distinct criteria: it should have low
Temporal aspects have been considered in some classi- history cost, meaning it should be similar to the previous
fication problems. Some work in online machine learning clustering in the sequence, and it should have high snapshot
considers learning tasks such as classification in which the quality, meaning it should be a high-quality clustering of the
model evolves over time, and the algorithm is penalized for data that arrived during the current timestep. The snapshot
both mistakes and shifts in the model from one timestep to quality simply reflects the underlying figure of merit driving
the next [11, 3]. However, it is unclear how this could be a particular clustering algorithm in a non-evolutionary set-
used in the unsupervised learning setting. ting. The history cost, however, must allow a comparison of
Temporal aspects have also been considered in online doc- clusterings, in order to determine how much the later one has
ument clustering setting. Online document clustering ap- shifted from the earlier. This comparison must address the
plies clustering sequentially to a time series of text docu- issue that some data will appear for the first time in the later
ments, in order to detect novelty in the form of certain types clustering, while some data will be seen in the earlier clus-
tering but not the later one, and so forth. There are many rameter which trades-off between the two. As cp increases,
examples of measures for comparing clusterings — see [13] more and more weight is placed on matching the historical
for a discussion of the complexities that arise even in the clusters.
case of flat clusterings. But this comparison may be more Given this framework, an evolutionary clustering algo-
sophisticated than simply a comparison of two partitions of rithm takes as input M1 , . . . , Mt and produces C1 , . . . , Ct .
the universe. The comparison may occur at the data level, We can compute the quality of the resulting cluster se-
for instance by comparing how similar pairs of data objects quence, and hence we may compare evolutionary cluster-
are clustered, or at the model level, for instance by compar- ing algorithms and speak of optimal algorithms and optimal
ing the best matching between two sets of centroids in the quality.
k-means setting. In our setting, comparisons at the model As we stated earlier, our focus is on the online setting.
level make the most intuitive sense, and this is what we use. Clearly a sequence of online decisions to find C1 , . . . , Ct may
First, we require a few high-level definitions. Let U = not be the best offline solution to (1), but nevertheless, it is
{1, . . . , n} be the universe of objects to be clustered. Let a defensible alternative given no knowledge about the future
Ut ⊆ U be the set of all objects present at timestep t, and data. Our algorithms therefore try to find an optimal cluster
let U≤t = ∪t′ ≤t Ut′ be the set of all objects present in any sequence by finding at each timestep t a clustering Ct that
timestep up to and including step t. optimizes the incremental quality
An evolutionary clustering algorithm must behave as fol-
lows. At each timestep t, it should produce a clustering Ct of sq(Ct , Mt ) − cp · hc(Ct−1 , Ct ). (2)
U≤t , the objects seen by the algorithm so far. The distance
from Ct to Ct−1 will be evaluated with respect to U≤t−1 in 3.3 Constructing Mt from raw data
order to determine the historical accuracy of Ct . Specifi- In a traditional clustering setting, the data objects are all
cally, if new data arrived for the first time during timestep available at once to be clustered, and some measure of sim-
t, the clustering will not be penalized for deviating from the ilarity or distance between objects may be applied. This is
previous clustering with respect to the new data unless this also true in our setting: all other timesteps may be ignored,
deviation also impacts the clustering of historical data. The and local information may be used to compute similarity be-
history cost is computed by projecting Ct onto U≤t−1 . tween objects during a particular timestep. However, there
At the same time, the snapshot quality of Ct will be eval- is also another type of similarity which is unique to our set-
uated with respect to Ut , the objects that actually appear ting — temporal similarity. If objects recur over time, the
during step t. These two measures, over all timesteps, will algorithm to cluster them during a particular timestep may
provide an overall evaluation of the entire cluster sequence. make use of their historical occurrence patterns. This type
Observe that, in order to perform well, clustering Ct must of similarity should not be confused with similarity between
include objects in U≤t \Ut , that is, the objects that have been time series, as in a time series clustering problem: the ob-
seen in the past but have not appeared during the current ject in our universe is the point of the time series, rather
timestep. So either implicitly or explicitly, an evolutionary than the series itself. Below, we describe these, and how to
clustering algorithm must “carry along” information about combine them to form the input matrix Mt .
the appropriate location of historical information.
Local information similarity. In some cases, the input
3.2 Input to the framework to evolutionary clustering might already be in the form of
Recall that U = {1, . . . , n} is the universe of objects to an inter-object similarity matrix St ; in other cases, we must
be clustered. At each timestep t where 1 ≤ t ≤ T , a new infer it. One common scenario involves input as a graph,
set of data arrives to be clustered. We assume that this for example, a bipartite graph linking the data objects of
data can be represented as an n × n matrix Mt that ex- interest to a set of features. Then, similarity between objects
presses the relationship between each pair of data objects. is related to the number of features they share.
The relationship expressed by Mt can be either based on Let B(t) represent this bipartite graph at time t, and let
similarity or based on distance depending on the require- R(t) = rj,j ′ encode the number of features shared by both
ments of the particular underlying clustering algorithm. If objects j and j ′ . Then, we can say
the algorithm requires similarities (resp., distances), we will
write sim(i, j, t) (resp., dist(i, j, t)) to represent the similar- R(t) = B(t)B′ (t).
ity (resp., distance) between objects i and j at time t.
At each timestep t, an online evolutionary clustering algo- However, notice that a similarity measure based purely on
rithm is presented with a new matrix Mt , either sim(·, ·, t) R(t) is “memoryless,” i.e., if an object fails to appear on one
or dist(·, ·, t), and must produce Ct , the clustering for time snapshot, all information about its similarities with other
t, based on the new matrix and the history so far. objects is lost completely. This is not desirable since we
A user of the framework must specify a snapshot quality wish to capture history to some extent. Therefore, we use
function sq(Ct , Mt ) that returns the quality of the clustering
Ct at time t with respect to Mt . The user must also provide a R(t) = (1 − β) · B(t)B′ (t) + β · R(t − 1), for t > 0
history cost function hc(Ct−1 , Ct ) that returns the historical
where the parameter β is chosen to retain enough infor-
cost of the clustering at time t. The total quality of a cluster
mation on relative similarities of old objects with all other
sequence is then
objects, but not swamp the data from the current snapshot.
T T
X X In our experiments, we set β = 0.1.
sq(Ct , Mt ) − cp · hc(Ct−1 , Ct ), (1) Observe that this approach incorporates an exponentially
t=1 t=2 decaying moving average of data, and could easily be mod-
where the “change parameter” cp > 0 is a user-defined pa- ified to support other windowing techniques.
 Now, we can use cosine similarity between objects to gen- obtained by first removing all leaves in leaf(T )\leaf (T ′ ) and
erate the local similarity matrix St : then collapsing all unary internal nodes.
We define the snapshot quality of T to be the sum of the
hrj , rj ′ i
St (j, j ′ ) = . qualities of all merges performed to create T :
|rj | · |rj ′ | X
sq(T, M ) = simM (m).
Similar steps could be followed if the underlying clustering m∈in(T )
algorithm required a distance matrix instead of a similarity
matrix. We now define the history cost, which is the distance be-
tween two trees T, T ′ with leaf(T ) ⊇ leaf(T ′ ). First, we
Temporal similarity. This is given by the standard defi- define the distance between objects i, j ∈ leaf(T ′ ) to be the
nition of correlation of the two time series up to and includ- squared-error distance:
ing time t0 : ` ´2
Pt0 dT ′ ,T (i, j) = dT ′ (i, j) − dT ′ |T (i, j) (3)
t=1 (xi,t − µ(i, t))(xj,t − µ(j, t))
Corr(i, j, t0 ) = p , Then, the distance between T and T ′ is defined to be the
Var(i, t) · Var(j, t)
average distance between all pairs of objects
where xi,t represents the number of occurrences of data ob-
hc(T ′ , T ) = E (dT ′ ,T (i, j)). (4)
ject i in timestep t, and the means and variances are defined i,j∈leaf (T ′ )
i6=j
on xi,1 , . . . , xi,t0 and xj,1 , . . . , xj,t0 .
As stated earlier, the goal is to find a clustering Ct that
Total similarity. We combine these two types of similar- minimizes (2). To do this, we first note that (4) can be
ity information into the final similarity matrix Mt , taking rewritten as a sum of contributions from each internal node,
the overall similarity between two objects at time t to be where the contribution covers all pairs of points for whom
Mt (i, j) = α · St (i, j) + (1 − α) · Corr(i, j, t), that internal node is the least common ancestor. Thus,
0 1
where α controls the contribution of correlation and tempo-
ral similarities. hc(T ′ , T ) = E @ E (dT ′ ,T (i, j))A .
m∈in(T ′ ) i∈leaf (mℓ )
j∈leaf (mr )

4. ALGORITHMS Using this reformulation of history cost, we may write the

incremental quality in (2) as
We now present two instantiations of our framework. Sec-
tion 4.1 describes an evolutionary version of the bottom-
0 1
X
up agglomerative hierarchical clustering algorithm and Sec- simM (m) − @cp · E (dT ′ ,T (i, j))A .
tion 4.2 discusses an evolutionary version of the traditional m∈in(T )
m∈in(T )
i∈leaf (mℓ ),j∈leaf (mr )
k-means. The above two choices were motivated by the sig-
nificant differences between the underlying clustering algo- We propose four greedy heuristics to choose the order of
rithms: k-means produces a flat rather than a hierarchi- merges. Let T = Tt and T ′ = Tt−1 .
cal clustering, implicitly requires the data to lie in a vector In the first heuristic, we choose the merge m whose con-
space, and creates a model based on pseudo-objects that tribution to this expression is maximal. In other words, pick
lie in the same space as the actual objects being clustered. the merge m that maximizes
These two very different approaches thus show the generality 0 1
of our framework.
simM (m) − @cp · E (dT ′ ,T (i, j))A . (5)
i∈leaf (mℓ )
4.1 Agglomerative hierarchical clustering j∈leaf (mr )

To develop an evolutionary hierarchical clustering, we first We refer to this heuristic as Squared, since it greedily mini-
describe a standard agglomerative clustering at a particular mizes the squared error in Equation 3.
fixed timestep t. Let M = Mt = sim(·, ·, t), U = U≤t . First, However, we observe that a merge with a particular squared
we select the pair i, j of objects that maximizes M (i, j). error may become better or worse if it is put off until later.
Next, we merge these two objects, creating a new object; To wit, if two objects are far away in T ′ , then perhaps we
we also update the similarity matrix M by replacing the should delay the merge until they are similarly far away in
rows and columns corresponding to objects i and j by their T . However, if two objects are close in T ′ but merging them
average that represents the new object. We then repeat the would already make them far in T then we should encour-
procedure, building a bottom-up binary tree T whose leaves age the merge despite their high cost, as delaying will only
are the objects in U ; the tree Ct = Tt = T represents the make things worse. Based on this observation, we consider
clustering of the objects at timestep t. the cost of merge based on what would change if we de-
Let the internal nodes of T be labeled m1 , . . . , m|U |−1 , layed the merge until the two merged subtrees became more
and let simM (mi ) represent the similarity of objects that distant from one another (due to intermediate merges).
were merged to produce the internal node mi . Let in(T ) be Thus, consider a possible merge of subtrees S1 and S2 .
the set of all internal nodes of T . For an internal node m, Performing the merge incurs a penalty for nodes that are
let mℓ be the left child of m, mr be the right child of m, and still too close, and a benefit for nodes that are already
leaf (m) be the set of leaves in the subtree rooted at m. Let too far apart. The benefit and penalty are expressed in
d(i, j) be the tree distance in T between nodes i and j. If terms of the change in cost if either S1 or S2 participates
T ′ , T are binary trees with leaf(T ) ⊇ leaf (T ′ ), then the tree in another merge, and hence the elements of S1 and S2 in-
T ′ |T is the projection of T ′ onto T , i.e., T ′ |T is a binary tree crease their average distance by 1. This penalty may be
written by taking the partial derivative of the squared cost The algorithm proceeds during several passes, during each
with respect to the distance of an element to the root. At of which it updates each centroid based on the data elements
any point in the execution of the algorithm at time t, let currently assigned to that centroid:
root(i) be the root of the current subtree containing i. For
cj ← E | (x),
i ∈ S1 and j ∈ S2 , let dm T (i, j) be the merge distance of x∈closest(j)
i and j at time t, i.e., dm T (i, j) is the distance between i
and j at time t if S1 and S2 are merged together. Then, after which cj is normalized to have unit length. The al-
dm gorithm terminates after sufficiently many passes and the
T (i, j) = dT (i, root(i)) + dT (j, root(j)) + 2 The benefit of
merging now is given by: clustering Ct = C is given by the set {c1 , . . . , ck } of k cen-
0 1 troids.
We define the snapshot quality of a k-means clustering to
simM (m) − @cp · E (dT ′ (i, j) − dm
T (i, j))
A. (6) be
i∈leaf(mℓ ) X
j∈leaf (mr )
sq(C, M ) = (1 − min ||c − x||).
c∈C
We refer to this heuristic as Linear-Internal. Notice that, x∈U
as desired, the benefit is positive when the distance in T ′ is (Since all points are on the unit sphere, distances are bounded
large, and negative otherwise. Similarly, the magnitude of above by 1.)
the penalty depends on the derivative of the squared error We define the history cost, i.e., the distance between two
(Equation 3). clusterings, to be
As another heuristic, we may also observe that our deci-
sion about merging S1 with S2 may also depend on objects hc(C, C ′ ) = min ||ci − c′f (i) ||,
f :[k]→[k]
that do not belong to either subtree. Assume that elements
of S1 are already too far apart from some subtree S3 . Then where f is a function that maps centroids of C to centroids
merging S1 with S2 may introduce additional costs down- of C ′ . That is, the distance between two clusterings is com-
stream that are not apparent without looking outside the puted by matching each centroid in C to a centroid in C ′
potential merge set. In order to address this problem, we in the best possible way, and then adding the distances for
modify (6) to penalize a merge if it increases the distance these matches.
gap (i.e., the distance at time t versus the distance at time As stated earlier, we use a greedy approximation algo-
t−1) between elements that participate in the merge and el- rithm to choose the next cluster in the sequence. How-
ements that do not. Similarly, we give a benefit to a merge if ever, in the case of k-means, the greedy algorithm becomes
it decreases the distance gap between elements in the merge particularly easy. At time t, for a current centroid ctj , let
and elements not in the merge. The joint formulation is then ct−1
f (j) ∈ Ct−1 be the closest centroid in Ct−1 . Let nj =
t

as follows: |closest(j)| be the number of points belonging to cluster j

simM (m) − cp · E (dT ′ (i, j) − dm
T (i, j))
at time t; let nt−1 be the corresponding number for ct−1 .
i∈leaf (mℓ )
“ f (j) ” f (j)
t t t−1 t
j∈leaf (mr ) Let γ = nj / nj + nf (j) . Then, update cj as
+ cp · E (dT ′ (i, j) − dm
T (i, j)) . (7)
i∈leaf(m)
j6∈leaf (m)
ctj ← (1 − γ) · cp ct−1
f (j)

This heuristic considers the internal cost of merging ele- +γ · (1 − cp) E (x).
x∈closest(j)
ments i ∈ S1 and j ∈ S2 , and the external cost of merging
elements i ∈ S1 ∪ S2 and j 6∈ S1 ∪ S2 ; therefore, we refer to In words, the new centroid ctj lies in between the centroid
it as Linear-Both. For completeness, we also consider the suggested by non-evolutionary k-means and its closest match
external cost alone: from the previous timestep, weighted by the cp and the rel-
ative sizes of these two clusters. Again, this is normalized
simM (m) + cp · E (dT ′ (i, j) − dm
T (i, j)) . (8) to unit length, and we continue with the usual k-means it-
i∈leaf (m)
j6∈leaf (m)
erations.
We refer to this final heuristic as Linear-External.
5. EXPERIMENTS
4.2 k-means clustering
In this Section, we perform an extensive study of our al-
Let the objects to be clustered be normalized to unit vec- gorithms under different parameter settings. We show how
tors in the Euclidean space, i.e., the objects at time t are distance from history can be reduced significantly while still
given by Ut = {x1,t , . . .} where each xi,t ∈ ℜℓ and the dis- maintaining very high snapshot quality. For our experi-
tance matrix Mt (i, j) = dist(i, j, t) = ||xi,t − xj,t ||. (See, for ments, we use the collection of timestamped photo–tag pairs
instance, [7].) from flickr.com indicating that at a given time, a certain
We begin with a description of the traditional k-means tag was placed on a photo. A bipartite tag-photo graph is
algorithm. Let t be a fixed timestep and let U = U≤t , xi = formed for each week, and two tags are considered to be sim-
xi,t , M = Mt . The algorithm begins with a set of k cluster ilar if they co-occur on the same photo at the same timestep,
centroids, c1 , . . . , ck , with ci ∈ ℜℓ ; these centroids can be as described before in Section 3. Our goal is to apply evo-
initialized either randomly, or by using the results of the lutionary clustering algorithms to this space of tags.
previous clustering Ct−1 (which is exactly “incremental k-
means”). Let closest(j) be the set of all points that are k-means clustering over time. For this experiment, we
closest to centroid cj , i.e., selected the most commonly occurring 5000 tags that in the
Flickr data and proceeded to study their clustering. We ran
closest(j) = {x ∈ U | j = arg min ||cj ′ − x||}. k-means with k = 10 centroids over time t = 0 . . . 67, for
j ′ =1,...,k
several values of cp. Recall that cp = 0 is exactly the same 6. CONCLUSIONS
as applying k-means independently to each snapshot, but We considered the problem of clustering data over time
with the clusters found in the previous step as the starting and proposed an evolutionary clustering framework. This
seed; it is “incremental k-means,” in other words. framework requires that the clustering at any point in time
Figure 1 shows the results. We observe the following: should be of high quality while ensuring that the clustering
Both the snapshot quality and the distance from history de- does not change dramatically from one timestep to the next.
crease as cp increases. In fact, incremental k-means (cp = 0) We presented two instantiations of this framework: k-means
gives the best snapshot quality and worst distance from his- and agglomerative hierarchical clustering. Our experiments
tory. This is to be expected since clustering each snapshot on Flickr tags showed that these algorithms have the desired
independently should give the best quality performance, but properties — obtaining a solution that balances both the
at the cost of high distance from history. Also, even low val- current and historical behavior of data.
ues of cp lower the distance from history significantly. For It will be interesting to study this framework for a larger
example, even when cp is as low as 0.125, k-means incorpo- family of clustering algorithms. It will also be interesting to
rates history very well, which results in a significant drop in investigate tree-based clustering algorithms that construct
distance from history. non-binary and weighted trees.
Agglomerative clustering over time. We empirically 7. REFERENCES
find that Linear-Both and Linear-Internal significantly out- [1] C. Aggarwal, J. Han, J. Wang, and P. S. Yu. A framework for
perform both Linear-External and Squared, so in Figure 2, clustering evolving data streams. In Proceedings of the
International Conference on Very Large Data Bases, pages
we plot only the performance of Linear-Both and Linear- 852–863, 2003.
Internal over the top 2000 tags. The plots for Linear-Both [2] J. Allan, J. Carbonell, G. Doddington, J. Yamron, and
are smoother than those for Linear-Internal, for all values Y. Yang. Topic detection and tracking pilot study: Final
report. Proc. of the DARPA Broadcast News Transcription and
of the change parameter cp. This demonstrates that the ex- Understanding Workshop, 1998.
tra processing for Linear-Both improves the cluster tracking [3] P. Auer and M. Warmuth. Tracking the best disjunction. In
ability of the algorithm. Also note that the distance from Proceedings of the 36th Annual Symposium on Foundations
history plot shows very high values for a few timesteps. We of Computer Science, pages 312–321, 1995.
[4] D. Blei, T. Griffiths, M. Jordan, and J. Tenenbaum.
suspect this is due to increased activity during that time- Hierarchical topic models and the nested Chinese restaurant
frame; that was when Flickr “took off.” Note that this peak process. In S. Thrun, L. Saul, and B. Schölkopf, editors,
also appears during k-means clustering (Figure 1(b)), rein- Advances in Neural Information Processing Systems 16, 2004.
forcing the idea that this is an artifact of the data. [5] C. Chatfield. The Analysis of Time Series. Chapman and
Hall, 1984.
Effect of cp on snapshot quality. Figure 3(a,b) shows [6] S. Chien and N. Immorlica. Semantic similarity between search
engine queries using temporal correlation. In Proceedings of the
the dependence of snapshot quality on cp. The snapshot International Conference on the World-Wide Web, pages
quality values at time t are normalized by the corresponding 2–11, 2005.
value for cp = 0 to remove the effects of any artifacts in the [7] I. Dhillon and D. S. Modha. Concept decompositions for large
data itself. We observe that the snapshot quality is inversely sparse text data using clustering. Machine Learning,
42:143–175, 2001.
related to cp. I.e., higher the cp, more the weight assigned to [8] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern
the distance from history, and thus worse the performance Classification. Wiley-Interscience, 2000.
on snapshot quality. [9] D. Fisher. Knowledge acquisition via incremental conceptual
However, while the snapshot quality decreases linearly and clustering. Machine Learning, 2:139–172, 1987.
[10] S. Guha, N. Mishra, R. Motwani, and L. O’Callaghan.
is well-behaved as a function of cp for k-means, the situa- Clustering data streams. In IEEE Symposium on Foundations
tion is different for agglomerative clustering. The snapshot of Computer Science, pages 359–366, 2000.
quality takes a hit as soon as history is incorporated even [11] M. Herbster and M. K. Warmuth. Tracking the best linear
a little bit, but the degradation after that is gentler. This predictor. Journal of Machine Learning Research, 1:281–309,
2001.
suggests that k-means can accommodate more of history [12] J. Lin, M. Vlachos, E. Keogh, and D. Gunopulos. Iterative
without compromising the snapshot quality. incremental clustering of time series. In Proceedings of the
International Conference on Extending Database Technology,
Effect of cp on distance from history. Figure 3(c,d) pages 106–122, 2004.
shows the dependence of distance from history on the change [13] M. Meila. Comparing clusterings by the variation of
parameter cp. The y-axis values are normalized by the cor- information. In Proceedings of the ACM Conference on
Computational Learning Theory, pages 173–187, 2003.
responding value for cp = 0 at that timestep to remove [14] P. Smyth. Clustering sequences with hidden Markov models. In
any data artifacts. We see that the distance from history Advances in Neural Information Processing Systems,
is inversely related with cp. I.e., as the value of cp is in- volume 9, page 648, 1997.
creased, our algorithms weigh the distance higher, and re- [15] M. Vlachos, C. Meek, Z. Vagena, and D. Gunopulos. Identifying
similarities, periodicities and bursts for online search queries.
ducing the distance from history becomes relatively more In Proceedings of the ACM SIGMOD International
important than increasing snapshot quality. Thus, higher Conference on Management of Data, pages 131–142, 2004.
cp leads to lower distance from history. [16] I. Witten and E. Frank. Data Mining: Practical Machine
Learning Tools and Techniques. Morgan Kaufmann, 2005.
While k-means gets closer to history for small values of cp,
[17] Y. Yang, T. Pierce, and J. Carbonell. A study on retrospective
the situation is more dramatic with agglomerative cluster- and on-line event detection. In Proceedings of the 21st ACM
ing. Even values of cp as small as 0.05 reduce the distance International Conference on Research and Development in
from history in a dramatic fashion. This suggests that the Information Retrieval, pages 28–36, 1998.
[18] J. Zhang, Z. Ghahramani, and Y. Yang. A probabilistic model
agglomerative clustering algorithm is easily influenced by for online document clustering with applications to novelty
history. detection. In Proceedings of Advances in Neural Information
Processing Systems, 2005.
 2 0.8
cp = 0 cp = 0
1.8 cp = 0.125 0.7 cp = 0.125
cp = 0.25 cp = 0.25
1.6 cp = 0.5 cp = 0.5

Distance from history

0.6
cp = 0.75 cp = 0.75

Snapshot quality
1.4
0.5
1.2
0.4
1
0.3
0.8
0.6 0.2

0.4 0.1

0.2 0
0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70
Time Time
(a) Snapshot quality over time (b) Distance from history, over time

Figure 1: k-means clusters over time: As the change parameter cp increases, both the snapshot quality and
the distance from history decrease. The case of cp = 0 is “incremental k-means.”

0.16 10000
cp = 0 cp = 0
cp = 0.05 cp = 0.05
0.14 cp = 0.1 cp = 0.1
cp = 0.2 cp = 0.2

Distance from history

1000
0.12 cp = 0.3 cp = 0.3
Snapshot quality

0.1
100
0.08

0.06
10
0.04

0.02 1
0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70
Time Time
(a) Linear-Both snapshot quality (log-linear) (b) Linear-Both distance from history (log-linear)
0.16 1e+06
cp = 0 cp = 0
0.14 cp = 0.05 100000 cp = 0.05
cp = 0.1 cp = 0.1
cp = 0.2 10000 cp = 0.2
Distance from history

0.12
cp = 0.3 cp = 0.3
Snapshot quality

1000
0.1
100
0.08
10
0.06
1
0.04 0.1
0.02 0.01
0 0.001
0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70
Time Time
(a) Linear-Internal snapshot quality (log-linear) (b) Linear-Internal distance from history (log-linear)

Figure 2: Performance of agglomerative clustering over time: The plots for Linear-Both are far smoother than
those of Linear-Internal.
1 1 1 1
0.95 0.9
Distance from history (wrt cp=0)

Distance from history (wrt cp=0)

0.9 0.9
Snapshot quality (wrt cp=0)

Snapshot quality (wrt cp=0)

0.8 0.9 0.8

0.8
0.85 0.7
0.7
0.8 0.7 0.6
0.6
0.75 0.6 0.5
0.5
0.7 0.5 0.4
0.4
0.65 0.3
0.3 0.4
0.6 0.2
0.2 0.55 0.3 0.1
0.1 0.5 0.2 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 0.05 0.1 0.15 0.2 0.25 0.3
Change parameter (cp) Change parameter (cp) Change parameter (cp) Change parameter (cp)

(a) k-means snapshot (b) Agglomerative snapshot (c) k-means distance (d) Agglomerative distance
quality vs. cp quality vs. cp from history vs. cp from history vs. cp

Figure 3: Snapshot quality and distance from history, versus the change parameter cp.