On Similarity-Based Queries for Time Series Data

Davood Rafiei
Department of Computer Science, University of Toronto
E-mail: [email protected]

Abstract tween two time sequences is to compute the Euclidean dis-

tance (or any other distance, such as the city-block distance)
We study similarity queries for time series data where between the two sequences, and call the two sequences sim-
similarity is defined in terms of a set of linear transforma- ilar if their distance is less than some user-defined thresh-
tions on the Fourier series representation of a sequence. We old. However, there are many similarity queries that such
have shown in an earlier work that this set of transforma- a simple notion of similarity fails to capture; for example,
tions is rich enough to formulate operations such as moving one may consider two stocks similar if they have almost the
average and time scaling. same price fluctuations, even though one stock might sell for
In this paper, we present a new algorithm for process- twice as much as the other. Consider the following motivat-
ing queries that define similarity in terms of multiple trans- ing examples.
formations instead of a single one. The idea is, instead COMPV 94/06/15 NYV 94/06/15 normalized and 9−day MV

of searching the index multiple times and each time apply- 100

80
500 1

400 0.5
ing a single transformation, to search the index only once 60
300 0
and apply a collection of transformations simultaneously to 40

200 −0.5

the index. Our experimental results on both synthetic and 20

0 100 −1
0 50 100 150 0 50 100 150 0 50 100 150
real data show that the new algorithm for simultaneously
COMPV 94/06/15 DECL 94/06/15 normalized and 19−day MV

processing multiple transformations is much faster than se- 100 2000 1

80

quential scanning or index traversal using one transforma- 60

1500
0.5

0
tion at a time. We also examine the possibility of composing 40
1000
−0.5
transformations in a query or of rewriting a query expres- 20

0 500 −1

sion such that the resulting query can be efficiently evalu- 0 50 100 150 0 50 100 150 0 50 100 150

ated.
Figure 1. On the top from left to right,
daily closings of Dow Jones 65 Composite Volume
(COMPV) index, NYSE Volume (NYV) index and
1. Introduction both put together, normalized and smoothed
using 9-day moving average. On the bot-
Time-series data are of growing importance in many new tom from left to right, again daily closings of
database applications, such as data mining or data warehous- COMPV index, NYSE Declining Issues (DECL) in-
ing. A time series is a sequence of real numbers, each num- dex and both put together, normalized and
ber representing a value at a time point. For example, the se- smoothed using 19-day moving average.
quence could represent stock or commodity prices, sales, ex-
change rates, weather data, biomedical measurements, etc.
We are often interested in similarity queries on time-series
data [3, 2]. For example, we may want to find stocks that be- Example 1.1 Figure 1 shows daily closings of three in-
have in approximately the same way (or approximately the dices: Dow Jones 65 Composite Volume (COMPV), NYSE
opposite way, for hedging); or products that had similar sell- Volume (NYV) and NYSE Declining Issues (DECL). It is dif-
ing patterns during the last year; or years when the temper- ficult to see any similarity between these sequences. The
ature patterns in two regions of the world were similar. In Euclidean distance between closes of COMPV and NYV is
queries of this type, approximate, rather than exact, match- 2873 and that between COMPV and DECL is 12939. On the
ing is required. other hand, if we normalize 1 closes of COMPV and NYV
A simple approach to determine a possible similarity be- 1 This operation is described in Section 3.
and compare their 9-day moving averages, they look similar. on February 8th. No value is recorded for February 4th, 5th
The Euclidean distance between 9-day moving averages of and 6th. If we shift the momentum of PCG two days to the
normalized closes of COMPV and NYV is less than 3. Sim- right, the spikes will overlap and the Euclidean distance will
ilarly, if we normalize the closes of COMPV and DECL and reduce to 5.65.
compare their 19-day moving averages, they also look simi-
lar. In fact, ‘19-day moving average’ is the shortest moving The momentum of a sequence describes the rate at which
average that reduces the Euclidean distance between nor- its value (such as the price in the preceding example) is ris-
malized closes of COMPV and DECL to less than 3. ing or falling and it is seen as a measure of strength be-
hind upward or downward movements. On the other hand,
Moving averages are widely used in stock data analysis shifting a sequence horizontally before comparing it to an-
(for example, see [5]). Their primary use is to smooth out other sequence removes any possible delay between the two
short term fluctuations and depict the underlying trend of sequences which can arise, for example in the stock mar-
a stock. Given two sequences to be compared, we usually ket domain because of different reactions of two stocks to
do not know what moving average can make them similar. the same piece of news or recording errors. Both momen-
There can be several moving averages that reduce the dis- tum and shifting can be formulated as linear transformations
tance between two sequences to less than a threshold. We over the Fourier representation of a sequence (see the ex-
are often interested in the shortest moving average mainly tended version of this paper for details [10]). In general,
because it leaves more details to the distance computation there can be several possible linear transformations (or time
process 2 . Moving averages can be formulated as linear shifts, as an example) to be applied to sequences and each
transformations over the Fourier representation of a time se- transformation can either reduce or increase the distance be-
quence [12]. tween sequences. However, for every pair of sequences we
are usually interested in finding transformations that reduce
PCG 941102 PCL 941102
6 8

6
the distance between them to a minimum.
4
4 In this paper, we propose a fast algorithm to process
2 2

0
0 queries that specify more than one transformation as the ba-
−2

−2
0 50 100 150
−4
0 50 100 150
sis for similarity. The idea is, instead of processing a single
5
Momentum of PCG
6
Momentum of PCL transformation at a time, to process a collection of them at
4 once. To achieve this goal, we construct a minimum bound-
2

0 0 ing rectangle (MBR) for transformations. We show that the

−2

−4
minimum bounding rectangle for transformations can be ap-
−5
0 50 100 150
−6
0 50 100 150 plied to a multidimensional index constructed on sequences,
thus reducing the number of searches over the index to one.
Figure 2. The daily closing price of Pacific Gas Our experiments show that this algorithm performs much
and Electric Co. (PCG) and that of Plum Creek Tim- better than both sequentially scanning all sequences and also
ber Co. (PCL), both starting from 94/11/02 for the index traversal using one transformation at a time. We
128 days, represented in normal forms and also examine the possibility of composing transformations
their momentums. in a query or of rewriting a query expression such that the
resulting query can be efficiently evaluated.
The organization of the rest of the paper is as follows. In
the next section we review the related work. The benefits
Example 1.2 Figure 2 shows in normal form the daily clos- of using transformations for expressing similarity queries is
ing prices of stocks of Pacific Gas and Electric Co. (PCG) discussed in Section 3. In Section 4 we propose algorithms
and Plum Creek Timber Co. (PCL) both starting from for fast processing queries that express similarity in terms
November 2nd, 1994 for 128 days. One way to compare the of multiple transformations. Section 5 contains experiments
change rates of two stocks is to compare their “momenta”, that show the effectiveness of our algorithms. Section 6 is
which are obtained for every stock by subtracting the price the conclusion.
at time t from the price at time t+1 (or, in general, t+n for
some ). The Euclidean distance between the two momenta
is 13.01. The series representing the price of PCG has a 2. Related Work
spike on February 3rd while the series of PCL has a spike
An indexing technique for the fast retrieval of similar
2 Although



it seems if two sequences are similar w.r.t. -day moving
time sequences is proposed by Agrawal et al. [1]. The idea
average, they should be similar w.r.t. -day moving average, this is
not true in general; a counter example can be found in the extended version is to use Discrete Fourier Transform (DFT) to map time se-
of this paper [10]. quences (stored in a database) into the frequency domain.


    ,

:@?
BA <

The DFT of time sequence
denoted by , is given by
for maps to . This class of transformations can eas-
ily express operations such as moving average, momentum,
 
    !
'&)(+*#,'2 -/.10 3 4 5  (1)
time shift, etc. The expressions of these transformations and
more can be found somewhere else [12, 11].
# "%$
 7
where 6 
3.1. Transformations - Normal Form
is the imaginary unit. Keeping only the

first k Fourier coefficients, each sequence becomes a point in An efficient way to compare two time sequences is to
a k-dimensional feature space. To allow a fast retrieval, the compare their normal forms. Given a time sequence
C D
8 F E D  G C F E D =
authors keep the first k Fourier coefficients of a sequence in


of mean and standard deviation , the transformation
a R-tree index. In an earlier work [13], as a major improve- applied to gives its normal form. Due to the
ment over this technique, we show that the last few Fourier linearity property of DFT, the same transformation is appli-
coefficients of a sequence are as important as the first few cable to the Fourier representation of a sequence.
coefficients due to the symmetry property of DFT. We also Although it is not required by the algorithms given in this
show that using the symmetry property improves the search paper, we assume time sequences are normalized and for ev-
time of the index by more than a factor of 2 without increas- ery sequence, its normal form along with its mean and stan-
ing its dimensionality. dard deviation are stored in a relation. This is mainly be-
In another work [12], we use this indexing method and cause of efficiency (as is noted by Goldin et al. [7]) and the
propose techniques for retrieving similar time sequences following two attractive properties of the normal form se-
whose differences can be removed by a linear transforma- quences which are not mentioned by Goldin et al. [7].
tion such as moving average, time scaling and inverting. In
this paper, we generalize our earlier work and allow queries
 IGK% L INM =
8 J
1. It minimizes the Euclidean distance with respect to the
H
that express similarity in terms of multiple transformations. scalar shift, i.e.
INK NI M
has its minimum
Our work here can be seen as an efficient implementation of
a special case of the query language described by Jagadish of and 4.
O
when and respectively are chosen to be the means

et al. [9] for time-series data.

2. The Euclidean distance between two normalized se-
There are other related works on time series data. An ex-
quences is directly related to their cross-correlation 5 .

HP 8   L = 4Q8  B ;R;8   L =S=

tension of the indexing technique of Agrawal et al. [1] for
subsequence matching is proposed by Faloutsos et al. [6]. (2)
Goldin et al. [7] show that the similarity retrieval will be
invariant to simple shifts and scales if sequences are nor- This can be derived by expanding the Euclidean dis-
malized before being stored in the index. Yi et al. [18] use tance formula and replacing the mean and the standard
time warping as a distance function and present algorithms deviation respectively by 0 and 1 in both the Euclidean
for retrieving similar time sequences under this function. A distance and the cross-correlation formulas.
query language for time series data in the stock market do-
main is developed by Roth [14]. More related works include The second property can be quite useful in formulating
the work of Agrawal et al. [3] on querying shapes of histo- similarity queries or translating one query to another. Since
ries, the work of Shatkay and Zdonik [17] on representing the Euclidean distance between two sequences can range
sequences in terms of some functions and the work of Se- from zero to infinity, it is usually difficult to specify a thresh-
shadri et al. [16] on querying sequences in general. old for this distance. Instead, we can specify a threshold for
cross-correlation which is between 0 and 1 and plug it into
3. Similarity Queries and Transformations Equation 2 to find a threshold for the Euclidean distance.
Using Equation 2, we can also translate any expression that
uses the cross-correlation in a query to one that uses the Eu-
A transformation can be seen as a way to remove certain
clidean distance or vice versa.
variations before aligning two sequences. Although many
kinds of variations may be present in each sequence, we con-
sider only those that can be removed using linear transfor-
3.2. Composing Transformations in a Query
mations on the Fourier series representation of the sequence.
98 :; <>=
More formaly, a transformation is a pair of real 3 vectors, de- In a query, we may specify a sequence of transformations
noted by . The transformation applied to a point to be applied to a time sequence. For example, we may want
4 This TVU
TW andZ 7equating
3 We have shown earlier [12] how a transformation described in terms can be verified by taking the first derivatives of D w.r.t. and

X Y[ \ Y] ^`_bc%a d i e a c f h _gi cbe a h _%e a

of two complex vectors can be mapped into one expressed in terms of two them to zero.

a a
5
real vectors under a safety constraint.
age”, for I 4




and    
to apply a “s-day shift” followed by an “m-day moving aver-



 , to a sequence.
of IBM. A solution for processing this query is to scan the
whole stocks relation, compute the  -day moving average
We claim the queries expressed in terms of such a sequence for the closing price of every stock and

determine if the re-
of transformations also benefit from the algorithms given in sulting sequence is within distance of the  -day moving
this paper. We show this by giving a method to translate average of the close of IBM. The distance predicate needs
any query expression that uses a sequence of transforma- to be checked for all possible transformations. We refer to
tions into one that uses only a set of transformations. The this algorithm as the sequential-scan method. The cost of
resulting query can then be processed using the same tech- this algorithm includes one scan of the whole relation and
I I ?
  8:  < =  8: < =
nique that we present for multiple transformations. computing the distance predicate  times.
Given transformations and
for example respectively corresponding to “2-day shift” and
P,
P P Another approach is for every  , apply t to the in-
dex built on the first few Fourier coefficients of the closing
8 =
“10-day moving average”, suppose we want to apply fol-
  price and do a range query on the new index [12]. The union
P
lowed by , which we denote by
P 
, to sequence . We
can construct the new transformation as follows:
of these results for all  gives the query answer. We
call this algorithm ST-index, where ST stands for ‘a Single
8 8  ==  : ? 8 :  ?  A <  = A <
P 
Transformation at a time’. The cost of this algorithm in-

 : P ? :  ?  A : ? <  AP <
(3) cludes traversing the index times. Next, we describe a
P P P new algorithm that requires a single scan of the index and

8 =
 equivalently can be expressed as  
performs much better than both the sequential-scan and ST-

8 :   <  = where :  P  : ? :  and <   : ? <  A < .

Transformation index algorithms. We shall refer to this new algorithm by
P P
We can use this result to compose two sets of trans-
P MT-index, where MT stands for ‘Multiple Transformations
at a time’.
formations. Given two transformation sets  and
P, 4.1. MT-Index Algorithm for Multiple Transforma-
for I  


 and “m-day moving average” for  
for example respectively corresponding to “s-day shift”
8 =,
tions



 , we can construct transformation set  
P  8 :; <G= :
which corresponds to a “s-day shift” followed by an “m-day
I  <
A transformation is of the form where
moving average” for all possible values of and , as fol- and are -dimensional real vectors. Thus, a transforma-
Q
lows:
    8  =      tion can be represented as a point in a -dimensional space.

= P P P (4) Given a query that requires a set of transformations to be ap-

where 8  is defined by Equation 3. Using Equations 3
plied to a set of data sequences (or points), we first construct
P
and 4, we can simplify a query by replacing any expression
a minimum bounding rectangle (MBR) for all transforma-
tions. Having a multidimensional index for time sequences,
that uses a sequence of transformations with one that uses
we can apply the transformation rectangle to entries of the
only a single or a set of transformations. We can process the
index. For a point data set, entries of a multidimensional
resulting query using the techniques described in the next
index (such as R-tree) are usually in the form of points or
section.
rectangles. Since a point can be seen as a special kind of a
rectangle with its lower bound equal to its upper bound in
4. Processing Similarity Queries every dimension, we only consider applying a transforma-
tion rectangle to a data rectangle.
We consider spatial queries, namely range queries, spa-
Q
To apply a transformation rectangle to a data rectangle,
tial join queries and nearest neighbor queries and allow our
transformations to be used in those queries. We discuss the
we decompose the -dimensional transformation rectan-
gle into two -dimensional MBRs, one corresponding to :
issue of processing range queries in more detail and the two <
which we denote by mult-MBR, and the other correspond-
others very briefly. We start with the following range query:  8    =
ing to which we denote by add-MBR. Given mult-MBR:
  "!




# and add-MBR: %$  $ ! 8    =
Query 1: “Given the closing price of a stock data rectangle X:

8    = 
!



# and



# , the result of apply-

find every stock I  I

 I
q and a set of transformations denoted by T,
and transforma-
ing
 8L  L  = 
mult-MBR and add-MBR to rectangle X is rectangle Y:
 !


# where
/  E  / E L'&)( *
H 8 8 I I& =  8 I & =S=

tion


such that the Euclidean distance

   $ &A
,+ 8 &  ? -&  . ? -& ! . ? - &  . ? -& ! =
.” (5)
& & &
 ! !
As a specific example, T could
 
 
be the set of  -day mov- L &! /  $ &! A
 :
8 &  ? -&  0 ? -& ! 1 ? -&  0 ? 2& ! =
ing averages for 


and we may want to find
& & &
all stocks that have an  -day moving average similar to that  ! !
for all dimensions + . As an example, consider the points of
m-day moving average for 


 . Figure 3 shows    Algorithm 1 : Given an R-tree index which is built on the
first k Fourier coefficients of a sequence and whose root is

the magnitudes and the angles of these points at the second N, a transformation set , a threshold , and a search point
DFT coefficient and their decompositions into mult-MBR  , use the index to find all sequences that become within dis-

and add-MBR. It can be observed that points inside mult- tance of  after being transformed by a member of .
MBR make a horizontal line at 1. This is due to the fact that
1. Build MBR for points in and project into a mult-
a data point angle is multiplied by 1. Similarly points inside
MBR and an add-MBR as described above.
add-MBR make a vertical line at 0 to show the fact that a

data point magnitude is added by 0. The result of applying 2. Build a search rectangle of width around  . We call
these MBRs to a data rectangle is shown in Figure 4. this rectangle 
 


 
 .

MV1−40 mult−MBR for MV1−40 add−MBR for MV1−40
3. If is not a leaf, apply the mult-MBR and the add-

0 2 0

MBR to every (rectangle) entry of using Equation 5


−0.1 1.8 −0.1

−0.2 1.6 −0.2

and check if the resulting rectangle intersects 
 . For
−0.3 1.4 −0.3

−0.4 1.2 −0.4

every intersecting entry, go to step 3 and do this step on
the index rooted at the node of the intersecting entry.
angle(F2)

angle(F2)

−0.5 1 −0.5

−0.6 0.8 −0.6 

4. If is a leaf, apply the mult-MBR and the add-MBR
−0.7 0.6 −0.7

−0.8

−0.9
0.4

0.2
−0.8

−0.9
to every (point) entry of
 and check if the resulting
rectangle intersects 
 . If so, the entry is a candidate.
−1 0 −1
0.8 0.9 1 0.8 0.9 1 −1 0 1
abs(F2) abs(F2)
5. For every candidate entry, retrieve its full database
record, apply all transformations inside to the se-
Figure 3. The second DFT coefficients of  -
day moving averages (for 


 ) and    quence, and determine transformations that reduce the
Euclidean distance between  the data sequence and the
their decompositions into mult-MBR and add- query sequence to less than .
MBR
This algorithm is guaranteed not to miss any qualifying
sequence (the proof is given in the extended version of this
paper [10]). We can develop similar algorithms for effi-
ciently processing spatial join and nearest neighbor queries.
In a spatial join query, we apply the transformation MBR to
angle(F 2 )

3 3 all data items used in the join predicate before computing

the predicate. For example, we may want to find all pairs
1

7 17
1 * 1 - 0.96
0.85 * 7 + 0 17
 -day moving average for some    
of stocks that have similar closing prices with respect





to an
. Having
abs(F2 ) an R-tree index for the closing prices, we can use any well-
known spatial join algorithm for R-tree and change the join
Figure 4. A data rectangle before and after condition such that the transformation rectangle be applied
being transformed to both data rectangles involved in the join before testing
them for a possible overlap. Similarly in a nearest neighbor
query, as we walk down the tree, we apply the transforma-
To develop an algorithm for answering Query 1, suppose tion MBR to all entries of the node we visit. We can then use
an R-tree index is available on sequences. We can apply the any kind of metric (such as MINDIST or MINMAXDIST
transformation rectangle to every data rectangle in the index [15]) to prune the search.
and construct a new index on the fly. The new index is con-
structed one index rectangle at a time, and each time the new 4.2. Performance Improvement
rectangle is checked to see whether it intersects the query re-
gion. This process retrieves a set of candidate data items that A potential problem with the MT-index algorithm is if
includes all qualifying data items plus some false positives. transformations make several clusters or a few of them
The last step of the algorithm removes false positives by ap- spread all over the space, then the minimum bounding rect-
plying every member of the transformation set to every can- angle of transformations will cover a large area. This MBR,
didate data item and selecting data items that intersect the when applied to a data rectangle, can easily make the data
query region. We can write the search algorithm more for- rectangle intersect the query region. This can reduce the fil-
mally as follows: tering power of the index dramatically. A solution for this
problem is to allow more than one transformation rectan- need to apply all scale factors to sequences. Instead, we need
gle. As the number of MBRs goes up, the area of each MBR
I
to find the largest scale factor that makes the distance predi-
&
gets smaller, and as a result the filtering power of the MBR
increases; but, on the other hand, the same index needs to I
cate true. Suppose is such a scale factor. One way to find
&
is to do a binary search on the set of scale factors. Defi-
be traversed several times. In the worst case, the number of
MBRs is the same as the number of transformations, i.e. ev- I
nition 1 easily implies that the distance predicate is true for
all scale factors less than .
&

ery MBR includes only one transformation point. In such a We can use the binary search technique in all three al-
case, both ST-index and MT-index perform exactly the same. gorithms described earlier. In the case of the sequential
Now the question is how we should optimally choose scan method, we still need to scan the whole stocks rela-
MBRs for a given set of transformations such that the cost
of Algorithm 1 (in terms of the number of disk accesses) be- I
to "I ?
tion. However, the number of sequence comparisons drops
 . Similarly in the case of the MT-index
comes minimum. One solution is to estimate the cost for algorithm, the number of disk accesses still will be the same,
any possible set of MBRs and choose the set with minimum but the number of comparisons for every candidate sequence
cost. A first attempt in estimating the cost for a given set of drops to  . The ordering assumption reduces the num-
MBRs is to use the total area of MBRs. However, the total ber of index traversals for ST-index to  .
area is minimum if every MBR includes only one transfor- On the other hand, the ordering assumption does not hold
mation point, i.e. the ST-index algorithm is used. Another in general. There are useful transformations that are not or-
approach for estimating the cost of a given set of MBRs is dered w.r.t. time sequences and the Euclidean distance. For
to apply MBRs for a fixed data rectangle, say a unit square, example, we can show that no ordering is possible for a set
then compute the total area of the resulting data rectangles. of moving averages w.r.t. time sequences and the Euclidean
Due to this estimation, the best performance should be ob- distance (see the extended version of this paper for a proof
tained using only one transformation rectangle. [10]).
However, our experiments showed that using one trans-
formation rectangle did not necessarily give the best per- 5. Experimental Results
formance. The worst performance for MT-index, which is
close to that of ST-index, is when we pack two clusters
We implemented both ST-index and MT-index, on top of
of transformations into one rectangle. A solution to avoid
Norbert Beckmann’s Version 2 implementation of the R*-
this problem is to use a cluster detection algorithm (such as
tree [4]. We ran experiments on both stock prices data ob-
CURE [8]) and avoid packing two clusters into one rectan-
tained from the ftp site “ftp.ai.mit.edu/pub/stocks/results”
gle.
and synthetic data. All our experiments were conducted on
a 168MHZ Ultrasparc station. The stock prices database
4.3. Ordering Assumption on Transformations
consisted of 1068 stocks and for each stock its daily clos-
So far, we have made no assumption on any possible or-
9

 
   A N N
ing prices for 128 days. Each synthetic sequence was in the
 and  is a uniformly
dering among transformations. In this section, we define a
form of where
distributed random number in the range      

.
notion of ordering among transformations and show that it For every time series, we first transformed it to the nor-
can be quite useful in guiding the search process more effec- mal form for reasons described in Section 3.1, and then we
tively. found its Fourier coefficients. Since the mean of a normal

#    form series is zero by definition, the first Fourier coefficient
 P  
Definition 1 We 
call an ordering of



is always zero, so we can throw it away. For every sequence,
&
tion D if     
    , 
w.r.t. value domain dom and distance func-
we stored the magnitudes and the angles of the second and

8  8  & =   8  ==  8  8  & =   8  ==

the third DFT coefficients in the index. We used the symme-
  try property of DFT in all our experiments over the index.
We report our experiments in two parts. In the first part,
Once an ordering is established among transformations, we we compare MT-index to ST-index and sequential scan. In
can use this ordering to guide the search more cleverly.
 this part, we made the choice of packing all transformations
+Q



To give an example,
 consider Query 1 and assume
represents a set of scaling factors. It is easy
into one rectangle though it did not necessarily give us the
best possible performance of MT-index. In the second part,
to show that “less than” defines an ordering among mem- we varied the number of transformation rectangles from one
bers of T w.r.t. the domain of time sequences and the Eu- to its maximum to see the effect of having multiple transfor-
clidean distance (see the extended version of this paper for mation rectangles on the performance of MT-index. In all
a proof [10]). To find all transformations that make a data our experiments over range queries, we ran each experiment
sequence to become similar to a query sequence we do not 100 times and each time we chose a random query sequence
 2.5
from the  data set and searched for all other sequences within
: sequential−scan
distance of the query sequence. We averaged the execution : ST−index

times from these runs. We also set the correlation threshold 2

: MT−index

fixed to 0.96 for all range queries. We plugged this thresh-

running time (seconds)

old in Equation 2 to find a value for the Euclidean distance 1.5

threshold.
1

5.1. Comparing MT-index to ST-index and Sequen-

tial Scan 0.5

12
0
0 5 10 15 20 25 30
: sequential−scan number of transformations

10 : ST−index

: MT−index

8
Figure 6. Time per query varying the number
of transformations for range queries
running time (seconds)

4
ble performance. To show this, we ran Query 1 using MT-
2
index algorithm on real stock prices data, but this time we
varied the number of transformations per MBR from one to
its maximum. The transformation set consisted of  -day
0
0 2000 4000 6000
number of sequences
8000 10000 12000
moving averages for 


. We equally partitioned 
 >Q
subsequent transformations and built an MBR for each parti-
Figure 5. Time per query varying the number tion. As is shown in Figure 7, despite the fact that collecting
of sequences for range queries all transformations in one rectangle resulted in the minimum
number of disk accesses, it did not necessarily give us the
best performance mainly because of the increased number
Figure 5 shows the running time of Query 1 using three of false positives.
algorithms sequential-scan, ST-index, and MT-index. In
1600 14000
the experiment, we set the number of transformations fixed : mv6−mv29 : mv6−mv29

: mv6−mv29 and : mv6−mv29 and

to 16, but we varied the number of sequences from 500 1400 their inverted forms 12000 their inverted forms

to 12,000. The experiment ran on synthetic sequences of 1200 10000

length 128. The transformations were a set of moving aver-

running time (msec)

# of disk accesses

1000 8000

ages ranging from 10-day moving average to 25-day mov-

800 6000
ing average. The average output size was 7 or more de-
pending on the number of input sequences. The figure 600 4000

shows that MT-index performs better than both ST-index and 400 2000

sequential-scan. 200 0
0 10 20 30 40 50 0 10 20 30 40 50
Figure 6 shows the running time of Query 1 again using # of transformations per MBR # of transformations per MBR

three algorithms sequential-scan, ST-index, and MT-index.

In the experiment, we set the number of sequences fixed to Figure 7. Both the running time and the num-
1068, but we varied the number of transformations from 1 ber of disk accesses varying by the number
to 30. The transformations were a set of moving averages of transformations per MBRs
ranging from 5-day moving average to 34-day moving aver-
age. The experiment ran on real stock prices data. The av-
erage output size was 11 or more depending on the number We later added the inverted version of each transforma-
of transformations. The figure shows that MT-index outper- tion, which was obtained by multiplying every coefficient
forms both ST-index and sequential-scan. by -1, to the transformation set. This created two clusters
in a multidimensional space. Again, we equally partitioned
5.2. Multiple Transformation Rectangles subsequent transformations and built an MBR for each par-
tition. We varied the number of transformations per MBR
In this section, we show that grouping all transformations from one to 48 which was the size of the transformation
in one rectangle does not necessarily give us the best possi- set. As is shown in Figure 7, the running time shows bumps
when we pack one third or all of the transformations in a [3] R. Agrawal, G. Psaila, E. L. Wimmers, and M. Zait. Query-
rectangle. The same bumps are also observed in the num- ing shapes of histories. In Proc. of the 21st Intl. Conf. on
ber of disk accesses. This is due to the fact that in these two Very Large Data Bases (VLDB ’95), pages 502–514, Zurich,
cases the gap between two clusters is included in a transfor- September 1995.
[4] N. Beckmann, H.-P. Kriegel, R. Schneider, and B. Seeger.
mation rectangle.
The R* tree: an efficient and robust index method for points
These experiments show that as we start packing trans- and rectangles. In Proc. of the ACM SIGMOD Intl. Conf.
formations into rectangles, we see a major performance im- on Mgmt. of Data (SIGMOD ’90), pages 322–331, Atlantic
provement which continues up to a certain point (six to eight City, May 1990.
transformations per rectangle here). The performance after [5] R. D. Edwards and J. Magee. Technical analysis of stock
this point either stays the same or goes down. The worst per- trends. John Magee, Springfield, Massachusetts, 1969.
formance for MT-index, which was even worse than that of [6] C. Faloutsos, M. Ranganathan, and Y. Manolopoulos. Fast
ST-index, was when we packed two clusters of transforma- subsequence matching in time-series databases. In Proc. of
tions into one rectangle. A solution to avoid this problem the ACM SIGMOD Intl. Conf. on Mgmt. of Data (SIGMOD
’94), pages 419–429, Minneapolis, May 1994.
is to use a cluster detection algorithm in advance and avoid [7] D. Q. Goldin and P. C. Kanellakis. On similarity queries
packing more than one cluster to a rectangle. for time-series data: constraint specification and implemen-
tation. In 1st Intl. Conf. on the Principles and Practice of
6. Summary Constraint Prog., pages 137–153. LNCS 976, Sept. 1995.
[8] S. Guha, R. Rastogi, and K. Shim. CURE: an efficient clus-
tering algorithm for large databases. In Proc. of the ACM
We have proposed an efficient method for processing SIGMOD Intl. Conf. on Mgmt. of Data (SIGMOD ’98), pages
similarity queries that specify multiple transformations as 73–84, Seattle, June 1998.
the basis for similarity. We have shown that, instead of ap- [9] H. V. Jagadish, A. O. Mendelzon, and T. Milo. Similarity-
plying many single transformations to the index, we can based queries. In Proc. of the 14th ACM SIGACT-SIGMOD-
group transformations and apply a group of them simulta- SIGART Symp. on Principles of Database Systems (PODS
neously to the index. We have discussed the possibility of ’95), pages 36–45, San Jose, May 1995.
grouping transformations into multiple rectangles and its ef- [10] D. Rafiei. On similarity-based queries for time
series data. (extended version). Available at
fects on the performance of the algorithm. We have also
ftp.db.toronto.edu/pub/papers/icde99-ext.ps.Z.
shown that in the presence of some ordering among trans- [11] D. Rafiei. Similarity-based Queries on Time Series Data.
formations, the search can be guided more efficiently. We PhD thesis, University of Toronto, 1998.
evaluated our method over both real stock prices data and [12] D. Rafiei and A. Mendelzon. Similarity-based queries for
synthetic data. Our experiments confirm that the given al- time series data. In Proc. of the ACM SIGMOD Intl. Conf.
gorithm for handling multiple transformations outperforms on Mgmt. of Data (SIGMOD ’97), pages 13–24, Tucson, Ari-
both the sequential scanning and the index traversal using zona, May 1997.
one transformation at a time. [13] D. Rafiei and A. Mendelzon. Efficient retrieval of similar
time sequences using DFT. In Proc. of the 5th Intl. Conf.
on Found. of Data Org. and Alg. (FODO ’98), Kobe, Japan,
Acknowledgments November 1998.
[14] W. G. Roth. MIMSY: A system for analyzing time series
The author would like to thank Professor Alberto data in the stock market domain. University of Wisconsin,
Madison, 1993. Master Thesis.
Mendelzon for valuable suggestions and helpful discus-
[15] N. Roussopoulos, S. Kelley, and F. Vincent. Nearest neigh-
sions. This work was supported by the Natural Sciences bor queries. In Proc. of the ACM SIGMOD Intl. Conf. on
and Engineering Research Council of Canada and the Mgmt. of Data (SIGMOD ’95), pages 71–79, San Jose, May
Information Technology Research Centre of Ontario. 1995.
[16] P. Seshadri, M. Livny, and R. Ramakrishnan. Sequence
query processing. In Proc. of the ACM SIGMOD Intl. Conf.
References
on Mgmt. of Data (SIGMOD ’94), pages 430–441, Min-
neapolis, May 1994.
[1] R. Agrawal, C. Faloutsos, and A. Swami. Efficient similarity [17] H. Shatkay and S. Zdonic. Approximate queries and repre-
search in sequence databases. In Proc. of the 4th Intl. Conf. sentations for large data sequences. In Proc. of the 12th Intl.
on Found. of Data Org. and Alg. (FODO ’93), pages 69–84, Conf. on Data Eng. (ICDE ’96), pages 536–545, New Or-
Chicago, October 1993. leans, March 1996.
[2] R. Agrawal, K.-I. Lin, H. S. Sawhney, and K. Shim. Fast sim- [18] B.-K. Yi, H. V. Jagadish, and C. Faloutsos. Efficient retrieval
ilarity search in the presence of noise, scaling, and translation of similar time sequences under time warping. In Proc. of the
in time-series databases. In Proc. of the 21st Intl. Conf. on 14th Intl. Conf. on Data Eng. (ICDE ’98), pages 201–208,
Very Large Data Bases (VLDB ’95), pages 490–501, Zurich, Orlando, February 1998.
September 1995. Morgan Kaufmann Publishers.