A feature mining based approach for the classication of

text documents into disjoint classes

Salvador Nieto Saanchez
a
, Evangelos Triantaphyllou
a,
*
, Donald Kraft
b
a
Department of Industrial and Manufacturing Systems Engineering, 3128 CEBA Building,
Louisiana State University, Baton Rouge, LA 70803, USA
b
Department of Computer Science, 286 Coates Hall, Louisiana State University, Baton Rouge,
LA 70803, USA
Received 10 July 2000; accepted 7 September 2001
Abstract
This paper proposes a new approach for classifying text documents into two disjoint classes. The new
approach is based on extracting patterns, in the form of two logical expressions, which are dened on
various features (indexing terms) of the documents. The pattern extraction is aimed at providing de-
scriptions (in the form of two logical expressions) of the two classes of positive and negative examples. This
is achieved by means of a data mining approach, called One Clause At a Time (OCAT), which is based on
mathematical logic. The application of a logic-based approach to text document classication is critical
when one wishes to be able to justify why a particular document has been assigned to one class versus the
other class. This situation occurs, for instance, in declassifying documents that have been previously
considered important to national security and thus are currently being kept as secret. Some computational
experiments have investigated the eectiveness of the OCAT-based approach and compared it to the well-
known vector space model (VSM). These tests also have investigated nding the best indexing terms that
could be used in making these classication decisions. The results of these computational experiments on a
sample of 2897 text documents from the TIPSTER collection indicate that the rst approach has many
advantages over the VSM approach for solving this type of text document classication problem. More-
over, a guided strategy for the OCAT-based approach is presented for deciding which document one needs
to consider next while building the training example sets. 2002 Elsevier Science Ltd. All rights reserved.
Keywords: Document classication; Document indexing; Vector space model; Data mining; One Clause At a Time
(OCAT) algorithm; Machine learning
*
Corresponding author. Tel.: +1-225-578-5372; fax: +1-225-578-5990.
E-mail addresses: [email protected] (S. Nieto Saanchez), [email protected] http://www.imse.lsu.edu/vangelis/ (E.
Triantaphyllou), [email protected] (D. Kraft).
0306-4573/02/$ - see front matter 2002 Elsevier Science Ltd. All rights reserved.
PII: S0306- 4573( 01) 00049- 8
Information Processing and Management 38 (2002) 583604
www.elsevier.com/locate/infoproman
1. Introduction and background information
This paper investigates the problem of classifying text documents into two disjoint classes. Two
sample sets of training examples (text documents) are assumed to be available. An approach is
developed that uses indexing terms to form logical expressions (patterns) that next are used to
classify unseen text documents. This is a typical case of supervised crisp classication.
A typical application of this type of classication problem occurs in the declassication process
of vast amounts of documents originally produced by the US Federal Government. For reasons of
national security, today there are huge numbers of documents that remain classied as secret.
These documents are being kept in secured places because once they were considered to be im-
portant to national security. However, high maintenance costs and new laws dictate that these
documents should be re-evaluated, and the ones that are not critical any more should become
public. Thus, such documents need to be (roughly speaking) classied into the two disjoint cate-
gories of secret and non-secret. In this context, when a document becomes non-secret after being
secret, it is termed as declassied. In reality, documents have been classied into multiple levels of
criticality to the national security, but in this study we will consider only two classes as described
above. It should also be stated here that once a document becomes public (i.e., it has been declassied),
then there is no way to make it secret again (especially now with the proliferation of the Internet).
In order to highlight the complexity of this kind of classication problem, consider the eval-
uation of the following three illustrative sentences: An atomic test is to be conducted at Site X,
An atomic test is to be conducted at 1:00 p.m., and An atomic test is to be conducted at Site X at
1:00 p.m. which come from three hypothetical documents, A, B, and C, respectively. According
to (DynMeridian, 1996) and (DOE, 1995), only document C is both specic and sensitive and
should not be declassied and instead should continue to be kept secret. The reason for this DOE
(US Department of Energy) classication rule is because document C includes a sentence with
specic reference to the place and time of an atomic test. On the other hand, documents A
and B can be declassied (assuming that the rest of their contents is not critical) and become
available to the general public. In this illustrative example some key text features that can be used
to characterize the two classes are references to place, time and atomic test.
Traditionally, this declassication process is carried out by employing vast numbers of human
experts. However, the sheer amount of documents under consideration can make this process
extremely ineective and inecient. Although there are guidelines of how to declassify secret
documents, directly computerizing the human eort would require developing sophisticated
parsers. Such parsers would have to analyze syntactically a document and then determine which,
if any, guideline is applicable. The poor quality of the documents (many of which are decades old)
and the complexity of the declassication guidelines could make such an approach too risky to
national security. Thus, a reasonable alternative is to seek to employ machine-learning techniques.
More specically, techniques that use learning from examples approaches might be appropri-
ate. Thus, this study is centered on the following three inter-related research problems:
1. Employ data mining techniques for extracting (mining) from two sets of examples text features
that could be used to correctly group documents into two disjoint classes.
2. Use such features to form logical expressions (patterns) that could explain how the training ex-
amples are grouped together, and accurately classify unseen documents.
584 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
3. When considering a guided learning strategy for extracting the logical expressions, determine
the next training document to include in the sets with the training examples so that accurate
logical expressions are extracted as quickly as possible.
Since being able to justify this kind of classication decisions is important (given the severity of
wrongly releasing a sensitive document to the public), methods that do not clearly allow for an
explanation of the decision making process are not appealing here. Therefore, an impetus for this
research is to seek to develop an approach that is based on mathematical logic, versus approaches
that do not provide satisfactory explanation capabilities.
Traditional text classication and information retrieval (IR) techniques may have some limi-
tations in solving this problem because they group documents that share a similar content. The
prime example of such techniques is the vector space model (VSM) (Salton, 1989), which according
to the literature (Buckley & Salton, 1995; Shaw, 1995) is the most eective methodology for this
type of classication. The limitation of this technique is that it is based on similarity measures and
thus it may not be able to distinguish between the previous three illustrative sentences in terms of
the critical classication issues despite them all having similar contents. Other techniques, such as
fuzzy set approaches (FSAs), neural networks (NNs), nearest neighbor, and computational se-
mantic analysis (SA), have limitations in addressing these research problems, either because of
their time complexity or because the resulting sizes of their outputs are still unacceptable and do
not possess satisfactory explanatory capabilities (Chen, 1996; Scholtes, 1993).
An alternative approach to address the present research problems is the One Clause At a Time
(or OCAT) algorithm (Triantaphyllou, 1994; Triantaphyllou, Soyster, & Kumara, 1994). This is a
new data mining approach based on mathematical logic. This approach extracts (mines) key
features from the training examples and next uses them to construct logical expressions (patterns)
that can be used in classifying the training examples into the two original disjoint classes. These
logical expressions can also easily be transformed into the IF-THEN type of decision rules. The
OCAT approach applies to examples that can be represented by binary vectors (although attri-
butes with continuous values can be transformed into ones with binary values (Triantaphyllou,
2001). However, this is not a limitation because it is the mere presence or absence of certain key
words that can cause a document to be grouped in one class or another. On the other hand, the
typical document classication done by traditional IR systems uses term frequencies (which are
continuous numbers usually normalized in the interval 0; 1) of keywords to group together
documents of seemingly similar context.
This paper is organized as follows. Section 2 presents an overview of the document clustering
process. Section 3 introduces the OCAT algorithm. Section 4 presents an overview of the VSM
algorithm. Section 5 presents an overview of the guided learning approach (GLA). Section 6
describes the methodology of this investigation. Section 7 presents and summarizes the results.
Finally the paper ends with some concluding remarks.
2. A brief overview of the document clustering process
The traditional process for automatic clustering of text documents results in a grouping of
documents with similar content into meaningful groups in order to facilitate their storage and
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 585
retrieval (see, for example, Salton, 1989). This is a four-step process as follows. In the rst step a
computerized systemcompiles a list of the unique words that co-occur in a sample of the documents
from various classes (see, for example, Cleveland & Cleveland, 1983; Salton, 1989). In the second
step, the co-occurring frequency of these words is analyzed and the best set of indexing terms is
extracted. Usually, indexing terms (also known as keywords or content descriptors) are selected
among the words with moderate frequency. The most common and the most rare words (i.e., the
most frequent and infrequent words, respectively) are discarded as keywords because they convey
little lexical meaning (see, for example, Cleveland & Cleveland, 1983; Fox, 1990; Luhn, 1957, 1958;
Meadow, 1992; Salton, 1968; Zip, 1949). Some examples of common words are: a, an, and,
and the (Fox, 1990); rare words depend on a documents subject domain (Meadow, 1992).
In the third step, a document is indexed by axing it with the set of keywords that only occur in
its text. According to Cleveland & Cleveland (1983), this assignment is correct because authors
usually repeat words that conform with the documents subject. A list (vector) of keywords
represents the content of a document and usually it is referred to as a document representation or
surrogate. An example of such a surrogate is the list of the seven words or phrases: {Document
classication, document indexing, vector space model, data mining, OCAT algorithm,
and machine learning}. This surrogate could be used to represent the content of this paper,
which is composed of thousands of words, symbols, and numbers. Hence, the goal of the third
step is to construct a surrogate for representing the content of each document.
An advantage of using such surrogates is that they can be further simplied by expressing them
as numerical vectors (Salton, 1989). One way to construct such vectors is by expressing their
elements as binary values to indicate the presence (denoted by 1) or absence (denoted by 0) of
certain keywords in a document. For instance, the vectors element w
ij
1 (or 0) expresses the
presence (or absence) of keyword T
i
i 1; 2; 3; . . . ; t in document D
j
j 1; 2; 3; . . . ; N. Thus,
the surrogate D
j
011110 of six binary elements indicates the presence of keywords (terms) T
2
,
T
3
, T
4
, and T
5
and the absence of keywords (terms) T
1
and T
6
in D
j
.
Another way to construct these numerical vectors is by expressing (i.e., weighting) their ele-
ments using real values from the range 0; 1. In this case, the value of an element w
ij
indicates the
relative occurrence frequency of a keyword within a document. For instance, a hypothetical
surrogate such as D
j
0:00 1:00 0:10 0:75 0:90 1:00 may indicate that term T
3
occurs
little, terms T
4
and T
5
occur moderately, and terms T
2
and T
6
occur with high frequency in D
j
. In
the remainder of this paper, however, only binary surrogates will be considered. As stated above
the reason for considering binary vectors as surrogates is because the mere presence or absence of
a keyword (or some pattern of keywords) may be detrimental in assigning a document to one of
the two disjoint classes considered in this paper.
In the last step of the (traditional) classication process, documents sharing similar keywords
(i.e., content) are grouped together. This classication follows from the pairwise comparison of all
the surrogates (Salton, 1989).
3. An overview of the OCAT algorithm
The OCAT algorithm (Triantaphyllou, 1994; Triantaphyllou et al., 1994) is an inductive
learning (data mining) approach for the classication of examples (documents in this study) into
586 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
two disjoint classes. Each example is represented by a binary vector, although it can be generalized
into vectors dened on continuous variables (Triantaphyllou, 2001). The ith element of such a
vector represents the presence (1) or absence (0) of a key characteristic (also called a feature,
variable, atom, or attribute) pertinent to the phenomenon under study. For this study, such bi-
nary features represent the presence or absence of the index terms (keywords) in the document
surrogates. It is a good idea for the analyst to rst try to dene these examples by using as many
features as possible.
The OCAT algorithm systematically identies (mines) a small set of features while simul-
taneously constructing a logical expression of small size dened on these features that can be used
to group together the training examples into the two original disjoint classes. This logical ex-
pression is a Boolean function that evaluates to true when it is given training examples from one
of the two disjoint classes (the positive class) and false when it is given training examples from
the other class (the negative class). The assignment of the terms positive and negative is
arbitrary. Hopefully, if the training examples are representative enough, then this logical ex-
pression can be used to accurately classify unseen examples. Moreover, this logical expression can
be used to extract new knowledge pertinent to the system under study. The extracted Boolean
function is in conjunctive normal form (CNF) or in disjunctive normal form (DNF). Fig. 1 il-
lustrates this algorithm for the CNF case.
From Fig. 1 it becomes evident that the OCAT algorithm is greedy in nature. This allows it to
return logical expressions of small size. That is, it attempts to return a logical expression that is
comprised of a small number of CNF (or DNF) clauses (Triantaphyllou & Soyster, 1996a). This
algorithm is greedy in the sense that in the rst iteration, it forms a clause that for the CNF case
accepts (i.e., it evaluates to 1) all the examples in one of the classes E

and rejects (i.e., it

evaluates to 0) as many examples in the other class E

as possible. Similarly, in the second it-

eration it forms another clause that again accepts all the examples in E

and rejects as many

Fig. 1. The OCAT algorithm for CNF expressions.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 587
examples as possible in E

that were accepted by the previous clause(s). The algorithm repeats this
process until all the examples in E

are rejected by the generated sequence of clauses.

Triantaphyllou et al. (1994) and Triantaphyllou (1994) have implemented a branch-and-bound
(B&B) approach to construct a set of clauses of minimal cardinality (i.e., a logical expression
comprised by a minimum number of clauses) to solve the problem in Step 2 of the OCAT al-
gorithm in Fig. 1. This approach, however, is limited to problems of small to medium size because
of the extensive CPU times it takes to nd an optimal or next to optimal set of clauses (i.e., a set of
clauses of minimal cardinality). In Triantaphyllou & Soyster (1995) the authors present a simple
transformation approach that can be used to extract DNF functions from algorithms that infer
CNF functions and vice versa.
More recently, Deshpande & Triantaphyllou (1998) have implemented a faster heuristic of
quadratic time complexity as it is highlighted in Fig. 2. The version of the OCAT approach de-
picted in Fig. 2 returns a CNF expression (Boolean function) which accepts all of the positive
examples while rejecting all of the negative examples. This algorithm can also be enhanced with a
randomized approach which essentially solves this problem many times (in a randomized fashion)
and at the end it selects the smallest (in terms of the number of clauses in the Boolean expression)
of the extracted functions. This solution allows for larger classication problems and delivers a set
of (CNF or DNF) clauses of small size (as opposed to the minimal or near to minimal size sets
delivered by the B&B approach).
The notation POSA
j
and NEGA
j
in Fig. 2 denotes the number of positive and negative
examples that will be accepted, respectively, if feature (atom) A
j
is introduced in the current CNF
clause. That is, POSA
j
jE

A
j
j and NEGA
j
jE

A
j
j. Please note that Steps 1 and 2 in
Fig. 2 also consider the negations (denoted as

AA
j
) of the features A
j
. The ratio POSA
j
=NEGA
j

in Steps 1 and 2 is used to quickly identify features that can form a clause that would accept all the
Fig. 2. A fast heuristic for forming CNF clauses for the OCAT approach.
588 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
positive examples while rejecting many of the remaining negative ones (for the CNF case). By
selecting a feature that maximizes this ratio, it is likely to have a feature that has a large POSA
j

value and a low NEGA

j
value. If the value of NEGA
j
is equal to zero, then a large value is
assigned to the ratio in Step 1 (in Fig. 2) in order to make the choice of including the feature
(atom) A
j
very appealing.
In order to help illustrate the previous issues consider the two sets of training examples depicted
in Fig. 3. The set with the positive examples is comprised of four examples, while the set of the
negative examples is comprised of six examples. These examples (document surrogates in our
context) are dened on four binary features (i.e., index terms or atoms) or their negations. A value
of 1 indicates the presence of the corresponding index term, while a value of 0 indicates the ab-
sence of the index term.
In the experiments described in this paper, the OCAT approach (as depicted in Fig. 1) employs
the fast heuristic shown in Fig. 2. However, in other implementations a B&B approach (Tri-
antaphyllou, 1994) may be used in Step 2. When the fast heuristic in Fig. 2 is employed, the rst
CNF clause that is derived is A
2
_ A
4
. To follow this, observe that the ratio with the maximum
value is given by POSA
2
=NEGA
2
2=2 1:00 (this is a tie with the ratio that corresponds to
A
4
). When feature A
2
(which is arbitrarily selected over A
4
) is included in the rst clause (which
initially is nil), then the rst and the second positive examples will be accepted by that clause. The
next iteration of the fast heuristic will consider the same negative examples, but the updated set of
positive examples consists of the remaining positive examples (i.e., the third and fourth). This will
result in the next feature to be selected being A
4
. Now, the CNF clause that is comprised of these
two features (i.e., A
2
and A
4
) will accept all the positive examples in E

while rejecting the rst,

fourth and fth examples from the set of the negative examples in E

.
The next iteration of the OCAT approach (in Fig. 1) will consider the original four positive
examples, but now the set of the negative examples consist of the ones not rejected so far. That is,
the second, third, and sixth negative examples, in terms of the original set E

. Working as above,
it can be seen that the next application of the fast heuristic will return the clause :

AA
2
_

AA
3
. The
loop in Fig. 1 needs to be repeated once more, and a third (and nal) clause is derived. That clause
is A
1
_ A
3
_

AA
4
. In other words, the logical expression (in CNF) which is derived from the
training examples depicted in Fig. 3 is as follows:
A
2
_ A
4
^

AA
2
_

AA
3
^ A
1
_ A
3
_

AA
4
: 1
A fundamental property of expression (1) is that it accepts (i.e., evaluates to 1) all the exam-
ples in E

, while it rejects (i.e., evaluates to 0) all the examples in E

. However, since such an

Fig. 3. A set of four positive and a set of six negative examples.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 589
expression is usually constructed from a relatively small collection of training examples, it is
possible that the expression to be inaccurate when it classies unseen examples. The error may
occur if either the unseen example is positive, and the expression rejects it, or if the example is
negative and the expression accepts it. More details on the OCAT approach can be found at:
http://www.imse.lsu.edu/vangelis
4. Improving the classication accuracy of the OCAT algorithm by using two logical expressions
Let us consider generating a second logical expression for classifying unseen examples. Please
recall that the rst expression is derived by treating the rst set of training examples as positive
and the second as the negative examples. However, the second expression is derived by treating
the second set of training examples as positive and the rst as the negative training examples. For
instance, Fig. 4 depicts the same training examples as the ones in Fig. 3, but now they have reverse
roles.
When the OCAT approach is applied on the new inference problem, the following expression
(2) is derived:
A
3
_

AA
2
^

AA
4
_ A
2
_

AA
1
^ A
1
_

AA
3
: 2
As with expression (1), a property of the corresponding Boolean function f x A
3
_

AA
2
^

AA
4
_ A
2
_

AA
1
^ A
1
_

AA
3
is to accept (i.e., to evaluate to 1) the former negative examples and to
reject (i.e., to evaluate to 0) the former positive examples. For convenience, following the setting
of the examples in Figs. 3 and 4, expression (1) will be called the positive rule (denoted as R

) while
expression (2) the negative rule (denoted as R

).
The disadvantage of using only one rule (logical expression) can be overcome by considering
the combined decisions of R

and R

when classifying an unseen example e. If e is a positive

example it will be denoted as e

, while if it is a negative example it will be denoted as e

. Under
this setting, the classication of e can only be:
1. Correct if and only if:
(a) R

1 and R

0;
(b) R

0 and R

1.
2. Incorrect if and only if:
(c) R

0 and R

1;
(d) R

1 and R

0.
Fig. 4. The training example sets in reverse roles.
590 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
3. Undecided if and only if:
(e) R

1 and R

1;
(f) R

1 and R

1;
(g) R

0 and R

0;
(h) R

0 and R

0.
Cases (a) and (b) are called correct classications because both rules perform according
to the desired properties described above. However, as indicated above it is possible that the rules
could incorrectly classify an example (cases (c) and (d)). Or the rules could simultaneously ac-
cept (cases (e) and (f)) or reject (cases (g) and (h)) the example. Cases (e)(h) are called unde-
cided because one of the rules does not possess enough classication knowledge, and thus such
a rule must be reconstructed. Therefore, undecided situations open the path to improve
the accuracy of a classication system. This paper exploits the presence of undecided situa-
tions in order to guide the reconstruction of the rule that triggered an erroneous classication
decision.
5. An overview of the vector space model
The VSM is a mathematical model of an IR system that can also be used for the classication of
text documents (Salton, 1989; Salton & Wong, 1975). It is often used as a benchmarking method
when dealing with document retrieval and classication related problems. Fig. 5 illustrates a
typical three-step strategy of the VSM approach to clustering.
To address Step 1 Salton (1989) indicates that a suitable measure for pairwise comparing any
two surrogates X and Y is the cosine coecient (CC) as dened in Eq. (3) (other similarity
measures are listed in Salton (1989, Chapter 10):
CC
jX \ Y j
jXj
1=2
jY j
1=2
: 3
In this formula, X x
1
; x
2
; x
3
; . . . ; x
t
and Y y
1
; y
2
; y
3
; . . . ; y
t
, where x
i
indicates the presence
(1) or absence (0) of the ith indexing term in X and similarly for y
i
in respect to Y. Moreover,
jXj jY j t is the number of indexing terms, and jX \ Y j is the number of indexing terms ap-
pearing simultaneously in X and Y. To be consistent with the utilization of binary surrogates,
formula (3) provides the CC expression for the case of Boolean vectors. This coecient measures
the angle between two surrogates (Boolean vectors) in a Cartesian plane. Salton (1989) indicates
Fig. 5. The VSM approach.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 591
that the magnitude of this angle can be used to measure the similarity between any two docu-
ments. This statement is based on the observation that two surrogates are identical, if and only if
the angle between them is equal to 0.
In Step 2 the VSM clusters together documents that share a similar content based on their
surrogates. According to Salton (1989), any clustering technique can be used to group documents
with similar surrogates. A collection of clustering techniques is given in Anderberg (1973), Ald-
enderfer (1984), Spaath (1985), and Van Rijsbergen (1979). However, it is important to mention
here that with any of these techniques, the number of generated classes is always a function of
some predened parameters. This is in contrast with the requirements of our problem here in
which the number of classes is exactly equal to two. When the VSM works under a predened
number of classes, it is said to perform a pseudo-classication. In this study the training examples
are already grouped into two (disjoint) classes. Thus, the VSM is applied on the examples
(documents) in each class and the corresponding centroids are derived. Hence, we will continue to
use this kind of pseudo-classication.
To address Step 3 Salton (1989), Salton & Wong (1975), and Van Rijsbergen (1979) suggest the
computation of a class centroid to be done as follows. Let w
rj
j 1; 2; 3; . . . ; t be the jth element
of the centroid for class C
r
which contains q documents. Also, the surrogate for document D
i
is
dened as fD
ij
g. Then, w
rj
is computed as follows:
w
rj
1=q
X
q
i1
D
ij
for j 1; 2; 3; . . . ; t: 4
That is, the centroid for class C
r
is also a surrogate (also known as the average document)
dened on t keywords.
Finally, the VSM classies a new document by comparing (i.e., computing the CC) its surrogate
against the centroids that were created in Step 3. A new document will be placed in the class for
which the CC value is maximum.
In the tests to be described later in this paper, the VSM is applied on the documents (training
examples) available for each class. In this way, the centroid of each one of the two classes is
derived. For instance, consider the training examples depicted in Figs. 3 and 4. The VSM is now
applied on these data. The centroids in expression (5) have been constructed from the data in Fig.
3 and the centroids in expression (6) from the data in Fig. 4. Obviously, the centroids for the
second set are in reverse order of those for the rst set of data.
C

1=2; 1=2; 1=4; 1=2;

C

2=3; 1=3; 1=2; 1=3;

5
C

2=3; 1=3; 1=2; 1=3;

C

1=2; 1=2; 1=4; 1=2:

6
The notation C

and C

stands for the centroids for the data in Fig. 3, and C

and C

stand for
the centroids for the data in Fig. 4, respectively. In order to match the names of the positive and
negative rules described for the OCAT algorithm, the two centroids for the data in Fig. 3 will be
called the positive centroids while the centroids for the data in Fig. 4 will be called the negative
centroids. As with the OCAT algorithm, the utilization of two sets of centroids has been inves-
592 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
tigated in order to tackle the new classication problem by using the VSM as new examples
become available.
6. Guided learning for the classication of text documents
The central idea of the GLA can be illustrated as follows. Suppose that the collection to be
classied contains millions of documents. Also, suppose that an oracle (i.e., an expert classier) is
queried in order to classify a small sample of examples (documents) into classes E

and E

. Next,
suppose that the OCAT algorithm is used to construct the positive and negative rules, such as was
the case with expressions (1) and (2). As indicated earlier, these rules may be inaccurate when
classifying examples not included in the training set, and therefore they will result in one of the
classication outputs provided in cases (a)(g), as described earlier. One way to improve the
classication accuracy of these rules is to add one more document to the training set (either in E

or E

) and have them reconstructed. Therefore, the question GLA attempts to answer is: What is
the next document to be inspected by the expert classier so that the classication performance of the
derived rules can be improved as fast as possible?
One way to provide the expert with this document is to randomly select one from the remaining
unclassied documents. We call this the RANDOM input learning strategy. A drawback of this
strategy may occur if the oracle and incumbent rules frequently classify a document in the same
class. If this occurs frequently, then the utilization of the oracle and the addition of the new ex-
amples to the training set is of no benet. An alternative and more ecient way to provide the
expert with a document is to select one in an undecided situation. This strategy (in a general
form) was rst introduced in Triantaphyllou & Soyster (1996b). This approach appears to be a
more ecient way of selecting the document because an undecided situation implies that one of
the rules misclassied the document. Therefore, the experts verdict will not only guide the re-
construction of the rule that triggered the misclassication, but it may also improve the learning
rate of the two rules. We call this the GUIDED input learning strategy. An incremental learning
version of the OCATapproach is described in Nieto Sanchez, Triantaphyllou, Chen, &Liao (2002).
7. Experimental data
In order to determine the classication performance of the OCAT approach in addressing this
new problem, the OCATs classication accuracy was compared with that of the VSM. Both
approaches were tested under three experimental settings:
1. a Leave-One-Out Cross-Validation (or CV) (also known as the Round-Robin test);
2. a 30/30 Cross-Validation (or 30CV), where 30 stands for the number of training documents in
each class; and
3. in an experimental setting in which the OCAT algorithm was studied under a random and a
guided learning strategy.
These will be dened below. This multiple testing strategy was selected in order to gain a more
comprehensive assessment of the eectiveness of the various methods.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 593
For these tests, a sample of 2897 documents was randomly selected from four document classes
of the TIPSTER collection (Harman, 1995; Voorhees, 1998). The previous numbers of documents
in each class were determined based on memory limitations on the computing platform used (an
IBM Pentium II PC running Windows 95). The TIPSTER collection is a standard data set for
experimentation with IR systems. The four document classes were as follows:
1. Department of Energy (DOE) documents,
2. Wall Street Journal (WSJ) documents,
3. Associated Press (AP) documents, and
4. ZIPFF class documents.
We chose documents from this collection because for security reasons we did not have access to
actual secret DOE documents.
Table 1 shows the number of documents that were used in the experimentation. These docu-
ments were randomly extracted from the four classes of the TIPSTER collection.
We simulated two mutually exclusive classes by forming the following ve class-pairs: (DOE vs.
AP), (DOE vs. WSJ), (DOE vs. ZIPFF), (AP vs. WSJ), and (WSJ vs. ZIPFF). These ve class-
pairs were randomly selected from all possible class-pairs combinations. To comply with the
notation presented in the previous sections, the rst class of each class-pair was denoted as E

,
while the second class was denoted as E

. Thus, we try to nd a Boolean expression to classify a

document surrogate into the proper TIPSTER class.
Table 2 shows the average number of keywords that were extracted from the ve class-pairs
mentioned above. The data in this table can be interpreted as follows. For the class-pair (DOE vs.
AP), the average number of keywords used in all the experiments was 511 under the CV validation
and 803 under the 30CV validation. A similar interpretation applies to the data in the other col-
umns.
It should be stated at this point that we used a number of alternative indexing terms. Besides
single words, we also used sequences of two words at a time, sequences of three words at a time,
Table 1
Number of documents randomly extracted from each class
Class: DOE AP WSJ ZIPFF Total
Number of documents: 1407 336 624 530 2897
DOE, AP, WSJ, and ZIPFF stand for Department of Energy, Associated Press, and the Wall Street Journal, re-
spectively; ZIPFF is a collection of technical documents of various topics.
Table 2
Average number of indexing words used in each experiment
Type of experiment DOE vs. AP DOE vs. WSJ DOE vs. ZIPFF AP vs. WSJ WSJ vs. ZIPFF
CV 511 605 479 448 501
30CV 803 911 890 814 811
In order to keep the size reasonable for our computing environment, only the rst hundred words from each document
were considered. Stop words were always removed.
594 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
and sequences of four words at a time. However, some pilot studies indicated that the best results
would be derivable by using as indexing terms single words only. Thus, the atoms (binary vari-
ables) in the derived logical expressions are single keywords and not sequences of them.
8. Testing methodology
This section rst summarizes the methodology for the Leave-One-Out Cross-Validation and the
30/30 Cross-Validation. These two alternative testing methods have been employed in order to
gain a better understanding of the various procedures used to classify text documents. The same
section also presents the statistical tests employed to determine the relative performance of the
VSM and the OCAT algorithm. This section ends with the methodology for the GLA.
8.1. The Leave-One-Out Cross-Validation
The cross-validation (CV) testing was implemented on samples of 60 documents as follows.
First, 30 documents from each class were randomly selected. Please note that the size 60 was used
due to storage limitations in our computing environment. Then, one document was removed from
these sets of documents with its class noted. After that, the positive and negative rules under the
OCAT approach and the positive and negative centroids under the VSM were constructed using
the remaining 59 documents. In the third step the class of the document left out was inferred by
both algorithms. Then, the correctness of the classication was determined according to the cases
(a)(h), as dened in Section 4. The previous second and third steps were repeated with dierent
sets of training examples until all 60 documents had their class inferred one at a time. This ex-
perimental setting was replicated 10 times with dierent subsets of the training data, at which
point the results of the two algorithms were tested for statistical dierences.
8.2. The 30/30 Cross-Validation
The 30/30 Cross-Validation (30CV) was implemented on samples of 254 documents as follows.
The number of 254 documents was used to avoid excessive computational time. Initially, the
positive and negative rules under the OCAT approach and positive and negative centroids under the
VSM were constructed by using only 30 documents (randomly selected) from each class. Then,
the classication of the remaining 194 documents was inferred. As before, the correctness of
this classication was determined according to the cases (a)(g), as dened earlier. As with the
rst experimental setting, the 30CV validation was replicated 10 times, at which point the results
of the two algorithms were tested for statistical dierence.
8.3. Statistical performance of both algorithms
To determine the statistical performance of both algorithms, the following hypotheses were
tested. The rst test was needed to determine the relative dominance of the algorithms. In the
second test we implemented a sign test in order to determine the consistency of the dominance of
the algorithms.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 595
1: H
0
:

PP
OCAT
6

PP
VSM
H
1
:

PP
OCAT
>

PP
VSM
2: H
0
: p 0:50
H
1
: p 6 0:50
where P
OCAT
and P
VSM
are the numbers of documents with correct classication under the two
algorithms divided by the total number of documents in the experiment. In addition, p is the
probability of nding an equal number of positive and negative dierences in a set of outcomes.
More on how these tests were performed is provided in the following sub-sections which present
the computational results.
8.4. Experimental setting for the Guided Learning Approach
Consider the question: What is the best next document to be given to the oracle in order to
improve the performance of the classication rule? Three samples of 510 documents (255 from each
class) from the three class-pairs: (DOE vs. ZIPFF), (AP. vs. DOE), and (WSJ vs. ZIPFF) were
used. The number of 510 documents was determined by the available RAM memory on the
Windows PC we used. The previous three class-pairs were processed by the OCAT algorithm
(only) under the RANDOM and the GUIDED learning approaches.
These two learning approaches were implemented as follows. At rst, 30 documents from each
class in the experiment were randomly selected, and the positive and negative rules (logical ex-
pressions) were constructed. Next, the class membership of all 510 documents in the experiment
was inferred based on the two classication rules. The criteria expressed as cases (a)(g) in Section
4 were used to determine the number of correct, incorrect, and undecided classications.
Next, a document was added to the initial training sample as follows. For the case of the
RANDOM approach, this document was selected at random from among the documents not
included in the training sets yet (i.e., neither in E

nor in E

).
In contrast, under the GUIDED approach this document was selected from the set of docu-
ments which the positive and negative rules had already termed as undecided. However, if the
two rules did not detect an undecided case, then the GUIDED approach was replaced by the
RANDOM approach until a new undecided case was identied. This process for the RAN-
DOM and GUIDED approaches was repeated until all 510 documents were included in the two
training sets E

and E

. The results of this experimentation are presented next.

9. Results for the Leave-One-Out and the 30/30 Cross-Validations
Table 3 summarizes the experimental results for the CV validation, while Table 4 summarizes
the results for the 30CV validation. The abbreviations C:, I:, and U: in the rst column of
both tables correspond to the correct, incorrect, and undecided classication outcomes
which can be obtained by using the positive and the negative rules (for the OCAT case) or the
positive and the negative centroids (for the VSM case). For instance, the data in Table 3, column 2
596 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
(i.e., class-pair (DOE vs. AP)) indicate that the VSM identied 334 correct, 261 incorrect,
and 5 undecided cases.
Similarly, the data in Table 3, column 3 (i.e., class-pair (DOE vs. AP)) indicate that the OCAT
algorithm identied 410 correct, 5 incorrect, and 185 undecided classications. The data in
the other columns can be interpreted in a similar manner. The last two columns of these two tables
summarize the results across all ve class-pairs. Fig. 6 compares the proportions of these results
for both algorithms.
Two key observations can be derived from the size of the dark areas (or areas of undecided
classications) in Fig. 6 which was derived from Tables 3 and 4. First, it can be observed that the
proportion of undecided cases detected by the VSM algorithm is almost 0%. These have oc-
curred when the two positive and the two negative centroids accepted the same document and,
therefore, the classes predicted by both sets of centroids have to be selected randomly. More
specically, these undecided instances occurred even when these randomly predicted classes
were identical. The VSM was implemented using the CC coecient, following the suggestions in
Salton (1989).
In contrast, as the second observation, we have large proportions of undecided classications
with the OCAT algorithm. Please recall that this type of classication decision is desired because
such cases demonstrate that either the positive or the negative rules are unable to classify correctly
new documents. Therefore, in these results the large dark areas in the above gure show that both
rules were unable to classify correctly a large proportion of the documents in the experiments.
More importantly, the size of these areas indicates that positive or negative rules may be improved
if they are modied when an undecided situation is detected.
Consider the proportion of the incorrect classications (i.e., the white areas in Fig. 6). One
can derive two conclusions. First, the number of incorrect classications the VSM made
amounts to 49.77% (or 1493/3000) with the CV validation and to 46.65% (or 4548/9750) with the
30CV validation. These large proportions of incorrect classications can be attributed to the
Table 4
Summary of the second experimental setting: 30/30 Cross-Validation
DOE vs. AP DOE vs. WSJ DOE vs. ZIPFF AP vs. WSJ WSJ vs. ZIPPF TOTAL
VSM OCAT VSM OCAT VSM OCAT VSM OCAT VSM OCAT VSM OCAT
C: 975 1406 975 1266 1035 1320 1088 1283 1088 1145 5161 6420
I: 975 70 975 134 915 124 846 140 837 176 4548 644
U: 0 474 0 550 0 506 16 527 25 41 41 2686
Table 3
Summary of the rst experimental setting: Leave-One-Out Cross-Validation
DOE vs. AP DOE vs. WSJ DOE vs. ZIPFF AP vs. WSJ WSJ vs. ZIPPF TOTAL
VSM OCAT VSM OCAT VSM OCAT VSM OCAT VSM OCAT VSM OCAT
C: 334 410 286 296 280 358 316 365 286 303 1502 1732
I: 261 5 314 66 320 25 284 47 314 76 1493 219
U: 5 185 0 238 0 217 0 188 0 221 5 1049
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 597
inability of the positive and negative centroids to distinguish between incorrect and unde-
cided classications. Second we have 7.30% (or 219/3000) and 6.61% (or 644/9750) of incorrect
classications the OCAT algorithm made with the two test settings. In this case, these small error
rates can be attributed to the utilization of positive and negative rules of small size that enabled
the OCAT algorithm to distinguish between the incorrect and undecided classications.
Despite the disparate proportions of the inaccurate and undecided classications for both
of these algorithms, their performances were statistically compared using only the proportions
with the correct classications. That is, the undecided cases were not considered here. In this
way the VSM approach was not placed in an unfair setting when it was compared with the OCAT
approach. This comparison was needed in order to determine which algorithm better addressed
the classication problem studied in this paper. For this comparison, it was assumed that no
Fig. 6. Comparison of the classication decisions under the OCAT and VSM approaches.
Table 5
Statistical dierence in the classication accuracy of the VSM and OCAT approaches
Type of experiment P
OCAT
a
P
VSM
b
P
VSM
P
OCAT
Binomial test
Half-length
c
Interval
CV 0.577 0.501 )0.0760 0.025 0:035; 0:085
30CV 0.658 0.529 )0.1287 0.014 11:47; 14:27
a
1732/n and 6420/n; where n is 3000 for CV and 9750 for 30CV.
b
1502/n and 5161/n; where n is 3000 for CV and 9750 for 30CV.
c
Denotes that both approaches performed statistically dierently.
598 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
additional improvement of the two algorithms was possible under the CV and 30CV cross-vali-
dations. Furthermore, the incorrect and undecided outcomes were considered as incorrect
classications.
The results of these tests (as shown in Table 5) indicate that the OCAT approach is more
accurate in both types of computational experiments than the VSM. Furthermore, the very low p-
values in Table 6 indicate that it is extremely unlikely to nd a similar number of positive and
negative dierences in the proportions of the correct classications under the two approaches
Fig. 7. Results when the GUIDED and RANDOM approaches were used on the (DOE vs. ZIPFF) class-pair.
Table 6
Data for sign test to determine the consistency in the ranking of the OCAT and VSM approaches
Type of experiment
CV 30CV
Number of + signs 4 7
Number of ) signs 46 43
p-value
X
m
i0
50
i

p
i
1 p
50i
;
p-value 2:23 10
10
p-value 1:04 10
7
where m 4 for CV and 7 for 30CV and p 0:50
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 599
(Barnes, 1994). Therefore, the results of these two statistical tests indicate that the OCAT ap-
proach is better suited to address the document classication problem studied in this paper.
10. Results for the Guided Learning Approach
Figs. 79 show the results of the OCAT algorithm under the RANDOM and GUIDED input
learning approaches. The horizontal axis indicates the percentage of training documents used
during the experiment. For example, at the beginning of the experiment there were 60 training
documents or 11.76% of the 510 documents in the experiment. Next, when one more document
was added to the training set, following the recommendation of the GUIDED and RANDOM
approaches, there were 12.16% of the documents in the experiment.
The vertical axis shows the proportions of correct, incorrect, and undecided classi-
cation for the various percentages of training documents used in the experiment. The abbrevia-
tions Rc, Ri, Ru and Gc, Gi, Gu stand for the proportions of correct, incorrect, and
undecided outcomes for the RANDOM and GUIDED approaches, respectively. For instance,
Rc is the proportion of Correct classications under the RANDOM approach, and Gu is the
proportion of Undecided classication under the GUIDED approach.
Table 7 shows the percentage of training documents the OCAT algorithm needed before it
classied all 510 documents in each class-pair correctly (i.e., it became 100% accurate). The po-
Fig. 8. Results when the GUIDED and RANDOM approaches were used on the (AP vs. DOE) class-pair.
600 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
sition of the dotted line in the above three gures corresponds to the percentages shown in this
table. For instance, in Fig. 7 (class-pair (DOE vs. ZIPFF)) this line is at 65.69% on the horizontal
axis.
Some important observations can be made regarding the proportions of correct, incorrect,
and undecided classications in Figs. 79. First, the rate of correct classications under the
GUIDED approach, Gc, was higher than the rate Rc under the RANDOM approach. Actually,
the last row in Table 7 indicates that the OCAT algorithm needed on the average about 34% less
training documents to classify correctly all 510 document under the GUIDED approach than
under the RANDOM approach.
Fig. 9. Results when the GUIDED and RANDOM approaches were used on the (WSJ vs. ZIPFF) class-pair.
Table 7
Percentage of documents from the population that were inspected by the oracle before an accuracy of 100% was reached
Class-pairs % Under GUIDED % Under RANDOM
(DOE vs. ZIPFF) 65.69 100.00
(AP vs. DOE) 60.98 99.80
(WSJ vs. ZIPFF) 71.18 99.80
Average 65.95 99.87
100% accuracy was achieved when the number of Incorrect and Undecided classications were 0%.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 601
These results are very interesting for a number of reasons. They conrm the assumption stated
in Section 5 which indicated that the utilization of documents with the undecided classication
could increase the accuracy of the OCAT algorithm. These results are also encouraging because
they help to answer the second question stated in the introduction of this section. That is, queries
to the oracle should stop when about 66% of the 510 documents from the three class-pairs of the
TIPSTER collection had been inspected and were included in the training sets. More importantly,
these results are important because they suggest that the OCAT algorithm can be employed for
the classication of large collections of text documents.
The other two observations are related to the rates at which the incorrect and undecided
classications were eliminated. It can be observed from the previous three gures that these rates
were a direct consequence of improving the classication rules. The gures show that the rates Gi
and Gu reach 0% when about 66% (336 documents) of the 510 documents in the experiment have
been included in E

and E

. On the other hand, it can be seen that under the RANDOM learning
approach, the rates Ri and Ru reached 0% when 99.8% (509 documents) of the documents are
processed.
11. Concluding remarks
This paper has examined a classication problem in which a document must be classied into
one of two disjoint classes. As an example of the importance of this type of classication, one can
consider the possible release to the public of documents that may aect national security. The
method proposed in this paper (being an automatic method) is not infallible. This is also true
because its performance depends on how representative the training examples (documents) are.
The application of such an approach to a problem of critical importance (such as the one high-
lighted in the introduction) can be seen as an important and useful automatic tool for a pre-
liminary selection of the documents to be classied.
We considered an approach to this problem based on the VSM algorithm and compared it with
an algorithm which is based on mathematical logic, called the OCAT algorithm. We tested these
two approaches on almost 3000 documents from the four document classes of the TIPSTER
collection: Department of Energy (DOE), Wall Street Journal (WSJ), Associated Press (AP), and
the ZIPFF class. Furthermore, these documents were analyzed under two types of experimental
settings: (i) Leave-One-Out Cross-Validation and (ii) a 30/30 Cross-Validation (where 30 indicates
the initial number of training documents from each document class). The experimental results
suggest that the OCAT algorithm performed signicantly better in classifying documents into two
disjoint classes than the VSM.
Moreover, the results of a third experiment suggested that the classication eciency of the
OCAT algorithm can be improved substantially if a GLA is implemented. Actually, experiments
on samples of 510 documents from the previous four classes of the TIPSTER collection indicated
that the OCAT algorithm needed only about 336 (i.e., 66% of the) training documents before it
correctly classied all of the documents.
The results presented here, although limited to a relatively small collection of almost 3000
documents, are encouraging because they suggest that the OCAT algorithm can be used in the
classication of larger collections of documents.
602 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604
Acknowledgements
The authors are very appreciative for the very thoughtful and constructive comments made by
two anonymous referees of an earlier version of the paper. Those comments played an important
role in improving the context and presentation of this paper. The rst and second authors wish to
thank the US Department of Energy for providing the nancial funds that supported this re-
search. The rst author is pleased to recognize the partial nancial support from Consejo Nac-
ional de Ciencia y Technologia-Mexico. The second author is also very appreciative to the US
Navy, Oce of Naval Research, for the support from grants N00014-95-1-0639 and N00014-97-1-
0632.
References
Aldenderfer, M. S. (1984). Cluster analysis. Beverly-Hill, CA: Sage.
Anderberg, M. R. (1973). Cluster analysis for applications. New York: Academic Publishers.
Barnes, J. W. (1994). Statistical analysis for engineers and scientists, a computer-based approach. New York: McGraw-
Hill.
Buckley, C., & Salton, G. (1995). Optimization of relevance feedback weights. In Proceedings of SIGIR 1995 (pp. 351
357).
Chen, H. (1996). Machine learning approach to document retrieval: An overview and an experiment. Technical Report,
University of Arizona, MIS Department, Tucson, AZ, USA.
Cleveland, D., & Cleveland, A. D. (1983). Introduction to indexing and abstracting. Littleton, CO: Libraries Unlimited.
Deshpande, A. S., & Triantaphyllou, E. (1998). A greedy randomized adaptive search procedure (GRASP) for inferring
logical clauses from examples in polynomial time and some extensions. Mathematical and Computer Modelling,
27(1), 7599.
DOE (1995). General Course on Classication/Declassication, Student Syllabus, Handouts, and Practical Exercises. US
Department of Energy, Germantown, MD, USA.
DynMeridian (1996). Declassication Productivity Initiative Study Report. DynCorp Company, Report Prepared for the
US Department of Energy, Germantown, MD, USA.
Fox, C. (1990). A stop list for general text. ACM Special Interest Group on Information Retrieval, 24(12), 1935.
Harman, D. (1995). Overview of the second text retrieval conference (TREC-2). Information Processing and
Management, 31(3), 271289.
Luhn, H. P. (1958). The automatic creation of literature abstracts. IBM Journal of Research and Development, 5(3),
155165.
Luhn, H. P. (1957). A statistical approach to mechanized encoding and searching of literary information. IBM Journal
of Research and Development, 4(4), 600605.
Meadow, C. T. (1992). Text information retrieval systems. San Diego, CA: Academic Press.
Nieto Sanchez, S., Triantaphyllou, E., Chen, J., & Liao, T. W. (2002). An incremental learning algorithm for
constructing Boolean functions from positive and negative examples. Computers and Operations Research (in press).
Salton, G. (1968). Automatic information organization and retrieval. New York: McGraw-Hill.
Salton, G., & Wong, A. (1975). A vector space model for automatic indexing. Information retrieval and language
processing, 18(11), 613620.
Salton, G. (1989). Automatic text processing. The transformation analysis, and retrieval of information by computer.
Reading, MA: Addison-Wesley.
Scholtes, J. C. (1993). Neural networks in natural language processing and information retrieval. The Netherlands: North-
Holland.
Spaath, H. (1985). Cluster dissection and analysis: theory, Fortran programs, and examples. Chichester, UK: Ellis
Harwood.
S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604 603
Shaw, W. M. (1995). Term-relevance computations and perfect retrieval performance. Information Processing and
Management, 31(4), 312321.
Triantaphyllou, E. (2001). The OCAT approach for data mining and knowledge discovery, Working Paper, IMSE
Department, Louisiana State University, Baton Rouge, LA 70803-6409, USA.
Triantaphyllou, E., & Soyster, A. L. (1996a). On the minimum number of logical clauses inferred from examples.
Computers and Operations Research, 23(8), 783799.
Triantaphyllou, E., & Soyster, A. L. (1996b). An approach to guided learning of Boolean functions. Mathematical and
Computing Modelling, 23(3), 6986.
Triantaphyllou, E., & Soyster, A. L. (1995). A relationship between CNF and DNF systems which are derived from the
same positive and negative examples. ORSA Journal on Computing, 7(3), 283285.
Triantaphyllou, E., Soyster, A. L., & Kumara, S. R. T. (1994). Generating logical expressions from positive and
negative examples via a branch-and-bound approach. Computers and Operations Research, 21(2), 185197.
Triantaphyllou, E. (1994). Inference of a minimum size Boolean function from examples by using a new ecient
branch-and-bound approach. Journal of Global Optimization, 5, 6494.
Van Rijsbergen, C. J. (1979). Information retrieval (2nd ed.). London: Butterworths.
Voorhees, E. (1998). Overview of the sixth text retrieval conference (TREC-6). In Proceedings of the sixth text retrieval
conference (TREC-6), Gaithersburg, MD, USA (pp. 127).
Zip, H. P. (1949). Human behavior and the principle of least eort. Menlo Park, CA: Addison-Wesley.
604 S. Nieto Saanchez et al. / Information Processing and Management 38 (2002) 583604