Modern Information Retrieval

Chapter 3
Modeling

Part I: Classic Models

Introduction to IR Models
Basic Concepts
The Boolean Model
Term Weighting
The Vector Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 1
IR Models
Modeling in IR is a complex process aimed at
producing a ranking function
Ranking function: a function that assigns scores to documents
with regard to a given query

This process consists of two main tasks:

The conception of a logical framework for representing
documents and queries
The definition of a ranking function that allows quantifying the
similarities among documents and queries

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 2
Modeling and Ranking
IR systems usually adopt index terms to index and
retrieve documents
Index term:
In a restricted sense: it is a keyword that has some meaning on
its own; usually plays the role of a noun
In a more general form: it is any word that appears in a document
Retrieval based on index terms can be implemented
efficiently
Also, index terms are simple to refer to in a query
Simplicity is important because it reduces the effort of
query formulation

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 3
Introduction
Information retrieval process

index terms

1
docs terms
documents match
2

query terms 3
information ...
need ranking

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 4
Introduction
A ranking is an ordering of the documents that
(hopefully) reflects their relevance to a user query
Thus, any IR system has to deal with the problem of
predicting which documents the users will find relevant
This problem naturally embodies a degree of
uncertainty, or vagueness

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 5
IR Models
An IR model is a quadruple [D, Q, F , R(qi , dj )] where
1. D is a set of logical views for the documents in the collection
2. Q is a set of logical views for the user queries
3. F is a framework for modeling documents and queries
4. R(qi , dj ) is a ranking function

D
d












R(d ,q )
Q

q







Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 6
A Taxonomy of IR Models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 7
Retrieval: Ad Hoc x Filtering
Ad Hoc Retrieval:

Q1 Q3

Q2 Collection










Q4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 8
Retrieval: Ad Hoc x Filtering
Filtering

user 1


!



&
$
"




%




user 2


!



'
$
"




%




documents stream

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 9
Basic Concepts
Each document is represented by a set of
representative keywords or index terms
An index term is a word or group of consecutive words
in a document
A pre-selected set of index terms can be used to
summarize the document contents
However, it might be interesting to assume that all
words are index terms (full text representation)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 10
Basic Concepts
Let,
t be the number of index terms in the document collection
ki be a generic index term

Then,
The vocabulary V = {k1 , . . . , kt } is the set of all distinct index
terms in the collection

V= k k k k

5
0

6
-.

/3

-
,

4
(

+
?

<
8

=>
;

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 11
Basic Concepts
Documents and queries can be represented by
patterns of term co-occurrences

V=
C

C
D

G
@

@
@

JN

P
HI

KL

OK

M
S
A

JN

JN

JN
Z
@

KL

KL

MQ

KL
S

[
^
^
^

JN

J
HI

KL

OK

TK

O
S
A

JN

J
I

KL

KL

O
S

]
@

Each of these patterns of term co-occurence is called a

term conjunctive component
For each document dj (or query q ) we associate a
unique term conjunctive component c(dj ) (or c(q))

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 12
The Term-Document Matrix
The occurrence of a term ki in a document dj
establishes a relation between ki and dj
A term-document relation between ki and dj can be
quantified by the frequency of the term in the document
In matrix form, this can written as
d1 d2
 
k1 f1,1 f1,2
 
k2 f f
 2,1 2,2 
k3 f3,1 f3,2

where each fi,j element stands for the frequency of

term ki in document dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 13
Basic Concepts
Logical view of a document: from full text to a set of
index terms
e

`
a

e
`

`
fgh

h
e

k
`

`
j

ab

cad

ia

ld

pb

il
`

`
v
h

m
`
b

fg

il

na

q
j
`

`
d

pd

`
d

pd

`
h

t
k

k
`
d

i
`

`
`

`
d

pd

ta

q
p

e
h

cad

_
Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 14
The Boolean Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 15
The Boolean Model
Simple model based on set theory and boolean
algebra
Queries specified as boolean expressions
quite intuitive and precise semantics
neat formalism
example of query
q = ka ∧ (kb ∨ ¬kc )

Term-document frequencies in the term-document

matrix are all binary
wij ∈ {0, 1}: weight associated with pair (ki , dj )
wiq ∈ {0, 1}: weight associated with pair (ki , q)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 16
The Boolean Model
A term conjunctive component that satisfies a query q is
called a query conjunctive component c(q)
A query q rewritten as a disjunction of those
components is called the disjunct normal form qDN F
To illustrate, consider
query q = ka ∧ (kb ∨ ¬kc )
vocabulary V = {ka , kb , kc }
Then
qDN F = (1, 1, 1) ∨ (1, 1, 0) ∨ (1, 0, 0)
c(q): a conjunctive component for q

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 17
The Boolean Model
The three conjunctive components for the query
q = ka ∧ (kb ∨ ¬kc )

Ka
Kb
(1,1,1)

Kc

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 18
The Boolean Model
This approach works even if the vocabulary of the
collection includes terms not in the query
Consider that the vocabulary is given by
V = {ka , kb , kc , kd }
Then, a document dj that contains only terms ka , kb ,
and kc is represented by c(dj ) = (1, 1, 1, 0)
The query [q = ka ∧ (kb ∨ ¬kc )] is represented in
disjunctive normal form as

qDN F = (1, 1, 1, 0) ∨ (1, 1, 1, 1) ∨

(1, 1, 0, 0) ∨ (1, 1, 0, 1) ∨
(1, 0, 0, 0) ∨ (1, 0, 0, 1)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 19
The Boolean Model
The similarity of the document dj to the query q is
defined as
(
1 if ∃c(q) | c(q) = c(dj )
sim(dj , q) =
0 otherwise

The Boolean model predicts that each document is

either relevant or non-relevant

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 20
Drawbacks of the Boolean Model
Retrieval based on binary decision criteria with no
notion of partial matching
No ranking of the documents is provided (absence of a
grading scale)
Information need has to be translated into a Boolean
expression, which most users find awkward
The Boolean queries formulated by the users are most
often too simplistic
The model frequently returns either too few or too many
documents in response to a user query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 21
Term Weighting

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 22
Term Weighting
The terms of a document are not equally useful for
describing the document contents
In fact, there are index terms which are simply vaguer
than others
There are properties of an index term which are useful
for evaluating the importance of the term in a document
For instance, a word which appears in all documents of a
collection is completely useless for retrieval tasks

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 23
Term Weighting
To characterize term importance, we associate a weight
wi,j > 0 with each term ki that occurs in the document dj
If ki that does not appear in the document dj , then wi,j = 0.

The weight wi,j quantifies the importance of the index

term ki for describing the contents of document dj
These weights are useful to compute a rank for each
document in the collection with regard to a given query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 24
Term Weighting
Let,
ki be an index term and dj be a document
V = {k1 , k2 , ..., kt } be the set of all index terms
wi,j > 0 be the weight associated with (ki , dj )

Then we define d~j = (w1,j , w2,j , ..., wt,j ) as a weighted

vector that contains the weight wi,j of each term ki ∈ V in
the document dj
k w
V d

|
w
k w

•
x

|
x
k w

{


’



~

€„
}



y

|
y
†
‡

Š

~

€

~

‹
{
... ...


“

‹

Ž


k w

”
|
z
z

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 25
Term Weighting
The weights wi,j can be computed using the frequencies
of occurrence of the terms within documents
Let fi,j be the frequency of occurrence of index term ki in
the document dj
The total frequency of occurrence Fi of term ki in the
collection is defined as
N
X
Fi = fi,j
j=1

where N is the number of documents in the collection

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 26
Term Weighting
The document frequency ni of a term ki is the number
of documents in which it occurs
Notice that ni ≤ Fi .
For instance, in the document collection below, the
values fi,j , Fi and ni associated with the term do are

f (do, d1 ) = 2 To do is to be.
f (do, d2 ) = 0 To be is to do. To be or not to be.
I am what I am.

d1
f (do, d3 ) = 3
f (do, d4 ) = 3 d2
I think therefore I am.
F (do) = 8 Do be do be do. Do do do, da da da.
Let it be, let it be.

d3
n(do) = 3
d4
Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 27
Term-term correlation matrix
For classic information retrieval models, the index term
weights are assumed to be mutually independent
This means that wi,j tells us nothing about wi+1,j

This is clearly a simplification because occurrences of

index terms in a document are not uncorrelated
For instance, the terms computer and network tend to
appear together in a document about computer
networks
In this document, the appearance of one of these terms attracts
the appearance of the other
Thus, they are correlated and their weights should reflect this
correlation.

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 28
Term-term correlation matrix
To take into account term-term correlations, we can
compute a correlation matrix
~ = (mij ) be a term-document matrix t × N where
Let M
mij = wi,j
~ =M
The matrix C ~M~ t is a term-term correlation matrix
Each element cu,v ∈ C expresses a correlation between
terms ku and kv , given by
X
cu,v = wu,j × wv,j
dj

Higher the number of documents in which the terms ku

and kv co-occur, stronger is this correlation

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 29
Term-term correlation matrix
Term-term correlation matrix for a sample collection

d1 d2 k1 k2 k3
 
k1 w w1,2 " #
 1,1  d1 w1,1 w2,1 w3,1
k2 
 w2,1 w2,2 
 d2 w1,2 w2,2 w3,2
k3 w3,1 w3,2
M × MT
| {z }
⇓
 k1 k2 k3 
k1 w w + w1,2 w1,2 w1,1 w2,1 + w1,2 w2,2 w1,1 w3,1 + w1,2 w3,2
 1,1 1,1 
k2 
 w2,1 w1,1 + w2,2 w1,2 w2,1 w2,1 + w2,2 w2,2 w2,1 w3,1 + w2,2 w3,2 

k3 w3,1 w1,1 + w3,2 w1,2 w3,1 w2,1 + w3,2 w2,2 w3,1 w3,1 + w3,2 w3,2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 30
TF-IDF Weights

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 31
TF-IDF Weights
TF-IDF term weighting scheme:
Term frequency (TF)
Inverse document frequency (IDF)
Foundations of the most popular term weighting scheme in IR

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 32
Term Frequency (TF) Weights
Luhn Assumption. The value of wi,j is proportional to
the term frequency fi,j
That is, the more often a term occurs in the text of the document,
the higher its weight

This is based on the observation that high frequency

terms are important for describing documents
Which leads directly to the following tf weight
formulation:

tfi,j = fi,j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 33
Term Frequency (TF) Weights
A variant of tf weight used in the literature is
(
1 + log fi,j if fi,j > 0
tfi,j =
0 otherwise

where the log is taken in base 2

The log expression is a the preferred form because it
makes them directly comparable to idf weights, as we
later discuss

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 34
Term Frequency (TF) Weights
Log tf weights tfi,j for the example collection

Vocabulary tfi,1 tfi,2 tfi,3 tfi,4

1 to 3 2 - -
2 do 2 - 2.585 2.585
3 is 2 - - -
4 be 2 2 2 2
5 or - 1 - -
6 not - 1 - -
7 I - 2 2 -
8 am - 2 1 -
9 what - 1 - -
10 think - - 1 -
11 therefore - - 1 -
12 da - - - 2.585
13 let - - - 2
14 it - - - 2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 35
Inverse Document Frequency
We call document exhaustivity the number of index
terms assigned to a document
The more index terms are assigned to a document, the
higher is the probability of retrieval for that document
If too many terms are assigned to a document, it will be retrieved
by queries for which it is not relevant

Optimal exhaustivity. We can circumvent this problem

by optimizing the number of terms per document
Another approach is by weighting the terms differently,
by exploring the notion of term specificity

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 36
Inverse Document Frequency
Specificity is a property of the term semantics
A term is more or less specific depending on its meaning
To exemplify, the term beverage is less specific than the
terms tea and beer
We could expect that the term beverage occurs in more
documents than the terms tea and beer
Term specificity should be interpreted as a statistical
rather than semantic property of the term
Statistical term specificity. The inverse of the number
of documents in which the term occurs

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 37
Inverse Document Frequency
Terms are distributed in a text according to Zipf’s Law
Thus, if we sort the vocabulary terms in decreasing
order of document frequencies we have

n(r) ∼ r−α

where n(r) refer to the rth largest document frequency

and α is an empirical constant
That is, the document frequency of term ki is an
exponential function of its rank.

n(r) = Cr−α

where C is a second empirical constant

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 38
Inverse Document Frequency
Setting α = 1 (simple approximation for english
collections) and taking logs we have

log n(r) = log C − log r

For r = 1, we have C = n(1), i.e., the value of C is the

largest document frequency
This value works as a normalization constant

An alternative is to do the normalization assuming

C = N , where N is the number of docs in the collection

log r ∼ log N − log n(r)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 39
Inverse Document Frequency
Let ki be the term with the rth largest document
frequency, i.e., n(r) = ni . Then,
N
idfi = log
ni

where idfi is called the inverse document frequency

of term ki
Idf provides a foundation for modern term weighting
schemes and is used for ranking in almost all IR
systems

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 40
Inverse Document Frequency
Idf values for example collection

term ni idfi = log(N/ni )

1 to 2 1
2 do 3 0.415
3 is 1 2
4 be 4 0
5 or 1 2
6 not 1 2
7 I 2 1
8 am 2 1
9 what 1 2
10 think 1 2
11 therefore 1 2
12 da 1 2
13 let 1 2
14 it 1 2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 41
TF-IDF weighting scheme
The best known term weighting schemes use weights
that combine idf factors with term frequencies
Let wi,j be the term weight associated with the term ki
and the document dj
Then, we define
(
(1 + log fi,j ) × log nNi if fi,j > 0
wi,j =
0 otherwise

which is referred to as a tf-idf weighting scheme

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 42
TF-IDF weighting scheme
Tf-idf weights of all terms present in our example
document collection

d1 d2 d3 d4
1 to 3 2 - -
2 do 0.830 - 1.073 1.073
3 is 4 - - -
4 be - - - -
5 or - 2 - -
6 not - 2 - -
7 I - 2 2 -
8 am - 2 1 -
9 what - 2 - -
10 think - - 2 -
11 therefore - - 2 -
12 da - - - 5.170
13 let - - - 4
14 it - - - 4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 43
Variants of TF-IDF
Several variations of the above expression for tf-idf
weights are described in the literature
For tf weights, five distinct variants are illustrated below

tf weight

binary {0,1}

raw frequency fi,j

log normalization 1 + log fi,j

f
double normalization 0.5 0.5 + 0.5 maxi,j
i fi,j

f
double normalization K K + (1 − K) maxi,j
i fi,j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 44
Variants of TF-IDF
Five distinct variants of idf weight

idf weight

unary 1
N
inverse frequency log ni
N
inv frequency smooth log(1 + ni )
maxi ni
inv frequeny max log(1 + ni )
N −ni
probabilistic inv frequency log ni

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 45
Variants of TF-IDF
Recommended tf-idf weighting schemes

weighting scheme document term weight query term weight

N f N
1 fi,j ∗ log ni (0.5 + 0.5 maxii,qfi,q ) ∗ log ni
N
2 1 + log fi,j log(1 + ni )
N
3 (1 + log fi,j ) ∗ log ni (1 + log fi,q ) ∗ log nNi

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 46
TF-IDF Properties
Consider the tf, idf, and tf-idf weights for the Wall Street
Journal reference collection
To study their behavior, we would like to plot them
together
While idf is computed over all the collection, tf is
computed on a per document basis. Thus, we need a
representation of tf based on all the collection, which is
provided by the term collection frequency Fi
This reasoning leads to the following tf and idf term
weights:
XN
tfi = 1 + log fi,j N
idfi = log
j=1 ni

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 47
TF-IDF Properties
Plotting tf and idf in logarithmic scale yields

We observe that tf and idf weights present power-law

behaviors that balance each other
The terms of intermediate idf values display maximum
tf-idf weights and are most interesting for ranking

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 48
Document Length Normalization
Document sizes might vary widely
This is a problem because longer documents are more
likely to be retrieved by a given query
To compensate for this undesired effect, we can divide
the rank of each document by its length
This procedure consistently leads to better ranking, and
it is called document length normalization

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 49
Document Length Normalization
Methods of document length normalization depend on the
representation adopted for the documents:
Size in bytes: consider that each document is represented
simply as a stream of bytes
Number of words: each document is represented as a single
string, and the document length is the number of words in it
Vector norms: documents are represented as vectors of
weighted terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 50
Document Length Normalization
Documents represented as vectors of weighted terms
Each term of a collection is associated with an orthonormal unit
vector ~ki in a t-dimensional space
For each term ki of a document dj is associated the term vector
component wi,j × ~ki

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 51
Document Length Normalization
The document representation d~j is a vector composed
of all its term vector components

d~j = (w1,j , w2,j , ..., wt,j )

The document length is given by the norm of this vector,

which is computed as follows
v
u t
uX
|d~j | = t 2
wi,j
i

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 52
Document Length Normalization
Three variants of document lengths for the example
collection

d1 d2 d3 d4
size in bytes 34 37 41 43
number of words 10 11 10 12
vector norm 5.068 4.899 3.762 7.738

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 53
The Vector Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 54
The Vector Model
Boolean matching and binary weights is too limiting
The vector model proposes a framework in which
partial matching is possible
This is accomplished by assigning non-binary weights
to index terms in queries and in documents
Term weights are used to compute a degree of
similarity between a query and each document
The documents are ranked in decreasing order of their
degree of similarity

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 55
The Vector Model
For the vector model:
The weight wi,j associated with a pair (ki , dj ) is positive and
non-binary
The index terms are assumed to be all mutually independent
They are represented as unit vectors of a t-dimensionsal space (t
is the total number of index terms)
The representations of document dj and query q are
t-dimensional vectors given by

d~j = (w1j , w2j , . . . , wtj )

~q = (w1q , w2q , . . . , wtq )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 56
The Vector Model
Similarity between a document dj and a query q
j d

–
q

i cos(θ) = d~j •~
q
|d~j |×|~q|

Pt
wi,j ×wi,q
sim(dj , q) = qP
t
i=1
2
qP
t 2
i=1 wi,j × j=1 wi,q

Since wij > 0 and wiq > 0, we have 0 6 sim(dj , q) 6 1

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 57
The Vector Model
Weights in the Vector model are basically tf-idf weights
N
wi,q = (1 + log fi,q ) × log
ni
N
wi,j = (1 + log fi,j ) × log
ni

These equations should only be applied for values of

term frequency greater than zero
If the term frequency is zero, the respective weight is
also zero

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 58
The Vector Model
Document ranks computed by the Vector model for the
query “to do” (see tf-idf weight values in Slide 43)

doc rank computation rank

1∗3+0.415∗0.830
d1 5.068
0.660
1∗2+0.415∗0
d2 4.899
0.408
1∗0+0.415∗1.073
d3 3.762
0.118
1∗0+0.415∗1.073
d4 7.738
0.058

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 59
The Vector Model
Advantages:
term-weighting improves quality of the answer set
partial matching allows retrieval of docs that approximate the
query conditions
cosine ranking formula sorts documents according to a degree of
similarity to the query
document length normalization is naturally built-in into the ranking
Disadvantages:
It assumes independence of index terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 60
Probabilistic Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 61
Probabilistic Model
The probabilistic model captures the IR problem using a
probabilistic framework
Given a user query, there is an ideal answer set for
this query
Given a description of this ideal answer set, we could
retrieve the relevant documents
Querying is seen as a specification of the properties of
this ideal answer set
But, what are these properties?

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 62
Probabilistic Model
An initial set of documents is retrieved somehow
The user inspects these docs looking for the relevant
ones (in truth, only top 10-20 need to be inspected)
The IR system uses this information to refine the
description of the ideal answer set
By repeating this process, it is expected that the
description of the ideal answer set will improve

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 63
Probabilistic Ranking Principle
The probabilistic model
Tries to estimate the probability that a document will be relevant
to a user query
Assumes that this probability depends on the query and
document representations only
The ideal answer set, referred to as R, should maximize the
probability of relevance

But,
How to compute these probabilities?
What is the sample space?

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 64
The Ranking
Let,
R be the set of relevant documents to query q
R be the set of non-relevant documents to query q
P (R|d~j ) be the probability that dj is relevant to the query q
P (R|d~j ) be the probability that dj is non-relevant to q

The similarity sim(dj , q) can be defined as

P (R|d~j )
sim(dj , q) =
P (R|d~j )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 65
The Ranking
Using Bayes’ rule,

P (d~j |R, q) × P (R, q) P (d~j |R, q)

sim(dj , q) = ∼
P (d~j |R, q) × P (R, q) P (d~j |R, q)

where
P (d~j |R, q) : probability of randomly selecting the document
dj from the set R
P (R, q) : probability that a document randomly selected
from the entire collection is relevant to query q
P (d~j |R, q) and P (R, q) : analogous and complementary

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 66
The Ranking
Assuming that the weights wi,j are all binary and
assuming independence among the index terms:
Q Q
( ki |wi,j =1 P (ki |R, q)) × ( ki |wi,j =0 P (k i |R, q))
sim(dj , q) ∼ Q Q
( ki |wi,j =1 P (ki |R, q)) × ( ki |wi,j =0 P (k i |R, q))

where
P (ki |R, q): probability that the term ki is present in a
document randomly selected from the set R
P (k i |R, q): probability that ki is not present in a document
randomly selected from the set R
probabilities with R: analogous to the ones just described

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 67
The Ranking
To simplify our notation, let us adopt the following
conventions
piR = P (ki |R, q)
qiR = P (ki |R, q)
Since
P (ki |R, q) + P (k i |R, q) = 1
P (ki |R, q) + P (k i |R, q) = 1
we can write:
Q Q
( ki |wi,j =1 piR ) × ( ki |wi,j =0 (1 − piR ))
sim(dj , q) ∼ Q Q
( ki |wi,j =1 qiR ) × ( ki |wi,j =0 (1 − qiR ))

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 68
The Ranking
Taking logarithms, we write

Y Y
sim(dj , q) ∼ log piR + log (1 − piR )
ki |wi,j =1 ki |wi,j =0
Y Y
− log qiR − log (1 − qiR )
ki |wi,j =1 ki |wi,j =0

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 69
The Ranking
Summing up terms that cancel each other, we obtain

Y Y
sim(dj , q) ∼ log piR + log (1 − pir )
ki |wi,j =1 ki |wi,j =0
Y Y
− log (1 − pir ) + log (1 − pir )
ki |wi,j =1 ki |wi,j =1
Y Y
− log qiR − log (1 − qiR )
ki |wi,j =1 ki |wi,j =0
Y Y
+ log (1 − qiR ) − log (1 − qiR )
ki |wi,j =1 ki |wi,j =1

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 70
The Ranking
Using logarithm operations, we obtain

Y piR Y
sim(dj , q) ∼ log + log (1 − piR )
(1 − piR )
ki |wi,j =1 ki
Y (1 − qiR ) Y
+ log − log (1 − qiR )
qiR
ki |wi,j =1 ki

Notice that two of the factors in the formula above are a

function of all index terms and do not depend on
document dj . They are constants for a given query and
can be disregarded for the purpose of ranking

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 71
The Ranking
Further, assuming that
∀ ki 6∈ q, piR = qiR
and converting the log products into sums of logs, we
finally obtain

P    
piR 1−qiR
sim(dj , q) ∼ ki ∈q∧ki ∈dj log 1−piR + log qiR

which is a key expression for ranking computation in the

probabilistic model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 72
Term Incidence Contingency Table
Let,
N be the number of documents in the collection
ni be the number of documents that contain term ki
R be the total number of relevant documents to query q
ri be the number of relevant documents that contain term ki

Based on these variables, we can build the following

contingency table
relevant non-relevant all docs
docs that contain ki ri ni − ri ni
docs that do not contain ki R − ri N − ni − (R − ri ) N − ni
all docs R N −R N

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 73
Ranking Formula
If information on the contingency table were available
for a given query, we could write
ri
piR = R
ni −ri
qiR = N −R

Then, the equation for ranking computation in the

probabilistic model could be rewritten as
 
X ri N − ni − R + ri
sim(dj , q) ∼ log ×
R − ri ni − ri
ki [q,dj ]

where ki [q, dj ] is a short notation for ki ∈ q ∧ ki ∈ dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 74
Ranking Formula
In the previous formula, we are still dependent on an
estimation of the relevant dos for the query
For handling small values of ri , we add 0.5 to each of
the terms in the formula above, which changes
sim(dj , q) into
 
X ri + 0.5 N − ni − R + ri + 0.5
log ×
R − ri + 0.5 ni − ri + 0.5
ki [q,dj ]

This formula is considered as the classic ranking

equation for the probabilistic model and is known as the
Robertson-Sparck Jones Equation

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 75
Ranking Formula
The previous equation cannot be computed without
estimates of ri and R
One possibility is to assume R = ri = 0, as a way to
boostrap the ranking equation, which leads to

P  
N −ni +0.5
sim(dj , q) ∼ ki [q,dj ] log ni +0.5

This equation provides an idf-like ranking computation

In the absence of relevance information, this is the
equation for ranking in the probabilistic model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 76
Ranking Example
Document ranks computed by the previous probabilistic
ranking equation for the query “to do”

doc rank computation rank

4−2+0.5 4−3+0.5
d1 log 2+0.5 + log 3+0.5 - 1.222

d2 log 4−2+0.5
2+0.5 0

d3 log 4−3+0.5
3+0.5 - 1.222

d4 log 4−3+0.5
3+0.5 - 1.222

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 77
Ranking Example
The ranking computation led to negative weights
because of the term “do”
Actually, the probabilistic ranking equation produces
negative terms whenever ni > N/2
One possible artifact to contain the effect of negative
weights is to change the previous equation to:
 
X N + 0.5
sim(dj , q) ∼ log
ni + 0.5
ki [q,dj ]

By doing so, a term that occurs in all documents

(ni = N ) produces a weight equal to zero

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 78
Ranking Example
Using this latest formulation, we redo the ranking
computation for our example collection for the query “to
do” and obtain

doc rank computation rank

d1 log 4+0.5
2+0.5 + log
4+0.5
3+0.5 1.210
4+0.5
d2 log 2+0.5 0.847
4+0.5
d3 log 3+0.5 0.362
4+0.5
d4 log 3+0.5 0.362

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 79
Estimaging ri and R
Our examples above considered that ri = R = 0
An alternative is to estimate ri and R performing an
initial search:
select the top 10-20 ranked documents
inspect them to gather new estimates for ri and R
remove the 10-20 documents used from the collection
rerun the query with the estimates obtained for ri and R

Unfortunately, procedures such as these require human

intervention to initially select the relevant documents

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 80
Improving the Initial Ranking
Consider the equation

   
X piR 1 − qiR
sim(dj , q) ∼ log + log
1 − piR qiR
ki ∈q∧ki ∈dj

How obtain the probabilities piR and qiR ?

Estimates based on assumptions:
piR = 0.5
ni
qiR = N where ni is the number of docs that contain ki
Use this initial guess to retrieve an initial ranking
Improve upon this initial ranking

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 81
Improving the Initial Ranking
Substituting piR and qiR into the previous Equation, we
obtain:
 
X N − ni
sim(dj , q) ∼ log
ni
ki ∈q∧ki ∈dj

That is the equation used when no relevance

information is provided, without the 0.5 correction factor
Given this initial guess, we can provide an initial
probabilistic ranking
After that, we can attempt to improve this initial ranking
as follows

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 82
Improving the Initial Ranking
We can attempt to improve this initial ranking as follows
Let
D : set of docs initially retrieved
Di : subset of docs retrieved that contain ki
Reevaluate estimates:
piR = D
D
i

ni −Di
qiR = N −D

This process can then be repeated recursively

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 83
Improving the Initial Ranking
 
X N − ni
sim(dj , q) ∼ log
ni
ki ∈q∧ki ∈dj

To avoid problems with D = 1 and Di = 0:

Di + 0.5 ni − Di + 0.5
piR = ; qiR =
D+1 N −D+1

Also,
Di + nNi ni − Di + nNi
piR = ; qiR =
D+1 N −D+1

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 84
Pluses and Minuses
Advantages:
Docs ranked in decreasing order of probability of
relevance
Disadvantages:
need to guess initial estimates for piR
method does not take into account tf factors
the lack of document length normalization

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 85
Comparison of Classic Models
Boolean model does not provide for partial matches
and is considered to be the weakest classic model
There is some controversy as to whether the
probabilistic model outperforms the vector model
Croft suggested that the probabilistic model provides a
better retrieval performance
However, Salton et al showed that the vector model
outperforms it with general collections
This also seems to be the dominant thought among
researchers and practitioners of IR.

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 86
Modern Information Retrieval

Modeling

Part II: Alternative Set and Vector Models

Set-Based Model
Extended Boolean Model
Fuzzy Set Model
The Generalized Vector Model
Latent Semantic Indexing
Neural Network for IR

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 87
Alternative Set Theoretic Models
Set-Based Model
Extended Boolean Model
Fuzzy Set Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 88
Set-Based Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 89
Set-Based Model
This is a more recent approach (2005) that combines
set theory with a vectorial ranking
The fundamental idea is to use mutual dependencies
among index terms to improve results
Term dependencies are captured through termsets,
which are sets of correlated terms
The approach, which leads to improved results with
various collections, constitutes the first IR model that
effectively took advantage of term dependence with
general collections

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 90
Termsets
Termset is a concept used in place of the index terms
A termset Si = {ka , kb , ..., kn } is a subset of the terms in
the collection
If all index terms in Si occur in a document dj then we
say that the termset Si occurs in dj
There are 2t termsets that might occur in the documents
of a collection, where t is the vocabulary size
However, most combinations of terms have no semantic meaning
Thus, the actual number of termsets in a collection is far smaller
than 2t

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 91
Termsets
Let t be the number of terms of the collection
Then, the set VS = {S1 , S2 , ..., S2t } is the vocabulary-set
of the collection
To illustrate, consider the document collection below

To do is to be.
To be is to do. To be or not to be.
I am what I am.

d1
d2
I think therefore I am.
Do be do be do. Do do do, da da da.
Let it be, let it be.

d3
d4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 92
Termsets
To simplify notation, let us define
ka = to kd = be kg = I kj = think km = let
kb = do ke = or kh = am kk = therefore kn = it
kc = is kf = not ki = what kl = da

Further, let the letters a...n refer to the index terms

ka ...kn , respectively

abcad
adcab adefad
ghigh
d1
d2
gjkgh
bdbdb bbblll
mndmnd
d3
d4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 93
Termsets
Consider the query q as “to do be it”, i.e. q = {a, b, d, n}
For this query, the vocabulary-set is as below

Termset Set of Terms Documents

Sa {a} {d1 , d2 }
Sb {b} {d1 , d3 , d4 }
Notice that there are
Sd {d} {d1 , d2 , d3 , d4 }
11 termsets that occur
Sn {n} {d4 }
in our collection, out
Sab {a, b} {d1 }
of the maximum of 15
Sad {a, d} {d1 , d2 }
Sbd {b, d} {d1 , d3 , d4 }
termsets that can be
Sbn {b, n} {d4 } formed with the terms
Sabd {a, b, d} {d1 } in q
Sbdn {b, d, n} {d4 }

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 94
Termsets
At query processing time, only the termsets generated
by the query need to be considered
A termset composed of n terms is called an n-termset
Let Ni be the number of documents in which Si occurs
An n-termset Si is said to be frequent if Ni is greater
than or equal to a given threshold
This implies that an n-termset is frequent if and only if all of its
(n − 1)-termsets are also frequent
Frequent termsets can be used to reduce the number of
termsets to consider with long queries

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 95
Termsets
Let the threshold on the frequency of termsets be 2
To compute all frequent termsets for the query
q = {a, b, d, n} we proceed as follows
1. Compute the frequent 1-termsets and their inverted lists:
Sa = {d1 , d2 }
Sb = {d1 , d3 , d4 }
abcad
adefad
Sd = {d1 , d2 , d3 , d4 }
adcab
2. Combine the inverted lists to ghigh
d1
compute frequent 2-termsets: d2
Sad = {d1 , d2 } gjkgh
bdbdb bbblll
Sbd = {d1 , d3 , d4 } mndmnd
d3
3. Since there are no frequent 3- d4
termsets, stop

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 96
Termsets
Notice that there are only 5 frequent termsets in our
collection
Inverted lists for frequent n-termsets can be computed
by starting with the inverted lists of frequent 1-termsets
Thus, the only indice that is required are the standard inverted
lists used by any IR system

This is reasonably fast for short queries up to 4-5 terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 97
Ranking Computation
The ranking computation is based on the vector model,
but adopts termsets instead of index terms
Given a query q , let
{S1 , S2 , . . .} be the set of all termsets originated from q
Ni be the number of documents in which termset Si occurs
N be the total number of documents in the collection
Fi,j be the frequency of termset Si in document dj

For each pair [Si , dj ] we compute a weight Wi,j given by

(
N
(1 + log Fi,j ) log(1 + N ) if Fi,j > 0
Wi,j = i
0 Fi,j = 0

We also compute a Wi,q value for each pair [Si , q]

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 98
Ranking Computation
Consider
query q = {a, b, d, n}
document d1 = ‘‘a b c a d a d c a b’’

Termset Weight
Sa Wa,1 (1 + log 4) ∗ log(1 + 4/2) = 4.75
Sb Wb,1 (1 + log 2) ∗ log(1 + 4/3) = 2.44
Sd Wd,1 (1 + log 2) ∗ log(1 + 4/4) = 2.00
Sn Wn,1 0 ∗ log(1 + 4/1) = 0.00
Sab Wab,1 (1 + log 2) ∗ log(1 + 4/1) = 4.64
Sad Wad,1 (1 + log 2) ∗ log(1 + 4/2) = 3.17
Sbd Wbd,1 (1 + log 2) ∗ log(1 + 4/3) = 2.44
Sbn Wbn,1 0 ∗ log(1 + 4/1) = 0.00
Sdn Wdn,1 0 ∗ log(1 + 4/1) = 0.00
Sabd Wabd,1 (1 + log 2) ∗ log(1 + 4/1) = 4.64
Sbdn Wbdn,1 0 ∗ log(1 + 4/1) = 0.00

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 99
Ranking Computation
A document dj and a query q are represented as
vectors in a 2t -dimensional space of termsets

d~j = (W1,j , W2,j , . . . , W2t ,j )

~q = (W1,q , W2,q , . . . , W2t ,q )

The rank of dj to the query q is computed as follows

P
~
dj • ~q Si Wi,j × Wi,q
sim(dj , q) = =
|d~j | × |~q| |d~j | × |~q|

For termsets that are not in the query q , Wi,q = 0

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 100
Ranking Computation
The document norm |d~j | is hard to compute in the
space of termsets
Thus, its computation is restricted to 1-termsets
Let again q = {a, b, d, n} and d1
The document norm in terms of 1-termsets is given by

q
|d~1 | = 2 + W2 + W2 + W2
Wa,1 b,1 c,1 d,1
p
= 4.752 + 2.442 + 4.642 + 2.002
= 7.35

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 101
Ranking Computation
To compute the rank of d1 , we need to consider the
seven termsets Sa , Sb , Sd , Sab , Sad , Sbd , and Sabd
The rank of d1 is then given by

sim(d1 , q) = (Wa,1 ∗ Wa,q + Wb,1 ∗ Wb,q + Wd,1 ∗ Wd,q +

Wab,1 ∗ Wab,q + Wad,1 ∗ Wad,q + Wbd,1 ∗ Wbd,q +
Wabd,1 ∗ Wabd,q ) /|d~1 |
= (4.75 ∗ 1.58 + 2.44 ∗ 1.22 + 2.00 ∗ 1.00 +
4.64 ∗ 2.32 + 3.17 ∗ 1.58 + 2.44 ∗ 1.22 +
4.64 ∗ 2.32)/7.35
= 5.71

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 102
Closed Termsets
The concept of frequent termsets allows simplifying the
ranking computation
Yet, there are many frequent termsets in a large
collection
The number of termsets to consider might be prohibitively high
with large queries

To resolve this problem, we can further restrict the

ranking computation to a smaller number of termsets
This can be accomplished by observing some
properties of termsets such as the notion of closure

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 103
Closed Termsets
The closure of a termset Si is the set of all frequent
termsets that co-occur with Si in the same set of docs
Given the closure of Si , the largest termset in it is called
a closed termset and is referred to as Φi
We formalize, as follows
Let Di ⊆ C be the subset of all documents in which termset Si
occurs and is frequent
Let S(Di ) be a set composed of the frequent termsets that occur
in all documents in Di and only in those

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 104
Closed Termsets
Then, the closed termset SΦi satisfies the following
property

6 ∃Sj ∈ S(Di ) | SΦi ⊂ Sj

Frequent and closed termsets for our example

collection, considering a minimum threshold equal to 2

frequency(Si ) frequent termset closed termset

4 d d
3 b, bd bd
2 a, ad ad
2 g, h, gh, ghd ghd

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 105
Closed Termsets
Closed termsets encapsulate smaller termsets
occurring in the same set of documents
The ranking sim(d1 , q) of document d1 with regard to
query q is computed as follows:
d1 =’’a b c a d a d c a b ’’
q = {a, b, d, n}
minimum frequency threshold = 2

sim(d1 , q) = (Wd,1 ∗ Wd,q + Wab,1 ∗ Wab,q + Wad,1 ∗ Wad,q +

Wbd,1 ∗ Wbd,q + Wabd,1 ∗ Wabd,q )/|d~1 |
= (2.00 ∗ 1.00 + 4.64 ∗ 2.32 + 3.17 ∗ 1.58 +
2.44 ∗ 1.22 + 4.64 ∗ 2.32)/7.35
= 4.28

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 106
Closed Termsets
Thus, if we restrict the ranking computation to closed
termsets, we can expect a reduction in query time
Smaller the number of closed termsets, sharper is the
reduction in query processing time

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 107
Extended Boolean Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 108
Extended Boolean Model
In the Boolean model, no ranking of the answer set is
generated
One alternative is to extend the Boolean model with the
notions of partial matching and term weighting
This strategy allows one to combine characteristics of
the Vector model with properties of Boolean algebra

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 109
The Idea
Consider a conjunctive Boolean query given by
q = kx ∧ ky
For the boolean model, a doc that contains a single
term of q is as irrelevant as a doc that contains none
However, this binary decision criteria frequently is not
in accordance with common sense
An analogous reasoning applies when one considers
purely disjunctive queries

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 110
The Idea
When only two terms x and y are considered, we can
plot queries and docs in a two-dimensional space

A document dj is positioned in this space through the

adoption of weights wx,j and wy,j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 111
The Idea
These weights can be computed as normalized tf-idf
factors as follows
fx,j idfx
wx,j = ×
maxx fx,j maxi idfi

where
fx,j is the frequency of term kx in document dj
idfi is the inverse document frequency of term ki , as before

To simplify notation, let

wx,j = x and wy,j = y
d~j = (wx,j , wy,j ) as the point dj = (x, y)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 112
The Idea
For a disjunctive query qor = kx ∨ ky , the point (0, 0) is
the least interesting one
This suggests taking the distance from (0, 0) as a
measure of similarity

r
x2 + y 2
sim(qor , d) =
2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 113
The Idea
For a conjunctive query qand = kx ∧ ky , the point (1, 1) is
the most interesting one
This suggests taking the complement of the distance
from the point (1, 1) as a measure of similarity

r
(1 − x)2 + (1 − y)2
sim(qand , d) = 1−
2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 114
The Idea

r
x2 + y 2
sim(qor , d) =
2
r
(1 − x)2 + (1 − y)2
sim(qand , d) = 1 −
2

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 115
Generalizing the Idea
We can extend the previous model to consider
Euclidean distances in a t-dimensional space
This can be done using p-norms which extend the notion
of distance to include p-distances, where 1 ≤ p ≤ ∞
A generalized conjunctive query is given by
qand = k1 ∧p k2 ∧p . . . ∧p km
A generalized disjunctive query is given by
qor = k1 ∨p k2 ∨p . . . ∨p km

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 116
Generalizing the Idea
The query-document similarities are now given by
 p p p
 p1
sim(qor , dj ) = x1 +x2 m
+...+xm

 p p p
 p1
sim(qand , dj ) = 1 − (1−x1 ) +(1−x2m) +...+(1−xm )

where each xi stands for a weight wi,d

If p = 1 then (vector-like)
x1 +...+xm
sim(qor , dj ) = sim(qand , dj ) = m

If p = ∞ then (Fuzzy like)

sim(qor , dj ) = max(xi )
sim(qand , dj ) = min(xi )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 117
Properties
By varying p, we can make the model behave as a
vector, as a fuzzy, or as an intermediary model
The processing of more general queries is done by
grouping the operators in a predefined order
For instance, consider the query q = (k1 ∧p k2 ) ∨p k3
k1 and k2 are to be used as in a vectorial retrieval
while the presence of k3 is required
The similarity sim(q, dj ) is computed as
   1 p
 p1
(1−x1 )p +(1−x2 )p p p
 1 − 2 + x 3
 
sim(q, d) =  
 2 

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 118
Conclusions
Model is quite powerful
Properties are interesting and might be useful
Computation is somewhat complex
However, distributivity operation does not hold for
ranking computation:
q1 = (k1 ∨ k2 ) ∧ k3
q2 = (k1 ∧ k3 ) ∨ (k2 ∧ k3 )
sim(q1 , dj ) 6= sim(q2 , dj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 119
Fuzzy Set Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 120
Fuzzy Set Model
Matching of a document to a query terms is
approximate or vague
This vagueness can be modeled using a fuzzy
framework, as follows:
each query term defines a fuzzy set
each doc has a degree of membership in this set
This interpretation provides the foundation for many IR
models based on fuzzy theory
In here, we discuss the model proposed by
Ogawa, Morita, and Kobayashi

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 121
Fuzzy Set Theory
Fuzzy set theory deals with the representation of
classes whose boundaries are not well defined
Key idea is to introduce the notion of a degree of
membership associated with the elements of the class
This degree of membership varies from 0 to 1 and
allows modelling the notion of marginal membership
Thus, membership is now a gradual notion, contrary to
the crispy notion enforced by classic Boolean logic

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 122
Fuzzy Set Theory
A fuzzy subset A of a universe of discourse U is
characterized by a membership function

µA : U → [0, 1]

This function associates with each element u of U a

number µA (u) in the interval [0, 1]
The three most commonly used operations on fuzzy
sets are:
the complement of a fuzzy set
the union of two or more fuzzy sets
the intersection of two or more fuzzy sets

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 123
Fuzzy Set Theory
Let,
U be the universe of discourse
A and B be two fuzzy subsets of U
A be the complement of A relative to U
u be an element of U
Then,

µA (u) = 1 − µA (u)
µA∪B (u) = max(µA (u), µB (u))
µA∩B (u) = min(µA (u), µB (u))

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 124
Fuzzy Information Retrieval
Fuzzy sets are modeled based on a thesaurus, which
defines term relationships
A thesaurus can be constructed by defining a term-term
correlation matrix C
Each element of C defines a normalized correlation
factor ci,` between two terms ki and k`
ni,l
ci,l =
ni + nl − ni,l
where
ni : number of docs which contain ki
nl : number of docs which contain kl
ni,l : number of docs which contain both ki and kl

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 125
Fuzzy Information Retrieval
We can use the term correlation matrix C to associate a
fuzzy set with each index term ki
In this fuzzy set, a document dj has a degree of
membership µi,j given by

Y
µi,j = 1 − (1 − ci,l )
kl ∈ d j

The above expression computes an algebraic sum over

all terms in dj
A document dj belongs to the fuzzy set associated with
ki , if its own terms are associated with ki

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 126
Fuzzy Information Retrieval
If dj contains a term kl which is closely related to ki , we
have
ci,l ∼ 1
µi,j ∼ 1
and ki is a good fuzzy index for dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 127
Fuzzy IR: An Example
Da
cc cc Db
cc

Dq = cc1 + cc2 + cc3

Dc

Consider the query q = ka ∧ (kb ∨ ¬kc )

The disjunctive normal form of q is composed of 3
conjunctive components (cc), as follows:
~qdnf = (1, 1, 1) + (1, 1, 0) + (1, 0, 0) = cc1 + cc2 + cc3
Let Da , Db and Dc be the fuzzy sets associated with the
terms ka , kb and kc , respectively

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 128
Fuzzy IR: An Example
Da
cc cc Db
cc

Dq = cc1 + cc2 + cc3

Dc

Let µa,j , µb,j , and µc,j be the degrees of memberships of

document dj in the fuzzy sets Da , Db , and Dc . Then,
cc1 = µa,j µb,j µc,j
cc2 = µa,j µb,j (1 − µc,j )
cc3 = µa,j (1 − µb,j )(1 − µc,j )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 129
Fuzzy IR: An Example
Da
cc cc Db
cc

Dq = cc1 + cc2 + cc3

Dc
µq,j = µcc1 +cc2 +cc3 ,j
3
Y
= 1− (1 − µcci ,j )
i=1
= 1 − (1 − µa,j µb,j µc,j ) ×
(1 − µa,j µb,j (1 − µc,j )) × (1 − µa,j (1 − µb,j )(1 − µc,j ))

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 130
Conclusions
Fuzzy IR models have been discussed mainly in the
literature associated with fuzzy theory
They provide an interesting framework which naturally
embodies the notion of term dependencies
Experiments with standard test collections are not
available

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 131
Alternative Algebraic Models
Generalized Vector Model
Latent Semantic Indexing
Neural Network Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 132
Generalized Vector Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 133
Generalized Vector Model
Classic models enforce independence of index terms
For instance, in the Vector model
A set of term vectors {~k1 , ~k2 , . . ., ~kt } are linearly independent
Frequently, this is interpreted as ∀i ,j ⇒ ~ki • ~kj = 0

In the generalized vector space model, two index term

vectors might be non-orthogonal

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 134
Key Idea
As before, let wi,j be the weight associated with [ki , dj ]
and V = {k1 , k2 , . . ., kt } be the set of all terms

If the wi,j weights are binary, all patterns of occurrence

of terms within docs can be represented by minterms:

(k1 , k2 , k3 , . . . , kt )
m1 = (0, 0, 0, . . . , 0)
m2 = (1, 0, 0, . . . , 0)
For instance, m2 indi-
m3 = (0, 1, 0, . . . , 0) cates documents in which
m4 = (1, 1, 0, . . . , 0) solely the term k1 occurs
..
.
m2 t = (1, 1, 1, . . . , 1)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 135
Key Idea
For any document dj , there is a minterm mr that
includes exactly the terms that occur in the document
Let us define the following set of minterm vectors m
~ r,

1, 2, . . . , 2t
m
~1 = (1, 0, . . . , 0) Notice that we can associate
m
~2 = (0, 1, . . . , 0) each unit vector m ~ r with a
.. ~i•m
minterm mr , and that m ~j =
. 0 for all i 6= j
m
~ 2t = (0, 0, . . . , 1)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 136
Key Idea
Pairwise orthogonality among the m
~ r vectors does not
imply independence among the index terms
On the contrary, index terms are now correlated by the
m
~ r vectors
For instance, the vector m
~ 4 is associated with the minterm
m4 = (1, 1, . . . , 0)
This minterm induces a dependency between terms k1 and k2
Thus, if such document exists in a collection, we say that the
minterm m4 is active

The model adopts the idea that co-occurrence of terms

induces dependencies among these terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 137
Forming the Term Vectors
Let on(i, mr ) return the weight {0, 1} of the index term ki
in the minterm mr
The vector associated with the term ki is computed as:

P
~ki ∀r on(i, mr ) ci,r m ~r
= qP
on(i, m ) c 2
∀r r i,r
X
ci,r = wi,j
dj | c(dj )=mr

Notice that for a collection of size N , only N minterms

affect the ranking (and not 2t )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 138
Dependency between Index Terms
A degree of correlation between the terms ki and kj can
now be computed as:
X
~ki • ~kj = on(i, mr ) × ci,r × on(j, mr ) × cj,r
∀r

This degree of correlation sums up the dependencies

between ki and kj induced by the docs in the collection

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 139
The Generalized Vector Model
An Example

K1 K2 K3
K1 d1 2 0 1
d2
d6 d7 K2 d2 1 0 0
d4
d5 d3 0 1 3
d1 d3 d4 2 0 0
d5 1 2 4
d6 1 2 0
K3
d7 0 5 0
q 1 2 3

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 140
Computation of ci,r

K1 K2 K3 K1 K2 K3 c1,r c2,r c3,r

d1 2 0 1 d1 = m6 1 0 1 m1 0 0 0
d2 1 0 0 d2 = m2 1 0 0 m2 3 0 0
d3 0 1 3 d3 = m7 0 1 1 m3 0 5 0
d4 2 0 0 d4 = m2 1 0 0 m4 0 0 0
d5 1 2 4 d5 = m8 1 1 1 m5 0 0 0
d6 0 2 2 d6 = m7 0 1 1 m6 2 0 1
d7 0 5 0 d7 = m3 0 1 0 m7 0 3 5
q 1 2 3 q = m8 1 1 1 m8 1 2 4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 141
 −
→
Computation of ki

−
→ (3m
~√2 +2m
~ 6 +m
~ 8) c1,r c2,r c3,r
k1 = 32 +22 +12 m1 0 0 0
−
→ (5m +3m
~ 3√ ~ 7 +2m
~ 8) m2 3 0 0
k2 = 5+3+2 m3 0 5 0
−
→ (1m +5m
~ 6√ ~ 7 +4m
~ 8) m4 0 0 0
k3 = 1+5+4 m5 0 0 0
m6 2 0 1
m7 0 3 5
m8 1 2 4

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 142
Computation of Document Vectors

−
→ −
→ − →
d1 = 2 k1 + k3
K1 K2 K3
→ −
− →
d2 = k1 d1 2 0 1
→ −
− → →
− d2 1 0 0
d3 = k2 + 3 k3
d3 0 1 3
→
− −
→
d4 = 2 k1 d4 2 0 0
→ −
− → →
− →
− d5 1 2 4
d5 = k1 + 2 k2 + 4 k3 d6 0 2 2
→
− −
→ →
−
d6 = 2 k2 + 2 k3 d7 0 5 0
→
− −
→ q 1 2 3
d7 = 5 k2
→
− →
− →
− →
−
q = k1 + 2 k2 + 3 k3

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 143
Conclusions
Model considers correlations among index terms
Not clear in which situations it is superior to the
standard Vector model
Computation costs are higher
Model does introduce interesting new ideas

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 144
Latent Semantic Indexing

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 145
Latent Semantic Indexing
Classic IR might lead to poor retrieval due to:
unrelated documents might be included in the
answer set
relevant documents that do not contain at least one
index term are not retrieved
Reasoning: retrieval based on index terms is vague
and noisy
The user information need is more related to concepts
and ideas than to index terms
A document that shares concepts with another
document known to be relevant might be of interest

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 146
Latent Semantic Indexing
The idea here is to map documents and queries into a
dimensional space composed of concepts
Let
t: total number of index terms
N : number of documents
M = [mij ]: term-document matrix t × N

To each element of M is assigned a weight wi,j

associated with the term-document pair [ki , dj ]
The weight wi,j can be based on a tf-idf weighting scheme

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 147
Latent Semantic Indexing
The matrix M = [mij ] can be decomposed into three
components using singular value decomposition

M = K · S · DT

were
K is the matrix of eigenvectors derived from C = M · MT
DT is the matrix of eigenvectors derived from MT · M
S is an r × r diagonal matrix of singular values where
r = min(t, N ) is the rank of M

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 148
Computing an Example
Let MT = [mij ] be given by

K1 K2 K3 q • dj
d1 2 0 1 5
d2 1 0 0 1
d3 0 1 3 11
d4 2 0 0 2
d5 1 2 4 17
d6 1 2 0 5
d7 0 5 0 10
q 1 2 3

Compute the matrices K, S, and Dt

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 149
Latent Semantic Indexing
In the matrix S, consider that only the s largest singular
values are selected
Keep the corresponding columns in K and DT
The resultant matrix is called Ms and is given by

Ms = Ks · Ss · DTs

where s, s < r, is the dimensionality of a reduced

concept space
The parameter s should be
large enough to allow fitting the characteristics of the data
small enough to filter out the non-relevant representational details

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 150
Latent Ranking
The relationship between any two documents in s can
be obtained from the MTs · Ms matrix given by

MTs · Ms = (Ks · Ss · DTs )T · Ks · Ss · DTs

= Ds · Ss · KTs · Ks · Ss · DTs
= Ds · Ss · Ss · DTs
= (Ds · Ss ) · (Ds · Ss )T

In the above matrix, the (i, j) element quantifies the

relationship between documents di and dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 151
Latent Ranking
The user query can be modelled as a
pseudo-document in the original M matrix
Assume the query is modelled as the document
numbered 0 in the M matrix
The matrix MTs · Ms quantifies the relationship between
any two documents in the reduced concept space
The first row of this matrix provides the rank of all the
documents with regard to the user query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 152
Conclusions
Latent semantic indexing provides an interesting
conceptualization of the IR problem
Thus, it has its value as a new theoretical framework
From a practical point of view, the latent semantic
indexing model has not yielded encouraging results

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 153
Neural Network Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 154
Neural Network Model
Classic IR:
Terms are used to index documents and queries
Retrieval is based on index term matching
Motivation:
Neural networks are known to be good pattern
matchers

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 155
Neural Network Model
The human brain is composed of billions of neurons
Each neuron can be viewed as a small processing unit
A neuron is stimulated by input signals and emits output
signals in reaction
A chain reaction of propagating signals is called a
spread activation process
As a result of spread activation, the brain might
command the body to take physical reactions

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 156
Neural Network Model
A neural network is an oversimplified representation of
the neuron interconnections in the human brain:
nodes are processing units
edges are synaptic connections
the strength of a propagating signal is modelled by a
weight assigned to each edge
the state of a node is defined by its activation level
depending on its activation level, a node might issue
an output signal

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 157
Neural Network for IR
A neural network model for information retrieval

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 158
Neural Network for IR
Three layers network: one for the query terms, one for
the document terms, and a third one for the documents
Signals propagate across the network
First level of propagation:
Query terms issue the first signals
These signals propagate across the network to
reach the document nodes
Second level of propagation:
Document nodes might themselves generate new
signals which affect the document term nodes
Document term nodes might respond with new
signals of their own

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 159
Quantifying Signal Propagation
Normalize signal strength (MAX = 1)
Query terms emit initial signal equal to 1
Weight associated with an edge from a query term
node ki to a document term node ki :
wi,q
wi,q = qP
t 2
i=1 wi,q

Weight associated with an edge from a document term

node ki to a document node dj :
wi,j
w i,j = qP
t 2
i=1 wi,j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 160
Quantifying Signal Propagation
After the first level of signal propagation, the activation
level of a document node dj is given by:
t Pt
X
i=1 wi,q wi,j
wi,q wi,j = qP qP
t 2 × t 2
i=1 i=1 wi,q i=1 w i,j

which is exactly the ranking of the Vector model

New signals might be exchanged among document
term nodes and document nodes
A minimum threshold should be enforced to avoid
spurious signal generation

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 161
Conclusions
Model provides an interesting formulation of the IR
problem
Model has not been tested extensively
It is not clear the improvements that the model might
provide

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 162
Modern Information Retrieval

Chapter 3
Modeling

Part III: Alternative Probabilistic Models

BM25
Language Models
Divergence from Randomness
Belief Network Models
Other Models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 163
BM25 (Best Match 25)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 164
BM25 (Best Match 25)
BM25 was created as the result of a series of
experiments on variations of the probabilistic model
A good term weighting is based on three principles
inverse document frequency
term frequency
document length normalization

The classic probabilistic model covers only the first of

these principles
This reasoning led to a series of experiments with the
Okapi system, which led to the BM25 ranking formula

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 165
BM1, BM11 and BM15 Formulas
At first, the Okapi system used the Equation below as
ranking formula
X N − ni + 0.5
sim(dj , q) ∼ log
ni + 0.5
ki ∈q∧ki ∈dj

which is the equation used in the probabilistic model,

when no relevance information is provided

It was referred to as the BM1 formula (Best Match 1)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 166
BM1, BM11 and BM15 Formulas
The first idea for improving the ranking was to introduce
a term-frequency factor Fi,j in the BM1 formula
This factor, after some changes, evolved to become
fi,j
Fi,j = S1 ×
K1 + fi,j
where
fi,j is the frequency of term ki within document dj
K1 is a constant setup experimentally for each collection
S1 is a scaling constant, normally set to S1 = (K1 + 1)

If K1 = 0, this whole factor becomes equal to 1 and

bears no effect in the ranking

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 167
BM1, BM11 and BM15 Formulas
The next step was to modify the Fi,j factor by adding
document length normalization to it, as follows:
0 fi,j
Fi,j = S1 ×
K1 ×len(dj )
avg _doclen + fi,j

where
len(dj ) is the length of document dj (computed, for instance, as
the number of terms in the document)
avg_doclen is the average document length for the collection

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 168
BM1, BM11 and BM15 Formulas
Next, a correction factor Gj,q dependent on the
document and query lengths was added

avg _doclen − len(dj )

Gj,q = K2 × len(q) ×
avg _doclen + len(dj )

where
len(q) is the query length (number of terms in the query)
K2 is a constant

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 169
BM1, BM11 and BM15 Formulas
A third additional factor, aimed at taking into account
term frequencies within queries, was defined as
fi,q
Fi,q = S3 ×
K3 + fi,q

where
fi,q is the frequency of term ki within query q
K3 is a constant
S3 is an scaling constant related to K3 , normally set to
S3 = (K3 + 1)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 170
BM1, BM11 and BM15 Formulas
Introduction of these three factors led to various BM
(Best Matching) formulas, as follows:

 
X N − ni + 0.5
simBM 1 (dj , q) ∼ log
ni + 0.5
ki [q,dj ]
 
X N − ni + 0.5
simBM 15 (dj , q) ∼ Gj,q + Fi,j × Fi,q × log
ni + 0.5
ki [q,dj ]
 
X 0 N − ni + 0.5
simBM 11 (dj , q) ∼ Gj,q + Fi,j × Fi,q × log
ni + 0.5
ki [q,dj ]

where ki [q, dj ] is a short notation for ki ∈ q ∧ ki ∈ dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 171
BM1, BM11 and BM15 Formulas
Experiments using TREC data have shown that BM11
outperforms BM15
Further, empirical considerations can be used to
simplify the previous equations, as follows:
Empirical evidence suggests that a best value of K2 is 0,
which eliminates the Gj,q factor from these equations
Further, good estimates for the scaling constants S1 and S3
are K1 + 1 and K3 + 1, respectively
Empirical evidence also suggests that making K3 very large
is better. As a result, the Fi,q factor is reduced simply to fi,q
For short queries, we can assume that fi,q is 1 for all terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 172
BM1, BM11 and BM15 Formulas
These considerations lead to simpler equations as
follows

 
X N − ni + 0.5
simBM 1 (dj , q) ∼ log
ni + 0.5
ki [q,dj ]

 
X (K1 + 1)fi,j N − ni + 0.5
simBM 15 (dj , q) ∼ × log
(K1 + fi,j ) ni + 0.5
ki [q,dj ]

 
X (K1 + 1)fi,j N − ni + 0.5
simBM 11 (dj , q) ∼ K1 len(dj )
× log
+ fi,j ni + 0.5
ki [q,dj ] avg _doclen

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 173
BM25 Ranking Formula
BM25: combination of the BM11 and BM15
The motivation was to combine the BM11 and BM25
term frequency factors as follows
(K1 + 1)fi,j
Bi,j = h i
len(dj )
K1 (1 − b) + b avg_doclen + fi,j

where b is a constant with values in the interval [0, 1]

If b = 0, it reduces to the BM15 term frequency factor

If b = 1, it reduces to the BM11 term frequency factor
For values of b between 0 and 1, the equation provides a
combination of BM11 with BM15

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 174
BM25 Ranking Formula
The ranking equation for the BM25 model can then be
written as
 
X N − ni + 0.5
simBM 25 (dj , q) ∼ Bi,j × log
ni + 0.5
ki [q,dj ]

where K1 and b are empirical constants

K1 = 1 works well with real collections
b should be kept closer to 1 to emphasize the document length
normalization effect present in the BM11 formula
For instance, b = 0.75 is a reasonable assumption
Constants values can be fine tunned for particular collections
through proper experimentation

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 175
BM25 Ranking Formula
Unlike the probabilistic model, the BM25 formula can be
computed without relevance information
There is consensus that BM25 outperforms the classic
vector model for general collections
Thus, it has been used as a baseline for evaluating new
ranking functions, in substitution to the classic vector
model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 176
Language Models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 177
Language Models
Language models are used in many natural language
processing applications
Ex: part-of-speech tagging, speech recognition, machine
translation, and information retrieval
To illustrate, the regularities in spoken language can be
modeled by probability distributions
These distributions can be used to predict the likelihood
that the next token in the sequence is a given word
These probability distributions are called language
models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 178
Language Models
A language model for IR is composed of the following
components
A set of document language models, one per document dj of the
collection
A probability distribution function that allows estimating the
likelihood that a document language model Mj generates each of
the query terms
A ranking function that combines these generating probabilities
for the query terms into a rank of document dj with regard to the
query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 179
Statistical Foundation
Let S be a sequence of r consecutive terms that occur
in a document of the collection:

S = k1 , k2 , . . . , kr

An n-gram language model uses a Markov process to

assign a probability of occurrence to S :
r
Y
Pn (S) = P (ki |ki−1 , ki−2 , . . . , ki−(n−1) )
i=1

where n is the order of the Markov process

The occurrence of a term depends on observing the
n − 1 terms that precede it in the text

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 180
Statistical Foundation
Unigram language model (n = 1): the estimatives are
based on the occurrence of individual words
Bigram language model (n = 2): the estimatives are
based on the co-occurrence of pairs of words
Higher order models such as Trigram language
models (n = 3) are usually adopted for speech
recognition
Term independence assumption: in the case of IR,
the impact of word order is less clear
As a result, Unigram models have been used extensively in IR

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 181
Multinomial Process
Ranking in a language model is provided by estimating
P (q|Mj )
Several researchs have proposed the adoption of a
multinomial process to generate the query
According to this process, if we assume that the query
terms are independent among themselves (unigram
model), we can write:
Y
P (q|Mj ) = P (ki |Mj )
ki ∈q

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 182
Multinomial Process
By taking logs on both sides
X
log P (q|Mj ) = log P (ki |Mj )
ki ∈q
X X
= log P∈ (ki |Mj ) + log P6∈ (ki |Mj )
ki ∈q∧dj ki ∈q∧¬dj
 
X P∈ (ki |Mj ) X
= log + log P6∈ (ki |Mj )
ki ∈q∧dj
P6∈ (ki |Mj ) ki ∈q

where P∈ and P6∈ are two distinct probability distributions:

The first is a distribution for the query terms in the document
The second is a distribution for the query terms not in the
document

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 183
Multinomial Process
For the second distribution, statistics are derived from
all the document collection
Thus, we can write

P6∈ (ki |Mj ) = αj P (ki |C)

where αj is a parameter associated with document dj

and P (ki |C) is a collection C language model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 184
Multinomial Process
P (ki |C) can be estimated in different ways
For instance, Hiemstra suggests an idf-like estimative:
ni
P (ki |C) = P
i ni

where ni is the number of docs in which ki occurs

Miller, Leek, and Schwartz suggest

Fi
P (ki |C) = P
i Fi
P
where Fi = j fi,j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 185
Multinomial Process
Thus, we obtain
 
X P∈ (ki |Mj ) X
log P (q|Mj ) = log + nq log αj + log P (ki |C)
αj P (ki |C)
ki ∈q∧dj ki ∈q
 
X P∈ (ki |Mj )
∼ log + nq log αj
αj P (ki |C)
ki ∈q∧dj

where nq stands for the query length and the last sum
was dropped because it is constant for all documents

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 186
Multinomial Process
The ranking function is now composed of two separate
parts
The f rst part assigns weights to each query term that
appears in the document, according to the expression
 
P∈ (ki |Mj )
log
αj P (ki |C)

This term weight plays a role analogous to the tf plus idf

weight components in the vector model
Further, the parameter αj can be used for document
length normalization

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 187
Multinomial Process
The second part assigns a fraction of probability mass
to the query terms that are not in the document—a
process called smoothing
The combination of a multinomial process with
smoothing leads to a ranking formula that naturally
includes tf , idf , and document length normalization
That is, smoothing plays a key role in modern language
modeling, as we now discuss

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 188
Smoothing
In our discussion, we estimated P6∈ (ki |Mj ) using P (ki |C)
to avoid assigning zero probability to query terms not in
document dj
This process, called smoothing, allows fine tuning the
ranking to improve the results.
One popular smoothing technique is to move some
mass probability from the terms in the document to the
terms not in the document, as follows:
(
P∈s (ki |Mj ) if ki ∈ dj
P (ki |Mj ) =
αj P (ki |C) otherwise

where P∈s (ki |Mj ) is the smoothed distribution for

terms in document dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 189
Smoothing
P
Since i P (ki |Mj )
= 1, we can write
X X
s
P∈ (ki |Mj ) + αj P (ki |C) = 1
ki ∈dj ki 6∈dj

That is,
P s (k |M )
1− P
ki ∈dj ∈ i j
αj = P
1 − ki ∈dj P (ki |C)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 190
Smoothing
Under the above assumptions, the smoothing
parameter αj is also a function of P∈s (ki |Mj )
As a result, distinct smoothing methods can be
obtained through distinct specifications of P∈s (ki |Mj )
Examples of smoothing methods:
Jelinek-Mercer Method
Bayesian Smoothing using Dirichlet Priors

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 191
Jelinek-Mercer Method
The idea is to do a linear interpolation between the
document frequency and the collection frequency
distributions:
fi,j Fi
P∈s (ki |Mj , λ) = (1 − λ) P +λ P
i fi,j i Fi

where 0 ≤ λ ≤ 1

It can be shown that

αj = λ

Thus, the larger the values of λ, the larger is the effect

of smoothing

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 192
Dirichlet smoothing
In this method, the language model is a multinomial
distribution in which the conjugate prior probabilities are
given by the Dirichlet distribution
This leads to
fi,j + λ PFiFi
P∈s (ki |Mj , λ) = P i

i fi,j + λ

As before, closer is λ to 0, higher is the influence of the

term document frequency. As λ moves towards 1, the
influence of the term collection frequency increases

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 193
Dirichlet smoothing
Contrary to the Jelinek-Mercer method, this influence is
always partially mixed with the document frequency
It can be shown that
λ
αj = P
i fi,j +λ

As before, the larger the values of λ, the larger is the

effect of smoothing

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 194
Smoothing Computation
In both smoothing methods above, computation can be
carried out efficiently
All frequency counts can be obtained directly from the
index
The values of αj can be precomputed for each
document
Thus, the complexity is analogous to the computation of
a vector space ranking using tf-idf weights

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 195
Applying Smoothing to Ranking
The IR ranking in a multinomial language model is
computed as follows:
compute P∈s (ki |Mj ) using a smoothing method

compute P (ki |C) using Pni or PFi

i ni i Fi
P
1− ki ∈dj P∈s (ki |Mj )
compute αj from the Equation αj = 1−
P
P (ki |C)
ki ∈dj

compute the ranking using the formula

 s 
X P∈ (ki |Mj )
log P (q|Mj ) = log + nq log αj
k ∈q∧d
αj P (ki |C)
i j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 196
Bernoulli Process
The first application of languages models to IR was due
to Ponte & Croft. They proposed a Bernoulli process for
generating the query, as we now discuss
Given a document dj , let Mj be a reference to a
language model for that document
If we assume independence of index terms, we can
compute P (q|Mj ) using a multivariate Bernoulli process:
Y Y
P (q|Mj ) = P (ki |Mj ) × [1 − P (ki |Mj )]
ki ∈q ki 6∈q

where P (ki |Mj ) are term probabilities

This is analogous to the expression for ranking
computation in the classic probabilistic model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 197
Bernoulli process
A simple estimate of the term probabilities is
fi,j
P (ki |Mj ) = P
` f`,j

which computes the probability that term ki will be

produced by a random draw (taken from dj )
However, the probability will become zero if ki does not
occur in the document
Thus, we assume that a non-occurring term is related to
dj with the probability P (ki |C) of observing ki in the
whole collection C

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 198
Bernoulli process
P (ki |C) can be estimated in different ways
For instance, Hiemstra suggests an idf-like estimative:
ni
P (ki |C) = P
` n`

where ni is the number of docs in which ki occurs

Miller, Leek, and Schwartz suggest

Fi X
P (ki |C) = P where Fi = fi,j
` F` j

This last equation for P (ki |C) is adopted here

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 199
Bernoulli process
As a result, we redefine P (ki |Mj ) as follows:

fi,j
 i fi,j if fi,j > 0

 P

P (ki |Mj ) =

 PFi

Fi if fi,j = 0
i

In this expression, P (ki |Mj ) estimation is based only on

the document dj when fi,j > 0
This is clearly undesirable because it leads to instability
in the model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 200
Bernoulli process
This drawback can be accomplished through an
average computation as follows
P
j|ki ∈dj P (ki |Mj )
P (ki ) =
ni

That is, P (ki ) is an estimate based on the language

models of all documents that contain term ki
However, it is the same for all documents that contain
term ki
That is, using P (ki ) to predict the generation of term ki
by the Mj involves a risk

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 201
Bernoulli process
To fix this, let us define the average frequency f i,j of
term ki in document dj as
X
f i,j = P (ki ) × fi,j
i

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 202
Bernoulli process
The risk Ri,j associated with using f i,j can be
quantified by a geometric distribution:
! !fi,j
1 f i,j
Ri,j = ×
1 + f i,j 1 + f i,j

For terms that occur very frequently in the collection,

f i,j  0 and Ri,j ∼ 0
For terms that are rare both in the document and in the
collection, fi,j ∼ 1, f i,j ∼ 1, and Ri,j ∼ 0.25

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 203
Bernoulli process
Let us refer the probability of observing term ki
according to the language model Mj as PR (ki |Mj )
We then use the risk factor Ri,j to compute PR (ki |Mj ),
as follows

P (k |M ) (1−Ri,j ) × P (k )Ri,j if f

 i j i i,j > 0
PR (ki |Mj ) =

 PFi
Fi otherwise
i

In this formulation, if Ri,j ∼ 0 then PR (ki |Mj ) is basically

a function of P (ki |Mj )
Otherwise, it is a mix of P (ki ) and P (ki |Mj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 204
Bernoulli process
Substituting into original P (q|Mj ) Equation, we obtain
Y Y
P (q|Mj ) = PR (ki |Mj ) × [1 − PR (ki |Mj )]
ki ∈q ki 6∈q

which computes the probability of generating the query

from the language (document) model
This is the basic formula for ranking computation in a
language model based on a Bernoulli process for
generating the query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 205
Divergence from Randomness

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 206
Divergence from Randomness
A distinct probabilistic model has been proposed by
Amati and Rijsbergen
The idea is to compute term weights by measuring the
divergence between a term distribution produced by a
random process and the actual term distribution
Thus, the name divergence from randomness
The model is based on two fundamental assumptions,
as follows

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 207
Divergence from Randomness
First assumption:
Not all words are equally important for describing the content of
the documents
Words that carry little information are assumed to be randomly
distributed over the whole document collection C
Given a term ki , its probability distribution over the whole
collection is referred to as P (ki |C)
The amount of information associated with this distribution is
given by

− log P (ki |C)

By modifying this probability function, we can implement distinct

notions of term randomness

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 208
Divergence from Randomness
Second assumption:
A complementary term distribution can be obtained by
considering just the subset of documents that contain term ki
This subset is referred to as the elite set
The corresponding probability distribution, computed with regard
to document dj , is referred to as P (ki |dj )
Smaller the probability of observing a term ki in a document dj ,
more rare and important is the term considered to be
Thus, the amount of information associated with the term in the
elite set is defined as
1 − P (ki |dj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 209
Divergence from Randomness
Given these assumptions, the weight wi,j of a term ki in
a document dj is defined as

wi,j = [− log P (ki |C)] × [1 − P (ki |dj )]

Two term distributions are considered: in the collection

and in the subset of docs in which it occurs
The rank R(dj , q) of a document dj with regard to a
query q is then computed as
P
R(dj , q) = fi,q × wi,j
ki ∈q

where fi,q is the frequency of term ki in the query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 210
Random Distribution
To compute the distribution of terms in the collection,
distinct probability models can be considered
For instance, consider that Bernoulli trials are used to
model the occurrences of a term in the collection
To illustrate, consider a collection with 1,000 documents
and a term ki that occurs 10 times in the collection
Then, the probability of observing 4 occurrences of
term ki in a document is given by
  4  6
10 1 1
P (ki |C) = 1−
4 1000 1000

which is a standard binomial distribution

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 211
Random Distribution
In general, let p = 1/N be the probability of observing a
term in a document, where N is the number of docs
The probability of observing fi,j occurrences of term ki
in document dj is described by a binomial distribution:
 
Fi fi,j ×
P (ki |C) = p (1 − p)Fi −fi,j
fi,j

Define

λi = p × Fi

and assume that p → 0 when N → ∞, but that

λi = p × Fi remains constant

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 212
Random Distribution
Under these conditions, we can aproximate the
binomial distribution by a Poisson process, which yields

e−λi λfi i ,j
P (ki |C) =
fi,j !

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 213
Random Distribution
The amount of information associated with term ki in
the collection can then be computed as
!
e−λi
λfi i ,j
− log P (ki |C) = − log
fi,j !
≈ −fi,j log λi + λi log e + log(fi,j !)
   
fi,j 1
≈ fi,j log + λi + − fi,j log e
λi 12fi,j + 1
1
+ log(2πfi,j )
2

in which the logarithms are in base 2 and the factorial

term fi,j ! was approximated by the Stirling’s formula
√ (fi,j +0.5) −fi,j (12fi,j +1)−1
fi,j ! ≈ 2π fi,j e e

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 214
Random Distribution
Another approach is to use a Bose-Einstein distribution
and approximate it by a geometric distribution:

P (ki |C) ≈ p × pfi,j

where p = 1/(1 + λi )
The amount of information associated with term ki in
the collection can then be computed as
   
1 ×
λi
− log P (ki |C) ≈ − log − fi,j log
1 + λi 1 + λi

which provides a second form of computing the term

distribution over the whole collection

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 215
Distribution over the Elite Set
The amount of information associated with term
distribution in elite docs can be computed by using
Laplace’s law of succession
1
1 − P (ki |dj ) =
fi,j + 1

Another possibility is to adopt the ratio of two Bernoulli

processes, which yields
Fi + 1
1 − P (ki |dj ) =
ni × (fi,j + 1)

where ni is the number of documents in which the term

occurs, as before

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 216
Normalization
These formulations do not take into account the length
of the document dj . This can be done by normalizing
the term frequency fi,j
Distinct normalizations can be used, such as

0 avg _doclen
fi,j = fi,j ×
len(dj )
or
 
0 × log 1 +
avg _doclen
fi,j = fi,j
len(dj )

where avg _doclen is the average document length in the

collection and len(dj ) is the length of document dj

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 217
Normalization
To compute wi,j weights using normalized term
0
frequencies, just substitute the factor fi,j by fi,j
In here we consider that a same normalization is
applied for computing P (ki |C) and P (ki |dj )
By combining different forms of computing P (ki |C) and
P (ki |dj ) with different normalizations, various ranking
formulas can be produced

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 218
Bayesian Network Models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 219
Bayesian Inference
One approach for developing probabilistic models of IR
is to use Bayesian belief networks
Belief networks provide a clean formalism for combining
distinct sources of evidence
Types of evidences: past queries, past feedback cycles, distinct
query formulations, etc.

In here we discuss two models:

Inference network, proposed by Turtle and Croft
Belief network model, proposed by Ribeiro-Neto and Muntz

Before proceeding, we briefly introduce Bayesian

networks

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 220
Bayesian Networks
Bayesian networks are directed acyclic graphs
(DAGs) in which
the nodes represent random variables
the arcs portray causal relationships between these variables
the strengths of these causal influences are expressed by
conditional probabilities

The parents of a node are those judged to be direct

causes for it
This causal relationship is represented by a link
directed from each parent node to the child node
The roots of the network are the nodes without parents

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 221
Bayesian Networks
Let
xi be a node in a Bayesian network G
Γxi be the set of parent nodes of xi

The influence of Γxi on xi can be specified by any set of

functions Fi (xi , Γxi ) that satisfy
X
Fi (xi , Γxi ) = 1
∀xi
0 ≤ Fi (xi , Γxi ) ≤ 1

where xi also refers to the states of the random variable

associated to the node xi

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 222
Bayesian Networks
A Bayesian network for a joint probability distribution
P (x1 , x2 , x3 , x4 , x5 )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 223
Bayesian Networks
The dependencies declared in the network allow the
natural expression of the joint probability distribution

P (x1 , x2 , x3 , x4 , x5 ) = P (x1 )P (x2 |x1 )P (x3 |x1 )P (x4 |x2 , x3 )P (x5 |x3 )

The probability P (x1 ) is called

the prior probability for the
network
It can be used to model previ-
ous knowledge about the se-
mantics of the application

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 224
Inference Network Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 225
Inference Network Model
An epistemological view of the information retrieval
problem
Random variables associated with documents, index
terms and queries
A random variable associated with a document dj
represents the event of observing that document
The observation of dj asserts a belief upon the random
variables associated with its index terms

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 226
Inference Network Model
An inference network for information retrieval

Nodes of the network

documents (dj )
index terms (ki )
queries (q, q1 , and q2 )
user information need (I)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 227
Inference Network Model
The edges from dj to the nodes ki indicate that the
observation of dj increase the belief in the variables ki
dj has index terms k2 , ki , and kt
q has index terms k1 , k2 , and ki
q1 and q2 model boolean formulation
q1 = (k1 ∧ k2 ) ∨ ki )
I = (q ∨ q1 )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 228
Inference Network Model
Let ~k = (k1 , k2 , . . . , kt ) a t-dimensional vector
ki ∈ {0, 1}, then k has 2t possible states
Define
(
1 if ki = 1 according to ~k
on(i, ~k) =
0 otherwise

Let dj ∈ {0, 1} and q ∈ {0, 1}

The ranking of dj is a measure of how much evidential
support the observation of dj provides to the query

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 229
Inference Network Model
The ranking is computed as P (q ∧ dj ) where q and dj are
short representations for q = 1 and dj = 1, respectively
dj stands for a state where dj = 1 and ∀l6=j ⇒ dl = 0,
because we observe one document at a time
X
P (q ∧ dj ) = P (q ∧ dj |~k) × P (~k)
∀~
k
X
= P (q ∧ dj ∧ ~k)
∀~
k
X
= P (q|dj ∧ ~k) × P (dj ∧ ~k)
∀~
k
X
= P (q|~k) × P (~k|dj ) × P (dj )
∀~
k

P (q ∧ dj ) = 1 − P (q ∧ dj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 230
Inference Network Model
The observation of dj separates its children index term
nodes making them mutually independent
This implies that P (~k|dj ) can be computed in product
form which yields
X
P (q ∧ dj ) = P (q|~k) × P (dj ) ×

∀~
k
 
Y Y
 P (ki |dj ) × P (k i |dj )
∀i|on(i,~
k)=1 ∀i|on(i,~
k)=0

where P (k i |dj ) = 1 − P (ki |dj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 231
Prior Probabilities
The prior probability P (dj ) reflects the probability of
observing a given document dj
In Turtle and Croft this probability is set to 1/N , where
N is the total number of documents in the system:
1 1
P (dj ) = P (dj ) = 1 −
N N
To include document length normalization in the model,
we could also write P (dj ) as follows:
1
P (dj ) = P (dj ) = 1 − P (dj )
|d~j |

where |d~j | stands for the norm of the vector d~j

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 232
Network for Boolean Model
How an inference network can be tuned to subsume the
Boolean model?
First, for the Boolean model, the prior probabilities are
given by:
1 1
P (dj ) = P (dj ) = 1 −
N N
Regarding the conditional probabilities P (ki |dj ) and
P (q|~k), the specification is as follows
(
1 if ki ∈ dj
P (ki |dj ) =
0 otherwise
P (k i |dj ) = 1 − P (ki |dj )

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 233
Network for Boolean Model
We can use P (ki |dj ) and P (q|~k) factors to compute the
evidential support the index terms provide to q :
(
1 if c(q) = c(~k)
P (q|~k) =
0 otherwise
P (q|~k) = 1 − P (q|~k)

where c(q) and c(~k) are the conjunctive components

associated with q and ~k , respectively

By using these definitions in P (q ∧ dj ) and P (q ∧ dj )

equations, we obtain the Boolean form of retrieval

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 234
Network for TF-IDF Strategies
For a tf-idf ranking strategy
Prior probability P (dj ) reflects the importance of
document normalization
1
P (dj ) = P (dj ) = 1 − P (dj )
|d~j |

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 235
Network for TF-IDF Strategies
For the document-term beliefs, we write:

P (ki |dj ) = α + (1 − α) × f i,j × idf i

P (k i |dj ) = 1 − P (ki |dj )

where α varies from 0 to 1, and empirical evidence

suggests that α = 0.4 is a good default value
Normalized term frequency and inverse document
frequency:

fi,j log nNi

f i,j = idf i =
maxi fi,j log N

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 236
Network for TF-IDF Strategies
For the term-query beliefs, we write:

X
P (q|~k) = f i,j × wq
ki ∈q

P (q|~k) = 1 − P (q|~k)

where wq is a parameter used to set the maximum

belief achievable at the query node

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 237
Network for TF-IDF Strategies
By substituting these definitions into P (q ∧ dj ) and
P (q ∧ dj ) equations, we obtain a tf-idf form of ranking
We notice that the ranking computed by the inference
network is distinct from that for the vector model
However, an inference network is able to provide good
retrieval performance with general collections

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 238
Combining Evidential Sources
In Figure below, the node q is the standard
keyword-based query formulation for I
The second query q1 is a Boolean-like query formulation
for the same information need

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 239
Combining Evidential Sources
Let I = q ∨ q1
In this case, the ranking provided by the inference
network is computed as

X
P (I ∧ dj ) = P (I|~k) × P (~k|dj ) × P (dj )
~k
X
= (1 − P (q|~k) P (q 1 |~k)) × P (~k|dj ) × P (dj )
~k

which might yield a retrieval performance which

surpasses that of the query nodes in isolation
(Turtle and Croft)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 240
Belief Network Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 241
Belief Network Model
The belief network model is a variant of the inference
network model with a slightly different network topology
As the Inference Network Model
Epistemological view of the IR problem
Random variables associated with documents, index
terms and queries
Contrary to the Inference Network Model
Clearly defined sample space
Set-theoretic view

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 242
Belief Network Model

By applying Bayes’ rule, we can write

P (dj |q) = P (dj ∧ q)/P (q)

X
P (dj |q) ∼ P (dj ∧ q|~k) × P (~k)
∀~
k

because P (q) is a constant for all documents in the collection

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 243
Belief Network Model
Instantiation of the index term variables separates the
nodes q and d making them mutually independent:
X
P (dj |q) ∼ P (dj |~k) × P (q|~k) × P (~k)
∀~k

To complete the belief network we need to specify the

conditional probabilities P (q|~k) and P (dj |~k)
Distinct specifications of these probabilities allow the
modeling of different ranking strategies

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 244
Belief Network Model
For the vector model, for instance, we define a vector ~ki
given by
~ki = ~k | on(i, ~k) = 1 ∧ ∀j6=i on(i, ~k) = 0

The motivation is that tf-idf ranking strategies sum up

the individual contributions of index terms
We proceed as follows

 qPwi,q if ~k = ~ki ∧ on(i, ~q) = 1
t
P (q|~k) =
2
i=1 wi,q
 0 otherwise
P (q|~k) = 1 − P (q|~k)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 245
Belief Network Model
Further, define


 qPwi,j if ~k = ~ki ∧ on(i, d~j ) = 1
t
P (dj |~k) =
2
i=1 wi,j
 0 otherwise
P (dj |~k) = 1 − P (dj |~k)

Then, the ranking of the retrieved documents coincides

with the ranking ordering generated by the vector model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 246
Computational Costs
In the inference network model only the states which
have a single document active node are considered
Thus, the cost of computing the ranking is linear on the
number of documents in the collection
However, the ranking computation is restricted to the
documents which have terms in common with the query
The networks do not impose additional costs because
the networks do not include cycles

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 247
Other Models
Hypertext Model
Web-based Models
Structured Text Retrieval
Multimedia Retrieval
Enterprise and Vertical Search

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 248
Hypertext Model

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 249
The Hypertext Model
Hypertexts provided the basis for the design of the
hypertext markup language (HTML)
Written text is usually conceived to be read
sequentially
Sometimes, however, we are looking for information that
cannot be easily captured through sequential reading
For instance, while glancing at a book about the history of the
wars, we might be interested in wars in Europe

In such a situation, a different organization of the text is

desired

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 250
The Hypertext Model
The solution is to define a new organizational structure
besides the one already in existence
One way to accomplish this is through hypertexts, that
are high level interactive navigational structures
A hypertext consists basically of nodes that are
correlated by directed links in a graph structure

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 251
The Hypertext Model
Two nodes A and B might be connected by a directed
link lAB which correlates the texts of these two nodes
In this case, the reader might move to the node B while
reading the text associated with node A
When the hypertext is large, the user might lose track of
the organizational structure of the hypertext
To avoid this problem, it is desirable that the hypertext
include a hypertext map
In its simplest form, this map is a directed graph which displays
the current node being visited

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 252
The Hypertext Model
Definition of the structure of the hypertext should be
accomplished in a domain modeling phase
After the modeling of the domain, a user interface
design should be concluded prior to implementation
Only then, can we say that we have a proper hypertext
structure for the application at hand

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 253
Web-based Models

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 254
Web-based Models
The first Web search engines were fundamentally IR
engines based on the models we have discussed here
The key differences were:
the collections were composed of Web pages (not documents)
the pages had to be crawled
the collections were much larger

This third difference also meant that each query word

retrieved too many documents
As a result, results produced by these engines were
frequently dissatisfying

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 255
Web-based Models
A key piece of innovation was missing—the use of link
information present in Web pages to modify the ranking
There are two fundamental approaches to do this
namely, PageRank and Hubs-Authorities
Such approaches are covered in Chapter 11 of the book (Web
Retrieval)

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 256
Structured Text Retrieval

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 257
Structured Text Retrieval
All the IR models discussed here treat the text as a
string with no particular structure
However, information on the structure might be
important to the user for particular searches
Ex: retrieve a book that contains a figure of the Eiffel tower in a
section whose title contains the term “France”
The solution to this problem is to take advantage of the
text structure of the documents to improve retrieval
Structured text retrieval are discussed in Chapter 13 of
the book

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 258
Multimedia Retrieval

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 259
Multimedia Retrieval
Multimedia data, in the form of images, audio, and
video, frequently lack text associated with them
The retrieval strategies that have to be applied are quite
distinct from text retrieval strategies
However, multimedia data are an integral part of the
Web
Multimedia retrieval methods are discussed in great
detail in Chapter 14 of the book

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 260
Enterprise and Vertical Search

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 261
Enterprise and Vertical Search
Enterprise search is the task of searching for
information of interest in corporate document collections
Many issues not present in the Web, such as privacy,
ownership, permissions, are important in enterprise
search
In Chapter 15 of the book we discuss in detail some
enterprise search solutions

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 262
Enterprise and Vertical Search
A vertical collection is a repository of documents
specialized in a given domain of knowledge
To illustrate, Lexis-Nexis offers full-text search focused on the
area of business and in the area of legal

Vertical collections present specific challenges with

regard to search and retrieval

Chap 03: Modeling, Baeza-Yates & Ribeiro-Neto, Modern Information Retrieval, 2nd Edition – p. 263