Introduction to

Information
Retrieval
Term-document incidence matrices
 Sec.
1.1

Unstructured data in 1620

• Which plays of Shakespeare contain the words
Brutus AND Caesar but NOT Calpurnia?
• One could grep all of Shakespeare’s plays for Brutus
and Caesar, then strip out lines containing
Calpurnia?
• Why is that not the answer?
– Slow (for large corpora)
– NOT Calpurnia is non-trivial
– Other operations (e.g., find the word Romans near
countrymen) not feasible
– Ranked retrieval (best documents to return)
• Later lectures
9
 Sec.
1.1

Term-document incidence matrices

Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth
Antony 1 1 0 0 0 1
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0

Brutus AND Caesar BUT NOT 1 if play

Calpurnia contains word,
0 otherwise
 Sec.
1.1

Incidence vectors
• So we have a 0/1 vector for each term.
• To answer query: take the vectors for Brutus,
Caesar and Calpurnia (complemented) ➔
bitwise AND.
– 110100 AND
– 110111 AND Antony
Antony and Cleopatra
1
Julius Caesar
1
The Tempest
0
Hamlet
0
Othello
0
Macbeth
1

– 101111 =
Brutus 1 1 0 1 0 0
Caesar 1 1 0 1 1 1
Calpurnia 0 1 0 0 0 0

– 100100
Cleopatra 1 0 0 0 0 0
mercy 1 0 1 1 1 1
worser 1 0 1 1 1 0

4
 Sec.
1.1

Answers to query
• Antony and Cleopatra, Act III,
Scene ii
Agrippa [Aside to DOMITIUS ENOBARBUS]: Why,
Enobarbus, When Antony found Julius
Caesar dead,
He cried almost to roaring; and he wept When at
Philippi he found Brutus slain.

• Hamlet, Act III, Scene ii

Lord Polonius: I did enact Julius Caesar I was killed i’
the Capitol; Brutus killed me.

5
 Sec.
1.1

Bigger collections
• Consider N = 1 million documents, each with about
1000 words.
• Avg 6 bytes/word including
spaces/punctuation
– 6GB of data in the documents.
• Say there are M = 500K distinct terms among these.

6
 Sec.
1.1

Can’t build the matrix

• 500K x 1M matrix has half-a-trillion 0’s and 1’s.

• But it has no more than one billion 1’s. Why

?
– matrix is extremely sparse.

• What’s a better representation?

– We only record the 1 positions.

7
Introduction to
Information
Retrieval
The Inverted Index
The key data structure underlying
modern IR
 Inverted index Sec.
• It is an index data structure storing a mapping content, such as 1.2

words or numbers, to its location or a set of documents.

• It is a hashmap like data structure that directs you from a word to a
document or a web page.
• For each term t, we must store a list of all documents that contain
t. – Identify each doc by a docID, a document serial number.
Can we use fixed size array for this ?

Brutus 1 2 4 11 31 45 173 174

Caesar 1 2 4 5 6 16 57
Calpurn 132

ia 2 31 54101
What happens if the word
Caesar is added to document
14? 16
 Sec.
1.2

Inverted index
• We need variable-size postings lists
– On disk, a continuous run of postings is normal
and best
– In memory, can use linked lists or variable l Posting
arrays• Some tradeoffs in size/ease of insertion engt
h
Brutu 1 2 4 11 31 45
s
Caes 173 174
1 2 4 5 6 16 57
ar
Calpurni 132
a
2 31 54101
Dictionary Postin
gs
Sorted by docID (more later on why).
17
 Sec.
1.2

Inverted index construction

Documents Friends, Romans, countrymen.
to be
indexed
Tokenizer

Token stream Friends Romans Countrymen

Linguistic modules

Modified friend roman countryman

tokens
Indexer frien 2 4
d
roma 1 2
Inverted n
index countrym
an 13 16
 Initial stages of text processing
• Text Cleaning and Preprocessing
This step involves removing any unnecessary characters, such as
punctuation, special symbols, and numbers, and correcting any
misspellings.
• Tokenization
– Cut character sequence into word tokens
• Deal with “John’s”, a state-of-the-art solution
• Normalization
– Map text and query term to same form
• You want U.S.A. and USA to match
• Stemming
– We may wish different forms of a root to match
• authorize, authorization
• Stop words
– We may omit very common words (or not)
• the, a, to, of
 Sec.
1.2

Indexer steps: Token sequence

• Sequence of (Modified token, Document ID) pairs.

Doc Doc
1 2
I did enact So let it be with
Julius Caesar I Caesar. The noble
was killed i’ the Brutus hath told
Capitol; Brutus you Caesar was
killed me. ambitious
 Sec.
1.2

Indexer steps: Sort

• Sort by terms
– And then
docID

Core indexing
step
 Sec.
1.2

Indexer steps: Dictionary & Postings

• Multiple term entries
in a single document
are merged.
• Split into Dictionary
and Postings
• Doc. frequency
information is added.

Why frequency?
Will discuss later.
Introduction to
Information
Retrieval
Query processing with an inverted index
 Sec.
1.3

The index we just built

• How do we process a query? Our focus

– Later - what kinds of queries can we process?

17
 Sec.
1.3

Query processing: AND

• Consider processing the query:
Brutus AND Caesar
– Locate Brutus in the Dictionary;
• Retrieve its postings.
– Locate Caesar in the Dictionary;
• Retrieve its postings.
– “Merge” the two postings (intersect the document
sets): 2 4 8 16 32 64 128 Brutus
1 2 3 5 8 13 21 Caesa
34 r
18
 Sec.
1.3

The merge
• Walk through the two postings
simultaneously, in time linear in the total
number of postings entries
2 4 8
32 64 128
16
Brutus
1 2 3 5
8 13 21 34 Caesar
If the list lengths are x and y, the merge takes
O(x+y) operations.
Crucial: postings sorted by docID.
19
Intersecting two postings lists (a “merge”
algorithm)

20
Introduction to
Information
Retrieval
The Boolean Retrieval Model
& Extended Boolean
Models
 Sec.
1.3

Boolean queries: Exact match

• The Boolean retrieval model is being able to ask a
query that is a Boolean expression:
– Boolean Queries are queries using AND, OR and NOT
to join query terms
• Views each document as a set of words
• Is precise: document matches condition or not.
– Perhaps the simplest model to build an IR system on
• Primary commercial retrieval tool for 3 decades.
• Many search systems still use are Boolean:
– Email, library catalog, Mac OS X Spotlight
22
 Sec.
1.4

Example: WestLaw http://www.westlaw.c

om/

• Largest commercial (paying

subscribers) legal search service
(started 1975; ranking added 1992;
new federated search added 2010)
• Tens of terabytes of data; ~700,000
users
• Majority of users still use boolean
queries
• Example query:
– What is the statute of limitations in
cases involving the federal tort
claims act? 23
– LIMIT! /3 STATUTE ACTION /S
FEDERAL /2 TORT /3 CLAIM
 Sec.
1.4

Example: WestLaw http://www.westlaw.c

om/

• Another example query:

– Requirements for disabled people to be able to
access a workplace
– disabl! /p access! /s work-site work-place
(employment /3 place)
• Note that SPACE is disjunction, not
conjunction!
• Long, precise queries; proximity operators;
incrementally developed; not like web search
• Many professional searchers still like Boolean
search
– You know exactly what you are getting
• But that doesn’t mean it actually works
 Sec.
1.3
Boolean queries:
More general merges
• Exercise: Adapt the merge for the queries:
Brutus AND NOT
Caesar Brutus OR
NOT Caesar

• Can we still run through the merge in time

O(x+y)? What can we achieve?

25
 Merging Sec.
1.3

What about an arbitrary Boolean formula?

(Brutus OR Caesar) AND NOT (Antony OR
Cleopatra)
• Can we always merge in “linear” time?
– Linear in what?
• Can we do better?

26
 Sec.
1.3

Query optimization
• What is the best order for query
processing?
• Consider a query that is an AND
of n terms.
• For each of the n terms, get its
Brutuspostings, then AND them 16

together 32
Caesa
64 128
r
Calpurni 1 2 3 5 8 16 21
a 34 Calpurnia AND
Query: Brutus AND 3
Caesar 13 16 5
 Sec.
1.3

Query optimization example

• Process in order of increasing freq:
– start with smallest set, then keep cutting further.

This is why we kept

document freq. in dictionary

Brutu 2 4 8 16 32 64 128
s
Caesa 1 2 3 5 8 16 21
r 34
Calpurni 13 16
a Execute
the query as (Calpurnia AND Brutus) AND
Caesar. 28
 Sec.
1.3

More general optimization

• e.g., (madding OR crowd) AND (ignoble
OR
strife)
• Get doc. freq.’s for all terms.
• Estimate the size of each OR by the sum of its
doc. freq.’s (conservative).
• Process in increasing order of OR sizes.

29
 Exercise
• Recommend a query
processing order for

(tangerine OR trees) AND Term Freq

(marmalade OR skies) AND eyes 213312
(kaleidoscope OR eyes) kaleidoscope 87009
marmalade 107913
• Which two terms should we skies 271658
process first? tangerine 46653
trees 316812

30
 Query processing exercises
• Exercise: If the query is friends AND romans
AND (NOT countrymen), how could we use the
freq of countrymen?
• Exercise: Extend the merge to an arbitrary
Boolean query. Can we always guarantee
execution in time linear in the total postings size?
• Hint: Begin with the case of a Boolean formula
query: in this, each query term appears only once
in the query.
31
 Exercise
• Try the search feature at http://www.rhy
mezone.com/shakespeare/
• Write down five search features you think it
could do better

32
Introduction to
Information
Retrieval
Phrase queries and positional indexes
 Phrase queries Sec.
• A phrase query is a type of search query where 2.4

multiple terms must appear together in a specific

order in the retrieved documents. Unlike single-term
queries or Boolean queries, phrase queries enforce
positional constraints to ensure the exact sequence
of words is found.
• We want to be able to answer queries such as
“stanford university” – as a phrase
• Thus the sentence “I went to university at Stanford”
is not a match.
– The concept of phrase queries has proven easily understood
by users; one of the few “advanced search” ideas that works
– Many more queries are implicit phrase queries
• For this, it no longer suffices to store only
<term : docs> entries
 A first attempt: Biword indexes Sec.
2.4.1
• A biword index is a simple approach to solving the
phrase query problem by treating every
consecutive pair of terms in the document as a
single unit (biword).
• Index every consecutive pair of terms in the text as a
phrase
• For example the text “Friends, Romans,
Countrymen” would generate the biwords
– friends romans
– romans countrymen
• Each of these biwords is now a dictionary term
• Two-word phrase query-processing is now
immediate.
 Advantages of Biword Indexing:
1.Fast query processing:
1. Two-word phrase queries are as efficient as standard term
lookups.
2. No need for positional comparisons, just direct lookups.
2.Reduced processing complexity:
1. Eliminates the need for positional indexes and adjacency checks.
3.Simple implementation:
1. Easily integrates with existing inverted index structures.

Challenges and Limitations:

1.Scalability for longer phrases:
•Biword indexing handles two-word phrases well, but longer queries like "stanford
university campus" require additional processing.
•A workaround is to break longer queries into overlapping biwords:
•"stanford university campus" → "stanford university" AND "university campus"
2.Storage overhead:
•The number of unique biwords increases significantly, leading to larger index sizes.
3.Vocabulary growth:
•Compound words and variations can expand the dictionary size dramatically (e.g., "New
York City" vs. "New York").
4.Limited flexibility:
•If a user searches for "romans friends", it won't match the biword "friends romans" because
order matters.
 Sec.
2.4.1

Longer phrase queries

• Longer phrases can be processed by breaking
them down
• stanford university palo alto can be broken
into the Boolean query on biwords:
stanford university AND university palo AND
palo alto

Without the docs, we cannot verify that the docs

matching the above Boolean query do contain
the phrase.

Can have false positives!

 Sec.
2.4.1

Issues for biword indexes

• False positives, as noted before
• Index blowup due to bigger dictionary
– Infeasible for more than biwords, big even for
them

• Biword indexes are not the standard solution

(for all biwords) but can be part of a
compound strategy
 Sec.
2.4.2

Solution 2: Positional indexes

• In the postings, store, for each term the
position(s) in which tokens of it
appear:

<term, number of docs containing term;

doc1: position1, position2 … ;
doc2: position1, position2
… ; etc.>
 Sec.
2.4.2

Positional index example

<be: 993427;
1: 7, 18, 33, 72, 86, 231;
Which of docs 1,2,4,5
2: 3, 149; could contain “to be
4: 17, 191, 291, 430, 434; or not to be”?

5: 363, 367, …>

• For phrase queries, we use a merge

algorithm recursively at the document level
• But we now need to deal with more than
just equality
 Processing a phrase query Sec.
2.4.2

• Extract inverted index entries for each distinct

term: to, be, or, not.
• Merge their doc:position lists to enumerate all
positions with “to be or not to be”.
– to:
• 2:1,17,74,222,551;
• 4:8,16,190,429,433;
• 7:13,23,191; ...
– be:
• 1:17,19;
• 4:17,191,291,430,434;
• 5:14,19,101; ...
• Same general method for proximity searches
 Sec.
2.4.2

Proximity queries
• LIMIT! /3 STATUTE /3 FEDERAL /2 TORT
– Again, here, /k means “within k words of”.
• Clearly, positional indexes can be used for
such queries; biword indexes cannot.
 Sec.
2.4.2

Positional index size

• A positional index expands postings storage
substantially
– Even though indices can be compressed
• Nevertheless, a positional index is now
standardly used because of the power and
usefulness of phrase and proximity queries …
whether used explicitly or implicitly in a
ranking retrieval system.
 Positional Index Size

The size of a positional index depends on several

factors:
1.Vocabulary Size (V): The number of distinct terms
in the collection.
2.Document Collection Size (N): The total number
of documents.
3.Average Document Length (L): The average
number of terms per document.
4.Average Term Frequency (Tf): The average
frequency of terms in a document.
 Components Affecting the Size:

•Term Storage: For each term, the index stores its occurrences
across all documents.

•Document ID Storage: For each occurrence, the document ID

needs to be stored.

•Position Storage: For each occurrence of the term in a

document, the position(s) of the term in that document must be
stored.
 Estimating the Size:
The size of a positional index can be estimated using the
following components:

•Document ID: Stored for each occurrence of a term.

•Position: Each position where the term occurs in a
document is stored.
•Given a total of T term occurrences in the collection, if
each term occurrence requires storing the document ID
and position, the total size can be estimated as:

Size≈T×(Size of DocID+Size of Position)

 Sec.
2.4.2

Positional index size

• Need an entry for each occurrence, not just
once per document
• Index size depends on average document
sizeWhy?
– Average web page has <1000 terms
– SEC filings, books, even some epic poems … easily
100,000 terms
• Consider
Document a
sizeterm with frequency 0.1%
Postings Positional postings
1000 1 1
100,000 1 100
 Sec.
2.4.2

Rules of thumb
• A positional index is 2–4 as large as a non-
positional index

• Positional index size 35–50% of volume of

original text

– Caveat: all of this holds for “English-like”

languages
 Sec.
2.4.3

Combination schemes
• These two approaches can be profitably
combined
– For particular phrases (“M ichael Jackson”,
“Britney Spears”) it is inefficient to keep on
merging positional postings lists
• Even more so for phrases like “The Who”
• Williams et al. (2004) evaluate a more
sophisticated mixed indexing scheme
– A typical web query mixture was executed in ¼ of the
time of using just a positional index
– It required 26% more space than having a positional
index alone
Key Idea of the Hybrid Approach
Instead of relying solely on either biword or positional indexes, a
combination scheme strategically uses:

1.Biword Indexing for Frequent/Fixed Phrases:

•Common or well-known phrases (e.g., "Michael Jackson", "New
York City", "The Who") are stored as biword entries for fast lookup,
avoiding unnecessary positional checks.

2.Positional Indexing for Flexible/Arbitrary Queries:

•For phrases not pre-indexed as biwords, positional indexes ensure
precise matching and support long or less common phrases.
By combining these approaches, search engines can reduce query
processing time while maintaining accuracy and minimizing index size
growth.