Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?

id=443868

Site: University of the People Printed by: Patrick Rolemodel Asante

Course: CS 3308-01 Information Retrieval - AY2025-T2 Date: Tuesday, 10 December 2024, 12:03 PM
Book: Learning Guide Unit 6

1 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Learning Guide Unit 6

2 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

3 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

• Evaluating of the e�ectiveness of Information retrieval

• Measures of information retrieval e�ectiveness including precision and recall
• Assessing relevance
• User utility of information retrieval system

By the end of this Unit, you will be able to:

1. Describe methodologies for evaluating information retrieval systems

2. Implement techniques to evaluate unranked retrieval sets including:
◦ Precision
◦ Recall
3. Compute the F Measure which balances precision and recall
4. Recognize various techniques to compute recall precision
◦ Interpolated precision
◦ 11-Point interpolated average precision
◦ Mean Average Precision
◦ Precision at k
◦ R-Precision

• Peer assess Unit 5 Development Assignment

• Read the Learning Guide and Reading Assignments
• Participate in the Discussion Assignment (post, comment, and rate in the Discussion Forum)
• Make entries to the Learning Journal
• Take the Self-Quiz

4 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Unit six explores the area of evaluation of the information retrieval system. Up to this point we have learned how to build the inverted
index, we have learned how to develop a weighted term retrieval scheme by computing the cosine similarity between a query and the
documents that are candidates to be relevant to the query and we have successfully put all of these components together to create a
complete search system that can successfully search our corpus of 2,476 Reuters news article documents.

What we have yet learned is how to determine if our search engine is really accomplishing the goal of searching the corpus and retrieving
the most relevant results. The weighting provided by the calculation of the tf-idf (term frequency inverse document frequency) metric
combined with the calculation of the cosine similarity is intended to identify those documents are likely to be relevant to the query terms
that the user of the system is using to search for information.

We really don’t know, however, if they in fact WERE relevant. In chapter 8 of our text we explore techniques to determine just how
e�ective our information retrieval system is at returning relevant documents. Some key metrics that will be used to determine relevance
is recall which is a ratio of relevant items received over the total number of potential relevant items. Obviously the higher the recall the
better.

Another key metric is precision which provides a ratio of the number of relevant items retrieved over the total number items retrieved.
This metric is a measure of the relevance retrieved in a query, where recall measures how e�ective the search was to get the relevant
items that are in the collection. Obviously a higher recall is a better more relevant query.

These two metrics are combined to form the F Measure which is an overall measure of the relevance and e�ciency of information
retrieval performance.

What is important to understand from this unit is how a corpus along with standard queries and results are used to test and validate the
e�ectiveness of an IR system. In chapter 8 of the text, we learn that there are a number of document collections (corpus) that are used for
this purpose. These document collections are used along with well known queries (keep in mind that a query is the terms that are used to
search the collection). An IR system to be evaluated �rst indexes the corpus and then the queries are used to test the results that the IR
system returns. What is important about this process is that for these queries there are known metrics such as the number of documents
in the collection that SHOULD be relevant. Because we have these standard queries for which the actual number of relevant documents
has been determined, we can use these same queries to determine how e�ective our IR system is. We will not be using most of these
corpus in this course because they are relatively large and would require considerable processing time.

5 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Manning, C.D., Raghaven, P., & Schütze, H. (2009). An Introduction to Information Retrieval (Online ed.). Cambridge, MA: Cambridge
University Press. Available at http://nlp.stanford.edu/IR-book/information-retrieval-book.html

Chapter 8: Evaluation in Information Retrieval

• Relevance
• Gold Standard Around Truth
• Precision
• Recall
• Accuracy
• F-Measure
• Interpolated precision

6 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Unit six looks at how to evaluate the e�ectiveness of an information retrieval system. Precision, recall, accuracy, and the F measure are all
discussed as metrics that can be used to measure the e�ectiveness of results retrieved from an IR system. In chapter 8 of the text, we
learn that there are a number of document collections (corpus) that are used for this purpose. These document collections are used along
with well known queries (keep in mind that a query is the terms that are used to search the collection). An IR system to be evaluated �rst
indexes the corpus and then the queries are used to test the results that the IR system returns. What is important about this process is
that for these queries there are known metrics such as the number of documents in the collection that SHOULD be relevant.

These measures of e�ectiveness are calculated based upon such known information and the results returned from a query submitted to
an IR system.

'For example, consider . We know that it contains documents which

Suppose that we have known metrics such as the fact that there are documents in this collection that are relevant to a query for
the terms .

Assume that the IR system that we have developed returns 8 relevant documents and 10 documents that are not relevant. Using this
information and the formulas for Precision, Recall, F-Measure, and Accuracy, calculate what each of these measures would be for the
example presented above. When you have determined the metric for each post a response that includes:

1. The Precision, Recall, F-Measure, and Accuracy e�ectiveness metrics which you will calculate using the metrics provided above.
2. Discuss which approach provides the most valid measure of the e�ectiveness of the IR system and why.

Keep in mind that Precision and Recall are used together a measure of e�ectiveness, the F-Measure provides a single measure that
balances Precision and Recall metrics and Accuracy provides a measure of the accuracy of classi�cations in the collection.

7 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Your learning journal entry must be a re�ective statement that considers the following questions:

• Describe what you did. This does not mean that you copy and paste from what you have posted or the assignments you have
prepared. You need to describe what you did and how you did it.
• Describe your reactions to what you did
• Describe any feedback you received or any speci�c interactions you had. Discuss how they were helpful
• Describe your feelings and attitudes
• Describe what you learned

Another set of questions to consider in your learning journal statement include:

• What surprised me or caused me to wonder?

• What happened that felt particularly challenging? Why was it challenging to me?
• What skills and knowledge do I recognize that I am gaining?
• What am I realizing about myself as a learner?
• In what ways am I able to apply the ideas and concepts gained to my own experience?

Your Learning Journal must be a minimum of 500 words.

8 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

The Self-Quiz gives you an opportunity to self-assess your knowledge of what you have learned so far.

The results of the Self-Quiz do not count towards your �nal grade, but the quiz is an important part of the University’s learning process
and it is expected that you will take it to ensure understanding of the materials presented. Reviewing and analyzing your results will help
you perform better on future Graded Quizzes and the Final Exam.

Please access the Self-Quiz on the main course homepage; it will be listed inside the Unit.

9 of 10 12/10/2024, 12:03 PM
Learning Guide Unit 6 | Home https://my.uopeople.edu/mod/book/tool/print/index.php?id=443868

Peer assess Unit 5 Development Assignment

Read the Learning Guide and Reading Assignments

Participate in the Discussion Assignment (post, comment, and rate in the Discussion Forum)

Make entries to the Learning Journal

Take the Self-Quiz

10 of 10 12/10/2024, 12:03 PM