Lecture-C

Lecture-C - Lecture C Retrieval Evaluation Introduction s...

Info iconThis preview shows pages 1–9. Sign up to view the full content.

View Full Document Right Arrow Icon
Lecture C Retrieval Evaluation
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Introduction The performation evaluation of an IR system includes Response time (IR, as well as DBMS) Space complexity (IR, as well as DBMS) Relevance ranking: preciseness of the answer set (IR) The evaluation is based on 1. A collection of documents 2. A set of example queries 3. A set of relevant documents of each query in (2) The evaluation measure qualifies the similarity between a) the set of retrieved documents Text Reference Collection
Background image of page 2
3 Introduction Two most commonly used retrieval measures : Precision Recall It is assumed that any query in an IR system is processed in batch mode in a lab, and thus the quality of the answer set is very important Another execution mode: the interface query session in which each user specifies the needed information through a series of interactive steps with the system, the original query is further expaned
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
4 Precision and Recall Let R be the set of relevant documents, and | R | be the number of documents in R for a given query Let A be the document answer set , and | A | be the number of documents in A Let R a = R A , and | R a | be the number of documents in R a Recall = , fraction of relevant documents retreived Precision = , fraction of retrieved docs that are relevant |R a | | R | |R a | | A | Collection Relevant Docs | R | Answer Set | A | Relevant Docs In Answer Set | R a |
Background image of page 4
5 Precision and Recall Example . Given the following evaluation, where prediction denotes the number of (non-)retrieved documents for finding all the spam documents by an IR system on a set of documents: Compute the Precision: Recall: False Positives (No. of retrieved docs should not be retrieved): False Negatives (No. of non-retrieved docs should be retrieved): d / ( b + d ) d / ( c + d ) b c Prediction by an IR System Non-Spam Spam Actual Non-Spam Spam a b c d Documents Retrieved Not Retrieved b + d a + c
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
6 Precision and Recall True/False Postives/Negatives Compute the Precision: Recall: Accuracy (Fraction of the correct classications): TP / ( TP + FP ) TP / ( TP + FN ) False Negatives ( FN ) Relevant Non-Relevant Retrieved Not Retrieved ( TP + TN ) / ( TP + FP + FN + TN ) True Positives ( TP ) True Negatives ( TN ) False Positives ( FP )
Background image of page 6
7 Precision and Recall Precision versus Recall Curve A visual display of the evaluation measure of a retrieval strategy such that documents are ranked accordingly Example . Let q be a query in the text reference collection Let R q = { d 3 , d 5 , d 9 , d 25 , d 39 , d 44 , d 56 , d 71 , d 89 , d 123 } be the set of relevant documents for q Assume the following ranking of retreived documents in the answer set of q : 1. d 123 6. d 9 11. d 38 2. d 84 7. d 511 12. d 48 3. d 56 8. d 129 13. d 250 4. d 6 9. d 187 14. d 113 5. d 8 10. d 25 15. d 3
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
8 Example . | R
Background image of page 8
Image of page 9
This is the end of the preview. Sign up to access the rest of the document.

This document was uploaded on 10/18/2011.

Page1 / 35

Lecture-C - Lecture C Retrieval Evaluation Introduction s...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online