lecture8-evaluation-handout-6-per

Lecture8-evaluation-handout-6-per

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: que of pure relevance Can we avoid human judgment?   Relevance vs Marginal Relevance Sec. 8.6.3   No   Makes experimental work hard         A document can be redundant even if it is highly relevant Duplicates The same informa)on from different sources Marginal relevance is a beUer measure of u)lity for the user.   Especially on a large scale   In some very specific seZngs, can use proxies   Using facts/en))es as evalua)on units more directly measures true relevance.   But harder to create evalua)on set   See Carbonell reference 35   E.g.: for approximate vector space retrieval, we can compare the cosine distance closeness of the closest docs to those found by an approximate retrieval algorithm   But once we have test collec)ons, we can reuse them (so long as we don’t overtrain too badly) 36 6 Introduc)on to Informa)on Retrieval Sec. 8.6.3 Introduc)on to Informa)on Retrieval Evalua)on at large search engines A/B tes)ng   Search engines have test collec)ons of queries and hand ­ranked results   Recall is difficult to measure on the web   Search engines o|en use precision at top k, e.g., k = 10   . . . or measures that reward you more for geZng rank 1 right than for geZng...
View Full Document

This document was uploaded on 02/26/2014.

Ask a homework question - tutors are online