Lecture CS630 9: End of Probabilistic Retrieval, Introduction to Relevance Feedback
Lecture by Lillian Lee Scribed by Randy Au, Nick Gerner, Blazej Kot February 23, 2006
In this lecture, we nish treating the basic retrieval paradigms, and move onto introducing the relevance feedback paradigm.
1
Meta-lessons
During the past several lectures, we have been trying to develop the following skills and important meta-tools to do mathematical modeling. These are tools with broad application within and outside of information retrieval. Bayes ip : Apply Bayes Rule when appropriate. (Re)-factorizations : Consider independent random variables and decomposing into products thereof. This rephrases the problem to hopefully eliminate terms to appropriately simplify a derivation. Insert R.V.s : Capture the system you are trying to model by inserting (mathematically appropriately) random variables Ground models : Incorporate semantic knowledge and scenarios to yield intuition and take reasonable next steps (sometimes making appropriate assumptions). Aesthetic sense : Many choices during a derivation may seem arbitrary, but must be taken with care to move towards a desirable result. These choices rely on your intuition and aesthetic sense Check/challenge preconceptions : Important, interesting and valuable new results are often achieved by questioning previous assumptions and models (consider pivoted length normalization or the language modeling approach to IR ranking). Math formalisms : By formalizing a model mathematically, assumptions and preconceptions are made explicit, making the applicability of a technique clear and suggesting directions for future work. Also consider taking advantage of previous formalisms (see challenging preconceptions above) 1
2
Simple re-derivation of a probabilistic scoring function
TD D
Recall we have the topic models and independent query models TQ Q We can now dene what we mean by R = y to be that the topic model that generates a document is the same as the topic model which generates the query. Our scoring function now (relying on the above intuition of R = y) is: P (R = y|D = d, Q = q) Again relevance status would seem to be determined by d and q and so apparently there is no room for probabilities. To address this issue (which we have faced before in probabilistic models of information retrieval) we consider the contents of documents rather than the documents themselves. Here, two documents might have been generated by dierent topic models but have the same content. This gives us a binning intuition which we considered in the RSJ model to give a random choice. We can rewrite our scoring function as follows: P (R = y, TQ = t, TD = t |D = d, Q = q)
t,t
Now we want to move Q to the LHS, so use Bayes ip: P (Q = q|R = y, TQ = t, TD = t , D = d)P (R = y, TQ = t, TD = t |D = d)
t,t
P (Q = q|D = d) Note that the P (R = y, TQ = t, TD = t |D = d) term is zero if t = t since our interpretation of relevance is that these two topic models must be the same (equal the same t). Also, by the independence of the query and document the denominator term becomes P (Q = q), which is constant for all documents, so it can be ignored under rank. This now gives: P (Q = q|R = y, TQ = t, TD = t, D = d)P (R = y, TQ = t, TD = t|D = d)
t
Firstly, notice R = y is in both cases implied because of TQ = t, TD = t - that is our very denition of relevance. Therefore, we can drop this term from both parts of the equation. Secondly, notice Q = q does not depend on the document choice at all - therefore we can remove TD = t and D = d. Together, this now gives: 2
P (Q = q|TQ = t)P (TQ = t, TD = t|D = d)
t
Now, notice that in the right-most term TQ = t is independent of D = d. However, we have this annoying term TD = t. Note that from basic probability, this term can be re-written in a split form: P (TQ = t|TD = t, D = d)P (TD = t|D = d) Plugging this back into our scoring function gives: P (Q = q|TQ = t)P (TQ = t)P (TD = t|D = d)
t
In practice, we assume t (d) is an MLE/MAP for the topic model of d and that P (TD = t (d)|D = d) = 1 removing the necessity to sum over all possible topic models for d. This gives: P (Q = q|TQ = t (d))P (TQ = t (d)) Note that this model has a problem in that it looks for exact matching between the the topic model for the document and the topic model for the query. We might address this with some notion of scoring by the distance between the models to allow for partial matching.
3
3.1
Relevance Feedback (prelude)
Introduction to Relevance Feedback
In our coverage of classic probabilistic information retrieval we had to overcome the issue of missing relevance information: we always needed some way to gather statistics conditioned on the (hidden) value of R. Now we will consider the case when some of this relevance information is available. Vector space models have no a priori method of incorporating this information and rely on further ad hoc methods. The probabilistic retrieval framework depends on some probabilistic scoring function that can (and does) R include = y explicitly.
3.2
Relevance Feedback Intuition
For a given query we assume that some relevance-labeled documents are available. This may seem counter-intuitive (this assumes that the user chooses relevant documents rather than the system!) Here are some possible scenarios in which the availability of such information is plausible: The user is very interested in recall for novelty verication (paper publishing or plagiarism detection) 3
Mitigate sample bias: to use a returned set of relevant documents as a random sample we must account for ranking bias If a system is giving no relevant documents, but some partially relevant documents, feedback can improve results and so the user might be willing to provide it. This may arise if the system was unable to interpret the query (because of user or system error is left to the readers judgement)
4
Questions
The scoring function derived in section 2 results in the same function as derived in the previous lecture. Specically we can use the same multinomial estimation model for document and query generation. Recall an important parameter of this model: . Lets consider how these two derivations dier qualitatively by exploring this parameter. Imagine a world in which we have the following corpus. Weve included topic models which can generate the documents; however, notice how we have not been concrete in dening the topic models. They are, for your reference in parameter decisions and to help conceptualize the corpus, as follows: t1 embodies the idea of dogs hate cats which generates d1 = dogs chase cats d2 = cats f ear dogs d3 = dogs hate cats and t2 embodies the idea of dogs love cats which generates d4 = dogs chase cats d5 = dogs like cats d6 = cats hate dogs We now have a case of 1 unique document per topic, and 2 shared documents between the two that are semantically dierent but the same when treated as a bag of words. The entire vocabulary space is 6 words: V = dogs, cats, chase, f ear, like, hate
4
4.1
Question 1
In your own words describe the function of the parameter . (a) Quantitatively (but in words), for the scoring function used in both derivations, compare the ranking function as varies from zero. (b) Qualitatively, what is the meaning of this parameter for this derivation with respect to topic model document generation? Perhaps reference the corpus in question (above).
4.2
Question 2
Now that you have an idea of the function of the parameter , what might be one good choice for this parameter? Provide a value and a short justication with respect to the corpus in question (above).
4.3
Question 3
Now, suppose some user out there has the information need t3 embodies the idea of what do dogs hate which generates the query q = dogs hate Calculate the rank of each document using your choice of from before.
5
5.1
Answer
Question 1
(a) is a smoothing parameter. Conceptually it changes emphasis from the relative-frequency distribution in the document to the observed relative-frequency distribution in the corpu...