Redundancy Exploiting in Question Answering
Summary
Submitted by: Kuljit Singh Lesley Ponneri Phebu George Pradeep Nayar
Introduction The World Wide Web today is so vast that it has some data to answer almost every question we search for. The main aim of the paper is to find concise answers for short Questions and Exploiting Redundancy in Question Answering . Example: Question Who is the President of the US?
Answer
George Bush.
Question Answers (QA) has recently received attention from both Information Retrieval (IR) and Information Extraction (IE) communities, both as an extension to their traditional interests and as an area of intersection between the two. In contrasts with traditional IR tasks, it is not acceptable for a QA system to retrieve a full document, or even a paragraph or a large text fragment, in response to a question. In contrast to the traditional IE tasks, no pre specified domain restrictions are placed on the questions, which may be of any time and deal with any topic.
1. QUESTION ANSWERING The Figure below provides a simplified overview of our QA system
The basic process is as follows- First, a parser analyzes a submitted question to extract two types of information: a query which can be submitted to a passage retrieval component and a set of selection rules that guide the process of extracting answers from the passages. The passage retrieval component runs the submitted
query over the target corpus, and retrieves a ranked list of the top k passages for further analysis by the answer selection component. The selection rules include an answer category (\person", \monetary value", \date", etc.) that identifies the general type of information the question asks. The answer selection component identifies possible answers ("candidates") from the passages and then ranks these candidates using a variety of methods to select the best possible solution. This considers the number of times each candidate answer appears in the retrieved passages, the location of the candidate in the retrieved passages, the rank of the passages in which the candidate appears, the likelihood that the candidate matches the assigned answer category, and other special-case information provided by the selection rules. Question Answering (QA) is a combination of the Information Retrieval (IR) and Information Extraction (IE) communities. A typical QA system looks for precise answers to short factual questions, where a large target corpus is used as the source for these answers. Answers may be of the form of values, names, phrases, sentences, or brief text fragments. We don't want a QA system to retrieve a full document, or even a paragraph or large text fragment, in response to a question. In contrast with traditional IE tasks, no pre-specified domain restrictions are placed on the questions
The TREC conference series, which is sponsored annually by the U.S. National Institute of Standards and Technology (NIST) [22], over the past two years, has worked on QA. For TREC-9, the most recent conference, the task consisted of 682 questions posed
over a 3GB target corpus comprised of newspaper articles, where an answer for each question was guaranteed to appear in the corpus [21]. The answers were of the form of text fragments extracted from the target corpus. For each experimental run, five attempts were given to answer each question. Each attempt was ranked, and the primary evaluation measure was based on the rank of the first fragment containing a correct answer. Two types of experimental runs were permitted, which differed only in the maximum allowed length of the text fragment, which was either 50 or 250 bytes.
There are three specific features that influence greatly the performance of our system. One is the flexibility of the parser, which makes reasonable choices in response to unexpected question types and formats. The second is the basic passage retrieval strategy used, which retrieves high-quality passages for further analysis. The third is the use of candidate redundancy in selecting answers. Redundancy is the combination of evidence from multiple passages to identify the most likely answer. If a candidate answer appears in several of the top ranking passages, it is a strong indication of the importance of a candidate. This "exploitation of redundancy in question answering" is looked into here. We also look into some details of our passage retrieval technique and also the role of redundancy in our TREC-9 experiments and discusses its contribution to the performance of our QA system. Then we examine the impact of redundancy in a broader context by removing the guarantee that the answer appears in the corpus and by requiring an exact answer, rather than a text fragment containing the answer. Further we use trivia questions drawn from
a popular game show to test, the value of redundancy. The paper concludes with an overview of additional related research and a discussion of future work.
2. PASSAGE RETRIEVAL FOR QA Different document-oriented retrieval techniques are used in retrieval where Paragraphs, sentences or n-word segments are treated as separate documents. This is not necessarily the bestsuited method for QA. Text fragments can often supply the answer to a question even when the topic of the question is incident to the topic of the document or paragraph containing the fragment. For example, evidence supporting \ James Boswell" as the answer to the question \Who was Samuel Johnson's friend and biographer?" is supplied by the text fragment \...self-appointed cultural commentator Stanley Crouch played Boswell to Marsalis's Samuel Johnson..." although jazz music is the primary concern of the document from which it is drawn. We look into such trivia questions later on. For QA we use a passage retrieval technique that does not depend on predefined passages, but can retrieve any document sub string in the target corpus. This was used both in TREC-8 and TREC-9. Here we discuss why we use this technique and dwell upon its efficient implementation. Algorithm Each document D in the corpus is treated as an ordered sequence of terms: D = d1, d2, d3...dm.
A passage from D is represented by an extent (u, v), an ordered pair of coordinates with 1 u v m that corresponds to the subsequence of D beginning at position u and ending at position v du, du+1, du+2 ...d v . A query Q is generated from the original question and takes the form of a term set: Q = {q1, q2, q3 ...}. An extent (u, v) satisfies a term set T Q if the subsequence of D defined by the extent contains at least one occurrence of each of the terms from T. An extent (u, v) is a cover for T if (u, v) satisfies T and the subsequence corresponding to (u, v) contains no subsequence that also satisfies T. That is, there does not exist an extent (u 0, v 0) with either u < u v v or u u v < v that satisfies T . Given an extent (u, v) that is a cover for a term set T ={t1, t2 ...} we want to compute a score for the extent with respect to T with the probability that an answer to the question is contained in the extent or appears in its close proximity. First, we model a document as a sequence of independently generated terms and assume that there is a fixed probability pt of a term t T matching at any given document location. This assumption allows multiple terms from T to match at a particular location. Given an extent (u; v) with length l = v-u + 1, the probability P (t, l) that the extent contains one or more occurrences of t is
P (t, l) = 1-(1-pt )l = 1-(1-l pt + O(pt 2)) l pt : The probability that an extent (u, v) contains all the terms from T is then P (T, l) = P (t, l) t T l pt t T = l | T | . pt t T Finally, we estimate p t from the term frequencies within the target corpus p t = f t / N, where f t is the total number of times t appears in the target corpus and N is the total length of all documents in the corpus. The score for an extent of length l containing the terms T Q is the selfinformation of P (T, l) log (N/f t ) - | T | log (l) ..................................eqn 1. t T The above Equation 1 assigns a higher score to a passage whose probability of occurrence is lower. While a higher score does not directly imply a greater likelihood that the answer will appear in close proximity, empirical evidence suggests that this relationship holds. The TREC-8 QA experiments were based on this passage retrieval technique. Instead of parsing the questions, the stop words were eliminated. Instead of answer selection, a 250byte fragment centered on each extent was submitted. Despite the simplicity of the technique, answers for 63% of the questions
appeared in the five fragments submitted and the system was among the six best performing systems overall. For the given query Q we generate all covers for all subsets of Q and rank them using equation 1. All but the highest-ranking passage from each document is discarded and the top k ones are used for further analysis. An extent (u, v) is said to i-satisfy a query Q if the subsequence of D defined by the extent contains exactly i distinct terms from Q. An extent (p, q) is an i-cover for Q if and only if it i-satisfies Q and does not contain a shorter extent that also i-satisfies Q. The set of covers for all subsets of Q is simply the union of the i-covers N U J i. t T
3. TREC 9 EXPERIENCE
An analysis of TREC-9 QA results revealed the importance of redundancy. Our experimental runs for TREC-9 used the overall approach outlined in Figure 1. For passage retrieval we, retrieved the top k = 10 passages and symmetrically expanded each about its center point C = (u + v)=2 to 200 terms for further analysis. The aim of the TREC-9 QA experiments was to select 50- and 250-byte answer fragments from the retrieved passages and not to find candidates that are exact or complete answers. Rather single terms, where the nature of these terms depends on the category of the
question. For example, if a question asks for a proper noun, the candidates consist of those terms that match a simple syntactic pattern for proper nouns; if a question asks for a length, the candidates consist of those numeric values that precede appropriate units; and if a question cannot be classified, the candidates simply consist of all non-query and non-stop word terms appearing in the retrieved passages. After identification, each candidate term t is assigned a weight that takes into account the number of distinct passages in which the term appears, as well as the relative frequency of the term in the database, w t = c t log (N/f t ) , where 1 c t k is the number of distinct passages in which t appears and represents the redundancy associated with the candidate. The weights of the candidates are used to select 50- and 250byte answer fragments from the retrieved passages. Each 50- or 250-byte fragment from the retrieved passages is considered as a possible answer. A score of each fragment was computed by summing the weights of the candidate terms that appear within it. Given a text fragment F and a set of candidates K the score for a fragment is wt t F ^ t K ..............................eqn 2.
This score is modified using a number of heuristics that consider the rank of the passage in which the fragment appeared, the location of the fragment relative to the center point of the passage, and the selection rules generated by the parser. These
heuristics had only a small (but positive) effect on the system's performance. The highest-scoring fragment is selected and the weights of the candidates appearing in that fragment are reduced to zero. All fragments are re-scored and the highest-scoring fragment is again selected. This process is repeated until five fragments have been selected. The weight assigned to a candidate combines a redundancy factor and a term-frequency factor. The individual contributions can be ascertained by setting one, or the other to a constant value and measuring the resulting impact on question answering performance. Figure 2 compares three different formulations for the weight w t using the TREC-9 QA test collection. The candidate weight w t = ct log(N/f t ) was used in the TREC-9 experiments. The weight w t = log (N=f t ) does not consider candidate redundancy, essentially as if all candidates appeared an equal number of times in the passages. The weight w t = c t treats all candidates as having the same term frequency and considers only redundancy. For comparison, the final row of the figure 2 represents results for the 50- or 250-byte fragment centered on each of the top five extents. Here we also show two effectiveness measures. Mean reciprocal rank (MRR) is the standard TREC effectiveness measure reported by NIST for each TREC QA run. To compute MRR, each question is assigned a score that is the inverse of the rank of the first fragment that is judged to contain a correct answer. If no fragment contains an answer, the question is assigned a score of zero. The average of these scores of the questions give the MRR value for the run. In addition to MRR, the figure shows the number and the percent of questions for which a correct answer was found in the
five fragments. An automatic judging script developed by Voorhees computed these measures. The value of redundancy is clear in the 50-byte runs. Eliminating the redundancy factor reduces the MRR by 38% and the number of questions answered correctly by 28%. Eliminating the term frequency factor has a lesser effect, reducing MRR by 12% and the number of correct questions by 10%. The raw passages have an MRR 50% lower and contain the answers to 40% fewer questions. In contrast, the raw 250-byte passages exhibit good performance, which is harmed by answer selection if either the redundancy or term frequency factors are used alone.
4. REDUNDANCY In the TREC experiments, text fragments were considered as answers. The experimental confounds associated with parsing, question categorization, and the use of selection rules were eliminated. A single category of question, that requires the name of a person as the answer was explored and redundancy was used as a means of isolating the required name from the top ranking passages. A simple syntactic pattern was used to identify candidate answers that match most names in written English, but which often matches other text, including many other proper nouns. We assumed that redundancy could be used as a substitute for further analysis; that we identify correct answers from other candidates solely by their repeated occurrence in high-ranking passages. 87 questions were used from the TREC-9 QA collection that required the name of a person as the answer. Typical examples are "Who is
the emperor of Japan?", "Name a female figure skater," and "Who wrote the book Huckleberry Finn?". For each question, a query was generated by eliminating stop words. A larger corpus, the TREC 100GB VLC2 corpus was used here. Since the QA corpus consists of newspaper articles and the VLC2 corpus of arbitrary Web pages, we might generally expect documents in the VLC2 corpus to be of lower quality. It is not necessary that answers to all questions be present in the VLC2 corpus. Here a relatively small collection of Web data was being used to answer a set of independently created questions. A simple syntactic pattern was used to identify candidate answers. Any sequence of at least two-name tokens separated by white space or a single hyphen was considered a candidate. A name token defined is as either an uppercase letter followed by one or more lowercase letters or an uppercase letter followed by a period (an initial). In addition, a capitalized stop word is not considered to be a name token, and a candidate answer cannot consist entirely of initials. This pattern rejected many valid names ("Wm. Paterson", "k. d. lang", "Samuel To", "Pius XI", "Anne of Cleaves", "Fabio") and included many inappropriate candidates. While the identification of proper names, including the names of people, is a wellunderstood problem it was assumed that the benefits of transparency would outweigh the benefits of more accurate candidate identification. Rather than find the answer using any and all means, the value of redundancy was examined, where the ability of redundancy to overcome the low precision inherent in this crude candidate-identification technique was looked into.
For each query the top k passages were retrieved according to Equation 1. Each passage was expanded symmetrically about its center point to w bytes. For the experiment the parameters k and w, the depth and width respectively were varied. Candidate answers were identified in the passages, and those, which consisted entirely of query terms, were eliminated. Each remaining candidate was assigned a score which is the count of the number of distinct passages in which the candidate appeared (ct ). Candidates were ranked according to this score, and ties were broken by applying a rule that takes into account the distance of each candidate from the center point of the passages, in which it appears. For each passage in which a candidate appears, the distance from the center point of the passage to the center point of the closest occurrence of the candidate is recorded. The average of these distances is used to break ties, with candidates having smaller averages ranked higher. Experiments with TREC-8 questions suggested that a depth of k = 50 and width of w = 1000 would produce reasonable results. Using these parameters, 49 (56%) of the 87 questions are answered correctly and for 34 (39%) a correct answer is ranked first. For example, the top five candidates for the question "Who is the emperor of Japan?" are: 1. Emperor Hirohito 2. World War 3. Emperor Akihito 4. Hong Kong 5. Prime Minister where the first and third candidates are considered correct. The mixture of invalid proper nouns and the inclusion of multiple valid answers are typical of the results. The top five candidates for the question "Name a female figure skater." are
1. Kristi Yamaguchi 2. Followup Name 3. Follow Ups 4. Tonya Harding 5. Nancy Kerrigan where the first, fourth and fifth candidates are considered correct. The second and third candidates are answers retrieved by the inappropriate inclusion of the term "name" in the query and appear to be generated by a series of related postings to a message board. Additional independent data might tend to reduce the rank of these candidates while increasing the rank of the remaining candidates. Finally, the top five candidates for the question "Who wrote the book, Huckleberry Finn?" are 1. Mark Twain 2. Tom Sawyer 3. Huck Finn 4. Samuel Clemens 5. Connecticut Yankee which includes the author's pen name and his real name, as well as the principal characters from the novel. Only exact answers identifying people were counted as correct. For example, the candidate "Anne Morrow Lindbergh Foundation" was not accepted as the answer to the question "Who was Charles Lindbergh's wife?", the candidate "Time Warner" was not accepted as the answer to "Who owns CNN?", and the candidate "Prime Minister" was not accepted as the answer to "Who is the leader of India?". The TREC-9 judging script was not designed to identify exact answers, and the determination of correct answers required a combination of automatic and manual processing. An automatic script based on the TREC-9 judging script analyzed the output of
each experimental run and potentially correct answers. The results of the script were manually checked to determine that the answers consisted solely of a person's name. Finally, the runs were automatically checked for consistency, verifying that answers were judged in the same way for all experimental runs.
These questions run for a range of depth and width values and their results are reported in Figure 3. In addition to the runs using redundancy the figure reports two other runs for comparison. Both runs represent other simple strategies for extracting candidate answers from the retrieved passages. The first, labeled "best 5 from top passage", consists of the five distinct candidates appearing closest to the center point of the top passage retrieved for each question, with the candidates ranked according their distance from the center point. This run is equivalent to using parameters k = 1 and w = 1. The second, labeled "best from top 5 passages" consists of a single candidate selected from each of the top five passages. For each passage, the candidate closest to the center
point is selected. If a candidate has already been selected from a higher-ranking passage, the next closest candidate is selected instead. Within the range shown, changes to k and w produce only minor changes in the results, with a slight tendency to improve as k and w are increased. This stability might be considered surprising given the range of data sizes represented by the retrieved passages. With k = 25 and w = 250, less than 7KB of data is analyzed per question; with k = 100 and w = 2000 over 195KB of data is analyzed per question.
A 100GB target corpus was chosen to increase the chances of finding multiple occurrences of answers in contexts that could be
retrieved by the unmodified question keywords. In order to ascertain the impact that database size has on the results the questions were run over various subsets of the full corpus. The documents in the database were ordered randomly and increasingly larger subsets of the database were used as the target corpus. The results are presented in Figure 4. The potential remains for additional improvement if the database size were further increased. For 76% of the questions, a correct answer was reported in the top five by at least one of the runs reported in this section, and additional data may force more of these correct answers into the top five ranks. Several of the questions required answers that the candidate selection pattern failed to match ("Dr. Suess", "Brutus", "William the Conqueror"). While the crude techniques used for query generation and candidate identification could easily be improved, the effects of these improvements would be largely orthogonal to the effects of redundancy and would be re in improvements to all runs. 5. ADDIT IO NAL WORK Trivia questions from the popular game show "Who wants to be a Millionaire?" have been used applying the approach of the previous section. These questions are presented in a multiple-choice format. For example, the question "Who was the first Prime Minister of Canada?" might be given with candidate answers A) Pierre Elliot Trudeau", B) John Graves Simcoe", C) John A. MacDonald" and D) Louis Riel".
The prize money awarded for correctly answering a question ranges from $100 to $1,000,000. The first question is valued at $100, and while the contestants continue to answer questions correctly the value roughly doubles until the final $1,000,000 question is reached. Contestants are eliminated if they answer a question incorrectly. They may also quit voluntarily before attempting to answer a question, keeping as prize money the value of the highest question correctly answered. Here there exists a closed set of candidate answers, which simplifies the voting process but complicates the passage retrieval process. Each question is treated as a "bag of words" for retrieval purposes and each candidate answer is matched as a phrase and the terms in the answer do not contribute to the score of a passage. The highest scoring passages (according to Equation 1) returned by the Retrieval Engine also contain a phrase match to one of the answers. For these questions the weights of answer terms will be higher for incorrect answers, since some of the choices are deliberately bizarre, and are intended as jokes or red herrings. The answers can be treated as phrases, since it is not necessary to compute weights for their terms and the phrasing is usually the "common" form appearing in everyday speech and writing. The top k = 20 passages were retrieved for each question. Since each passage is guaranteed to contain a match to at least one of the candidate answers, the passages were not expanded in width. A passage that contained a match for a candidate answer was counted as a vote for that candidate. Candidates were then ranked in terms of the number of votes each received. The 108 questions asked on the programs first broadcast on the ABC television network in the United States on October 31, November 1 and November 9, 2000 were evaluated. The results are shown in
Figure 5. The figure splits the que...