124.11.lec12 - CS 124/LINGUIST 180: From Languages to...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: CS 124/LINGUIST 180: From Languages to Information Dan Jurafsky Lecture 12: Ques4on Answering Thanks to Jim Martin and Mihai Surdeanu Question Answering   One of the oldest problem in NLP   early systems were working (on punched cards) in 1961   Two kinds of ques4on answering:   Simple (factoid) ques4ons   Who wrote the Declara-on of Independence?   What is the average age of the onset of au-sm?   Where is Apple Computer based?   Complex (narra4ve) ques4ons   What do scholars think about Jefferson’s posi-on on dealing with pirates?   What is a Hajj?   In children with an acute febrile illness, what is the efficacy of single medica-on therapy with acetaminophen or ibuprofen in reducing fever? 2 IR-Based (Corpus-based) Approaches   I’ll describe IR ­based approaches to both kinds   1.  2.  As opposed to knowledge ­based ones We assume an IR system with an index into documents that plausibly contain the answer(s) to likely ques4ons. And that the IR system can find plausibly relevant documents in that collec4on given the words in a users ques4on. 3 Corpus-Based Approaches   Factoid ques4ons   From a smallish collec4on of relevant documents   Extract the answer (or a snippet that contains the answer)   Complex ques4ons   From a smallish collec4on of relevant documents   Summarize those documents to address the ques4on   Query ­focused summariza4on 4 Factoid questions Factoid Q/A 6 Full-Blown System   QUESTION PROCESSING   Parse and analyze the ques4on   Formulate queries suitable for use with an IR system (search engine)   PASSAGE RETRIEVAL   Retrieve ranked results   Break into suitable units   ANSWER PROCESSING   Perform NLP on those   Rank snippets based on NLP processing Question Processing   Two tasks  Answer Type Detec4on   what kind of en4ty (person, place) is the answer?  Query Formula4on   what is the query to the IR system   We extract both of these from the ques4on  keywords for query to the IR system  an answer type 8 Factoid Q/A 9 Answer Type Detection   Who founded Virgin Airlines?   PERSON.   What Canadian city has the largest popula-on?   CITY. 10 Answer Types   Factoid ques4ons…   Who, where, when, how many…   Who ques4ons are going to be answered by…   Where ques4ons…   Simplest: use Named En44es Answer Type Taxonomy (from Li & Roth)   Two ­layered taxonomy   6 coarse classes   ABBEVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC_VALUE   50 fine classes   HUMAN: group, individual, 4tle, descrip4on   ENTITY: animal, body, color, currency…   LOCATION: city, country, mountain… 12 Answer Type Taxonomy (2/2) distance size speed money … weight date other NUMERI C … country LOCATION abbreviation ABBREVIATION expression DESCRIPTION definition city group … HUMAN manner individual … ENTITY reason description animal body color … vehicle 13 Answer Types 2/10/11 14 More Answer Types 2/10/11 15 Answer Type Detection   Hand ­wriden rules   Machine Learning   Hybrids:   Regular expression ­based rules can get some cases:   Who {is|was|are|were} PERSON   PERSON (YEAR – YEAR) Answer Type Detection   Regular expression ­based rules can get some cases:   Who {is|was|are|were} PERSON   PERSON (YEAR – YEAR)   Make use of answer ­type word: Answer Types   S4ll, regular expressions can get some cases:   Who {is|was|are|were} PERSON   PERSON (YEAR – YEAR)   Other rules use the ques-on headword:   = headword of first noun phrase ajer wh ­word:   Which city in China has the largest number of foreign financial companies?   What is the state flower of California? Answer Type Detection   Most ojen, we treat the problem as machine learning classifica4on  Define a taxonomy of ques4on types  Annotate training data for each ques4on type  Train classifiers for each ques4on class using a rich set of features.   these features can include those hand ­wriden rules! 19 Factoid Q/A Query Terms Extraction   Grab terms from the query that will help us find answer passages Question (from TREC QA track) keywords Q002: What was the monetary value of the Nobel Peace Prize in 1989? monetary, value, Nobel, Peace, Prize Q003: What does the Peugeot company manufacture? Peugeot, company, manufacture Q004: How much did Mercury spend on advertising in 1993? Mercury, spend, advertising, 1993 Q005: What is the name of the managing director of Apricot Computer? name, managing, director, Apricot, Computer Keyword Selection Algorithm Select all non ­stop words in quota4ons 10 Select all NNP words in recognized named en44es 9 Select all complex nominals with their adjec4val modifiers 8 Select all other complex nominals 7 Select all adjec4val modifiers 6 Select all other nouns 5 Select all verbs 4 Select all adverbs 3 Select the QFW word (which was skipped in all previous steps) 2 Select all other words 1 22 Walk-through Example Who coined the term “cyberspace” in his novel “Neuromancer”? 10 7 10 7 4 cyberspace/10 Neuromancer/10 term/7 novel/7 coined/4 23 Keyword Selection Examples   What researcher discovered the vaccine against Hepa44s ­B?   Hepa44s ­B, vaccine, discover, researcher   What is the name of the French oceanographer who owned Calypso?   Calypso, French, own, oceanographer   What U.S. government agency registers trademarks?   U.S., government, trademarks, register, agency   What is the capital of Kosovo?   Kosovo, capital 24 Passage Extraction Loop   Passage Extrac4on Component   Extracts passages that contain all selected keywords   Passage quality and keyword adjustment   In the first itera4on use the first 6 keyword selec4on heuris4cs   If the number of passages is lower than a threshold ⇒ query is too strict ⇒ drop a keyword   If the number of passages is higher than a threshold ⇒ query is too relaxed ⇒ add a keyword Factoid Q/A 26 Passage Processing   Output of IR:   Ranked Documents, according to simialrity with keywords   Problems:   Documents aren’t best unit for QA   Ranking for IR may not be appropriate for QA   So passage retrieval:   extracts and reranks shorter units from the returned set of documents 27 Passage Processing   Step 1: Segmenta4on   something like paragraphs   Step 2: Passage ranking   Use answer type to help rerank passages Passage Ranking   Number of Named En44es of the right type   Number of query words   Longest sequence of ques4on words   Rank of the document from which the passage was extracted   Query term proximity   N ­Gram overlap Named Entities   If we are looking for ques4ons whose Answer Type is CITY   We have to have a “CITY” named ­en4ty detector   So if we have a rich set of answer types   We also have to build answer ­detectors for each of those types! Factoid Q/A 31 Answer Extraction   Padern ­extrac4on methods:   Run the answer ­type tagger on the passages:   Return the string that is the right type:   Who is the prime minister of India” (PERSON)   Manmohan Singh, Prime Minister of India, had told leV leaders that the deal would not be renego-ated.   “How tall is Mt. Everest? (LENGTH)   The official height of Mount Everest is 29035 feet Answer Extraction   For answers that are not a par4cular named en4ty type:   Use regular expression paderns   These can be wriden by hand or learned automa4cally (as we’ll see next week when we talk about rela4on extrac4on) Answer Extraction   Some4mes padern ­extrac4on methods are insufficient   We can’t write rules   There is more than one poten4al answer in the passage Ranking Candidate Answers Q066: Name the first private citizen to fly in space.     Answer type: Person Text passage: “Among them was Christa McAuliffe, the first private citizen to fly in space. Karen Allen, best known for her starring role in “Raiders of the Lost Ark”, plays McAuliffe. Brian Kerwin is featured as shuttle pilot Mike Smith...” Ranking Candidate Answers Q066: Name the first private citizen to fly in space.     Answer type: Person Text passage: “Among them was Christa McAuliffe, the first private citizen to fly in space. Karen Allen, best known for her starring role in “Raiders of the Lost Ark”, plays McAuliffe. Brian Kerwin is featured as shuttle pilot Mike Smith...”   Best candidate answer: Christa McAuliffe Answer Extraction   In these cases we use machine learning to combine many rich features about which phrase is the answer Features for Answer Ranking 2/10/11 38 Evaluation   NIST has been running Q /A evalua4ons as part of it’s TREC program. Both generic Q /A and applica4on specific (bio ­ etc.).   Typical metric is Mean Reciprocal Rank.   Assumes that systems return a ranked list of N possible answers.   Your score is based on 1/Rank of the first right answer. 39 Mean Reciprocal Rank   What is the reciprocal rank for this ques4on?   Q: Who is the president of Stanford?   Ranked Candidate Answers:   John Etchemendy   Dmitry Medvedev   John Hennessy   Richard Saller Is the Web Different?   TREC and commercial applica4ons:   retrieval performed against small closed collec4on of texts.   The diversity/crea4vity in how people express themselves necessitates all that work to bring the ques4on and the answer texts together.   But… The Web is Different   On the Web popular factoids are likely to be expressed in a gazzilion different ways.   At least a few of which will likely match the way the ques4on was asked.   So why not just grep (or agrep) the Web using all or pieces of the original ques4on. AskMSR   Process the ques4on by…   Simple rewrite rules to rewri4ng the original ques4on into a statement   Involves detec4ng the answer type   Get some results   Extract answers of the right type based on   How ojen they occur AskMSR Step 1: Rewrite the questions   Intui4on: The user’s ques4on is ojen syntac4cally quite close to sentences that contain the answer   Where is the Louvre Museum located?   The Louvre Museum is located in Paris   Who created the character of Scrooge?   Charles Dickens created the character of Scrooge. Query rewriting Classify ques4on into seven categories       Who is/was/are/were…? When is/did/will/are/were …? Where is/are/were …? a. Hand ­crajed category ­specific transforma4on rules e.g.: For where ques4ons, move ‘is’ to all possible loca4ons Look to the right of the query terms for the answer. “Where is the Louvre Museum located?” → “is the Louvre Museum located” → “the is Louvre Museum located” → “the Louvre is Museum located” → “the Louvre Museum is located” → “the Louvre Museum located is” Step 2: Query search engine   Send all rewrites to a Web search engine   Retrieve top N answers (100 ­200)   For speed, rely just on search engine’s “snippets”, not the full text of the actual document Step 3: Gathering N-Grams   Enumerate all N ­grams (N=1,2,3) in all retrieved snippets   Weight of an n ­gram: occurrence count, each weighted by “reliability” (weight) of rewrite rule that fetched the document   Example: “Who created the character of Scrooge?” Dickens Christmas Carol Charles Dickens Disney Carl Banks A Christmas Christmas Carol Uncle 117 78 75 72 54 41 45 31 Step 4: Filtering N-Grams   Each ques4on type is associated with one or more “data ­type filters” = regular expressions for answer types   Boost score of n ­grams that match the expected answer type.   Lower score of n ­grams that don’t match.   For example   The filter for   How many dogs pull a sled in the Iditarod?   prefers a number   So disprefer candidate n ­grams like   Dog race, run, Alaskan, dog racing   Prefer canddiate n ­grams like   Pool of 16 dogs Step 5: Tiling the Answers Scores 20 Charles Dickens 15 10 Dickens merged, discard old n-grams Mr Charles Score 45 Mr Charles Dickens Harder Questions   Factoid ques4on answering is really predy silly.   A more interes4ng task is one where the answers are fluid and depend on the fusion of material from disparate texts over 4me.   Who is Condoleezza Rice?   Who is Stephen Harper?   Why did San Francisco have to hand ­count ballots in the last elec4on? Query-Focused Summarization   What is water spinach?   A: Water spinach (ipomoea aqua4ca) is a semi ­aqua4c leafy green plant characterized by long hollow stems and spear ­shaped or heart ­shaped leaves which is widely grown throughout Asia as a leaf vegetable. The leaves and stems are ojen eaten s4r ­fried as greens with salt or salty sauces, or in soups. Other common names include morning glory vegetable, kangkong (Malay), rau muong (Vietnamese), ong choi (Cantonese), and kong xin cai (Mandarin). It is not related to spinach, but is closely related to sweet potato and convolvulus. Query-Focused Summarization   Q: In children with an acute febrile illness, what is the efficacy of single medica4on therapy with acetaminophen or ibuprofen in reducing fever?   A: Ibuprofen provided greater temperature decrement and longer dura4on of an4pyresis than acetaminophen when the two drugs were administered in approximately equal doses. (PubMedID: 1621668, Evidence Strength: A) How to do query-focused summarization   The Snippet method   Use cosine and y ­idf to find sentences and rank them, as we discussed last week   The informa4on extrac4on method   build specific answerers for different ques4on types:   defini4on ques4ons,   biography ques4ons,   certain medical ques4ons The Information Extraction method   a good biography of a person contains:   a person’s birth/death, fame factor, educa-on, na-onality and so on   a good defini-on contains:   genus or hypernym   The Hajj is a type of ritual   a medical answer about a drug for a disease:   the problem (the medical condi4on),   the interven-on (the drug or procedure), and   the outcome (the result of the study). Information that should be in the answer for 3 kinds of questions Architecture for complex question answering: definition questions R1C.(),4)(C&)D.EESR T)U)5V)))W)U)X !"#$%&'() *&(+,&-./ !"!#$%&'()*+,-,+#.*-,&,/,%&01#2*&/*&+*2 00)1&2)3"#$%&'(4 0056)("(./) 4&'(&'#&4 7+&3,#.(&) 83&'(,9,#.(,"' 3#4*&52'()*+,*2#(*&/*&+*2 BC&)D.EE>)"+)?,/@+,%.@&)(")F.GG.C)HF&##.I>),4)(C&)#&'(+./)3$(<)"9)84/.%J BC&)D.EE),4).)%,/&4("'&)&-&'(),').)F$4/,%K4)/,9&J BC&)C.EE),4)"'&)"9)9,-&)?,//.+4)(C.()%.G&)$?)(C&)9"$'3.(,"')"9)84/.%J JJJ BC&)D.EE>)"+)?,/@+,%.@&)(")F.GG.C)LF&##.M>),4)(C&)#&'(+./)3$(<)"9)84/.%J)F"+&)(C.')(N")%,//,"')F$4/,%4).+&)&O?&#(&3)(")(.G&) (C&)D.EE)(C,4)<&.+J)F$4/,%4)%$4()?&+9"+%)(C&)C.EE).()/&.4()"'#&),')(C&,+)/,9&(,%&),9)?C<4,#.//<).'3)9,'.'#,.//<).2/&J)BC&)D.EE),4).) %,/&4("'&)&-&'(),').)F$4/,%K4)/,9&J)BC&).''$./)C.EE)2&@,'4),')(C&)(N&/9(C)%"'(C)"9)(C&)84/.%,#)<&.+)HNC,#C),4)/$'.+>)'"()4"/.+>) 4")(C.()C.EE).'3)*.%.3.')9.//)4"%&(,%&4),')4$%%&+>)4"%&(,%&4),')N,'(&+IJ)BC&)D.EE),4).)N&&G:/"'@)?,/@+,%.@&)(C.()2&@,'4) ,')(C&)05(C)%"'(C)"9)(C&)84/.%,#)/$'.+)#./&'3.+J);'"(C&+)#&+&%"'<>)NC,#C)N.4)'"()#"''&#(&3)N,(C)(C&)+,(&4)"9)(C&)P.K2.) 2&9"+&)(C&)+,4&)"9)84/.%>),4)(C&)D.EE>)(C&).''$./)?,/@+,%.@&)(")K;+.9.(>).2"$()(N")%,/&4)&.4()"9)F&##.>)("N.+3)F,'.Q !.(.:!+,-&' );'./<4,4 =&'(&'#&)#/$4(&+4>) 8%?"+(.'#&)"+3&+,'@ !&9,',(,"' A+&.(,"' Summary   factoid ques4on answering   answer type detec4on   query formula4on   passage retrieval   passage ranking   answer extrac4on   web ­based factoid ques4on answering   AskMSR system   complex ques4on answering   query ­focused summariza4on ...
View Full Document

This document was uploaded on 06/01/2011.

Ask a homework question - tutors are online