a Choosing Parser for Anaphora Resolution
Judita Preiss
Computer Laboratory JJ Thomson Avenue Cambridge CB3 0FD United Kingdom [e-mail] Abstract
We investigate the performance changes in the Lappin and Leass (1994) anaphora resolution algorithm when we switch the parser. We compare anaphora resolution results obtained using the Briscoe and Carroll (1993) probabilistic LR parser, the maximum entropy based Charniak (2000) parser, and two versions of the Collins (1997) parser.
1. Introduction
The Lappin and Leass (1994) anaphora resolution algorithm requires its input to be fully parsed, and its performance is thus affected by which parser is chosen to do this preprocessing step. In this paper, we investigate to what extent the performance varies by switching between the Briscoe and Carroll (1993) probabilistic LR parser,1 the Charniak (2000) parser2 and two versions of the Collins (1997) parser. 3 Anaphora resolution algorithms rarely use the same tools, often they are not even evaluated on the same anaphorically resolved corpus. These factors mean that it is usually difcult to compare performances of anaphora resolution algorithms. For example, consider the results of the Kennedy and Boguraev algorithm in the authors paper (Kennedy and Boguraev, 1996), in a paper by Barbu and Mitkov (2001) and in our previous work (Preiss, 2002). We use the four parsers in their entirety, including the recommended taggers etc. wherever this is appropriate. Thus we end up with a comparison involving complete changes in the parser component, but all other components of the anaphora resolution algorithm stay constant. In Section 2. we describe the Lappin and Leass anaphora resolution algorithm. Section 3. outlines the four parsers, with Section 4. introducing our grammatical role extraction system. An evaluation, including a description of the evaluation corpus can be found in
This work was supported by UK EPSRC project GR/N364 62/93: Robust Accurate Statistical Parsing. 1 Available from http://www.cogs.susx.ac. uk/lab/nlp/rasp/ 2 Available from http://www.cs.brown.edu/ people/ec/ 3 Available from http://www.cis.upenn.edu/ mcollins/home.html
Factor Sentence recency Subject emphasis Existential emphasis Accusative emphasis Indirect object/oblique Head noun emphasis Non-adverbial emphasis
Weight 100 80 70 50 40 80 50
Table 1: Salience weights Section 5. We draw our conclusions in Section 6.
2. Lappin and Leass (1994) Anaphora Resolution Algorithm
We re-implement a non-probabilistic algorithm due to Lappin and Leass (1994). For each pronoun, this algorithm uses syntactic criteria to rule out noun phrases that cannot possibly corefer with it. An antecedent is then chosen according to a ranking based on salience weights. For all pronouns, noun phrases cannot corefer if they have incompatible agreement features. Pronouns are split into two classes, lexical (reexives and reciprocals) and non-lexical anaphors. There are additional syntactic lters for both of the two types of anaphors. Candidates which remain after ltering are ranked according to their salience. A salience value corresponding to a weighted sum of the relevant feature weights (summarized in Table 1) is computed. If we consider the sentence John walks, the salience of John will be: sal John
sent subj head
non-adv
Parser Charniak Coll 1 Coll 2 Charniak Coll 1 Coll 2 B&C
LR LP 40 words WSJ 90.1 90.1 WSJ 87.52 87.92 WSJ 88.07 88.35 100 words WSJ 89.6 89.5 WSJ 87.01 87.41 WSJ 87.60 87.89 B&C evaluation Susanne 74.0 73.0
Corpus
CB 0.74 0.96 0.95 0.88 1.11 1.09 1.03
0CB 70.1 64.86 65.84 67.6 62.17 63.20 59.6
2CB 89.6 86.19 86.64 87.7 83.86 84.60
Table 2: Summary of Published Results The weights are scaled by a factor of where is the distance (number of sentences) of the candidate from the pronoun. We select the candidate with the highest salience to be the antecedent.
3.
Parser Descriptions
Tagger: comes with parser. Training corpus: sections 221 of the Penn Wall Street Journal (WSJ) tree-bank (Marcus et al., 1993). 3.3. Collins (1997)
In this experiment we use four state of the art probabilistic parsers, freely downloadable from the Internet. They were given the same tokenization for the test corpus4 and their output was post-processed into a consistent format. In the following sections we will list the parsers grammar, parsing algorithm, tagger and training corpus. This information is relevant to their use in the present anaphora resolution experiment, and will help when we interpret the results of the experiment. 3.1. Briscoe and Carroll (1993)
Grammar: unication-based grammar of part of speech and punctuation labels. Parsing algorithm: LR parser. Tagger: Acquillex HMM tagger (using the CLAWSII labels) (Elworthy, 1994). Training corpus: the Susanne corpus (Sampson, 1995).
However, Charniaks parser contains a tokenization component, so some words were further (wrongly) tokenized. For example, the text contained the word Sánchez instead of S nchez. The Charniak parser a created three words out of this: S&aacute, ;, and nchez. The tokenization of the words was also sometimes already pre-processed for the Briscoe and Carroll parser, for example so that is tokenized as so blank that which was used in the other three parsers too.
4
Available from http://www.cis.upenn.edu/ adwait/
3.2.
Charniak (2000)
Grammar: Generative parsing model using a Markov grammar. Parsing algorithm: Standard bottom-up best-rst chart parser.
Grammar: Both parsers are generative models of lexicalized context-free grammar. Parsing algorithm: CKY-style dynamic programming chart parser. Tagger: Ratnaparkhi (1996).5 Training corpus: sections 221 of the Penn Treebank. For our experiment we used two versions of the Collins parser, with different parsing models: Collins 1: a sequence of decisions is used to generate the parse tree in a top-down fashion. Collins 2: extends Collins 1 by introducing subcategorization frames, adjunct/complement distinction.
5
Sentence: John gave Mary the book. Full parse: (|T/txt-sc1/---| (|S/np_vp| (|NP/n1_name/-| (|N1/n| |John:1_NP1|)) (|V/np_np| |give+ed:2_VVD| (|NP/n1_name/-| (|N1/n| |Mary:3_NP1|)) (|NP/det_n| |the:4_AT| (|N1/n| |book:5_NN1|))))) Grammatical relations: (|ncsubj| |give+ed:2:2_VVD| |John:1:1_NP1| _) (|dobj| |give+ed:2:2_VVD| |Mary:3:3_NP1| _) (|obj2| |give+ed:2:2_VVD| |book:5:5_NN1|) (|detmod| _ |book:5:5_NN1| |the:4:4_AT|)
Figure 1: Briscoe and Carroll output for John gave Mary the book Sentence: John gave Mary the book Full parse: (S1 (S (NP (NNP John)) (VP (VBD gave) (NP (NNP Mary)) (NP (DT the) (NN book))) (. .)))
Figure 2: Charniak output for John gave Mary the book 3.4. Summary of Published Results Performance results, taken from the authors original papers, are presented in Table 2. We present the following information: Labelled Precision (LP) no. of correct constituents in proposed parse number of constituents in proposed parse Labelled Recall (LR) no. of correct constituents in proposed parse number of constituents in treebank parse Crossing Brackets (CB) = number of constituents which violate constituent boundaries with a constituent in the treebank parse The Charniak and Collins parsers output Penn Treebank style trees, and are trained on sections 2 21 of the Penn Wall Street Journal treebank. The reported results are found from evaluating on section 23. The Briscoe and Carroll parser does not output a Penn Treebank style tree, and is therefore evaluated on 250 sentences bracketed in the Susanne style. dates grammatical roles. In this section, we explain how we extracted this grammatical information from the output of the parsers we used. Dependency information of this type is already provided by the Briscoe and Carroll parser, which for example for the sentence John gave Mary the book produces the output shown in Figure 1. However, the same information is not available for the Charniak or Collins parsers. The parse for the same sentence (obtained from Charniaks parser) is shown in Figure 2. We therefore a program which identies the necessary grammatical roles in a Penn Treebank style parse by walking over the constructed tree and applying a set of hand-crafted rules. As distinction the between a subject and object is often the difference between choosing the correct candidate or not, we required our system to have high precision. We present a comparison of the grammatical roles extracted by our program to those extracted by the Briscoe and Carroll parser. The extraction software was evaluated on the 500 sentence grammatical relation evaluation corpus described in the work of Carroll et al. (1999). 6 The
6
4.
Grammatical Role Extraction
The Lappin and Leass anaphora resolution algorithm requires information about the suggested candi-
This
evaluation
corpus
is
available
from
subj
dobj
inobj
precision (%) recall (%) F-measure precision (%) recall (%) F-measure precision (%) recall (%) F-measure
B&C 84 88 0.86 86 84 0.85 39 84 0.53
Charniak 91 85 0.88 82 67 0.74 60 32 0.41
Collins 1 89 80 0.84 83 62 0.71 50 32 0.39
Collins 2 90 83 0.86 83 55 0.66 50 32 0.39
Table 3: Grammatical Relation Evaluation
results of this comparison are presented in Table 3.
5. Evaluation
5.1. Evaluation Corpus The lack of anaphorically resolved corpora (reported by (Mitkov, 1999)) led us to manually annotate our own. We chose the initial 2400 sentences of the BNC, which we split into ve roughly equal sized texts for a 5-fold cross validation. The full details of the annotation can be found in our earlier work (Preiss, 2002). 5.2. Results We present the results of the anaphora resolution algorithm experiment in Table 4. The rst ve lines tell us the number of sentences, pronouns and the performance results in each of the ve cross-validation texts. The table also includes the average performance and variance values. The baseline column describes the most recent subject baseline performance with the Briscoe and Carroll parser. We also carried out a one-tailed -test comparison of the performances, presented in Table 5. We illustrate how to interpret the table using the last line: 1 Collins 2 2 3 70% 4 70%
5
This states that the anaphora resolution algorithm using Collins 2 does not outperform the Briscoe and Carroll parser (columns 2 and 5). However, it outperforms both the Charniak and Collins 1 parser components with condence 70% (columns 3 and 4). 5.3. Interpretation of the Results The results obtained are surprising: intuitively, we would expect the performances of the anaphora resolution algorithm to mirror the performances of the
http://www.cogs.susx.ac.uk/lab/nlp/ carroll/greval.html
parsers (see Table 2). In particular, we would expect the performance with the Charniak parser to be better than the performance with the Collins parsers. In this section, we suggest some possible reasons for the difference. Already in Section 4., we can see that the performance of our complement to verb dependency results are lower than those reported by Collins in his thesis (Collins, 1999). He obtains 93.76% precision with 92.96% recall on the Penn Treebank. Now adding the observation that the performance of the anaphora resolution algorithm with various parsers does not match the parsers relative performance, suggests to us that we may be dealing with what Gildea (2001) describes as strong corpus effects with parsers. Due to being trained and tested on different corpora (and especially due to the fact that the test corpus is not as nicely uniform as the WSJ), we may be observing deviations from the cited parser performance. It is however very interesting to note that the differences in performance of the anaphora resolution algorithm are not great. The change in performance between using the Briscoe and Carroll parser versus using the Charniak parser is only three percent. This may imply that it is difcult for different parsers to introduce much new information that would benet the Lappin and Leass anaphora resolution algorithm. The addition of adjunct information generated during parsing (as is the case in the Collins 2 parser), improves the performance of the anaphora resolution system only by one percent. The more detailed structure of the Briscoe and Carroll parse also does not yield great improvements to the accuracy of the anaphora resolution system. It might be argued that we are not using the parsers in an optimal manner and if they were used differently we may obtain much better results with anaphora resolution. The following three problematic areas were noted during development:
File 1 2 3 4 5 Average Variance
Sents 495 489 487 482 480 487 28
Prons 93 44 54 198 224 123 5545
B&C 70% 61% 61% 64% 63% 64% 10
Collins 2 68% 66% 57% 70% 55% 63% 34
Collins 1 71% 55% 61% 66% 55% 62% 41
Charniak 68% 57% 61% 62% 58% 61% 14
Baseline 31% 49% 45% 44% 36% 41 43
Table 4: Percentage Accuracy with Various Parsers B&C Collins 2 60% Collins 1 80% 70% Charniak 97.5% 70% 55%
B&C Collins 2 Collins 1 Charniak
Table 5: One Tailed -test Signicance Results 1. As noted in Footnote 3., Charniaks algorithm may be decreasing its own performance b...