ALEMBIC
MITRE-Bedford : MUC-4 Test Results and Analysi s
John Aberdeen, John Burger, Dennis Connolly, Susan Roberts, & Marc Vilai n aberdeen john decon @ mitre. org suzi mbv
The MITRE Corporatio n 202 Burlington Road Bedford, MA 01730 PRELIMINARIE S
This note embodies our analyses of the performance of the ALEMBIC system in the MUC-4 evaluation task . These analyses have provided us with a reasonably good understanding of the principal factors contributing to the system's correct responses and to its errors . This understanding is based in part on interpretations of the performance measures provided by the MUC-4 scoring software; in addition, we performed a number of qualitative and quantitative investigations into linguistic aspects of the messages that underly the system ' s performance . It should be noted however, that ALEMBIC iS still in very early stages of development, and that the analyse s we give here should be taken as just presenting a snapshot of the system's performance. In the weeks since the MUC-4 evaluation runs, the system has of course remained under development, and its performance score s have improved steadily . One consequence of our system's relative youth is that it embodies many oportunitie s for improvement, and even minor implementational tweaks can yield significant performance gains .
OVERALL PERFORMANCE MEASURE S
Looking first at our system's overall performance, the following table reproduces the f-measures for our run s on TST3 and TST4 .
P&R TST3 T5T4 9 .6 13.75 2P&R 8 .57 11 .22 P&2 R 10.91 17.74
Table 1 : Overall f-score s These are clearly fairly humble scores, but we offer them for consideration with a certain measure of pride . They represent the very first results of a text understanding project that was barely begun six months prior to the evaluation runs, as fielded by a group that had no prior experience with the MUC data extraction task . As one might note, raw f-measures are a fairly coarse performance statistic ; despite this, some trends are apparent . First, our system seems better at recall than at precision, an issue which we will address below . Second, the system scored uniformly better on TST4 than on TST3, which is in line with the general observatio n among MUC-4 participants that TST4 is the easier of the two test sets.
116
RECALL MEASURE S
Turning to a more detailed analysis of our recall measures, the principal determinant of our overall recal l performance is rather clear. That is, we only attempted to fill about half of all possible template slots, thos e for the incident and perpetrator. Although we had slot-filling strategies prepared for the remaining slots, the y were never incorporated into the system prior to the evaluation runswe just simply ran out of time to do so . It is illuminating, however, to consider ALEMBIC'S performance on the slots that the system actually did fill . The more interesting of these slots are those that the system filled by meaningfully extracting information fro m the free text ; their slotwise recall scores are shown in Table 2 below . Slots that do not appear in the tabl e were simply not filled at all, or were only filled by default strategies (more on this later) . T5T3
inc-date inc-loc inc-type inc-instr-id inc-instr-type perp-ind-id perp-org-id 3 23 10 30
T5T4
35 16 53
27
16
22
40
48
40
Table 2 : Recall scores on meaningfully filled slots A quick glance at the table reveals that our scores ranged fairly widely . On string fills, ALEMBIC obtained scores ranging for TST3 from 8 (instrument ID) to 40 (perpetrator organization ID) ; for set fills the range was 3 (instrument type) to 30 (incident type) . Similar patterns held for TST4, but with higher individual slot scores , reflecting the fact that this was the easier of the two test sets . As an estimate of the average recall for th e slots in Table 2 , we calculated a restricted overall recall score (based only on these slots) of approximatel y 20 for TST3 and 34 for TST4. 1 On a slot-by-slot basis, the following qualitative observations apply . Incident date : We derived this slot from the free text, and only used the dateline as a last recourse in case we failed to identify any date phrases . The date grammar we used for MUC-4 treats date phrases as functors, which were often left unattached due to the fragmentar y nature of our parses . This made it harder to actually locate temporal phrases when they did not appear as modifiers of events, resulting in a fair number of invocations of the heuristi c fallback strategy of using the dateline . Incident location : Recall errors for this slot were due in good part to locational modifiers no t being attached to events, as well as to a number of infelicities in the locational knowledg e representation . Among the more amusing: the lexical item Bogota maps to a number of possible locations, but the one that was picked by default was the Bogota Air Force Base . Incident type : This was our most accurate set-fill slot . ALEMBIC derives the filler of this slot from the heads of violent events ; missing cases are due in part to gaps in the lexicon .
'Not too much should be made of these scores . They admittedly exclude slots that are easy to fill using default values , but they also don't include slots that are hard to fill, i .e., the target slots .
117
Incident instrument in : We expected to get better recall scores for this slot. Eyeballing the actual fillers that ALEMBIC produced, part of the problem was grammatical incompleteness . For example, "a powerful dynamite charge" ended up only being parsed as "a powerfu l dynamite," due to a grammar bug involving noun-noun modification . Since we attempted to use full noun phrases to fill string slots, we ended up being penalized for cases where we ha d nearly parsed the complete instrument phrase, but where our fragmentary filler failed to b e matched by the scoring program . Incident instrument type : This slot was only filled when an instrument ID filler was obtained . We never implemented implicit fills for this slot, i .e ., fills that could be derived from verb s such as shoot even if no gun is ever mentioned . Perpetrator individual ID : As mentioned in the system overview, our strategy for filling thi s slot was heuristic . In case the violent event associated with the template lacked an agentiv e argument, plausible candidates were looked for elsewhere in the neighboring text . Once again, the fragmentary nature of the parses led to the heuristic strategy fallback bein g invoked fairly often, with very mixed results . Perpetrator organization ID : We obtained comparatievely high recall scores for this slot . This is a relatively easy slot to fill, however, because likely perpetrator organizations ar e readily identified . The remaining incident and perpetrator slots ended up being filled by default values . As a result, although we obtained some reasonable recall scores for these individual slots, these scores are of little real interest .
ISSUES WITH PRECISIO N
Our precision error rate is largely accounted for by overly eager template generation . As we note in th e system description, the version of ALEMBIC fielded at MUC-4 generates a template for every seemingly distinc t violent event . Our strategy for distinguishing such events from each other was heavily dependent on ou r reference resolution module, which turned out to be quite unreliable, and as a result generated multipl e (nearly) identical templates for the same event . Consequently, we ended up with overall low precision an d high overgeneration scores, as demonstrated by Table 3 . Precision TST3 T5T4 8 10 Overgeneratio n 90 87
Table 3 : Overall precision scores (all templates row) The effects of this template generation strategy on our precision scores were fairly dramatic . We ended up actually making relatively few incorrect fills for those templates that were mapped by the scoring program . Specifically, ignoring spurious templates (as in the matched/missing row) we obtain precision scores of 72 an d 75 for TST3 and TST4 respectively . However, because our spurious fills ended up outnumbering our incorrec t fills by 20 to 1, our official scores from the all templates row were considerably weaker . Aside from this principal cause of our precision errors, another significant factor is that ALEMBIC failed to filter out templates that corresponded to military clashes between guerrilla groups and the armed forces . We failed to incorporate such a filter largely because to do so presupposes filling some slots that we were simpl y leaving blank. Anecdotally, among the worst offenders of this sort was one message for which we generate d two legitimate templates and ten templates corresponding to military clashes .
11 8
A CLOSER LOOK AT SYNTAX AND REFERENC E
A common thread to both our recall and precision problems is fragmentation of the parses . With respect t o recall, fragmentation lead to the syntactic arguments of event verbs (and of their nominalizations) being lef t unattached; this caused the system to fall back frequently on unreliable backup strategies . In addition, th e fragmentation confused our reference resolution module, because it introduced far too many top-level nou n phrases or event verbs, each of which was potentially a candidate for reference resolution . This caused ALEMBIC to miss co-references needed to fill slots, and it also led to the system's poor ability to distinguis h identical events on the basis of reference resolution . We performed a number of post-hoc analyses to estimate the relative weight on fragmentation of variou s linguistic factors for which the MUC-4 version of ALEMBIC had incomplete grammatical coverage . These turned out to include a traditional and unsurprising cast of linguistic characters : coordination, PP attachment, noun noun modification, subgrammars for title, date, location, etc . None of these factors seems particularl y dominant, however ; they all need to be eventually addressed in some way (linguistically principled o r otherwise) . Because fragmentation played such a compromising role with respect to our reference resolution module , we also performed a number of quantitative analyses to clarify the nature of the problem . To begin with, w e looked at the set of candidates that were considered when resolving an anaphoric expression . Many of these candidates were spurious ; they had b...