This preview shows page 1. Sign up to view the full content.
Unformatted text preview: different categorical hierarchies of about 800 and
2,300 categories (Paaß & deVries 2005). Due to conﬁdentiality the results can be
published only in anonymized form.
For the corpus with 2,300 categories the best system achieved at an F1-value
of 39%, while for the corpus with 800 categories an F1-value of 79% was reached.
In the latter case a partially automatic assignment based on the reliability score
was possible for about half the documents, while otherwise the systems could
only deliver proposals for human categorizers. Especially good are the results
for recovering persons and geographic locations with about 80% F1-value. In
general there were great variations between the performances of the systems.
In a usability experiment with human annotators the formal evaluation results
were conﬁrmed leading to faster and more consistent annotation. It turned out,
that with respect to categories the human annotators exhibit a relative large
disagreement and a lower consistency than text mining systems. Hence the
support of human annotators by text mining systems offers more consistent
annotations in addition to faster annotation. The Deutsche Presse-Agentur now
is routinely using a t...
View Full Document
- Summer '11