Named Product Entity Recognition Based on Hierarchical Hidden Markov Model
Feifan Liu, Jun Zhao, Bibo Lv, Bo Xu Hao Yu National Laboratory of Pattern Recognition FUJITSU R&D Institute of Automation Chinese Academy of Sciences Xiao Yun Road No.26 Beijing P.O. Box 2728, 100080 Chao Yang District, Beijing, 100016 {ffliu,jzhao,bblv,xubo}@nlpr.ia.ac.cn [e-mail]
Abstract
A hierarchical hidden Markov model (HHMM) based approach of product named entity recognition (NER) from Chinese free text is presented in this paper. Characteristics and challenges in product NER is also investigated and analyzed deliberately compared with general NER. Within a unied statistical framework, the approach we proposed is able to make probabilistically reasonable decisions to a global optimization by leveraging diverse range of linguistic features and knowledge sources. Experimental results show that our approach performs quite well in two different domains.
entity recognition which can be crucial and valuable in many business IE applications, especially with the increasing research interest in Market Intelligence Management(MIM), Enterprise Content Management (ECM) [Pierre 2002] and etc. This paper describes a prototype system for product named entity recognition, ProNER, in which a HHMM-based approach is employed. Within a unied statistical framework, the approach based on a mixture model is able to make probabilistically reasonable decisions to a global optimization by exploiting diverse range of linguistic features and knowledge sources. Experimental results show that ProNER performs quite well in two different domains.
2 Related Work
Up to now not much work has been done on product named entity recognition, nor systematic analysis of characteristics for this task. [Pierre 2002] developed an English NER system capable of identifying product names in product views. It employed a simple Boolean classier for identifying product name, which was constructed from the list of product names. The method is similar to token matching and has a limitation for product NER applications. [Bick et al. 2004] recognized named entities including product names based on constraint grammar based parser for Danish. This rule-based approach is highly dependent on the performance of Danish parser and suffers from its weakness in system portability. [C. Niu et al. 2003] presented a bootstrapping approach for English named entity recognition using successive learners of parsing-based decision
1 Introduction
Named entity recognition(NER) plays a signicantly important role in information extraction(IE) and many other applications. Previous study on NER is mainly focused either on the proper name identication of person(PER), location(LOC), organization(ORG), time(TIM) and numeral(NUM) expressions almost in news domain, which can be viewed as general NER, or other named entity (NE) recognition in specic domain such as biology. As far as we know, however, there is little prior research work conducted by far on product named
0 This work was supported by the Natural Sciences Foundation of China(60372016,60272041) and the Natural Science Foundation of Beijing(4052027).
40
System [Zhang et al. 2003] [Sun et al. 2002] [Tsai et al. 2004]
Statistical Model HMM class-based LM ME model
Linguistic Feature semantic role, tokens word form, NE category tokens
Combinative Points pattern rules cue words list knowledge representation
Table 1: Comparison between several Chinese NER systems1 list and HMM, and promising experiment results (F-measure: 69.8%) on product NE (corresponding to our PRO) were obtained. Its main advantage lies in that manual annotation of a sizable training corpus can be avoided, but it suffers from two problems, one is that it is difcult to nd sufcient concept-based seeds needed in bootstrapping for the coverage of the variations of PRO subcategories, another it is highly dependent on parser performance as well. Research on product NER is still at its early stage, especially in Chinese free text collections. However, considerable amount of work has been done in the last decade on the general NER task and biological NER task. The typical machine learning approaches for English NE are transformation-based learning[Aberdeen et al. 1995], hidden Markov model[Bikel et al. 1997], maximum entropy model[Borthwick, 1999], support vector machine learning[Eunji Yi et al. 2004], unsupervised model[Collins et al. 1999]and etc. For Chinese NER, the prevailing methodology applied recently also lie in machine learning combining other knowledge base or heuristic rules, which can be compared on the whole in three aspects showed in Table 1. In short, the trend in NER is to adopt a statistical framework which try to exploit some knowledge base as well as different level of text features within and outside NEs. Further those ideas, we present a hybrid approach based on HHMM [S. Fine et al. 1998] which will be described in detail. Brand Name(BRA), Product Type(TYP), Product Name(PRO), and BRA and TYP are often embedded in PRO. In the following two examples, there are two BRA NEs, one TYP NE and one PRO NE all of which belong to the family of product named entities. Exam 1: (Benq)/BRA (brand) (market shares) (steadily) (ascend) Exam 2: (corporation) (will) (deliver) [Canon/BRA 334 (ten thousand) (pixels) (digital) (camera) Pro90IS/TYP]/PRO Brand Name refer to proper name of product trademark such as (Benq) in Exam 1. Product Type is a kind of product named entities indicating version or series information of product, which can consist of numbers, English characters, or other symbols such as + and etc.In our study, two principles should be followed. (1) Chinese characters are not considered to be TYP, nor subpart of TYP although some of them can contain version or series information. For instance, in 2005 (happy new year) (version) (cell phone), here (happy new year) (version)should not be considered as a TYP. (2) Numbers are essential elements in product type entity. For instance, in PowerShot (series) (digital) (camera), PowerShot is not considered as a TYP, however, in PowerShot S10 (digital) (camera), PowerShot S10 can make up of a TYP. Product Name, as showed above in Exam 2, is a kind of product named entities expressing selfcontained proper name for some specied product in real world compared to BRA and TYP which only express one attribute of product. i.e. a PRO NE must be assigned with distinctly discriminative information which can not shared with other general product-related expressions.
3 Problem Statements and Analysis
3.1 Task Denition 3.1.1 Denition of Product Named Entity In our study, only three kinds of product named entities are considered, namely
1
Note: LM(language model); ME(maximum entropy).
41
(1) Product-related expressions which are embedded with either BRA or TYP can be qualied to be a PRO entity. e.g. BenQ (ash) (disk) is a PRO entity, but the general product-related expression (ash) (market) (investigation) cannot make up of a PRO entity. (2) Product-related expressions indicating some specic version or series information which is unique for a BRA can also be considered as a PRO entity. e.g. DIGITAL IXUS (series) (digital) (camera) is a PRO because DIGITAL IXUS series is unique for Canon product, but (intelligent) (version) (cell phone) is not a PRO because the attribute of intelligent version can be assigned to any cell phone product. 3.1.2 Product Named Entity Recognition Product named entity recognition involves the identication of product-related proper names in free text and their classication into different kinds of product named entities, referring to PRO, TYP and BRA in this paper.In comparison with general NER, nested product NEs should be tagged separately rather than being tagged just as a single item, shown as Figure 1. 3.2 Challenges for Product Named Entity Recognition For general named entities, there are some cues which are very useful for entity recognition, such as (city), (Inc.), and etc. In comparison, product named entities have no such named conventions and cues, resulting in higher boundary ambiguities and more complex NE candidate triggering difculties.
In comparison with general NER, more challenges in product NER result from miscellaneous classication ambiguities. Many entities with identical form can be a kind of general named entity, a kind of product named entity, or just common words. In comparison with general named entities, product named entities show more exible variant forms. The same entity can be expressed in several different forms due to spelling variation, word permutation and etc. This also compounds the difculties in product named entity recognition. In comparison with general named entities, it is more frequent that product named entities are nested as Figure 1 illustrates. More efforts have to be made to identify such named entities separately. 3.3 Our Solutions We adopt the following strategies in triggering and disambiguating process respectively. (1) As to product NER, its pivotal to control the triggering candidates efciently for the balance between precision and recall. Here we use the knowledge base such as brand word list, and other heuristic information which can be easily acquired. (2)After triggering candidates, we try to employ a statistical model to make the most of multi-level context information mentioned above in disambiguation. We choose hierarchical hidden Markov model (HHMM) [S. Fine et al. 1998] for its more powerful ability to model the multiplicity of length scales and recursive nature of sequences.
42
4 Hybrid Approach for Product NE Recognition
4.1 Overall Workow of ProNER Preprocessing: Segment, POS tagging and general NER is primarily conducted using our offshelf SegNer2.0 toolkit on input text. Generating Product NE Candidates: First, BRA or ORG and TYP are triggered by brand word list and some word features respectively. Here we categorize the triggering word features into six classes: alphabet string, alphanumeric string, digits, alphabet string with fullwidth, digits with fullwidth and other symbols except Chinese characters. Then PRO are triggered by BRA and TYP candidates as well as some clue words indicating type information to some extent such as (version), (series). In this step the model structure(topology) of HHMM[S. Fine et al. 1998] is dynamically constructed, and some conjunction words or punctuations and specied maximum length of product NE are used to control it. Disambiguating Candidates: In this module, boundary and classication ambiguities between candidates are resolved simultaneously. And Viterbi algorithm is applied for most-likely state sequences based on the HHMM topology. 4.2 Integration with Heuristic Information To get more efcient control in triggering process above, we try to integrate some heuristic information. The heuristic rules we used are as domainindependent as possible in order that they can be integrated with statistical model systematically rather than just some tricks on it. (1) Stop Word List: Common English words, English brand word, and some punctuations are extracted automatically from training set to make up of stop word list for TYP; by co-occurrence statistics between ORG and its contexts, some words are extracted from the contexts to make up of stop word list for PRO in order to overcome the case that brand word is prone to bind its surroundings to be a PRO. (2) Constrain Rules: Rule 1: For the highly frequent pattern + (number + English quantier
0.7
IS0
0.3 0.2 0.5 0.7 0.3 0.5 0.3
ES
IS1
0.2 0.3
IS2
PS5
ES
PS1
PS2
ES
PS3
PS4
Figure 2 Structure of Hierarchical Hidden Markov Model (HHMM)
word), all the corresponding TYP candidates triggered by categorized word features(CWF) should be removed. Rule 2: Product NE candidates in which some binate symbols dont match each other should be removed. Rule 3: Unreasonable symbols such as - or : should not occur in the beginning or end of product NE candidates. 4.3 HHMM for product NER application By HHMM [S. Fine et al. 1998] the product NER can be formulated as a tagging problem using Viterbi algorithm. Unlike traditional HMM in POS tagging, here the topology of HHMM is not xed and internal states can be also a similar stochastic model on themselves, called internal states compared to production states which will emit only observations. Our HHMM structure actually consists of three level approximately illustrated as gure 2 in which IS denotes internal state, PS denotes production state and ES denote end state at every level. For our application, an input sequence from our SegNer2.0 toolkit can be formalized as w1 /t1 w2 /t2 . . . wi /ti . . . wn /tn , among which wi and ti is the ith word and its part-ofspeech, n is the number of words. The POS tag set here is the combination of tag set from Peking University(PKU-POS) and our general NE categories(GNEC) including PER(person), LOC(location), ORG(organization), TIM(time expression), NUM(numeric expression). Therefore we can construct our HHMM by model the state set {S} consisting of {GNEC}, {BRA, PRO, TYP}, and {V} as well as the observation set {O} consisting of {V} which is the word set from training data. That is to say, the word forms
43
in {V} which are not included in NEs are also viewed as production states. In our model, only PRO are internal state which may activate other production states such as BRA and TYP resulting in recursive HMM. In consisd tence with S. Fines work, qi (1 d D) is used to indicate the ith state in the dth level of hierarchy. So, the product NER problem is to nd the most-likely state activation sequence Q*, a multiscale list of states, based on the dynamic topology of HHMM given a observation sequence W = w1 w2 . . . wi . . . wn , formulated as follows based on Bayes rule (P (W )=1).
arriving at an end state, thus hierarchical computation is implemented; k (3) If qj = BRA, we assign equation (3) a constant value in that BRA candidates consist of only a single brand word in our method. In addition brand word can also generate ORG candidates, thus we can assign equation (3) as follows.
k p([wqk begin ...wqk end ]|qj = BRA) = 0.5 (4)
j j
Q= arg max P (Q|W )= arg max P (Q)P (W |Q)
Q Q
(1) From the root node of HHMM, activity ows to all other nodes at different levels according to their transition probability. For description convenience, we take the kth level as example(activated by the mth state at the k-1th level).
horizontal transition |q k | k k1 k k P (Q) p(q1 |qm ) p(q2 |q1 ) = vertical transition k k k p(qj |qj1 , qj2 ) j=3
k (4) If qj = T Y P , categorized word features(CWFs) dened in section 4.1 are applied, i.e. the words associated with the current state are replaced with their CWFs (WC) acting as observations. Then we can compute the emission probability of this TYP production state as the followk ing equation, among which |qj | is the length of observation sequence associated with the current state.
k p([wqk begin ...wqk end ]|qj = T Y P )
j j
k =p(wc1 |begin)p(end|wc|qj | )
k |qj |
p(wcm |wcm1 )
m=2
(2)
=
k |qP S |
All the parameters in every level of HHMM can be acquired using maximum likelihood method with smoothing from training data. 4.4 Mixture of Two Hierarchical Hidden Markov Models Now we have implemented a simple HHMM for product NER. Note that in the above model(HHMM-1), we exploit both internal and external features of product NEs only at levels of simply semantic classication and just word form. To achieve our motivation in section 3.3, we construct another HHMM(HHMM2) for exploiting multi-level contexts by mixing with HHMM-1. In HHMM-2, the difference from HHMM-1 lies in the state set SII and observation set OII . Because the input text will be processed by segment, POS tagging and general NER, as a alternative, we can also take T=t1 t2 . . . ti . . . tn as observation sequence, i.e. OII ={PKU-POS}. Accordingly, SII = {{PKU-POS}, {GNEC}, BRA, TYP,
j=1
k p([wqk begin ...wqk end ]|qj ) j j
P (W |Q)=
k / if qj {IS} activate other states recursively k
if qj {IS}
(3)
Where and is the number of production states in the kth level; wqk begin ...wqk end indicates the word sequence
j j
|q k | is the number of all states
k |qP S |
k corresponding to the state qj . k (1) In equation (3), if qj {{GNEC},{V}}, k )=1, because we asp([wqk begin ...wqk end ]|qj j j sume that the general NER results from the preceding toolkit are correct; k (2) If qj = P RO, production states in the (k+1)th level will be activated by this internal state through equation (2),(3) and go back when
44
Data Sets DataSetPRO1.2 OpenTestSet CloseTestSet
PRO 12,432 1800 1553
BRA 5,047 803 513
TYP 10,606 1364 1296
PER 424 39 55
LOC 1,733 207 248
ORG 4,798 614 619
Table 2: Overview of Data Sets
PRO}, among which PRO is internal state. Similarly, the problem is formulated as follows with HHMM-2. Q = arg max P (QII |T ) II
QII
5 Experiments and analysis
5.1 Data Set Preparation A large number of web pages in mobile phone and digital domain are compiled into text collections, DataSetPRO, on which multi-level processing were performed. Our nal version, DataSetPRO1.2, consists of 1500 web pages, roughly 1,000,000 Chinese characters. Randomly selected 140 texts (digital 70, mobile phone 70) are separated from DataSetPRO1.2 as our OpenTestSet, the rest as TrainingSet, from which 160 texts are extracted as CloseTestSet. Table 2 illustrates the overview of them. 5.2 Experiments Due to various and exible forms of product NEs, though some boundaries of recognized NEs are inconsistent with manual annotation, they are also reasonable. So soft evaluation is also applied in our experiments to make the evaluation more reasonable. The main idea is that a discount score will be given to recognized NEs with wrong boundary but correct detection and classication. However, strict evaluation only score completely correct ones. All the results is conducted on OpenTestSet unless it is particularly specied. Also, the evaluation scores used below are obtained mainly by
= arg max P (QII )P (T |QII )
QII
(5)
The description and computation of HHMM-2 is similar to HHMM-1 and is omitted here. We can see that besides making use of semantic classication of NEs in common, HHMM-1 and HHMM-2 exploit word form and part-of-speech (POS) features respectively. Word form features make the model more discriminative, while POS features result in robustness. Intuitively, the mixture of these two models is desirable for higher performance in product NER by balancing the robustness and discrimination which can be formulated in logarithmic form as follows. (Q , Q ) II = arg max{log(P (Q)) + log(P (W |Q))
Q,QII
+ [log(P (QII )) + log(P (T |QII ))]} (6) Where is a tuning parameter for adjusting the weight of two models.
45
Product NEs PRO TYP BRA Product NEs PRO TYP BRA
Digital Domain ( 8) Close Test Open Test Precision Recall F-measure Precision 0.864 0.799 0.830 0.762 0.903 0.906 0.905 0.828 0.824 0.702 0.758 0.723 Mobile Phone Domain ( 8) Close Test Open Test Precision Recall F-measure Precision 0.917 0.935 0.926 0.799 0.959 0.976 0.967 0.842 0.911 0.741 0.818 0.893
Recall 0.744 0.944 0.705
F-measure 0.753 0.882 0.714
Recall 0.856 0.886 0.701
F-measure 0.827 0.864 0.785
Table 3: Experimental Results in Digital and Mobile Phone Domain soft metrics, and strict scores are also given for comparison in experiment 3. 1. Evaluation on the Inuence of in the Mixture Model. In the mixture model denoted as equation (6), the value reects the different contribution of two individual models to the overall system performance. The larger , the more contribution made by HHMM-2. Figure 3, 4, 5 illustrate the varying curves of recognition performance with the value on PRO, TYP, BRA respectively. Note that, if equal to 1 then two models are mixed with equivalent weight. We can see that, as goes up, the F-measures of PRO and TYP increase obviously rstly, and begin to go down slightly after a period of growing at. It can be explained that HHMM-2 mainly exploits part-of-speech and general NER features which can relieve the sparseness problem to some extent, which is more serious in HHMM-1 due to using lower level of contextual information such as word form. However, as becomes larger, the problem of imprecise modeling in HHMM2 will be more salient and begin to illustrate a side-effect in the mixture model. Whereas, the inuence of on BRA is negligible because its candidates are triggered by the relatively reliable knowledge base and its sub-model in HHMM is assigned a constant as shown in equation(4). Summings-up: (1) Mixture with HHMM-2 can make up the weakness of HHMM-1. (2) HHMM-2 can make more contributions to the mixture model under the conditions that
limited annotated data is available at present. In our system, is assigned t...