Classication Text in Asian Languages without Word Segmentation
Fuchun Peng Xiangji Huang Dale Schuurmans Shaojun Wang School of Computer Science, University of Waterloo, Ontario, Canada Department of Computer Science, University of Massachusetts, Amherst, MA, USA Department of Statistics, University of Toronto, Ontario, Canada f3peng, jhuang, dale, sjwang @ai.uwaterloo.ca
Abstract
We present a simple approach for Asian language text classication without word segmentation, based on statistical -gram language modeling. In particular, we examine Chinese and Japanese text classication. With character -gram models, our approach avoids word segmentation. However, unlike traditional ad hoc -gram models, the statistical language modeling based approach has strong information theoretic basis and avoids explicit feature selection procedure which potentially loses signicantly amount of useful information. We systematically study the key factors in language modeling and their inuence on classication. Experiments on Chinese TREC and Japanese NTCIR topic detection show that the simple approach can achieve better performance compared to traditional approaches while avoiding word segmentation, which demonstrates its superiority in Asian language text classication.
Yang, 1999). Text classication in Asian languages such as Chinese and Japanese, however, is also an important (and relatively more recent) area of research that introduces a number of additional difculties. One difculty with Chinese and Japanese text classication is that, unlike English, Chinese and Japanese texts do not have explicit whitespace between words. This means that some form of word segmentation is normally required before further processing. However, word segmentation itself is a difcult problem in these languages. A second difculty is that there is a lack of standard benchmark data sets for these languages. Nevertheless, recently, there has been signicant notable progress on Chinese and Japanese text classication (Aizawa, 2001; He et al., 2001). Many standard machine learning techniques have been applied to text categorization problems, such as naive Bayes classiers, support vector machines, linear least squares models, neural networks, and knearest neighbor classiers (Sebastiani, 2002; Yang, 1999). Unfortunately, most current text classiers work with word level features. However, word identication in Asian languages, such as Chinese and Japanese, is itself a hard problem. To avoid the word segmentation problems, character level gram models have been proposed (Cavnar and Trenkle, 1994; Damashek, 1995). There, they used grams as features for a traditional feature selection process and then deployed classiers based on calculating feature-vector similarities. This approach has many shortcomings. First, there are an enormous number of possible features to consider in text categorization, and standard feature selection ap
1 Introduction
Text classication addresses the problem of assigning a given passage of text (or a document) to one or more predened classes. This is an important area of information retrieval research that has been heavily investigated, although most of the research activity has concentrated on English text (Dumais, 1998;
Proceedings of the Sixth International Workshop on Information Retrieval with Asian Languages, July 2003, pp. 41-48.
The simplest and most successful basis for language modeling is the -gram model. Note that by the chain rule of probability we can write the probability of any sequence as
4 6 5& 3 & 90 1 % 6 90 1
An -gram model approximates this probability by assuming that the only words relevant to predicting are the previous words; that is, it assumes the Markov -gram independence assumption
4 6 F& ) D B 4 EC5& @ A 3 & 20 1 4 6 5& 4 6 5& 3 & 3 & 20 1 90 1
A straightforward maximum likelihood estimate of -gram probabilities from a corpus is given by the observed frequency
D B 4 67 45& EC5& 1 G D B 4 EC5& 1 6 & 4 6 5& D B 4 ECF& 3 & 90 1
if
6 4 7 5&
)
3 &
20 1
' (&
$
!
Perplexity
(1)
otherwise
4 U V6 F&
DEBC5& 3 & 120 6 5& 4 4 b Y `WX6 & DEBC45& 1
4 U V6 5&
4 6 5&
D B 4 ECF&
G
D B 4 IC5&
D B 4 IC5&
3 &
90 1 T
1 a
3 &
PQQ
QQS
R
90 1
The goal of language modeling is to predict the probability of natural word sequences; or more simply, to put high probability on word sequences that actually occur (and low probability on word sequences that never occur). Given a word sequence to be used as a test corpus, the quality of a language model can be measured by the empirical perplexity (or entropy) on this corpus
% "# #
(4)
2 Language Model Text Classiers
where #(.) is the number of occurrences of a specied gram in the training corpus. Unfortunately, using grams of length up to entails estimating the probability of events, where is the size of the word vocabulary. This quickly overwhelms modern computational and data resources for even modest choices of (beyond 3 to 6). Also, because of the heavy tailed nature of language (i.e. Zipfs law) one is likely to encounter novel -grams that were never witnessed during training. Therefore, some mechanism for assigning non-zero probability to novel grams is a central and unavoidable issue. One standard approach to smoothing probability estimates to cope with sparse data problems (and to cope with potentially missing -grams) is to use some sort of back-off estimator
H B H
G
' (&
8
proaches do not always cope well in such circumstances. For example, given a sufciently large number of features, the cumulative effect of uncommon features can still have an important effect on classication accuracy, even though infrequent features contribute less information than common features individually. Therefore, throwing away uncommon features is usually not an appropriate strategy in this domain (Aizawa, 2001). Another problem is that feature selection normally uses indirect tests, such as or mutual information, which involve setting arbitrary thresholds and conducting a heuristic greedy search to nd a good subset of features. Moreover, by treating text categorization as a classical classication problem, standard approaches can ignore the fact that texts are written in natural language, which means that they have many implicit regularities that can be well modeled by specic tools from natural language processing. In this paper, we present a simple text categorization approach based on statistical -gram language modeling to overcome the above shortcomings in a principled fashion. An advantage we exploit is that the language modeling approach does not discard low frequency features during classication, as is commonly done in traditional classication learning approaches. Also, the language modeling approach uses -gram models to capture more contextual information than standard bag-of-words approaches, and employs better smoothing techniques than standard classication learning. These advantages are supported by our empirical results on Chinese and Japanese data.
The goal of language modeling is to obtain a small perplexity. 2.1 -gram language modeling
(2)
(3)
where
6 & G 4 D B G4 EC5& 1 rphfd 6 5& s q i g e c 4 6 5& t u D B 4 EC5& 1 4 6 5& 4 6 5& D B 4 IC5& I D B 4 ECF& D B 4 ECF& 1 a 3 590 d 1 T 3 590 d 1 T 4 6 5& D B 4 IC5& D B 4 IC5& 3 & @ @ ) ) 90 1 T
Using Bayes rule, this can be rewritten as
i c v 6 3 1 6 i 1
Here, is the likelihood of under category , which can be computed by -gram language modeling. The likelihood is related to perplexity by can be computed from Equ. (1). The prior training data or can be used to incorporate more assumptions, such as a uniform or Dirichelet distribution. Therefore, our approach is to learn a separate back-off language model for each category, by training on a data set from that category. Then, to categorize a new text , we supply to each language model, evaluate the likelihood (or entropy) of under the model, and pick the winning category according to Equ. (9). The inference of an -gram based text classier is very similar to a naive Bayes classier (to be dicussed below). In fact, -gram classiers are a straightforward generalization of naive Bayes (Peng and Schuurmans, 2003).
3 Traditional Text Classiers
We introduce the three standard text classiers that we will compare against below. 3.1 Naive Bayes classiers A simple yet effective learning algorithm for text classication is the naive Bayes classier. In this model, a document is normally represented by a vector of attributes . The naive Bayes model assumes that all of the attribute values , are independent given the category label
6 U U 51 c c
2.2
Language models as text classiers
Text classiers attempt to identify attributes which distinguish documents in different categories. Such attributes may include vocabulary terms, word average length, local -grams, or global syntactic and semantic properties. Language models also attempt capture such regularities, and hence provide another natural avenue to constructing text classiers.
c
c
c
6
i
1
v
~
c
i c v 6 3 1
~
i
(6) The discounted probability (5) can be computed using different smoothing approaches including Laplace smoothing, linear smoothing, absolute smoothing, Good-Turing smoothing and WittenBell smoothing (Chen and Goodman, 1998). The language models described above use individual words as the basic unit, although one could instead consider models that use individual characters as the basic unit. The remaining details remain the same in this case. The only difference is that the character vocabulary is always much smaller than the word vocabulary, which means that one can normally use a much higher order, , in a character level -gram model (although the text spanned by a character model is still usually less than that spanned by a word model). The benets of the character level model in the context of text classication are multi-fold: it avoids the need for explicit word segmentation in the case of Asian languages, and it greatly reduces the sparse data problems associated with large vocabulary models. In this paper, we experiment with character level models to avoid word segmentation in Chinese and Japanese.
4 6 5&
D B 4 IC5&
6
c
3
3 &
i
1
v
1 | 0
~ o
} { t | x w v t zyIu
' (&
% f ~
6
i
1
v
v
r si
~
~
} { t | x w v t zIu
} { t | x w v t zIu
o
o
r i
4 6 5&
is the discounted probability, and is a normalization constant calculated to be
1 a
qn n po i
i
i
j kii h
(5)
Our approach to applying language models to text categorization is to use Bayesian decision theory. Assume we wish to classify a text into a category . A natural choice is to pick the category that has the largest posterior probability given the text. That is,
f g I e cU U l m
(7)
(8)
w x 5 rEy Iw v w x 5 rEy Iw v
(9)
(10)
subject to
s&
In
our
settings.
4 Empirical evaluation
We now present our experimental results on Chinese and Japanese text classication problems. The Chinese data set we used has been previously investigated in (He et al., 2001). The corpus is a subset of the TREC-5 Peoples Daily news corpus published by the Linguistic Data Consortium (LDC) in 1995. The entire TREC-5 data set contains 164,789 documents on a variety of topics, including international and domestic news, sports, and culture. The corpus was originally intended for research on information retrieval. To make the data set suitable for text categorization, documents were rst clustered into 101 groups that shared the same headline (as indicated by an SGML tag). The six most frequent groups were selected to make a Chinese text categorization data set. For Japanese text classication, we consider the Japanese text classication data investigated by (Aizawa, 2001). This data set was converted from the NTCIR-J1 data set originally created for Japanese text retrieval research. The conversion process is similar to Chinese data. The nal text classication dataset has 24 categories which are unevenly distributed. 4.1 Experimental paradigm
In this method a test document and a class label are both represented by vectors of -gram features, and a distance measure between the representations of and is dened. The features to be used during classication are usually selected by employing heuristic methods, such as or mutual information scoring, that involve setting cutoff thresholds and conducting a greedy search for a good feature subset. We refer this method as ad hoc -gram based text classier. The nal classication decision is made according to
q 6 i U
distance
l
(12)
Different distance metrics can be used in this approach. We implemented a simple re-ranking distance, which is sometimes referred to as the out-outplace (OOP) measure (Cavnar and Trenkle, 1994). In this method, a document is represented by an gram prole that contains selected -grams sorted by decreasing frequency. For each -gram in a test document prole, we nd its counterpart in the class prole and compute the number of places its location differs. The distance between a test document and a class is computed by summing the individual out-of-place values. 3.3 Support vector machine classiers linearly separable training exam, where each
q 8 U U U ) e 3 B h 8 l & d
Given a set of ples
i
c
1
c
} x w v t (9| Iu
o
3.2
Ad hoc -gram text classiers
Both of the Chinese and Japanese data sets involve classifying into a large number of categories, where each document is assigned a single category. Many classication techniques, such as SVMs, are intrinsically dened for two class problems, and have to be extended to handle these multiple category data
is the frequency of attribute in , , and . A special case of Laplace smoothing is add one smoothing, obtained . We use add one smoothing in our by setting experiments below.
) | 8 8 | 8
where
experiments below, we use the (Joachims, 1998) toolkit with default
)
(11)
|
3 3
y6 & d & 1 )
3 3
To cope with features that remain unobserved during training, the estimate of is usually adjusted by Laplace smoothing
i 6 3 590 1 | | 8 i 6 3 s520 1
minimize
(13)
h
&
Y
d q
)
@
U
)
l
. Thus, a maximum a posteriori (MAP) classier can be constructed as follows.
i ' E6 3 520 1 % 8 b X6 i 1 20 S P R } { t | x w v t 9Iu o r si | i r si c i
sample belongs to one of the two classes, , the SVM approach seeks the optimal hyperplane that separates the positive and negative examples with the largest margin. The problem can be formulated as solving the following quadratic programming problem (Vapnik, 1995).
sets. For SVMs, we employ a standard technique of rst converting the category classication problem to binary classication problems. For the experiments on Chinese data, we follow (He et al., 2001) and convert the problem into 6 binary classication problems. In each case, we randomly select 500 positive examples and then select 500 negative examples evenly from among the remaining negative categories to form the training data. The testing set contains 100 positive documents and 100 negative documents generated in the same way. The training set and testing set do no overlap and do not contain repeated documents. For the experiments on Japanese data, we follow (Aizawa, 2001) and directly experiment with a 24-class classication problem. The NTCIR data sets are unevenly distributed across categories. The training data consists of 310,355 documents distributed unevenly among the categories (with a minimum of 1,747 and maximum of 83,668 documents per category), and the testing set contains 10,000 documents unevenly distributed among categories (with a minimum of 56 and maximum of 2,696 documents per category). 4.2 Measuring classication performance
3 j 3 3 j 3
4.3
Results on Chinese data
Table 1 gives the results of the character level language modeling approach, where rows correspond to different smoothing techniques. Columns corre. The spond to different -gram order entries are the micro-average F-measure. (Note that the naive Bayes result corresponds to -gram order 1 with add one smoothing, which is italicized in the table.) The results the ad hoc OOP classier, and for the SVM classier are shown in Table 2 and Table 3 respectively, where the columns labeled Feature # are the number of features selected. Add-one Absolute Good-Turing Linear Witten-Bell 1 0.856 0.856 0.856 0.857 0.857 2 0.802 0.868 0.863 0.861 0.860 3 0.797 0.867 0.861 0.861 0.865
U U U )
4 0.805 0.868 0.862 0.865 0.864
Table 1: Results of character level language modeling classier on Chinese data.
In the Chinese experiments, where 6 binary classication problems are formulated, we measured classication performance by micro-averaged F-measure scores. To calculate the micro-averaged score, we formed an aggregate confusion matrix by adding up the individual confusion matrices from each category. The micro-averaged precision, recall, and Fmeasure can then be computed based on the aggregated confusion matrix. For the Japanese experiments, we measured overall accuracy and the macro-averaged F-measure. Here the precision, recall, and F-measures of each individual category can be computed based on a confusion matrix. Macro-averaged scores can be computed by averaging the individual scores. The overall accuracy is computed by dividing the number of correctly identied documents (summing the numbers across the diagonal) by the total number of test documents.
3 j 3 b 3 j 3
Feature # 100 200 300 400
Micro-F1 0.7808 0.8012 0.8087 0.7889
Feature # 500 1000 1500 2000
Micro-F1 0.7848 0.7883 0.7664 0.7290
Table 2: Results of the character level OOP classier on Chinese data. Feature # 100 200 300 400 Micro-F1 0.811 0.813 0.817 0.816 Feature # 500 1000 1500 2000 Micro-F1 0.817 0.817 0.815 0.816
Table 3: Results of the character level SVM classier on Chinese data. 4.4 Results on Japanese data
For the Japanese data, we experimented with byte level models (where in fact each Japanese character is represented by two bytes). We used byte level
models to avoid possible character level segmentation errors that might be introduced, because we lacked the knowledge to detect misalignment errors in Japanese characters. The results of byte level language modeling classiers on the Japanese data are shown in Table 4. (Note that the naive Bayes result corresponds to -gram order 2 with add one smoothing, which is italicized in the table.) The results for the OOP classier are shown in Table 5. Note that SVM is not applied in this situation since we are conducting multiple category classication directly while SVM is designed for binary classication. However, Aizawa (Aizawa, 2001) reported a performance of abut 85% with SVMs by converting the problem into a 24 binary classication problem and by performing word segmentation as preprocessing. Feature # 100 200 300 400 500 1000 2000 3000 4000 5000
Accuracy 0.2044 0.2830 0.3100 0.3616 0.3682 0.4416 0.4990 0.4770 0.4462 0.3706
Macro-F 0.1692 0.2308 0.2677 0.3118 0.3295 0.4073 0.4510 0.4315 0.3820 0.3139
other approaches for both the Chinese and Japanese data, while avoiding word segmentation. The SVM result on Japanese data is obtained from (Aizawa, 2001) where word segmentation was performed as a preprocessing. Note that SVM classiers do not perform as well in our Chinese text classication as they did in English text classication (Dumais, 1998), neither did they in Japanese text classication (Aizawa, 2001). The reason worths further investigations. Overall, the language modeling approach appears to demonstrate state of the art performance for Chinese and Japanese text classication. The reasons for the improvement appear to be three-fold: First, the language modeling approach always considers every feature during classication, and can thereby avoid an error-prone feature selection process. Second, the use of -grams in the model relaxes the restrictive independence assumption of naive Bayes. Third, the techniques of statistical language modeling offer better smoothing methods for coping with features that are unobserved during training. LM 0.868 0.84 NB OOP SVM Chinese Character Level 0.856 0.8087 0.817 Japanese Byte Level 0.66 0.4990 85% (Aizawa, 2001)
Table 6: Comparison of best classier results 5.2 Inuence of the -gram order
Table 5: Results of byte level OOP classier on Japanese data.
5 Discussion and analysis
We now give a detailed analysis and discussion based on the above results. We rst compare the language model based classiers with other classiers, and then analyze the inuence of the order of the -gram model, the inuence of the smoothing method, and the inuence of feature selection in tradition approaches. 5.1 Comparing classier performance
Table 6 summarizes the best results obtained by each classier. The results for the language model (LM) classiers are better than (or at least comparable to )
The order is a key factor in -gram language modeling. An order that is too small will not capture sufcient information to accurately model character dependencies. On the other hand, a context that is too large will create sparse data problems in training. In our Chinese experiments, we did not observe signicant improvement when using higher order gram models. The reason is due to the early onset of sparse data problems. At the moment, we only have limited training data for Chinese data set (1M in size, 500 documents per class for training). If more training data were available, the higher order models may begin to show an advantage. For example, in the larger Japanese data set (average 7M size, 12,931 documents per class for training) we
1 2 3 4 5 6 7 8
Add-one Accu. F-Mac 0.33 0.29 0.66 0.63 0.77 0.68 0.74 0.51 0.69 0.42 0.66 0.42 0.64 0.38 0.62 0.31
Absolute Accu. F-Mac 0.33 0.29 0.66 0.62 0.75 0.72 0.81 0.77 0.83 0.77 0.84 0.76 0.84 0.75 0.83 0.74
Good-Turing Accu. F-Mac 0.34 0.29 0.66 0.61 0.75 0.72 0.81 0.76 0.83 0.76 0.83 0.75 0.83 0.74 0.83 0.73
Linear Accu. F-Mac 0.34 0.29 0.66 0.63 0.76 0.73 0.82 0.76 0.83 0.76 0.83 0.75 0.83 0.74 0.83 0.73
Witten-Bell Accu. F-Mac 0.34 0.29 0.66 0.62 0.75 0.72 0.81 0.77 0.83 0.77 0.84 0.77 0.84 0.76 0.84 0.76
observe an obvious increase in classication performance with higher order models (Table 4). However, here too, when becomes too large, overtting will begin to occur, as better illustrated in Figure 1.
0.8
0.7
Accuracy
0.6
Overall accuracy
0.5
0.4
Accuracy
0.3
0.2
1
2
3
Figure 1: Effects of order of -gram language models 5.3 Inuence of smoothing techniques
Smoothing plays an ...