Companion The Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, July 2003, pp. 121-124.
Automatic Collection of Related Terms from the Web
Satoshi Sato and Yasuhiro Sasaki Graduate School of Informatics Kyoto University Sakyo, Kyoto, 606-8501 Japan [e-mail], [e-mail]
corpus Cs
ATR
Abstract
This paper proposes a method of collecting a dozen terms that are closely related to a given seed term. The proposed method consists of three steps. The rst step, compiling corpus step, collects texts that contain the given seed term by using search engines. The second step, automatic term recognition, extracts important terms from the corpus by using Nakagawas method. These extracted terms become the candidates for the nal step. The nal step, ltering step, removes inappropriate terms from the candidates based on search engine hits. An evaluation result shows that the precision of the method is 85%.
a seed term s
Compiling corpus
the Web
related terms Filtering T
candidates X
Figure 1: System conguration Automatic acquisition of technical terms in a certain domain has been studied as automatic term recognition (Kageura and Umino, 1996; Kageura and Koyama, 2000), and the methods require a large corpus that are manually prepared for a target domain. In contrast, our system, which is proposed in this paper, requires only a seed word; from this seed word, the system compiles a corpus from the Web by using search engines and produces a dozen technical terms that are closely related to the seed word.
1
Introduction
This study aims to realize an automatic method of collecting technical terms that are related to a given seed term. In case natural language processing is given as a seed term, the method is expected to collect technical terms that are related to natural language processing, such as morphological analysis, parsing, information retrieval, and machine translation. The target application of the method is automatic or semi-automatic compilation of a glossary or technical-term dictionary for a certain domain. Recursive application of the method enables to collect a list of terms that are used in a certain domain: the list becomes a glossary of the domain. A technical-term dictionary can be compiled by adding an explanation for every term in the glossary, which is performed by term explainer (Sato, 2001).
2
System
Figure 1 shows the conguration of the system. The system consists of three steps: compiling corpus, automatic term recognition (ATR), and ltering. This system is implemented for Japanese language. 2.1 Compiling corpus The rst step, compiling corpus, produces a corpus Cs for a seed term s. In general, compiling corpus is to select the appropriate passages from a document set. We use the Web for the document set and select the passages that describe s for the corpus. The actual procedure of compiling corpus is:
1. Web page collection For a given seed term s, the system rst makes four queries: s toha, s toiu, s ha, and s, where toha, ha, and toiu are Japanese functional words that are often used for dening or explaining a term. Then, the system collects the top K (= 100) pages at maximum for each query by using a search engine. If a collected page has a link whose anchor string is s, the system collects the linked page too. 2. Sentence extraction The system decomposes each page into sentences, and extracts the sentences that contain the seed term s. The reason why we use the additional three queries is that they work efciently for collecting web pages that contain a denition or an explanation of s. We use two search engines, Goo1 and Infoseek2 . We send all four queries to Goo but only the query s to Infoseek, because Infoseek usually returns the same result for the four queries. A typical corpus size is about 500 sentences. 2.2 Automatic term recognition The second step, automatic term recognition (ATR), extracts important terms from the compiled corpus. We use Nakagawas ATR method (Nakagawa, 2000), which works well for Japanese text, with some modications. The procedure is as follows. 1. Generation of term list To make the term list L by extracting every term that is a noun or a compound noun from the compiled corpus. 2. Selection by scoring To select the top N (= 30) terms from the list L by using a scoring function. For the scoring function of a term x, we use the following function, which is multiplying Nakagawas Imp1 by a frequency factor F (x, L). score(x, L) = Imp1 (x, L) F (x, L) F (x, L) =
1 2
While Nakagawas Imp1 does not consider term frequency, this function does: is a parameter that controls how strongly the frequency is considered. We use = 0.5 in experiments. The result of automatic term recognition for (natural language processing) is shown in the column candidate in Table 1. 2.3 Filtering The ltering step is necessary because the obtained candidates are noisy due to the small corpus size. This step consists of two tests: technical-term test and relation test. 2.3.1 Technical-term test The technical-term test removes the terms that do not satisfy conditions of technical terms. We employ the following four conditions that a technical term should satisfy. 1. The term is sometimes or frequently used in a certain domain. 2. The term is not a general term. 3. There is a denition or explanation of the term. 4. There are several technical terms that are related to the term. We have implemented the checking program of the rst two conditions in the system: the third condition can be checked by integrating the system with term explainer (Sato, 2001), which produces a denition or explanation of a given term; the fourth condition can be checked by using the system recursively. There are several choices for implementing the checking program. Our choice is to use the Web via a search engine. A search engine returns a number, hit, which is an estimated number of pages that satisfy a given query. In case the query is a term, its hit is the number of pages that contain the term on the Web. We use the following notation. H(x) = the number of pages that contain the term x The number H(x) can be used as an estimated frequency of the term x on the Web, i.e., on the hugest set of documents. Based on this number, we can infer whether a term is a technical term or not: in case the number is very small, the term is not a
1 if x is a single noun frequency of x L in otherwise
www.goo.ne.jp www.infoseek.co.jp
Table 1: Result for natural language processing
candidate (natural langauge processing; NLP) (NLP technology) (NLP system) (NLP research) (NLP study) (processing) (text processing) (research and development) (Information Processing Society of Japan; IPSJ) (semantic processing) (speech processing) (speech information processing) (information processing) (NLP domain) (research eld) (parsing) (information retrieval) (SIGNLP) (speech recognition) (machine translation) (morphological analysis) (information processing system) (research) (semantic analysis) (chair of NLP) (NLP symposium) (application system) (knowledge information processing) (language) (information) Tech. Rel.
x is a broader or narrower term of s; or (2) relation degree between x and s is high enough, i.e., above a given threshold. The candidate terms can be classied from the viewpoint of term composition. Under a given seed term, we introduce the following ve types for classication. Type 0 the given seed term s: e.g., (natural language processing) Type 1 a term that contains s: e.g., (natural language processing system) Type 2 a term that is a subsequence of s: e.g., (natural language) Type 3 a term that contains at least a component of s: e.g., (language analysis) Type 4 others: e.g., (parsing) The reason why we introduce these types is that the following rules are true with a few exception: (1) A type-1 term is a narrower term of the seed term s; (2) A type-2 term is a broader term of the seed term s. We assume that these rules are always true: they are used to determine whether x is a broader or narrower term of s. To measure the relation degree, we use conditional probabilities, which are calculated from search engine hits. P (s|x) = P (x|s) = where H(s x) = the number of pages that contain both s and x One of two probabilities is equal to or greater than a given threshold Z, the system decides that x is closely related to s. We use Z = 0.05 as the threshold. In summary, our relation test is: If x is type-1 or type-2; or P (s|x) 0.05 or P (x|s) 0.05 then x is closely related to s. The result of the ltering step for (natural language processing) is in Table 1; a H(s x) H(x) H(s x) H(s)
*
*
technical term because it does not satisfy the rst condition; in case the number is large enough, the term is probably a general term so that it is not a technical term. Two parameters, M in and M ax, are necessary here. We have decided that we use search engine Goo for H(x), and determined M in = 100 and M ax = 100, 000, based on preliminary experiments. In summary, our technical-term test is: If 100 H(x) 100, 000 then x is a technical term. 2.3.2 Relation test The relation test removes the terms that are not closely related to the seed term from the candidates. Our conditions of x is closely related to s is: (1)
Table 2: Experimental Result
domain natural language processing Japanese language information technology current topics persons in Japanese history Total Evaluation I correct incorrect 101 (93%) 8 ( 7%) 71 (81%) 17(19%) 113 (88%) 15 (12%) 106 (91%) 10 ( 9%) 128 (76%) 41 (24%) 519 (85%) 91(15%) total 109 88 128 116 169 610 S 6 7 10 2 18 43 F 3 0 5 0 0 8 Evaluation II A C R 14 11 8 19 5 1 27 13 0 13 19 5 23 1 0 96 49 14 total 43 32 55 39 42 210
check mark indidates that the term passed the test. Twenty terms out of the thrity candidate terms passed the rst techinical-term test (Tech.) and sixteen terms out of the twenty terms passed the second relation test (Rel.). The nal result includes two inappropriate terms, which are indicated by *.
C: the target term existed in the collected web pages, but did not exist in the compiled corpus. R: the target term did not exist on the collected web pages. Only 43 terms (20%) out of 210 terms were collected by the system. This low recall primarily comes from the failure of automatic term recognition (case A in the above classication). Improvement of this step is necessary. We also examined whether each of the 210 target terms passes the ltering step. The result was that 133 (63%) terms passed; 44 terms did not satisfy the condition H(x) 100; 15 terms did not satisfy the condition H(x) 100, 000; and 18 terms did not pass the relation test. These experimental results suggest that the ATR step may be replaced with a simple and exhaustive term collector from a corpus. We have a plan to examine this possibility next.
3
Experiments and Disucssion
First, we examined the precision of the system. We prepared fty seed terms in total: ten terms for each of ve genres; natural language processing, Japanese language, information technology, current topics, and persons in Japanese history. From these fty terms, the system collected 610 terms in total; the average number of output terms per input is 12.2 terms. We checked whether each of the 610 terms is a correct related term of the original seed term by hand. The result is shown in the left half (Evaluation I) of Table 2. In this evaluation, 519 terms out of 610 terms were correct: the precision is 85%. From this high value, we conclude that the system can be used as a tool that helps us compile a glossary. Second, we tried to examine the recall of the system. It is impossible to calculate the actual recall value, because the ideal output is not clear and cannot be dened. To estimate the recall, we rst prepared three to ve target terms that should be collected from each seed word, and then checked whether each of the target terms was included in the system output. We counted the number of target terms in the following ve cases. The right half (Evaluation II) in Table 2 shows the result. S: the target term was collected by the system. F: the target term was removed in the ltering step. A: the target term existed in the compiled corpus, but was not extracted by automatic term extraction.
References
Kyo Kageura and Teruo Koyama. 2000. Special issue: Japanese term extraction. Terminolgy, 6(2). Kyo Kageura and Bin Umino. 1996. Methods of automatic term recognition: A review. Terminology, 3(2):259289. Hiroshi Nakagawa. 2000. Automatic term recognition based on statistics of compound nouns. Terminology, 6(2):195210. Satoshi Sato. 2001. Automated editing of hypertext r sum from the world wide web. In Proceedings e e of 2001 Symposium on Applications and the Internet (SAINT 2001), pages 1522.