Traditional statistical approaches for identifying multi-word terms have to handle a large amount of noisy data and are extremely time consuming. This paper presents a new statistical approach for identifying multi-word terms based on the co-related text-segments existing in a group of documents. The approach involves three stages: (a) using a short predefined stoplist as an initial input to segment a set of text documents into text-segments, (b) calculating the segment-weights of all text-segments, and (c) applying the short text-segments to segment the longer text-segments based on the weight values. The newly generated text-segments then segment each other again until all text-segments cannot be further divided. The resultant text-segments are identified as terms based on a specified threshold. The initial experimental result on a set of traditional Chinese documents shows that this approach can achieve a minimum of 76.39% of recall rate and a minimum of 91.05% of precision rate on retrieving multiple occurrences terms, including 18.30% of new identified terms. The approach can be applied to identify multi-word terms in any languages.
|Title of host publication||Seventh International Conference on Web-Age Information Management Workshops, WAIM 2006|
|Publication status||Published - 1 Dec 2006|
|Event||7th International Conference on Web-Age Information Management Workshops, WAIM 2006 - Hong Kong, China|
Duration: 17 Jun 2006 → 19 Jun 2006
|Conference||7th International Conference on Web-Age Information Management Workshops, WAIM 2006|
|Period||17/06/06 → 19/06/06|