n-gram A Large-scale N-gram Search System Using Inverted Files Susumu Yata, 1, 2 Kazuhiro Morita, 1 Masao Fuketa 1 and Jun-ichi Aoe 1 TSUBAKI 2)

Transcription

1 n-gram 1, n-gram n-gram A Large-scale N-gram Search System Using Inverted Files Susumu Yata, 1, 2 Kazuhiro Morita, 1 Masao Fuketa 1 and Jun-ichi Aoe 1 Recently, a search system has been proposed for large-scale n-gram data. The system uses tries for indexing n-gram data because of its retrieval performance. However, the disk usage and the construction cost of the trie index are very large. In contrast, inverted files could be easily constructed but require random access to secondary storage in retrieval. Therefore, this paper reports the actual performance of inverted files for large-scale n-gram data. 1. 1) 1 University of Tokushima 2 JSPS Research Fellow TSUBAKI 2) Wikipedia Google n-gram 3),4) n-gram 5) 7) 8) 9) 10) Document Frequency DF 11) n-gram n-gram 100 GB n-gram 1-gram 6) MySQL 9) n-gram Minimal Prefix MP 12) 13),14) 1-gram n-gram k-gram k C k/ gram 4) GB 15) n-gram n-gram 2 3 n-gram 4 1 c 2009 Information Processing Society of Japan

2 1 D Table 1 Word segmentation of documents in D ID Document Segmented words (separated by / ) 1 <S> / / / / / / </S> 2 <S> / / / / / / </S> 3 <S> / / / / / / </S> ID DocID 1 </S> 1, 2, 3 2 <S> 1, 2, 3 3 1, 2, 3 2 Table 2 An inverted index using segmented words ID DocID 4 1, 2, 3 5 1, 2, 3 6 1, 2, ID DocID n-gram D = {d 1, d 2, d 3 } ID ID DocID 2 ID ID ID DocID 2 DocID 1, 2, 3 DocID 1 d 1 DocID Variable Byte Code DocID 15) 2.2 n-gram n-gram 4) 1-gram 3 D 2-gram Table 3 2-gram data extracted from documents in D ID ID Frequency 1 2, 6 <S> 3 2 3, 1 </S> 3 3 4, , , 7 1 ID ID Frequency 6 5, , , , , 4 1 n-gram D 2-gram 3 1-gram n-gram DocID n-gram n-gram n-gram n-gram 3 Solid State Disk SSD 4 3. n-gram 3.1 n-gram n-gram 2 c 2009 Information Processing Society of Japan

3 n-gram 40% ( 1 ) 1-gram ID ( 2 ) n-gram ID ( 3 ) n-gram n-gram 1 n-gram ( i ) Dictionary lookup : ID ( ii ) Index lookup : n-gram ( iii ) N-gram lookup : n-gram ( iv ) Decode : n-gram Index lookup n-gram n-gram 1 N-gram database : n-gram n-gram Inverted index : N-gram database n-gram ID ID dictionary : ID ID 16) ID 3.2 n-gram n-gram n-gram n-gram Query Inverted index Fig. 1 Dictionary lookup ID Index lookup dictionary N-gram addresses Encoded n-grams N-gram lookup Decode 1 n-gram An n-gram search system using an inverted index N-grams N-gram database Search result n-gram DocID n-gram 1-gram n-gram n-gram n-gram 3 n-gram n-gram 2 1 Index lookup N-gram lookup Database lookup Database lookup : n-gram 2 Inverted database 1 Inverted Index N-gram database Inverted database : n-gram ID ID 3 c 2009 Information Processing Society of Japan

4 4 n-gram kb Query Dictionary lookup ID dictionary Database lookup Decode Encoded n-grams N-grams Search result Table 4 Disk usage of large-scale n-gram search systems (kb) Dic Inverted Idx N-gram DB Inverted DB Total (a) Uncompressed 76,818 29,757, ,221, ,055,762 (b) Compressed 148,126 26,969,347 29,033, ,151,370 (c) Extended 148, ,720, ,720,463 Fig. 2 Inverted database 2 n-gram An n-gram search system using an inverted database 5 Table 5 Average, median and maximum search time (s) 1-gram 3-gram 5-gram Ave Med Max Ave Med Max Ave Med Max (a) hdd (b) hdd (b ) ssd (c) hdd n-gram n-gram n-gram n-gram 64-bit Linux Google n-gram 4) 1-gram 7-gram 32 n-gram 3 4 ( a ) Uncompressed : n-gram ( b ) Compressed : n-gram ( c ) Extended : n-gram 1-gram, 3-gram, 5-gram 1,000 Core 2 Quad 2.83 GHz, 8 GB RAM, 1.5 TB HDDx2 (RAID 0), 256 GB SSD 24 HDD Compressed SSD SSD b SSD b n-gram SSD a b n-gram 40 % % b b HDD SSD % SSD b c b 3 4 c 2009 Information Processing Society of Japan

5 10 3 Fig. 3 Inverted indices 10-4 (a) Uncompressed (b ) Compressed Inverted database 10-4 (c) Extended gram n-gram Dependencies between the number of results and the search time for unigram queries Inverted indices (a) Uncompressed (b ) Compressed 10 3 Fig. 4 Inverted database (c) Extended gram n-gram Dependencies between the number of results and the search time for 5-gram queries c b 1/10 b n-gram c HDD SSD b 1-gram n-gram 3 a b n-gram c n-gram 5-gram 4 3 a b n-gram 13) 6 Table 6 6 Differences between an inverted database and a trie index 13) 1-gram 7-gram 32 7-gram GB 490 GB 1 GB 24 4 GB gram AND gram AND SSD n-gram 5. n-gram 5 c 2009 Information Processing Society of Japan

6 n-gram 13) 1) Christopher D. Manning and Hinrich Schütze. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, Massachusetts, London, England, 6 edition, ) Keiji Shinzato, Tomohide Shibata, Daisuke Kawahara, Chikara Hashimoto, and Sadao Kurohashi. TSUBAKI: An Open Search Engine Infrastructure for Developing New Information Access Methodology. In IJCNLP2008, pp , Hyderabad, India, January ) Thorsten Brants and Alex Franz. Web 1T 5-gram Version 1. upenn.edu/catalog/catalogentry.jsp?catalogid=ldc2006t13, ),. Web N 1. GSK2007-C/catalog.html, ) Andrew Finch, Etienne Denoual, Hideo Okuma, Michael Paul, Hirofumi Yamamoto, Keiji Yasuda, Ruiqiang Zhang, and Eiichiro Sumita. The NICT/ATR Speech Translation System for IWSLT In IWSLT 2007, pp , Trento, Italy, October ) Marcello Federico and Mauro Cettolo. Efficient Handling of N-gram Language Models for Statistical Machine Translation. In the 2nd Workshop on Statistical Machine Translation, pp , Prague, Czech Republic, June ) Jing Zheng, Wen Wang, and Necip Fazil Ayan. Development of SRI s Translation Systems for Broadcast News and Broadcast Conversations. In Interspeech 2008, Brisbane, Australia, September ) Ghinwa F. Choueiter, Stephanie Seneff, and James R. Glass. New Word Acquisition Using Subword Modeling. In Interspeech 2007, pp , Antwerp, Belgium, August ) Andrew Carlson and Ian Fette. Memory-Based Context-Sensitive Spelling Correction at Web Scale. In ICMLA 2007, pp , Cincinnati, Ohio, USA, December ) Steven Bethard and James H. Martin. Learning Semantic Links from a Corpus of Parallel Temporal and Causal Relations. In ACL-08: HLT, Short Papers (Companion Volume), pp , Columbus, Ohio, USA, June ) Martin Klein and Michael L. Nelson. Correlation of Count and Document Frequency for Google N-Grams. In ECIR 2009, pp , Toulouse, France, April ) John A. Dundas III. Implementing Dynamic Minimal Prefix Tries. Software Practice and Experience, Vol.21, No.10, pp , October ). N Google 7. 14, pp , March ) Satoshi Sekine. A Linguistic Knowledge Discovery Tool: Very Large Ngram Database Search with Arbitrary Wildcards. In Coling 2008: Companion volume: Demonstrations, pp , Manchester, UK, August ) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, ).., Vol. J71 D, No.9, pp , c 2009 Information Processing Society of Japan