Clustering of Text and Image for Grouping Similar Contents

Transcription

1 University of Aizu, Graduation Thesis. August, 2003 s Clustering of Text and Image for Grouping Similar Contents of Web Data Keigo Hirai s Supervised by Prof. Ryuichi Oka Abstract 2 System for Clustering of Text and Image The object of this paper is to design a searching system that uses relation between two vectors. The data files are random text (Hyper Text Markup Language, HTML) and image files. This paper explains how the data files were classified and retrieved. 1 Introduction This paper supports searching system that uses relation between two vectors. The Google image searching system searches images with text keywords that are near the tag< img... > and the image captions [1]. This paper s image searching system is more effective than the Google one. The system uses image vector information and text vector information. The system can search for images from input text or image as well as texts from input text or image. The position of this paper is between word analysis and image processing. Word analysis is needed to make text feature vectors from texts. This paper uses KAKASI as word analysis method. KAKASI is abbreviation of KAnji KAna Simple Inverter. Word analysis method is included the invert process of KAKASI [2]. Image processing is needed to make image feature vectors from images. The Prewitt method is adopted this research as edge analysis method [3]. This searching system needs a clustering method for retrieving object data. K-means clustering is grouping method that arranges similar data files close groups [4]. Images that are used at web sites on mostly Joint Photographic Experts Group (JPEG) image. JPEG images adopted for this research too. JPEG image data is stored 2-dimension arrays. Image processing processes the RED, GREEN, and BLUE (RGB) values of the 2- dimension array. In this research program, a dynamic link library (DLL) imgctl.dll was adopted as the JPEG image loading method [5]. Figure 1: Outline of the system for multimedia data organization and retrieval Figure 1 shows this research system. The raw data 1 of images and text is hard to process for computer. The need for feature vector extraction stems from this raw data property. Feature vectors are easy to process for computers. 2.1 Feature Vector Extraction from Image Data There are several ways to get image feature vectors, such as color feature vectors, edge feature vector, and position feature vectors. Color feature vectors and edge feature vectors were adopted for this research Color Feature Vector Extraction 1. Separate the image into 64 cells. The cell width is image width image height 8. The cell height is Compute the RGB mean value of the respective cells. The value range is from 0 to non-processed original data

2 University of Aizu, Graduation Thesis. August, 2003 s Quantized value Original value 0 (0-31) 1 (32-63) 2 (64-95) 3 (96-127) 4 ( ) 5 ( ) 6 ( ) 7 ( ) M k is a mask element value. M 0 corresponds to the first element of the mask. I k is an image intensity value under the mask. The edge direction at the pixel is the mask direction that has maximum MD value. 4. Run above process for per pixel. Table 1: Quantize table for color feature vector extraction 3. Quantize the RGB mean value (0-255) to 8 levels (0-7). The quantized value is computed at the respective cells (Table 1). 4. Prepare the 24-dimension array Vc(R 0,..., R 7, G 0,..., G 7, B 0,..., B 7 ) to store color data. 64 cells are processed with the above 2 and 3 rules. The specific process is follows: (a) If the RGB mean value is (R, G, B) = (10, 150, 255), the quantized value is (0, 4, 7) (b) Add 1 to R 0, G 4, B 7 (c) Process these quantizations and increment all 64 cells. Those processes extract the 24-dimension color feature vector Vc. 5. Apply the above method to all image files (JPEG) under a directory (include subdirectories) Edge Feature Vector Extraction The Prewitt method, a method of Template Matching, is used in the edge feature extraction process. It uses the kinds 3 x 3 pixel masks. Those masks detect where edge direction is. The method detects edge direction with the 3 x 3 pixels mask that moves a pixel at a time. 1. Change the image into gray-scale. This research considers the image intensity element. 2. Prepare eight 3 x 3 pixels masks that correspond to eight directions respectively (Figure 2). 3. Compute the matching degree (MD) for the eight masks (eight directions) as follows. (i=mask identifier (0-7) ). MD i = 7 M k I k k=0 Figure 2: Mask set for Prewitt method 5. Separate the image into 80 x 80 blocks for extracting the broad edge vectors. 6. Compute the image intensity mean value for the respective blocks. In broad edge vector extraction, the Prewitt method regards a block as a pixel. 7. Prepare the array of the number of edge directions frequency Ve(E 0, E 1,..., E 15 ). (E 0,..., E 7 ) are elements for storing the number of edge direction frequencies. (E 8,..., E 15 ) are elements for storing the number of broad edge direction frequencies. 8. Apply the above method to all image files (JPEG) under a directory (include subdirectories). The image feature vector is defined as Vi(I 0,..., I 31 )= (Vc, Ve). (Vi is 40-dimension. Color feature vector Vc is 24-dimension. Edge feature vector Ve is 16- dimension.) All image feature vectors (Vi) under a directory are recorded to an image feature data file. The data file is prepared for clustering. 2.2 Feature Vector Extraction from Text Data The text feature vector Vt(word 0,..., word n ) is an array of word frequency within the text (HTML). n is the num-

3 University of Aizu, Graduation Thesis. August, 2003 s ber of the word type. The first element of array (word 0 ) is first word frequency within the text. The KAKASI project members had researched originally the KAKASI program for transforming Japanese from kanji to hiragana, katakana, and Romaji. (Romaji is Romanized Japanese pronunciation.) The KAKASI program uses a function that divides text into words. Extract method of text feature vector is written as follows. 1. Get rid of HTML tags<... > and comments <! > from input text. 2. Insert spaces between words with a KAKASI function. 3. Prepare a linked-list of structures which has string data (word) and integer data (frequency). 4. Add a word of text to a linked-list node in occurrence order. If a word already exists within the linked-list, the occurrence number of the word is added All text files (HTML) under a directory (obtain subdirectories) are processed by above method. The text feature vector dimension becomes larger by reading new text file. This dimension number is the word kind number of whole texts. (In practice, the dimension number is 3658 for 118 text files.) This large dimension vector is hard to handle for clustering. 6. Set the appropriate high value and low value to extract useful middle word data. Words that occurs too frequently and words that occurs too infrequently are cut with the high value and the low value. The example of the word that occurs too frequently is basic verb, kakuzyosi 2, period ( ), and comma ( ). They are cut because of the meaninglessness. Words occurs too infrequently have the possibility that include many kinds of part of speech. Most of those words have meaning themselves, but they are not useful for clustering because of the small frequency. Words occurs too infrequently has weak relation to other texts. 7. Cut words that occur too frequently and words that occur too infrequently from the linked-list. Figure 3 shows the linked-list elements after cutting. 2 the Japanese part of speech that shows the relation between word. For example, (ga), (no), (wo), (ni), (he), (to), (yori), (kara), and (de). It resembles English preposition. Figure 3: The part of the linked-list elements after cutting This figure shows text feature vectors frequency of the web data under a directory. Left elements are word type. Right elements are word frequency. 8. Recount the word frequency for respective texts. The counter notes only the linked-list words. The recounted number array is the text feature vector. 9. Apply the above method to all text files (HTML) under a directory (include subdirectories). All text feature vectors (Vt) under a directory are recorded to a text feature data file. The data file is prepared for clustering. 3 Clustering of Two Types of Feature Vectors Both feature vectors, image feature vectors and text feature vectors, are classified by k-means clustering method by using the image feature data file and the text feature data file. The k-means method is defined as follows. 1. Set random K initial center points of class. 2. Classify all data to clusters. If there is a center point of class nearest for a data, this data is classified to this class. 3. Move the center point of class to the new class s center of gravity. 4. If the all new positions of center points of class equal to old ones, the process ends. If not, the process goes back to 2. The k-means method gathers resemble data to a resemble class. Those data are feature vector set written in feature data file. Image feature data file and text feature data file are classified individually.

4 University of Aizu, Graduation Thesis. August, 2003 s Retrieval Algorithm of Multimedia Data 4.2 Text Query The retrieval method in searching texts by a text is follows: 1. Extract the input text s feature vector 2. Examine the input text class 3. Texts in the class are results. The retrieval method in searching images by a text is follows: Figure 4: Outline of retrieval algorithm of multimedia data Figure 4 shows outline of retrieval. This section explains the retrieval way to get results when the keyword is entered. There are 4 retrieval routes. Classified data are prepared already in this section. When the system gets the text from image keyword, when the system gets the image from text keyword, this situation needs bridge data that explains the relation between texts and images. Bridge data are constructed the set of image file names (src=...jpg ) within img tag in the HTML. The bridge data are recorded to a bridge data file. 4.1 Image Query The retrieval method in searching images by an image is follows: 1. Extract the input image s feature vector 2. Examine the input image class 3. Images in the class are results. The retrieval method in searching texts by an image is follows: 1. Extract the input image s feature vector 2. Examine the input image class 3. Select another image of the class 4. A (Several) class (classes) of texts is (are) determined by the selected image using by text-image bridge data 5. Texts in the class (classes) are results. When several classes of texts are determined, the input image is used several HTML. 1. Extract the input text s feature vector 2. Examine the input text class 3. Select another text of the class 4. A (Several) class (classes) of images is (are) determined by the selected text using by text-image bridge data 5. Images in the class (classes) are results. When several classes of images are determined, the input text (HTML) uses several JPEG images. 5 Experimental Results This section shows results of the searching system. Web data (130 texts (HTML) and 91 images) are prepared for this searching system. The number of the image class and the text class is twelve in this paper. This number (twelve) are determined by this research experience. They are retrieved by four methods. Those web data are completely same as Mr. Yanagita s paper s web data. 5.1 Image Query Mr. Yanagita s (s ) paper shows image query results. The paper title is Extraction of Text-Image Correlation for Web Image Data with Text. 5.2 Text Query The web data have words, 5221 word types. The low value for cutting is decided 10. The high value for word types cutting is decided 30 = 174. This text feature has 215 elements, in other words, this feature dimension is 215.

5 University of Aizu, Graduation Thesis. August, 2003 s Text Query Text Results Figure 5: Example of the web page having only the image and caption (text results for the short text query) Short texts (HTML) tend to gather at the same class. Those HTML have the strong possibility of the web page having only the image and caption (Figure 5), in practice, all such web pages of this research data gather at the same class. Therefore, if the query text length is short, text results (HTML) are like such web pages. Figure 7: Example of the web page having the word kamosika (text results for the long text query with figure 6) Long texts (HTML) are classified by word frequency (Figure 6). Therefore, if the query text is long and has the word kamosika 3, text results (HTML) have the possibility of having the word kamosika (Figure 7), in practice, such web pages tend to gather at the same class Text Query Image Results The input texts is used figure 6. The image results are follows. Figure 8: Retrieved nine images by the input text query shown in figure 6 Figure 6: Example of long texts word frequency (input text) Three pictures of kamosika, (b), (f), and (h), are retrieved. Six pictures of monkey, (a), (c), (d), (e), (g), and (i), are retrieved too, because colors and edges of monkey picture resemble to kamosika one. 3 Japanese of serow

6 University of Aizu, Graduation Thesis. August, 2003 s Discussion of the Text Query Short texts tend to position the center point of vector space because of those Euclidean distances are so shorter than long texts one. Those searching system results depend on the word type and the text length too. The text length is one of text features, so this system supports the text length feature without notice. This property will contribute to this system improvement. 6 Conclusion This paper presented the searching system that uses relation between two vectors. This system realizes the retrieving system that superiors to two data types ( Images and Texts ). The old searching system supports the combination of words as keyword. This system supports images and texts as keyword. This improvement supports the fuzzy retrieving. This property helps to search wide relational informations. The additional improvements can be made to this searching system in the future. 6.1 Future work Images clustering is constructed in this research completely, but programs of the texts clustering and the retrieving system are not constructed completely. The program of texts clustering has a bug to stop reading from large-scale number of HTML. Above results are produced by the handwork. The future works for this research are follows: Acknowledgment I want to thank Prof. Oka, Mr. Suenaga, Mr. Murakami and Mr. Yanagita for their helpful comments and programs during this research. I use Mr. Yanagita s image processing program in this research. References [1] Google, Google Frequently Asked Questions: Image Search, images.html (current Jan. 2003) [2] KAKASI project, KAKASI - Kanji Kana Simple Inverter, (current Dec. 2002) [3] S. Inoue, N. Yagi, M. Hayashi, E. Nakasu, K. Mitani, M. Okui, C gengo de Manabu Jissen Gazoushori, Ohmsha, [4] H. Kasuga, Color Quantization Using Fast K- means Algorithm, kasuga/kmeans.html (current Jan. 2003) [5] Ruche, Ruche s Homepage, (current Jan. 2003) Construct the complete programs of the texts clustering and the retrieving system Prepare the user friendly GUI (Graphical User Interface) system for retrieve Expand the destination data from local data to Internet data Visualize web data by using a classification method Adopt audio data as keyword and retrieve destination. Adopt texts in images as bridge data. Adopt the position feature as the image feature vector. If above works will be researched, this searching system will be more useful for the people all around the world.