Subjective Feature Space

Transcription

1 Learning of Personal Visual Impression for Image Database Systems akio Kurita oshikazu Kato Mathematical Informatics Section Interactive Interface Systems Section Information Science Division Intelligent Systems Division Electrotechnical Laboratory Electrotechnical Laboratory Umezono, sukuba, 305 JAPAN Umezono, sukuba, 305 JAPAN Abstract Visual impression may dier with each person. User-friendly interfaces for image database systems require special retrieval methods which can adapt to the visual impression of each user. his paper describes algorithms for learning personal visual impression on visual objects. he algorithms are based on multivariate data analysis methods. hese algorithms provide a model on visual perception process of each user from a small set of training examples. his model is referred to as a personal index to retrieve desired images for the user. hese algorithms were implemented and examined in our graphical symbol database systems called RADEMARK and our full color image database called AR MUSEUM. 1. Introduction An intelligent image database management which provides a content based data operation is one of the key issues of future multimedia information systems. In conventional database systems, query languages, such as SQL, have been the only interface to retrieve desired data from the database. Even in image database systems, users have to describe the desired image data in query languages with the combination of index terms, which are assigned to each image in the database by the database manager. However, it is dicult to describe the content of the image properly only by index terms. Several visual interfaces have been proposed to provide visual query mechanisms from the viewpoint of database schema. For example, SNAP [1] and QPE [2] provide visual query in an ER diagram form and in a tabular form, respectively. he hypermedia-type image browser [3] navigates the user into the image database space following the links assigned by the database manager. In the icon based system [4], icons are used not only as the visual index of the image data but also as the element of the visual query language. Although these approaches gave new ideas on GUI, we have to meet another sort of problems in content based image retrieval. Visual impression may dier with each person. User-friendly interfaces for image database systems require special retrieval methods which can adapt to the visual impression of each user. If we try to attach these facilities on a conventional database system, each user, or a database manager, has to assign suitable keywords to all the image data in the database at rst. hus, when a new image is registered into the database, each user has to assign suitable keywords to the image again. If the keyword index is managed only by the database manager, he has to understand the subjective view, i.e. the way of thinking and the way of giving keywords, of each user. It is hardly a possible way. We need a new idea on understanding the subjective view of each user. We have been developing a graphical symbol database called RADEMARK [5, 6] and a full color image database called AR MUSEUM [7, 8, 9]. hose systems provide the following facilities in their visual interface. 1. he systems accept an instance image in a query as an example. 2. he systems provide a learning mechanism to adapt the visual impression of each user. his paper describes the algorithms for learning subjective view of each user adopted in these image database systems. In our applications, a subjective view means the way of feeling visual impressions on image data, such as graphical symbols and artistic paintings. We have developed three learning algorithms which refer the small set of user-given examples. he algorithms are based on multivariate analysis methods. hese algorithms create an approximated model of visual perception process of each user from a small set of training examples. he model is referred to as the personal index of the user in order to evaluate the user's query and to retrieve the target image data. Section 2 outlines RADEMARK and AR MU- SEUM systems. hen, section 3 describes each of the learning algorithms. Section 4 shows some results of experiments of content based image retrieval. 2. Image Database Systems 2.1. RADEMARK he RADEMARK 1 database is a collection of graphical symbols and designs. Currently, about 2,000 1 RAdemark and DEsign database with Multimedia Abstracted image Representation on Knowledge base

2 Figure 1. Examples of graphical symbols. graphical symbols are stored in the database as binary image data. Figure 1 shows some examples of graphical symbols. At the Patent Oce, examiners compare all of the applied gures with tens of thousands of existing registered graphical symbols. his burdensome task can be avoided if an image database system accepts a query by visual example (QVE). In the RADEMARK system, a user can provide a visual example as a key image through an image scanner. he system evaluates similarities between graphical features of the visual example and those of graphical symbols in the database. he system retrieves similar graphical symbols with the key image according to the similarity values. We have used the following graphical feature which characterize graphical symbols. Spatial distribution of the black pixels: he distribution of black pixels represents the outline of the graphical symbol. he numbers of black pixels of a binary image in the meshes give an approximation of the outline. Spatial frequency: he spatial frequency is a measure of the complexity of graphical symbols. We will approximate this by a run-length distribution in the 4 meshes. Spatial structural features: he local correlation and contrast show the spatial relationships such as relative position of partial gures. We will approximate this by local correlation and contrast of black pixels among meshes. hese graphical features are extracted from each graphical symbol stored in the database. hey are parameterized in a vector form on a multidimensional vector space. We refer to such a vector space as a graphical feature (GF) space. We expect that graphical symbols which look similar with each other may have similar graphical features and they will be mapped to the near points in the GF space. For example, a ne copy of a graphical symbol and its rough sketch may be mapped into the near points in the GF space. In the RADEMARK system, a user can retrieve image data which have similar graphical features by showing a rough sketch to the system. We call this style of information retrieval sketch retrieval. his is suitable when a user wants to retrieve image data whose graphical shapes are similar with the key image. However, such a GF space gives an objective criterion to describe the graphical symbols. We experience that the impression of similarity sometimes dier with each person while viewing the same graphical symbols. It means that the similarity measure on visual objects varies with each user, which is not reected in the GF space. o manage such subjectivity of visual impressions, we need another mapping mechanism for subjective indexing. he RADEMARK system also provides learning facility from a small set of training examples to construct such additional mapping from the GF space to the subjective feature (SF) space of each user. his mapping enables the evaluation of similarity between the key image and an image in the database on SF space where visual impression of each user is reected. We call this style of information retrieval similarity retrieval AR MUSEUM he AR MUSEUM 2 database is a collection of full color paintings of landscapes. Currently, about 200 impressionist paintings are stored in the database. he AR MUSEUM system accepts a query by visual example (QVE) and a query by subjective descriptions (QBD). In QVE, a user can provide a rough sketch as a key image. he system evaluates the similarity between the sketch and each image in the database. According to the similarity values the system displays the paintings whose general composition is similar with that of the key image. he similarity measure is dened on the matching score of edge images [10, 11]. In QBD a user gives artistic impressions by a combination of adjectives to retrieve desired paintings. While visual impressions on artistic paintings originate from a motif, a general composition, a color arrangement, and so on, Chijiiwa, et al, found that coloring is dominant through their many psychological experiments and analyses [12]. hus, we may expect there is a reasonable correlation between the coloring of a painting, user's visual impression and the subjective description by words. Since such impressions may dier with each user, the system should analyze and learn the correlation, with each user, between the subjective descriptions and the image data. However, we cannot directly compare the subjective descriptions and the coloring feature of paintings in the database because they belong to dierent domains. We have represented the subjective descriptions as binary vectors; adjective feature (AF) vectors. he number of elements of an AF vector is equal to the number of adjectives available in the system. Each element is set to 1 if the corresponding adjective is included in the subjective description, and 0 otherwise. Currently, the system can treat 30 adjectives listed in table 1. he coloring feature is represented as a graphical feature (GF) vector calculated from each image. In the AR MUSEUM system, we have used the follow- 2 MUlitimedia database with SEnse of color and construction Upon the Matter of AR

3 able 1. Adjectives available in AR MUSEUM. clear gorgeous fresh ethnic rened dynamic & active modern soft authentic heavy dandy warm classic cool natural sober chic bright elegant/tasteful quiet elegant/relaxed hard romantic clean pretty sporty casual/relaxed Japanesque casual/pleasant country No.1 * 1 1 No.2 * * 1 No.4 No.3 1 * * No.5 Figure 2. Patterns of the displacements for local autocorrelations. Classifications of training images ing graphical features which are based on local autocorrelations. Let be an image ff(i; j) = (r(i; j); g(i; j); b(i; j)) ji 2 I; j 2 Jg; where r(i; j), g(i; j), b(i; j) denote the red, green and blue components of the pixel at (i; j), respectively. hen 0-th order autocorrelation is dened as! r X g = 1 f(i; j): (1) b IJ i2i;j2j his consists of three components that are the averages of red, green and blue signals. he rst order autocorrelations with displacement a = (a i ; a j ) are dened as r rg rb! rg g gb X = 1 rb gb b IJ i2i;j2j (2) While there are many rst order autocorrelations depending on the displacement, we restrict the range of displacements within a local window. By eliminating the displacement which are equivalent by shifting, we can reduce the number of the patterns of the displacements to only ve shown in gure 2. he rst order autocorrelations are 30 features. hus the GF vector consists of these 33 features. o adjust to the visual impressions of each user, we have developed a method to construct a unied feature (UF) space, such as to maximize the correlation between AF vectors and GF vectors of training samples of each user. he method is based on canonical correlation analysis (CCA). On the UF space, we can expect that a painting and words to describe the visual impression on the painting are mapped into the near points. hus, a user can retrieve a picture giving some subjective descriptions imaging the picture. We call such retrieval method sense retrieval. f(i; j)f(i+a i ; j+a j ) : Discriminant Alalysis Subjective Feature Space Figure 3. Learning from classications. 3. Learning Algorithms 3.1. Learning from Classications o adjust to the visual impression of each user for similarity retrieval, we construct a mapping from the GF space to the SF space in which the subjective similarity of each user is reected. hus we have to collect data on the subjective similarity of each user. A user can easily classify the graphical symbols, which are selected from the database as the training set, into several clusters judging similarity. From this classication of the training images we can construct a mapping from the GF space to the SF space (Figure 3). Let the training samples of graphical symbols from the RADEMARK database be G = fg i ji = 1;... ; N g and the GF vectors of each sample be ff i ji = 1;... ; N g. Suppose that the training samples are classied by a user into K groups fc1;... ; C K g. For the simplicity, we will consider to construct the following linear mapping from the GF space to the SF space s = A (f 0 f ); (3) where A denotes the transpose of the coecients matrix A and the total mean vector of GF vectors is given by f = 1 N f i : (4)

4 he coecients matrix A is determined such that in SF space the discriminant criterion J = tr(^6 01 W ^6 B ) (5) Similarities between training images is maximized, where ^6 W and ^6 B are the within- and between-class covariance matrices dened on the SF space. his criterion takes large value when SF vectors in the same class are close and SF vectors of dierent classes are away. he optimal coecients are obtained by solving the following eigen equation 6 B A = 6 W A3 (6) A 6 W A = I; where 3 is a diagonal matrix of eigenvalues and I denotes the unit matrix and 6 B = KX! k x k x k 0 x x (7) k=1 KX 6 W =! k 6 k k=1 6 k = E Ck xx 0 x k x k (k = 1;... ; K) x k = E Ck x (k = 1;... ; K)! k = N k N (k = 1;... ; K): Once the system has learned the linear mapping of eq. (3) for the user, it can automatically construct the SF vectors of all graphical symbols in the database from their GF vectors. he SF space and the SF vectors are referred to in similarity retrieval as the personal index of the user. he SF space is constructed such that the classication result of the user is approximated by the linear mapping as much as possible. We can expect that the system can retrieve similar graphical symbols by searching those whose SF vectors are close to the SF vector of the key image Learning from Similarities In the previous subsection, we have described a learning algorithm from classication results of a set of training images. Next, we will consider the case where we can obtain similarities between all pairs of the training images. We have to construct a mapping from the GF space to the SF space in which similarities given by the user are approximated as much as possible (Figure 4). Let the similarity value given by a user between the images g i and g j in the set of the training images G be e ij, (e ij = e ji ; 0 e ij 1; e ii = 1; i; j = 1;... ; N) and the matrix which consists of the elements e ij be E. Again we will use a linear model s = A (f 0 f ): (8) hen the similarity between the images g i and g j in G can be evaluated by the inner product s i s j of the Subjective Feature Space Figure 4. Learning from similarities. SF vectors s i and s j. o reect the user's similarity values e ij in the SF space as much as possible, we have maximized the following criterion Q = e ij s i s j: (9) j=1 he optimal coecients are given by solving the following eigen equation where 6 E A = 6 F A3 (10) A 6 F A = 3 6 E = [F 0 1 f ] E[F 0 1 f ] (11) 6 F = [F 0 1 f ] [F 0 1 f ] F = [f1;... ; f N ] 1 = (1;... ; 1) : Once this linear mapping has learned for a user, the personal index of the user can be constructed by applying this mapping to the GF vectors of all graphical symbols in the database. he SF space is constructed such as to reect similarities given by the user for pairs of the training images as much as possible. We can expect that the system can retrieve similar images in the SF space based on the user's subjective criterion Learning of Correlation o adjust to the visual impression of each user for QBD in the AR MUSEUM system, we construct a UF space by canonical correlation analysis (Figure 5). Let the training images from the AR MUSEUM database be G = fg i ji = 1;... ; N g and the GF vectors of each image be ff i ji = 1;... ; N g. he AF vector assigned by a user to the image g i is denoted as h i. hen the following linear mappings are constructed s i = A f i t i = B h i (12)

5 Subjective Description raining Images Canonical Correlation Analysis Unified Feature Space Figure 5. Learning of correlation. Figure 6. An example of similarity retrieval in RADEMARK database. such that the correlation between s i and t i is maximized. We can use one of these as a UF space for sense retrieval. he optimal coecients matrices A and B are given as the solution of the following eigen equations R F H R 01 H R HF A = R F A3 2 R HF R 01 F R F HB = R H B3 2 (13) where 3 2 is a diagonal matrix of eigenvalues and R F = R H = R F H = f i fi h i h i f i h i = RHF : (14) here are the following relations between the matrices A and B R F H B = R F A3 R HF A = R H B3: (15) hen the statistics related with s or t are given by R S = R = R S = s i s i = A R F A = I L t i t i = B R H B = I L s i t i = 3 = R S : (16) hus the linear regression from s to t is given by ~ t = 3s; (17) and the linear regression from t to s is ~s = 3t: (18) Once the system has learned these linear mappings for a user, the system can compute the UF vectors of all images in the database from their GF vectors. hey are used as the personal index of the user for QBD. Since the UF space is constructed such as to maximize the correlation between the subjective descriptions and the colorings of the images in the training samples, it is expected that the system can retrieve desired images from the subjective description by searching the images whose UF vectors are close to the corresponding UF vector of the query. We can use the UF space for other purposes. For example, we can retrieve paintings that gives the user a similar impression by showing a painting as a visual key. It also can be used to estimate the adjectives suitable for an unknown painting. 4. Experiments 4.1. Similarity Retrieval We can expect that the neighboring graphical symbols on the personal index are similar patterns for the user. In similarity retrieval, the system searches for graphical symbols which have similar values on the SF space, i.e., the personal index. hen the system shows them as candidates in the descending order of the similarity values. We have used classication results of the 227 training images for learning. he candidates were searched out of about 1600 images in RADEMARK database. For the experiments using training images as key, the system retrieved at least one similar graphical symbol with a precision ratio of 99%. Figure 6 shows an example of similarity retrieval in the RADEMARK database for a user. he fteen candidates appear in this gure. he symbol at the top left is the key image. he symbol at left in the second line is the rst candidate. his gure also shows the graphical symbols classied by the user in the same group with the key image on the right. In this case, the system

6 (a) (b) Figure 7. Example of sense retrieval in AR MU- SEUM database. retrieved these eight symbols in the best sixteen candidates. We can also nd other similar symbols, such as the tenth, twelfth, and thirteenth candidates, which are not included in the training samples Sense Retrieval We have evaluated the algorithm described in the subsection paintings, out of 200 paintings, were selected as the training pictures from the AR MU- SEUM database. he UF space was constructed according to the average values of visual impressions of female examinees. Figure 7 (a) shows the best eight candidates for the adjectives cool, clear, clean and Figure 7 (b) shows the result for the adjectives warm, romantic, soft. hese paintings roughly satised the visual impressions of the examinees. 5. Summary his paper described learning methods of personal visual impression for content based image retrieval facilities. hese algorithms create a model of visual perception process of each user. Such a model is referred to as a personal index to retrieve images for the user. his paper showed how to construct linear models as the rst approximations of the personal visual perception process. his idea can be easily extended to nonlinear models by using neural networks. References [1] Bryce, D. and Hull, R.: \SNAP: A Graphic-based Schema Manager," Proc. of Int. Conf. on Data Engineering, pp , [2] Chang, N.S. and Fu, K.S.: \Query-by-Pictorial Example," IEEE rans. on Software Engineering, Vol.SE-6, No.6, pp , [3] Kasahara, H., Onomura, Y. and Kishimoto,.: \Image Mnemonics by a Computer Visual hinking Augmentation," Proc. of Int. Symp. Computer World'89, pp.63-70, [4] Chang, S.K., Yan, C.W., Dimitro, D.C. and Arndt,.: \An Intelligent Image Database System," IEEE rans. on Software Engineering, Vol.SE-14, No.5, pp , [5] Kato,., Shimogaki, H. and Fujimura, K.: \RADE- MARK: Multimedia Image Database System with Intelligent Human Interface, \ rans. of IEICE, Vol.J72- D-II, No.4, pp , 1989 (in Japanese). [6] Kurita,., Shimogaki, H. and Kato,.: \A Personal Interface for Similarity Retrieval on an Image Database System," rans. of IPSJ, Vol.31, No.2, pp , 1990 (in Japanese). [7] Kato,., Kurita,. and Sakakura, A.: \Electronic Art Museum: Full Color Image Database with Visual Interaction on Color and Sketch," IEICE ech. Rep., IE88-118, pp.31-38, 1989 (in Japanese). [8] Kato,. and Kurita,.: \Visual Interaction with Electronic Art Gallery," Proc. of Int. Conf. on Data Base and Expert Systems Application DEXA'90, [9] Kurita,., Kato,., Fukuda, I. and Sakakura, A.: \Sense Retrieval on a Image Database of Full Color Paintings," rans. of IPSJ, Vol.33, No.11, 1992 (in Japanese). [10] Hirata, K. and Kato,.: \Query by Visual Example," Proc. of Int. Conf. on Extending Database echnology EDB'92, pp.56-71, [11] Kato,., Kurita,., Otsu, N. and Hirata, K.: \A Sketch Retrieval Method for Full Color Image Database Query by Visual Example," Proc. of Int. Conf. on Pattern Recognition, pp , [12] Chijiwa, H.:\Chromatics," Fukumura Printing Co., 1983 (in Japanese).