Matching and Retrieval Based on the Vocabulary and Grammar of Color Patterns

Transcription

1 38 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 Matchng and Retreval Based on the Vocabulary and Grammar of Color Patterns Aleksandra Mojslovć, Member, IEEE, Jelena Kovačevć, Senor Member, IEEE, Janyng Hu, Member, IEEE, Robert J. Safranek, Senor Member, IEEE, and S. Kcha Ganapathy, Member, IEEE Abstract We propose a perceptually based system for pattern retreval and matchng. There s a need for such an ntellgent retreval system n applcatons such as dgtal museums and lbrares, desgn, archtecture, and dgtal stock photography. The central dea of the work s that smlarty judgment has to be modeled along perceptual dmensons. Hence, we detect basc vsual categores that people use n judgment of smlarty, and desgn a computatonal model that accepts patterns as nput, and dependng on the query, produces a set of choces that follow human behavor n pattern matchng. There are two major research aspects to our work. The frst one addresses the ssue of how humans perceve and measure smlarty wthn the doman of color patterns. To understand and descrbe ths mechansm we performed a subjectve experment. The experment yelded fve perceptual crtera used n comparson between color patterns (vocabulary), as well as a set of rules governng the use of these crtera n smlarty judgment (grammar). The second research aspect s the actual mplementaton of the perceptual crtera and rules n an mage retreval system. Followng the processng typcal for human vson, we desgn a system to: 1) extract perceptual features from the vocabulary and 2) perform the comparson between the patterns accordng to the grammar rules. The modelng of human percepton of color patterns s new startng wth a new color codebook desgn, compact color representaton, and texture descrpton through multple scale edge dstrbuton along dfferent drectons. Moreover, we propose new color and texture dstance functons that correlate wth human performance. The performance of the system s llustrated wth numerous examples from mage databases from dfferent applcaton domans. Index Terms Color and texture classfcaton, color and texture extracton, mage database retreval. I. INTRODUCTION FLEXIBLE retreval and manpulaton of mage databases has become an mportant problem wth applcaton n vdeo edtng, photo-journalsm, art, fashon, catalogung, retalng, nteractve CAD, geographc data processng, etc. Untl recently, content-based retreval systems (CBR s) have asked people for key words to search mage and vdeo databases. Unfortunately, ths approach does not work well snce dfferent people descrbe what they see or what they search for n dfferent ways, and even the same person mght descrbe the same mage dfferently dependng on the context n whch t wll be used. These problems stmulated the development of Manuscrpt receved November 20, 1998; revsed July 7, The assocate edtor coordnatng the revew of ths manuscrpt and approvng t for publcaton was Dr. B. S. Manjunath. The authors are wth Bell Laboratores, Lucent Technologes, Murray Hll, NJ USA (e-mal: saska@research.bell-labs.com). Publsher Item Identfer S (00) nnovatve content-based search technques as well as new types of queres. A. Prevous Work One of the earlest CBR systems s ART MUSEUM [1], where retreval s performed entrely based on edge features. The frst commercal content-based mage search engne wth profound effects on later systems was QBIC [2]. As color representaton, ths system uses a k-element hstogram and average of (R; G; B), (Y; ; q), and (L; a; b) coordnates, whereas for the descrpton of texture t mplements Tamura s feature set [3]. In a smlar fashon, color, texture, and shape are supported as a set of nteractve tools for browsng and searchng mages n the Photobook system developed at the MIT Meda Lab [4]. In addton to these elementary features, systems such as VsualSeek [5], Netra [6], and Vrage [7] support queres based on spatal relatonshps and color layout. Moreover, n the Vrage system [7], the user can select a combnaton of mplemented features by adjustng the weghts accordng to hs own percepton. Ths paradgm s also supported n RetrevalWare search engne [8]. A dfferent approach to smlarty modelng s proposed n the MARS system [9], where the man focus s not n fndng a best representaton, but rather on the relevance feedback that wll dynamcally adapt multple vsual features to dfferent applcatons and dfferent users. Hence, although great progress has been made, none of the exstng search engnes offers a complete soluton to the general mage retreval problem, and there are stll many open research ssues, preventng ther use n a real applcaton. Why s that so? B. Motvaton Whle t s recognzed that mages can be descrbed at a metalevel through color, texture, and shape of the objects wthn the mage, general mage understandng s a hard problem. Thus, one challenge s to accomplsh mage retreval based on smlartes n the feature space wthout necessarly performng full-fledged scene analyss. Many of the exstng systems [7], [8], accomplsh ths task by expectng the user to assgn a set of weghts to color, shape, and texture features, thus specfyng the way these attrbutes are gong to be combned n the algorthm. Unfortunately, certan problems arse from ths approach: Frst, ths s certanly not the way matchng s performed n the human vsual system. Further, humans have no general noton of smlarty; nstead, they possess a functonal noton of smlarty wthn a partcular doman. Therefore, to /00$ IEEE

2 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 39 Fg. 1. Overvew of the system. The two man parts deal wth feature extracton and smlarty measurement. Both the feature extracton and smlarty measurment parts mmc the behavor of the human vsual system. Wthn the feature extracton part, color and texture are processed separately. perform smlarty matchng n a human-lke manner one has to: 1) choose a specfc applcaton doman, 2) understand how users judge smlarty wthn that doman, and then 3) buld a system that wll replcate human performance. Snce color and texture are fundamental aspects of human vsual percepton, we developed a set of technques for search and manpulaton of color patterns. Moreover, there are a great many applcatons for pattern retreval n: arts and museums, fashon, garment and desgn ndustry, dgtal lbrares, and dgtal stock photography. Therefore, there s a need for an ntellgent vsual nformaton retreval system that wll perform pattern matchng n these applcatons. However, regardless of the applcaton doman, toaccomplshretrevalsuccessfully ts necessary tounderstand what type of color and texture nformaton humans actually use and how they combne them n decdng whether two patterns are smlar. In ths paper, we are focusng on the ntegraton of color and texture features for pattern retreval and matchng. Our am s to detect basc vsual categores that people use n judgment of smlarty, and then to desgn a computatonal model whch accepts one (or more) texture mages as nput, and dependng on the type of query, produces a set of choces that follow human behavor n pattern matchng. There are two major research aspects n our work: The frst one addresses the ssue of how humans perceve and measure smlarty wthn the doman of color patterns. To understand and descrbe ths mechansm we performed a subjectve experment. The experment yelded fve perceptual crtera mportant for the comparson between the color patterns, as well as a set of rules governng the use of these crtera n the smlarty judgment. The fve perceptual crtera are consdered to be the basc vocabulary, whereas the set of rules s consdered as the basc grammar of the color pattern language. The second research aspect s the actual mplementaton of the perceptual crtera and rules n the mage retreval system llustrated n Fgs. 1 and 2. Followng the processng typcal for human vson, we desgn a system to 1) extract perceptual features from the vocabulary and 2) perform the comparson between the patterns accordng to the grammar rules. The modelng of human percepton of color patterns s new startng wth a new color codebook desgn, compact color representaton, and texture descrpton through multple scale edge dstrbuton along dfferent drectons. Fnally, to model the human behavor n pattern matchng, nstead of usng the tradtonal Eucldean metrc to compare color and texture feature vectors, we propose new dstance functons that correlate wth human performance. The outlne of the paper s as follows. Secton II descrbes the subjectve experment and analytcal tools we used to nterpret the data. At the end of ths secton we lst and descrbe n detal the fve perceptual categores (vocabulary) and fve rules (grammar) used by humans n comparson of color patterns. Secton III gves an overvew of the system together wth ts psychophyscal background. Sectons IV and V present the mplementaton of feature extracton based on color and texture, respectvely, and the development of new color and texture metrcs. Secton VI descrbes how these features and dstances are used n smlarty measurement and presents numerous examples. Secton VII gves examples of dfferent queres and the correspondng search results. The fnal secton ncludes dscusson and conclusons. II. VOCABULARY AND GRAMMAR OF COLOR PATTERNS Our understandng of color patterns s very modest compared to our understandng of other vsual phenomena such as color, contrast or even gray-level textures. That s manly due to the fact that the basc dmensons of color patterns have not yet been dentfed, a standardzed set of features for addressng ther mportant characterstcs does not exst, nor are there rules defnng how these features are to be combned. Prevous nvestgatons n ths feld concentrated manly on gray-level natural textures [3], [10], [11]. Partcularly nterestng s work of Rao and Lohse [11]: ther research focused on how people classfy textures n meanngful, herarchcally structured categores, dentfyng relevant features used n the percepton of gray-level textures. Smlarly, here we determne the basc categores vocabulary used by humans n judgng smlarty of color patterns, ther relatve mportance and relatonshps, as well as the herarchy of rules grammar. Later n the paper, through numerous search examples (see Fgs. 8 13), we wll show that these attrbutes are applcable to a broad range of textures, startng from smple patterns, all the way up to complex, hgh-level vsual texture phenomena. Ths secton descrbes the subjectve experment, and gves a bref overvew of multdmensonal scalng and herarchcal clusterng technques we used to nterpret the expermental data. Multdmensonal scalng was appled to determne the most mportant dmensons of pattern smlarty, whle herarchcal clusterng helped us understand how people combne these dmensons when comparng color patterns. The results obtaned are lsted and explaned at the end of ths secton, whle the detals can be found n [14]. A. Expermental Setup Durng the subjectve testng, we used 25 patterns from nteror desgn catalogs. Twenty patterns were used n the actual study, whle fve patterns were used as a warm-up before

3 40 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 (a) (b) Fg. 2. Two basc blocks of the feature extracton part from Fg. 1. (a) Color representaton and modelng. (b) Texture representaton and modelng. At the end of experment, half of the subjects were presented wth pars they thought the most smlar, and asked to explan why. Ther explanatons were used later as an ad n the nterpretaton of the expermental results, as well as for the development of the retreval system. Expermental data were nterpreted usng multdmensonal scalng technques yeldng the vocabulary and the herarchcal clusterng analyss whch, n turn, led to the grammar rules. Fg. 3. Pattern set used n the experment. The patterns are obtaned from an nteror desgn database, contanng 350 patterns. Twenty were selected capturng a varety of features. Another fve were used as a warm up n the study. The patterns are numbered from 1 through 20, startng at the upper left-hand corner. each tral. Ths allowed the subjects to get comfortable wth the testng procedure and to sharpen ther own understandng of smlarty. The dgtzed verson of the twenty patterns selected are dsplayed n Fg. 3. We selected patterns that capture a varety of dfferent mage features and ther combnatons. The selecton of stmul s crucal for nterpretaton of the data. Snce we postulated that vsual smlarty needs to be modeled by a hgh number of dmensons, t was vtal for ths experment to select the stmul so that there s suffcent varaton of potental dmensons. Twenty eght subjects partcpated n the study. The subjects were not famlar wth the nput data. They were presented wth all 190 possble pars of stmul. For each par, the subjects were asked to rate the degree of overall smlarty on a scale rangng from zero for very dfferent to 100 for very smlar. There were no nstructons concernng the characterstcs on whch these smlarty judgments were to be made snce ths was the very nformaton we were tryng to dscover. The order of presentaton was dfferent for each subject and was determned through the use of a random number generator. Ths was done to mnmze the effect on the subsequent ratngs of both the same presentaton order for all the subjects (group effect) as well as the presentaton order for one subject (ndvdual effect). B. Multdmensonal Scalng Multdmensonal scalng (MDS) s a set of technques that enables researchers to uncover the hdden structures n data [12]. MDS s desgned to analyze dstance-lke data called smlarty data; that s, data ndcatng the degree of smlarty between two tems. Tradtonally, smlarty data s obtaned va subjectve measurement. It s acqured by askng people to rank smlarty of pars of objects stmul on some scale (as n our experment). The obtaned smlarty value connectng stmulus to stmulus j s denoted by j. Smlarty values are arranged n a smlarty matrx 1, usually by averagng j obtaned from all measurements. The am of MDS s to place each stmulus from the nput set nto an n-dmensonal stmulus space (the optmal dmensonalty of the space, n, should be also determned n the experment). The ponts x = [x 1 x x n ] representng each stmulus are obtaned so that the Eucldean dstances d j between each par of ponts n the stmulus space match as closely as possble the subjectve smlartes j between correspondng pars of stmul. The coordnates of all stmul (.e., the confguraton) are stored n the matrx X, also called the group confguraton matrx. Dependng on the type of the MDS algorthm, one or several smlarty matrces are analyzed. The smplest algorthm s the classcal MDS (CMDS), where only one smlarty matrx s analyzed. The central concept of CMDS s that the dstance d j between ponts n an n-dmensonal space wll have the strongest possble relaton to the smlartes j from a sngle matrx 1. The tradtonal way to descrbe a desred relatonshp between the dstance d j and the smlarty j s by the relaton d = f () such as d = f() =a + b (1)

4 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 41 where, for a gven confguraton, values a and b must be dscovered usng numercal optmzaton. There are many dfferent computatonal approaches for solvng ths equaton [12]. Once the best f s found, we then search for the best confguraton X of ponts n the stmulus space. Ths procedure s repeated for dfferent n s untl further ncrease n the number of dmensons does not brng a reducton n the followng error functon (also known as stress formula 1 or Kruskal s stress formula): stress(1; X; f)= vu XX j u t XX [f( j ) 0 d j ] 2 j f( j ) 2 : (2) A detaled ntroducton to the CMDS together wth many mportant mplementaton aspects can be found n [12]. Once the CMDS confguraton s obtaned we are left wth the task of nterpretng and labelng the dmensons we have. Usually, we am to nterpret each dmenson of the space. However, the number of dmensons does not necessarly reflect all the relevant characterstcs. Also, although a partcular feature exsts n the stmulus set, t may not contrbute strongly enough to become vsble as a separate dmenson. Therefore, one useful role of MDS s to ndcate whch partcular features are mportant. Another mportant MDS type s weghted multdmensonal scalng (WMDS). It generalzes CMDS Eucldean dstance model, so that several smlarty matrces can be used. Ths model assumes that ndvduals vary n the mportance they attach to each dmenson of the stmulus space. In that way WMDS accounts for ndvdual dfferences n human responses. WMDS analyzes several smlarty matrces, one for each of m subjects. In the WMDS model, jk ndcates the smlarty between stmul and j, as judged by subject k. The noton of ndvdual taste s ncorporated nto the model through weghts w kl, for each subject k =1; 111;mand each dmenson l =1; 111;n. Just as n CMDS, WMDS determnes the confguraton of ponts n the group stmulus space X. However, n order to fnd the best possble confguraton, WMDS does not use dstances among the ponts n the group space. Instead, a confguraton for each subject s made by alterng the group confguraton space accordng to the weghts w kl. Algebracally, gven a pont x from the group space, the ponts for subject k are obtaned as x lk = p w lk 1 x l : (3) In WMDS, the formula for stress s based on the squared dstances calculated from each of m ndvdual smlarty matrces vu u stress(1; X k ;f)= t 1 m XX X j XX k [f( jk ) 0 d jk ] 2 j f( jk ) 2 (4) where d jk are weghted Eucldean dstances between stmul and j, for the subject k. In that way, the WMDS model accommodates very large dfferences among the ndvdual ratngs, and even very dfferent data from two subjects can ft nto the same space. An mportant characterstc of CMDS s that once a confguraton of ponts s obtaned, t can be rotated, mplyng that the dmensons are not meanngful. Thus, when nterpretng the results, hgher-dmensonal CMDS soon becomes mpractcal. As opposed to CMDS, due to the algebra of the weghted Eucldan model, once the WMDS confguraton s obtaned, t cannot be rotated [12], [28]. However, the stablty of confguraton depends heavly on the accuracy of the model; f the model fts that data well, the dmensons are meanngful whch makes our job of nterpretng them much easer. C. Herarchcal Cluster Analyss Gven a smlarty matrx, herarchcal cluster analyss (HCA) organzes a set of stmul nto smlar unts [13]. Therefore, HCA help us dscover the rules and the herarchy we use n judgng smlarty and pattern matchng. Ths method starts from the stmulus set to buld a tree. Before the procedure begns, all stmul are consdered as separate clusters, hence there are as many clusters as there are ranked stmul. The tree s formed by successvely jonng the most smlar pars of stmul nto new clusters. At every step, ether an ndvdual stmulus s added to the exstng clusters, or two exstng clusters are merged. The groupng contnues untl all stmul are members of a sngle cluster. How the smlarty matrx s updated at each stage of the tree s determned by the jonng algorthm. There are many possble crtera for decdng how to merge clusters. Some of the smplest methods use nearest neghbor technque, where the frst two objects combned are those that have the smallest dstance between them. Another commonly used technque s the farthest neghbor technque where the dstance between two clusters s obtaned as the dstance between ther farthest ponts. The centrod method calculates the dstances between two clusters as the dstance between ther means. Also, snce the mergng of clusters at each step depends on the dstance measure, dfferent dstance measures can result n dfferent clusterng solutons for the same clusterng method [13]. Clusterng technques are often used n combnaton wth MDS, to clarfy the obtaned dmensons. However, n the same way as wth the labelng of the dmensons n the MDS algorthm, nterpretaton of the clusters s usually done subjectvely and strongly depends on the qualty of the data. D. Vocabulary: Most Important Dmensons of Color Patterns The frst step n the data analyss was to arrange subjects ratngs nto a smlarty matrx 1 to be an nput to the two-dmensonal (2 D) and three-dmensonal (3 D) CMDS. Also, WMDS procedure was appled to the set of 28 ndvdual smlarty matrces. WMDS was performed n two, three, four, fve, and sx dmensons. The stress ndex (4) for the 2-D soluton was 0.31, ndcatng that a hgher-dmensonal soluton s necessary, that s, the error s stll substantal. The stress values for the three-, four-, fve-, and sx-dmensonal confguratons were: 0.26, 0.20, 0.18, and 0.16, respectvely. We stopped at sx dmensons snce further ncrease dd not result n a notceable decrease of the stress value. The 2 D CMDS confguraton s shown n Fg. 4. Dmensons derved from ths confguraton are: 1) presence/absence of a domnant color, or as we are gong to call t the dmenson of overall color, and 2) color purty. It

5 42 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 Fg. 4. Multdmensonal scalng results. Two-dmensonal CMDS confguraton s shown. Horzontal axs represents the dmenson of color pruty whereas the vertcal axs s the dmenson of domnant color. s nterestng that both dmensons are purely color based, ndcatng that, at the coarsest level of judgment, people prmarly use color to judge smlarty. As wll be seen later, these dmensons remaned n all solutons. Moreover, the 2-D confguraton strongly resembles one of the perpendcular projectons n the three-, four-, and fve-dmensonal solutons. The same holds for all three dmensons from the 3-D soluton, ndcatng that these features could be the most general n human percepton. Both for CMDS and WMDS, the same three dmensons emerged from 3-D confguratons. They are 1) overall color,; 2) color purty; 3) regularty and placement. The four-dmensonal (4-D) WMDS soluton revealed followng dmensons: 1) overall color; 2) color purty,; 3) regularty and placement; 4) drectonalty. The fve-dmensonal (5 D) WMDS soluton came wth the same four domnant characterstcs wth the addton of a dmenson that we called pattern heavness. Hence, as a result of the experment, the followng fve mportant smlarty crtera emerged. Dmenson 1 Overall Color: Overall color can be descrbed n terms of the presence/absence of a domnant color. At the negatve end of ths axs are patterns wth an overall mpresson of a sngle domnant color (patterns 4, 5, 8, 15). Ths mpresson s created mostly because the percentage of one color s truly domnant. However, a multcolored mage can also create an mpresson of domnant color. Ths happens when all the colors wthn ths mage are smlar, havng smlar hues but dfferent ntenstes or saturaton (pattern 7). At the postve end of ths dmenson are patterns where no sngle color s perceved as domnant (such as n true multcolored patterns 16 20). Dmenson 2 Drectonalty and Orentaton: Ths axs represents a domnant orentaton n the edge dstrbuton, or a domnant drecton n the repetton of the structural element. The lowest values along ths dmenson have patterns wth a sngle domnant orentaton, such as strpes and then checkers (2, 4, 11 13). Mdvalues are assgned to patterns wth a notceable but not domnant orentaton (5, 10), followed by the patterns where a repetton of the structural element s performed along two drectons (3, 8, 9, 15). Fnally, completely nonorented patterns and patterns wth unform dstrbuton of edges or nondrectonal placement of the structural element are at the postve end of ths dmenson. Dmenson 3 Regularty and Placement Rules: Ths dmenson descrbes the regularty n the placement of the structural element, ts repetton and unformty. At the negatve end of ths axs are regular, unform, and repettve patterns (wth repetton completely determned by a certan set of placement rules), whereas at the opposte end are nonrepettve or nonunform patterns. Dmenson 4 Color Purty: Ths dmenson arose somehow unexpectedly, but t remaned stable n all MDS confguratons, clusterng results, even n the subjects explanatons of ther rankngs. Ths dmenson dvdes patterns accordng to the degree of ther colorfulness. At the negatve end are pale patterns (1, 10), patterns wth unsaturated overtones (7), patterns wth domnant sandy or earthy colors (5, 6, 11). At the postve end are patterns wth very saturated and very pure colors (9, 13, 19, etc.). Hence, ths dmenson can also be named the dmenson of overall chroma or overall saturaton wthn an mage. Dmenson 5 Pattern Complexty and Heavness: Ths dmenson showed only n the last, 5 D confguraton, hence t can be seen as optonal. Also, as we wll show n the next secton, t s not used n judgng smlarty untl the very last level of comparson. For that reason we have also named t a dmenson of general mpresson. At one end of ths dmenson are patterns that are perceved as lght and soft (1, 7, 10) whle at the other end are patterns descrbed by subjects as heavy, busy, and sharp (2, 3, 5, 17, 18, 19). E. Grammar: Rules for Judgng Smlarty Havng determned the dmensons of color patterns, we need to establsh a set of rules governng ther use. HCA acheves that by orderng groups of patterns accordng to the degree of smlarty, as perceved by subjects. Fg. 5 shows the orderng of clusters obtaned as a result of the HCA, arsng from the complete smlarty matrx for 20 patterns used n the study. As a result of the HCA, we derved a lst of smlarty rules and the sequence of ther applcaton based on the analyss gven below. For example, we observed that the very frst clusters were composed of pars of equal patterns (clusters 21 23). These were followed by the clusters of patterns wth smlar color and domnant orentaton. Thus, from the early stages of clusterng we were able to determne the ntal rules used by humans n judgng smlarty (Rules 1 and 2). These were followed by rules emergng from the mddle stages (Rules 3 and 4). Fnally, at the

6 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 43 Fg. 5. Result of the HCA appled to the complete set of stmul. Clusters 1 to 20 are orgnal patterns, clusters 21 to 37 represent successve nodes of the tree. In the last step, clusters 36 and 38 are joned to form the top cluster. The orderng of clusters was used to determne the rules and the sequence of ther applcaton n pattern matchng. coarsest level of comparson we use Rule 5 (clusters n Fg. 5). In addton, to confrm the stablty of rules, we have splt the orgnal data n several ways and performed separate HCA s for each part. As suggested n [12], we elmnated some of the stmul from the data matrx and determned the HCA trees for the remanng stmul. The rules remaned stable through varous solutons; thus we conclude that the 5 D confguraton can be used for modelng the smlarty metrcs of the human vsual system, together wth the followng rules: Rule 1: The strongest smlarty rule s that of equal pattern. Regardless of color, two textures wth exactly the same pattern such as pars (17, 18), (2, 11), and (3, 15) are always judged to be the most smlar. Hence, ths rule uses Dmensons 3 and 2 (pattern regularty and drectonalty). Rule 2: The second rule n the herarchy s that of overall appearance. It uses the combnaton of Dmenson 1 (domnant color) and Dmenson 2 (drectonalty). Two patterns that have smlar values n both dmensons, such as pars (10, 11), (1, 7), and the trplet (2, 4, 5) are also perceved as smlar. Rule 3: The thrd rule s that of smlar pattern. It concerns ether dmenson 2 (drectonalty) or dmenson 3 (pattern regularty and placement rules). Hence, two patterns whch are domnant along the same drecton (or drectons) are seen as smlar, regardless of ther color. One such example s the cluster (12 14). In the same manner, seen as smlar are patterns wth the same placement or repetton of the structural element, even f the structural element s not exactly the same (see patterns 8 and 9, or 17, 18 and 19). Rule 4: In the mddle of the herarchy comes the rule of domnant color. Two multcolored patterns are perceved as smlar f they possess the same color dstrbutons regardless of ther content, drectonalty, placement, or repetton of a structural element (patterns 16 20). Ths also holds for patterns that have the same domnant or overall color (patterns 2 6). Hence, ths rule nvolves only the Dmenson 1 (domnant color). Rule 5: Fnally, at the very end of the herarchy, comes the rule of general mpresson (Dmensons 4 and 5). Ths rule dvdes patterns nto dm, smooth, earthy, romantc, or pale (at one end of the correspondng dmenson) as opposed

7 44 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 to bold, brght, strong, pure, sharp, abstract, or heavy patterns (at the opposte end). Ths rule represents the complex combnaton of color, contrast, saturaton, and spatal frequency, and therefore apples to patterns at the hghest, abstract level of understandng. Ths set of rules represents the basc grammar of pattern matchng. For actual mplementaton of the grammar t s mportant to observe the way these rules are appled: Each rule can be expressed as a logcal combnaton (logcal OR, AND, XOR, NOT) of the pattern values along the dmensons nvolved n t. For example, consder cluster 24 composed of patterns 4 and 5 n Fg. 5. These patterns have smlar overall color and domnant orentaton, thus ther values both along the dmensons 1 and 2 are very close. Consequently, they are perceved as smlar accordng to the Rule 2, whch s expressed n the followng way: (DIM 1 (pattern 4) smlar to DIM 1 (pattern 5)) AND(DIM 2 (pattern 4) smlar to DIM 2 (pattern5)): (5) III. OVERVIEW OF THE SYSTEM We wll summarze our fndngs thus far. To model the human percepton of smlarty: 1) we determned the basc vocabulary V of color patterns consstng of dmensons 1 5: V = fdim 1 ; 111;DIM 5 g; 2) we determned the grammar G, that s, the rules governng the use of the dmensons from the vocabulary V. Fve rules (R 1 R 5 ) were dscovered so that G = fr 1 ;R 2 ;R 3 ;R 4 ;R 5 g. Havng found the vocabulary and grammar, we need to desgn a system that wll, gven an nput mage A and a query Q: 1) measure the dmensons DIM (A) from the vocabulary, = 1; 111; 5; 2) for each mage B from the database, apply rules R 1 R 5 from G and obtan correspondng dstance measures dst 1 (A; B); 111, dst 5 (A; B), where dst (A; B) s the dstance between the mages A and B accordng to the rule ; Therefore, the system has two man parts: 1) the feature extracton part, measurng the dmensons from V and 2) the search part, where smlar patterns are found accordng to the rules from G. The feature extracton part s desgned to extract dmensons 1 to 4 of pattern smlarty. Dmenson 5 (pattern complexty and heavness) s not mplemented, snce our experments have shown that people use ths crteron only at a hgher level of judgment, whle comparng groups of textures [14]. Feature extracton s followed by judgment of smlarty accordng to Rules 1 4 from G. Rule 5 s not supported n the current mplementaton, snce t s only used n combnaton wth dmenson 5 at a hgher level of pattern matchng (such as subdvdng a group of patterns nto romantc, abstract, geometrc, bold, etc.). Let us now examne the system n more detal. It s mportant to note that the feature extracton part s developed accordng to the followng assumptons derved from psychophyscal propertes of the human vsual system and conclusons extracted from our experment. 1) The overall percepton of color patterns s formed through the nteracton of lumnance component L, chromnance component C and achromatc pattern component AP. The lumnance and chromnance components approxmate sgnal representaton n the early vsual cortcal areas whle the achromatc pattern component approxmates sgnal representaton formed at hgher processng levels [15]. Our expermental results confrm ths fact: we found that at the coarsest level of judgment only color features are used (2-D MDS) whereas texture nformaton s added later and used n the detaled comparson. Therefore, our feature extracton smulates the same mechansm; t decomposes the mage map nto lumnance and chromnance components n the ntal stages, and models pattern nformaton later n the system. 2) As n the human vsual system the frst approxmaton s that each of these components s processed through separate pathways [16], [29]. Whle lumnance and chromnance components are used for the extracton of colorbased nformaton, the achromatc pattern component s used for the extracton of purely texture-based nformaton. However, f we want to be more precse, we need to account for resdual nteractons along the pathways [17]. As wll be shown n Secton V, we accomplsh ths by extractng the achromatc pattern component from the color dstrbuton, nstead of usng the lumnance sgnal as n prevous models. Moreover, the dscrete color dstrbuton s estmated through the use of a specally desgned perceptual codebook allowng the nteracton between the lumnance and chromnance components (see Secton IV). 3) Features are extracted by combnng three major domans: a) nonorented lumnance doman represented by the lumnance component of an mage, b) orented lumnance doman represented by the achromatc pattern map, and c) nonorented color doman represented by the chromnance component. The frst two domans are essentally color blnd, whereas the thrd doman carres only the chromatc nformaton. These three domans are well documented n the lterature [18] and expermentally verfed n perceptual computatonal models for segregaton of color textures [19]. Purely color-based dmensons (1 and 4) are extracted n the nonorented domans and are measured usng the color feature vector. Texture-based dmensons (2 and 3) are extracted n the orented lumnance doman, through the scale-orentaton processng of the achromatc pattern map. In summary, our computatonal model s mplemented as n Fg. 1 and contans the followng parts. 1) Feature extracton block wth the followng components. Image Decomposton: Input mage s transformed nto the Lab color space and decomposed nto lumnance L and chromnance C =(a; b) components. Estmaton of Color Dstrbuton: Both L and C maps are used for the color dstrbuton estmaton and

8 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 45 extracton of color features. We are thus performng feature extracton along the color-based dmensons 1 and 4. Pattern Map Generaton: Color features extracted n the second stage are used to buld the achromatc pattern map. Texture Prmtve Extracton and Estmaton: The achromatc pattern map s used to estmate the spatal dstrbuton of texture prmtves. We are thus performng feature extracton along the texture-based dmensons 2 and 3. 2) Smlarty Measurement: Here smlar patterns are found accordng to the rules from G. Gven an nput mage A, for every mage B n the database, rules R 1 R 4 are appled and correspondng dstance measures are computed. Then, dependng on a query Q, a set of best matches s found. IV. FEATURE EXTRACTION BASED ON COLOR INFORMATION The color nformaton s used both for the extracton of colorrelated dmensons (color features), and for the constructon of the achromatc pattern map (used later n texture processng), therefore we am for compact, perceptually-based color representaton. As llustrated n Fg. 2(a), ths representaton s obtaned through the followng steps. 1) The nput mage s transformed nto the Lab color space. 2) Its color dstrbuton s determned usng a vector quantzaton-based hstogram technque,. 3) Sgnfcant color features are determned from the hstogram. 4) These color features are used n conjuncton wth a new dstance measure to determne the perceptual smlarty between two color dstrbutons. A. Color Representaton Our goal s to produce a system that performs n accordance wth human percepton, hence we need a representaton (color space) based on human color matchng. CIE Lab s such a color space, snce t was desgned so that ntercolor dstances computed usng the kk 2 norm correspond to subjectve color matchng data [20]. After transformng an nput mage nto the Lab color space, the next step s to estmate the color dstrbuton by computng a hstogram of the nput color data. Snce lnear color spaces (such as RGB) can be approxmated by 3-D cubes, hstogram bn centers can be computed by performng separable, equdstant dscretzatons along each of the coordnate axes. Unfortunately, by gong to the nonlnear Lab color space, the volume of all possble colors dstorts from cube to an rregular cone and consequently, there s no smple dscretzaton that can be appled to ths volume. To estmate color dstrbutons n the Lab space, we have to determne the set of bn centers and decson boundares that mnmze some error crteron. In the Lab color system, kk 2 norm corresponds to perceptual smlarty, thus representng the optmal dstance metrc for that space [20]. Therefore, to obtan an optmal set of bn centers and decson boundares, we have to fnd Lab coordnates of N bn centers so that the overall mean-square classfcaton error s mnmzed. Ths s exactly the underlyng problem n vector quantzaton (VQ). Hence, we used the LBG vector quantzaton algorthm [21] to obtan a set of codebooks whch optmally represent the vald colors n the Lab space. In any VQ desgn, the tranng data have a large effect on the fnal result. A commonly used approach s to select tranng mages that are ether representatve of a gven problem so the codebook s optmally desgned for that partcular applcaton, or span enough of the nput space so the resultng codebook can be used n dfferent applcatons. The followng problem occurs wth both approaches: In order to obtan an accurate estmaton for the color dstrbuton, a large number of tranng mages s requred, resultng n a computatonally expensve and possbly ntractable desgn task. To overcome ths problem, we have taken a dfferent approach. Snce we are dealng wth an arbtrary nput, we can assume that every color s equprobable. Hence, a synthetc set of tranng data was generated by unformly quantzng the XY Z space. The data was transformed nto the Lab space and used as nput to the standard VQ desgn algorthm. Ths resulted n a set of codebooks rangng n sze from 16 to 512 colors. When used n the standard mage retreval task, these codebooks performed qute well. For our task, however, these codebooks have one drawback; They are desgned as a global representaton of the entre color space and consequently, there s no structure to the bn centers. Our purpose s to desgn a system whch allows a user to nteract wth the retreval process. Therefore, the color representaton must provde manpulaton wth colors n a human-frendly manner. To smulate human performance n color percepton, a certan amount of structure on the relatonshps between the L, a, and b components must be ntroduced. One possble way to accomplsh ths s by separatng the lumnance L, from the chromnance (a; b) components. Startng from ths assumpton, we frst appled one-dmensonal (1 D) quantzaton on lumnance values of the tranng data (usng a Lloyd Max quantzer). Then, after parttonng the tranng data nto slces of smlar lumnance, a separate chromnance codebook was desgned for each slce by applyng the LBG algorthm to the approprate (a; b) components. Ths color representaton better mmcs human percepton and allows the formulaton of functonal queres such as lookng for same but lghter color, paler, contrastng, etc. For example, the formulaton of a query vector to search for a lghter color can be accomplshed through the followng steps: 1) extract the lumnance LQ and the (aq, bq) par for the query color; 2) fnd the codebook for a hgher lumnance level L>LQ; 3) n ths codebook, fnd the cell whch corresponds to (a; b) entry whch s the closest to (aq; bq) n the kk 2 sense; 4) retreve all mages havng (L; a; b) as a domnant color. Moreover, startng from the relatonshp between L; a, and b values for a partcular color, and ts hue H and saturaton S H = arctan b a ; S = pa 2 + b 2 : (6) Smlar procedures can be appled to satsfy queres such as paler color, bolder color, contrastng color, etc. Fnally,

9 46 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 n applcatons where the search s performed between dfferent databases or when the query mage s suppled by the user, separaton of lumnance and chromnance allows for elmnaton of the unequal lumnance condton. Snce the chromnance components contan the nformaton about the type of color regardless of the ntensty value, color features can be extracted only n the chromnance doman C(; j) = fa(; j); b(; j)g, for the correspondng lumnance level, thus allowng for comparson between mages of dfferent qualty. B. Color Feature Extracton Color hstogram representatons based on color codebooks have been wdely used as a feature vector n mage segmentaton and retreval [22], [23]. Although good results have been reported, a feature set based solely on the mage hstogram may not provde a relable representaton for pattern matchng and retreval. Ths s due to the fact that most patterns are perceved as combnatons of a few domnant colors. For example, subjects who partcpated n our prevously reported subjectve experments [14], were not able to perceve nor dstngush more than sx or seven colors, even when presented wth very busy or multcolored patterns. For that reason, we are proposng color features and assocated dstance measures consstng of the subset of colors (whch best represent an mage), augmented by the area percentage n whch each of these colors occur. In our system we have used a codebook wth N = 71 colors denoted by C 71 = fc 1 ;C 2 ; 111;C 71 g where each color C = fl ;a ;b g s a three-dmensonal Lab vector. As the frst step n the feature extracton procedure (before hstogram calculaton) nput mage s convolved wth a B-splne smoothng kernel. Ths s done to refne contours of texture prmtves and foreground regons, whle elmnatng most of the background nose. The B-splne kernel s used snce t provdes an optmal representaton of a sgnal n the kk 2 sense, hence mnmzng the perceptual error [24]. The second step (after the hstogram of an mage was bult) nvolves extracton of domnant colors to fnd colors from the codebook that adequately descrbe a gven texture pattern. Ths was done by sequentally ncreasng the number of colors untl all colors coverng more than 3% of the mage area have been extracted. The remanng pxels were represented wth ther closest matches (n kk 2 sense) from the extracted domnant colors. Fnally, the percentage of each domnant color was calculated and the color feature vectors were obtaned as f c = f( j ;p j )jj 2 [1; N]; p j 2 [0; 1]g (7) where j s the ndex n the codebook, p j s the correspondng percentage and N s the number of domnant colors n the mage. Another smlar representaton has been successfully used n mage retreval [25]. The proposed feature extracton scheme has several advantages: It provdes an optmal representaton of the orgnal color content by mnmzng the MSE ntroduced when usng a small number of colors. Then, by explotng the fact that the human eye cannot perceve a large number of colors at the same tme, nor s t able to dstngush close colors well, we provde a very compact feature representaton. Ths greatly reduces the sze of the features needed for storage and ndexng. Furthermore, because of the codebook used, ths representaton facltates queres contanng an overall mpresson of patterns expressed n a natural way, such as fnd me all blue-yellow fabrcs, fnd me the same color, but a bt lghter, etc. Fnally, n addton to storng the values of the domnant colors and ther percentages, we are also storng the actual number of domnant colors. Ths nformaton s useful n addressng the more complex dmensons of pattern smlartes as suggested n [14]. Namely, by usng ths feature we can search for smple and sngle colored patterns, versus heavy, multcolored ones. C. Color Metrc The color features descrbed above, represented as color and area pars, allow the defnton of a color metrc that closely matches human percepton. The dea s that the smlarty between two mages n terms of color composton should be measured by a combnaton of color and area dfferences. Gven two mages, a query mage A and a target mage B, wth N A and N B domnant colors, and feature vectors f c (A) = f( a ;p a )j8a 2 [1; N A ]g, and f c (B) = f( b ;p b )j8b 2 [1; N B ]g, respectvely, we frst defne the smlarty between these two mages n terms of a sngle domnant color. Suppose that s the domnant color n mage A. Then, we measure the smlarty between A and B n terms of that color usng the mnmum of dstance measures between the color element (; p) and the set of color elements f( b ;p b )j8 b 2 [1; N B ]g: d(; B) = mn D((; p); ( b ;p b )) (8) b2[1;nb] where D((; p); ( b ;p b )) p = jp 0 p b j + (L 0 L b ) 2 +(a 0 a b ) 2 +(b 0 b b ) 2 : (9) Once the dstance d(; B) has been calculated, besdes ts value we also use ts argument to store the color value from B that, for a partcular color from A, mnmzes (8). We denote ths color value by k(; B) as k(; B) = arg d(; B): (10) Note that the dstance between two color/area pars s defned as the sum of the dstance n terms of the area percentage and the dstance n the Lab color space, both wthn the range [0, 1]. In [25], Ma et al. used a dfferent defnton where the overall dstance s the product of these two components. That defnton, whle beng more ntutve, has the drawback that when ether component dstance s very small the remanng component becomes rrelevant. Consder the extreme case, when the color dstance between two color/area pars s zero. Ths s not unusual, snce the color space has been heavly quantzed. Then, even f the dfference between the two area percentages s very large, the overall dstance s zero yeldng a measure that does not match human percepton. Our defnton s a smple and effectve remedy to that problem t guarantees that both color

10 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 47 and area components contrbute to the percepton of color smlarty. Gven the dstance between two mages n terms of one domnant color as defned above, the dstance n terms of overall color composton s defned as the sum over all domnant colors from both mages, n the followng way. 1) For mage A, for 8 a 2 [1; N A ] fnd k A ( a ;B) and the correspondng dstance d( a ;B). 2) Repeat ths procedure for all domnant colors n B, that s, for 8 b 2 [1; N B ] fnd k B ( b ;B) and d( b ;A). 3) calculate the overall dstance as dst(a; B) = X a2[1;na] d( a ;B)+ X b2[1;nb] d( b ;A): (11) V. FEATURE EXTRACTION BASED ON TEXTURE INFORMATION Havng obtaned the color feature vector, the extracton of texture features nvolves the followng steps [see Fg. 2(b)]: 1) spatal smoothng, to refne texture prmtves and remove background nose; 2) buldng the achromatc pattern map; 3) buldng the edge map from the achromatc pattern map; 4) applcaton of a nonlnear mechansm to suppress nontextured edges; 5) orentaton processng to extract the dstrbuton of pattern contours along dfferent spatal drectons; 6) computaton of a scale-spatal texture edge dstrbuton. Spatal smoothng of the nput mage s performed durng the extracton of color features. Then, the color feature representaton s used for constructon of the achromatc pattern map. The achromatc map s obtaned n the followng manner: For a gven texture, by usng the number of ts domnant colors N, a gray level range of s dscretzed nto N levels. Then, domnant colors are mapped nto gray levels accordng to the followng rule: Level 0 s assgned to the domnant color wth the hghest percentage of pxels, the next level s assgned to the second domnant color, etc., untl the level 255 has been assgned to a domnant color wth the lowest area percentage. In other words, the achromatc pattern map models the fact that human percepton and understandng of form, shape, and orentaton s completely unrelated to color. Furthermore, t resolves the problem of secondary nteractons between the lumnance and chromnance pathways. As an example, consder a par of textures n Fg. 6(a). The values n the lumnance map are much hgher for the texture on top, hence the edge ampltudes, and edge dstrbutons are dfferent for these two mages [see Fg. 6(b)]. Moreover, the domnant colors are not close, whch makes the classfcaton of these two patterns as smlar (ether usng lumnance, chromnance, or color features) extremely dffcult. However, n our model, the way that lumnance and chromnance are coupled nto a sngle pattern map guarantees that both textures wll have dentcal achromatc pattern maps [see Fg. 6(c)], leadng to almost dentcal texture feature vectors. The objectve of edge and orentaton processng s to extract nformaton about the pattern contours from the achromatc pattern map. Instead of applyng a bank of orented flters, as n Fg. 6. Human percepton and understandng of form, shape, and orentaton s unrelated to color. The system models ths through the use of the achromatc pattern map. (a) Two dentcal textures wth dfferent color dstrbutons are perceved as dentcal. (b) However, modelng of these patterns by ther lumnance components results n dfferent feature vectors. (c) The soluton s to map domnant colors from both patterns nto the same gray-scale values, resultng n an achromatc pattern map. Ths representaton corresponds to human percepton. Consequently, the feature vectors extracted from the achromatc pattern maps are almost dentcal. prevous models, we decded to compute polar edge maps and use them to extract dstrbuton of edges along dfferent drectons. Ths approach allowed us to obtan the edge dstrbuton for an arbtrary orentaton wth low computatonal cost. It also ntroduced certan flexblty n the extracton of texture features snce, f necessary, the orentaton selectvty can be enhanced by choosng an arbtrary number of orentatons. In our system, we used edge-ampltude and edge-angle maps, calculated at each mage pont. Edge maps were obtaned by convolvng an nput achromatc pattern map wth the horzontal and vertcal dervatves of a Gaussan and convertng the result nto polar coordnates. The dervatves of a Gaussan along x and y axes were computed as g x (; j) =e 0(2 +j 2) ; g y (; j) =je 0(2 +j 2 ) (12) whle the dervatves of the achromatc pattern map along x and y axes were computed as A x (; j) =(g x 3 AP )(; j); A y (; j) =(g y 3 AP )(; j) (13) where 3 stands for 2-D convoluton. These dervatves were then transformed nto ther polar representaton as q A(; j) = A x (; j) 2 + A y (; j) 2 ; (; j) = tan 01 A y(; j) A x (; j) ; (; j) ; 2 : (14) Texture phenomenon s created through the percepton of mage edgeness along dfferent drectons, over dfferent scales. Hence, to estmate the placement and organzaton of texture prmtves, we do not need nformaton about the edge strength at a certan pont; rather, we only need to know whether an edge exsts at ths pont and the drecton of the edge. Therefore, after the transformaton nto the polar representaton, the ampltude map s nonlnearly processed as 1; med (A(; j)) T A Q (; j) = (15) 0; med (A(; j)) <T

11 48 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 where med(1) represents the medan value calculated over a 5 5 neghborhood. Nonlnear medan operaton was ntroduced to suppress false edges n the presence of stronger ones, and elmnate weak edges ntroduced by nose. The quantzaton threshold T s determned as T = A 0 2 q 2 A (16) where A and A 2 are the mean and varance of the edge ampltude, estmated on a set of 300 mages. Ths selecton allowed all the major edges to be preserved. After quantzng the ampltude map, we perform the dscretzaton of the angle space, dvdng t nto the sx bns correspondng to drectons 0,30, 60,90, 120, and 150, respectvely. For each drecton an ampltude map A (; j) s bult as A (; j) 1; A Q (; j) =1^(; j)) 2 = ;=1; 111; 6: 0; A Q (; j) =0_ (; j)) =2 (17) To address the textural behavor at dfferent scales, we estmate mean and varance of edge densty dstrbuton, by applyng overlappng wndows of dfferent szes to the set of drectonal ampltude maps. For a gven scale, along a gven drecton, edge densty s calculated smply by summng the values of the correspondng ampltude map wthn the wndow, and dvdng that value by the total number of pxels n the wndow. We used four scales, wth the followng parameters for the sldng wndow: Scale 1: WS 1 = 3 4 W H; N 1 =30; Scale 2: WS 2 = 2 5 W H; N 2 =56; Scale 3: WS 3 = 1 5 W H; N 3 =80; Scale 4: WS 4 = 1 10 W H; N 4 = 224 where WS and N are wndow sze and number of wndows for scale, and W and H are the wdth and heght of the nput texture. Note that the above approach s scale (zoom) nvarant. In other words, the same pattern at dfferent scales wll have smlar feature vectors. Hence, at the output of the texture processng block, we have a texture feature vector of length 48: f t =[ ] (18) Fg. 7. Dscrmnaton of textures based on the mean and varance of texture edge dstrbuton. (a) If two textures have dfferent degrees of regularty, characterzed by dfferent varances, they are mmedately perceved as dfferent. (b) However, f two textures have smlar degrees of regularty, characterzed by smlar varances, percepton of smlarty depends on pattern qualty, whch s modeled by the mean values of edge dstrbuton. the standard devaton estmates the unformty, regularty and repettveness at ths scale, thus addressng the dmenson of pattern regularty. A. Texture Metrc As prevously mentoned, at any partcular scale, the mean values measure the overall edge pattern and the standard devatons measure the unformty, regularty and repettveness at ths scale. Our experments [14] demonstrate that the perceptual texture smlarty between two mages s a combnaton of these two factors n the followng way: If two textures have very dfferent degrees of unformty [as n Fg. 7(a)] they are mmedately perceved as dfferent. On the other hand, f ther degrees of unformty, regularty and repettveness are close [as n Fg. 7(b)], ther overall patterns should be further examned to judge smlarty. The smooth transton between these two factors can be mplemented usng the logstc functon, commonly used as an exctaton functon n artfcal neural networks [26]. Thus, the dstance between the query mage A and the target mage B, wth texture feature vectors f t (A) =[ 1 1A A ] and f t(b) =[ 1 1B B ] (19) respectvely, s defned as M j D j = j j A 0 j B j; = j j A 0 j B j (20) where j and j stand for mean and standard devaton of texture edges at scale along the drecton j. Each feature component s normalzed so that t assumes the mean value of zero and standard devaton of one over the whole database. In that way ths feature vector essentally models both texture-related dmensons (drectonalty and regularty): The dstrbuton estmates along the dfferent drectons address the dmenson of drectonalty. At any partcular scale, the mean value can be understood as an estmaton of the overall pattern qualty, whereas d j = w M (; j )M j = e0(d 1+e 0(D j + j 0Do) 1 0Do) + w D (; j )D j M j 1+e 0(D j0do) dst(a; B) = X X j D j ; (21) d j : (22)

12 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 49 Fg. 8. Examples of the search mechansm usng Rule 1 (the rule of equal pattern). Ths s the strongest rule people use when judgng smlarty. The leftmost mage s the query pattern followng by four best matches. (a) Example from the Interor Desgn database. (b) Example from the Corel database: bark textures. At each scale and drecton j, the dstance functon d j s the weghted sum of two terms: the frst M j, measurng the dfference n mean edge densty and the second D j, measurng the dfference n standard devaton, or regularty. The weghtng factors, w M (; j ) and w D (; j ), are desgned such that when the dfference n standard devaton s small, the frst term s more domnant; as t ncreases, the second term becomes domnant, thus matchng human percepton as stated above. The parameters and Do control the behavor of the weghtng factors, where controls the sharpness of the transton, and Do defnes the transton pont. These two parameters are currently traned usng 40 mages taken from an nteror desgn database, n the followng way: Frst, ten mages were selected as representatves of the database. Then, for each representatve, three comparson mages were chosen as the most smlar, close, and least smlar to the representatve. For each representatve mage I, =1; 111; 10, the comparson mages C ; j ;j=1; 111; 3 are ordered n decreasng smlarty. Thus, sets fi g and fc ; j g represent the ground truth. For any gven set of parameters (, Do), the rankngs of the comparson mages as gven by the dstance functon can be computed. Let rank j (; Do) represents the rankng of the comparson mage C ; j for representatve mage I. Ideally, we would lke to acheve rank j (; Do) =j; 8 ; jj 2 [1; 10]; j2 [1; 3]: (23) The devaton from ground truth s computed as where 3X d (; Do) = j=1 D(; Do) = 10 X d (; Do) (24) =1 dst(i ;C ; j ) 0 dst(i ;C ; rankj(; Do)) : (25) The goal of parameter tranng s to mnmze functon D(; Do). Many standard optmzaton algorthms can be used to acheve ths. We used Powell s algorthm [27] and the optmal parameters derved were: =10and Do =0:95. VI. SIMILARITY MEASUREMENT In ths part of the system, we perform smlarty measurement based on the rules from our grammar G. The system was tested on the followng databases: Corel (more than 2000 mages), nteror desgn (350 mages), archtectural surfaces (600 mages), stones (350 mages), hstorc ornaments (110 mages), and orental carpets (100 mages). The current mplementaton of our system supports four strongest rules for judgng the smlarty between patterns. Here we brefly summarze the rules and ther mplementaton n the system. For more detals on rules, see Secton II or [14]. Applyng Rule 1: The frst smlarty rule s that of equal pattern. Regardless of color, two textures wth exactly the same pattern are always judged to be smlar. Hence, ths rule concerns the smlarty only n the doman of texture features, wthout actual nvolvement of any color-based nformaton. Therefore, ths rule s mplemented by comparng texture features only, usng the texture metrc (20) (22). The same search mechansm supports Rule 3 (equal drectonalty, regularty or placement) as well. Accordng to that rule, two patterns that are domnant along the same drectons are seen as smlar, regardless of ther color. In the same manner, seen as smlar are textures wth the same placement or repetton of the structural element, even f the structural element s not exactly the same. Hence, the value of the dstance functon n the texture doman reflects ether pattern dentty or pattern smlarty. For example, very small dstances mean that two patterns are exactly the same (mplyng that the rule of dentty was used), whereas somewhat larger dstances mply that the smlarty was judged by the less rgorous rules of equal drectonalty or regularty. Examples of the equal pattern search mechansm are gven n Fg. 8, whle the examples of smlar pattern search mechansm are gven n Fg. 10. Applyng Rule 2: The second n the herarchy of smlartes s the combnaton of domnant colors and texture drectonalty,

13 50 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 Fg. 9. Examples of the search mechansm usng Rule 2 (the rule of smlar overall appearance). Ths s the second strongest rule people use when judgng smlarty. Ths rule comes nto play when there are no dentcal patterns. The leftmost mage s the query pattern followed by four best matches. (a) Example from the Hstorc Ornaments database. (b) Example from the Stones database: varous types of green marble. Fg. 10. Examples of the search mechansm usng Rule 3 (the rule of smlar pattern). The leftmost mage s the query pattern followng by four best matches. (a) Example from the Orental Carpets database. (b) Example from the Archtectural Surfaces database. yeldng mages wth smlar overall appearance. The actual mplementaton of ths rule nvolves comparson of both color and texture features. Therefore the search s frst performed n the texture doman, usng texture features and metrcs (20) (22). A set of selected patterns s then subjected to another search, ths tme n the color doman, usng color features (7) and color metrc (8) (11). Examples of ths search mechansm are gven n Fg. 9. Applyng Rule 3: The same mechansm as n Applyng Rule 1 s used here, and the search examples are gven n Fg. 10. Applyng Rule 4: Accordng to the rule of domnant color, two patterns are perceved as smlar f they possess the same color dstrbutons regardless of texture qualty, texture content, drectonalty, placement, or repetton of a structural element. Ths also holds for patterns that have the same domnant or overall color. Hence, ths rule concerns only smlarty n the color doman and s appled by comparng color features only. An example of the search s gven n Fg. 11. VII. QUERY TYPES AND OTHER SEARCH EXAMPLES As explaned n the ntroducton, one of the assumptons about the model s that chromatc and achromatc components are processed through mostly separate pathways. Hence, by separatng color representaton and color metrc from texture representaton and texture metrc, we add a sgnfcant amount of flexblty nto the system n terms of manpulaton of mage features. Ths s an extremely mportant ssue n many practcal applcatons, snce t allows for dfferent types of queres. As nput nto the system the user s expected to supply: a) a query and b) patterns to begn the search. The rules explaned n the prevous secton model typcal human queres, such

14 MOJSILOVIĆ et al.: VOCABULARY AND GRAMMAR OF COLOR PATTERNS 51 Fg. 11. Example of the search mechansm usng Rule 4 (the rule of domnant color). The leftmost mage s the query pattern followed by four best matches. Example s from the Hstorc Ornaments database: Islamc desgns wth letterng from an llumnated Koran, 14th or 15th century. Fg. 12. Dfferent types of queres supported by the system. (a) Query by sketch. The user supples a sketch (btmap mage) of a desred pattern (the leftmost mage). Four best matches are gven from the nteror Desgn database. (b) Combnaton query. The desred pattern (strpes) s taken from one nput mage (frst from left) and the desred color (blue) from another (second from left). Four best matches are gven on the rght. as: fnd the same pattern (Rule 1), fnd all patterns wth smlar overall appearance (Rule 2), fnd smlar patterns (Rule 3), fnd all patterns of smlar color, fnd all patterns of a gven color, fnd patterns that match a gven pattern (Rule 4). Moreover, due to the way the color codebook s desgned, the system supports addtonal queres such as fnd darker patterns, fnd more saturated patterns, fnd smple patterns, fnd multcolored patterns, and fnd contrastng patterns. The nput pattern the user provdes can be suppled by the user, selected from a database, or gven n the form of a sketch. If the user has color preferences, they can be specfed ether from the color codebook, or from another pattern. As an example, let us dscuss query by sketch. There are certan stuatons when the user s unable to supply an mage of the pattern he s tryng to fnd. Hence, nstead of browsng through the database manually, our system provdes tools for sketchng the pattern and formulatng a query based on the obtaned btmap mage. In that case, wthout any lowpass preflterng, only texture feature vector s computed for the btmap mage and used n the search. One such query and four best matches are gven n Fg. 12(a). Furthermore, ths search mechansm allows the user to specfy a desred color, by selectng a color = fl ;a ;b g from the codebook. Then, the search s performed n two teratons. Frst a subset of patterns s selected based on color smlarty. Color smlarty between the color and target mage B, wth the color feature vector f c (B) = f( b ;p b )j8 b 2 [1; N B ]g s calculated as d(; B) = mn D c (; b ); b2[1; NB ] p D c (; b )= (L 0 L b ) 2 +(a 0 a b ) 2 +(b 0 b b ) 2 : (26) Next, wthn the selected set, a search based on texture features s performed to select the best match. A smlar search mechansm s appled for combnaton query, where the desred pattern s taken from one nput mage and the desred color from another mage [see Fg. 12(b)], or n a search where the desred pattern s specfed by an nput mage and the desred color s selected from the color map. To conclude ths secton, we present retreval results on general class of mages from the Corel database. Although our system was desgned specfcally for color patterns, the search results demonstrate robustness of the algorthm to other types of mages (such as natural scenes and mages wth homogeneous regons as n Fg. 13). VIII. DISCUSSION AND CONCLUSIONS It s our belef that a good workng system for mage retreval must accomplsh vsual smlarty along perceptual dmensons. Wth ths as the central thrust of our research, we performed subjectve experments and analyzed them usng multdmensonal scalng technques to extract the relevant dmensons. We then nterpreted these dmensons along perceptual categores, and used herarchcal clusterng to determne how these categores are combned n measurng smlarty of color patterns. Havng dscovered the psychophyscal bass of pattern matchng, we developed algorthms for feature extracton and mage retreval n the doman of color patterns. As part of ths research we realzed a need for dstance metrcs that are better matched to human percepton. Dstance metrcs that we developed for color matchng (8) (11) and texture matchng (20) (22) satsfy ths crteron. Whle most of our research has been drected at color patterns, we beleve that the underlyng methodology has greater sgnfcance beyond color and texture. We beleve that such a methodology, f appled to other retreval tasks (such as shape and object understandng), wll result n a system that s better matched to human expectatons. A major advantage of such an approach s that t elmnates the need for selectng the vsual prmtves for mage retreval and expectng the user to assgn weghts to them, as n most current systems. Furthermore, as can

15 52 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 1, JANUARY 2000 Fg. 13. Examples of the serch algorthms appled to the general class of mages. The leftmost mage s the query pattern followed by four best matches. (a) Applcaton of Rule 2 (the rule of overall appearance). Example from the Corel database: Tulps. (b) Applcaton of Rule 3 (the rule of smlar pattern). Example from the Corel database: Alaska. (c) Applcaton of Rule 4 (the rule of domnant color). Example from the Corel database: Vegetables. be seen from the results, our rules of pattern matchng are robust enough to work n varous domans, ncludng dgtal museums [Fgs. 9(a) and 11], archtecture [Fgs. 8(b) and 10(b)], nteror desgn [Fg. 9(b)], and fashon and desgn ndustry [Fgs. 8(a) and 12]. In general, as long as there s no meanng attached to the patterns (or even mages) our approach should work well. However, when buldng any system dealng wth mage smlarty, one should be aware of the mportance of mage content or doman specfc nformaton, and addtonal studes addressng ths ssue need to be conducted. The mportant reason for the success of our system s that t mplements the followng expermental, bologcal, and physologcal observatons. 1) The percepton of color patterns can be modeled by a set of vsual attrbutes and rules governng ther use. 2) Ths same percepton s formed through the nteracton of lumnance and chromnance components (n the early stages of the human vsual system), and achromatc pattern component (n the later stages of the human vsual system). 3) Each of these components s processed through separate pathways. 4) Percepton and understandng of patterns s unrelated to color and relatve lumnance. 5) Patterns are perceved through the nteracton of mage edges of dfferent orentatons and at dfferent scales. Each of these assumptons has ts equvalent n the system, and s accomplshed by 1) determnng the basc vocabulary and grammar of color patterns through a subjectve experment; 2) decomposng an mage nto lumnance, chromnance, and pattern maps; 3) processng the color nformaton frst, and then texture; 4) modelng an mage pattern wth ts achromatc pattern map; 5) extractng texture features from edge representaton of the achromatc pattern map at dfferent scales, along dfferent drectons. Ths has been the approach we have taken toward buldng an mage retreval system that has human lke performance and behavor. Besdes mage retreval, the proposed model can be utlzed n other areas such as perceptually based segmentaton and codng, pattern recognton and machne vson as well as for effectvely employng perceptual characterstcs n scentfc vsualzaton of large data sets. ACKNOWLEDGMENT The authors wsh to thank A. Stanley-Marbell for hs work on color search, D. Kall for helpng desgn the experment, J. Hall for hs tremendous help wth the multdmensonal scalng and for many useful suggestons, D. Davs for provdng software for the subjectve experment, and J. Pnhero for hs help wth the statstcal analyss of the data. The authors also thank F. Juang for techncal dscussons and J. Mazo for nsghtful comments. REFERENCES [1] K. Hrata and T. Katzo, Query by vsual example, content based mage retreval, n Advances n Database Technology-EDBT 92, vol. 580, A. Protte, C. Delobel, and G. Gottlob, Eds., [2] W. Nblack et al., The QBIC project: Querng mages by content usng color, texture and shape, n Proc. SPIE Storage and Retreval for Image and Vdeo Data Bases, 1994, pp [3] H. Tamura, S. Mor, and T. Yamawak, Textural features correspondng to vsual percepton, IEEE Trans. Syst., Man, Cybern.,, vol. 8, pp , [4] A. Pentland, R. W. Pcard, and S. Sclaroff, Photobook: Content-based manpulaton of mage databases, Int. J. Comput. Vs., vol. 18, pp , 1996.