Colour-texture analysis of paintings using ICA filter banks

Transcription

1 Colour-texture nlysis of pintings using ICA filter nks Nnne vn Noord Tilurg University, Wrndeln 2, 5037 AB Tilurg, The Netherlnds Eri Postm Tilurg University, Wrndeln 2, 5037 AB Tilurg, The Netherlnds Ell Hendriks Vn Gogh Museum, Postus 75366, 1070 AJ Amsterdm, The Netherlnds Keywords: olour-texture nlysis, independent omponent nlysis, imge segmenttion, olour Astrt In humn nd omputer vision, the nlysis of visul texture is of pivotl relevne for ojet reognition nd sene understnding. The nlysis of texture defined y oth intensity nd olour vritions ontriutes to the utomti lssifition in stellite imges, medil imges, or food proessing. Biologilly motivted studies hve proposed Independent Component Anlysis (ICA) s model for the nlysis of olour texture in humn vision. To im of this pper is to determine the effetiveness of ICA for olour texture nlysis in omputer vision. The effetiveness of ICA on the segmenttion of pintings is ssessed in unsupervised nd supervised settings. The results re enourging nd suggest ICA to e suitle sis for olour texture nlysis. 1. Introdution Both in iologil nd rtifiil vision systems, visul texture provides n importnt ue for the reognition of ojets nd senes (Tueryn & Jin, 1993). Visul texture refers to the visul pperne of, for instne, textile or surfes. Texture nlysis enles the mesurement nd quntifition of visul texture. Two ommon pplitions of texture nlysis re texture lssifition nd texture segmenttion. In texture lssifition, imges or imge pthes re lssified into useful lsses. For instne, the texture of Copy- Appering in Proeedings of BENELEARN right 2013 y the uthor(s)/owner(s). piees of fruit my e lssified ording to the type of fruit. In texture segmenttion imges re sudivided into similrly-textured regions. For instne, texture segmenttion n e used to utomtilly sudivide stellite imges into different regions. Although texture is trditionlly nlysed y onsidering the sptil vrition in pixel intensity (grey sle vlues) (Tueryn & Jin, 1993), the pplition exmples given underline the ft tht most nturl textures re defined in terms of oth intensity nd olour (see, e.g., (Cheng et l., 2001)). Suh textures re lled olour textures (Drimren & Wheln, 2001). In reent yers, olour texture nlysis eme n tive reserh field. The min hllenge in this field is to find n effetive wy to integrte intensity nd olour ues in olour texture nlysis (Fernndez & Vnrell, 2012; Wng et l., 2011). The urrent study is prt of reserh projet tht ims to support pinting onservtors in their ssessments of the geing of pints. Pintings onsist of different types of pigment pplied to nvs. A highresolution digitl reprodution of pinting n e onsidered to form mixture of olour texture omponents, orresponding to the vrious pigments, rush strokes, nd the nvs. From signl- or imgeproessing perspetive, the prolem of reovering the omponents is so-lled lind soure seprtion prolem (Kleinsteuer & Shen, 2012). A widely used lgorithm for lind soure seprtion is Independent Component Anlysis (ICA) (Hyvärinen & Oj, 2000). ICA is onerned with finding n unmixing mtrix y whih mixed signl n e deomposed into its soure signls. Whtler, Lee, & Sejnowski (2001) showed tht ICA is iologilly plusile method for the nlysis of nt-

2 Colour-texture nlysis of pintings using ICA filter nks url senes nd tht ICA, when pplied to olour imges, yields integrted representtions of intensity nd olour informtion. Suh ICA representtions my e invoked s filter nks for olour texture nlysis (Jenssen & Eltoft, 2003; Chen et l., 2006). While filter nks for imge (or texture) segmenttion ommonly onsist of pre-defined Gor or Wvelet filters (Ni et l., 2013; Fn, 2012; Jin & Frrokhni, 1991; Wng et l., 2011). ICA-sed filter nks hve een shown to outperform Gor filters (Chen et l., 2006). To im of this pper is to determine the effetiveness of ICA for olour texture nlysis in oth n unsupervised nd supervised setting. Unsupervised ICAsed olour texture segmenttion hs een studied efore (Cheng et l., 2003), ut the uthors report only qulittive results. To the est of our knowledge, we re the first to study ICA-sed olour texture nlysis in supervised nd quntittive setting. The outline of the reminder of the ppers is s follows; in Setion 2 the ICA method is desried. Then, in Setion 3 the experimentl setup is desried, detiling the dtset, experiments nd evlution proedure. The experimentl results re presented in Setion 4 nd the disussion of the results nd onlusions re given in Setion Independent Component Anlysis for olour texture nlysis Our method for performing Independent Component Anlysis to olour texture nlysis onsists of two steps: (1) finding the independent omponents nd tret them s filters, nd (2) onvolving the ICA filters with n imge. In this work the fixed-point lgorithm FstICA y Hyvärinen nd Oj (2000) is used for ll ICA opertions. ICA is vrint of ftor nlysis tht diretly finds the independent omponents of ny non-gussin distriution (Hyvärinen & Oj, 1997) Finding the independent omponents The underlying ssumption of ICA is tht olletion of oserved signls or vetors x onsist of sttistilly independent omponents (Hyvärinen & Oj, 1997). These omponents re denoted y s, nd n e found y mens of liner trnsformtion of the oservtions x with weight mtrix W; s = Wx. (1) Although the weight mtrix W nd the independent omponents s re unknown, the mixing model n e rewritten s x = As, (2) where A represents the mixing mtrix. It is possile to estimte oth the mixing mtrix A nd the omponents s from the oserved signls x using n unsupervised lerning proedure. A is otined suh tht its (pseudo)inverse W multiplied y x i is n estimte of s i. In this work ICA is pplied to otin filter nk g i, onsisting of size D D 3 filters. A weight mtrix W is lerned for eh imge I i y pplying ICA to olletion of rndomly smpled pthes of size D D 3. Eh olletion onsists of fixed mount of 50, 000 imge pthes, smpled from its respetive imge I i. The resulting filter nk is onstruted y reshping eh row of weight mtrix W into D D 3 filter Convolving the ICA filters with n imge The otined filter nks re pplied y onvolving eh imge I i with its respetive filter nk g i s follows; G i (x, y, z) = I i (x, y, z) g i. (3) The outome of the onvolution n e summed ross ll dimensions in order to lulte the energy distriution s follows; f i = N M y=1 x=1 z=1 P G i (x, y, z). (4) The energy vlue desries how strong the intertion of filter is with the imge t given lotion. The energy distriution for two similr res is expeted to e more similr thn the energy distriution of two dissimilr res. The resulting feture mtrix f i onsists of MN g i mtrix, where eh row desries the energy distriution ross olour hnnels for given pixel. The size of the filter nk depends on how mny independent omponents re hosen. For our experiments we vried the numer of independent omponents depending on the olour spe nd for omprility to other results. 3. Experimentl setup The experimentl evlution of our ICA-sed olour texture nlysis method is performed y segmenting pinting into pints nd primed-nvs regions. Both

3 Colour-texture nlysis of pintings using ICA filter nks Figure 1. Duigny s Grden y Vn Gogh (1890). n unsupervised nd supervised setting will e exmined Dtset The dtset is extrted from digitl reprodution of pinting y Vn Gogh: Duigny s Grden, m pinting tht ws reted in Figure 1 shows reprodution of Duigny s Grden. The key hrteristi of this pinting is tht the primednvs is visile in some res of the pinting. Ground truth on these primed-nvs lotions is ville in the form of n imges-sized msk onsisting of inry lel for eh pixel of the imge of Duigny s Grden. The dtset onsists of rndom seletion of 50, 000 D D pthes (D = 8). The imges in I i onsist of two typil res of size , exmples shown in Figure 2, ropped from the high-resolution imge of Duigny s Grden Unsupervised setting We employ two unsupervised methods: k-mens lustering (Cheng et l., 2003) nd grph-sed lustering (Felzenszwl & Huttenloher, 2004), see lso (Peng et l., 2013). We pply k-mens lustering to the energy distriution f i otined y onvolving n imge with the filter nk. The vlue of k determines the extent of the segmented regions (lrge for low vlues of k nd smll for high vlues of k). Segmenting n imge in few lrge regions will often result in high rell, due to mny of the ground lyer pixels eing enpsulted inside the sme region. However, lrge regions re lso muh more orse, nd will thus ontin mny unwnted pixels, resulting in low preision. The enefit of hving few lrge regions is tht it is trivil to mnully selet the est regions. In our experiments, k ws set to 5, 10, nd 100. The grph-sed lustering method hs three djustle prmeters; σ, t nd min. The first prmeter σ is used to smooth the imge using Gussin filter efore segmenting it. The seond prmeter t whih is used to determine the sle of oservtion, higher vlue of t will result in lrger omponents. The third prmeter min is used during post-proessing to enfore minimum region size, preventing too mny smll regions. All experiments were performed with the reommended vlues of 0.8 for σ nd 20 for min. The vlue of t is vried etween the reommended vlue of 500 nd the, in preliminry experiments determined to e pproprite, vlue of Supervised setting Supervised lgorithms mke use of trining dt, or leled dt in order to lern model tht desries the dt nd llows for preditions to e mde out unseen smples. A supervised lssifier opertes y lerning mpping, for eh instne, of the fetures to the provided lel. For the experiments onduted the instnes re represented y single pixels. For eh instne the ground-truth is noted y inry lel, inditing whether or not it is prt of the ground lyer. The fetures onsist of the energy distriution f i. Idelly lssifier n e trined on suset of the dtset nd n e used to urtely segment n entire imge. Severl supervised lssifiers re evluted using stndrd k-fold ross vlidtion pproh whih divides the dt in trining nd test set Clssifiers Three lssifiers were used in order to tkle the prolem t hnd, nmely the MATLAB implementtions of the k-nerest neighour (k-nn), nive Byes, nd RUSBoost lssifiers. k-nn ssigns the unseen smples the (mjority) lel of the k nerest neighouring smples in the trining set (Hstie et l., 2003). For ll experiments desried in this pper the vlue of k ws kept t 1. Nive Byes is proilisti lssifier tht ssigns lels to unseen smple y using Byes rule (Hstie et l., 2003). RUSBoost is n ensemle lerner tht employs wek lerner, tree, nd is prtiulrly well-suited for deling with lss imlne (Seiffert et l., 2010). RUSBoost omines the AdBoost oosting proedure with rndom under-smpling in order to overome lss imlne, results re otined using 10,

4 Colour-texture nlysis of pintings using ICA filter nks () () Figure 2. Imge regions I () nd I () extrted from Duigny s Grden 50 nd 100 lerners Evlution The evlution of our olour texture nlysis method is split into n evlution proedure for the unsupervised setting nd one for the supervised setting. Unsupervised setting. The unsupervised setting my yield more thn two different regions. For instne, k-mens lssifier results in segmenttion into k different regions, lthough our tsk requires only inry segmenttion into nvs nd non-nvs regions. To del with this issue, we inlude only suset of the k regions, nmely those tht ontriute to the overll performne. A region ontriutes to the performne when the numer of true positives for region (rtp ) is greter thn the numer of flse positives for tht region (rf p ) multiplied y. As suh the solution to finding the est omintion of ll regions ri, rest eomes rest = {r 0 < (rtp rf p )}. (5) The vlue of ws optimised in preliminry experiments yielding vlue of Supervised setting. Two ommonly used mesures for evluting the performne of lssifition systems in supervised setting re preision nd rell, whih re lso used to evlute segmenttion performne (Mrtin et l., 2004). Preision is mesure of the orretness of the lssifition. Fewer mistkes led to higher preision. Rell is mesure of ompleteness. The preision of generted segmenttion is lulted y dividing the numer of pixels orretly lssified s ground lyer (true positives) y the totl numer of pixels lssified s ground lyer (true positive + flse tp positive), preision = tp+f p. Rell is lulted y dividing the true positives y the true positives nd the numer of ground lyer pixels tht were not lssified tp s suh (flse negtives), rell = tp+f n. As joint mesure of preision nd rell, we use the F mesure, whih is the (hrmoni) men of preision nd rell: F = 2 (preision rell)/(preision + rell). 4. Results We strt the presenttion of results y showing n exmple of the filter nk otined y pplying ICA to our dtset. Figure 3 is otined y reshping eh row of mtrix W into n-dimensionl filter (where n denotes the numer of olour hnnels). Evidently, the filters pture sptil nd olour informtion from the dtset Unsupervised results The results for the unsupervised experiments re presented in Tles 1 nd 2, for Figures 2() nd 2(), respetively. The fetures tht served s input for the unsupervised experiments were otined using filter nk of 64 filters, lerned from RGB pthes. For the initil experiments we hve hosen n under-omplete sis of 64 rther thn 192 independent omponents in order to filitte fir omprison with the results otined on grey sle imges. For the k-mens lgorithm lrger vlue of k results in higher F-sore nd preision. In the se of I the rell does not hnge muh when hnging the vlue of k. While for I smller vlues of k give etter result. A possile explntion for this n e found in the differene etween the imge regions. The perfor-

5 Colour-texture nlysis of pintings using ICA filter nks Tle 2. Results of unsupervised lgorithms on imge I Method k-mens grph-sed Setting k=5 k = 10 k = 100 t = 350 t = 500 F-sore preision rell () Figure 3. Exmple of ICA filter nk otined for D = 8. Tle 1. Results of unsupervised lgorithms on imge I Method k-mens grph-sed Setting k=5 k = 10 k = 100 t = 350 t = 500 F-sore preision rell mne of the grph-sed lgorithm improves with the lower t vlue. While the performne with the reommended vlue (t = 500) is on pr with k-mens (with k = 100), the lower vlue (t = 350) results in etter segmenttion. In Figure 4() the grey-sle result of the segmenttion using k-mens with k = 100 is shown, whih t first glne ppers to e more omplex thn the olour result of the grph-sed lgorithm shown in Figure 4(). However, every region in the output of the grph-sed lgorithm hs rndomly-ssigned lel, whih mens tht identil setions of the imge in different lotions will e leled differently. This is not the se for the k-mens lgorithm tht lels identil setions in onsistent wy. () Figure 4. Segmenttion result of Figure 2(), () is the outome of k-mens with k = 100, () is the outome of the grph-sed lgorithm. Eh setion is indited y unique olour Supervised results The results for the supervised experiments re presented in Tle 3 nd 4, for Figures 2() nd 2(), respetively. As with the results presented in Setion 4.1 the results shown in Tle 3 nd Tle 4 re the result of onvolutions with filters otined from the 64 independent omponents lerned from RGB imge pthes. The performne of the first lssifier, RUSBoost, is out the sme for oth imges. Inresing the numer of lerners, the min RUSBoost prmeter, results in lower rell, inditing tht upsling the numer of of lerners will eventully result in lower performne. Similrly to RUSBoost the height of the F-

6 Colour-texture nlysis of pintings using ICA filter nks Tle 3. Results of supervised lssifiers on imge I Clssifier RUSBoost(10) RUSBoost(50) RUSBOOST(100) Nive Byes 1-NN F-sore preision rell Tle 4. Results of supervised lssifiers on imge I Clssifier RUSBoost(10) RUSBoost(50) RUSBoost(100) Nive Byes 1-NN F-sore preision rell sore for Nive Byes stems lrgely from the high rell, with the preision eing very low. Inditing neither lssifier is le to urtely distinguish etween the different lsses nd thus ssigns the ground-lyer lel to too lrge regions. The preision nd rell of the k-nn lssifier re very similr, resulting in the highest F-sore mongst ll lssifiers onsidered. While the rell is lower ompred to the other lssifiers the higher preision indites tht the k-nn lssifier is ple of distinguishing etween the mjority of the ground-lyer pixels nd the rest of the pinting. 5. Disussion Figure 5. Imge region I extrted from Duigny s Grden Tle 5. F-sores of the preliminry supervised experiments. Pure refers to results without ny PCA preproessing, PCA(10) re results otined on the 10 remining fetures fter PCA. Filters Colour spe Grey ICA RGB LAB Grey Log-Gor LAB While the presented results re enourging we feel tht two points re not dequtely delt with, whih we ddress y presenting seletion of preliminry results of our ongoing experiments. The first point onerns the influene of the type of imge on the performne. In our urrent investigtions we employ imge regions, suh s I, shown in Figure 5. This imge hs stronger distintion etween the primed-nvs nd pint lyers thn imges I nd I. The seond point is the impt of type of filter nk, olour spes, nd numer of fetures, on segmenttion performne. In our studies we fous on new filter nks, suh s Log-Gor filters (Field, 1987), dditionl olour spes (e.g., grey-sle nd CIELAB), nd dimensionlity redution (using PCA). To provide n impression of how these vritions ffet the segmenttion performne, Tle 5 shows the results otined with the k-nn lssifier. The results Ii Pure PCA(10) on imge I re muh etter thn those on the other two imge regions. We suspet tht this is due to the greter distintion etween the lyers present in this imge, used y the inresed thikness of the pints pplied in this re, ompletely msking the underlying nvs. The Log-Gor filters generlly perform etter thn he ICA filters to degree tht depends on the settings. Dimensionlity redution is detrimentl for the Log-Gor filters, ut not for ICA. These preliminry results hint t diretions for future study. 6. Conlusion Our experiments show tht ICA in supervised setting yields etter results thn in n unsupervised set-

7 Colour-texture nlysis of pintings using ICA filter nks ting. This finding is not surprising, euse using the ground truth for modeling texturl differenes yields stronger model. More importntly, the k-nn performne otined with ICA is resonle nd results otined when nlysing olour texture outperform those otined on grey-sle texture. While Log-Gor filters show etter results in the setting without PCA, with PCA the ICA filters outperform the Log-Gor filters. Whih is notle s PCA drstilly redues the mount of time required y the lssifier. As we extend the method to the entire pinting, the size of the dtset inreses y ftor of 40, whih would put the required time well over wht would e fesile for ny prtil pplition. Furthermore, there is no differene etween the grey sle nd olour Log-Gor settings, whih implies tht the Log-Gor method does not effiiently inorporte the dditionl hromti informtion. Although we hve otined resonle performne on olour texture nlysis tsk using n ICA filter nk, there re two points of onern. First, it is unler to wht extent the omplexity of the tsk influenes the performne. A omprle experiment on pinting with less fine-grined distintion etween lyers would help in nswering this question. Seond, the inry leling of the trining dt might not hve een detiled enough for the lssifiers to lern ler distintion. Further experiments re required to determine whether riher leling of the trining dt would improve performne. Notwithstnding these onerns, our results re enourging nd suggest ICA to e suitle sis for olour texture nlysis. Future reserh will fous on exploring wys to improve lssifition performne y experimenting with onditionl rndom fields, other olour spes nd n extended dtset. We expet tht inorporting n intertive user feedk model will improve performne s well s e prtil solution to otin leled trining dt with reltively little effort. Aknowledgments The reserh reported in this pper is performed s prt of the REVIGO projet, funded y NWO in the ontext of the Siene4Arts reserh progrm. The uthors thnk the nonymous reviewers for their helpful omments. Referenes Chen, X.-w., Zeng, X., & vn Alphen, D. (2006). Multi-lss feture seletion for texture lssifition. Pttern Reognition Letters, 27, Cheng, H. D., Jing, X. H., Sun, Y., & Wng, J. L. (2001). Color imge segmenttion: Advnes nd prospets. Pttern Reognition, 34, Cheng, J., Chen, Y.-W., Lu, H., & Zeng, X.-Y. (2003). Color- nd texture-sed imge segmenttion using lol feture nlysis pproh. Pro. SPIE 5286, Third Interntionl Symposium on Multispetrl Imge Proessing nd Pttern Reognition, Drimren, A., & Wheln, P. F. (2001). Experiments in olour texture nlysis. Pttern Reognition Letters, 22, Fn, J. (2012). Feture lerning sed multi-sle wvelet nlysis for texturl imge segmenttion. In D. Jin nd S. Lin (Eds.), Advnes in omputer siene nd informtion engineering, vol. 168 of Advnes in Intelligent nd Soft Computing, Springer Berlin Heidelerg. Felzenszwl, P. F., & Huttenloher, D. P. (2004). Effiient grph-sed imge segmenttion. Interntionl Journl of Computer Vision, 59, Fernndez, S. A., & Vnrell, M. (2012). Texton theory revisited: g-of-words pproh to omine textons. Pttern Reognition, 45, Field, D. J. (1987). Reltions etween the sttistis of nturl imges nd the response properties of ortil ells. J. Opt. So. Am. A, 4, Hstie, T., Tishirni, R., & Friedmn, J. H. (2003). The Elements of Sttistil Lerning. Springer. Correted edition. Hyvärinen, A., & Oj, E. (1997). A fst fixed-point lgorithm for independent omponent nlysis. Neurl Computing Surveys, 9, Hyvärinen, A., & Oj, E. (2000). Independent omponent nlysis: lgorithms nd pplitions. Neurl Networks, 13, Jin, A. K., & Frrokhni, F. (1991). Unsupervised texture segmenttion using gor filters. Pttern Reognition, 24, Jenssen, R., & Eltoft, T. (2003). Independent omponent nlysis for texture segmenttion. Pttern Reognition, 36,

8 Colour-texture nlysis of pintings using ICA filter nks Kleinsteuer, M., & Shen, H. (2012). Blind soure seprtion with ompressively sensed liner mixtures. IEEE Signl Proessing Letters, 19, Mrtin, D. R., Fowlkes, C. C., & Mlik, J. (2004). Lerning to detet nturl imge oundries using lol rightness, olor, nd texture ues. IEEE Trnstions on Pttern Anlysis nd Mhine Intelligene, 26, Ni, W., Go, X., & Yn, W. (2013). Sr imge segmenttion sed on gor filter nk nd tive ontours. In J. Yng, F. Fng nd C. Sun (Eds.), Intelligent siene nd intelligent dt engineering, vol of Leture Notes in Computer Siene, Springer Berlin Heidelerg. Peng, B., Zhng, L., & Zhng, D. (2013). A survey of grph theoretil pprohes to imge segmenttion. Pttern Reognition, 46, Seiffert, C., Khoshgoftr, T. M., Vn Hulse, J., & Npolitno, A. (2010). Rusoost: A hyrid pproh to lleviting lss imlne. IEEE Trnstions on Systems, Mn, nd Cyernetis, Prt A, 40, Tueryn, M., & Jin, A. K. (1993). texture nlysis. In C. H. Chen, L. F. Pu nd P. S. P. Wng (Eds.), Hndook of pttern reognition & omputer vision, hpter Texture nlysis, River Edge, NJ, USA: World Sientifi Pulishing Co., In. Whtler, T., won Lee, T., & Sejnowski, T. J. (2001). The hromti struture of nturl senes. The Journl of the Optil Soiety of Ameri A, 18, Wng, X.-Y., Wng, T., & Bu, J. (2011). Color imge segmenttion using pixel wise support vetor mhine lssifition. Pttern Reognition, 44,