Paintings at an Exhibition EE368 Group 17 Project Report

Transcription

1 1 Pantngs at an Exhbton EE368 Group 17 Project Report Mthun Kamat Stanford Unversty mkamat at stanford dot edu Abstract An algorthm s developed and mplemented to recognze pantngs on dsplay at the Cantor Arts Center based on snapshots taken wth a camera phone. The tranng mages were used to create a database of low dmensonal fsher mages. The mage processng was carred out n three steps. Frst, the tranng mages were gray scaled and down sampled from the hgh resoluton color patterns whle preservng the key features. The mages were also adjusted for brghtness by normalzaton. Second, the tranng mages were segmented to dentfy edges correspondng to the frames of the pantngs. The locaton of the frames of the pantngs was used to crop and resze the tranng mages to a consstent scale. Fnally, the tranng mages were transformed to a lower order database usng Fsher Lnear Dscrmnant (FLD) transformaton. The test mage s compared aganst the database usng Eucldean dstance nearest neghbor smlarty matchng. The algorthm demonstrated reasonably good computatonal effcency and accuracy. Ths algorthm was developed as part of the project requrements for EE368 class at Stanford Unversty. The algorthm was selected by tradng off the computatonal and accuracy requrements for the project. The code was mplemented n Matlab TM verson 6.5 usng the Image Processng Toolbox. Ths work could be useful as part of an electronc museum gude wheren the users would pont ther camera phone at a pantng of nterest and would hear commentary based on the recognton result. Keywords augmented realty, egen mages, FLD, fsher mages, mage processng, pantng recognton I I.INTRODUCTION MAGE RECOGNITION s a complex problem due to the dffculty n recognzng and matchng mage features n the presence of varable orentaton, llumnaton, and scale among other thngs. Addtonal complexty s nvolved n the case of recognton of pantngs because no two pantngs are fundamentally alke unlke faces where certan facal features (e.g. nose, eyes) tend to be smlar between dfferent ndvduals. The applcatons for mage recognton are numerous and growng. For example, an electronc museum gude could be envsoned where the user would pont ther camera phone at a pantng of nterest and would hear commentary based on the recognton result. Such an applcaton would nvolve processng of the mage and comparng wth a known database of lbrary mages. The mages could vary from pantngs, art dsplays to hstorcal dsplays such as World War II artfacts. The matched mage could provde the user the relevant audo vsual nformaton about the mage at hand. Applcatons of ths knd are usually referred to as "augmented realty" applcatons. Implemented on hand held moble devces, they are called "moble augmented realty." The task of the project [1] s to focus on the mage recognton part of the above applcatons. Photographs were taken n the European Gallery of the Cantor Arts Center n Calforna. Three photos were taken of each of 33 pantngs. The photos were taken wth a Noka N93 wth a resoluton of 3 mega pxels. The pantngs dd not represent any certan type and ncluded portrats as well as landscapes. An mage recognton algorthm wll perform the necessary processng on the tranng mages such that a new mage from the same set of pantngs could be dentfed accurately n a computatonally effcent manner. II.ALGORITHM DESCRIPTION A.ALGORITHM SELECTION A lterature search was performed to dentfy the most relevant mage processng algorthm. A large body of lterature exsts for mage processng related to face recognton [2 5]. The most cted algorthms nclude the egen faces or fsher faces method that adopt Prncpal

2 Components Analyss (PCA) and Fsher Lnear Dscrmnant Analyss (FLD). PCA technques, also known as Karhunen Loeve methods [2], choose a dmensonalty reducng lnear projecton that maxmzes the scatter of all tranng mages. A drawback of ths approach s that the scatter beng maxmzed s due not only to the between class scatter that s useful for classfcaton, but also to the wthn class scatter that, for classfcaton purposes, s unwanted nformaton [3]. Fsher's Lnear Dscrmnant (FLD) [6] s an example of a class specfc method, n the sense that t tres to shape the scatter n order to make t more relable for classfcaton. Ths method selects a transformaton of the raw mages such a way that the rato of the between class scatter and the wthn class scatter s maxmzed. Both FLD and PCA methods are not nvarant to mage algnment and scalng [7]. Hence, preprocessng would be nvolved before applyng these methods. More recently, a more robust approach has been developed for feature dentfcaton. The Scale Invarant Feature Transform (SIFT) algorthm [8] detects key features that are nvarant to mage scale and rotaton, and are shown to provde robust matchng across a substantal range of affne dstorton, change n 3D vewpont, addton of nose, and change n llumnaton. The smplest algorthm that can be chosen for ths partcular problem s to do a nearest neghbor search of par wse correlaton of the mage wth every mage n the tranng database. Obvously such an algorthm wll requre very hgh computatonal space and may be too senstve to dsturbances. Selecton of the mage processng algorthm nvolved consderaton to the followng elements..the mages presented challenges due to low lght condtons. The algorthm had to adjust for llumnaton and brghtness. b. The mages were taken wth a hand held camera and were qute nosy and moton blurs and/or defocus. c. The algorthm had to be nvarant to rotaton or algnment, and scalng as the mages were taken from dfferent vantage ponts and orentaton vews. d. The algorthm had to be computatonally nexpensve as the turn around on matchng had to be on the order of less than a mnute. A SIFT based mage recognton method [8] would automatcally cover a c. However, the algorthm s patented (not readly usable n a commercal settng), and s not computatonally effcent n Matlab wthout the usage of external nterfaces such as MEX fles, compared wth other avalable methods. On the other hand, FLD approach seemed to be computatonally very effcent and academcally free to use. Based upon the dscusson provded n [1], FLD seemed more relevant than PCA. However, both those algorthms are not nvarant to scalng and algnment and hence preprocessng would be requred before applyng the FLD transformaton. B.PROCEDURE The proposed algorthm s made up of three steps. The step wse procedure s descrbed n Fgure c b c02f9021a e c000000fb bc d00021a d5c0000a b030c d c000000fb02ceff d e f6d616e d d a2d a04f70220d d Fgure 1: Step wse Image Processng Algorthm for detecton of pantngs In the frst step, the raw tranng mages are pre processed to account for the varatons n lghtng, nose, blur and scalng. Also, mages are down sampled n order to reduce the computatonal tme. In the second step, the mages are decomposed ntally based upon the PCA method (Karhunen Loeve Transform KLT) to a reduced order dmenson. Ths step avods sngularty of the wthn class scatter matrx n FLD. Subsequently, the reduced order mages are used to compute the transformaton matrx usng the FLD transformaton approach. The transformaton s appled on the tranng mages and the reduced order tranng mages are stored n the database for smlarty matchng. In the fnal step, the test mages are processed usng the smlar approach dentfed n the frst step, transformed usng the transformaton matrx dentfed n step two, and fnally compared wth the reduced order mage database usng the nearest neghbor Eucldean dstance smlarty matchng method. The algorthm s coded n Matlab TM verson 6.5. Commands from the Image Processng Toolbox wthn the Matlab lbrary are used heavly n ths algorthm. C.IMAGE PRE PROCESSING 1)Convert to Gray Scale The tranng mages obtaned from the lbrary were n the n the form of RGB array. The mage s converted nto gray scale wth pxel values depctng ntenstes between 0 to 255. Fgure 2 shows one of the orgnal pantngs n gray scale. The gray scale mage retaned the key features of the color mages

3 wthout compromsng the effcency of the algorthm. The gray scale mage has sgnfcant computatonal advantage due to lower storage requrement. 3)Normalzaton In order to elmnate dfferences wth regard to contrast and brghtness, the mages are normalzed usng a dynamc range expanson. For example, f the ntensty range of a gray scale mage s 30 to 190, then the pxel values are frst subtracted by 30 and then scaled by a factor of 255/190 n order to acheve normalzaton from 0 to 255. Fgure 4 shows mage The Resurrecton after normalzaton. 3 Fgure 2: Gray scale mage of The Resurrecton. 2)Down Sample The gray scale mage s then down sampled to 15% of the orgnal mage sze. Ths step was performed to avod the computatonal burden of a hgh resoluton mage on a desktop PC. As wll be descrbed below, the KLT method nvolves the calculaton of covarance matrx. Such matrx multplcaton operatons were found to be computatonally extremely expensve especally on a desktop PC. Wth regard to the objectve of ths work, t was found that a down sampled mage retaned the features to a suffcent level provded the correct down samplng scheme s adopted. In ths work, the mage was down sampled usng a blnear nterpolaton. Fgure 3 shows one of the pantngs down szed by a factor of 0.15 usng the blnear nterpolaton. Fgure 3: Down sampled verson of The Resurrecton Fgure 4: Normalzed The Resurrecton mage. 4)Isolaton of the pantng Ths s one of the most mportant steps n the pre processng algorthm. The orgnal mages were taken at dfferent vantage ponts. See Fgure 5a and 5b for two mages of the same pantng taken from a dfferent dstance and angle from each other. Clearly, the mages could possbly contan the pantngs as well as other background nformaton such as corners n the buldng, clps of other pantngs or objects. Frst, the key edges n the mages are detected usng an edge detecton algorthm. In ths work, the canny edge detecton algorthm s used as t optmzes the ablty of several types of edge detectors and also demonstrates nvarance to nose. The masked mage wth edges s then dlated usng a structurng element of a 3x3 matrx of ones n order to clearly segment the edges of the pantng. Note that the edges of the pantng are bascally the frames of the pantngs. Second, the edges correspondng to the frames of the pantngs are used to segment the pantng tself from the rest of the background. Ths s performed usng a smple segmentaton operaton usng labelng. Fgure 5c and 5d show the segmented mages after edge detecton and labelng. A vsual nspecton of Fgure 3 shows that the down sampled mage has retaned the key features of the orgnal mage.

4 ths step proves to be qute robust to changes n dmensons, orentaton as well as scale. Fgure 5a Fgure 5c Fgure 5b Fgure 5d Fgure 5: [a, b] Two gray scale mages of the same pantng; [c, d] segmented mages usng Canny edge detecton and segmentaton usng labelng. 5)Image scalng The locaton of the four corners of the edges solated n the prevous step are computed and then the gray scaled mage from Step 3 s cropped to the sze of the box correspondng to the edges. The cropped mage s then reszed to the orgnal sze thereby ensurng that all mages are of the same scale and sze. D.PROJECTION TRANSFORMATION 1)Karhunen Loeve Pre Transform The Karhunen Loeve Transform method s used to perform a Prncpal Components Analyss (PCA) on the mages. The PCA analyss maxmzes the scatter between the mages thus extractng dstnct features from the mage. Assumng a set of mages n vector form wth each mage of sze MxN, F = f, f,..., f } { 1 2 L The KLT transformaton can reduce the dmensonalty of the matrx F of sze MNxL to JxL by representng the mage by J coeffcents. C = WF Where W s the transformaton matrx. The transformaton matrx s computed usng the Srovch and Krby method [9] L L of computng egen mages correspondng to the 0 Sv largest egen values accordng as where S s the tranng set matrx, S = { f1 µ, f2 µ,..., f L µ } v, and are the egen vectors of S T S. Ideally, L0 would be chosen to mnmze the number of egen mages requred to represent the tranng set. However, snce our prmary objectve s to perform a Fsher transform, we retan the frst n egen mages, where n represents the number of class of mages. Fgure 6a Fgure 6b Fgure 6: [a, b] Cropped and rescaled mages of the same pantng Fgure 6 shows the scaled mages of a pantng that were taken from two dfferent angles. Ths step ensures that the scale nvarant weakness of the egen mages algorthm s not exploted. As wll be descrbed n the Results secton below, 2)Fsher Lnear Dscrmnant Transform The KL transform maxmzes scatter wthn the lnear subspace over the entre mage set regardless of the classfcaton task at hand. However, the set of tranng mages ncludes multple mages of the same pantngs. Specfcally, three mages of 33 pantngs are provded n total. The FLD algorthm s desgned to explot the nformaton wth each class of pantng. The FLD approach, frst proposed n 1936, maxmzes the between class scatter whle mnmzng the wthn class scatter. Assume R B and R W as the between class and wthn class scatter matrces. R R B W = = c = 1 c N ( µ µ )( µ µ ) = 1 f Class( ) ( f µ )( f H µ ) The FLD approach nvolves calculatng the egen values and egen vectors correspondng to the followng soluton H

5 5 R w = λ R w Where w are the egen vectors. B The KL transform was used to generate a reduced order set of 33 egen mages. Ths ensures that for cases where L << MN, the R W matrx s not sngular. The egen values and egen R 1 ( ) vectors of the matrx, R W B were computed usng sngular value decomposton. The egen values of the FLD approach were vsually nspected to decde the reduced order dmenson of the FLD transformaton matrx W E.SIMILARITY MATCHING 1)Test Image Transformaton The test mage s subjected to the preprocessng outlned n the Step 1 of the prevous secton. The processed mage s projected on to a lower dmensonal space of sze 10 by convoluton wth the FLD transformaton matrx 2)Mnmal Eucldean dstance matchng The lower dmensonal vector of test mage s compared to every mage n the tranng database by computng the Eucldean dstance between the test mage and every tranng mage. The values are sorted based upon the nearest neghbor dstance. 2 ξ = mn( q Where q s the reduced test mage, and p k s the reduced tranng mage n the database, k = 1 to L (number of mages) p k 2 ) Egen value number Fgure 7: Egen values of the reduced mage. X axs denotes the number of egen values. Fgure 7 shows the dfferent egen values of the FLD approach. Clearly, most of the hgh energy content of the mages s condensed n the frst 10 egen values. A senstvty analyss was performed by varyng the dmenson of the transformaton matrx from eght to 15, and t was found that the dmenson of 10 s suffcent to preserve the hgh energy characterstcs whle mantanng algorthm accuracy. Therefore, the frst 10 sorted egen vectors consttute the FLD transformaton matrx. 3)Database Generaton and Storage The PCA and FLD transforms are appled n seres to mages that were pre processed usng steps outlned n Step 1. Snce the FLD transform s appled on the mean mages wthn each class, the fnal sze of the database of mages was 10x33 where 10 represents the dmenson of the egen (fsher) mages and 33 represent the number of types of pantngs n the database. Thus, the FLD transform was successful n reducng the dmenson of an mage of sze MN to a vector of 10 fsher coeffcents. 3)Output A two step approach s utlzed to match the pantng. Frst the nearest neghbor value s used to detect a possble matchng canddate n the database. Next, the nearest neghbor Eucldean dstance s compared wth a pre set value of theta. 1 θ = max( 2 p p j For, j = 1 to L (L = number of mages) The followng rejecton system s appled. f ξ > θ, pantng not recognzed else, pantng recognzed The followng functonal nterface provded the results [pantng_ttle] = dentfy_pantng(mg) Where mg: heght by wdth by 3 unt8 array representng a 24 bt RGB mage obtaned through mread( ). pantng_ttle: A character array ndcatng the ttle of the domnant pantng contaned n the nput mage. III.RESULTS The algorthm was tested n a seres of three steps. Frst, the tranng mages used n the algorthm were used as test mages to test the self relablty and computatonal speed. As expected, the algorthm successfully dentfed 100% of the nput mages. Each call to the functon approxmately took 33 seconds for processng. Second, ten of the tranng mages were altered by nducng a combnaton of nose, blur and algnment changes. The algorthm successfully detected 95% )

6 of the nput mages. The mages that were not successfully recognzed were the ones that were algned dfferently (vertcal angle was altered). Ths provded feedback on the weakness of the approach to changes n mage orentaton. Fnally, to test the lmts of the algorthm, a set of 38 test Fgure 8a 8b mages [10] completely dfferent from the tranng database were tested. Some of the test mages ndeed were very challengng because the camera used to capture the mage was a low resoluton camera and the vantage ponts were delberately chosen to be extremely strngent. Fgure 8: [a] Gray scale test mage (courtesy [xx]); [b] Processed test mage usng the algorthm Fgure 8 shows a test mage and the output of the preprocessng of the test mage. The algorthm accurately detected 78% of the test mages. Also, the average tme for the computaton of the results was only 3.35 seconds. The dfference n the computatonal speed between the test mages and the tranng mages was prmarly attrbuted to the sze of the nput mage. The test mages were generated wth a low resoluton camera and hence were of smaller sze. The result showed that the algorthm ndeed was robust to most of the varatons n the nput mages ncludng blurrng, nose, scalng, and brghtness. A study of the cases where the algorthm faled to detect the test mage revealed the common theme of the ssue wth detecton of unalgned mages. IV.CONCLUSIONS An mage processng algorthm was developed as part of the project requrements for EE368 class at Stanford Unversty. The algorthm processed mages of pantngs of dfferent types, scale and orentaton. The mages were transformed nto a reduced dmensonal space usng Fsher Lnear Dscrmnant (FLD) transformaton. The algorthm demonstrated reasonably good computatonal effcency and accuracy. The algorthm suffered from accuracy ssues related to mage orentaton. The success rate of the algorthm n matchng mages needs to be mproved further before the code could be mplemented n commercal software for mage recognton of pantngs. Future work has been dentfed to mprove the features of the algorthm. V.LIMITATIONS AND FUTURE WORK The algorthm was not desgned to be fully robust aganst mage algnment. FLD and PCA based methods are known to be susceptble to scale nvarant ssues. The testng of the algorthm aganst the test mages revealed ths flaw. There are two proposals for future work wth ths algorthm. Frst, the pre processng of the mages before the FLD transformaton should nvolve some sort of algnment of the mages. One method s to combne SIFT based feature detecton wth fsher mages. One can use the SIFT based algorthm to detect key features ncludng the scale and the orentaton. Ths nformaton could be used to detect the major orentaton of the mage and the mage could then be rotated to adjust for the orentaton. Second, the computatonal speed of the algorthm could be mproved by creatng C code for some of the major sub routnes (e.g. scalng sub routne). The C code could be converted nto a Matlab MEX fle, whch wll mprove the computatonal effcency of the overall algorthm. ACKNOWLEDGMENT The author would lke to thank Prof. Bernd Grod and EE368 Sprng Teachng Assstants for ther help throughout the course and especally durng trouble shootng. The author also acknowledges Mr. Jacob Mattngley, a fellow student, for gracously sharng test mages from the Canton Arts Center. Those mages were used to evaluate the lmts of the algorthm. REFERENCES 1] B. Grod, Lecture notes for EE368: Dgtal Image Processng, Sprng 2007, Stanford Unversty. 2] Javer Ruz del Solar, and Pablo Navarrete, Egenspace based face recognton: A comparatve study of dfferent approaches, Applcatons and Revews, vol. 35, no. 3, Aug ] P. Belhumeur, J. P. Hespanha, and D. Kregman, Egenfaces vs. Fsherfaces: Recognton usng class specfc lnear projecton, IEEE transactons on PAMI, July ] R. Chellappa, C. Wlson, and S. Srohey, Human and machne

7 7 recognton of faces: A survey, Proceedngs of the IEEE, vol. 83, no. 5, pp , ] A. Samal and P. Iyengar, Automatc recognton and analyss of human faces and facal expressons: A survey, Pattern recognton, vol. 25, pp , ] R. Fsher, The use of multple measures n taxonomc problems, Ann. Eugencs, vol. 7, pp , ] M. Turk and A. Pentland, Egenfaces for recognton, J. Cogn. Neurosc. vol. 3, no. 1, pp , ] Davd G. Lowe, Dstnctve mage features from scale nvarant feature keyponts, Internatonal journal of computer vson, ] L. Srovtch and M. Krby, Low dmensonal procedure for the characterzaton of human faces, J. Optcal Soc. of Amerca A, vol. 2, pp , ] J. Mattngley, test mages from the Canton Arts Center, clcked on personal camera, Sprng 2007.