IMAGE MATCHING WITH SIFT FEATURES A PROBABILISTIC APPROACH

IMAGE MATCHING WITH SIFT FEATURES A PROBABILISTIC APPROACH Jyot Joglekar a, *, Shrsh S. Gedam b a CSRE, IIT Bombay, Doctoral Student, Mumba, Inda jyotj@tb.ac.n b Centre of Studes n Resources Engneerng, IIT Bombay, Assocate Professor, Mumba, Inda shrsh@tb.ac.n Commsson III - WG III/5 KEY WORDS: Image, Matchng, Feature, Extracton, Reconstructon. ABSTRACT: An mage matchng algorthm s presented n ths paper. A set of nterest ponts known as SIFT features are computed for a par of mages. Every keypont has a descrptor based on hstogram of magntude and drecton of gradents. These descrptors are the prmary nput for the mage correspondence algorthm. Intal probabltes are assgned for categores (probable matches) consderng a feature pont assgnment to one of the category as a classfcaton problem. Baye s theorem s used for assgnng ntal probabltes. For selectng the neghbours for the left keypont, a fxed number of xels around the keypont, consdered as a wndow, are selected. The neghbours of the rght keypont are based on nspecton of par of mages and the dsparty range. The probablstc estmates are teratvely mproved by a relaxaton labelng technque. The neghbour keyponts whch wll contrbute to mprove the probablty s based on consstency property. The algorthm s effectve for matchng stereo mage par and these correspondences can be used as nput for 3D reconstructon.. 3D reconstructon. INTRODUCTION 3D reconstructon s a problem of recoverng depth nformaton from ntensty mages. Physcal pont n space s projected onto dfferent locatons on mages f the vewpont for capturng the mages s changed. The depth nformaton s nferred from the dfference n the projected locatons. Two computatonal problems are assocated wth 3D reconstructon from two or more mages:. Feature Correspondence 2. Structure estmaton Feature correspondence conssts of followng steps: Affne nvarant nterest ponts are detected n each mage. A descrptor s assgned to a regon (whch s affne nvarant) around each nterest pont. Image correspondence algorthm fnds a set of potental match par between two adjacent mages. Prunng of these matches s done by usng consstency flters lke symmetry, dsparty gradent etc. Problems of structure estmaton consst of followng steps: Fundamental matrces are computed for every par of consecutve mages n an overlapng mage sequence. A set of potental matches obtaned from mage correspondence algorthm are used to compute fundamental matrx. Potental trple matches are obtaned from fundamental matrces of consecutve mages. These trple matches are used to compute the tr-lnear tensor. The tensor computed ths way encodes the nformaton of three mage pars. Therefore correspondence produced wll be correct and robust methods used for dscardng correspondence gve more accurate results. Dense reconstructon of the scene s a fnal goal for 3D reconstructon. So after computng tr-lnear tensor followng steps are undertaken (Roth and Whtehead, 2000): Rectfcaton of mage sequence so that epolar lnes are made horzontal. A stereo algorthm s run to compute dense depth from rectfed mage pars. Calbraton of mage sequence to move from projectve to metrc reconstructon. In many applcatons lke, ndustral assembly and nspecton, robot obstacle detecton and medcal mage analyss the mportant computer vson task s recovery of three dmensonal structures from two dmensonal dgtal camera mages. In mage formaton process of the camera the depth nformaton about the scene or object s lost. The 3D structure or depth nformaton has to be nferred by analyss of 2D ntensty mages..2 Structure from stereo Structure from stereo method uses camera mages that are taken from dfferent vewponts. For bnocular stereo, a sngle par of mages of the same scene or objects s taken smultaneously by two cameras located at two dfferent spatal locatons and sometmes wth dfferent orentaton. Use of stereos for depth percepton n human vson s a well known phenomenon. Structure from stereo smply refers to the class of computer vson algorthm that apples the same prncple for nferrng depth nformaton from mages taken from dfferent vew ponts. The left and rght camera captures a par of mages I L (f), I R (f) smultaneously when no change occurred n the scene or object between the acquston of two mages. The dfference between projected postons of a pont n the left & rght mages s referred as dsparty. For a whole mage a collecton of dsparty s computed and known as dsparty map. 7

The feature correspondence problem can be best explaned by an example. For nstance a physcal 3D pont s projected on to Image X as pont and mage Y as pont 2. Then pont and pont 2 are sad to be correspondences. Hence the feature correspondence or feature matchng problem s to fnd the pont 2 on mage Y gven the locaton of pont on mage X. Human vson s superb n solvng ths problem. Solvng the problem by automaton of process by computers s rather dffcult. It searches the whole mage Y for a pont on mage X. Some constrants can narrow down the search, but f suffcent constrants are not there the problem becomes dffcult. The second problem of structure estmaton s relatvely easy n comparson. After solvng the correspondence problem a set of ponts are computed. If ntrnsc and extrnsc parameters of the camera are known then exact reconstructon n absolute coordnates s possble. However the accuracy of the reconstructon depends on accuracy of these parameters. In addton, any errors n solvng the correspondence problem between two mages also affect the accuracy of the reconstructon. So, even f ntrnsc and extrnsc parameters are known, the challenge remans for develong the matchng algorthm that reduces the errors n the preprocessng steps to estmate the structure. 2. DETECTOR AND DESCRIPTOR 2. Pont Detectors and Descrptors Parts of the mage that have specal propertes and have some structural sgnfcance are usually referred as mage features. The regons havng vsually dentfable textures are also referred as mage features. Some of the examples are edges, corners, mage gradents etc. Many computer vson applcatons have feature extracton process as an ntermedate step for locatng partcular elements on an mage. Whle extractng features some of the mportant factors to be consdered are nvarance, detectablty, nterpretablty and accuracy. Many applcatons n the area of photogrammetry and computer vson use feature extracton as prmary nput for further processng and analyss. Features are used for mage regstraton, 3D reconstructon, moton trackng etc. Invarance property of feature extractor s very mportant as under dfferent transformatons (geometrc and radometrc) the same features should be detectable n par of stereo mages, so that they wll be useful for matchng process. (Remondno, 2006) 2D locatons n the mages are located by the detectors. After analyzng the regon around the locaton a descrptor s assgned to the locaton (nterest pont) whch characterzes the nterest pont under consderaton wth respect to ts neghborng ponts, usng nformaton about neghborng ponts lke ntensty varaton, change n gradent, hstogram consderng gradent drecton and magntude. 2.2 SIFT Algorthm In SIFT descrptor (Lowe, 2004) DoG detector s used to detect nterest ponts and the extracted regons are descrbed by a vector of dmenson 28. The descrptor s normalzed by dvdng the descrptor vector by square root of sum of the squared components, so that the descrptor becomes llumnaton nvarant. A 3D hstogram of gradent locaton and orentaton s used as a descrptor. Wth varous measures t s demonstrated that SIFT descrptors outperform (Mkolajczyk and Schmd, 2003). Extended verson of SIFT descrptor was presented n (Mkolajczyk and Schmd, 2004). It s known as gradent locaton and orentaton hstogram (GLOH). As number of drectons chosen to represent the hstogram n GLOH are more than SIFT the sze of the descrptor s large n GLOH descrptor. The sze s reduced usng prncple component analyss. Nowadays many detectors and descrptors algorthms are avalable for detectng corners edges and regons of nterest. A vector s assocated wth t as a descrptor. The SIFT algorthm by Lowe s explaned here whch s used to provde prmary nput to the mage matchng algorthm explaned n secton 3. The detected regon should have a shape whch s a functon of the mage. To characterze the regon nvarant descrptor s computed for the extracted regon. For computng SIFT features and assgnng descrptors to the features followng procedure s used. : A pyramd of mages s constructed wth dfferent scales of Gaussan functon. From these Gaussan smoothed mages Dfference of Gaussan mages are computed at dfferent scales. Dfference of Gaussan functon detects the nterest ponts nvarant to scale and orentaton n scale-space. The Dfference of Gaussan functon wll have strong response along edges, though the locaton along the edge s poorly determned, as these locatons are unstable to small amount of nose. A poorly defned peak n the Dfference of Gaussan functon wll have a large prncple curvature across the edge but small along perpendcular drecton. Over all scales mage locatons are found to detect the extrema. Scales of keypont s used to select the Gaussan smoothed mage of closest scale, so that the computatons are performed n scale nvarant manner. For keypont localzaton a model based on Taylor seres s ft to every keypont locaton and scale. Here stablty (.e. nvarance to transformaton) s the measure of selectng the nterest ponts. For each mage sample for the scale L, gradent magntude and orentaton s computed usng xel dfference. Consderng mage gradent at every keypont one or more orentatons are assgned to the keyponts. These orentatons and the respectve magntude at selected scales are used to construct a 3D hstogram for the regon around the keypont. The descrptor computed usng these gradent magntude and orentaton at each mage sample pont s weghted by Gaussan wndow. A Gaussan weghtng functon wth σ equal to one half of the wdth of the descrptor wndow s used to assgn a weght to the magntude of each sample pont. Ths Scale nvarant feature descrptor for every keypont s of dmenson 28 (Lowe, 2004). 3. MATCHING MODEL When the set of keyponts are found next step s to construct a set of possble matches. Ideally we want to match each keypont n the left mage, whch s consdered as reference mage, wth a keypont n the rght mage. But n realty n a stereo mage par we can fnd vald matches for some of the keyponts n the left mage. A prmary nput to the matchng algorthm s a set of keypont wth ther descrptors computed usng SIFT algorthm, explaned n the secton 2.2. An mage correspondence algorthm s proposed and presented n detal steps as below. 8

. Key-ponts selecton n both the mages (left and rght) wth SIFT. 2. For every keypont from both the mages a descrptor s computed as below ) Around every keypont a xel area of sze 6 x 6 s consdered. ) For each sample of sze 4 x 4 gradent magntude & orentaton are assgned. ) A hstogram of Gradent orentaton showng 8 bns gves a descrptor for every 4 x 4 sample sze. v) For a 6 x 6 sample sze around the keypont a descrptor vector of dmenson 4 x 4 x 8 s obtaned. 3. An approxmate maxmum dsparty range s found by vsual nspecton of few matchng keyponts n the stereo mage par. The dsparty s present n the left and rght mage as the stereo mages are captured from dfferent vewponts and orentatons. 4. An area s selected around every rght keypont node, consderng possble maxmum dsparty range. 5. All the keyponts are found n the area selected n step 4, around a rght keypont node n the rght mage. 6. The procedure of step 4 and 5 s teratvely performed for all the rght keyponts and the area around each rght keypont s selected consderng the approxmate maxmum dsparty range. 7. The procedure n steps 4 and 5 s repeated for all left keyponts from left mage teratvely. But here area around the left keypont s a fxed sample area of sze 6 x 6 8. As shown n fgure, the left keypont node b s pared wth every rght keypont node c and the par s called as category par 9. For every category par, Eucldan dstance between the descrptors of the keyponts s calculated. 0. Weght s assgned to every rght keypont c, n the selected area. The weght s nversely proportonal to the Eucldan dstance between the correspondng descrptors.. For every category c the weght s calculated as w k ε + =, c c k s a postve constant 2. A dsparty category whch assocates hghly smlar pars of regon wll have large weght value. 3. w (c) wll be n the nterval [0, ] and weght s nversely proportonal to Eucldan dstance. 4. For every category set c, c s undefned dsparty category. 5. Consder weght w (c) for c whch s () undefned.e. keypont b (x,y) from left mage does not correspond to any keypont n the rght selecton area of rght mage. 6. The weghts can not be used as probablty estmates as w (c) s undefned and weghts wll not sum up to. 7. Consderng the keypont matchng as a classfcaton problem, b s classfed to one of the category c. Intal probablty for undefned category s gven as o = max ( w ) (2) c c 8. By applyng Baye s rule o o ( c ) = ( c ) ( ), c c (3) p ( c ) : condtonal probablty that b has category c as matchng, gven that b s matchable ( ) : pror probablty that b s p o 9. Estmatng p ( c ) matchable as below w ( c ) = (4) L w ( c ) c' = c' c 20. Intal probabltes are assgned to every category c from rght selecton by equaton (2), (3) and (4). 2. Intal probabltes whch depend only on the smlarty of neghborhood of canddate matchng ponts can be mproved usng consstency property. 22. The probablty updatng rule should have followng property : The new probablty k+ should tend to ncrease when descrptors wth hghly probable category consstent wth c are found nearby the keypont regon. 23. Categores are consdered consstent f they represent nearly the same dsparty.e. d ( c ) d( cm ) < Threshold: The threshold has to be decded emrcally by nspecton of stereo mage par. 24. For computng new probablty k+ for all c n category set c, lkelhood estmaton s done. The degree to whch the c j of c strengthen p (c) should be related to estmated lkelhood. 9

k j q = m= p( cm), m j (5) where d( c ) d( c ) < Th L m qj k (c) : Estmated lkelhood consderng the neghborhood of c j L : Number of neghbours n the category set. 25. Rule for updatng category probablty s p k+ j qj = (6) L k k p ( c' ) q ( c' ) c = k k where denomnator acts as normalzng factor j N k+ k+ = a j j (7) j= Here category probablty s updated teratvely. The values n the k th teraton are used to calculate values n k+ th teraton. a j are the weghts assocated wth contrbuton of dfferent neghbors of b. N s the number of neghborhood ponts of b. As shown n fgure there are four neghborhood ponts n the left selected area. Hence N = 4. a j can be constant for all the neghbors or vary. 26. For updatng probabltes of categores n the rght mage, equatons (5), (6), (7) are used teratvely. 27. After few teratons most possble matchng categores wll have very low probablty. The category wth hghest probablty s the most perfect match. 4. RESULTS AND DISCUSSION Interestng property of ths matchng algorthm s that t works for any range of dsparty between a par of stereo mages and does not requre nformaton regardng camera orentaton. Dsparty n the par of mages of the same scene or object s due to translaton or rotaton of the sensor. A photogrammetrc model could translate ths dsparty nformaton n to quanttatve measurement of depth, so that 3D reconstructon of the scene s possble. In the presented algorthm, the probablty of the vald match s mproved by consderng relatve postons of the neghbors usng the dstance between the neghbors. As the relatve dstance between the neghbors s used to mprove the probablty of the correct match, the accuracy to choose a correct match among the neghbors also mproves. Fgure 2 s a test stereo mage par. Fgure 3 shows the performance of the algorthm usng Eucldan dstance and Best Bn Frst approach to compare closest neghbor to that of second closest neghbor. All matches where dstance rato s greater than 0.7 are rejected. The matches found wth ths method are 244 for the gven test stereo mage par n fgure 2. Fgure 4 shows the performance of the matchng algorthm wth probablstc approach. For a specfc left keypont, the set of rght keyponts probabltes are evolved, through the teratons, usng consstency property and relaxaton labelng technque. The neghbourng keyponts contrbute n decdng the fnal selecton of keypont from the rght mage. Over sx teratons the probabltes of rght keyponts are evolved. The result of 2 nd, 4 th and 6 th teratons s shown n fgure 4. The canddate matchng ponts selected wth every teraton are supermposed on the mages. The equaton () n the proposed algorthm computes the weghts usng the Eucldan dstance between the left and rght keypont descrptor. The descrptors are computed by the bnary code for SIFT provded by Lowe and freely avalable on ste http://www.cs.ubc.ca/~lowe/keyponts/ If the correct dsparty range s known then the tme complexty of the algorthm mproves as the selecton of the number of keyponts for left and rght mage wll be mnmum wth the exact dsparty range. Although f the dsparty range s not known accurately, the algorthm works effcently gvng the correct and more number of vald matches as compared to the Eucldan dstance (BBF algorthm) method. Number of correct matches s counted consderng lmtng canddate probablty greater than 0.7. Comparson of number of matched ponts wth the method usng only Eucldan dstance crtera (BBF algorthm) and the method presented n the algorthm of secton 3 s shown n Table. The presented matchng algorthm s robust to 2D rotaton n mage plane, as rotaton and scale nvarant SIFT algorthm s used for selectng the keyponts. In case of rotaton, dfferent neghbourng keyponts are selected as the rght mage plane s rotated. But number of keyponts selected n the rght selecton area wll not vary much as the dsparty range does not change. In case of scalng of the rght mage, f rght mage s enlarged the dsparty range ncreases. Hence the sample sze around the keypont under consderaton ncreases. But t hardly affects the computng speed, as number of neghbourng keyponts found n the selected area are n same numbers. Fgure : b s the keypont n the selected area from the left mage and c j s the keypont n the selected area from the rght mage. b and c j together makes a category par 0

Image Test Image (shown n Fgure 2) Test mage 2 Method Algorthm consderng Eucldan dstance only Algorthm wth Probablstc approach Algorthm consderng Eucldan dstance only Algorthm wth Probablstc approach Left Image Selected keyponts Rght Image Matches found 378 376 244 378 376 286 855 239 843 855 239 020 Table. Comparson of matched ponts Fgure 2: A par of stereo mages wth keyponts supermposed on t Fgure 3: Result of matchng algorthm wth Eucldan dstance (BBF algorthm). The joned lnes show few matchng pont pars of left and rght mages. There are total 244 matches. ( (a) (b) (c) Fgure 4: Iteraton (a), (b), (c) shows matched ponts n the rght mage after 2 nd, 4 th and 6 th teratons.

5. CONCLUSION Dsparty map between mages s very useful nput for 3D reconstructon. Conventonally cross-correlaton approach s used to fnd mage correspondences n a stereo mage par, but t s prone to errors caused by dstorton n the magng process. 7. ACKNOWLEDGEMENT Sncere thanks to Dr. B. K. Mohan, IIT Bombay, Inda, for the gudance n the area, Relaxaton labelng technque. The probablstc approach explaned n the presented algorthm s usng consstency property to mprove the canddate probabltes usng relaxaton labelng technque. Instead of keypont by keypont matchng, the approach of fndng neghbourng keyponts and selectng the match for the keypont under consderaton n the selected area of left mage, usng neghbourng keyponts contrbuton from selected area of rght mage, mproves accuracy of the vald match. Expensve two dmensonal search over the entre mage s reduced by applyng nterest pont operator to both the mages, and t also greatly mproves n search space. The algorthm converges quckly wth few teratons and can be appled to mages havng wde dsparty range. It s robust over a large range of dsparty. The method s robust to 2D rotaton n mage plane and scalng. 6. REFERENCES B. Krshna Mohan, Neural Networks And Fuzzy Logc In Remote Sensng, n Landslde Dsaster Assessment and Montorng, R. Nagarajan (ed.) Anmol Publshers Pvt. Ltd., New Delh, 2004. C. P. Jeran and R. Jan, Structure from moton a crtcal analyss of methods, IEEE Trans. Systems, Man, and Cybernetcs, 2(3):572 588, 99. Lowe, D., Dstnctve mage features from scale-nvarant keyponts, Internatonal Journal of Computer Vson, Vol. 60(2), pp. 9-0, 2004. Lucas B. D. and Kanade T., An teratve mage regstraton technque wth an applcaton to stereo vson, Proc. 7th Int. Jont Conf. on Artfcal Intellgence, 674 679, 98. Marr D and Poggo T, Cooperatve computaton of stereo dsparty, Scence, vol. 94, pp. 283-287, Oct. 5, 976. Mkolajczyk, K. and Schmd, C., A performance evaluaton of local descrptors, Proc. of CVPR, 2003. Mkolajczyk, K. and Schmd, C., Scale and Affne Invarant Interest Pont Detectors, Int. Journal Computer Vson, Vol. 60(), pp. 63-86, 2004. Remondno F., Detectors and descrptors for photogrammetrc applcatons, Photogrammetrc and computer vson ISPRS symposum, Bonn, Germany, 2006. Rosenfeld A, Hummel R. A., and Zucker S. W., Scene labelng by relaxaton operatons, IEEE Trans. Syst., Man, Cybern., vol. SMC-6, June 976. Roth G., Whtehead A., Usng Projectve vson to fnd Camera Postons n an Image Sequence, Proc. of Vson Interface, pp.225-232, 2000. 2