MULTI-IMAGE MATCHING USING NEURAL NETWORKS AND PHOTOGRAMMETRIC CONDITIONS

Transcription

1 MULTI-IMAGE MATCHING USING NEURAL NETWORKS AND PHOTOGRAMMETRIC CONDITIONS Ahmed F. Elaksher Faculty of Engneerng, Caro Unversty, Egypt, Commsson III, WG III/1 KEYWORDS: Image matchng, Neural networks, coplanarty condton, collnearty condton, correlaton. ABSTRACT: Automatc determnaton of three dmensonal nformaton from dgtal mages s a fundamental problem n dgtal photogrammetry and computer vson. The hardest part of the problem s fndng conugate ponts n two mages. Despte the wealth of nformaton contaned n dgtal mages, factors such as occluson and dscontnuty weaken several matchng algorthms. However, mage matchng usng more than a par of stereo mages enhance the relablty of the mage matchng process. Ths paper presents an alternatve approach to match mage ponts across several vews. For each par of mages, the coplanarty condton and the correlaton coeffcent of mage ntenstes are computed for each par of mage ponts. These two measures are feed nto a feedforward neural network used to solve the mult-mage correspondent. The collnearty condton s then used to valdate the outputs of the neural network and to compute the 3D coordnates of the matched ponts. The detecton rate of the neural network s about 95% to 98% and the false alarm rate s about 7% to 4%. In addton, the collnearty condton elmnated several of the ncorrect matches and reduced the false alarm rate to less than 2%. The RMS errors of the ground coordnates are seven to eght centmetres. 1. INTRODUCTION Retrevng 3D spatal nformaton usng photogrammetrc models depends extremely on solvng the correspondence problem between mage features (Crotoru and Vncent, 2003). However, feature correspondence s not a trval task and several algorthms are presented n the lterature to solve ths problem (Lane and Thacker, 2007). Several factors affect ths task such as: lack of texture, feature topology, condton of surroundng features, and nose. Researchers have proposed a varety of technques to solve and mprove the performance of matchng technques. These technques are categorzed nto stereo matchng and mult vew matchng. In ether case, a varety of constrants, approaches, crtera are used to fnd conugate features dependng upon the propertes of the mage (Vncent and Laganère, 2001). Technques that uses only a par of stereo mages, usually suffer from mssng or hdden nformaton. Although there has been several attempts to solve ths problem (Mordoha and Medon, 2004; Ztnck and Kanada, 2000; Sun et al., 2005), mssng features contnue to challenge stereo matchng. On the other hand, mult vew matchng can overcome such problem. Ths paper presents an alternatve approach to solve the mult vew feature correspondence problem usng neural network and photogrammetrc condton. The approach take advantage of the fact that neural network can be used to nfer a functon from observatons (Haykn, 1994). Neural networks are beng used n several scentfc applcatons to solve a varety of problems n pattern recognton, predcton, optmsaton assocatve memory and control (Russell and Norvg, 2002). None of the conventonal approaches to these problems s flexble enough to perform well outsde ther doman. The mage matchng technque presented n ths paper conssts of three maor steps. In the frst step, feature vectors are computed for all mage ponts n all mage pars. The elements of the feature vectors are computed, for each mage pont, usng both mage coordnates and local ntenstes. These two quanttes are computed for all pars of mage ponts. The second step ncludes the use of a feed-back neural network to fnd conugate ponts. The algorthm assumes that all the mages are trangulated. In the last step, the collnearty condton s used to check the outputs of the neural network and compute the ground coordnates of the mage ponts. The remanng of the paper s organzed n the followng order. Frst, a background on recent research n mage matchng s summarzed. The proposed approach s then llustrated. Expermental results are then presented and dscussed. The last secton then states the research conclusons. 2. BACKGROUND Mult vew matchng has been addressed by several researchers n the photogrammetrc and computer vson communtes. The technque presented n Maas (1996) shows a mult mage matchng algorthm usng dscrete ponts extracted by an nterest operator. Image matchng s then carred out usng eppolar lne ntersecton. The technque presented n Brown et al. (2005) starts by locatng nterest ponts usng Harrs corner detecton (Harrs and Stephens, 1998). Matchng s then performed usng a fast nearest neghbour algorthm that ndexes features based on ther low frequency Haar wavelet coeffcents. Moreover, an outler reecton procedure s also ntroduced that verfes a parwse feature match based on a background dstrbuton of ncorrect feature matches. Feature matches are then refned usng the Random Sample Consensus RANSAC (Fschler and Bolles, 1981). 39

2 An adaptve mult mage matchng technque was presented n Paterak and Baltsavas (2002) usng the 3 panchromatc channels of the ADS40 dgtal camera. Edge pxel matchng s performed based on cross-correlaton and smlarty measures to provde pxel approxmate postons. These postons are subsequently refned usng sub-pxel matchng technques. The geometry of the sensor s used to apply matchng constrants va a modfed eppolar geometry desgned for the pushbroom sensor. In addton a modfed mage pyramd approach s used for dervaton of approxmatons. Another mult mage matchng method s presented n Gruen and L (2002) to generate DSM usng Three-Lne-Scanner (TLS) raw mages. The proposed method combnes matchng procedures based on grd pont matchng and feature pont matchng. The three mages are matched to provde pxel and obect coordnates for grd ponts smultaneously. An addtonal feature-pont matchng procedure s then performed to compensate for the dsadvantage of modellng the terran usng grd ponts. Ths was performed va a modfed Mult-photo Geometrcally Constraned (MPGC) matchng algorthm. 3. METHODOLOGY The mage matchng technque presented n ths paper depends on generatng all matchng hypotheses between each par of mages. For each hypothess, a feature vector s computed usng mage coordnates and ntenstes. A neural network s then used to fnd the approprate correspondent ponts n all mages. Once the mage pont matchng s solved the collnearty equaton s used to compute the 3D ground coordnates of the pont and to dscard any false matchng. Image ntenstes s also used to assst n removng false matches. Fgure 1 shows a flow chart of the proposed framework. The next three sectons provde a detaled explanaton of the algorthm. The process presented n D'Apuzzo (2002) s used to model human faces n a mult mage envronment. The mult mage matchng process s based on the geometrcally constraned least squares matchng algorthm presented n Gruen (1985). The process produces a dense set of correspondng ponts n the fve mages. Neghborhood flters are then appled on the matchng results to remove outlers. After flterng the data, the three dmensonal coordnates of the matched ponts are computed by forward ntersecton usng the camera nteror and exteror parameters. The researchers n Kang et al. (2001) provded two technques to assst n solvng the mult mage matchng problem. Frst, they mplemented a combnaton of movable wndows and a dynamcally selected subset of the neghborng mages to perform the matches. Secondly, they explctly labeled occluded pxels wthn a global energy mnmzaton framework, and reasoned about vsblty wthn ths framework so that only truly vsble pxels are matched. The researchers n Wang and Hsao (1999) used neural networks n stereo matchng. Two dfferent types of neural networks were used. The frst network utlzes ntensty, varaton, orentaton, and poston of each mage pxel to facltate self-development n network growng. The network then classfy the nput mage nto several clusters, and results are then used by a second network to acheve accurate matchng. The second network s used to generate an ntal dsparty map. Wth the clusterng results and the ntal map, a matchng algorthm that ncorporates a back propagaton network s then appled to recursvely refne the dsparty map. In the matchng process, useful constrants, such as eppolar, orderng, geometry and contnuty, are employed to reduce the occurrence of msmatchng. Researchers n Memony and Khanz (2001) and Mendonça (2002) proposed a method to establsh the relatonshp between 3D coordnates and the mage coordnates of a pont usng neural networks. A three layer neural network was used. The nput layer receve the mage coordnates n both mages. One hdden layer was used n both researches. The out layer presents the ground coordnates of each pont. The networks were traned usng several control ponts wth known mage and ground coordnates and then the network was used to compute the ground coordnates of a number of test ponts from there mage coordnates. The computed ground coordnates of the test ponts where compared wth those computed usng the tradtonal mage trangulaton technque and the dfferences n the coordnates were nsgnfcant. Fgure 1. The proposed framework for mult mage matchng 3.1 Generatng the parwse matchng hypotheses For each par of mage ponts, there obect space rays should ntersect provded that they represent the same ground target. Ths geometrc relaton s represented by the coplanarty equaton, equaton 1. The determnate of (F), as presented n equaton 1, should be zero provded that the second mage pont lay on the eppolar lne of the frst mage (Mkhal et al., 2001). Ths fact have been used n recent computer vson and photogrammetrc research to restrct the search space to a 1D space. In ths research, the determnate of (F) s computed for all pont pars n each mage par. The computed values are stored n an nxm matrx (F), where n s the number of ponts n the frst mage and m s the number of ponts n the second mage. For N mages,!n/2 matrces are generated. bx by bz F = u1 v1 w1 = 0 (1) u2 v2 w2 40

3 where b x, b y, b z = elements of the base vector representng the dsplacement between the perspectve centers of two cameras, u 1, v 1, w 1 = elements of the vector that representng the obect space vector from the mage ponts n the 1 st mage, u 2, v 2, w 2 = elements of the vector representng the obect space vector from the mage ponts n the 2 nd mage. The next step n ths research s to compute the correlaton coeffcents for all pont pars n each mage par. The correlaton coeffcent s a measure used to quantfy the smlarty of the ntenstes between two mage wndows. It has been used wdely to support parwse mage matchng and to solve matchng ambgutes (Helava, 1978). For each mage par, the values wll be stored n another nxm matrx,.e. the matrx of the correlaton coeffcents (C). 3.2 Computng the feature vector elements The data nput nto the neural network s presented as a feature vector. The elements of the feature vectors affect the performance of any neural network sgnfcantly. Hence, they have to be carefully selected. The elements of the two matrces,.e. the matrx of the correlaton coeffcents and the matrx of the coplanarty values are used to generate the feature vector as presented n equatons (2a and 2b). Two dfferent feature vectors (FV) are tested. In the frst feature vector (equaton 2a), both mage ntenstes and pont geometry,.e. the values of (F), are utlzed. The second feature vector (equaton 2b) uses only the pont geometry. 3.3 Implementng the neural network The mult mage matchng problem s defned as fndng the correspondng ponts n all mages. Ths problem could be consdered as mnmzng an L2 functon of the values of the F and maxmzng an L2 functon of the correlaton values. The mathematcal soluton of such system s hard to mplement. However, Hornk et al. (1989) showed that for any gven ε>0 and any L2 functon, there exsts a three-layer back-propagaton neural network that can approxmate the functon wthn ε mean squared error accuracy. Thus neural networks provde exctng soluton for the mult mage matchng problem. The neural network mplemented n ths research s a feedforward back-propagaton network. The network conssts of three layers; an nput layer, one hdden layer, and an output layer. The number of neurons n the frst layer s the same as the number of elements n the feature vectors. For the frst case,.e. usng the mage ntenstes and the pont geometry, the number of neurons n the nput layer s sx, whle for the second case the number of neurons s three fgure 2. Several experments were conducted to determne the optmum number of neurons n the second layer. Results showed no sgnfcant changes n the output of the networks. Hence, the number of neurons n ths layer was selected to be ten. The number of neurons n the thrd,.e. last, layer s constraned to one. The output of ths neuron s ether one n case the ponts n all mage match or zero n case the ponts do not match. The actvaton functons for all neurons n the frst and second layers, s the sgmod functons (Haykn, 1994). For the output neuron, the step functon s chosen as the actvaton functon. The results of the neural network are evaluated usng the Mean Square Error computed usng equaton 3. FV k =[ k k k k C ] t (2a) 12 F12 C13 F13 C23 F23 FV k k k =[ F 12 F 13 F 23 ] t (2b) 2 ( T O) MSE = n (3) where C 12, C 13 k, C 23 k = elements of the correlaton matrces for ponts,, and k, n mages 1, 2, and 3 respectvely, F 12, F 13 k, F 23 k = elements of the coplanarty matrces for ponts,, and k, n mages 1, 2, and 3 respectvely, dots n both formulas represent the ablty for more mages. where T = one for correct matches or zero for false matches, O = output value from the neural network, n = number of ponts used to evaluate the network. Several remarks are observed n the selected feature vector. The values of the coplanarty relaton between each par of mages are dependent. However, neural networks overcome ths problem and doesn t run nto sngularty. Moreover, for cases where one of the values s zero the others wll only be zero f and only f the ponts represent the same ground obect. In addton, snce automatc pont locatng s not perfect, the quanttes wll be mnmum, for matched ponts, and not equal to zero. The correlaton coeffcent values are effected by several factors such as the sze of the local wndow, the orentaton of the mages, and the topology of the obect. For ths research the local wndow sze s fxed to seven by seven pxels. On the other hand, the correlaton coeffcents could reach ther maxmum values even f the mage ponts don t correspond to the same obect pont. Fgure 2. The mplemented neural network (2 nd case) 3.4 Computng ground coordnates and valdatng the neural network outcomes The collnearty equaton, equaton 4, s used to compute the ground coordnates of any pont gven t s mage coordnates n at least two mages. For N mages, 2N equatons are wrtten per 41

4 pont, however, only three equatons are requred to compute the ground coordnates of the pont. Ths provdes an over determnate system of equatons. Hence, the least squares adustment technque s used to compute the 3D coordnates. the results. The dstrbuton of the ponts n one mage s shown n fgure 3. U F = x x 1 o + f W V F = y y f 2 o + W (5) Where x o, y o, f = camera nteror parameters, x, y = coordnates of vertex () n mage (), U X X c V = R Y Y, c W Z Z c R = rotaton matrx for mage (), X c, Y c, and Z c = exposure staton coordnates for mage (), X, Y, and Z = obect space coordnates of pont (). Two measures are used to valdate the matchng outcomes. The frst measure s the quadratc form of the resduals,.e. (v t v) of the mage coordnate. In addton, the sum of the par-wse correlaton coeffcents s also used. If the quadratc value s larger than a gven threshold and the sum of the correlaton coeffcents s small than another threshold, the canddate matchng s removed. These two measures provde a tool for reectng false matches that result from the neural network. 4. EXPERIMENTS AND RESULTS 4.1 Dataset descrpton The dataset used n ths research s for the cty hall of Zurch buldng. The data s avalable on the web page of the Internatonal Socety of Photogrammetry and Remote Sensng (ISPRS). The dataset s provded wth a complete descrpton of the nteror orentaton of the used cameras, the mages, and the coordnates and the descrpton of the reference ponts measured on the facades of the buldng by geodetc means (Strelen et al., 1999). The mages acquston was performed usng an Olympus C1400L camera of about ten mllmeters focal length and a progressve scan CCD of 1280x1024 pxel resoluton. Nne ground control ponts are used to compute the exteror orentaton parameters. The Root Mean Square Errors (RMSE) of the check ponts n the three drectons s about seven to eght centmeters. In order to test and evaluate the proposed technque, four mages are used. Harrs corner detector was appled to all four mages. The mage coordnates of the ponts are stored n an nput fle. In addton, the ground coordnates are computed and stored n another fle to evaluate 4.2 Data processng Fgure 3. Ponts detected n one mage The next step s to compute the parwse matrces usng both the coplanarty condton and the correlaton coeffcent values for each par of ponts for each mage par. For the four mages, used n ths research, sx coplanarty matrces and sx correlaton coeffcent value matrces are generated. Fgure 4 shows the coplanarty matrx and the correlaton coeffcent matrx between the frst and second mages. The next step s to generate the feature vector for each par of mage ponts n each par of mages. Tranng samples are then selected randomly from all avalable samples. Several experments were conductng usng a tranng sample sze of 50 to 250 samples. The tranng dataset samples contan 10% samples of correctly matched ponts and 90% samples of false matchng ponts. Two cases are tested,.e. usng both the correlaton and coplanarty values and usng only the coplanarty values. Fgures 5 and 6 show the results for both cases. The mage coordnates of the correctly matched ponts are then feed to the least squares adustment model. For each pont eght equatons are wrtten. Ths provdes a redundancy degree of fve. The values of the quadratc form (v t v) and the sum of the par-wse correlaton coeffcents are computed and compared aganst two selected thresholds. The threshold for the quadratc form s selected to be 10e -3 and the threshold of the sum of the correlaton coeffcents s selected to 0.5. The fnal results showed that out of the 134 ponts only one ponts were mssed. In addton, only two ncorrect matchng ponts passed the two thresholds and provded two fnal mss match ponts. The fnal set of matched ponts are shown n fgure CONCLUSIONS Ths paper presents an alternatve approach to solve the mult mage matchng problem. Usng any number of mages, the process starts by computng the coplanarty condton between all pars of mages n each mage par. In addton, for each pont par the correlaton coeffcent s computed usng local mage ntenstes. The two measures are then used to dscrmnate between correct and ncorrect mult mage matches usng a feed-forward neural network. Results of the neural network showed a detecton rate of about 95% to 98% and a 42

5 false alarm rate of about 4% to 7%. The collnearty condton was then used to compute 3D ground coordnates of all matched canddates. The resduals of the mage coordnate and the sum of the parwse correlaton coeffcents are then used to elmnate false matches. Ths process reduces the false alarm rate to less than 2%. Fgure 7. 3D coordnates (meters) for correctly matches (blue) and false (red) matches Fgure 4. The coplanarty and the correlaton coeffcent matrces between the 1 st and 2 nd mages % Case 1 Case Tranng Date Sze (Samples) Fgure 5. Percentage of correctly matched ponts usng both cases % 7 Case Case Tranng Date Sze (Samples) Fgure 6. Percentage of false matched ponts usng both cases REFERENCES Brown, M., Szelsk, R., Wnder, S., Mult-mage matchng usng mult-scale orented patches. Proceedngs of the Internatonal Conference on Computer Vson and Pattern Recognton, San Dego, Ca, USA, Vol. 2, pp Crotoru, A. and Vncent, T., An alternatve approach to the pont correspondence problem. Proceedngs of the ASPRS 2003 Annual Conference, Anchorage, Alaska, USA, CD-ROM. D'Apuzzo, N., Modellng human faces wth mult-mage photogrammetry. Proceedngs of SPIE, San Jose, Calforna, Vol. 4661, pp Fschler, M.A. and Bolles, R.C., Random sample consensus: a paradgm for model fttng wth applcatons to mage analyss and automated cartography. Communcatons of the ACM, Vol. 24(6), pp Gruen, A., Adaptve least squares correlaton: a powerful mage matchng technque. South Afrcan Journal of Photogrammetry, Remote Sensng and Cartography, Vol. 14(3), pp Gruen, A. and L, Z., Automatc DTM generaton from three-lne-scanner (TLS) mages. Proceedngs of the ISPRS Commsson III Symposum, Vol. XXXIV, part 3A, pp , Graz, Austra. Harrs, C. and Stephens, M., A combned corner and edge th detector. Proceedngs of the 4 Alvey Vson Conference, Manchester, England, pp Haykn, S., Neural networks: a comprehensve nd foundaton. 2 edton, Prentce Hall, Upper Saddle Rver, New Jersey, pp Helava, U.V., Dgtal correlaton n photogrammetrc nstruments, Photogrammetra, Vol. 34, pp Hornk, K., Stnchcombe, M., and Whte, H., Multlayer feed forward networks are unversal approxmators, Neural Networks, 2(5), pp Kang, S.B., Szelsk, R., and Cha, J., Handlng occlusons n dense mult-vew stereo. Proceedngs of the IEE Conference on Computer Vson and Pattern Recognton, Vancouver, Canada, Vol. 1, pp

6 Lane, R.A. and Thacker, N.A., Overvew of stereo matchng research. (accessed 21 Oct. 2007) Maas, H.-G., Automatc DEM generaton by mult-mage feature based matchng. The Internatonal Archves of Photogrammetry, Remote Sensng, and Spatal Informaton Scences, Vol. 31, Part B3, pp , Venna, Austra. Memony, Q. and Khanz, S., Camera calbraton and three-dmensonal world reconstructon of stereo-vson usng neural networks. Internatonal Journal of Systems Scence, Vol. 32(9), pp Mendonça, M., Camera calbraton usng neural networks. Proceedngs of the 10 th Internatonal Conference n Central Europe on Computer Graphcs, Vsualzaton and Computer Vson, Campus Bory, Plzen - Bory, Czech Republc. Mkhal, E., Bethel, J., and McGlone, J., Introducton to modern photogrammetry, Jon Wley & Sons. Inc., New York, pp.. Mordoha, P. and Medon, G., Stereo usng monocular cues wthn the tensor votng framework. Proceedngs of the 8 th European Conference on Computer Vson, Vol. 3024, pp , Prague, Czech Republc. Paterak, M. and Baltsavas, E., Adaptve mult-mage matchng algorthm for the arborne dgtal sensor ADS40. Proceedngs of Map Asa 2002, Bangkok, Thaland, (on CD- ROM). Russell, S. and Norvg, P., Artfcal ntellgence a modern approach. 2 nd edton, Prentce-Hall, Upper Saddle Rver, New Jersey, pp Strelen, A., Grussenmeyer, P., and Hanke, K., Zurch cty hall: a reference data set for dgtal close-range photogrammetry. Proceedngs of the CIPA Internatonal Symposum, Recfe/Olnda-PE, Brazl. Sun, J., L, Y., Kang, S., and Shum, H., Symmetrc stereo matchng for occluson handlng. Proceedngs of the Internatonal Conference on Computer Vson and Pattern Recognton, San Dego, CA, Vol. 2, pp Vncent, E. and Laganère, R., Matchng feature ponts n stereo pars: a comparatve study of some matchng strateges. Machne Graphcs and Vson, Vol. 10(3), pp Wang, J. and Hsao, C., On dsparty matchng n stereo vson va a neural network framework. Proceedngs of Natonal Scence Councl, Vol. 23(5), pp Ztnck, C.L. and Kanada, T., A cooperatve algorthm for stereo matchng and occluson detecton. Proceedngs of the IEEE Transactons on Pattern Analyss and Machne Intellgence, Vol. 22(7), pp