INVARIANT DESCRIPTOR LEARNING USING A SIAMESE CONVOLUTIONAL NEURAL NETWORK

Transcription

1 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc INVARIANT DESCRIPTOR LEARNING USING A SIAMESE CONVOLUTIONAL NEURAL NETWORK L. Chen *, F. Rottenstene, C. Hepke Insttute of Photogammety and GeoInfomaton, Lebnz Unvestät Hannove, Gemany - (chen, ottenstene, hepke)@p.un-hannove.de Commsson III, WG III/1 KEY WORDS: Descpto Leanng, CNN, Samese Achtectue, Nesteov's Gadent Descent, Patch Compason ABSTRACT: In ths pape we descbe eanng of a descpto based on the Samese Convoutona Neua Netwok (CNN) achtectue and evauate ou esuts on a standad patch compason dataset. The descpto eanng achtectue s composed of an nput modue, a Samese CNN descpto modue and a cost computaton modue that s based on the L Nom. The cost functon we use pus the descptos of matchng patches cose to each othe n featue space whe pushng the descptos fo non-matchng pas away fom each othe. Compaed to eated wok, we optmze the tanng paametes by combnng a movng aveage stategy fo gadents and Nesteov's Acceeated Gadent. Expements show that ou eaned descpto eaches a good pefomance and acheves stateof-at esuts n tems of the fase postve ate at a 95% eca ate on standad benchmak datasets. 1. INTRODUCTION Featue based matchng fo fndng pas of homoogous ponts n two dffeent mages s a fundamenta pobem n compute vson and photogammety, equed fo dffeent tasks such as automatc eatve oentaton, mage mosackng and mage eteva. In genea, fo a featue based matchng agothm one needs to defne a featue detecto, a featue descpto and a matchng stategy. Each of these thee modues s eatvey ndependent of the othes, theefoe a combnaton of dffeent detectos, descptos and matchng stateges s aways possbe and a good combnaton mght adapt to some specfc data confguatons o appcatons. The key pobem of mage matchng s to acheve nvaance aganst possbe photometc and geometc tansfomatons between mages. The st of photometc tansfomatons that affects the matchng pefomance compses umnaton change, dffeent efectons and the use of dffeent specta bands n the two mages. Geometc tansfomatons compse tansaton, otaton and scang as we as affne and pespectve tansfomaton; besdes, the matchng pefomance may aso be affected by occuson caused by a vewpont change. In most cases, featues fo matchng ae extacted ocay n the mage, and a featue vecto (descpto) used to epesent the oca mage stuctue s geneated fom a eatvey sma oca mage patch cented at each featue. Consequenty, t s usuay suffcent to desgn a matchng stategy that s nvaant to affne dstoton, because a goba pespectve tansfomaton can be appoxmated we by an affne tansfomaton ocay. Such dstotons ae key to occu n case of age changes of the vew ponts and the vewng dectons. Cassca descptos, ke SIFT (Lowe, 004) and SURF (Bay et a., 008) ae desgned manuay; they ae nvaant to shft, scae and otaton, but not to affne dstotons. Some authos (Mkoajczyk and Schmd, 005; Moees and Peona, 007; Aanæs et a., 01) have evauated the pefomance of detectos and descptos aganst dffeent types of tansfomatons n pana and 3D scenes, usng eca and matchng pecson as the man evauaton ctea (Mkoajczyk and Schmd, 005). As dscussed n (Moees and Peona, 007), the esuts show that the pefomance of cassca detectos and descptos dops shapy when the vewpont change becomes age, because the oca patches vay seveey n appeaance, so that the toeance of cassca featue detectos and descptos s exceeded. One stategy to mpove the nvaance of descptos to vew pont changes s to convet the descpto desgn and descpto matchng nto a patten cassfcaton pobem. By coectng the patches of the same featue n dffeent mages, one can captue the ea dffeences between these patches. The pocess of desgnng nvaant featue descptos s equa to fndng a mappng of those patches nto a pope featue space whee they ae ocated moe cosey to the descptos of the homoogous featues. By usng an appopate machne eanng mode, a oss based on the smaty of the eaned descptos s desgned. In ths case, deceasng the oss by eanng heps to acheve a hghe eve of nvaance. In ths pape, we pesent a new method fo defnng descptos based on machne eanng. It extends ou pevous descpto eanng wok on Convoutona Neua Netwoks (CNN; Chen et a., 015). As a CNN has a natua "deep" achtectue, we expect ths achtectue to have a stonge modeng abty whch can be used to poduce nvaance aganst moe chaengng tansfomatons, whch cassca manuay desgned descptos cannot cope wth. By conductng the tanng n a mn-batch manne, usng a movng aveage stategy on gadents and a momentum tem as we as Nesteov's Acceeated Gadent, we optmze the tanng paametes and acheve ou taned descpto. The man contbuton of ths pape s that we fst ntoduce ths tanng agothm nto descpto eanng tasks based on Samese CNN. * Coespondng autho Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

2 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc. RELATED WORK A substanta body of cassca descptos ae desgned n a manua manne, fo nstance SIFT (Lowe, 004) o SURF (Bay et a., 008). Moe ecent manuay desgned methods ke DAISY (Toa et a., 010) ntoduced a moe compex patten of poong opeatons. These descptos have been consdeed to be a standad fo qute some tme. Howeve, they cannot dea wth age vewpont changes. Ths s why affne-nvaant famewoks fo featue based matchng have been poposed, e.g. ASIFT (Moe and Yu, 009). By usng an affne vew-sphee smuaton stategy, ASIFT tansfoms the two ogna mages to many affne vesons, then featues and descptos ae computed based on those mages. Aftewads the descptos fom affne dstoted vesons of the two ogna mages ae matched. As each featue has many dffeent descptos that ae but on smuated affne vews, ASIFT can cope wth affne dstotons bette than othe matchng agothms that ony un on ogna mages. Howeve, ASIFT s computatonay expensve and benefts fom the vew-sphee smuaton matchng scheme athe than fom any mpovements on vewpont nvaance of the featue descpto. An atenatve to usng hand-cafted featues and stateges such as sampng many potenta vewponts synthetcay s descpto eanng (Bengo et a., 013). To test f machne eanng appoaches can acheve bette esuts, Bown et a. (011) poposed a descpto eanng famewok, n whch a descpto s composed of fou dffeent modues: 1) Gaussan smoothng; ) non-nea tansfomaton; 3) spata poong o embeddng; 4) nomazaton. New descptos can be deved by optmzng the confguaton of the second and the thd modues. An extenson of the wok whch aows convex optmzaton n the tanng pocess s gven n (Smonyan et a., 01; 014). In (Tzcnsk et a., 01; 015), a descpto eanng achtectue based on the combnaton of weak eanes by boostng s desgned, n whch the weak eanes ey on compasons of smpe featues. In the tanng pocess, the optma featues fo the weak eanes ae detemned aong wth the optma matchng scoe functon. The esutng descpto outpefoms SIFT unde neay evey type of tansfomaton on the benchmak data set of Mkoajczyk and Schmd (005). Anothe categoy of descpto eanng famewoks s but on CNN. CNN conssts of mutpe convoutona ayes (LeCun et a. 1998). Invaant featue epesentaton eanng based on a so-caed Samese CNN has ognay been poposed n (Bomey et a., 1993) to extact featue epesentatons fo sgnatue vefcaton, whee the sgnatues fom one peson may change n compex ways, whch ae neay mpossbe to captue wth expct modes. The tem Samese efes to the fact that the same CNN achtectue and the same paametes ae apped to two nput data sets wth compex eatve dstotons. In (Hadse et a., 006), the Samese CNN achtectue was used to ean featue epesentatons fo dgt ecognton; as the same dgt wtten by dffeent peope vaes consdeaby, a Samese CNN achtectue s used to fnd an nvaant featue epesentaton that can map the hgh dmensona nput data nto a moe dscmnatve featue space whee "sma" dgts ae ocated moe cosey to each othe. Ths featue space s defned by the output of the fna convonutona aye of the CNN. The use of mutpe ayes (.e., the deep achtectue) s the eason fo the stong modeng abty of CNNs. Ths popety fts we wth the equements fo eanng descptos that ae nvaant aganst vaous types of tansfomatons. Consequenty, CNN have been used to tan descptos fo patch compason. The fst patch compason wok based on the Samese CNN was pesented n (Jahe et a., 008). Jahe et a. (008) used the Samese CNN to tan the descpto and compae the patches, but the tanng data was geneated fom mage waps and dependent on nput mages, whch makes ths method ess pactca, because t aways needs a po smuaton and tanng befoe mage matchng. In (Osendofe et a., 013), a Samese CNN s used to tan a descpto; the pape focuses on the compason of fou dffeent types of oss functons. Moe ecenty, the Samese achtectue was used to tan patch descptos to cope wth dynamc ghtng condtons (Caevas-Banco and Eustce, 014), feedng patches wth sevee umnaton change nto a Samese CNN; umnaton nvaance that exceeds any hand-cafted descptos s acheved. In (Ln et a., 015), mages taken fom aea and teesta vews ae fed nto a Samese CNN netwok, foowed by appyng a smaty functon that ndcates whethe the two mages contan dentca scenes. Usng ths mode, aea and teesta vew ae nked, whch can be used to geneate a eaton gaph. Howeve, the descpto s apped to the whoe mage, not to patches cented aound featue ponts, theefoe t can ony bud ough connectons on the eve of compete mages, and t cannot fnd pecse pont coespondence. Ou wok s cosey nked to the wok n (Han et a., 015; Zagouyko and Komodaks, 015; Zbonta and Lecun, 015). Han et a. (015) and Zagouyko and Komodaks (015) dd not ony tan the descpto, but aso a cassfe to detemne the matchng abe, whch s caed the metc netwok (Han et a., 015) and decson aye (Zagouyko and Komodaks, 015). Ths makes the mode moe compcated than ous. Zbonta and LeCun (015) aso cacuate fou exta ayes of the metc netwok, but appy them to wde basene dense steeo matchng athe than to featue based matchng fo oentaton. They cuenty acheve the best esut on the KITTI benchmak. If one taned a metc functon fo pas of patches, then evey pa of featue patches shoud be fed nto the netwok wth metc ayes when ths descpto s apped n ea mage matchng o age scae mage eteva. In ths case, the hghy effcent seach stateges such as Best Bn Fst (Bes and Lowe, 1997) n a KD tee cannot be used and the matchng speed s seousy nfuenced. Ths educes the pactca vaue of a eaned descpto n featue based mage matchng. In contast to those woks, we theefoe tan a descpto wthout a metc functon fo the two patches. 3. METHODOLOGY In ths secton the Samese descpto eanng achtectue s descbed fst. Then, detas of the CNN used n ths achtectue ae pesented. Fnay, we descbe the method used to ean the paametes of the CNN. 3.1 Samese Descpto Leanng: Achtectue In ode to ean the CNN-based descpto, we need pas of tanng patches of whch we know whethe they epesent homoogous mage featues o not. In ths context, t s mpotant that the set of postve exampes (the pas that coespond to homoogous key ponts) contans tansfomatons that the eaned descpto shoud be toeant to. The basc dea of the Samese achtectue fo descpto eanng s to appy the same type of CNN usng the same set of paametes to each of the patches that shoud be checked fo coespondence Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

3 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc and detemne these paametes by optmsng a oss functon of the L nom of the dffeences of the esutant descptos. That s, by adjustng the paametes so that the L nom s as dscmnatve as possbe n sepaatng coect fom ncoect matches we obtan a descpto that shoud be toeant to the type of geometc dstotons that occu between postve exampes n the tanng data; efe to Fgue 1 fo an ustaton of the whoe achtectue. In the foowng secton, the paametes ae expaned n deta. pung the descptos of matchng pas cose to each othe. An ustaton of ths dea s shown n Fgue. Befoe eanng, the descptos ae dstbuted andomy n featue space, whe afte eanng the descptos fom patches coespondng to homoogous ponts e cose to each othe. Left Patch Rght Patch Befoe Leanng Afte Leanng CNN θ CNN push D D pu Fgue 1. The achtectue fo Samese CNN descpto eanng used n ths pape. Geen: nput patches; Red: a CNN as depcted n fgue 3; D, D : descptos fo the ght and the eft mage patch, espectvey. Bue: oss functon. The two CNNs shae the eaned paametes (oange). In the tanng pocess, the foowng oss functon based on the L nom of the dffeences of the CNN descptos of tanng patch pas s mnmsed: N L y max 0, D D pu 1 1 y max 0, D D s push whee L Nom Dstance Loss L (1) N = numbe of tanng sampes = ndex of a tanng sampe y = abe fo a patch pa: 1 fo matchng tanng pas, 0 fo unmatched pas. D k = CNN descptos fo patch k, wth k {, } ndcatng the eft o ght patch, espectvey D - D = L nom of the dffeences between the descptos of the two patches pu = Pu adus fo sma pas push = Push adus fo dssma pas = squaed L nom of the paametes s = weght of the eguasaton tem In Eq. 1, the ast tem coesponds to a eguasaton wth weght s, equed to decease the sk of ove-fttng. The oss functon ceates a magn between matchng and non-matchng pas. Fo matchng pas, a dstance age than a pu adus pu s penased, wheeas fo non-matchng pas (the negatve tanng exampes), penasaton occus fo dstances smae than a push adus push. Ths type of oss functon has been shown to be sutabe fo descpto eanng by Osendofe et a. (013). The two ad ae paametes that have to be set by the use. The CNN paametes ae ntased at andom, so that ntay the dstances of descptos fom matchng pas cannot be expected to be sma. The eanng pocedue then tes to fnd paametes of the CNN that push the descptos of unmatched pas away fom each othe n featue space, whe Featue Space Fgue. Descpto eanng. In the top pat, each cooued dot epesents a descpto; dentca coous ndcate homoogous patches fom mut-vew mages. In the owe pat, the adus of the nne concentc cce s pu and the adus of the oute one s push. 3. CNN Descpto Featue Space The concept of CNNs was poposed by (LeCun et a. 1998); t s a mut aye neua netwok. A CNN may have one o sevea stages consstng of a convouton aye, a nonnea aye and a featue poong aye each. Compaed to genea mut aye neua netwoks, thee ae two man dffeences: 1) In the convouton aye, the neuons of the nput aye ae not fuy connected to those of the next aye and weghts ae shaed, so that the same weghts ae epeatedy used acoss the dffeent poston of the nput aye. Ths s the eason fo usng the tem "convoutona" netwok. The weght shang stategy damatcay deceases the numbe of paametes and makes deep achtectues consstng of age numbes of stages tanabe. ) The netwok deceases the aye sze n successve stages by poong ayes. Theefoe, the nput can be compessed nto a meanngfu featue epesentaton, whch educes the dmenson of the ogna data consdeaby. In essence, a CNN can be seen as a nonnea mappng functon, tansfomng the nput (a gven mage patch) to a hghe-eve but owe dmensona featue epesentaton. In ths pape, we use a CNN achtectue consstng of thee stages to ean featue descptos (cf. fgue 3). Detas about the achtectue and the eanng paametes ae sted n tabe 1. The nput patch sze s 3 by 3 pxes. The CNN contans thee stages. The fst two stages have a [convouton - nonnea - poong] stuctue, wheeas the thd one ony contans a convouton aye. Fo each stage k wth k {1,, 3}, the paametes to be detemned ae the convouton kene w k and the bas tem b k, whch, thus, consttute the paametes shaed by the two CNNs n the Samese achtectue. Fo bevty, t s aso wtten as paametes w k and b k n the eman text. Wheeas n the fst convouton aye we tan fve D kenes of sze 5 x 5 to poduce fve featue maps, n the subsequent stages we detemne the paametes of 3D kenes (5 5 x 5 x 5 kenes n stage ; 15 5 x 5 x 5 kenes n stage 3). The nonneaty s Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

4 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc Input Convouton Nonnea Poong Output Leanng paametes kenes Stage 1 3 x 3 5 x 5 x 5 sgmod max ( x ) 14 x 14 x 5 w 1, b 1 Stage 14 x 14 x5 5 x 5 x 5 x 5 sgmod max ( x ) 5 x 5 x 5 w, b Stage 3 5 x 5 x 5 15 x 5 x 5 x 5 ~ ~ 1 x 1 x 15 w 3, b 3 Tabe 1. Detaed achtectue and eanng paametes fo the CNN used n ths pape. The numbes ndcate pxe numbes. Fgue 3. The CNN used n ths pape to ean the descpto based on the sgmod functon and we use max poong (wthout oveap,.e. stde = ), pesevng the agest vaue n a x neghbouhood as the output. The fna output of ou CNN s a 15 dmensona vecto. Ths 15 dmensona vecto s the eaned descpto that s used to epesent the content of a oca mage patch suoundng a featue. The CNN achtectue used n ths pape s dffeent fom (Han et a., 015; Zagouyko and Komodaks, 015). Fst, a smae nput wndow wth ony 3 x 3 pxes (nstead of 64 x 64 pxes, whch wee used n the epoted wok), s empoyed. When pocessng wde basenes mages, the appeaance of patches suoundng featue ponts changes moe seveey than n naow basene stuatons. By usng of smae context wndow, the poposed descpto can potentay cope wth age defomatons n a bette way. Addtonay, the sgmod functon s apped to acheve nonneaty because we found t to pefom bette than the Rectfed Lnea Unt (ReLU). Fnay, compaed to the eated wok, we use a moe advanced tanng agothm (see secton 3.3). 3.3 Tanng of the Samese CNN Tanng of the CNN s based on gadent descent to fnd the optmum of the oss functon. In ths context, the we-known back popagaton agothm (Rumehat et a., 1986) can be used to detemne devatves of the oss wth espect to the paametes. In ou netwok, back-popagaton s a tte moe compcated than usua, because the gadents ae nfuenced by both subnets n the Samese CNN. In Secton the onne gadent tanng pocedue s descbed, wheeas Secton 3.3. contans detas about the way n whch gadents ae computed Mn-batch Gadent Descent: In genea, afte cacuatng the gadent of the oss functon wth espect to the paametes to be eaned, the paametes ae updated accodng to the gadent, takng nto account a eanng ate. In the teatue one can fnd methods usng a tanng sampes to compute the gadents (batch tanng) and onne methods, usng ony one tanng sampe at a tme (Bshop., 006). The fst vaant can be vey sow n the pesence of many tanng sampes. On the othe hand, onne gadent descent can be unstabe because of sampng eos when computng the gadent ony fom one sampe. As a compomse we use mnbatch gadent descent, updatng the paametes on the bass of gadents computed fom eatvey sma goups of tanng sampes n each teaton. Each goup (mn-batch) typcay contans hundeds o sevea thousands of tanng sampes. The gadents used to update the paametes ae aveage gadents ove a sampes n the goup cuenty consdeed. One way of gadent descent s to consde a fxed eanng ate and update the paametes accodng to t+1 = t - g'( t ), whee g'( t ) s the gadent of oss functon wth espect to paametes and the suffx t ndcates the teaton step. Howeve, the seecton of the eanng ate s pobematc: a sma eanng ate eads to a athe sow decease of ou oss functon, wheeas a age vaue eads to oscatons. Ths can be consdeed by statng the teaton wth a eatvey age eanng ate 0 and deceasng the eanng ate n each teaton accodng to t+1 = t decease wth 0 < decease < 1. Howeve, ths has been found not to sove the pobem competey. A bette way of copng wth ths pobem s gven by the momentum method, whch updates the paametes accodng to t+1 = t - v t+1, whee the veocty v t+1 s based on the accumuated gadents of the pevous steps: v t+1 = β v t + t g'( t ) () whee the gadent s cacuated at the cuent poston g'( t ) and β wth 0 < < 1 s the momentum tem. At the begnnng of the teaton pocess, the veocty s assumed to be zeo (v 0 = 0). The top pat of Fgue 4 ustates the update ue of the standad momentum gadent descent. The bue vecto epesents the decton to adjust the paametes. Assumng that the accumuated veocty β v t w esut n a move that educes the vaue of the functon to be optmzed, t woud seem to be a bette choce to detemne the gadents afte appyng the accumuated veocty. That s, one detemnes new paamete vaues by t+1/ t - β v t, and then uses the gadent at poston t+1/, g'( t+1/ ) athe than g'( t ) fo the fna update. Ths s Nesteov's Acceeated Gadent (NAG, Nesteov, 1983) method, whee the veocty v t+1 s detemned accodng to: Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

5 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc v t+1 = β v t + t g' t - β v t ) (3) Ths update ue s ndcated by the owe pat of Fgue 4. The NAG method has been shown to be sutabe fo detemnng the paametes of deep neua netwoks n (Sutskeve et a., 013). g'( t) t t Fgue 4. (Top) Momentum method and (Bottom) Nesteov's Acceeated Gadent (NAG) (Sutskeve et a., 013). An atenatve to avod oscatng behavou of gadent descent s gven by the mspop method (Hnton et a., 016), n whch the gadent s nomased by the aveage gadent magntude. Ths eads to t vt 1 g( t) (4) t Whee t s the aveage squae gadent accumuated n the pevous teatons: t = (1- γ) g'( t ) γ t-1. (5) In equaton 5, γ wth 0 < γ < 1 s a weght that moduates the mpact of the accumuated magntude squaes eatve to the new one. Sma to (BRML, 013), we combne the mspop method wth the NAG momentum method n ode to acheve an mpoved convegence behavou. The tanng data ae andomy dvded nto a tanng and a coss vadaton set. The weghts ae ntased by andom vaues; both 0 and v 0 ae set to 0, and the eanng ate s set to an nta vaue 0. Tanng s caed out n epochs. In each epoch, the tanng data ae andomy dvded nto M nonoveappng subsets (the mn-batches), and each mn-batch s used to update the paametes once pe epoch. In each epoch m, the eanng ate m emans unchanged; that s, we use t = m. As soon as epoch m s fnshed, the eanng ate s updated accodng to m+1 = m decease, and a new andom dvson of the tanng data nto mn-batches s caed out, whch seves as the bass fo the next epoch. In each epoch, the paametes ae updated M tmes usng the foowng steps: 1) Fo the cuent poston t, appy the momentum by t+1/ = t - β v t and cacuate the gadent g'( t+1/ ). ) Compute t and v t+1 accodng to: t = (1- γ) g'( t+1/ ) γ t-1 t v v g( ) (6) t 1 t t 1 / t βv t 3) Update the cuent paametes accodng to t+1 = t - v t+1. v t+1 βv t v t+1 t+ 1 g'( t+ βv t) t+ 1 Note that the teaton counte t s ncemented afte pocessng each mn-batch, but t s not eset to 0 when a new epoch stats. The eanng agothm n ths pape s dffeent fom standad gadent decent because t stats wth a guess by movng the cuent paamete to a new poston t+1/ wth the accumuated gadents and momentum, foowed by a coecton (gadent cacuaton) at t+1/ and an update accodng to that gadent. We aso evauate the oss on the vadaton set afte each tanng epoch. If the oss does not decease fo thee subsequent epochs, we stop the tanng pocess and ecod the paametes n the cuent epoch as optmzed paametes. A pefomance compason of the method n ou pape and othe tanng methods s pesent n secton Gadent computaton: The oss functon s cacuated based on the dstances of the descptos, as descbed by equaton 1. The devatve of the oss wth espect to the dstance d = D - D s cacuated by: L y ( d ) ( d ) pu pu d (1 y) ( d ) ( d ) push push whee (.) s an ndcato functon; t equas to 1 f the agument s tue and 0 othewse. The devatves of the dstance d wth espect to the descptos D and D ae: d D d D ( D D ) ( D D ) The devatves of D and D wth espect to the paametes w k and b k wth k {1,, 3} ae cacuated by noma back popagaton. Snce both subnets contbute to the oss, the devatves of the oss functon wth espect to each paamete must be summed ove the two subnets: N d D d D [ 1 N d D d D [ 1 L 1 L L ( )] s wk w N d D w d D w k k k (9) L 1 L L ( )] s bk b N d D b d D b k k k 4. EXPERIMENTS In ths secton we fst ntoduce the expementa data and setup. Afte that, we compae the tanng agothm descbed n ths pape and to othe common tanng methods, whch s foowed by an evauaton of ou descpto. Fnay, we compae ou method to othe state-of-at descpto eanng technques. 4.1 Expementa Data and Setup Ou expements ae based on the Bown dataset (Bown et a., 011) s used. Ths dataset s wdey used n descpto eanng studes, e.g. (Tzcnsk et a., 01; 015; Han et a., 015; Zagouyko and Komodaks, 015). The dataset contans thee sepaate subsets - Note Dame (ND), Yosemte (Yos) and Statue of Lbety (Lb). A patches wee extacted n the vcnty of Dffeence of Gaussan (DoG) featue ponts on ea mut-vew (7) (8) Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

6 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc mages. Thus, ea vewpont changes ae contaned n those datasets. The ogna patch sze s 64 x 64 pxes. The esze these patches to 3 x 3 pxes wth ant-aasng snce the nput of ou mode s desgned as 3 x 3 pxes. Fgue 5 gves some exampes of the tanng pas fom the Note Dame dataset (Bown et a., 011). Fgue 5. Exampes fo tanng pas. The eft thee coumns ae postve (matchng) tanng pas and the ght thee coumns ae negatve (non-matchng) tanng pas. The hype-paametes fo tanng wee chosen empcay. In deta, we taned n 30 epochs and 450 mn-batches ae used fo tanng. Othe paametes used hee ae β = 0.9, γ = 0.9, α = 0.003, α decease = 0.9, pu = 5, push = 10. Each mn-batch contans 500 postve and 500 negatve tanng sampes. 4. Convegence Behavou In ths secton we compae the convegence behavou of ou tanng method to standad gadent decent, gadent decent wth momentum and to gadent decent wth Nesteov's momentum. In ths compason, the same postve and negatve tanng sampes fom the Note Dame dataset wee used fo a fou tanng methods n 10 epochs. The eanng ate and the momentum tem wee set to the vaues descbed n secton 4.1. The esuts ae pesented n Fgue 6. The fgue shows that the decease of oss does not beneft too much fom usng ony gadent descent o gadent descent wth momentum; howeve, the tanng benefts dstncty fom movng aveage gadents combned wth the NAG (geen cuve n Fgue 6), whch obvousy eads to a much faste decease of the oss functon. Aveage Loss Standad Gadent Descent Standad Gadent Descent wth Momentum Nesteov Momentum Method Method n Ou Pape Epoch Numbe Fgue 6. Resuts of oss functon fo standad gadent descent, standad gadent descent wth momentum, NAG and the method suggested n ths pape. 4.3 Resuts and evauaton In the set of expements epoted n ths secton, the descpto s taned usng one of the thee datasets, wheeas the othe two datasets ae used as fo testng. Ths expement was epeated thee tmes, so that each dataset was used fo tanng once. Each dataset contans 50,000 postve and 50,000 negatve tanng pas. The coss vadaton set conssted of 5,000 matchng pas and 5,000 non-matchng pas that wee andomy seected fom the dataset. Thus, the numbe of patch pas used fo gadent descent was 450,000 n each expement. The coss vadaton set was used to detemne the oss afte each epoch n ode to evauate the stoppng cteon: When the oss measued dd not mpove fo thee subsequent epochs, the tanng pocess was stopped. To mpement the whoe achtectue budng and eanng agothm expaned n secton 3, we used the matconvnet softwae 1 (Vedad and Lenc, 014) to conduct the convouton, poong, sgmod and back-popagaton of the basc CNN ayes. The ovea tanng pocedue of the Samese mode s based on ou own mpementaton. It uns on a 8-coe 3.40Ghz CPU; tanng fo one dataset takes about 11 hous. Fo each tanng dataset, the pefomance test s evauated on the othe two datasets, whch s a standad evauaton ue, aso suggested n (Bown et a., 011). In each test dataset, a the postve and negatve exampes ae used as evauaton dataset. The evauaton cteon s the fase postve ate at 95% eca ate. A owe fase postve ate at 95% eca ate means bette pefomance. Afte tanng, the descptos fo each patch n the test datasets ae detemned usng the paametes eaned wth the CNN. Then, the L Nom of the two descptos of each test pa s computed as the smaty measue of the patch pa. A patch pa wth an L Nom beow a theshod h s cassfed to be a match, othewse t s judged as a non-match. Thus, n essence, the eaned descpto can be consdeed to be a dect epacement of SIFT. As the tue abes (match o non-match) of a patch pas ae known, the tue postve and fase postve ate can be cacuated. By vayng the theshod h a ROC cuve s geneated. The v_oc functon n the vfeat softwae s used to obtan the ROC cuve of the fase postve ate aganst the tue postve ate. Tabe sts the esuts of ou wok, compang them to sevea state-of-at methods. None of the methods compaed n the tabe contans a decson aye,.e., a cassfe to detemne the matchng abe (matched o unmatched). The st consttutes a compason of cuent state-of-at methods fo descpto eanng. In the method SIM (Smonyan et a., 014), eanng s based on a convex optmzaton stategy. The eanng pocedue s an extenson of method BR (Bown et a., 011), whch s a benchmak n descpto eanng. Fo method TRC (Tzcnsk et a., 015), we chose the best pefomng descpto vaant fo ou compason, whch s the foatng pont veson wth 64 bts. In method OS (Osendofe et a., 013), a descpto eanng achtectue based on a Samese CNN sma to ou wok was used, but the authos concentated moe on the compason of dffeent foms of oss functons and the mode s taned by standad gadent descent. Fnay, SIFT (Lowe, 004) s used as a genea basene fo the 1 (accessed 05 Ap 016) (accessed 05 Ap 016) Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

7 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc Tanng Test Ous SIM BR TRC OS SIFT ND Yos ND Lb Lb ND Lb Yos Yos ND Yos Lb Mean Tabe. Fase postve ate [%] at 95% eca ate fo the dffeent methods beng compaed n ths wok usng dffeent tanng and test data subset combnatons (ND: Note Dame, Lb: Statue of Lbety, Yos: Yosmte). Compaed methods: SIM (Smonyan et a., 014); BR (Bown et a., 011); TRZ (Tzcnsk et a., 015); OS (Osendofe et a., 013), SIFT (Lowe, 004). descpto matchng, because t s wdey acknowedged as a good descpto n a featue engneeng manne. Among the sx combnatons of tanng and test dataset cases, ou method and (Smonyan et a., 014) acheve the best esuts n thee cases each. Fo the mean eo ate at 95% eca, ou method s sghty wose but compatbe wth (Smonyan et a., 014). Ou method exceeds the best descpto vaant n (Tzcnsk et a., 015), namey FPBoost 51-{64}, n tems of eo ate at 95% eca n a tanng and test data combnatons and a pefomance mpovement of neay 7.1% s acheved. To the best of ou knowedge, Osendofe et a. (013) pubshed the best esuts fo a method fo descpto eanng based on Samese CNN achtectue wthout cassfe so fa; t s the method most sma to ous n ou compason. Compaed to ths method, we acheved a pefomance mpovement of 3.5%. Compaed to SIFT, ou method, as we as the othe machne eanng based descptos, shows a dstnct mpovement n tems of the eo ate at 95% eca. Some of the andomy seected tue postve, fase postve, tue negatve and fase negatve patch pas ae shown n fgue 7. To pck those patch pas, the paametes ae taned fom the Statue of Lbety tanng data and the seected esuts ae a fom the Note Dame dataset. Tue Postve Pas Tue Negatve Pas When apped to ea mage matchng o mage eteva, a featue descpto needs to be matched aganst thousands of othes. Theefoe, as an extenson of ou wok we w adapt the method by adjustng the popoton of postve and negatve tanng sampes that the mode sees dung tanng. Anothe extenson ncudes appyng ths achtectue to tan descptos that ae abe to cope wth specfc stuatons ke obque aea mages whch contan moe compex geometc tansfomatons. ACKNOWLEDGEMENTS The autho Ln Chen woud ke to thank the Chna Schoashp Counc (CSC) fo fnancay suppotng hs PhD study at Lebnz Unvestät Hannove, Gemany. REFERENCES Aanæs, H., Dah, A. L., Pedesen, K. S., 01. Inteestng nteest ponts. Intenatona Jouna of Compute Vson, 97(1), pp Bay, H., Ess, A., Tuyteaas, T., et a., 008. Speeded-up obust featues (SURF). Compute Vson and Image Undestandng, 110(3), pp Bes, J. S., Lowe, D. G., Shape ndexng usng appoxmate neaest-neghbou seach n hgh-dmensona spaces. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton, pp Fase Postve Pas Fase Negatve Pas Bengo, Y., Couve, A., Vncent, P., 013. Repesentaton eanng: A evew and new pespectves. IEEE Tansactons on Patten Anayss and Machne Integence, 35(8), pp Fgue 7. Some esuts of test on Note Dame dataset. 5. CONCLUSIONS In ths pape we descbe tanng of a descpto based on a Samese CNN achtectue. In compason to othe wok based on Samese CNN, we use a moe advanced gadent descent tanng agothm and take a smae nput patch sze. Ou wok demonstates that wth advanced tanng stateges, descptos based on Samese CNN acheve state-of-at pefomance on the Bown dataset. Bshop, C. M., 006. Patten ecognton and machne eanng. Spnge, New Yok, pp BMRL (Bommetc Robotcs and Machne Leanng Goup, TU Munch, Gemany), 013. mspop. (accessed 05 Ap 016) Bomey, J., Bentz, J. W., Bottou, L., Guyon, I., LeCun, Y., Mooe, C., & Shah, R., Sgnatue vefcaton usng a Samese tme deay neua netwok. Intenatona Jouna of Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii

8 ISPRS Annas of the Photogammety, Remote Sensng and Spata Infomaton Scences, Voume III-3, 016 XXIII ISPRS Congess, 1 19 Juy 016, Pague, Czech Repubc Patten Recognton and Atfca Integence, 7(04), pp., Bown, M., Hua, G., Wnde, S., 011. Dscmnatve eanng of oca mage descptos. IEEE Tansactons on Patten Anayss and Machne Integence, 33(1), pp Chen L., Rottenstene F., Hepke, C., 015. Featue descpto by convouton and poong autoencodes. In: The Intenatona Achves of Photogammety, Remote Sensng and Spata Infomaton Scence, 40(3), pp Caevas-Banco, N., Eustce, R. M., 014. Leanng vsua featue descptos fo dynamc ghtng condtons. In Intenatona Confeence on Integent Robots and Systems (IROS 014), pp Hadse, R., Chopa, S., LeCun, Y., 006. Dmensonaty educton by eanng an nvaant mappng. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton, Vo., pp Han, X., Leung, T., Ja, Y., Sukthanka, R., Beg, A. C., 015. MatchNet: Unfyng Featue and Metc Leanng fo Patch- Based Matchng. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton. pp Hnton, G., Svastava, N., Swesky, K., 016. Neua netwoks fo machne eanng - Lectue 6e: mspop. ec6.pdf (accessed 05 Ap 016) Jahe, M., Gabne, M., and Bschof, H., 008. Leaned oca descptos fo ecognton and matchng. In Compute Vson Wnte Wokshop. Vo.. Moavske Topce, Sovena. LeCun, Y., Bottou, L., Bengo, Y., Haffne, P., Gadentbased eanng apped to document ecognton. Poceedngs of the IEEE, 86(11), Ln, T. Y., Cu, Y., Beonge, S., Hays, J., Tech, C., 015. Leanng Deep Repesentatons fo Gound-to-Aea Geoocazaton. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton, pp Lowe, D. G., 004. Dstnctve mage featues fom scaenvaant keyponts. Intenatona Jouna of Compute Vson, 60(), pp Nesteov, Y., A method of sovng a convex pogammng pobem wth convegence ate O (1/k). In Sovet Mathematcs Dokady, 7(), pp Osendofe, C., Baye, J., Uban, S., van de Smagt, P., 013. Convoutona Neua Netwoks ean compact oca mage descptos. In Neua Infomaton Pocessng, Spnge Ben Hedebeg, vo. 88, pp Rumehat, D. E., Hnton, G. E., Wams, R. J., Leanng epesentatons by back-popagatng eos. NATURE, 33(9), pp Smonyan, K., Vedad, A., Zsseman, A., 01. Descpto eanng usng convex optmsaton. In: Euopean Confeence on Compute Vson, pp Smonyan, K., Vedad, A., Zsseman, A., 014. Leanng oca featue descptos usng convex optmsaton. IEEE Tansactons on Patten Anayss and Machne Integence, 36(8), Sutskeve, I., Matens, J., Dah, G., Hnton, G., 013. On the mpotance of ntazaton and momentum n deep eanng. In Poceedngs of the 30 th ntenatona confeence on machne eanng (ICML-13), pp Toa, E., Vncent, L., Fua, P., 010. Dasy: An effcent dense descpto apped to wde-basene steeo. IEEE Tansactons on Patten Anayss and Machne Integence, 3(5), pp Tzcnsk, T., Chstoudas, M., Lepett, V. and Fua, P., 01. Leanng mage descptos wth the boostng-tck. Advances n neua nfomaton pocessng systems, In Advances n neua nfomaton pocessng systems, pp Tzcnsk, T., Chstoudas, M., Lepett, V., 015. Leanng mage descptos wth boostng. IEEE Tansactons on Patten Anayss and Machne Integence, 37(3) pp Vedad, A., Lenc, K MatConvNet-convoutona neua netwoks fo MATLAB. In Poceedngs of the 3d Annua ACM Confeence on Mutmeda Confeence, pp Zagouyko, S., Komodaks, N., 015. Leanng to compae Image Patches va Convoutona Neua Netwoks. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton, pp Zbonta, J., LeCun, Y., 015. Computng the steeo matchng cost wth a Convoutona Neua Netwok. In Poceedngs of the IEEE Confeence on Compute Vson and Patten Recognton, pp Mkoajczyk, K., Schmd, C., 005. A pefomance evauaton of oca descptos. IEEE Tansactons on Patten Anayss and Machne Integence, 7(10), pp Moees, P., & Peona, P., 007. Evauaton of featues detectos and descptos based on 3d objects. Intenatona Jouna of Compute Vson, 73(3), Moe, J. M., & Yu, G., 009. ASIFT: A new famewok fo fuy affne nvaant mage compason. SIAM Jouna on Imagng Scences, (), Ths contbuton has been pee-evewed. The doube-bnd pee-evew was conducted on the bass of the fu pape. do: /spsannas-iii