FAST AND ROBUST IMAGE FEATURE MATCHING METHODS FOR COMPUTER VISION APPLICATIONS

Transcription

1 FAST AND ROBUST IMAGE FEATURE MATCHING METHODS FOR COMPUTER VISION APPLICATIONS Vom Fachbeech fü Physk und Eektotechnk de Unvestät Bemen zu Eangung des akademschen Gades enes Dokto-Ingeneu (D.-Ing.) genehmgte Dssetaton von M.Sc.-Ing. Faaj Ahwan aus Syen Refeent: Koefeent: Pof. D.-Ing. Ae Gäse Pof. D. Ph. Nat. Dete Sbe Engeecht am:. Janua Tag des Pomatonskooquums: 6. Ap

2 Acknowedgements I woud fst of a ke to thank my doctoa fathe Pof. D. -Ing. Ae Gäse fo gvng me the vauabe oppotunty to wok n ths nteestng eseach fed, fo hs vauabe suggestons and gudance dung my doctoa eseach wok and vey thoughtfu comments fo the mpovements of ths thess. Futhe, I woud ke to thank Pof. D. Dete Sbe fo beng the second evewe of ths thess and hs vey thoughtfu comments. My thanks go aso to Pof. D. Ing. Wate Anhee and Pof. D. Ing. Abeto Gaca Otz fo showng nteest to be on my dssetaton commttee. I woud ke to thank my coeagues n the IAT nsttute, who wee aways thee to assst me n povdng ctcs and suggestons to my eseach wok. Especay I thank D. Danjea Rstc Duant fo he knd suppot dung the wtng of ths dssetaton. She spent much tme evsng the manuscpt and heped me wth he nsghtfu comments. I woud ke to thank my wfe Najed Ahwan fo beng so patent and undestandng dung the dffcut tmes whe gong though the doctoa eseach pogam. I appecate a the sacfces whch she has made fo me n ode to accompsh ths wok. Lasty, I woud ke to thank my famy fo the endess ove that fomed the most mpotant pat of my gowng-up and fo aways beng thee when I needed them, hepng me to face dffcutes. The endess suppot and encouagement has made the jouney ease and one that I w teasue fo many yeas to come. Bemen, Ma Faaj Ahwan

3 Abstact Sevce obotc systems ae desgned to sove tasks such as ecognzng and manpuatng objects, undestandng natua scenes, navgatng n dynamc and popuated envonments. It's mmedatey evdent that such tasks cannot be modeed n a necessay detas as easy as t s wth ndusta obot tasks; theefoe, sevce obotc system has to have the abty to sense and nteact wth the suoundng physca envonment though a muttude of sensos and actuatos. Envonment sensng s one of the coe pobems that mt the depoyment of mobe sevce obots snce estng sensng systems ae ethe too sow o too epensve. Vsua sensng s the most pomsng way to povde a cost effectve souton to the mobe obot sensng pobem. It's usuay acheved usng one o sevea dgta cameas paced on the obot o dstbuted n ts envonment. Dgta cameas ae nfomaton ch sensos and ae eatvey nepensve and can be used to sove a numbe of key pobems fo obotcs and othe autonomous ntegent systems, such as vsua sevong, obot navgaton, object ecognton, pose estmaton, and much moe. The key chaenges to takng advantage of ths powefu and nepensve senso s to come up wth agothms that can eaby and qucky etact and match the usefu vsua nfomaton necessay to automatcay ntepet the envonment n ea-tme. Athough consdeabe eseach has been conducted n ecent yeas on the deveopment of agothms fo compute and obot vson pobems, thee ae st open eseach chaenges n the contet of the eabty, accuacy and pocessng tme. Scae Invaant Featue Tansfom (SIFT) s one of the most wdey used methods that has ecenty attacted much attenton n the compute vson communty due to the fact that SIFT featues ae hghy dstnctve, and nvaant to scae, otaton and umnaton changes. In addton, SIFT featues ae eatvey easy to etact and to match aganst a age database of oca featues. Geneay, thee ae two man dawbacks of SIFT agothm, the fst dawback s that the computatona compety of the agothm nceases apdy wth the numbe of key-ponts, especay at the matchng step due to the hgh dmensonaty of the SIFT featue descpto. The othe one s that the SIFT featues ae not obust to age vewpont changes. These dawbacks mt the easonabe use of SIFT agothm fo obot vson appcatons snce they eque often ea-tme pefomance and deang wth age vewpont changes. Ths dssetaton poposes thee new appoaches to addess the constants faced when usng SIFT featues fo obot vson appcatons, Speeded up SIFT featue matchng, obust SIFT featue matchng and the ncuson of the cosed oop conto stuctue nto object ecognton and pose estmaton systems. The poposed methods ae mpemented and tested on the FRIEND II/III sevce obotc system. The acheved esuts ae vauabe to adapt SIFT agothm to the obot vson appcatons. Page II

4 Kuzfassung Sevce Robot-Systeme snd entwofen, um Aufgaben we das Ekennen und Beabeten von Objekten, das automatsche Vestehen natüche Szenen und de Navgaton n dynamschen, von Menschen bevökete Abetsumgebungen zu eedgen. Es st unmtteba enschtg, dass dese Aufgaben ncht n aen notwendgen Detas we de Fa mt Industeoboten modeet weden können. Deshab soen Sevceobote de Fähgket haben, mt de umgebenden physschen Umwet duch ene Vezah von Sensoen und Aktoen ageen und eageen zu können. De Umwetefassung st ene de wchtgsten Gundagen autonome Sevceobote, de den kommezeen Ensatz mobe Sevceobote beschänkt, we Wahnehmungssysteme entwede zu angsam ode zu teue snd. Vsuee Wahnehmung st de vespechendste Vaante, um ene kostengünstge Lösung fü das Wahnehmungspobem von moben Roboten dazusteen. Vsuee Wahnehmung st n de Rege mt ene ode meheen dgtaen Kameas auf dem Robote montet ode st n sene Abetsumgebung vetet. Dgtae Kameas snd Infomatonseche Sensoen und snd eatv günstg und können vewendet weden, um ene Rehe wchtge Pobeme fü de Robotk und andee Autonome Integente Systeme duchzufühen, we z. B. vsuee Sevong, Robote-Navgaton, Objektekennung, Poseschätzung, und vee andee Anwendungen. De zentae Heausfodeung st es, de dese estungsstaken und kostengünstgen Sensoen mt Agothmen zusammen kommen, de zuveässg und schne nützche vsueen Infomatonen etaheen und se automatsch ntepeteen können. Obwoh betächtche Foschungen den etzten Jahen duchgefüht woden snd, um de Entwckung von Agothmen fü Compute- und Robot Vson Pobeme zu ösen, gbt es noch offene Foschungsfagen m Zusammenhang mt de Zuveässgket, Genaugket und Aufwandzet. Skaennvaante Bdmekmaen (SIFT) st enes de am häufgsten vewendeten Methoden, de heutzutage ve Aufmeksamket n den Compute-Vson-Communty gewdmet weden, aufgund de Tatsache, dass SIFT Featues besondes ausgepägt snd, und nvaant bezügch auf Skaeung, Rotaton und de Beeuchtungsveändeungen snd. Daübe hnaus snd SIFT Featues eatv echt zu etaheen und gegen ene goße Datenbank von okaen Mekmaen zu vegechen. Im Agemenen, gbt es zwe wesentche Nachtee von SIFT- Agothmus: de este Nachte st das de Kompetät des Agothmus schne stegt mt de Anzah de Schüsse-Punkte, vo aem an dem Matchng-Schtt wegen de hohen Dmensonatät des SIFT Featue Deskptos. De andee st, dass de SIFT Featues ncht obust gegen goße Bckwnkeveändeungen snd. Dese Nachtee beschänken de venünftge Nutzung des SIFT- Agothmus fü Robot Vson- Anwendungen, da se häufg Echtzet-Lestung und den Umgang mt goße Bckwnkeveändeung efoden. Dese Dssetaton stet de neue Ansätze zu Bewätgung de Zwänge konfontet da, wenn de SIFT Featues fü Robot Vson-Anwendungen vewendet weden wd es de neue Ansätze geben: bescheungte SIFT Featue Matchng, obuste SIFT Featue Matchng und de Enbezehung des geschossenen Regekeses n de Objektekennung und Kameakabeungssysteme. De vogeschagenen Methoden snd mpementet und an dem FRIEND II/III Sevce- Robot-System getestet. De ezeten Egebnsse snd wetvo fü de Anpassung von SIFT- Agothmus an den Robote-Vson-Anwendungen. Page III

5 Contents. Intoducton..... Motvaton..... Contbutons Thess Oganzaton.... Robot Vson Tasks Sevce Robotc Camea Cabaton Intnsc Camea Paametes (Camea to Image) Etnsc Camea Paametes (Camea to Wod) Steeo Vson Eppoa Geomety Fundamenta Mat Tanguaton..... Vsua Sevong Poston-based Vsua Sevong Image-based Vsua Sevong Hybd Vsua Sevong Image Matchng Featue Detecton Edge Detectos Cone Detectos Bob Detectos Featue Descpton Coo Descptos Tetue Descptos Shape Descptos Featue Matchng Smaty Measues Matchng Stateges Seachng Technques SIFT Agothm SIFT Featue Etacton Scae-Space Etema Detecton Key-Ponts Locazaton Oentaton Assgnment Key-Ponts Descpton SIFT Featue Matchng SIFT Coespondences Seach Msmatches Dscadng Fast SIFT Featue Matchng Intoducton Ccua Random Vaabes PDF of Sum/Dffeence of Unfomy-Dstbuted ICRVs PDF of Sum/Dffeence of ICRVs... 5 Page IV

6 5.3. Spt SIFT Featue Matchng Etended SIFT Featue Matchng Speeded-Up Facto SIFT Featue Ange Etended SIFT Featues Matchng Epementa Resuts Vey Fast SIFT Featue SIFT Descpto Based Featue Anges Vey Fast SIFT Featues Matchng Epementa Resuts Concuson Robust SIFT Featue Matchng Intoducton Impoved SIFT Featues Matchng Scang Facto Cacuaton Reteva of The Coect Matches Compety and Cost of Tme Epementa Resuts Concusons Fuzzy Based Cosed Loop Conto System fo Object Recognton Intoducton Cosed Loop Conto System fo Object Recognton Dssmaty between Two Affne Tansfomatons Fuzzy Contoe Fuzzfcaton Infeence Defuzzfcaton Epementa Resuts Concusons Concuson and Outook...9 Bbogaphy... Page V

7 Lst of Fgues Fgue.: FRIEND III ehabtaton obotc system, deveoped at the Unvesty of Bemen, Insttute of Automaton... 5 Fgue.: components of the ehabtaton obotc system FRIEND II... 6 Fgue.3: The coodnate systems nvoved n camea cabaton... 7 Fgue.: The eppoa geomety... Fgue.5: Paae steeo vson system... 3 Fgue.6: Non-Paae steeo vson system.... Fgue.7: Vsua sevo conto system... 6 Fgue.8: Poston-based vsua sevong system Fgue.9: Image-based vsua sevong system Fgue.: Hybd vsua sevong system... Fgue 3.: The Has and Stephens cone detecto... 6 Fgue.: SIFT agothm (SIFT featue etacton and matchng) Fgue.: A Gaussan scae space conssts of 3 octaves, each octave has scae eves Fgue.3: Constuctng the DoG scae space fom the Gaussan scae space [] Fgue.: The Dffeence of Gaussan Scae Space Fgue.5: Scae-space etema detecton [].... Fgue.6: A 36 bns oentaton hstogam constucted usng oca mage gadent data aound key-pont Fgue.7: SIFT descpto constucton... 3 Fgue 5.: The ccua pobabty densty functon of the sum of two ndependent unfomy dstbuted ccua andom vaabes... 5 Fgue 5.: wappng the g aound the ccumfeence of a cce of unt adus... 5 Fgue 5.3: the Mama and Mnma SIFT featues etacted fom the same mage Fgue 5.: The vecto sum of the bns of an eght oentaton hstogam Fgue 5.5: The epementa PDFs of sum and tan, k fo SIFT featues etacted fom 6 test mages Fgue 5.6: The epementa PDF of the ange dffeence j fo ncoect and coect matches... 6 Fgue 5.7: Etended SIFT featue matchng pocedue Fgue 5.8: Matchng esut between two mages of the same scene maged fom two dffeent vewponts Fgue 5.9: Some of the standad dataset mages of scenes captued unde dffeent condtons: (a) vewpont, (b) ght changes, (c) zoom, (d) otaton Fgue 5.: Steeo mages fom a ea-wod obotc appcaton used n the epements Fgue 5.: Tade-off between matchng speedup and matchng pecson fo ea steeo mage matchng Fgue 5.: Tade-off between matchng speedup (SF) and matchng pecson fo mage goups (a) ght, (b) vewpont, (c) otaton, (d) zoom changes Fgue 5.3: (a) SOHs,(b):Vecto sum of the bns of a SOH, (c) anges computed fom SOHs Fgue 5.: The PDFs of anges estmated fom 6 SIFT featues etacted fom 7 mages Page VI

8 Fgue 5.5: The coeaton coeffcents between anges of SIFT featues. Fo eampe the top eft dagam pesents coeaton coeffcents between and a. The and y aes j pesent ndces and j espectvey whe z as pesent coeaton facto Fgue 5.6: The epementa PDFs of the ange dffeence j fo the possbe (a) and the coect matches (b) Fgue 5.7: Tade-off between matchng speedup (SF) and matchng pecson Fgue 5.8: Coect SIFT featue coespondences between two mages of the same scene captued unde two dffeent condtons Fgue 6.: Tansfomaton of both mode and test mage nto two coectons of SIFT featues; dvson of the featues sets nto subsets accodng to the octave of each featue Fgue 6.: Steps of the pocedue fo scae facto cacuaton... 8 Fgue 6.3: The scae ato hstogam F k Fgue 6.: Savng the coect matches that may eceed Lowe's theshod... 8 Fgue 6.5: Reca vesus -Pecson cuves fo the ogna and optmzed SIFT matchng methods Fgue 6.6: (eft coumn) matchng esut wth ogna SIFT, (ght coumn) matchng esut wth mpoved SIFT Fgue 7.: Goba featue-based object ecognton system... 9 Fgue 7.: Loca featue-based object ecognton system... 9 Fgue 7.3: poposed cosed oop object ecognton system Fgue 7.: Dssmaty between two affne tansfomatons Fgue 7.5: Stuctue of eatona fuzzy contoe Fgue 7.6: Fuzzy- based system fo affne tansfomaton seecton Fgue 7.7: Thee types of wdey used membeshp functons: (a) tangua, (b) tapezod, and (c) Gaussan type membeshp functons Fgue 7.8: Input and output membeshp functons and the anges Fgue 7.9: Gaphca epesentaton of centod aea method... Fgue 7.: Two eampes of the database mages (eft coumn) mode mages, (ght coumn) quey mages Fgue 7.: An eampe of used ea wod mages... 3 Fgue 7.: update of mage matchng and pose estmaton esuts dung tme. Left mage matchng esut and ght ts coespondng pose estmaton esut. In each teaton, the tansaton eos (E, Ey and Ez n mm) and otaton ange eos (E, E and E n degee) ae sted. Note that the numbe of matches s nceased, the dffeence of the both estmated poses s deceased and convegence to the pose of taget object... 6 Fgue 7.3: Matchng and pose esuts of the fna teaton fo some mode and quey mage pas Page VII

9 Lst of Tabes Tabe 5.: Compason between Standad and Spt SIFT Featue matchng Tabe 6.: The confuson Mat Tabe 6.: Compason of the steeo mages matchng tme Tabe 7.: The database of ngustc vaabes Tabe 7.: Rue base of poposed fuzzy contoe... Tabe 7.3: Fuzzy-epet ues n ngustc fom... Tabe 7.: Combned fuzzy-epet ues... Tabe 7.5: Compason between object poses estmated by Mnma and Mama SIFT matches... Page VIII

10 Intoducton. Intoducton The pmay goa n the fed of sevce obotcs s to desgn autonomous obots, whch ae capabe to move aound n the envonment, to avod obstaces, to ecognze objects and to nteact wth them. Theefoe sevce obotc system has to have the abty to sense and nteact wth the suoundng physca envonment though a vaety of sensos and actuatos. The fundamenta equement fo the souton of such pobems s the 3D econstucton of the envonment, whch means the detemnng the dstance between the obot and ts envonment ponts. Geneay, the 3D econstucton can be pefomed usng actve o passve sensng systems. The actve sensng systems can be cassfed based on the pncpe that s used to measue dstances nto tme of fght-based [] and tanguaton-based systems []. The tme-of-fght-based system s a scanne that uses ase ght to pobe a scene. The most popua type of tme-of fght-based system s ase angefnde. The ase angefnde fnds the dstance of a suface by tansmttng enegy as ase ght out nto the obot envonment, then measung the etun tme of efected enegy. Snce the speed of ght s known, the ound-tp tme detemnes the tave dstance of the ght, whch s twce the dstance between the scanne and the object suface. The accuacy of a tme-of-fght ase scanne depends on how eacty the tme can be measued. The ase angefnde ony detects the dstance of one pont n ts decton of vew. Thus, the scanne scans ts ente fed of vew one pont at a tme by changng the ange fnde s decton to scan dffeent ponts. The tanguaton-based system s aso a scanne that uses ase ght to nvestgate the envonment. In tems of tme-of-fght ase scanne the tanguaton ase shnes a ase on the subject and uses a camea to ook fo the poston of the ase dot. Dependng on how fa away the ase stkes a suface, the ase dot appeas at dffeent paces n the camea s fed of vew. Ths technque s caed tanguaton because the camea, ase emtte, and ase dot pojected onto the object fom a tange. Snce the dstance between the emtte and the camea s known and the ange of the ase emtte cone s aso known, the ange of the camea cone can be detemned by ookng at the ocaton of the ase dot n the camea s fed of vew. These thee peces of nfomaton fuy detemne the shape and sze of the tange and gves the ocaton of the ase dot cone of the tange. Scannng systems can poduce hghy accuate 3D measuements but tend to be epensve. Snce scannes opeate by scannng a snge pe wth evey pass and have mechanca components, they ae buky and sow especay when acqung a sgnfcant fed of vew at usefu esoutons. Despte the passve systems such as steeo vson have a ow ea-tme capabty and have no homogenous depth map, they ecenty eceved a ot of attenton due to the cheap costs. Steeo vson method woks sma to 3D pecepton n human vson by compang the smates and dffeences between two mages and s based on tanguaton between the pes that coespond to the same scene stuctue pojecton on each of the mages. Two mages of the scene ae suffcent n ode to compute 3D depth nfomaton. If a 3D pont n the wod can be dentfed as a pe ocaton n an mage, ths wod pont es on the ne Page

11 Intoducton passng by that pe ocaton and camea pojecton cente. If we use two cameas, we can obtan two nes. The ntesecton of these nes s the 3D ocaton of the wod pont. In ode to econstuct the 3D envonment of the obot usng steeo vson, two pobems have to be soved:. Identfy pes n mages that match the same wod pont. Ths pobem s known as the coespondence pobem.. Identfy the 3D coodnates of each pe n the mage and the camea pojecton cente. Ths pobem s known as the camea cabaton pobem. Camea cabaton ncudes the detemnaton of the optca paametes and the geometca ocaton of the camea. Both pobems ae soved by mage-matchng technques. Image matchng technques may fnd coespondences fo ony a spase set of featues n the mage (featue-based mage matchng), o attempt to fnd coespondences fo evey pe n the mage (dense mage matchng) [3]... Motvaton Steeo vson ees on fndng the coespondng ponts on two spatay sepaated mages and then usng tanguaton to get the 3D measuement. Ths pocess of fndng the coespondng ponts s senstve to geometc and photometc tansfomatons asng fom umnaton and vewpont changes. The accuacy of the 3D esuts of steeo matchng depends upon many factos such as mage tetue, mage esouton, foca ength and basene dstance. The ncease n basene mpoves the accuacy at ong ange but compcates the mage matchng pobem and naows the fed of vew (FoV). Hghe mage esouton nceases the accuacy of the esuts but aso may ncease the pocessng tme of mage matchng. The scae nvaant featue tansfom (SIFT) method poposed n [] s cuenty the most wdey used fo mage matchng due to the fact that SIFT featues ae hghy dstnctve, and nvaant to mage tansaton, scang, and otaton. SIFT featues ae aso patay nvaant to umnaton changes and affne 3D pojectons. In addton, SIFT featues ae eatvey easy to etact and to match. Geneay, thee ae two man dawbacks of SIFT agothm, the fst dawback s that the computatona compety of the agothm nceases apdy wth the numbe of key-ponts (hgh mage esouton), especay at the matchng step due to the hgh dmensonaty of the SIFT featue descpto. The othe one s that the SIFT featues ae not obust to age vewpont changes (wde-base ne). These dawbacks mt the easonabe use of SIFT agothm fo obot vson appcatons snce they eque often ea-tme pefomance and need to dea wth age vewpont changes. The goa of ths dssetaton s essentay to addess the SIFT dsadvantages pesevng a ts vey mpotant advantages. Specfcay, we ntend to mpove SIFT s obustness to vewpont changes and to acceeate SIFT featue matchng, whch s vey mpotant fo obot vson appcatons. Page

12 Intoducton.. Contbutons Ths thess makes thee man contbutons. Fsty, t poposes a new stategy fo fast SIFT featue matchng by etendng SIFT featue by some new attbutes. Secondy, t ntoduces new method fo obust SIFT featue matchng. Ths method s based on the potzed matchng. Fnay, t ncudes a fuzzy ogc based cosed oop system fo pecse object ecognton, pose estmaton, and camea cabaton.. Speeded up SIFT Featue Matchng. Fndng coespondences between SIFT featues s the pat of the matchng agothm that takes the most amount of pocessng tme, especay when the numbe of featues to be compaed s eatvey age. Most obot vson appcatons eque ea-tme esponse. Unfotunatey, the estng stateges fo speedng up featue matchng ae nadequate fo obot vson appcatons snce they ethe wok fo offne matchng such as Appomate Neaest Neghbo (ANN) seachng methods o gve nsuffcent acceeaton such as PCA- SIFT [5], Speeded Up Robust Featue (SURF) [6], Fast Appomated SIFT (FA-SIFT) [7] and Reduced SIFT (R-SIFT) [7]. Ths thess poposes a new stategy to speed up featue matchng. Ths stategy s based on the cassfcaton of SIFT featue nto sevea custes though featue etacton phase based on sevea new ntoduced attbutes computed fom SIFT oentaton hstogam (SIFT-OH) o SIFT descpto (SIFT-D). Thus, n the featue matchng phase ony featues ae compaed that shae amost the same coespondng attbutes. Ths stategy has speeded up mage matchng by a facto of about accodng to ehaustve seach, and has aso mpoved the matchng quaty sgnfcanty.. Potzed SIFT Featue Matchng Some obot vson tasks, such as camea cabaton and pose estmaton eque obust featue matchng. Even though SIFT featues ae easonaby nvaant, they can not accommodate age changes n vewpont, wtch s the coe pobem of camea cabaton and pose estmaton. Ths pobem s caused by ethe the absence of tue postve coespondences o the poton s nsuffcent fo fttng methods to wok coecty. Ths eseach ntoduces a new pocedue to detemne the scae facto between mages to be matched by dvdng SIFT featues nto dffeent sub-sets based on the octaves. Then the matchng pocess s done n potzed ode, so that ony the featues of the same scae ato ae compaed on each step. At the same tme a scae ato hstogam (SRH) s constucted. Ony matches of the step coespondng to the hghest SRH bn ae povded to the fttng method. Ths estcton deceases the poton of outes among postve matches eadng to mpove the pefomance of the fttng methods, such as Random Sampe Consensus (RANSAC) [5] o Least Medan of Squaes (LMS) methods. 3. Fuzzy ogc based cosed Loop Conto SIFT featue matchng In ths eseach, a fuzzy ogc-based cosed oop conto system s ncuded to ncease the accuacy of object ecognton, pose estmaton and camea cabaton. The dea s to etact two dffeent types of SIFT featues, fom mode and quey mages. These featues ae Page 3

13 Intoducton matched sepaatey povdng two ndependent affne tansfomatons. The smaty between these tansfomatons s used as a contoed vaue and passed to fuzzy contoe to seect one of these tansfomatons to wap the mode mage. The matchng pocess s epeated unt a temnaton cteon s met..3. Thess Oganzaton The thess s oganzed as foows: In chapte, basc concepts fom the fed of compute vson ae povded. Fsty, a genea descpton of the sevce obotc system FRIEND II/III s pesented. Futhemoe, backgounds of steeo vson and camea cabaton ae befy descbed, whch ae the common pobems n many compute vson appcatons. As an eampe of obot vson appcatons, vsua sevong s descbed. In chapte 3, mage matchng methods ae befy evewed befoe focusng on the featue-based methods. We aso evew genea aspects of featue etacton, descpton and matchng. Chapte pesents SIFT agothm n detas, snce t s the man concen of ths thess. In Chapte 5, fsty some aspects of the statstc of ccua andom vaabes ae descbed and n ths contet a new theoem has been ntoduced and poven. Based on ths theoem, sevea hashng methods ae poposed to speed up SIFT featue matchng. In Chapte 6, obust SIFT featue matchng based on potzed matchng s pesented to ncease the nvaance to affnty. In chapte 7 the ncuson of fuzzy-based cosed oop conto system fo object ecognton and pose estmaton s demonstated. Epementa esuts ae ncuded n each Chapte to demonstate the effcacy of the poposed methods. Fnay, Chapte 8 concudes ths thess and dscusses possbe etensons and futue eseach dectons. Page

14 . Robot Vson Tasks.. Sevce Robotc Robot Vson Tasks The pmay objectve n sevce obotcs s to desgn autonomous obots, whch ae abe to move aound n ts envonment, to ecognze cetan objects, to pan a moton to the destnaton of objects, possby to gab them and to conto the eecuton of the task. These systems shoud be abe to wok obusty n any envonment wthout econfguaton. The aea of sevce obotcs has ecenty eceved sgnfcant attenton. Sevce obots ae used fo many tasks such as ceanng, obsevng, and hepng human n the cayng out of dffcut tasks. In moe ecent tme, some of the most domnatng effots have been devoted to ehabtaton obotcs that ae desgned to hep edey and dsabed peope n the actvtes of day fe, such as pepang and sevng a cup of dnk, pckng up a teephone, o fetchng and handng a book. Fgue.: FRIEND III ehabtaton obotc system, deveoped at the Unvesty of Bemen, Insttute of Automaton FRIEND (Functona Robot am wth fiendy nteface fo Dsabed peope) [9] s a ehabtaton obot contoed based on vsua sensng and desgned to suppot dsabed and edey peope n the day fe actvtes (Fgue.). FRIEND system has been deveoped at the Insttute of Automaton of Unvesty Bemen snce 997. FRIEND s equpped wth an Page 5

15 Robot Vson Tasks eectc wheecha, a 7 degees of feedom (7-DoF) mounted manpuato wth a gppe and a muttude of sensos, ncudng steeo vson system as coe components attached to a pantt head. Besde steeo vson system, the obot has addtona oca sensos that can ncease ovea obustness of the task eecutons, fo nstance, a foce/toque senso s but n a gppe base, wtch can be used fo contact detecton when pacng an object on tabe. The system has an ntegent tay conssts of a sensoy suface wth nfaed emttes and eceves. -DOF pan-tt head Steeo Camea Chn joystck 7-DOF manpuato am TFT dspay Mn joystck Compute system Pannng am of TFT dspay Wheecha patfom Pannng am of manpuato Fgue.: components of the ehabtaton obotc system FRIEND II. Fo the human-machne nteface pupose, the system s equpped wth sevea nput devces such as chn joystck, hand joystck, voce conto, and ban compute nteface (BCI). The nput devces ae adapted accodng to the mpaments of the use o hs pefeences. The objectve of the ehabtaton obotc system FRIEND s to hep dsabed patents n the day fe actvtes. Thus, the obot opeates n a human, unstuctued envonment, as depcted n the scene fom Fgue. To pefom ts tasks autonomousy, the obot must be abe to sense ts envonment wtch s the task of the steeo vson system. The steeo vson system s a bumbebee steeo camea system wth but-n cabaton, synchonzaton and steeo pojectve cacuaton featues s used to acque nfomaton of the envonment. It s mounted at the top of the obot system on a pan-tt-head unt. Fgue. pesents the man components of the ehabtaton obotc system FRIEND II. Fo moe detas about FRIEND system the eade ae efeed to [9] [] and []... Camea Cabaton The pocess of budng the eatonshp between the wod coodnate system and that of a captued mage s caed camea cabaton. Camea cabaton s a necessay step fo many Page 6

16 Robot Vson Tasks compute vson appcatons especay fo the functonng of obots that ae meant to nteact vsuay wth the physca wod. These obots can then use a vdeo nput devce and cabate n ode to fgue out whee objects t sees mght actuay be n the ea wod, n actua tems of dstance and decton. The eatonshp between the 3-D wod coodnates and the coespondng mage coodnates s usuay descbed by two goups of paametes:. Intnsc camea paametes (Intena geometc and optca chaactestcs of the camea).. Etnsc camea paametes (poston and oentaton of the camea n the wod coodnate system). Z C C V m U M X Y C X C Y Y W W Z W X W Fgue.3: The coodnate systems nvoved n camea cabaton....intnsc Camea Paametes (Camea to Image) Intnsc camea paametes descbe the optca and geometca chaactestcs of the camea. The camea coodnate system has ts ogn at the cente of pojecton, ts z as aong the optca as, and ts and y aes paae to the and y aes of the mage, as shown n Fgue.3. Assumng that a pont M on an object wth coodnates T c, yc, zc measued n the camea coodnate system, s maged at the pont m, y n the mage pane. These coodnates ae wth espect to a coodnate system whose ogn s at the ntesecton of the optca as and the mage pane, and whose X and Y aes ae paae to the X c and Y c aes. Camea coodnates and mage coodnates ae eated by the pespectve pojecton equatons: f y f c c and y (.) zc zc Page 7

17 Robot Vson Tasks Whee f s the foca ength (dstance fom the cente of pojecton to the mage pane). The actua pe coodnates u v m, ae defned wth espect to an ogn n the top eft hand cone of the mage pane, and w satsfy: u y u and v v (.) w h whee w and h ae the wdth and the heght of the pe espectvey. By substtutng equatons (.) n (.) and mutpyng both sdes by z c z c yeds: c f yc f u zc u and zc v zc v (.3) w h The equatons (-3) can be wtten neay usng the homogeneous coodnates as: su f sv s w f h u v c yc z c (.) whee the scang facto s has vaue of z c. In shot hand notaton, we wte equaton (.) as ~ ~ U K (.5) M c whee U ~ epesents the homogeneous vecto of mage pe coodnates, K s the pespectve pojecton mat, and M ~ c s the homogeneous coodnates of a pont measued n the camea coodnate system. Thee ae fve camea paametes, namey the foca ength f, the pe wdth, the pe heght and the paametes u and v whch ae the u and v pe coodnate at the optca cente espectvey. Howeve, ony fou sepaabe paametes can be soved fo as thee s an abtay scae facto nvoved n f and n the pe sze. Thus we can ony sove fo the atos f w and f h. u v The paametes u, v, u and v do not depend on the poston and oentaton of the camea n space, theefoe they ae caed the ntnsc paametes....etnsc Camea Paametes (Camea to Wod) A cabaton taget can be maged to povde coespondences between ponts n the mage and ponts n space. It s, howeve, geneay mpactca to poston the cabaton taget accuatey wth espect to the camea coodnate system. As a esut, the eatonshp between the wod coodnate system and the camea coodnate system typcay aso needs to be ecoveed fom the coespondences. The wod coodnate system can be any system convenent fo the patcua desgn of the taget. Page 8

18 Robot Vson Tasks Etnsc camea paametes descbe the eatonshp between a wod coodnate system and the camea coodnate system. The tansfomaton fom wod to camea conssts of a otaton and a tansaton. Ths tansfomaton has s degees of feedom, thee fo otaton and thee fo tansaton. ae the coodnates of a 3D pont M measued n the wod coodnate c c c c ae the coodnates of the same pont n the camea coodnate system, then the eatonshp between M and M s: If M T w w, yw, zw system and M, y, z T c w M c R M T (.6) w The equaton (-6) can be ewtten n homogeneous coodnates as: 3 R 3, t t t y z ~ M c ~ RT M w T, R T (.7) R T 3 T whee T s the tansaton vecto captung the camea dspacement fom the wod fame ogn and R s the otaton mat encodes the camea oentaton wth espect to the wod coodnate system. By substtutng the equaton (-5) n (-7), we get the tansfomaton between mage and wod coodnate system. Ths tansfomaton s ca pojecton mat ncudes ntnsc and etnsc camea paametes. ~ U K P ~ R T M w (.8).3. Steeo Vson The human vson and depth pecepton s based, n pat, on the compason between the two eyes' mages. These two mages epesent two sghty dffeent pojectons of the wod n the etnas. The fuson of the two mages fom the ght and eft eye channe n the ban ceates the sensaton of depth. Compute steeo vson tes to mtate ths depth pecepton. The basc dea s to get two dffeent mages of the same scene acqued by steeo camea system fom two dffeent pespectves. A compute anayses the two mages and tes to match them. Once the mages have been bought nto pont-to-pont coespondence, ecoveng depth by tanguaton s staghtfowad; hence, the chaenge n steeo vson s to fnd coespondng ponts n steeo mages. Ths s a dffcut task and tme consumng; howeve, the compety of ths task can be educed by pecsey anaysng the geomety of the steeo system confguaton. The geomety descbng steeo vson s caed eppoa geomety. Page 9

19 Robot Vson Tasks.3.. Eppoa Geomety The eppoa geomety descbes the geometc eatons between a 3D pont and ts pojecton n two cameas. Any pont n the 3D wod space togethe wth the centes of pojecton of two cameas systems, defnes an eppoa pane. The ntesecton of such a pane wth an mage pane s caed an eppoa ne as shown n Fgue.. Evey pont of a gven eppoa ne must coespond to a snge pont on the coespondng eppoa ne. Theefoe, the eppoa geomety can be used to constant the seach fo coespondng mage pont n the fst mage to one dmensona neghbohood n the second mage. In ode to pesent eppoa geomety mathematcay, some defntons ae needed: M C m e e m C Fgue.: The eppoa geomety. Eppoe: The pojecton of the cente pont of the eft camea n the mage pane of the ght camea s caed eppoe. So, et e epesents the mage of ght camea s cente ( C ) n the eft mage. Smay, e epesents the mage of eft camea s cente ( C ) n the ght mage. These ponts, e and e, ae known as eppoes. Eppoa ne: m s the mage of M n the eft camea. The ne ( e ; m ) n the eft camea s caed an eppoa ne. Ths ne s the pojecton of ( C ; M ) n the eft camea. The patcuaty of ths ne s that t s seen by the ght camea as a pont and by the eft camea as a ne that goes though the eppoe e. So, a eppoa nes ntesect at the eppoe e. Symmetcay ( e ; m ) defnes an eppoa ne n the ght camea. Eppoa pane: C, C and M defne an eppoa pane. The eppoa nes assocated wth M can be seen as the ntesecton of the eppoa pane wth the mage panes of the cameas. Page

20 Robot Vson Tasks The geometca eatons between eppoes, eppoa nes and eppoa panes can be epessed mathematcay by ntoducng a mat caed fundamenta mat..3..fundamenta Mat Fundamenta Mat F s the agebac epesentaton of the eppoa geomety between two cameas. Ths mat captues the epesentaton of the pojectve map fom m n one mage to ts coespondng eppoa ne n the othe mage. The pojecton of any 3D pont M n the eft and ght pnhoe cameas can be wtten n mat fom: m m K K M M (.9) whee K and K ae espectvey the pojectve mat of the eft and ght camea. M and M ae the coodnates of M n the eft and ght camea coodnate systems espectvey. The coodnate system of the ght camea can be tansfomed nto the coodnate system of the eft camea though a otaton R and a tansaton T. Theefoe equaton (-9) can be ewtten as: m m I M K K R t M (.) These equatons can be combned to emove M. m H K (.) RT K m The mat that maps each pe n the eft mage to eacty one coespondng pe n the ght mage s caed the homogaphy mat H. Snce each eppoe ne has both the coespondng mage pont m and the eppoe e on t, t s defned as: e e m m e m e m Howeve, we just saw that H s the tansfe mappng of m to (.) m, ths can be wtten as: F H e F m e H m (.3) whee e s the vecto poduct mat assocated wth the eppoe e : Page

21 Robot Vson Tasks ez e y e ez e (.) e y e Ths mat depends on the ntnsc mat of the cameas ( C and C ) and on the eatve poston ( R and T ). Hence, n a statc setup whee the eatve poston of the cameas s known and whee the cameas have been cabated,.e. the ntnsc matces ae known, the fundamenta mat can be computed once and fo a. If the cameas ae not cabated, the fundamenta mat can be estmated usng a fttng agothm fom n>8 coespondences ponts. f f f3 F f f f 3 f T f f f3 f f f 3 f 3 f 3 f 33 (.5) f 3 f 3 f 33 Each coespondng pont pas,, and,, y gves an equaton: y y y y y y y f (.6) Stackng n equatons fom n pont coespondences gves nea system A f, whee A s an n 9 mat. If ank A 8 then the souton s unque (up to scae) but n eaty we seek a east-squaes (LS) souton wth n 8. Then LS souton s the ast coumn of the mat V n the sngua vaue decomposton (SVD) of mat A : whch coesponds to the smaest sngua vaue..3.3.tanguaton A U D V T (.7) The tanguaton s a pocess to econstuct the 3D coodnates of a pont fom ts D mages. Each pont n an mage pane coesponds to a 3D ne n wod space whch passes though ths pont and the cente of pojecton of the camea. If two coespondng ponts n two mages ae the pojecton of a common 3D wod pont M, then the assocated 3D nes must ntesect at M. In pactce, howeve, the coodnates of mage ponts cannot be measued wth abtay accuacy. Instead, vaous types of nose, such as geometc nose fom ens dstoton o nteest pont detecton eo ead to naccuaces n the measued mage coodnates. As a consequence, the 3D nes do not aways ntesect n wod space. The pobem, then, s to fnd a 3D pont whch optmay fts the measued mage ponts. In the teatue thee ae mutpe poposas fo how to defne optmaty and how to fnd the optma 3D pont such as Md-pont method o Dect Lnea Tansfomaton (DLT) []. The 3D poston X, Y, Z of a pont M, can be econstucted fom the pespectve pojecton Page

22 Robot Vson Tasks of M on the mage panes of the cameas, once the eatve poston and oentaton of the two cameas ae known. We choose the 3D efeence system to be the eft camea system. The ght camea s tansated and otated wth espect to the eft camea. Thee ae two key confguatons of steeo vson systems: paae and non-paae. In a paae confguaton, the optca aes of two cameas ae paae, and the tansaton of the ght camea s ony aong the X as. In a non-paae confguaton, the optca aes of two cameas ae non-paae and the ght camea can be ocated abtay wth espect to the eft camea.... Paae Cameas If the optca aes of two cameas ae paae, and the tansaton of the ght camea s ony aong the X as, the coespondence ponts e on the same hozonta ne; theefoe the coespondence pobem becomes a one-dmensona seach aong coespondng nes. M m C m C Fgue.5: Paae steeo vson system. The offset between a pe n the eft and ts coespondng pe n the ght mage s caed dspaty. D (.8) Once the dspaty vaues ae known, the wod coodnates of a pont can be computed as. b f Z D Z X f y Z Y f (.9) Page 3

23 Robot Vson Tasks whee f s the foca ength of both cameas and b s the dstance between the two camea pojecton centes ( basene ). In ths confguaton the matchng pocess s vey smpe, but the accuacy of 3D coodnates and the mamum depth that can be measued depend on the ength of the basene. Hgh accuacy woud eque a onge basene, whch causes a educton n the common fed of vew (FoV), so that ony a smae poton of the scene s vsbe. The contadcton between the accuacy of 3D econstucton and the sze of the common FoV can be eceeded usng the non-paae confguaton.... Non-Paae Cameas In the non-paae confguaton, the ght camea can be tansated and otated wth espect to the eft one n thee dectons. Gven the tansaton vecto T and otaton mat R descbng the tansfomaton fom eft camea to ght camea coodnates, the equaton to sove fo steeo tanguaton s: In the non-paae confguato, the ght camea can be tansated and otated wth espect to the eft one n thee dectons. Gven the tansaton vecto T and otaton mat R descbng the tansfomaton fom eft camea to ght camea coodnates the equaton to sove fo steeo tanguaton s: m T R m T (.) whee m and m ae the coodnates of M n the eft and ght camea coodnates T espectvey, and R s the tanspose (o the nvese) mat of R. If a pont M X Y, Z two ponts m, y and y, n 3D space s gven, wth two cameas t can be sepaatey pojected to m, espectvey. Eventuay f m and m ae known, then a ne can connect m and the pojecton cente of the eft camea C. Smay, an othe ne can connect m and C. It s obvous that M must be on the ne ntesecton. M m y m y C C Fgue.6: Non-Paae steeo vson system. Page

24 Robot Vson Tasks Page 5 The eatonshps between 3D wod pont and ts mages ae gven as: M P m M P m (-) whee P and P ae the eft and the ght pojecton matces espectvey The above equatons can be combned nto a fom AM whch s a nea equaton system n M. The homogeneous scae facto can be emnated by a coss poduct whch gves thee equatons fo each mage pont on the eft and ght steeo mages. Ths can be mathematcay epessed as: Epandng equatons (.), we get: whee T p and T p fo,,,3 ae the ows to consdeed the eft and the ght pojecton matces espectvey. Snce the equatons (.3) ae nea n the components of M, an equaton of fom AM can be then composed as descbed n equaton (.). As descbed pevousy, two equatons have been ncuded fom each steeo mages pa, gvng a tota of fou equatons n fou homogeneous unknowns. The souton of the above homogeneous equaton can be obtaned usng DLT agothm. Snce the vaue of A s known, a non-zeo souton fo M s found usng SVD method wtch satsfes the equaton AM. M P m M P m M P m M P m (.) M p M p M p M p y M p M p M p y M p M p M p y M p M p T T T T T T T T T T T T (.3) M p p y p p p p y p p A T T T T T T T T (.)

25 Robot Vson Tasks.. Vsua Sevong Vsua sevong s a agey used technque whch s abe to conto on-ne obots by usng the nfomaton povded by one o many cameas. Two typca tasks ae usuay pefomed usng vsua sevng, postonng and tackng. The fome ams at agnng the obot o the gppe wth the taget object, whe the atte ams at keepng a constant eatonshp between the obot and the movng taget object. In both cases, mage nfomaton s used to measue the eo between the cuent ocaton of the obot and ts desed ocaton. The desed ocaton s defned by an mage (caed desed mage) peceved n such confguaton. Though the matchng the vsua featues (such as ponts, nes and egons) etacted fom the desed and nta mages, the nta ocaton s obtaned accodng to the desed ocaton. The obot movement can be obtaned on-ne though the estmaton of coespondences between featues etacted fom mages taken sequentay fom dffeent postons. Contoe Robot System Camea System Refeence Image Image Pocessng Cuent Image Fgue.7: Vsua sevo conto system The basc concept of vsua sevong s theefoe based on the undestandng of the scene geomety by the camea. The scene geomety s used to epan the eaton between obot moton n the wod and eated mage moton. In ode to descbe the geomety of the scene, thee coodnate systems ae used: camea, obot and wod coodnate systems. In genea, vsua sevong systems can be cassfed nto two categoes: poston-based (IBVS) and mage-based vsua vevong (PBVS). In a poston-based vsua sevong, the system nput s computed n the thee-dmensona Catesan space [3]. The pose of the taget object wth espect to the camea s estmated fom mage featues coespondng to the pespectve pojecton of the taget object n the mage. The pose estmaton methods [] ae usuay based on the knowedge of a pefect geometc mode of the object and necesstate a cabated camea to obtan unbased esuts. On the othe hand, mage-based vsua sevong use optca fow aong wth Jacoban-based conto to conto the camea, n ths case, the nput s computed n the mage pane [5]. Recenty, a new appoach has been poposed n [3] that epot the combnaton of the two above methods to estmate the camea tansfomaton between the desed and the cuent pose. They combne the tadtona Jacoban-based conto wth othe technques to fom the cass of hybd vsua sevong (HVS). These methods yed a decouped, optma camea tajectoy and possess a age snguaty-fee task space. Page 6

26 Robot Vson Tasks...Poston-based Vsua Sevong In PBVS, the task functon s defned n tems of the pose tansfomaton between the cuent and the desed poston, whch can be epessed as the tansfomaton c T. d + - Catesan Conto Law Robot Pose Estmaton Featue Etacton Camea Fgue.8: Poston-based vsua sevong system. The nput mage s usuay used to estmate the camea to object tansfomaton c T O whch can be composed wth the object to desed pose tansfomaton O T d to fnd the tansfomaton fom the cuent to the desed pose. By decomposng the tansfomaton matces nto tansaton and otaton, ths can be epessed as: c T d c T O O T d c O RO R O c R O d c O R O O O t O R O c t d c d O t c t O R O d d c t d (.5) The task functon fo poston s then the vecto c t d. Fo oentaton, the otaton mat can be decomposed nto as of otaton and otaton ange, whch can be mutped to get the desed otatona movement. The otaton ange and otaton as can be cacuated fom the eements of the otaton mat R. If the eements of the otaton mat ae epessed as: a a a3 R a a a3 (.6) a 3 a3 33 The otaton ange and the decton of otaton as ae gven as: a accos a a a a a a a a T (.7) Page 7

27 Robot Vson Tasks In hs knd of conto, an eo between the cuent and the desed poston of the obot s cacuated and used by the ow eve contoe to geneate the conto commands to move the obot to the desed poston. d t T t T e P P (.8) Thus, the poston-based contoe can be wtten: d d d P d u P (.9) The man advantage of ths appoach s that t decty contos the camea tajectoy n Catesan space. The centa dsadvantage of PBVS s that the pose estmaton s usuay based on the knowedge of a pefect geometc mode of the object and necesstates a cabated camea to obtan unbased esuts. Theefoe, f the camea s coase cabated, o f eos est n the 3D mode of the taget object, the cuent and desed camea poses w not be accuatey estmated whch thus eads to sevong faue....image-based Vsua Sevong + - Featue Conto Law Robot Depth Estmaton Featue Etacton Camea Fgue.9: Image-based vsua sevong system. IBVS nvoves the estmaton of the obot s veocty scew, so as to move the mage pane featues m m T m... m n * * * * to a set of desed ocatons T m m m... m n whch epesents the desed obot poston. The eo functon s defned as a functon of e m m m m... m n m n. Ths eo functon s updated n each fame and used togethe wth the mage Jacoban to estmate the conto nput to the obot. * * * dstance between these measuements T Assumng that a pont M on a taget object wth coodnates T, y, z measued n the m u, v n the mage pane. camea coodnate system, s maged at the pont Usng a cassca pespectve pojecton mode, the eatonshp between each mage pont and ts coespondng 3D wod pont s gven by: Page 8

28 Robot Vson Tasks Page 9 whee,, u a a v u and v ae the ntnsc camea paametes. The equatons (.3) can be wtten as: When the tme devatve of ths equaton s taken we obtan the eatonshp between the mage pont veocty and a 3D veocty scew: whee M J s the mage Jacoban mat gven by: whee f s the foca ength of the camea. The mage Jacoban epesents the dffeenta eatonshp between the scene fame and the camea fame (whee ethe the scene o the camea fame s usuay attached to the obot). The mage pont veocty and the 3D scew veocty ae gven by: The mage Jacoban mat eates the moton of D ponts n the mage pane (whch s the effect) to the moton of the coespondng 3D ponts n the Catesan space (wtch s the cause). When consdeng n 3D ponts togethe wth the pojectons on the mage pane, the Jacoban mat J fo the compete set of featues s: v z y v u z u v u (.3) M f m (.3) M M J m t M M M f t m (.3) u f v u f u z v z f v f u f v u z u z f M M f M J (.33) z y T T T t M M v v u u t m m * * (.3) T M n J M J M J J... (.35)

29 Robot Vson Tasks In IBVS systems, the conto eo functon s defned decty n D mage pane. If mage postons of pont featues ae used as measuements, the eo functon s defned smpy as a dffeence between the cuent and the desed featue postons as foows: e m m (.36) * The most common appoach to geneate the conto sgna fo the obots s the use of a smpe popotona conto [8] fo an optma conto appoach: The conto aw can be obtaned fom equatons (.3) and (.33) fo at east thee coespondng featues: whee K s a constant gan mat. In genea, mage-based vsua sevong s known to be obust not ony wth espect to camea but aso to obot cabaton eos [9]. Howeve, ts convegence s theoetcay ensued ony n a egon aound the desed poston...3.hybd Vsua Sevong T J J T T T J m KJ J J e u KM K (.37) Mas et a. [6] poposed a hybd conto scheme (caed,5d vsua sevong). It combnes the cassca poston-based and mage-based appoaches n ode to ovecome the espectve dawbacks: contay to the poston based vsua sevong, t does not need any geometc 3D mode of the object. Conto Law Poston Robot + - Rotaton - Pata Pose Estmato Featue Etacton Camea Fgue.: Hybd vsua sevong system. In contast to the mage-based vsua sevong, t guaantees the convegence of the conto aw to zeo eo n the whoe task space and does not need fo depth estmaton when cacuatng the mage Jacoban. Ths conto s based on the estmaton of the pata camea tansfomaton fom the cuent to the desed camea poses. Page

30 Robot Vson Tasks In each teaton, the otaton and the scaed tansaton of the camea between the cuent and the desed vews of the object ae estmated fom the homogaphy mat. Vsua featues etacted fom the pata tansfomaton ae used to desgn a decouped conto aw. The featue pont veocty vecto s augmented wth depth and otaton nfomaton z whee st he ato and and ae the ange and otaton as of the otaton mat * z etacted fom the homogaphy mat. Futhemoe, can be decty cacuated fom the homogaphy mat as: Whe the otaton ange and the decton of otaton as of the otaton mat ae computed fom otaton mat accodng to equaton (.7) Mas et a [6] defne the moton conto aw as: ~ * m * u u v v og (.38) T T m * n * H (.39) T m n Wth ~ m M KJ (.) ~ J * dˆ J t * dˆ J I 3 t J (.) whee J t and J ae the tansatona and otatona potons of the mage Jacoban mat, * composed of the fst thee and ast thee coumns of the Jacoban espectvey, and ˆd s an estmate of the dstance between the foca pont and the featue pont pane. Page

31 Image Matchng 3. Image Matchng In ode to measue the smaty between two mages, the vsua content of each mage has to be tansfomed nto quanttatve chaactestcs that can be measued and compaed wth eatvey tte ambguty. These quanttatve chaactestcs ae usuay caed mage featues and the pocess of compang mage featues s aso efeed to as the mage matchng whch tes to fnd coespondng featues n two o moe mages. Image matchng s a necessay step fo many compute vson appcatons such as mage egstaton, camea cabaton, 3D econstucton, vsua sevong, and obot navgaton. In genea, Image matchng technques can be cassfed nto two categoes: ntensty-based and featue-based mage matchng. Intensty-based methods compae ntensty pattens n mages va coeaton metcs, whe featue-based methods fnd coespondences between mage featues such as cones, edges, and bobs. The Intensty-based methods ae usuay easy to mpement but they can ony be apped to matchng the mages wth sma vewng condtons. These condtons ae had to satsfy n pactce, especay n obot vson appcatons whee mages come wth many shapes and appeaances. In addton, these methods ae not obust to defomaton, occuson and backgound cutte. Featue-based methods ae based on the estabshment of the coespondences between a numbes of ponts n mages. Theefoe they ae moe obust to both cutte and occuson. The featue-based matchng appoaches typcay nvove the foowng steps: Featue Detecton Featue Descpton. Featue Matchng. 3.. Featue Detecton Featue detecton efes to pocess that ooks fo postons n a gven mage whee a patcua featue of a gven type can be ocated. Vsua featue s defned to be the descpton of an mage egon whch contans sgnfcant stuctua nfomaton, such as edges, cones, and othe pattens. In ode to detect nteest egons of an mage, a saency measue s defned and ooked fo ts oca Etema acoss the mage pes and acoss dffeent szes of the egon. The dea of checkng dffeent mage szes s to be abe to detect the same egon even f the egon s pesent at dffeent scaes n dffeent mages. Ths eads to so caed scae nvaant detecton. The seecton of the saency measue Etema s to make detecton pocess moe epeatabty. The featue epeatabty s defned as the pobabty that the same featue w be detected n two o moe dffeent mages of the same scene, even unde dffeent captung conatons. In teatue, thee ae many types of featues that can be etacted fom a dgta mage such as edges, cones, and bobs. Page

32 Image Matchng Edges mak the boundaes between dffeent aeas n the mage, fo eampe aeas of dffeent bghtness eves, o tetue statstcs. Cones ae found at the peaks n the autocoeaton functon o ponts whee edges ntesect. Bobs ae found n the stabe centes of unfom egons. Based on featue type, featue detectos can be dvded nto tee goups: Edge, cone and bob detectos. 3...Edge Detectos Edges ae ocated whee ntensty vaues n the two-dmensona mage functon undego a shap change fom one state to anothe, such as fom a whte squae to a back backgound. These ponts ae the oca mama of the gadent of the mage. Canny edge detecton [] s an effcent pocess that poduces a bnay edge mage n whch evey pont s abeed as an edge o othewse. Edge detecton s a pobem of fundamenta mpotance n mage anayss. In typca mages, edges chaacteze object boundaes and ae theefoe usefu fo segmentaton, egstaton, and object ecognton n a scene. An edge s a bounday between two mage egons epesented as a jump n ntensty. In genea, the coss secton of an edge can be of abtay shape (usuay amp). In pactce, edges ae usuay defned as sets of ponts n the mage whch have a stong gadent magntude. Fo a contnuous mage I (, y), whee and y ae the ow and coumn coodnates espectvey, we typcay consde the two dectona devatves g and g. Of patcua nteest n edge detecton ae two functons that can be epessed n tems of these dectona devatves: the gadent magntude and the gadent oentaton. The gadent magntude s defned as: And the gadent oentaton s gven by: m y g g, y (3.), y tan g g y (3.) I (, y ) I (, y) Whee g and g y y Loca mama of the gadent magntude justfy edges n I (, y) whch s the basc dea of the fst ode devatve- based edge detectos. An odd symmetc fte w appomate a fst devatve, and peaks n the convouton output w coespond to edges n the mage. Often, the fst devatve of the dgta mage s epessed as a convouton of the dgta mage wth a convouton mask whch s aso aways caed edge opeato, and then the esutng outputs ae pocessed to gve a gadent map. y Page 3

33 Image Matchng The magntude of the gadent map s cacuated and seves as nput of a non-mama suppesson pocess. Fnay the esutng map of oca mama s theshoded to poduce the edge map. Whe the fst devatve acheves a mamum, the second devatve s zeo. Fo ths eason, an atenatve edge-detecton stategy s to ocate zeos of the second devatves of I (, y). The dffeenta opeato used n these so-caed zeo-cossng edge detectos s the Lapacan: I I(, y) I(, y) y g g yy (3.3) The zeo cossng detectos such as Ma- Hdeth and Lapacan of Gaussan (LoG) edge detectos [] ook fo paces n the Lapacan of an mage whee the vaue of the Lapacan passes though zeo.e. ponts whee the Lapacan changes sgn. Such ponts often occu at edges n mages.e. ponts whee the ntensty of the mage changes apdy. The statng pont fo the zeo cossng detecto s an mage whch has been fteed usng the LoG fte. 3...Cone Detectos Geneay, a cone s defned as the ntesecton of two edges, but n mages, cones efeed to as pes that coespond to mama n the autocoeaton functon A numbe of agothms fo cone detecton have been epoted n ecent yeas. They can be dvded nto two goups. Agothms n the fst goup nvove etactng edges and then fndng the ponts havng mama cuvatue o seachng fo ponts whee edge segments ntesect. The second goup conssts of agothms that seach fo cones decty fom the gey-eve mage, so that cone can aso be defned as a pont fo whch thee ae two domnant and dffeent edge dectons n a oca neghbohood of the pont. The quaty of a cone detecto s often judged based on ts abty to detect the same cone n mutpe mages, whch ae sma but not dentca, fo eampe havng dffeent ghtng, tansaton, otaton and othe tansfoms. One of the eaest nteest pont detecton agothms s the Moavec cone detecto []. In the agothm, a sde wndow aound a pe s moved n fou dectons and the gay-eve change n fou dectons ae computed E, y. E, y s vey sma of each decton f the pe s on a smooth egon. At edges, E, y changes ony n one decton. Fo a cone pont, E, y changes geaty n a dectons Theefoe, the cone stength at a pe s defned as the smaest sum of squaed dffeences between the patch and ts neghbong patches. E y Iu, v Iu, v y, (3.) u v The pobem wth Moavec cone detecto s that the patches ony n hozonta, vetca, and dagona dectons ae consdeed; that s the agothm s not sotopc. Page

34 Image Matchng An atenatve appoach fo cone detecton used fequenty s based on a method poposed by Has and Stephens [3], whch n tun s an mpovement of a method by Moavec. The Has cone detecto s based on the oca auto-coeaton functon of a sgna; whee the oca auto-coeaton functon measues the oca changes of the sgna wth patches shfted by a sma amount n dffeent dectons. The Has cone detecto aso computes a coneness vaue, C, y, fo each pe n an mage. A pe s decaed as cone f the C, y s cacuated as: vaue of C s beow a cetan theshod. whee C, y wu, v Iu, v Iu, v y (3.5) u v I( u, v y) can be appomated by a Tayo epanson. Let I and I y be the pata devatves of I, y, such that By substtutng equaton (3.5) nto equaton (3.6), we obtan the foowng appomated coneness vaue. Though ewtng equaton (3.7) n mat fom we get: whee A s the Has mat. I( u, v y) I( u, v) I ( u, v) I ( u, v) y (3.6) C y wu, v I u, v I u, v, y y (3.7) u v y C, y y A (3.8) y I I I I I I y y A wu, v (3.9) u v I I y I y I I y I y In the equaton (3.9), the ange backets denote aveagng (.e. summaton ove (u,v)). If a ccua wndow (o ccuay weghted wndow, such as a Gaussan) s used, then the esponse w be sotopc. A cone s chaactezed by a age vaaton of y y C, n a dectons of the vecto. By anayzng the egenvaues of A, ths chaactezaton can be epessed n the foowng way: The mat A shoud have two age egenvaues fo a cone pont. Based on the magntudes of the egenvaues, the foowng nfeences can be made: Assumng that and ae the egenvaues of the mat A. Thee ae thee cases to be consdeed: Page 5

35 Image Matchng. If both and C, y changes sghty n any decton, the wndowed mage egon s of appomatey constant ntensty; ths ndcates a fat egon.. If one of the egenvaues s bg and the othe s sma, so the oca auto-coeaton functon s dge shaped, then ony oca shfts n one decton (aong the dge) cause weak change n C, y and sgnfcant change n the othogona decton; ths ndcates an edge. 3. If both egenvaues ae bg, so the oca auto-coeaton functon s shapy peaked, then shfts n any decton cause sgnfcant change; ths ndcates a cone. ae sma, so that the oca auto-coeaton functon Edge Cone Fat Edge Fgue 3.: The Has and Stephens cone detecto [3]. Has and Stephens note that eact computaton of the egenvaues s computatonay epensve, snce t eques the computaton of a squae oot, and nstead they suggest the vaue M gven by the equaton (3.): c M c A tac (3.) det A whee s a tunabe senstvty paamete Theefoe, the agothm does not have to actuay compute the egenvaue decomposton of the mat A and nstead t s suffcent to evauate the detemnant and tace of A to detect cones. The vaue of has to be detemned empcay, but n the teatue vaues n the ange. -.5 ae commony used Bob Detectos In the compute vson communty, a bob efes to unfom egon n the mage that s ethe bghte o dake than ts suoundng. Page 6

36 Image Matchng Thee ae two man casses of bob detectos, wateshed-based bob and dffeenta bob detectos. The wateshed-based detecto deveoped by Lndebeg [] s based on oca etemum n the ntensty. Detectng wateshed-based bobs n a one-dmensona functon s tva. In ths case t suffces to stat fom each oca mamum pont and ntate seach pocedues n each one of the two possbe dectons. Evey seach pocedue contnues unt t fnds a oca mnmum pont. As soon as a mnmum pont has been found the seach pocedue s stopped and the gey-eve vaue s egsteed. The base-eve of the bob s then gven by the mamum vaue of these two egsteed gey-eves. Fom ths nfomaton the gey-eve bob s gven by those pes that can be eached fom the oca mamum pont wthout descendng beow the base-eve. The two-dmensona case s moe eaboate, snce the seach then may be pefomed n a vaety of dectons. In [] Lndebeg poposed a methodoogy that avods the seach pobem by pefomng a goba bob detecton based on a pe-sotng of the gey-eves. In ode to etact both dak bobs and bght bobs, watesheds ae typcay etacted fom the gadent mage. In pactce, the botteneck of the wateshed-based detecto s the nheent nose senstveness whch eads typcay to ove segmented esuts. To ovecome ths, t woud be hepfu to ncopoate nfomaton about shape and sze of the desed bobs nto the pocess of wateshed detecton, whch s hady feasbe. The dffeenta detectos ae based on devatve epessons such as Lapacan of Gaussan (LoG), Defeence of Gaussan (DoG) and Detemnant of Hessan (DoH). The Lapacan of Gaussan fte (LoG) s a combnaton of a Lapacan and Gaussan fte. Ths fte fst appes a Gaussan bu, and then appes the Lapacan fte. The fst stage of the fte uses a Gaussan kene to bu the mage n ode to make the Lapacan fte ess senstve to nose. Then, the Lapacan opeato s computed, whch usuay esuts n stong postve esponses fo dak bobs of etent and stong negatve esponses fo bght bobs of sma sze. A man pobem when appyng ths opeato at a snge scae, howeve, s that the opeato esponse s stongy dependent on the eatonshp between the sze of the bob stuctues n the mage doman and the sze of the Gaussan kene used fo pe-smoothng. In ode to automatcay detect bobs of dffeent unknown sze, the scae-nomazed LoG s apped at the scae space epesentaton. whee I, y * g(, y, ) y g, y, e (3.) L s Gaussan bued mage. The scae space epesentaton s constucted by teatvey convovng the hgh esouton mage wth Gaussan based kenes of dffeent sze. L L L yy (3.) Page 7

37 Image Matchng Lndebeg [5] poposed a method fo detectng bobs ke featues n a scae-space epesentaton. In ode to detect bobs and compute the scae, a seach fo Etema of scaenomazed Lapacan of Gaussan s pefomed. The DoG opeato can be used as an appomaton to the LoG to fnd vey stabe nteest ponts n the cente of stabe bobs. In a sma way as fo the LoG, bobs can be detected fom scae-space Etema of DoG. Anthe bob detecto s based on the scae-nomazed detemnant of the Hessan (DoH) [6] as epaned by the equaton beow. whee L Ly H s the Hessan mat. Ly Lyy In tems of scae seecton, bobs defned fom scae-space Etema of the scae-nomazed DoH aso have sghty bette scae seecton popetes unde non-eucdean affne tansfomatons than the othe two popua bob detectos, LoG and DoG 3.. Featue Descpton Once the nteestng ocatons n the mage have been detected, the task emans s to descbe these ocatons quanttatvey. The obtaned quanttatve descptons ae caed featue descptos. The descptos ae usuay hstogams of mage measuements deved fom nteest oca egons. In ode to be effectve, the descpto has to be dstnctve and at the same tme obust to nose and to changes n both vewpont and photometc magng condtons, hence a good tade-off between obustness and dstnctveness shoud be acheved whe desgnng the descpton pocedue. It s n essence a tageted data educton whch gves patcua nfomaton about an aea n a compact fom. In compute vson, sevea vsua descptos have been poposed fo epesentng the vsua content of mages. These descptos can be geneay cassfed dependng on the eementay chaactestcs of nteest nto tee majo goups: coo, tetue and shape descptos Coo Descptos H L L L det (3.3) yy Coo s one of the most wdey used vsua featues n mage descpton, smaty, and eteva tasks. Coo featues ae nvaant to otaton, tansaton, and scang, but not nvaant to umnaton changes. An mpotant ssue fo coo featue descpton s the choce of the coo space. The coo space s a mut-dmensona coodnate system, and each dmenson epesents a specfc coo component such as RGB, HSV. In the ast two decades, many coo descptos fo mages and mage egons have been poposed [6] such as Coo Hstogam (CH), Coo Moments (CM) and Coo Coheence Vecto (CCV). The CH s the basc coo descpto, whch descbes the coo dstbuton of the mage o the mage egon. CH s computed by dvdng coo space nto n dscete epesentatve coos, and countng the numbe of pes havng the same coo. y Page 8

38 Image Matchng Howeve, the man dsadvantage of the coo hstogam s that t s not obust to sgnfcant appeaance changes because t does not ncude any spata eatonshps among coos. The CCV [7] s an etenson of coo hstogams, n that each pe s cassfed as coheent o non-coheent based on whethe the pe and ts neghbos have sma coos. Coo Coeogam s poposed to chaacteze how the spata coeaton of pas of coos s changng wth the dstance [8]. Coo Coeogam povdes much bette pefomance than CH and the CCV. 3...Tetue Descptos In genea, Tetue efes to the vsua popetes of suface such as smoothness o oughness. Tetue can be seen amost anywhee. Fo eampe, tees, gass, sky, oads and budngs appea as dffeent types of tetue. Descbng tetues n mages by appopate tetue descptos povdes powefu means fo smaty matchng. A wde vaety of tetue descptos have been ecenty poposed. Tetue descptos can be cassfed nto two categoes: homogeneous and non-homogeneous tetue descptos. The homogeneous tetue descpto (HTD) povdes a quanttatve chaactezaton of homogeneous tetue egons that has homogenous popetes. It s based on computng the oca spata-fequency statstcs of the tetue usng the Gabo tansfom [3]. Because non-homogeneous tetues have statstca and stuctua popetes, non homogeneous tetue descptos can be categozed nto statstca and stuctua tetue descptos [9]. In stuctua appoaches, statstca dstbutons of tetue pmtve such as edges ae used to descbe tetue pattens. As an eampe fo stuctua tetue descpto s edge hstogam descpto (EHD) [3]. Ths descpto captues spata dstbuton of edges n the mage. In ode to constuct EHD, edges ae cassfed n fve edge categoes: vetca, hozonta, 5, 35, and non-dectona edge. Hence EHD s epessed as a 5-bn hstogam. Theefoe EHD s scae and otaton nvaant. Fo statstca appoaches, statstca dstbutons of ndvdua pe vaues such as gay eve hstogam and co-occuence mat ae computed to dscmnate dffeent tetues. The cooccuence mat s a two dmensona hstogam of the dstbuton of the co-occuence between two gey eve vaues at a gven dstance [3]. Tetue descptos, ae usuay computed ove the ente mage and esut n one featue vecto pe mage, and theefoe ae not obust to occuson and cuttes. In ecent yeas, some vey dscmnatve oca tetue descptos have been poposed such as Scae Invaant Featue Tansfom (SIFT) [], Speeded Up Robust Featue (SURF)[6] and Gadent Locaton and Oentaton Hstogam (GLOH)[8]. Loca descptos ae computed at mutpe ponts n the mage and descbe mage patches aound these ponts, and thus ae moe obust to cutte and occuson Shape Descptos In many compute vson appcatons, the shape epesentatons povde powefu vsua featues fo smaty matchng. In mage matchng, t s usuay equed that the shape descpto s nvaant to scang, otaton, and tansaton. Thee ae geneay two types of shape epesentatons, bounday-based and egon-based. Bounday-based methods such as Page 9

39 Image Matchng chan codes [33] and Foue descptos [3] need ony contou pes. Bounday-based shape descptos may not be sutabe to descbe egons that have compe shapes. Regon-based methods, howeve, ey not ony on the contou pes but aso on a pes encosed wthn the egon of nteest, hence they ae moe sutabe fo descbng egons of compe shapes. A egon can be descbed by consdeng scaa measues based on ts geometc popetes. The smpest popety s gven by ts aea. Aea s otaton nvaant, but changes wth changes n scae. Anothe smpe popety s defned by the pemete of the egon. Based on the aea and pemete t s possbe to chaacteze the compactness of egon, whch s defned by the ato of pemete to aea. The most popua shape descptos ae based on moments, whch descbe the shape and the ntensty dstbuton n mages. A genea defnton of moment functons functon I, y can be gven as foows: m of ode p q pq, of an mage ntensty Geometc moments ae nvaant to otaton and scae changes, but not nvaant to tansaton snce the output woud depend on the eatve pe postons wthn the mage. To acheve tansaton nvaant, centa moments ae deved fom geometc moments by shftng the mage so that the mage centod, y concdes wth the ogn of the mage coodnate system: Whee m m and y m m y In [35] Hu used centa moments to deve seven nvaant moments that wee then wdey used n patten ecognton: p q m pq y I, y (3.) p q y y I, y pq (3.5) y h h h3 3 3 h 3 3 h h h (3.6) Page 3

40 Image Matchng Hu moments ae cacuated fom centa geometc moments of ode up to the 3d. The man dawback efes to the age vaues of geometc moments, whch ead to numeca nstabtes and nose senstvty. Snce the basc functon p y q fo geometc moments s not othogona. Thus, Hu moments ae not othogona and as a consequence, the cacuated moments w have edundant nfomaton and cause ess accuacy of epesentng mages. Teague poposed Zenke moments based on the bass set of othogona Zenke poynomas [36]. The othogona popety of Zenke poynoma avods any edundancy between moments of dffeent odes. Zenke poynomas povded vey usefu moment kenes. They pesent natve otatona nvaance and ae fa moe obust to nose. Scae and tansaton nvaance can be acheved usng moment nomazaton. Fo D mages, the Zenke moment of ode p wth epetton q s defned as foows: Z pq p I * pq y, yv, ddy (3.7) jq whee p, q y, tan y, V pq, R pq e, and R pq () s ada poynoma of ode p wth coeffcents dependng on both p and q [36]. p, 3.3. Featue Matchng As a consequence of mage featue detecton and descpton, each mage s abstacted as a set of oca featues. Featue descptos ae usuay epesented as hstogams. In ode to match two mage, t s needed a seachng technque that compaes each pa of featues fom each mage based on a smaty measue (Eucdan, o Mahaanobs) of the espectve descptons and then makes a decson based on a matchng stategy. The featue matchng pocedue theefoe conssts of thee pats: the smaty measue, the matchng stategy and the seachng technque Smaty Measues If the featue s epesented as a hstogam, the smaty between two featues can be evauated usng any dstance measue sutabe fo hstogams. Thee ae two man types of smaty measues: bn-by-bn and coss-bn measues [37]. Bn-by-bn technques, ke the Mnkowsk ony compae coespondng hstogam bns, wthout egadng nfomaton n neaby bns. The Mnkowsk dstance of ode p s defned by the foowng equaton: whee d p N p V V V and V ae featue descptos fom the fst and the second mage espectvey. p, v v (3.8) V v v v (3.9)... N Page 3

41 Image Matchng V v v... vn The Eucdean dstance, whch s a speca case of Mnkowsk dstance when p, s the most common dstance measues used n pactce. In contast, coss-bn technques take nto account non-coespondng bns as we, and ae thus moe powefu. As an eampe fo coss-bn measue s the quadatc fom dstance (QFD), whch computes the mnma cost fo fowng bn matte fom one hstogam to fom the othe. The QFD s defned as foows: d, T V V V V AV V whee A a j s a bn-smaty mat whose eements a j ae gven by: a j (3.) dj (3.) ma d j whee d j v v s the dstance between two hstogams bns. j If the bn-smaty mat s postve-defntve, then the QFD becomes the L -nom between the nea tansfomatons of V and V. A speca case of QFD when the bn-smaty mat s the nvese of the covaance mat s the Mahaanobs dstance. The Mahaanobs dstance s adapted bette than the Eucdan dstance to descbe smates n mutdmensona spaces when non-sotopc dstbutons ae nvoved Matchng Stateges Thee ae thee common stateges to make a decson whethe two featues ae coecty matched each othe accodng to the matchng measue: absoute theshod, theshoded neaest neghbo and neaest neghbo dstance ato. Absoute Theshod: Two featues ae consdeed as a coect match f the absoute dstance between them s ess than a pe-set theshod. Unde ths matchng stategy, each featue fom the fst featue set may match to moe than one featue fom the second featue set. Theshoded Neaest Neghbo (TNN): Each featue fom the fst featue set ae matched to ts neaest neghbo featue fom the second set f the absoute dstance between them s ess than a pe-set theshod. In ths case, ony some featues fom the fst featue set may fnd coespondng featues fom the second featue set. Neaest Neghbo Dstance Rato (NNDR): Fo each featue fom the fst featue set, ts dstances to the neaest and the second neaest neghbo featues of the second featue set ae fsty computed. If the ato between these dstances s ess than a pe-set theshod, then the featue and ts neaest neghbo featue ae consdeed as a match. Page 3

42 Image Matchng Seachng Technques The smpes seach agothm fo neaest neghbo (NN) s the ehaustve seach, whee each featue n the fst featue set s compaed wth a featues n the second featue set. The man dawback of ehaustve seach s ts vey hgh compety. In ode to ovecome ths pobem, many methods have been poposed fo appomate neaest neghbo (ANN) seach. Geneay ANN seachng technques can be cassfed nto two goups: Heachca space patton-based and hash-based methods. Heachca space patton-based methods The fst goup nvoves a tee-based appoaches such as k-d tee. The k-d tee was poposed by Bentey [38] and s key the most wdey used ANN method. The k-d tee s a bnay seach tee n whch each node epesents a patton of the k-dmensona space. The oot node epesents the ente space, and the chd nodes epesent sub-spaces whch ae pat of the paent node's space. Evey node has a key vaue assocated wth one of the k- dmensons. At each node, ts space s dvdes nto two pats, eft subspace contans a th featues whose k component s ess than the key vaue and the ght sub-space contans a th featues whose k components s geate than the key vaue. When the tee s seached, the coespondng component of quey featue q s compaed aganst the node key vaue, and the appopate banch s foowed. Once a eaf node s eached, the quey featue s tested aganst a the featues n the eaf node and the cosest featue p s detemned. It may happen that the tue neaest neghbo p es n a dffeent eaf node. Ths w occu when the dstance between q and the bounday of ts bn egon s ess than the dstance between q and p. Theefoe, p s guaanteed to be the tue neaest neghbo f the sphee centeed at q wth adus q p s competey contaned wthn the bn egon. Ths s known as the bawthn-bounds (BWB) test. If the BWB test fas, then p may not be the tue neaest neghbo, and t s necessay to backtack up the tee and test ponts contaned n atenate paths. Anothe test whch must be egaded when the tee s seached s caed the bounds- oveapba (BOB) test. BOB test detemnes whethe o not the sphee centeed at q ntesects wth some egon, whch may theefoe contan the tue neaest neghbo. A ponts contaned n a bn egons that pass the BOB test must be consdeed dung backtackng. If a new neaest neghbo s encounteed, then the sphee adus s adjusted downwad, the BWB test s epeated, and the backtackng esumes f necessay. Thee ae many othe methods based on heachca space patton fo ANN seachng such as R-tees [39] and B-tees []. Howeve, a the above methods do not wok we fo hgh dmensona seachng space, because the ncease of the dmensonaty of the seachng ead to hghy unbaanced tees due to most of the tee eaves ae empty. Hash-based methods The second categoy conssts of hash-based appoaches whch tade accuacy fo effcency, by etunng appomate cosest neghbos of a quey pont. The most popua hash-based method s ocaty senstve hashng (LSH) []. The basc dea of LSH s to use a set of hash Page 33

43 Image Matchng functons that map sma featues nto the same hash bucket wth a pobabty hghe than non-sma featues. At ndeng tme, a the featues of the dataset ae nseted n L hash tabes coespondng to L andomy seected hash functons. At quey tme, the quey featue q s aso mapped onto the L hash tabes and the coespondng L hash buckets ae seected as canddates to contan featues sma to the quey featue. A fna step s then pefomed to fte the canddate featues by computng the dstance to the quey featue. Moe fomay, et V be a dataset of N d-dmensona featues n Fo any pont v beong to Assumng that G g d k d unde the L - nom. d, the notaton v epesents the L - nom of the vecto v. : be a Goup of hash functons such as: g v h v h v... h v (3.) k whee the functons d H h :. h beongs to a ocaty senstve hashng functon famy The functon famy H s caed R cr, p p senstve p p,, fo hq hv p when q v R hq hv p when q v cr L - nom f fo any q, v (3.3) d whee c and p p. Intutvey, that means that neaby featues wthn dstance R have a geate chance of beng hashed to the same vaue than featues that ae fa away (dstance geate than cr ). Fo the L - nom, the typcay used LSH functons ae defned as: d whee a s a andom vecto wth entes chosen ndependenty fom a Gaussan dstbuton and, w. b a ea numbe chosen unfomy fom the ange Fo ANN seachng tasks, the LSH ndeng method woks as foows:. L hash functons g g... g L fom G ae seected ndependenty and unfomy at andom, so that each hash functon s the concatenaton of k LSH functons andomy geneated fom H. v h v h v g h... k a v b h v w (3.). Each one of the L hash functons s used to constuct one hash tabe (esutng n L hash tabes). 3. A ponts v V ae nseted n each of the L hash tabes by computng the coespondng L hash vaues. Page 3

44 Image Matchng Dung the ceaton of the LSH hash tabes, the agothm stoes each data pont n the dataset nto buckets v j, L. Then, dung the pocessng of a quey q, the agothm g j, fo a seaches a buckets g q g q... g q L. Fo each featue v found n a bucket, the agothm computes the dstance fom q to v, and epots the featues f and ony f the dstances to quey featue ae ess than cetan theshod Whe ths method s vey effcent n tems of tme, tunng such hash functons depends on the dstance of the quey pont to ts cosest neghbo. Page 35

45 SIFT Agothm.SIFT Agothm The Scae Invaant Featue Tansfom (SIFT) method, poposed by Lowe [] s one of the most wdey used methods fo mage matchng whch s usefu fo amost a compute vson tasks. The agothm ntends to detect sma featue ponts n each of the avaabe mages and then descbe these ponts wth a featue vecto whch s nvaant to scae and otaton, and patay nvaant to umnaton and vewpont changes. In addton to these popetes, SIFT featues ae hghy dstnctve and eatvey easy to etact and to match, but the etacton as we as the matchng of these featues nvoves a consdeabe computatona cost. In ode to use SIFT agothm fo matchng pupose, SIFT featues whch coespond to dffeent vews of the same scene shoud have sma featue vectos. The mage matchng methods that use SIFT featues, conssts of two pats, SIFT featue etacton and SIFT featue matchng. Etacton nvoves fndng and descbng nteest egons o ponts, whe matchng means fndng of the coespondences among featues n dffeent mages. Image SIFT Featue Etacton Set of SIFT Featues SIFT Featues Matchng Set of SIFT Matches Image SIFT Featue Etacton Set of SIFT Featues Fgue.: SIFT agothm (SIFT featue etacton and matchng)... SIFT Featue Etacton SIFT agothm etacts key-ponts nvaant to scae and otaton usng the Gaussan dffeence of the mages n dffeent scaes to ensue nvaance to scae. Rotaton nvaance s acheved by assgnng one o moe oentatons to each key-pont ocaton based on oca mage gadent dectons. The esut of a ths pocess s a 8 dmensona descpto of gadents aanged togethe accodng to the oentaton and ocaton, whch povdes an effcent too to descbe an nteest pont, aowng an easy matchng aganst a database of key-ponts. The etacton of SIFT featues can be decomposed nto fou majo stages:. Scae-space Etema detecton: The fst stage seaches ove scae space usng a Dffeence of Gaussan (DoG) functon to dentfy potenta nteest ponts.. Key-pont ocazaton: The sub-pe ocaton and scae of each canddate pont s detemned and key-ponts ae fteed by etanng ony those that ae obust to nose and umnaton changes. 3. Oentaton assgnment: One o moe oentatons ae assgned to each key-pont based on oca mage gadent dectons.. Key-pont descpto: A descpto vecto s geneated fo each key-pont fom oca mage gadent data at the key-pont scae. Page 36

46 SIFT Agothm... Scae-Space Etema Detecton The ocatons of potenta nteest ponts n the mage ae detemned by detectng the Etema (Mama and Mnma) of DoG scae space. In ode to constuct DoG scae space, t s needed fsty to bud a Gaussan scae-space epesentaton of the mage. The GSS s but fom the convouton of the nput mage I, y wth a vaabe-scae Gaussan: whee s the convouton opeato n and y dectons and, y, kene gven by: L, y, I, y G, y, (.) G s the Gaussan y G, y, ep (.) As ustated n Fgue., the G-SS conssts of a sees of smoothed mages at dscete vaues of ove a numbe of octaves whee the sze of the mage s down-samped by two at each octave. Because of the ecusve popety of the Gaussan functon, n each octave each mage can be cacuated fom the pevous one. Snce L, y, ae bued wth nceasng, mages of the net octave can be down-samped as shown n Fgue., wthout osng mpotant nfomaton. Ths educes the computatona compety sgnfcanty. In SIFT method, the of the Gaussan scae space s quantzed n ogathmc steps aanged n O octaves, whee each octave s futhe subdvded n S scae eves. The vaue of at a gven octave o and scae eve s s gven by: o s, s o S s, S, o, O (.3) whee s the base scae eve. Page 37

47 SIFT Agothm s= s= s= s=3 o= o= o= Fgue.: A Gaussan scae space conssts of 3 octaves, each octave has scae eves. Once the Gaussan scae space has been obtaned, the DoG scae space s computed by subtactng each two consecutve mages of each octave as shown n Fgue.3. Fgue.3: Constuctng the DoG scae space fom the Gaussan scae space []. D, y, o, s L, y, o, s L, y, o, s (.) The DoG functon can be teated as an appomaton to the scae-nomazed Lapacan of Gaussan [], whch s n fact the genea famy of soutons to the dffuson equaton (.5): Page 38

48 SIFT Agothm Fgue.: The Dffeence of Gaussan Scae Space Fgue. pesents the DoG esuted fom the Gaussan scae space ustated n Fgue.. L L (.5) Thus the DoG s an appomaton to the nomazed Lapacan, whch s needed fo tue scae nvaance., y, k L, y, L L D, y, k L (.6) k Ths ndcates that the DoG-SS has scaes dffeng by a constant facto, whe t ncopoates the scae nomazaton equed fo the scae-nvaant Lapacan. Inteest ponts ae chaactezed as the Etema (Mama and Mnma) n the 3 dmensona ea functon D, y,. Fo seachng scae space Etema, each pe n the DoG mages s compaed wth the pes of a ts 6 neghbos (8 neghbos at the same scae and 9 neghbos above and 9 neghbos beow that scae) as shown n Fgue (.5). If the pe s owe/age than a ts neghbos, then t s abeed as a canddate nteest pont. Page 39

49 SIFT Agothm Fgue.5: Scae-space etema detecton []. The scae space (SS) epesentaton of an mage mmcs the vsua pecepton of the maged scene vewed at dffeent dstances, theefoe the featue ponts etacted fom the SS ae scae nvaant.... Key-Ponts Locazaton Once a key-pont canddate has been found by compang a pe to ts neghbos, the net step s to pefom a detaed ft to the neaby data fo ocaton, scae, and ato of pncpa cuvatues. Ths nfomaton aows ponts to be ejected that have ow contast (and ae theefoe senstve to nose) o ae pooy ocazed aong an edge (and ae theefoe not enough dstnctve). Each of these key ponts s eacty ocazed by fttng a 3D quadatc functon computed usng a second ode Tayo epanson aound key-pont ocaton. D D z D z T z z Dz z z z z T z T (.7) whee D and ts devatves ae evauated at the key-pont ocaton, y T the offset fom ths pont. z, and z s The ocaton of the etemum ẑ, s detemned by takng the devatve of ths functon wth espect to z and settng t to zeo, gvng: D D z ˆ (.8) z z z z The offset ẑ may be estmated usng standad dffeence appomatons fom neghbong sampe ponts n the DoG esutng n a 3 3 nea system whch may be soved effcenty. If the offset ẑ s age than.5 n any dmenson, then t means that the etemum es cose to a dffeent sampe pont. In ths case, the sampe pont s changed and the ntepoaton pefomed nstead about that pont. The fna offset ẑ s added to the ocaton of ts sampe pont to get the ntepoated estmate fo the ocaton of the etemum. The functon vaue at Page

50 SIFT Agothm the etemum, D zˆ, s usefu fo ejectng unstabe Etema wth ow contast. Ths can be obtaned by substtutng equaton (.8) nto (.7), gvng: D A Etema wth a vaue of D (.9) z z ˆ, yˆ, ˆ Dz zˆ Dz zˆ Dzˆ ess than a cetan theshod ae dscaded. In the standad SIFT method, a theshod wth a vaue between. and. was used assumng that mage pe vaues ae n the ange [,] A fna test s pefomed to emove any featues ocated on edges n the mage snce these w suffe an ambguty f used fo matchng puposes. A peak ocated on an edge n the DoG w have a age pncpe cuvatue acoss the edge and a ow pncpe cuvatue aong t wheeas a we defned peak w have a age pncpe cuvatue n both dectons. The pncpa cuvatues can be computed fom a Hessan mat, H, computed at the ocaton and scae of the key-pont: The devatves ae estmated by takng dffeences of neghbong sampe ponts. The egenvaues of H ae popotona to the pncpa cuvatues of D. Boowng fom the appoach used by Has and Stephens [3], we can avod epcty computng the egenvaues, as we ae ony concened wth the ato. Assumng that s the egenvaue wth the agest magntude and s the smae one. Then, we can compute the sum of the egenvaues fom the tace of H and the poduct fom the detemnant as epaned n the foowng equatons: T D Dy H (.) Dy Dyy T Det H D D yy H D D yy D y (.) In the unkey event that the detemnant s negatve, the cuvatues have dffeent sgns so the pont s dscaded as not beng an etemum. Let be the ato between the agest magntude egenvaue and the smae one, so that Then: T Det H H (.) The quantty s at a mnmum when the two egenvaues ae equa and t nceases wth. Theefoe, to check that the ato of pncpa cuvatues s beow a cetan theshod, we ony need to check: Page

51 SIFT Agothm H H T Det (.3) whch s vey effcent to compute. The standad SIFT method use a vaue of, whch emnates key-ponts that have a ato between the pncpa cuvatues geate than...3. Oentaton Assgnment An oentaton s assgned to each nteest pont that combned wth the scae povdes a scae and otaton nvaant coodnate system fo the descpto. Oentaton s detemned by budng a hstogam of gadent oentatons fom the key-pont neghbohood. Fo each pe n a cetan egon R aound the key-pont ocaton, the fst ode gadents ae cacuated. The pe dffeence appomatons ae used to deve the coespondng gadent accodng to the foowng equatons:, y, L, y,, y, L, y, g L (.) g L y whee L, y, s the gey vaue of the pe y P, n the mage bued by a Gaussan kene whose sze s detemned by the scae of the keypont. The gadent magntude and oentaton fo each pe ae computed espectvey as foows: m, y g g y, y actng g y (.5) Fom gadent data (magntudes and oentatons) of pes wthn the egon R, a 36-bn oentaton hstogam s constucted coveng the ange of oentatons [-8, 8 ] (each bn coves ). The gadent oentaton detemnes whch bn n the hstogam shoud be used fo each pe. The vaue added to the bn s then gven by the gadent magntude weghted by a Gaussan-weghted ccua wndow wth that s.5 tmes of the scae of the key-pont centeed on the featue pont, thus mtng to oca gadent nfomaton. The hstogam s cacuated accodng to foowng fomuas: o mag whee, y, 36 and y gadent oentatons equa to O. nt, y/7 m, y m, y R m, ae gadent magntudes of pes that have dscete The oentaton of the SIFT featue s defned as the oentaton coespondng to the mamum bn of the oentaton hstogam accodng to: R (.6) Page

52 SIFT Agothm In ode to mpove the accuacy of detemnng the key-pont oentaton, a thee pont paaboa s ft to the peaks of the oentaton hstogam. o ag ma mag (.7) ma mag () ma 8 ma o () 8 Fgue.6: A 36 bns oentaton hstogam constucted usng oca mage gadent data aound keypont. If the hstogam has moe than one dstnct peak then mutpe copes of the featue ae geneated fo the decton coespondng to the hstogam mamum, and any othe decton wthn 8% of the mamum vaue. Fgue.6 epans an eampe of an oentaton hstogam fo a SIFT featue.... Key-Ponts Descpton The gadent mage patch aound key-pont s otated to agn the featue oentaton computed n the pevous secton wth the hozonta decton n ode to povde otaton nvaance. Fgue.7: SIFT descpto constucton Page 3

53 SIFT Agothm Afte that the egon aound key-pont wth sze eated to key-pont scae s seected and subdvded nto 6 squae sub-egons. Fo each sub-egon, an 8 bn oentaton hstogam s but fom pes wthn coespondng sub-egon. The weght of each pe s gven by the magntude of the gadent as we as a scae dependent Gaussan wndow centeed on the keypont. Dung the hstogam fomaton t-nea ntepoaton s used to add each vaue. Ths conssts of ntepoaton of the weght of the pe acoss the neghbong spata bns based on dstance to the bn centes as we as ntepoaton acoss the neghbong ange bns. Ths eads to educe bounday effects as sampes move between postons and oentatons. Fnay, a 6 esutng eght bn oentaton hstogams ae tansfomed nto 8-D vecto. The vecto s nomazed to unt ength to acheve the nvaance aganst umnaton changes. Fgue.7 shows the descpto geneated fom the gadent mage patch aound key-pont. Theefoe the SIFT featue conssts of fou attbutes, a ocaton P (, y) ( and y ae the coodnates of the key-pont n the mage), a scae (eve of scae space whee s the keypont), an oentaton ma and a 8-D descpto vecto V that descbes the oca mage egon aound the key-pont ocaton. Hence, SIFT featue can be wtten as F( P(, y),, ma, V )... SIFT Featue Matchng... SIFT Coespondences Seach In ode to match two mages usng SIFT agothm, SIFT featues w be etacted fom mages and stoed nto featue sets, then the coespondng featues ae found usng a neaest neghbo seach (NNS) method that s abe to detect the smates between SIFT descptos. The smaty measue between two SIFT featues s defned by the Eucdean dstance between ts descbng 8-vectos. Essentay each featue q F fom the quey mage s compaed to a the featues q t test mage by computng the Eucdean dstances F F j d,. k 8 q t q t F, F d d j k j j t F j n the The featue pas wth the smaest Eucdan dstances ae consdeed as possbe postve matches. Howeve, many featues fom the test mage w not have any coespondng featue n the quey mage because they ase pobaby fom backgound cutte o ae not detected n the quey mage. Theefoe, t s necessay to have a stategy to dscad msmatches. A goba theshod stategy on dstance to the cosest featue does not pefom we snce some descptos ae much moe dscmnatve than othes. Lowe poposed [] a stategy (caed Neaest Neghbo Dstance Rato (NNDR)) to dscad msmatches. In ths stategy, fo each featue fom the quey mage, the Eucdan dstances to the neaest and net neaest neghbo featues of the test mage, ae compaed. If the ato between the neaest and the second neaest dstances s beow a cetan theshod, then the match s consdeed as coect. Ths appoach povdes eabe featue matchng because the coect matches need to have the cosest neghbo sgnfcanty cose than the cosest k k d (.8) Page

54 SIFT Agothm ncoect one. Fo fase matches, t s moe key that the dstances to the neaest and net neaest neghbos ae sma to each othe due to the hgh dmensonaty of the featue space. The ehaustve seach fo the neaest neghbo s computatonay epensve when the featue ength and the numbe of featues ae age. The computatona epensve pobem can be soved by epacng the ehaustve seach by Appomate Neaest Neghbo (ANN) seach agothms. The most wdey used agothm fo ANN seach s the kd-tee [38,3], whch successfuy woks n ow dmensona seach space, but pefoms pooy when featue dmensonaty nceases. kd-tee agothm povdes no speedup ove ehaustve seach fo moe than about dmensona spaces. In [] Lowe used the Best-Bn-Fst (BBF) method, whch s epanded fom kd-tee by modfcaton of the seach odeng so that bns n featue space ae seached n the ode of the cosest dstance fom the quey featue and stoppng seach afte checkng the fst neaest-neghbos. Fo a database of, SIFT featues, the BBF povdes a speedup facto of tmes faste than ehaustve seach whe osng about 5% of coect matches.... Msmatches Dscadng Msmatches aways occu when featues ae matched. A set of matches between two mages ae fequenty used to cacuate geometca tansfomaton modes ke affne tansfomaton, homogaphy o the fundamenta mat. The geometca tansfomaton mode s used to dscad msmatches that do not ft t. Thee ae many agothms that have demonstated good pefomance n mode fttng, some of them ae the Least Medan of Squaes (LMeds) [] and Random Sampe Consensus (RANSAC) agothm [5]. Both ae andomzed agothms and ae abe to cope wth a age popoton of outes. Lowe [] used Hough Tansfom to custe eabe mode hypotheses to seach fo keys that agee upon a patcua mode pose. Hough tansfom dentfes custes of featues wth a consstent ntepetaton by usng each featue to vote fo a object poses that ae consstent wth the featue. The 6 DoF object pose can be appomated by an affne tansfom wth ony paametes. Theefoe, Lowe used boad bn szes of 3 degees fo oentaton, a facto of fo scae, and.5 tmes the mamum pojected tanng mage dmenson (usng the pedcted scae) fo ocaton. Each dentfed custe wth at east 3 matches s then subject to a vefcaton pocedue n whch a nea east squaes souton s pefomed fo the paametes of the affne tansfomaton eatng the mode to the mage. The affne tansfomaton of a mode pont y T to an mage pont u v T can be wtten as: u m t m (.9) v m m y t y whee the mode tansaton s t T t y and the affne otaton, scae, and stetch ae epesented by the paametes m, m, m and m. To sove fo the tansfomaton paametes the equaton above can be ewtten to gathe the unknowns nto a coumn vecto. Page 5

55 SIFT Agothm Page 6 Ths equaton shows 3 matches whch at east needed to povde a souton, but any numbe of futhe matches can be added, wth each match contbutng two moe ows to the fst and ast mat. Equaton (.) can be wtten n the shothand fom as: whee A s a known m-by-n mat (usuay wth m > n), s an unknown n-dmensona paamete vecto, and B s a known m-dmensona measuement vecto. The souton of the system of nea equatons s gven by the pseudo nvese of the mat A: whch mnmzes the sum of the squaes of the dstances fom the pojected mode ocatons to the coespondng mage ocatons. Outes can now be emoved by checkng fo ageement between each mage featue and the mode, gven the paamete souton. Gven the nea east squaes souton, each match s equed to agee wthn haf the eo ange that was used fo the paametes n the Hough tansfom bns. As outes ae dscaded, the nea east squaes souton s e-soved wth the emanng ponts, and the pocess teated. If fewe than 3 ponts eman afte dscadng outes, then the match s ejected. In addton, a top-down matchng phase s used to add any futhe matches that agee wth the pojected mode poston, whch may have been mssed fom the Hough tansfom bn due to the affne tansfom appomaton o othe eos v u v u v u t t m m m m y y y y y y y (.) B A (.) B A A A T T. (.)

56 Fast SIFT Featue Matchng 5. Fast SIFT Featue Matchng 5.. Intoducton Matchng a gven mage wth one o many othes s a key task n many compute vson appcatons such as object ecognton, mages sttchng and 3D steeo econstucton. These appcatons eque often ea-tme pefomance. The matchng s usuay done by detectng and descbng key ponts n the mages then appyng a matchng agothm to seach fo coespondences. Cassc key-pont detectos such as Dffeence of Gaussans (DoG) [], Has Lapacan [6], Lapacan of Gaussans (LoG) [7], Dffeence of Means (DoM) [6] and the Has cone detecto [3] use smpe attbutes ke bob-ke shapes o cones. Fo the key-pont descpton a vaety of key-pont descptos have been poposed such as the Scae Invaant Featue Tansfom (SIFT) [], Speeded Up Robust Featues (SURF) [6] and Gadent Locaton and Oentaton Hstogam (GLOH) [8]. To obusty match the mages, pont-to-pont coespondences ae detemned usng smaty measue fo Neaest Neghbou (NN) seach such as Mahaanobs o Eucdean dstance. Afte that, the RANdom Sampe Consensus (RANSAC) method [5] s apped to the postve coespondences set to estmate the coect coespondences (nes). The combnaton of the DoG detecto and SIFT descpto poposed n [] s cuenty the most wdey used n compute vson appcatons due to the fact that SIFT featues ae hghy dstnctve, and nvaant to scae, otaton and umnaton changes. In addton, SIFT featues ae eatvey easy to etact and to match aganst a age database of oca featues. Howeve, the man dawback of SIFT s that the computatona compety of the agothm nceases apdy wth the numbe of key-ponts, especay at the matchng step due to the hgh dmensonaty of the SIFT featue descpto. In ode to ovecome the man SIFT dawback, vaous modfcatons of the SIFT agothm have been poposed. In genea, the stateges deang wth the acceeaton of SIFT featues matchng can be cassfed nto thee dffeent categoes: educng the descpto dmensonaty, paaezaton and epotng the powe of hadwae (GPUs, FGPAs o mutcoe systems) and Appomate Neaest Neghbo (ANN) seachng methods. Ke and Thanka [5] apped Pncpa Components Anayss (PCA) to the SIFT descpto. The PCA-SIFT educes the SIFT featue descpto dmensonaty fom 8 to 36, so that the PCA-SIFT s fast fo matchng, but seems to be ess dstnctve than the ogna SIFT as demonstated n a compaatve study by Mkoajczyk et a. [8]. In [6] Bay et a. deveoped the Speeded Up Robust Featue (SURF) method that s a modfcaton of the SIFT method amng at bette un tme pefomance of featues detecton and matchng. Ths s acheved by two majo modfcatons. In the fst one, the Dffeence of Gaussan (DoG) fte s epaced by a Dffeence of Means (DoM) fte. The use of the DoM fte speeds up the computaton of featues detecton due to the epotng ntega mages fo a DoM mpementaton. The second modfcaton s the educton of the mage featue vecto ength to haf the sze of the SIFT featue descpto ength (fom 8 components down to 6), whch enabes qucke featues matchng. These modfcatons esut n an ncease computaton speed by a facto 3 Page 7

57 Fast SIFT Featue Matchng compaed to the ogna SIFT method. Howeve, ths s nsuffcent fo ea-tme equements. Addtonay, n contast to SIFT, SURF does not povde the numbe of coespondences whch ae equed fo some compute vson appcatons such as pose estmaton and 3D econstucton [8]. In ecent yeas, sevea papes [9,5] wee pubshed addessng the use of the paaesm of moden gaphcs hadwae (GPU) to acceeate some pats of the SIFT agothm, focused on featues detecton and descpton steps. In [5] GPU powe was epoted to acceeate featues matchng. These GPU-SIFT appoaches povde to tmes faste pocessng aowng ea-tme appcaton. Othe papes such as [5] addessed mpementaton of SIFT on a Fed Pogammabe Gate Aay (FPGA) acheved about tmes faste pocessng. Zhan et a. [53] pesented that SIFT featues etacton ate can be nceased by a facto of 6.7 by paaezng t on an 8-coe system, o by a facto 5 on a 3-coe chp mutpocesso (CMP) smuato. The matchng step can be speeded up by seachng fo the Appomate Neaest Neghbo (ANN) nstead of the eact neaest neghbo. The most wdey used agothm fo ANN seach s the kd-tee [5], whch successfuy woks n ow dmensona seach space, but pefoms pooy when featue dmensonaty nceases. In [] Lowe used the Best-Bn-Fst (BBF) method, whch s epanded fom kd-tee by modfcaton of the seach odeng so that bns n featue space ae seached n the ode of the cosest dstance fom the quey featue and stoppng seach afte checkng the fst neaest-neghbo canddates. The BBF povdes a speedup facto of tmes faste than ehaustve seach whe osng about 5% of coect matches. Spa-Anan et a. [55] poposed an mpoved veson of the kd-tee agothm n whch mutpe andomzed kd-tees ae ceated. In contast to ogna kd-tee agothm whch spts the data n haf at each eve of the tee on the dmenson fo whch the data has the geatest vaance, n mpoved veson the andomzed tees ae constucted by seectng the spt dmenson andomy fom among a few dmensons n whch the data has hgh vaance. In [] Gons et a poposed the Locaty Senstve Hashng (LSH) method, whch hashes featues usng sevea hash functons nto subsets (so caed buckets). The man dea s to ensue the coson of sma featues wth hgh pobabty. Lke KD-tees, LSH aso has a pobem when deang wth vey hgh dmensona data. In [56] Heng Yang et a poposed the Randomzed Sub-Vecto Hashng (RSVH) agothm fo hgh-dmensona featue matchng. The essenta dea of RSVH s that two featue vectos ae consdeed sma when the L noms of the coespondng andomzed sub-vectos ae appomatey same. RSVH can be eecuted aveagey about tmes faste than ehaustve seach fo databases of few ten thousands of SIFT featues. In [57] Eduado Vae et a. ntoduced mut-cuves scheme fo ndeng hgh dmensona featues to pefom ANN seach wth good compomse between pecson and speed. Ths technque s an mpovement to the space- fng cuves method amng at esove the bounday effects pobem. In [58] Mchae E. Houe et a. ntoduced a pactca nde fo appomate smaty quees of age mut-dmensona data sets, caed the Spata Appomaton Sampe Heachy (SASH), whch s a mut-eve stuctue of andom sampes, ecusvey constucted by budng a SASH on a age andomy seected sampe of data objects, and then connectng each emanng object to sevea of the appomate neaest neghbos fom wthn the sampe. Quees ae pocessed by fst ocatng appomate neghbos wthn the sampe, and then usng the pe-estabshed connectons to dscove neghbos wthn the emande of the data set. In [59] Muja and Lowe compaed Page 8

58 Fast SIFT Featue Matchng many dffeent agothms fo appomate neaest neghbo seach on datasets wth a wde ange of dmensonaty and they found that two agothms obtaned the best pefomance, dependng on the dataset and the desed pecson. These agothms used ethe the heachca k-means tee (HKMT) o mutpe andomzed kd-tees (MRKDTs). ANN seach agothms ae usuay based on constuctng a mut-ce data stuctue (eg. tee, hash tabe,..) n whch featues ae estoed, and then appyng a seach pocedue among the ces of ths data stuctue to answe a quey, whch eques not ony matchng tme but aso bud tme and an addtona memoy usage. Theefoe ANN agothms ae especay sutabe fo neaest neghbo seachng n age databases, snce they need offne tanng and compe data stuctues. In ths Chapte, a nove stategy whch s dstncty dffeent fom a thee of the above mentoned stateges, s ntoduced to acceeate the SIFT featues matchng step. The contbuton s summazed n two ponts. Fsty, n the key-pont detecton stage, the SIFT featues ae spt nto two types, Mama and Mnma, wthout eta computatona cost and at the matchng stage ony featues of the same type ae compaed. The dea behnd ths s that no match can be epected between two featues of dffeent types. Secondy, SIFT featue s etended by few new anges wthout eta computatona cost. These anges ae computed fom oentaton hstogam (OH) and/o sub-oentaton hstogams (SOHs) of the SIFT descpto.(sift-d). Hence SIFT featues ae dvded nto a few custes based on the anges and, at the matchng stage, ony featues that have amost the same anges ae compaed snce no match can be epected between two featues whose anges dffe fom each othe fo moe than a pe-defned theshod. In compason to the ogna SIFT method, whee ehaustve seach s used fo matchng, the poposed modfcatons aow moe than tmes faste pocessng n the matchng step wthout osng a notceabe poton of coect matches. In contast to ANN seach agothms, poposed stategy eques nethe bud tme no memoy ovehead, theefoe t s sutabe fo a appcatons, especay when onne matchng s equed. The poposed method can be geneazed fo a oca featue-based matchng agothms whch detect two o moe types of key-ponts (e.g. DoG, LoG, DoM) and whose descptos ae otaton nvaant, whee few dffeent oentatons can be assgned (e.g. SIFT, SURF, GLOH). Futhemoe, the pesented stategy can be combned wth othe above mentoned stateges to each a hghe facto of featues matchng speedup. Snce the poposed stategy s many based on the statstca dstbutons of ccua andom vaabes (anges), we fst gve a bef evew of the statstca anayss of ccua andom vaabes. 5.. Ccua Random Vaabes Ccua vaabes [6, 6] take vaues on the ccumfeence of a cce.e. they ae anges n the ange, adans. Many envonmenta data ae ccua n natue such as wnd decton, compass beang, cock and othes. To anayze ths type of data, t s needed to use technques dffeng fom those of the usua Eucdean type vaabes because the ccumfeence s a bounded cosed space, fo whch the concept of ogn s abtay o undefned. Thus, the technques that have been used fo contnuous nea data do not wok wth ccua vaabes because they assume that vaabes ae nea (the owest vaue s Page 9

59 Fast SIFT Featue Matchng fathest fom the hghest vaue). Theefoe, to anayze ccua vaabes, an ente fed of ccua statstcs has been deveoped. In ccua statstcs, each datum s defned by ts ength and ts ange fom a chosen pont on the cce. Ccua statstcs ncude tests of unfom decton aound the cce, confdence ntevas, ccua pobabty densty functons, coeatons, and egesson, among othes. In the foowng we w study the pobabty densty functon of the sum/ dffeence of two o moe ndependent ccua andom vaabes (ICRVs) PDF of Sum/Dffeence of Unfomy-Dstbuted ICRVs Fom the pobabty theoy, t s known that the pobabty densty functon g of the sum of two ndependent andom vaabes X and functon functons: g and X, each of whch has a pobabty densty g espectvey, s the convouton of the ndvdua densty g g d g g g * (5.) If X and X ae unfomy dstbuted n the nteva,, then the PDF of the sum X X X s tangua-dstbuted n the nteva, because the convouton of two ectangua functons s tangua. If X and X ae ccua vaabes wth peod, then the sum s aso peodc wth the same peod. Hence the eft pat of the PDF of the sum n the nteva, can be shfted to ght by the peod and summed to the ght pat n the nteva, to poduce the tota PDF of the sum X X X. Theefoe the sum of two ndependent unfomy-dstbuted ccua andom vaabes s aso unfomy-dstbuted. Ths outcome s gaphcay ustated by Fgue 5.. The same esut s aso vad fo the dffeence because the dffeence between two vaues can be epessed as the sum of the fst one and the negatve of the second: X X X X To pove ths, t s suffcent to pove that the PDF of X s equa to the PDF of X Because X s peodc wth peod, ts PDF g( ) can be shft to ght by ts peod. (5.) g ( ) g ( ) g ( ) (5.3) whch eads to the fact that the PDF of X s equa to the PDF of X. Hence the PDF of the sum/ dffeence of two ndependent unfomy dstbuted ccua vaabes s unfomy-dstbuted. The same esut can be easy geneazed to any numbe of ndependent unfomy ccua andom vaabes. Page 5

60 Fast SIFT Featue Matchng (a) (b) PDF of X PDF of X (c) PDF of X X (d) PDF of X X Fgue 5.: The ccua pobabty densty functon of the sum of two ndependent unfomy dstbuted ccua andom vaabes PDF of Sum/Dffeence of ICRVs The esut poven n the above can be poven even ony one of two ndependent ccua andom vaabes X and X s unfom. g(+6) g(+) g(+) / g() g(-) - Fgue 5.: wappng the g aound the ccumfeence of a cce of unt adus Fo eampe, f ony X s unfomy dstbuted n the nteva,, wheeas the othe X s abtay dstbuted n the same nteva, then the sum/dffeence of these two andom vaabes X X X s unfomy dstbuted n the same nteva. To pove ths, t s assumed that the pobabty densty functons of X and X on the ea ne ae g espectvey. g and - Page 5

61 Fast SIFT Featue Matchng Page 5 The pobabty densty functon of the sum s the convouton of the ndvdua PDFs Tansfomng the equatons (5.) nto the Lapace space yeds: Because the convouton of two functons n ea space s equvaent to the poduct of the Lapace tansfoms n Lapace space, the equaton (5.5) n Lapace space s epessed as: The PDF of the sum X X X on the ea ne g s obtaned by nvetng the Lapacespace epesson (5.7) back to ea space: othewse h g othewse g (5.) g g g (5.5) s G g e s s G g s s j s (5.6) s j e s G s G s s G s G s G (5.7) d g d g g s G d g d g d g d g d g g d g d g g (5.8)

62 Fast SIFT Featue Matchng Page 53 The ccua andom vaabes coespondng to X, s defned by. The pobabty densty functon f of s obtaned by wappng g aound the ccumfeence of a cce of unt adus [9]: Fo the nteva,, k, hence Fom equaton (5.8) yeds: By substtutng (5.8) and (5.) n (5.), yeds: Equaton (5.3) means that the pobabty densty functon of the sum/dffeence of two ndependent ccua andom vaabes, one of them s unfomy dstbuted n the nteva,, s unfomy dstbuted n the same nteva. Ths esut can be aso geneazed fo any numbe of ndependent ccua andom vaabes at east one of them s unfomydstbuted. mod X (5.9) k k g f (5.) g g f (5.) d g d g g d g d g g (5.) d g d g f (5.3)

63 Fast SIFT Featue Matchng 5.3. Spt SIFT Featue Matchng As sad n Chapte, the SIFT featue ocatons ae detected as the Etema of the scae space. Etema can be Mnma o Mama so that thee ae two types of SIFT featues, Mama and Mnma SIFT featues [63, 6] Though etacton of SIFT featues fom 6 dffeent mages of standad dataset [65], t was found that the numbe of Mama s amost equa to the numbe of Mnma SIFT featues etacted fom the same mage. Theefoe, when matchng ony Mama wth Mama and Mnma wth Mnma, the matchng tme s educed by 5% wth espect to the ehaustve seach wthout osng any coect matches because no coect match can be epected between two featues of dffeent types. The cam that thee ae no coect matches between Mnma and Mama SIFT featues s epementay suppoted. Namey, t was found that the featues of each coect match ae aways fom the same type. Fgue 5.3 pesents the Mama and the Mnma SIFT featues etacted fom the same mage. It can be seen fom Fgue 5.3 that the Mama SIFT featue ocatons ae the centes of dak bobs on the ght backgound and vce vesa fo the Mnma ocatons. Mama SIFT featues Mnma SIFT featues Fgue 5.3: the Mama and Mnma SIFT featues etacted fom the same mage. To decae the matchng tme educton by spttng the SIFT featues; t s assumed that the numbe of featues etacted fom the ght and the eft mage ae epessed as: ma ma mn mn (5.) whee ma ( ma ) and mn ( mn ) ae the numbes of Mama and Mnma SIFT featues espectvey. The matchng tme wthout egad to the type of featues, aso the tme of ehaustve seach s popotona to: Page 5

64 Fast SIFT Featue Matchng T eh (5.5) The matchng tme, n the case of compason of ony featues of the same type, s popotona to the foowng sum: T spt (5.6) ma ma mn mn Because the numbe of Mnma SIFT featues s amost equa to the numbe of Mama SIFT featues etacted fom the same mage: ma ma mn mn (5.7) By substtutng (5.7) n (5.6) one obtans: T spt Teh (5.8) whch means that the matchng tme s deceased by 5% n espect to ehaustve seach. To get ths matchng tme educton, t s suffcent that at east one of the two featue sets meets the assumpton that the numbe of Mama s amost equa to the numbe of Mnma. Fo eampe, f a SIFT featues of set R ae Mama ma, then they ae compaed ony wth the Mama-SIFT featues of the set L. Hence the equaton (5.8) becomes: T spt Teh ma (5.9) Theefoe, n the case of matchng a quey mage aganst a age database, thee ae no necessty to spt SIFT featues of the quey mage. In ode to eamne ths esut epementay, pas of steeo mages ae matched usng SIFT method wth and wthout spttng SIFT featues. Some esuts ae sted n Tabe 5. The test mages ae acqued fom wokng envonment of the obotc system FRIEND II wth ts steeo camea system (A Bumbebee steeo camea wth the esouton of. X768 pes) Tabe 5.: Compason between Standad and Spt SIFT Featue matchng N. of key-ponts Standad SIFT Featue Matchng Spt SIFT Featue Matchng Left mage Rght mage Matchng tme (sec) N. of nes Matchng tme (sec) N. of nes 65 73,686 37, ,79 6, ,76 5, Page 55

65 Fast SIFT Featue Matchng 67 6,8 5, As evdent fom Tabe 5., by spttng SIFT featues, not ony the matchng tme s educed to 5% but aso the numbe of nes (coect matches) s nceased, whch means that the matchng quaty s aso enhanced by Spt SIFT featue matchng. 5.. Etended SIFT Featue Geneay, f a scene s captued by two cameas o by one camea but fom two dffeent vewponts, the coespondng ponts, whch epesent mages of the same 3D pont, w have dffeent mage coodnates, dffeent scaes, and dffeent oentatons, though, they must have amost sma descptos that ae used to match the mages usng a smaty measues. Howeve, the hgh dmensonaty of the SIFT descpto V makes the featue matchng vey tme-consumng. In ode to speed up the featues matchng, t s assumed that two ndependent oentatons can be assgned to each featue so that the ange between them stays amost unchanged fo a coect coespondng featues even n the case of the mages captued unde dffeent condtons such as vewng geomety and umnaton changes. The dea of usng an ange between two ndependent oentatons s amed to avod compang of a geat poton of featues that can not be matched n any way. Ths eads to a sgnfcant acceeaton of the matchng step. Hence, the eason fo poposng SIFT featue ange s twofod. On the one hand, to fte the coect matches, so that a coect match M j can be estabshed between two featues F and dffeence between the anges F j, whch beong espectvey to mages and, f and ony f the and j s ess than a peset theshod vaue : On the othe hand, the eason fo poposng SIFT featue ange s to acceeate the SIFT featue matchng because thee s no necessty to compae two featues f the dffeence between the anges s age than the peset theshod Matchng Speeded-Up Facto Assumng that two mages to be matched whose featue anges and j ae consdeed as andom vaabes and espectvey. In the case of coect matches the andom vaabes and ae dependent on each othe snce the ange dffeences of coect matches ae equa to zeo whch coespond to the dea mage matchng case. In contast, the andom vaabes and ae ndependent of each othe fo ncoect matches whe the ange dffeences of ncoect matches ae somehow dstbuted n the ange,. Theefoe, the dffeence fo the ncoect matches has a pobabty densty functon (PDF) dstbuted ove the whoe ange ange,, wheeas the PDF of fo the coect matches s concentated n the so-caed ange of coect matches, whch s the naow ange about. Geneay, f the andom vaabes and ae ndependent and at east one of them s unfomy dstbuted n the ange,, the dffeence has an unfom PDF as t has been poven n Secton 5... j (5.) Page 56

66 Fast SIFT Featue Matchng If a matchng pocedue, whch compaes ony the featues havng ange dffeences n the ange of coect matches, s used n the case of unfom dstbuton of fo ncoect matches, then the matchng pocess s acceeated by a speed-up facto SF whch can be epessed as the ato between the wdth of the whoe ange ange w tota 36 and the wdth of the ange of coect matches w co. SF w w tota co 36 w co (5.) 5... SIFT Featue Ange It s suggested hee that a SIFT featue s etended wth an ange that meets the foowng condtons: - The ange has to be nvaant to the geometc and the photometc tansfomatons (the nvaance condton). - The ange has to be unfomy dstbuted n the ange condton)., (the equay key To assgn an ange to the SIFT featue, two oentatons ae equed. The nvaance condton s guaanteed ony f these oentatons ae dffeent, wheeas, as epaned n above Secton, the equay key condton s guaanteed f the oentatons ae ndependent and at east one of them s unfomy dstbuted n the ange, As mentoned n Chapte, the ogna SIFT featue has aeady an oentaton ma. Theefoe, t s ony necessay to defne a dffeent oentaton ndependent fom ma. Fsty, the oentaton sum coespondng to the vecto sum of a oentaton hstogam bns s consdeed and the dffeence between the suggested oentaton and the ogna SIFT featue oentaton sum sum ma s assgned to the SIFT featue as the SIFT featue ange sum. Fgue 5. pesents geometcay the vecto sum of an eght bns oentaton hstogam fo the sake of smpcty, wheeas the used oentaton hstogam has 36 bns fo the case of the ogna SIFT. Hence, mathematcay, the poposed oentaton sum s cacuated accodng to the foowng equaton: 8 mag sno 7 sum actan 8 (5.) mag coso 7 whee mag and o ae the amptude and the oentaton of the th bn of the oentaton hstogam Snce sum s dffeent fom ma and both ae cacuated fom the oentaton hstogam, then sum meets the nvaance condton. Page 57

67 Fast SIFT Featue Matchng sum ma sum Fgue 5.: The vecto sum of the bns of an eght oentaton hstogam. To eamne whethe sum meets the equay key condton, t s consdeed as a andom vaabe sum. The pobabty densty functon (PDF) of sum s estmated usng 6 SIFT featues etacted fom 7 dffeent mages ( 5 benchmak mages [65] and steeo mages fom a ea-wod obotc appcaton). Some eampes of used mages ae gven n Secton 5... sum tan, tan, tan, tan,3 tan, Pobabty(%) Ange(degee) Fgue 5.5: The epementa PDFs of mages. sum and tan, k fo SIFT featues etacted fom 7 test The PDF of sum was computed by dvdng the ange space [-8,8 ] nto 36 sub-anges, whee each sub-ange coves, and by countng the numbes of SIFT featues whose Page 58

68 Fast SIFT Featue Matchng anges sum beong to each sub-ange. Fo eampe, an estmate of the pobabty that a featue has an ange sum n the sub-ange [, + ) s : N sum, p sum, (5.3) N whee N sum, s the numbe of SIFT featues havng the ange n the consdeed subange and N tota s the tota numbe of featues 6 etacted fom 7 test mages n pefomed epements. As evdent fom Fgue 5.5, about 6% of SIFT featues have anges fang n the ange [- 3,3 ]. The eason of ths outcome s the hgh dependency between ma and sum due to the fact that the sum s defned as the vecto sum of a oentaton hstogam bns ncudng the bn whch coesponds to ma. The ma s the domnant oentaton n the patch aound the key-pont so that t has domnant nfuence to the sum. Due to the hgh dependency between ma and sum, sum does not meet the equay key condton, hene t can not povde the optmum speed up facto. To defne an appopate SIFT featue ange, oentatons ae futhe suggested to be tan, k consdeed as ndependent fom ma. These oentatons ae computed as the vecto sums of a oentaton hstogam bns ecudng the mamum bn and k of ts neghbo bns at the eft and at the ght sde as foows: tota... tan, tan, tan, actan actan actan 8 7 m 8 7 m 8 7 m, m 8 mag mag 7 m, m 8 7 m, m 8 7 m, m mag mag mag mag sn o cos o sn o cos o sn o cos o (5.) Page 59

69 whee. m ag ma mag. Fast SIFT Featue Matchng The PDFs of the andom vaabes coespondng to anges tan, k,, ae tan k tan k ma estmated n the same manne as the PDF of sum, pefomng the epements ove 6 SIFT featues etacted fom 7 test mages. The measued PDFs of tan, k (fo k,,,3,.) ae shown n Fgue 5.5. It s evdent fom Fgue 5.5 that the tan, has a PDF that s the cosest match to the unfom dstbuton. Theefoe, the ange tan, meets the both condtons, nvaance and equay key condtons, and t can be consdeed as a new attbute of the SIFT featue, that s F, y,, ma, V,.. Wth ths etenson the SIFT featue becomes tan, Etended SIFT Featues Matchng Assumng that two sets of etended SIFT featues R F :,,..., L F j :,,..., possbe M F, F matches s equa to j j, contanng espectvey and featues, ae gven, The numbe of and. Among these possbe matches a sma numbe of coect matches may est, whch ae detemned by Eucdan dstance between featue descptos foowed by the RANSAC method [5] to keep ony nes. A set of SIFT featue ange dffeences j j : j,,..., can be estabshed fom the anges :,,..., and j : j,,..., of the etended SIFT featues of the gven sets R and L. Consdeng the ange dffeences j as a andom vaabe j, the PDFs of j fo both coect and ncoect matches ae measued n epements ove consdeed 7 mages. The measued PDFs ae shown n Fgue 5.6. It can be seen fom Fgue 5.6 that about 98 % of coect and ony % of ncoect matches beong to [-, ]. Theefoe, n ode to fnd coect matches t s needed to teat ony % of possbe matches whch can speed up the featues matchng sgnfcanty. Page 6

70 Fast SIFT Featue Matchng PDF of fo ncoect matches 8 PDF of fo coect matches Pobabty (%) Ange (Degee) Fgue 5.6: The epementa PDF of the ange dffeence j fo ncoect and coect matches. To epot ths outcome, SIFT featues ae dvded nto sevea subsets based on the anges. The SIFT featues of each subset ae compaed ony wth the featues of some subsets, so that the esutng coespondences must have absoute dffeences of anges ess than a pe-set theshod. Hee a theshod of s seected because amost a coect matches have ange dffeences n the ange [-, ] as ustated n Fgue 5.6. Consde that each of the sets of featues R and L ae dvded nto b subsets, so that the fst subset contans ony the SIFT featues whose anges beong to 8, 8 36 b and th the subset contans featues whose anges beong to th 8 36 b, 8 36 b. Consequenty, the b subset contans featues 8 36 b b, 8. whose anges beong to The numbe of featues of both sets can be epessed as: b b (5.5) Because of the eveny dstbuton of featue anges ove the ange of the anges [- 8,8 ] as shown n Fgue 5.5, the featues ae amost equay dvded nto sevea subsets. Theefoe, t can be asseted that the featue numbes of each subset ae amost equa to each othe b b b b (5.6) Page 6

71 Fast SIFT Featue Matchng To ecude matchng of featues that have dffeences of anges outsde the ange a, a, each subset s matched to ts coespondng one and to n neghbong subsets to the eft and to the ght sde. In ths case the matchng tme s popotona to the foowng tem: T T etended etended b n b n j j n b jn (5.7) n b n b b Theefoe, the acheved speedup facto wth espect to ehaustve seach s equa to: b SF etended (5.8) n The eaton between n, a and b s as foows: 36 ab a n (5.9) b 36 whee epesents the fst ntege vaue age than o equa to. Substtutng equaton (5.8) nto equaton (5.7) yeds: 36 SF etended (5.3) a The matchng pocedue s ustated n Fgue 5.7 fo the case of compason of featues wth the anges fom few anges. Fo eampe, featues wth the anges n the ange of, 36 b, whch ae etacted fom the fst mage, ae compaed ony wth the featues etacted fom the second mage that have anges n the ange of n 36 b, n 36 b. It s mpotant to ndcate that the achevng of the above speedup facto eques the unfom dstbuton of SIFT featues based on the anges ony fo one of the featue sets. Fo eampe, f a SIFT featues of the set R fas n the nteva 8 36 b, 8 36 b the numbe of featues s. In ths case a SIFT featues of set R ae compaed ony wth the SIFT featues of the set L that fa n the coespondng nteva and ts specfed neghbos. Hence the equaton (5.7) becomes: T T etended etended n j jn b n b n jn (5.3) Page 6

72 Fast SIFT Featue Matchng Theefoe, n the case of matchng a quey mage aganst a age database, thee ae no necessty to spt SIFT featues of the quey mage based on the anges. In addton the assumpton that the SIFT featues of the database ae unfomy dstbuted based on the anges n the ange [-8, 8 ] s vad wth a hgh pobabty. mp mp... mpb (5.3) z z z b whee z s the sze of the database and subset. p s the pobabty that a featue beongs to the th n b, n b n b, n b n b, n b b,, b b, b b,, b b, b n b, n b n b, n b n b,n b Fgue 5.7: Etended SIFT featue matchng pocedue The esut (5.9) means that f t s amed to ecude matchng of featues that have ange dffeences outsde the ange [-, ], then the matchng step s acceeated by a facto 9. When ths modfcaton of ogna SIFT featue matchng s combned wth the spt SIFT featues matchng, the obtaned speedup facto s 8 wthout osng a notabe poton of coect matches. Ths s ustated wth the epementa esuts pesented n the net Secton. Fgue 5.8 pesents the coespondence SIFT featues etacted fom two mages of the same scene maged fom two dffeent vewponts. SIFT featue ae epesented by cooed cces (bue fo Mama and ed fo Mnma) wth adus popotona to the featue scae. Featue Page 63

73 Fast SIFT Featue Matchng ange s epesented by two dectons. It can be seen fom Fgue 5.8 that coespondence SIFT featues ae aways fom the same type and have amost the same anges.. Fgue 5.8: Matchng esut between two mages of the same scene maged fom two dffeent vewponts Epementa Resuts The poposed method fo speedng up featue matchng based on spt and etended SIFT featues was tested usng both a standad mage dataset, and ea wod steeo mages. (a) (b) (c) (d) Fgue 5.9: Some of the standad dataset mages of scenes captued unde dffeent condtons: (a) vewpont, (b) ght changes, (c) zoom, (d) otaton. Page 6

74 Fast SIFT Featue Matchng The used mage dataset [65] conssts of about 5 mages of 3 dffeent scenes. Each scene s epesented wth a numbe of mages taken unde dffeent photometc and geometc condtons. Some eampes of the mages used n the epements, whose esuts ae pesented hee, ae gven n Fgue 5.9. Steeo mages wee gabbed by the steeo camea system of the ehabtaton obotc system FRIEND (Functona Robot am wth fiendy nteface fo Dsabed peope) [9]. FRIEND s ntended to suppot the use n day fe actvtes whch demand object manpuaton such as sevng a dnk and pepang and sevng a mea. The cuca fo autonomous object manpuaton s pecse 3D object ocazaton. The key facto fo eabe 3D econstucton of object ponts s coect matchng of coespondence ponts n steeo mages. Hence, steeo obot vson s a typca appcaton whee fast and eabe featue matchng s of utmost nteest. Some eampes of steeo mages showng FRIEND envonment n sevng a dnk obot wokng scenao ae gven n Fgue 5.. Fgue 5.: Steeo mages fom a ea-wod obotc appcaton used n the epements. In ode to evauate the effectveness of the poposed method, ts pefomance was compaed wth the pefomances of two agothms fo ANN (heachca k-means tee and andomzed kd-tees) [59]. Spt&Etended SIFT K-Means Tee Rand, KD-Tees Speedup (SF) , Pecson(%) Fgue 5.: Tade-off between matchng speedup and matchng pecson fo ea steeo mage matchng. Page 65

75 Fast SIFT Featue Matchng Compasons wee pefomed usng the Fast Lbay fo Appomate Neaest Neghbos (FLANN) [66], whch s a bay fo pefomng fast appomate neaest neghbo seachng n hgh dmensona spaces. Fo a epements, the matchng pocess s caed out unde dffeent pecson degees makng tade off between matchng speedup and matchng accuacy. The pecson degee s defned as the ato between the numbe of coect matches etuned usng the consdeed agothm and the numbe of coect matches etuned usng ehaustve seach, wheeas the speedup facto s defned as the ato between the ehaustve matchng tme and the matchng tme fo the coespondng method. Fo both ANN agothms, heachca k-means tees and andomzed kd-tees, the pecson s adjusted by the numbe of nodes to be eamned, wheeas fo the poposed Spt and Etended SIFT method, the pecson s detemned by adjustng the wdth of the ange of coect matches w co (epaned n Secton 5..). The coect matches ae detemned usng the neaest neghbo dstance ato (NNDR) matchng stategy [] wth dstance ato equa to.6, foowed by RANSAC agothm [5] to keep ony nes. Two epements wee un to evauate poposed method, on ea steeo mages and on the mages of the dataset [65]. In the fst epement, SIFT featues ae etacted fom steeo mages. Each two coespondng mages ae matched usng a thee consdeed agothms unde dffeent degees of pecson. The epementa esuts ae shown n Fgue 5.. As can be seen fom Fgue 5., the pefomance of the poposed method outpefoms both ANN agothms fo a pecsons. Fo pecson aound 99% eve, the poposed method povdes a speedup facto of about. Fo the owe pecson degee speedup facto s much hghe. As evdent fom Fgue 5. by usng poposed Spt and etended SIFT the speedup facto eatve to ehaustve seach can be nceased to 8 tmes whe st etunng 7% of the coect matches. The second epement was caed out on the mages of the dataset [65]. As sad befoe, ths dataset conssts of about 5 mages of vaous contents. These mages epesent mages of 3 dffeent scenes taken unde dffeent condtons such as otaton, zoom, ght and vewpont changes. Fo the pefomed epements the mages of dataset ae gouped accodng to these dffeent condtons nto vewpont, zoom, otaton and ght goup. Fo each goup, SIFT featues ae etacted fom each mage and pas of two coespondng mages ae matched usng heachca k-means tee, andomzed kd-tees and poposed Spt and Etended SIFT, wth dffeent degees of pecson. The epementa esuts ae shown n Fgue 5.. As evdent fom Fgue 5., poposed Spt and Etended SIFT outpefoms the both othe consdeed ANN agothms n speedng up of featues matchng fo a pecson degees. Page 66

76 Fast SIFT Featue Matchng (a) (b) , (c) , (d), , Fgue 5.: Tade-off between matchng speedup (SF) and matchng pecson fo mage goups (a) ght, (b) vewpont, (c) otaton, (d) zoom changes Vey Fast SIFT Featue Geneay, f a scene s captued by two cameas o by one camea but fom two dffeent vewponts, the coespondng ponts n two esuted mages w have dffeent mage coodnates, dffeent scaes, and dffeent oentatons. Nevetheess, they must have amost sma descptos whch ae used to match the mages usng a smaty measue [,67,68]. The hgh dmensonaty of descpto makes the featue matchng vey tme-consumng. In ode to speed up the featues matchng, t s assumed that pawse ndependent anges can be assgned to each featue. These anges ae nvaant to vewng geomety and umnaton changes. When these anges ae used fo featue matchng togethe wth SIFT-D, we can avod the compason of a geat poton of featues that can not be matched n any way. Ths eads to a sgnfcant speed up of the matchng step as w be shown beow. Page 67

77 Fast SIFT Featue Matchng SIFT Descpto Based Featue Anges In Secton 5., a speedng up of SIFT featue matchng by 8 tmes compaed to ehaustve seach was acheved by etendng SIFT featue wth one unfomy-dstbuted ange computed fom the oentaton hstogam (OH) and by spttng featues nto Mama and Mnma SIFT featues. In ths Secton the attempts to etend SIFT featue by few anges computed fom SIFT descpto (SIFT-D). As descbed n Chapte, fo computaton of SIFT-D, the nteest egon aound key-pont s subdvded n sub-egons n a ectangua gd. Fom each sub-egon a sub-oentaton hstogam (SOH) s but. Theoetcay, t s possbe to etend a SIFT featue by a numbe of anges equa to the numbe of SOHs as these anges ae to be cacuated fom SOHs. In case of gd, the numbe of anges s then 6. Howeve, to each the vey hgh speed of SIFT matchng, these anges shoud be components of a mutvaate andom vaabe that s unfomy dstbuted n the 6-dmensona space [-8, 8 ] 6. In ode to meet ths equement, the foowng two condtons must be vefed [69]: Each ange has to be unfomy dstbuted n [-8, 8 ] (equay key condton). The anges have to be pa-wse ndependent (pa-wse ndependence condton). In ths secton, the goa s to fnd a numbe of anges that ae nvaant to geometca and photometca tansfomatons and that meet the above mentoned condtons. Fst, the anges between the oentatons coespondng to the vecto sum of a bns of each SOH and the hozonta oentaton ae suggested as the SIFT featue anges. Fgue 5.3.b pesents geometcay the vecto sum of a SOH. Mathematcay, the poposed anges { j ;, j,..,} ae cacuated accodng to the foowng equaton: whee mag j (k) and o j (k) ae the amptude and the ange of the k th bn of the j th hstogam espectvey. Now, these anges must be eamned, whethe they meet the equay key and pa-wse condtons. 7 magj k sn oj k k j actan 7 (5.33) magj k cosoj k k Page 68

78 Fast SIFT Featue Matchng Bode j 3 j j Cente Fgue 5.3: (a) SOHs,(b):Vecto sum of the bns of a SOH, (c) anges computed fom SOHs Equay Lkey Condton To eamne whethe the anges j ;, j, meet the equay key condton, they ae consdeed as andom vaabes ;, j, (a) PDFs of cente SIFT descpto anges. j. 8% 3 6% 3 % % % 8% 6% % % % (b) PDFs of bode SIFT descpto anges. 8% 6% % 3 3 % % 8% 6% % % % Fgue 5.: The PDFs of anges estmated fom 6 SIFT featues etacted fom 7 mages. Page 69

79 Fast SIFT Featue Matchng The pobabty densty functon (PDF) of each ange s estmated fom 6 SIFT featues etacted fom 5 benchmak mages [65] and steeo mages fom a ea-wod obotc appcaton. The PDFs of j was computed by dvdng the ange space 8,8 nto 36 sub-anges, whee each sub-ange coves, and by countng the numbes of SIFT featues whose anges beong to each sub-ange. j As evdent fom Fgue 5., t can be dstngush between two categoes of anges based on the PDFs fom, the anges computed fom the sub-egons that ae aound the cente of SIFT featue (cente sub-egons) and the anges computed fom the sub-egons that ae yng on the SIFT descpto egon boundaes (bode sub-egons). The anges that ae computed fom the SOHs aound the cente of SIFT featue (caed cente anges), have dstbutons concentated about, wheeas the anges that ae cacuated fom the SOHs of the gd bode (caed bode anges), tend to be equay key dstbuted ove the ange ange. The eason of ths outcome can be ntepeted as foows: On the one hand, the SOHs ae computed fom the nteest egon (whee OH s computed) afte ts otaton as descbed n Chapte. Theefoe the oentatons of the mamum bn of each cente SOH tend to be equa. On the othe hand, fo each SOH, the oentaton of the mamum bn and the oentaton of the vecto sum of a bns ae stongy dependent snce the vecto sum ncudes the mamum bn that has the domnant nfuence to the vecto sum []. In the contay, the bode SOHs and the OH do not shae the same gadent data, theefoe ony bode anges meet the equay key condton. Fgue 5.3 pesents the bode and the cente anges Pa-wse Independence Condton In ode to eamne whethe suggested anges j meet the pa-wse ndependence condton, t s needed to measue the dependence between each two anges. The most fama measue of dependence between two quanttes s the Peason poduct-moment coeaton coeffcent. It s obtaned by dvdng the covaance of the two vaabes by the poduct of the standad devatons. Assumng that two andom vaabes ae gven X and Y wth epected vaues and coeffcent y and standad devatons and y between them s defned as: whee E [] s the epected vaue opeato. then the Peason poduct-moment coeaton y E X Y y y (5.3) y The coeaton coeffcents between each two anges and ae computed usng 6 SIFT featues etacted fom the consdeed test mages. Page 7

80 Fast SIFT Featue Matchng (5.35) The estmated coeaton coeffcents ae epaned n Fgue 5.5 As evdent fom Fgue 5.5, anges that ae computed fom contguous SOHs, ae hghy coeated, wheeas thee s no o vey weak coeatons between two anges that ae computed fom non-contguous SOHs. The eason of ths outcome s caused by the t-nea ntepoaton that dstbutes the gadent sampes ove contguous SOHs. In othe wods, each gadent sampe s added to each SOH weghted by -d, whee d s the dstance of the sampe fom the cente of the coespondng sub-egon []. Hence fom the 6 anges at most anges can meet the pawse ndependence condton. Theefoe, ony fou anges can be pa-wse ndependent and ony bode anges can meet the equay key condton, hence the best choce ae the cone anges:,, 3, and. (, j ),8 (, j ) (, ),8 3 j,8 ( j, ),8,6,6,6,6,,,, 3 3, 3 3, 3 3, 3 3, ( j, ),8,6, ), ),8 ( j ( (, ) 3 j,8 j,6,6,8,6 3 3,, 3 3,, 3 3,, 3 3,, ( 3, j ),8 ( 3, j ) ( ),8 33 j,,8, ) ( 3 j,8,6,6,6,6,,,, 3, 3 3 (, j ) (, j ),8 3, 3,8, 3 3 3, ),8, ) ( 3 j ( j,,8,6,6,6,6,,,, 3, 3, 3,, Fgue 5.5: The coeaton coeffcents between anges of SIFT featues. Fo eampe the top eft dagam pesents coeaton coeffcents between and a. The and y aes pesent ndces and j j Page 7

81 Fast SIFT Featue Matchng espectvey whe z as pesent coeaton facto. As evdent fom Fgues 5. and 5.5, each two cone anges ae ndependent fom each othe and unfomy-dstbuted n the ange ange Theefoe, the anges { :,} meet both condtons, equay key and pa-wse ndependence condtons, hence they can be consdeed as new attbutes of the SIFT featue, that ae epoted to acceeate SIFT featue matchng., y,,, V F, y,,, V,,, F (5.36) ma ma 3, Vey Fast SIFT Featues Matchng Featue matchng pocess s the most computatonay epensve pat of many compute vson agothms. In ths Secton new dea s poposed to acceeate the matchng pocess by compason ony featues that shae the same coespondng anges whch may ead to coect matches. Assumng that two sets of etended SIFT featues R F ;,,... L F j ; j,,... possbe M F, F matches s equa to and, contanng espectvey and featues, ae gven,. The numbe of j j. Among these possbe matches a sma numbe of coect matches may est. Fo each possbe SIFT match, fou dffeent ange dffeences can be constucted: (5.37) Consdeng the ange dffeences,, 33,,,. 33, as andom vaabes The behavos of these andom vaabes vay dffeenty accodng to the type of matches (coect and fase matches) Fo fase match, ts featues ae ndependent, theefoe each two coespondng anges ae ndependent, whch ead to the fact that the fou andom vaabes ae unfomy dstbuted (accodng to the emma poven n Secton 5.) and ae pa wse ndependent. On the othe hand, fo coect match, each two coespondng anges tend to be equa, snce the featues of coect matches tend to have same SIFT descptos. Theefoe the fou andom vaabes tend to concentate n naow ange aound. The PDFs of j fo both coect and fase matches ae measued n epements ove consdeed 7 test mages. The estmated PDFs ae shown n Fgue 5.6. It can be seen fom Fgue 5.6 that fo each ange sepaatey about 99 % of coect and ony % of fase matches beong to the ange 36,36. Because the possbe matches ae Page 7

82 Fast SIFT Featue Matchng unfomy dstusted n the dmensona ange space 8,8 then the poton of possbe matches n the ange 36,36 7 s equa to %. Theefoe, n ode to 36 fnd coect matches t s needed to teat ony,6% of possbe matches whch can speed up the featue matchng sgnfcanty. (a): PDFs of fase matches 3,5% PDF(%) 3,%,5% Ange(Degee) (b): PDFs of coect matches 3 7% 6% 5% PDF(%) % 3% % % % Ange(Degee) Fgue 5.6: The epementa PDFs of the ange dffeence matches (b). j fo the possbe (a) and the coect To epot ths outcome, SIFT featues ae hashed nto dmensona tabe based on the anges. The SIFT featues of each ce ae compaed ony wth the featues of some ces, so that the coespondences must have absoute dffeences of anges ess than a pe-set Page 73

83 Fast SIFT Featue Matchng theshod. Hee a theshod of 36 s seected because amost a coect matches have ange dffeences n the ange [-36,36 ] as ustated n Fgue 5.6b. Consde that one of the sets of featues R o L (fo eampe R ) s hashed nto b buckets, th so that the jfg buckets S jfg contans ony the SIFT featues that meet the foowng condtons: 3 8 ( ) 36 b, 8 36 b 8 ( j ) 36 b, 8 8 ( f ) 36 b, 8 f j ( g ) 36 b, 8 g 36 b b b S jfg F,,, 3 (5.38) The numbe of featues of the set R can be epessed as: b b b b jfg j f g (5.39) Because of the eveny dstbuton of featue anges ove the ange of the anges [- 8,8 ] as shown n Fgue 5.6, the featues ae amost equay dvded nto b buckets. Theefoe, t can be asseted that the featue numbes of the buckets ae amost equa to each othe.,,... b:, j, f, g jfg b (5.) To ecude matchng of featues that have ange dffeences outsde the ange a, a, each bucket s matched to ts coespondng one and to n neghbong buckets to the eft and to the ght sde. In ths case the matchng tme s popotona to the foowng tem: T etended n jn f n gn opst on p jn s f n tg n b n jn f n g n on p jn s f n tg n () n b (5.) Theefoe, the acheved speedup facto wth espect to ehaustve seach s equa to: The eaton between n, a and b s as foows: b SF etended (5.) n 36 ab (n ) a n (5.3) b 36 whee epesents the fst ntege vaue age than o equa to. Substtutng of (5.) nto (5.) yeds: Page 7

84 Fast SIFT Featue Matchng The esut (5.3) means that f t s amed to ecude matchng of featues that have ange dffeences outsde the ange [-36,36 ], then the matchng step s acceeated by a facto of 65. When ths modfcaton of ogna SIFT featue matchng s combned wth the spt SIFT featues matchng, the obtaned speedup facto s 5 wthout osng a notabe poton of coect matches. Ths s ustated wth the epementa esuts pesented n the net secton Epementa Resuts The poposed method Vey Fast SIFT matchng (VF-SIFT) was tested usng a standad mage dataset [65] and ea-wod steeo mages. The used mage dataset conssts of about 5 mages of 3 dffeent scenes (some eampes ae shown n Fgue 5.9). Rea-wod steeo mages was captued usng obotc vson system (A Bumbebee steeo camea wth the esouton of. X768 pes), some eampes ae shown n Fgue 5.. In ode to evauate the effectveness of the poposed method, ts pefomance was compaed wth the pefomances of two agothms fo ANN (Heachca K-Means Tee (HKMT) and Randomzed KD-Tees (RKDTs)) [59]. Compasons wee pefomed usng the Fast Lbay fo Appomate Neaest Neghbos (FLANN) [66]. Fo a agothms, the matchng pocess s un unde dffeent pecson degees makng tade off between matchng speedup and matchng accuacy. The pecson degee s defned as the ato between the numbe of coect matches etuned usng the consdeed agothm and usng the ehaustve seach, wheeas the speedup facto s defned as the ato between the ehaustve matchng tme and the matchng tme fo the coespondng agothm. Fo both ANN agothms, the pecson s adjusted by the numbe of nodes to be eamned [66], wheeas fo the poposed VF-SIFT method, the pecson s detemned by adjustng the wdth of the ange of coect matches w. co To evauate the poposed method two epements wee un. In the fst epement, mage to mage matchng was studed. SIFT featues wee etacted fom steeo mage pas and then each two coespondng mages wee matched usng HKMT, RKDTs and VF-SIFT, unde dffeent degees of pecson. The epementa esuts ae shown n Fgue 5.7a. The second epement was caed out on the mages of the dataset [65] to study matchng mage aganst a database of mages. SIFT featues etacted fom quey mages ae matched aganst database of 5 SIFT featues usng a thee consdeed agothms, wth dffeent degees of pecson. The epementa esuts ae shown n Fgue 5.7b. As can be seen fom Fgue 5.7, VF-SIFT etemey outpefoms the two othe consdeed agothms n speedng up of featue matchng fo a pecson degees. Fo pecson aound 95%, VF-SIFT gets a speedup facto of about 5. Fo the owe pecson degees speedup facto s much hghe. Though compason between Fgues 5.7a and 5.7b, t can be seen that the poposed method pefoms smay fo both cases of mage matchng (mage to mage and mage 36 SF etended (5.) a Page 75

85 Fast SIFT Featue Matchng aganst database of mages), wheeas ANN agothms ae moe sutabe fo matchng mage aganst database of mages [66]. (a) (b) Speedup(SF) VF- SIFT HKMT RKDTs , Pecson (%) Pecson (%) Fgue 5.7: Tade-off between matchng speedup (SF) and matchng pecson. 3 Matchng esut wth umnaton changes Matchng esut wth otaton changes Fgue 5.8: Coect SIFT featue coespondences between two mages of the same scene captued unde two dffeent condtons. Page 76

86 Fast SIFT Featue Matchng Fgue 5.8 pesents two eampes of mage matchng unde otaton and umnaton changes. It s easy to educe that the coespondence SIFT featues ae aways fom the same type (mama o Mnma) and have amost the same coespondng anges Concuson In ths Chapte, a new method fo fast SIFT featue matchng s poposed. The dea behnd s to etend a SIFT featue by pa-wse ndependent anges, whch ae nvaant to otaton, scae and umnaton changes and unfomy-dstbuted n the ange ange. Dung etacton phase, SIFT featues ae cassfed based on the anges nto dffeent custes. Thus n matchng phase, ony SIFT featues that beong to custes whee coect matches may be epected ae compaed. The poposed method was tested on ea-wod steeo mages fom a obotc appcaton and standad dataset mages. The poposed method was compaed wth two agothms fo ANN seachng, heachca k-means and andomzed kd-tees. The pesented epementa esuts show that the pefomance of the poposed method outpefoms two othe consdeed agothms. Aso, the pesented esuts show that the featue matchng can be speeded up by 5 tmes wth espect to ehaustve seach wthout osng a notceabe poton of coect matches. Page 77

87 Robust SIFT Featue Matchng 6. Robust SIFT Featue Matchng 6.. Intoducton The matchng of mages n ode to estabsh a measue of the smaty s a key pobem n many compute vson tasks. Robot ocazaton and navgaton, object ecognton, budng panoamas and mage egstaton epesent just a sma sampe among a age numbe of possbe appcatons. In ths pape, the emphass s on object ecognton. In genea the estng object ecognton agothms can be cassfed nto two categoes: goba and oca featues based agothms. Goba featues based agothms am at ecognzng an object as a whoe. To acheve ths, afte the acquston, the test object mage s sequentay pe-pocessed and segmented. Then, the goba featues ae etacted and fnay statstca featues cassfcaton technques ae used. Ths cass of agothm s patcuay sutabe fo ecognton of homogeneous (tetueess) objects, whch can be easy segmented fom the mage backgound. Featues such as Hu moments [35] o the egenvectos of the covaance mat of the segmented object [7] can be used as goba featues. Goba featues based agothms ae smpe and fast, but thee ae mtatons n the eabty of object ecognton unde changes n umnaton and object pose. In contast to ths, oca featues based agothms ae moe sutabe fo tetued objects and ae moe obust wth espect to vaatons n pose and umnaton. In [7] the advantages of oca ove goba featues ae demonstated. Loca featues based agothms focus many on the so-caed key-ponts. In ths contet, the genea scheme fo object ecognton usuay nvoves thee mpotant stages: The fst one s the etacton of saent featue ponts (fo eampe cones) fom both test and mode object mages. The second stage s the constucton of egons aound the saent ponts usng mechansms that am to keep the egons chaactestcs nsenstve to vewpont and umnaton changes. The fna stage s the matchng between test and mode mages based on etacted featues. The deveopment of mage matchng by usng a set of oca key-ponts can be taced back to the wok of Moavec [7]. He defned the concept of "ponts of nteest" as beng dstnct egons n mages that can be used to fnd matchng egons n consecutve mage fames. The Moavec opeato was futhe deveoped by C. Has and M. Stephens [3] who made t moe epeatabe unde sma mage vaatons and nea edges. Schmd and Moh [73] used Has cones to show that nvaant oca featues matchng coud be etended to the genea mage ecognton pobem. They used a otatonay nvaant descpto fo the oca mage egons n ode to aow featue matchng unde abtay oentaton vaatons. Athough t s otatona nvaant, the Has cone detecto s howeve vey senstve to changes n mage scae so t does not povde a good bass fo matchng mages of dffeent szes. Lowe [, 67, 68] ovecome such pobems by detectng the ponts of nteest ove the mage and ts scaes though the ocaton of the oca Etema n a pyamda Dffeence of Gaussans (DOG). The Lowe s descpto, whch s based on seectng stabe featues n the scae space, s named the Scae Invaant Featue Tansfom (SIFT). Mkoajczyk and Schmd [8] epementay compaed the pefomances of sevea cuenty used oca descptos and they found that the SIFT descptos to be the most effectve, as they yeded the best Page 78

88 Robust SIFT Featue Matchng matchng esuts. SIFT mpovng technques deveoped ecenty tageted mnmzaton of the computatona tme [5][6][7][7], whe mted eseach amng at mpovng the accuacy has been done. The wok pesented n ths pape demonstates nceased matchng pocess pefomance obustness wth no addtona tme costs. Speca cases, sma scaed featues, consume even ess tme. The hgh effectveness of the SIFT descpto s the motvaton to use t fo object ecognton n sevce obotcs appcatons [75]. Though the pefomed epements t was found that SIFT key-ponts featues ae hghy dstnctve and nvaant to mage scae and otaton povdng coect matchng n mages subject to nose, vewpont and umnaton changes. Howeve, t was aso found that sometmes the numbe of coect matches s nsuffcent fo object ecognton, patcuay when the taget object, o pat of t, appeas vey sma n the test mage wth espect to ts appeaance n mode mage. In ths chapte, a new stategy to enhance the numbe of coect matches s poposed. The man dea s to detemne the scae facto of the taget object n the test mage usng a sutabe mechansm and to pefom the matchng pocess unde the constant ntoduced by the scae facto, as descbed n Secton Impoved SIFT Featues Matchng Fom the SIFT agothm descpton gven n Chapte t s evdent that n genea, the SIFTagothm can be undestood as a oca mage opeato whch takes an nput mage and tansfoms t nto a coecton of oca featues. To use the SIFT opeato fo object ecognton puposes, t s apped on two object mages, a mode and a test mage, as shown n Fgue 6. fo the case of a food package. As shown, the mode object mage s an mage of the object aone taken n pedefned condtons, whe the test mage s an mage of the object togethe wth ts envonment. Fgue 6.: Tansfomaton of both mode and test mage nto two coectons of SIFT featues; dvson of the featues sets nto subsets accodng to the octave of each featue. Page 79

89 Robust SIFT Featue Matchng To fnd coespondng featues between the two mages, whch w ead to object ecognton, dffeent featue matchng appoaches can be used. Accodng to the Neaest Neghbohood j pocedue fo each F featue n the mode mage featue set the coespondng featue F must be ooked fo n the test mage featue set. The coespondng featue s one wth the j smaest Eucdean dstance to the featue F. A pa of coespondng featues F, F s j caed a match M F, F. To detemne whethe ths match s postve o negatve, a theshod can be used. j If the Eucdean dstance between the two featues F and F s beow a cetan theshod, j the match M F, F s abeed as postve. Because of the change n the pojecton of the taget object fom scene to scene, the goba theshod fo the dstance to the net featue s not usefu. Lowe [67] poposed the usng of the ato between the Eucdean dstance to the neaest and the second neaest neghbos as a theshod. Unde the condton that the object does not contan epeatng pattens, one sutabe match s epected and the Eucdean dstance to the neaest neghbo s sgnfcanty smae than the Eucdean dstance to the second neaest neghbo. If no match s coect, a dstances have a sma, sma dffeence fom each othe. A match s seected as postve ony f the dstance to the neaest neghbo s.8 tmes age than that fom the second neaest one. Among postve and negatve matches, coect as we as fase matches can be found. Lowe cams [] that the theshod of.8 povdes 95% of coect matches as postve and 9% of fase matches as negatve. The tota amount of the coect postve matches must be age enough to povde eabe object ecognton. In the foowng an mpovement to the featue matchng obustness of the SIFT agothm wth espect to the numbe of coect postve matches s pesented. As mentoned above, the taget object n the test mage s pat of a cutteed scene. In a eawod appcaton the appeaance of the taget object n the test mage, ts poston, scae and oentaton, ae not known a po. Assumng that the taget object s not defomed, a featues of the taget mage can be consdeed as beng affected wth constant scang and otatona factos. Ths can be used to optmze the SIFT-featue matchng phase whee the outes' ejecton stage of the ogna SIFT-method s ntegated nto the SIFT-featue matchng stage Scang Facto Cacuaton As mentoned above, usng the SIFT-opeato, the two object mages (mode and test) ae tansfomed nto two SIFT-mage featue sets. These two featue sets ae dvded nto subsets accodng to the octaves n whch the featue ase. Hence, thee s a sepaate subset fo each mage octave as shown n Fgue 6.. To cay out the poposed new stategy of SIFT-featues matchng, the featues subsets obtaned ae aanged so that a subset of the mode mage featue set s agned wth an appopate subset of the test mage featue set. The pocess of agnment of the mode mage subsets wth the test mage subsets s ndcated wth aows n Fgue 6.. The agnment pocess s pefomed though the n m steps, whee n and m ae the tota numbe of octaves (subsets) coespondng to the mode and test mage espectvey. Fo each step a pas of agned subsets must have the same ato defned as: Page 8

90 Robust SIFT Featue Matchng Page 8 whee o and o ae the octaves of the mode mage subset and the test mage subset espectvey. 8 : 3 a : 3 b : 3 c : 3 3 d : 3 e : 3 f 8 : 3 g Fgue 6.: Steps of the pocedue fo scae facto cacuaton. Fo eampe at the fst step a, ony SIFT featues of mode mage etacted fom the octave o ae compaed wth the SIFT featues of the test mage etacted fom the o o (6.)

91 Robust SIFT Featue Matchng octave o 3. In ths case we can gantee that a possbe matches have the scae ato 3 8. In the step b, ony the mode SIFT featues of the octaves o and o ae compaed wth the test SIFT featues of the octaves o and o 3 espectvey. In both cases, 3 possbe matches have scae ato of, and so on fo the othe steps. At evey step, the tota numbe of postve matches s detemned fo each agned subsets pa. The tota numbe of postve matches wthn each step s ndeed usng the appopate shft nde k o o (6.) Shft nde can be negatve (Fgues 6.a, 6.b and 6.c), postve (6.e, 6.f and 6.g) o equa to zeo (Fgue 6.d). The hghest numbe of postve matches acheved detemnes the optma shft nde k opt and consequenty the scae facto: In ode to eaze the poposed pocedue mathematcay, a scae ato hstogam (SRH) F s defned as: M, s the numbe of postve matches between the th subset of the mode mage featue set M and the j th j subset of the test mage featue set M, and s the modfed shft nde ntoduced fo the sake of smpcty of equaton 6.. j whee M F j j jn M M mn M k S opt (6.3) n j n j n j, M, M, M j The dagam showng the dstbuton of F k ove the ange of the shft nde k fo the eampe shown n Fgue s pesented n Fgue 6.. j f f m j n m f n m (6.) n m ntk (6.5) Page 8

92 Robust SIFT Featue Matchng 5 5 F(k) Fgue 6.3: The scae ato hstogam F k. As evdent fom Fgue 6.3, the scae ato hstogam nde: - 3 Shft nde k F k eaches ts mamum at the shft k opt ac ma( F( k)) (6.6) whch coesponds to the scae facto The optma shft nde defnes a doman of coect matches. A matches outsde ths doman, ncudng postve matches, ae ecuded. The postve matches fom the doman of coect matches ae used to detemne the affne tansfomaton (otaton mat, and tansaton vecto) between the two featue sets, usng RANSAC method [5]. Once the tansfomaton s cacuated, evey match, ethe postve o negatve, wthn the doman of coect matches s eamned whethe t meets the aeady cacuated tansfomaton. If the match fufs the tansfomaton, t s abeed as a coect, othewse as a fase match Reteva of The Coect Matches k S opt (6.7) Among a found matches t can happen that a ot of coect matches eceed Lowe's theshod. In ode to eteve these coect matches, the ato between the Eucdean dstance to the neaest and the second neaest featue neghbo must be educed. Ths can be done ethe by j educng the smaest dstance d F, F o by nceasng the net smaest dstance j d F, F. In pactce, the fst atenatve s mpossbe whe the enagement of net smaest dstance can be acheved by mtng the seach aea fo both the neaest and net neaest featue to the featue F wthn a specfed doman. Fo a bette epanaton of ths Page 83

93 Robust SIFT Featue Matchng dea, suppose that a featue F fom the mode mage featue set s coecty assgned to the j j featue F fom the test mage featue set. Aso, suppose that F s the second neaest j featue to the F whe F s the second neaest featue to t when the seach s mted ony j to the octave n whch the F s found. F j F j F j F Fgue 6.: Savng the coect matches that may eceed Lowe's theshod. j j Snce F, F d F F d aways hods the foowng s obtaned: 3, j j j j F, F d F, F d F, F d F F d (6.8) 3, Thus, by educng the seach aea t s possbe to decease the ato eated to the featue and make t ess than theshod. In ths way the numbe of coect matches s nceased Compety and Cost of Tme An addtona esut of the eseach pesented n ths chapte s consdeaton of the mpovement of the ogna SIFT agothm wth espect to the pocessng tme. As fst, t can be shown that the ogna SIFT pocedue and the pocedue deveoped n ths wok compete the matchng pocedue n the same tme. Assumng that the numbe of featues n the mode object mage and n the test mage s: F Page 8

94 Robust SIFT Featue Matchng Page 85 whee n and m ae the tota numbe of octaves coespondng to the mode and test mage espectvey. Thus, the compety of ogna SIFT-matchng pocedue s popotona to the poduct The compety of the poposed appoach, whch can be seen fom Fgue 6., s popotona to the foowng sum of the poducts: Substtutng equaton (6.9) n equaton (6.) one obtans: whch s equa to the poduct P coespondng to the compety of the ogna SIFT matchng pocedue. The above condton epesents the compety of the poposed matchng pocedue when no a-poy nfomaton about the scang facto of coespondng featues s avaabe, that s when the pocedue conssts of a m n steps as epaned n Secton 6... Howeve, n some appcatons the compety s educed. Fo eampe, f the two mages to be matched ae mages of steeo camea system wth sma basene, a coespondng featues shoud have the same scae. Hence the poposed matchng pocedue s caed out wth ony one step coespondng to the shft nde. k m j j m n n h h h h h (6.9) h P (6.) n m j j n n n n m n m n m n m n m m m m h h h h h h h h h h h h h h h h P (6.) h h h h P n m j j n m j (6.)

95 Robust SIFT Featue Matchng Page 86 In ths case, the compety of the poposed pocedues s educed, snce t s popotona to the sum of the foowng poducts: In ode to detemne the amount of educed pocessng tme n compason to ogna SIFT pocedue, t s assumed that the numbe of etacted featues n the owe octave wth espect to the hghe octave s deceased tmes due to the down-sampng by the facto of n both mage dectons. Hence, t s assumed that: Substtutng equaton (6.) n both poducts P and 3 P, defned wth (6.) and (6.3) espectvey, one obtans: and Fom equatons (6.5) and (6.6) the ato 3 P P s gven as: It s known that Substtutng (6.8), the ato (6.7) becomes: 3... n n h h h P (6.3) h h (6.) 3 ) ( 3... n n h P h h h h P (6.5) n n n n n h P h h h P h h h h P (6.6) 3 n n h h P P (6.7) f (6.8)

96 Robust SIFT Featue Matchng P P 5 9 3,67 (6.9) Hence, the matchng tme cost n the case of matchng steeo mages s educed.67 tmes n compason to the ogna SIFT method Epementa Resuts In ths secton a pefomance evauaton of the poposed mpovement of the Lowe s SIFT featue matchng agothm s pesented. Snce the goa s to acheve a tade-off between the nceasng the numbe of coect matches and mnmzng the numbe of fase matches fo an object mage pa consstng of test and mode object mages, the pefomance of the poposed method s evauated usng the popua Reca-Pecson metc [76]. j As mentoned n Secton 6., two SIFT featues F and F ae matched when the SIFT j descpto of the featue F has the smaest dstance to the descpto of featue F among dstances coespondng to a othe etacted featues. If the ato between the Eucdan dstances to the neaest neghbo and to the second neaest neghbo s beow a theshod, the match s abeed as postve, othewse as negatve.. Among postve and negatve abeed matches, coect as we as fase matches can be found. Thus thee ae fou dffeent possbe combnatons though the foowng confuson mat: Tabe 6.: The confuson Mat Actua postve Actua negatve Pedcted postve TP FP Pedcted negatve FN TN Dung the matchng of an mage pa the eements of the confuson mat ae counted. The vaue of s vaed to obtan the Reca vesus -Pecson cuve, wth whch the esut ae pesented. Reca and - Pecson ae cacuated based on the foowng defntons [67]: Reca TP TP FN P ecson FP TP FN (6.) The agothms wee tested by matchng ea mages of the scenes fom wokng scenaos of the obotc system FRIEND II contanng dffeent taget objects to be ecognzed (bottes, packages, and etc), acqued wth the steeo camea system of FRIEND II obot. Page 87

97 Robust SIFT Featue Matchng Two man types of epements wee un to dscuss the dffeence between the ogna SIFT and the poposed optmzed SIFT matchng agothm. In the fst epement, the mode mages of two dffeent objects, a botte of the "mezzo m" dnk and a coffee ftes package, wee matched wth the coespondng test object mages usng the ogna and poposed mpoved SIFT matchng agothm. The epementa esuts ae ustated n Fgue 6.6. As evdent, the appeaance of the taget objects n the test mages s dffeent fom the appeaance n mode mages due to dffeent condtons such as umnaton dung the mage acquston, vewpont, pata occuson etc. the advantage of the poposed matchng technque ove the ogna SIFT matchng technque s evdent fom Fgue 6.6. Besde the eamnaton of the esuts ustaton n Fgue 6.6, pefomance evauaton can be done by eamnaton of the eca vesus -pecson cuve shown n Fgue 6.5. The cuves ae obtaned by vayng the theshod fom.5 t.. Incea sng Lowes ' Theshod,8 Reca,6,, Standad SIFT Impoved SIFT,,,6,8 -Pecson Fgue 6.5: Reca vesus -Pecson cuves fo the ogna and optmzed SIFT matchng methods. In the second epement mages of a scene fom the obot FRIEND II envonment, captued by the obot steeo camea system, wee matched to evauate the optmzng of the computatona matchng tme of the poposed appoach wth espect to the ogna SIFT. The epementa esuts ae gven n the Tabe 6.. The epementay obtaned atos of the pocessng tme of ogna SIFT and pocessng tme of poposed technque sghty dffe fom the ato deved n secton 6. because the assumpton assumed the poof does not necessay hod. The matchng pocess was caed out usng a Pentum IV GH pocesso wth, mages of sze X768 pes. Page 88

98 Robust SIFT Featue Matchng Tabe 6.: Compason of the steeo mages matchng tme. Key-ponts numbe n steeo mages Ogna SIFT matchng Impoved SIFT matchng ght eft Matchng tme (sec) Numbe of nes Matchng tme (sec) Numbe of nes Concusons In ths chapte an mpovement of the ogna SIFT-agothm deveoped by Lowe was poposed. Ths mpovement coesponds to enhancement of featue matchng obustness, so the numbe of coect SIFT featues matches s sgnfcanty nceased whe neay a outes ae dscaded. The dea s based on the detemnaton of the scae facto between mages to be matched and mtng the matchng pocess to featue pas that ft ths scae facto. In ode to detemne the scae facto, the featue sets ae dvded nto subsets accodng to the octaves n whch the featue ase. Afte that the featue matchng s pefomed n stepwse fashon so that wth each step ony the SIFT featues of the same scae ato s matched. The step wth the hghest numbe of postve matches detemnes the appomate scae facto between the mages beng matched. When no pe-nfomaton about scae facto ae avaabe then both matchng pocedues, the standad SIFT and the pocedue deveoped n ths wok compete the matchng pocess n the same tme. The new poposed appoach was tested usng ea mages acqued wth the steeo camea system of FRIEND II/III obotc system. The epementa esuts showed that the numbe of coect matches was nceased and, at the same tme, the numbe of outes was deceased n compason wth the ogna SIFT agothm. Compaed wth the ogna SIFT agothm, a % educton n pocessng tme was acheved fo the matchng of the steeo mages, snce the scae facto n case of steeo mage matchng s equa to. Page 89

99 Robust SIFT Featue Matchng Matchng esut fo coffee fte package Matchng esut fo botte of the "mezzo m" dnk Fgue 6.6: (eft coumn) matchng esut wth ogna SIFT, (ght coumn) matchng esut wth mpoved SIFT. Page 9

100 Fuzzy Based Cosed Loop Conto System fo Object Recognton 7. Fuzzy Based Cosed Loop Conto System fo Object Recognton 7.. Intoducton One of the most wdey eseached and an mpotant aea n compute vson s object ecognton. In genea, the object ecognton systems can be cassfed nto two majo categoes: the goba and the oca featue-based systems. The goba featue-based systems am at ecognzng the object n ts whoe. To ths end, the quey mage s acqued, pepocessed, segmented, and then goba featues ae etacted. Fnay, statstca cassfcaton technques ae used. Ths cass of agothms s especay sutabe fo homogeneous objects, whch can be easy segmented. Featues such as the Hu moments [85], the egenvectos of the covaance mat [86], centods, pemetes, aeas, and coos [87][88] can be used as goba featues. The goba featue-based agothms ae smpe and fast, but thee ae mtatons n ecognton unde umnaton and pose changes, Fgue 7. pesents the fow dagam of the goba featue-based object ecognton systems. Loca featue-based systems on the othe hand ae moe sutabe fo tetued objects and moe obust wth espect to vewpont and umnaton changes. Image Acquston Image Pe-pocessng Image Segmentaton Featues Etacton Cassfcaton Image Undestandng Fgue 7.: Goba featue-based object ecognton system The oca featue-based systems ae based on the dea of epesentng an object by a coecton of oca nvaant patches. Ths dea can be taced back to Schmd and Moh [89][9], whee the centes of patches ae ocated at ponts of nteest and ae nvaant to otaton. Lowe [67][68] deveoped an effcent object ecognton appoach based on scae nvaant featues (SIFT). Geneay, the stuctue of the oca featue-based object ecognton system many nvoves fou majo steps, as shown n Fgue 7.: Featues detecton: Etacton of saent ponts (typcay cones o bob-ke shapes), fom mages to be matched (quey and mode mages). Featues descpton: Constucton of descptos fom egons aound the saent keypont uses mechansms that am to keep the chaactestcs of these egons nsenstve to vewpont, umnaton changes and nvaant to otaton, scang and affne tansfomaton. Featues matchng: Computng the coespondence ponts between the quey and the mode mage based on etacted featues. Out of the matched ponts an affne tansfomaton between quey mage and mode mage can be computed usng a fttng method (such as Least of Squaes o RANSAC method). The matchng pocess s then teatvey ened by emovng those coespondence ponts whch do not t ths affne tansfomaton. Page 9

101 Fuzzy Based Cosed Loop Conto System fo Object Recognton Pose estmaton: Estmaton of the (, y, z)-tansaton components and (,, )- otaton anges of the object wth espect to the camea coodnate system usng the coespondence ponts, the taget object geomety and the ntnsc camea paametes. Mode Image Camea Intnsc Paamete Loca Featues Detecton Loca Featues Descpton Loca Featues Matchng Pose Estmaton Quey Image Loca featues etacted fom mode mage Loca featues etacted fom quey mage Object Geomety Fgue 7.: Loca featue-based object ecognton system It can be easy notced that both object ecognton systems pesented above ae open-oop, whch means that the esut of each step depends on the esut of the pevous one, theefoe the eos ae accumuated ove the ente ecognton system and popagated to the fna step. Hence the system fna esut tends to be eo pone and uneabe. Ths pobem s usuay soved usng cosed-oop conto technques. In the teatue, thee ae few pubcatons deang wth the usage of cosed oop conto stateges fo object ecognton and mage pocessng. Fo eampe, n [8] and [8] enfocement eanng has been used to nduce a mappng fom nput mages to coespondng segmentaton paametes by usng the confdence eve of mode matchng as a enfocement. In [8] conto stateges have been used at ow, ntemedate, and hgh eves of anayss fo mpovng on estabshed-snge-pass hypothess geneaton and vefcaton appoaches n object ecognton. In [83] and [8] the feedback conto of mage quaty at dffeent eves of mage pocessng chan, amng at goba featue-based object ecognton s ntoduced to mpove the mage quaty fo successfu mage segmentaton and featue etacton. The above-mentoned methods commony concentate on the goba featue-based object ecognton systems though optmzng of the segmentaton stage. In ths chapte, we popose a cosed-oop conto system fo object ecognton, pose estmaton and camea cabaton based on SIFT featues []. Ou wok concentates on usng the benefts of cosed oop stuctue to ncease the nvaance to affnty, theefoe to ncease the quaty and the quantty of the matchng pocess and to efne pose estmaton, whch s essenta fo sevce obotcs fo autonomous object manpuaton. The dea s to etact two ndependent paae featue steams (Mama and Mnma SIFT featues) fom both the mode and the quey mage, and then matchng between featues beong to coespondng steams to estmate two ndependent affne tansfomatons. The dssmaty between these tansfomatons s used as a feedback vaabe that seves to obseve and conto the matchng pocess. If ths vaabe s moe than a cetan theshod, one of the tansfomatons s seected usng fuzzy contoe to wap the mode mage. The pocedue s epeated unt the two Page 9

102 Fuzzy Based Cosed Loop Conto System fo Object Recognton tansfomatons become sma o one of them conveges to the dentty mat. The system has been vefed though epements on sevea ea-wod mages. The obtaned esuts ae shown n Secton Cosed Loop Conto System fo Object Recognton A typca oca featue-based object ecognton system as shown n Fgue 7. s used to dentfy and ocate an object of nteest captued by camea system n a scene.the nput of the system s a mode mage of the object of nteest and a quey mage. The mode mage s used to eamne the pesence of the object n the coespondng quey mage and to estmate ts pose wth espect to the camea coodnate system. At fst, key ponts ae etacted fom the mode and the quey mages and descbed by SIFT descpto [][67]. These SIFT featues ae then povded as nput to mage matchng pocess. In genea, mage matchng s defned as a pocess n whch the coespondences between subset of ponts n two mages ae detemned. Fom coespondences an affne tansfomaton (otaton, scang, and tansaton changes) s estmated that maps the two mages. Once the coespondence ponts ae estabshed, and the ntnsc camea paametes and the geomety of the object ae known, the pose of object can be estmated [9]. The accuacy and the eabty of the estmated pose depend stongy on the outcome of the mage matchng pocess. Hence the matchng esut pays a cuca oe n the eabty of the whoe system. The system ustated n Fgue 7. totay gnoes the effects of msmatches on the pefomance of the pose estmaton method. Ths pobem s sma to that occus n the open oop systems, whch ae affected by nose. In conto theoy, feedback oops have been used to sove these pobems. In ths chapte, we ty to use a sma pncpe fo mpovng the quaty and the quantty of the matchng pocess esut, whch eads to enhance the effcency of the object detecton and to efne the 3D pose of the taget object. To cose the oop, we need to defne a quanttatve measuement that descbes how good the matchng esut s, and to modfy the nput of mage matchng fo mpovng ts output when the matchng esut s not accepted. The defnton of ths quanttatve measuement s based on the fact that the SIFT featue ocatons ae effcenty detected by dentfyng Mama and Mnma of the Dffeence-of Gaussan (DoG) scae space as epaned n chapte. quey Each set of the SIFT featues of the quey mage GF and of the mode mage GF mod dvded nto two subsets, one fo the Mama and the othe fo the Mnma SIFT featues. e ae GF GF mod e quey GF GF mod e mn quey mn GF GF mod e ma quey ma (7.) By matchng Mama SIFT featues wth Mama and Mnma wth Mnma, two ndependent sets of postve matches GM ma and GM mn ae obtaned. GM GM mn ma match GF match GF mod e mn mod e ma, GF, GF quey mn quey ma (7.) Page 93

103 Fuzzy Based Cosed Loop Conto System fo Object Recognton Fom these sets of postve matches, two ndependent affne tansfomatons (Mama and Mnma affne tansfomatons) can be estmated usng RANSAC agothm [5]. T T mn ma RANSAC GM RANSAC GM mn ma (7.3) Snce both affne tansfomatons ae estmated though two dffeent channes affected by dffeent nose easons (Mama- and Mnma-msmatches), the degee of the dssmaty between them Ds T ma,t mn efects the degee of goodness of the matchng outcome. Geneay, when affne tansfomatons ae computed, t can be dstngushed between two cases: at east one of the tansfomatons s coect o both ae wong In the fst case, f the dssmaty Ds T ma,t mn s ess than a pe-defned theshod, whch means both tansfomatons ae coecty estmated snce two ndependent steams of matches etun the same nfomaton, hence the object s we-detected and ts pose s estmated wth a suffcent degee of accuacy. Othewse both tansfomatons ae gven as a feedback to a fuzzy contoe to seect the coect tansfomaton to wap the mode mage. The SIFT featues ae then etacted fom the new poduced mode mage and matched to the quey mage, hence two new affne tansfomatons ae estmated and the dssmaty s computed. The pocess s epeated unt the dssmaty between the cuent tansfomatons o the dssmaty between one of them and the dentty mat s ess than a cetan theshod. The ast temnaton condton s due to the feedback oop s desgned to make the taget object appeaance n mode mage as sma as possbe to n the quey mage. The second case can be dstngushed when the output of the cosed oop does not convege to the dentty mat, whch means that the quey mage does not nvove the taget object o t s vey dffcut to be detected. Page 9

104 Fuzzy Based Cosed Loop Conto System fo Object Recognton I Mama SIFT featues of MI Mnma SIFT featues of MI Mama SIFT featues of QI Mnma SIFT featues of QI MI Contoe QI FE FE: featues etacton FM: featues matchng ATE: affne tansfomaton estmaton PE: pose estmaton QI: quey mage MI: mode mage T ( k) ma FM ATE PE T ( k) mn Fgue 7.3: poposed cosed oop object ecognton system Dssmaty between Two Affne Tansfomatons In genea, because at east thee non-conea coespondng ponts between two mages ae equed to detemne the affne tansfomaton, t s aso needed at east thee non-conea ponts to compute the dssmaty between two affne tansfomatons T and T : Assumng that p a, a, p a, a and p a, a y, whee a s abtay vaue. 3 ae thee non-conea ponts at the pane p d T T p 3 d 3 p p 3 p d p p 3 p p Fgue 7.: Dssmaty between two affne tansfomatons. Each one of these ponts s mapped by each affne tansfomaton. p p T p T p (7.) Page 95

105 Fuzzy Based Cosed Loop Conto System fo Object Recognton whee,, 3 Hence the dssmaty Ds T,T s defned as: whee d s the Eucdan dstance between two ponts p and p p, p computed as foows: 7.. Fuzzy Contoe d 3 T, T d p, p Ds (7.5) 3 p p y, y, y, y (7.6) Geneay, a fuzzy knowedge-based system s composed of two modues, a knowedge base epesented by a set of condtona ues, and an nfeence engne, whch makes the ues wok n esponse to the system nputs. An mpotant appcaton of fuzzy knowedge-based system s the conto of compe, nonnea systems [93]. Conto agothms wth fuzzy contoes offe bette esponse and effcency n case of compe nonnea systems when compaed to conventona contoes. The basc dffeence between fuzzy and conventona contoes s that the atte ae desgned usng a mathematca mode of the pocess beng contoed. On the contay, fuzzy contoes ae based on the synthess of the knowedge whch s povded by human epetse to constuct a set of ues (n the fom of IF THEN statements) [9]. Dependng on the stuctue of the ues, two types of fuzzy contoe can be dstngushed: fuzzy eatona and fuzzy functona modes [95]. In the functona fuzzy contoe poposed by Takag and Sugeno [96], the ue consequents ae csp functons of the ngustc nput vaabes cacuated usng a weghtng method, wheeas by eatona fuzzy contoes, the mappng fom the nput to the output ngustc vaabes s epesented by a fuzzy eaton. The most wdey used eatona fuzzy mode s the Mamdan mode [97] ustated n fgue 7.5. Input Fuzzfcaton Infeece Defuzzfcaton Output Database: Lngustc vaabes, type and paamete of membeshp functons Rue base: ngustc If-Then Rues wth pemses and concusons Fgue 7.5: Stuctue of eatona fuzzy contoe. Because no mathematca mode fo the open oop object ecognton system s avaabe, whch s necessay fo cassca conto methods, the system s contoed by a fuzzy mode desgned to seect one of the feedbacked tansfomatons. The seected tansfomaton s used to poduce new mode mage fo matchng opeaton n the net teaton. Page 96

106 Fuzzy Based Cosed Loop Conto System fo Object Recognton Fo each channe (Mama and Mnma) of the object ecognton system, the eo e ma/ mn (dssmaty between the tansfomaton and the dentty mat computed accodng equaton 7.5) and the devaton of the eo ae chosen as nputs: e ma/ mn e ma/ mn e ma/ mn e Ds T ma/ ma/ mn mn, I k e k ma/ mn (7.7) whee I s the dentty tansfomaton gven by. I (7.8) The output s defned as a quaty nde, whch s a ea vaue n the ange [, ] epesentng how coect the coespondng affne tansfomaton s estmated. Once the quaty nde has been computed fo both channes, they ae compaed and the tansfomaton coesponded to the hghest quaty nde s seected fo the net matchng teaton as ong as the temnaton ctea ae not met. Fgue 7.6 pesents the bock dagam of the poposed fuzzy contoe. I T ma k T mn k k Ds T, I ma ma k mn k k Ds T, I mn Z Z e ma k e ma k mn k mn k Seecto e ma k e mn k e k e ma k e mn k Fuzzy Contoe Fuzzy Contoe ma mn Tansfomaton Seecto Fgue 7.6: Fuzzy- based system fo affne tansfomaton seecton. Geneay, the fuzzy contoe conssts of thee man stages: the fomaton of membeshp functons (fuzzfcaton), the defnton and the evauaton of fuzzy ues (Infeence) and seectng defuzzfcaton method (defuzzfcaton). Page 97

107 Fuzzy Based Cosed Loop Conto System fo Object Recognton Page Fuzzfcaton In fuzzfcaton, the csp nputs ae conveted nto fuzzy nputs usng the coespondng membeshp functons n the knowedge base. The seecton of membeshp functons depends on many aspects. Fo eampe, the use of Gaussan membeshp functons fo specfyng fuzzy sets s desed n many appcatons because they ehbt popetes that ae contnuousy, whch factates senstvty anayss ove the obtaned fuzzy nfeence system [98]. If the goa s to obtan smpe nea ntepoatons and smpe numeca evauatons, the tangua and tapezod membeshp functons ae pefeed. Fgue 7.7 ustates thee dffeent types of membeshp functons used fo fuzzfcaton. Fgue 7.7: Thee types of wdey used membeshp functons: (a) tangua, (b) tapezod, and (c) Gaussan type membeshp functons. The mathematca fomuas of tangua, tapezoda and Gaussan membeshps ae gven by the foowng equatons: In the poposed mode, tangua shape s seected as the man membeshp functon. But a few tapezoda membeshp functons ae used at the magna anges. e e m e e s s m s s Tan e e e e s s s s Tap m Tan e (7.9) Tan s m e s e Tap m G (a) (b) (c)

108 Fuzzy Based Cosed Loop Conto System fo Object Recognton The membeshp functons used fo fuzzfcaton and the anges of each nput and output ae pesented n Fgue The ange vaues ae detemned epementay. Fo each nputs, thee ngustc vaabes ae used: (S: sma, M: medum, and L: age fo the eo e ma/ mn ) and (Z: zeo, N: negatve and P: postve fo the eo devaton e ma/ mn ), whe fo the output fve ngustc vaabes ae defned as: vey sma (VS), sma (S), medum (M), age (L) and vey age(vl). e e y S M L N Z P VS S M L VL y Fgue 7.8: Input and output membeshp functons and the anges. The membeshp functons fo each vaabe consdeed n deveoped system and the defaut mt vaues coespondng to % and % of cetanty fo each membeshp functon ae stoed n the ngustc database as ustated n Tabe 7.. Tabe 7.: The database of ngustc vaabes. e Tangua s m e Tapezoda s e S M 8 L 8 e Tangua s m e Tapezoda s e N Z -6 6 P 6 Tangua s m e Tapezoda s e VS - -,5 Page 99

109 Fuzzy Based Cosed Loop Conto System fo Object Recognton S,5,5 M,5,5,75 L,5,75 VL, Infeence The eatonshp between the nputs and the outputs n a fuzzy mode s chaactezed by a set of ngustc statements caed as fuzzy ues [99]. They ae defned based on the human epet knowedge and obsevatons fom epementa wok. The numbe of fuzzy ues n a fuzzy system s eated to the numbe of nputs and the numbe of fuzzy sets fo each nput vaabe. In ths study fo each channe, thee ae thee nput vaabes, each of whch s cassfed nto thee ngustc vaabes. Theefoe, the numbe of ues fo ths mode s set to 9. The epementa and epet knowedge of the mode s descbed n the tabe 7.. Tabe 7.: Rue base of poposed fuzzy contoe. ma/ mn e ma/mn S M L ema/mn N M S VS Z L M S P VL L M The fuzzy ues ae used n fuzzy conto n ode to defne the map fom the fuzzfed nputs of the fuzzy contoe to ts fuzzy outputs []. In ths mode, knowedge s ntepeted IF THEN ues and mutpe statements ae joned by AND connectve. The fuzzy ues n ngustc fom ae shown n Tabe 7.3. Tabe 7.3: Fuzzy-epet ues n ngustc fom Rue IF (N s L) AND (e s S) AND (e s N) THEN( s M) Rue IF (N s L) AND (e s S) AND (e s Z) THEN( s L) Rue 3 IF (N s L) AND (e s S) AND (e s P) THEN( s VL) Rue IF (N s L) AND (e s M) AND (e s N) THEN( s S) Rue 5 IF (N s L) AND (e s M) AND (e s Z) THEN( s M) Rue 6 IF (N s L) AND (e s M) AND (e s P) THEN( s L) Rue 7 IF (N s L) AND (e s L) AND (e s N) THEN( s VS) Page

110 Fuzzy Based Cosed Loop Conto System fo Object Recognton Rue 8 IF (N s L) AND (e s L) AND (e s Z) THEN( s S) Rue 9 IF (N s L) AND (e s L) AND (e s P) THEN( s M) The ues that have eacty the same consequences must be combned nto a snge ue wth OR-opeatos so that fo each output ngustc vaabe, thee s eacty one consequent fo each possbe antecedent n the ue base: Tabe 7.: Combned fuzzy-epet ues. R VL IF (Rue3.) THEN( s VL) R L IF (Rue OR Rue 6) THEN( s L) R M IF (Rue OR Rue 5 OR Rue 9) THEN( s M) R S IF (Rue OR Rue 8) THEN( s S) R VS IF (Rue 7) THEN( s VS) The fuzzy nfeence method used s defned by a combnaton of two opeatos, the dsjunctve and conjunctve opeatos. In teatues, many such opeatos ae avaabe []. The most common used dsjunctve and conjunctve opeatos ae the AND- and the ORopeatos fo the conjuncton and the dsjuncton espectvey. Rues actvaton degees ae computed by evauatng the mnmum between two membeshp degees that ae combned wth AND-Opeato and the mamum between membeshp degees that ae combned wth OR-opeato. and y MIN, o y MAX, y y (7.) The actvated ues ae aggegated nto one fuzzy set fo the output vaabe by evauatng the mamum between ues actvaton degees Defuzzfcaton Output of a fuzzy pocess needs to be a snge scaa quantty as opposed to a fuzzy set. Defuzzfcaton s the conveson of a fuzzy quantty to a pecse quantty. In teatues, many defuzzfcaton methods have been poposed by nvestgatos n ecent yeas, such as: centod method, weght aveage method, mean of ma.-membeshp method, cente of sums method, cente of agest aea method, fst (o ast) of mama method. The seecton of the defuzzfcaton technque s ctca and has a sgnfcant mpact on the speed and accuacy of the fuzzy mode. In ths mode, centod of aea defuzzfcaton method s used because t has been used geneay and gves moe eabe esuts than the othes [][]. In ths method, the esutant membeshp functons ae deveoped by consdeng the unon of the output of each ue, whch means that the oveappng aea of fuzzy output sets s counted ony once, povdng moe esuts. Page

111 Fuzzy Based Cosed Loop Conto System fo Object Recognton VS S M L VL Fgue 7.9: Gaphca epesentaton of centod aea method. Fgue 7.9 shows the basc gaphca epesentaton of cente of aea defuzzfcaton method. In ths Fgue, the shape efes to the emanng aea of actve fuzzy sets that ae contoed by the eated fuzzy ues. The cente of gavty of the shape s mathematcay obtaned by the foowng equaton: e s e s d d (7.) 7.5. Epementa Resuts To evauate the pefomance of the poposed system, many epements wee conducted on dffeent pas of mages (mode and quey mages) of standad mage database [3] and ea wod mages. Each mode mage pesents snge taget object, whe the coespondng quey mage ncudes the taget object captued n cutteed backgound unde dffeent condtons (umnaton, vewpont, pata occuson). The system s evauated on mage pas of the database [3] ( two eampes ae shown n Fgue 7.) and ea wod mage pas fom wokng scenaos of the obotc system FRIEND II acqued wth ts steeo camea system (an eampe s shown n Fgue 7.). Page

112 Fuzzy Based Cosed Loop Conto System fo Object Recognton Fgue 7.: Two eampes of the database mages (eft coumn) mode mages, (ght coumn) quey mages. Mode mage quey mage Fgue 7.: An eampe of used ea wod mages. Page 3

113 Fuzzy Based Cosed Loop Conto System fo Object Recognton The pose estmaton of the object epesented by the mode mage n the scene epesented by coespondng quey mages s done paae fom two ndependent matchng channes (Mama and Mnma SIFT featue matchng). The matchng pocess s epeated unt both estmated poses ae neay equa. Because both poses ae povded fom dffeent ndependent nfomaton channes and each of them conssts of 6 ndependent paametes, the equaty of both poses means that both ae coecty estmated wth an eo owe than the dffeence. The esuts of pose estmaton fo some eampes ae sted n tabe 7.5. As evdent fom Tabe 7.5, the postona eos (the eo of the tansatons aong -as and y-as of the camea coodnate system) s ess that mm, whe the eo of the tansaton aong optca as s ess than 3 mm. The angua eos,.e. ptch, o and yaw ange eos ae ess than,5 degee. Tabe 7.5: Compason between object poses estmated by Mnma and Mama SIFT matches. Pose estmated fom Mama coespondences Pose estmated fom Mnma coespondences T T y T z T T y T z 3,76,8 3, -5,77 7,57 3,,7,57-5,7,63 7,3,59,38 57,3-6,77 8,3 6,,57,66 57,35-6, 9,9 6, -,87 -,9 5,36 6, -,56 -,3-5,6 -,3 56,5 5,5 -,78 -,37-9,9 8,79,35-6,76,77 -,5-8,99 8,6 9,6-65,5,79 -,6-6,8 7,69 97,7-6,33,6 -,7-6, 7,3 96,66-63,6,83 -,67 56, -,5,9 6,,,6 57, -,3 3, 5, 3,5 9,98 5,99, 9,9 -,9 -,9 7, 6,9,3 9,,7 -,9 73,58 An eampe fo the pogess of the mage matchng and pose estmaton of taget object (coffee fte package) s ustated n Fgue 7.6. One notes that the accuacy and convegence ate of estmated pose has been mpoved dung teatons. Page

114 Fuzzy Based Cosed Loop Conto System fo Object Recognton Iteaton : no pose s estmated because ony 3 matches ae found, but one affne tansfomaton s estmated, whch s used to wap the mode mage n the net teaton. Iteaton : E=63.69, Ey=5.9, Ez=75.8, E=38., E=9., E=6.76. Iteaton 3: E=.5, Ey=.395, Ez=.95, E=., E=.8, E=. Page 5

115 Fuzzy Based Cosed Loop Conto System fo Object Recognton Iteaton :E=.39, Ey=.5, Ez=.9, E =.59, E =.7, E =.5 Fgue 7.: update of mage matchng and pose estmaton esuts dung tme. Left mage matchng esut and ght ts coespondng pose estmaton esut. In each teaton, the tansaton eos (E, Ey and Ez n mm) and otaton ange eos (E, E and E n degee) ae sted. Note that the numbe of matches s nceased, the dffeence of the both estmated poses s deceased and convegence to the pose of taget object. Page 6

116 Fuzzy Based Cosed Loop Conto System fo Object Recognton Fgue 7.3: Matchng and pose esuts of the fna teaton fo some mode and quey mage pas. Page 7