GPU Accelerated Elevation Map based Registration of Aerial Images

Transcription

1 GPU Aelerated Elevaton Map based Regstraton of Aeral Images Joseph Frenh, Student Member, IEEE, Wllam Turr, Joseph Fernando, Member, IEEE and Er Balster Senor Member IEEE {joseph.frenh, Wllam.turr, Abstrat Ths paper proposes a lower lateny mplementaton of the georegstraton algorthm proposed by [5]. The algorthm has been modfed to mtgate the regstraton errors and has been parallelzed to map to a Graphal Proessor Unt (GPU). Also, the target mage offset and the pantng value omputatons have been ombned to a sngle loop to elmnate the use of shared memory. Modfatons to a urrent wdely used algorthm are proposed. The proposed modfed algorthm has been mplemented n ompute unfed deve (CUDA) arhteture to redue lateny. A fxed oordnate system s used to represent the mage, foal, and projeton planes. Expermental results show that the proposed algorthm s apable of generatng aurate georegstered mages for hgh flyng arborne vehles. Whle ths method has been tested usng aeral photographs, t an be extended to Satellte mages as well as other mage data. A speedup of over 10x has been aheved over the CPU verson. Index Terms Camera model, Dgtal Elevaton Maps, multple threads, Georegstraton, Orthoretfaton I. INTRODUCTION Ths paper s based on the methodology proposed by Jovanov et. al [5]. The orgnal algorthm has been modfed to mtgate errors and has been enhaned to map to a Graphal Proessor Unt (GPU) to lower lateny. Note that regstraton s the hghest lateny ausng funton n most arborne system mage proessng software han. It has a hgher lateny than preproessng and mage ompresson. The man reasons for seletng ths methodology are, frstly, the algorthm ould be mult threaded and parallelzed, seondly, t ould be modfed to ft the resoure n the GPU namely, the shared memory, and thrdly, t ould be hanged to mtgate errors to be wthn the requred threshold. In ths algorthm, to automatally regster aeral mages, a amera n whh the amera model s known, and a fxed oordnate system ndependent of the dreton of flght to represent the mage, foal and projeton planes s used. Global poston system (GPS)/Instrument navgaton system (INS) data n onjunton wth dgtal elevaton maps (DEM) are used, to perform aurate orthoretfaton and geo regstraton smultaneously, along wth the ntellgent seleton of parameters, to redue lateny and mtgate errors. The requred equatons are desrbed n detal, and the log leadng to these equatons have been explaned extensvely. The modfed algorthm targetng a GPU has been fully mplemented usng a Wndows based omputng system, usng the C language n a CUDA envronment. Ths exeutable program has been used to generate aurate orthoretfed and geo regstered maps usng many nput mages. A speedup of over 10x has been aheved over the CPU verson. Whle ths proposed methodology has been tested usng only aeral photographs aqured from hgh flyng arborne vehles, t an be extended to Satellte mages, as well as, other ases, easly. Many researhers have studed general mage regstraton [1]-[6], [8]-[21] extensvely durng the past deades wth revews of the methods provded n [4] [19]. These also gve the work done n the areas related to regsterng aeral mages. The orgnal algorthm has been used for varous applatons and s explaned n [1],[3] and [5]. Many others researhers have addressed regstraton of aeral mages [8] and have dealt wth orthoretfaton and georegstraton [9, 10] separately, as well as, a omposte problem. Shekh et. al. [12] has proposed orthoretfaton of aeral mages usng elevaton maps and a amera model. Gabor features of the orthoretfed mages are deteted and normalzed orrelaton s obtaned between a referene frame and the urrent frame. Feature lnkng loal regstraton and dret regstraton s performed and an adjustment for the amera model has been norporated n the proposed method. Three proesses for data retfaton (orthoretfaton), establshment of orrespondene, and model update are used n ths method. Shekh et. al. [13] has proposed somewhat of a smlar method to [12], usng elevaton maps and feature based orrespondene. An teratve method has been proposed to optmze global smlartes and a gradent based optmzaton algorthm has been used. Xong et. al. [16] has proposed a Harrs orner based approah that does not requre establshng any orrespondene. Yasen [17] has proposed to ompare smlartes of the referene and a urrent mage to establsh a orrespondene and regster them to eah other. Many have researhed the problem of regsterng an mage frame to another mage. In [8] a feature mathng to a prevous frame and lnear optmzaton method based on the Levenberg-Marquardt optmzaton algorthm has been

2 explaned. In [10] a general method of mathng geometr features has been explaned for any mage that ould be used for aeral mages also. In [6], a log polar transform based method to regster mages to regster gener mages. The advantage of usng log-polar over the Cartesan oordnate representaton s that any rotaton and sale n the Cartesan oordnates s represented as shftng n the angular and the log-radus dretons n the log-polar oordnates, respetvely. In [11], a smlar log polar transform s ombned wth the phase orrelaton to regster mages. In ths proposed method, the sale and the rotaton angle between the referene and the sensed mage s omputed and appled to obtan the regstered mage. Due to the lmted reovery of the rotaton angle by the Fast Fourer Transform and Log polar Transform n the frequeny doman, the regstraton method proposed by Zoka et al. [20] s performed entrely n the spatal doman. Ths work desrbes a novel tehnque to reover large smlarty transformatons (rotaton/sale/translaton) and moderate perspetve deformatons among mage pars. It ntrodues a hybrd algorthm that features log-polar mappngs and nonlnear least squares optmzaton usng the Levenberg Marquardt algorthm. In ths method the translaton parameter s reovered by usng the oarse-to-fne mult-resoluton framework, and the sale and rotaton parameters are obtaned by mathng the log-polar transformed mages usng a rossorrelaton funton. In [18] the haratersts of thn-plate splne, mult-quadr, peewse lnear and weghted mean transformaton funtons are explored and ther performanes n the regstraton of mages wth nonlnear geometr dfferenes are ompared for the purpose of regsterng two mages. The effetve use of GPS/INS data for mage regstraton has been shown by Sh et. al. n [15] and a hstogram based method has been explaned n [16]. II. PROPOSED METHODOLOGY A. CUDA Programmng [7] Compute unfed deve arhteture (CUDA) s a framework that allows users to map algorthms to Nvda GPUs and aelerate to lower the lateny. In ths unfed arhteture, all the hardware funtons of the GPU an be aessed through a sngle programmng model. CUDA ompatble GPUs are sngle nstruton multple data (SIMD) mahnes. They are apable of aeleratng loop bodes that do not have any nterloop data dependenes. Furthermore, algorthms that are order O(n 3 ) or more are very sutable to aelerate and hgher speedup an be aheved. The programmng model s based on ntatng multple hardware threads that exeute the same soure ode kernel on the GPU to leverage the benefts of the SIMD arhteture. There s a two level herarhy to spefy threads namely the blok and the grd. The sze of the blok, defnes the number of threads assgned to sngle mult proessor (MP) and exeuted n any stream proessor (SP) n that MP. The grd defnes the number of bloks that are n an algorthm. Lower lateny s aheved by omputng the threads n parallel n multple MPs. It s logal to assgn a number of threads that s greater than double the number of SPs n a MP to a MP. When a thread s aessng memory, whh s hgh lateny operaton, t an exeute another thread. At the hghest level of the memory herarhy s global memory, whh s used to ommunate wth the host memory. Shared memory s at the next level of the herarhy, and faltates the ommunaton between SP n a sngle MP. Loal memory s at the lowest level of the herarhy, and s reserved for loal varable wthn a SP. Varable assgned to the loal memory of an SP are not aessble from other SPs n a MP. Arrays that are oped from the host memory are loaded to the shared memory n sutable blok sze and all the SPs an have aess to that partular blok of the nput array. Basally, all the threads assgned to a MP an aess the bloks of memory of eah array that s assgned n the shared memory. The man reason to load a blok to the shared memory s that aessng shared memory auses lower lateny than aessng the global memory. Ths s one of the advantages of programmng n CUDA. By storng arrays n shared memory, global memory aesses are redued, and onsequently overall lateny s redued. However, whle there s a very large (n G. Bytes) of global memory, the avalable shared memory s very lmted (~2 MB). Threads assgned to a MP an be synhronzed to wth eah other by usng a thread barrer mehansm. At present, there s no mehansm to synhronze the thread n dfferent MPs. Therefore, t s essental that there be no data dependeny between bloks. Coalesed memory wrtng to the global memory s attempted, where ever possble, to redue lateny. Basally, all the threads n an MP should be synhronzed, before wrtng bak the outputs to the global memory. B. Enhanements to the orgnal methodology The followng enhanements were ntrodued to the orgnal ode proposed n [5]. These were done manly to mtgate regstraton errors, mprove parallelsm and redue lateny. 1. Changed the alulaton to determne the appromate sze of the pxel n the Earth Coordnates (redued under samplng) 2. Re-arranged the loops to ahe the memory more effently (lower lateny) 3. Removed nter dependeny between loops (for mproved parallelsm) 5. Fxed omputatons around edges (to mtgate errors) 4. The offset and the pantng value omputatons were ombned to be done n one loop. Ths redued memory and shared memory requred for the CPU and GPU mplementatons, respetvely. C. Image Projeton A georetfed mage has been projeted onto an Earth oordnate system whh gves a geo-loaton for eah pxel. However, the georetfed mage an ether have a perspetve that s not orthogonal to the projeton plane, or any heght assoated wth the loatons obtaned from some elevaton map. In general, Orthoretfaton s the proess of projetng

3 an mage so that the perspetve vew angle hanges to be orthogonal to the sene. However, when aeral mages are orthoretfed, they are typally projeted onto an Earth based map and assoated (DEM), along wth removng the perspetve dstortons. The result s essental for reatng more meanngful data from aeral magery as t an then be ted to a loaton. There are two types of projeton used for orthoretfaton. The frst type s forward projeton, where an mage oordnate s mapped to a projeton plane oordnate. Forward projeton sn t used wdely for orthoretfaton, espeally wth aeral magery beause the resultng pxels are not evenly spaed. Therefore, bak projeton s used more frequently. For bak projeton a world oordnate loaton s projeted onto the mage plane. The result of bak projeton s an evenly spaed mage. In photogrammetry, the oordnate system, and the assoated rotaton angle defntons are shown below n Fgure 1. The varables at the orgn of the oordnate system are the magng system enter denoted by (X, Y, Z ). amera foal length, f, the world oordnate beng projeted, (X,Y,Z), the magng system enter, (X, Y, Z ), and the ndvdual elements of the transform matrx n Equaton 1. m11( X X) + m12( Y Y ) + m13( Z Z) = f (2) m ( X X ) + m ( Y Y ) + m ( Z Z ) m21( X X ) + m22( Y Y ) + m23( Z Z) j = f (3) m ( X X ) + m ( Y Y ) + m ( Z Z ) 31 C. Iteratve Algorthm Desrpton 32 The followng equatons are based on the methodology desrbed n [5]. Before the orthoretfaton begns, the amera s albrated, whh determnes the foal length and ntal extrns amera parameters. One an mage s aptured wth the orrespondng alttude and GPS loaton, the extrns parameters an be omputed and orrespond to the transform matrx elements. The frst step n the orthoretfaton proess s to determne the regon of nterest (ROI) of the pre-loaded elevaton map. In ths step, the orners of the mage plane (the mage footprnt) are forward projeted onto the elevaton map. The ROI s then used to estmate the requred pxel sze, to redue the loss of data durng the projeton proess. The pxel sze estmated, determnes the nterpolaton fator requred for the rest of the orthophoto reaton. As Fgure 2 shows, the DEM has a relatvely oarse samplng. From the oarse samplng, the ROI requres a fner samplng. The fner samplng s aomplshed usng a blnear nterpolaton method Fgure 1. Projeton Geometry The rotatons an be multpled together usng the standard rotaton matres, desgnated at R to form an all enompassng transform matrx, M, and shown n Equaton 1. The world oordnate system an be multpled by M to ompute the orrespondng mage oordnates. m 11 m12 m13 M = m m m = R RR m m m ω φ κ One reason ths relatonshp works s that the magng system enter, the world oordnate, and the orrespondng mage plane loaton all are on a lne. Ths relatonshp an be modeled usng the ollnearty equatons shown n Equatons 2 and 3. The mage plane ndes, and j, are a funton of the (1) Fgure 2. Relatonshp between the Elevaton map, Regon of Interest (ROI) and Image footprnt. Fgure 2 shows the relatonshp between DEM, the ROI, and the mage footprnt. The equaton used for the omputaton of the nterpolaton fator, I f, s shown n Equaton 4. The sde of the mage footprnt nearest the magng system s measured n terms of the DEM samplng, gvng a dstane, r, n meters. The dstane s then dvded by the number of mage pxels along the orrespondng sde, N r. One ssue s that the nterpolaton fator determned from the nearest sde an lead to over samplng and alasng artfats along the furthest footprnt sde. To mtgate the alasng

4 artfats at the opposte sde of the footprnt, s to sale the nterpolaton fator to a slghtly hgher value, n ths ase a fator 1.2 s emprally determned to be the most sutable. r If 1. 2 N = (4) The ollnearty equatons shown n Equatons 1 and 2 are deal for parallelzaton as eah resultng pxel loaton s ndependent of the other pxel loatons. However, when performng the omputatons, lower memory aesses typally redue the omputatonal lateny. To redue the amount of memory aesses, all of the nterpolated pxels, wthn a sngle DEM pxel are projeted, before a new set of elevaton map pxels are loaded. The ollnearty equatons are modfed as shown n Equatons 17 and 18. The dstanes are alulated for eah nterpolated loaton wthn the ROI n all three dretons, D x for the dstane from the magng system n the horzontal dreton, D y for the dstane n the vertal dreton, and D z for the hange n alttude. The nterpolated ndes are ndated usng the subsrpt. m11dx[ ] + m12dy[ y ] + m13dz[ ] [ ] = f (17) m D [ x ] + m D [ y ] + m D [ x, y ] 31 x 32 m21dx [ ] + m22dy[ y ] + m23dz [ ] j[ ] = f (18) m31dx[ ] + m32dy[ y ] + m33dz [ ] The nput mage, amera model, the entre elevaton maps and ROI oordnates are passed as parameters. The output s the orthoretfed mage. The psuedoode for the urrent mplementaton on the CPU s shown below. for y north 0, max_pxels North -1 //of the elevaton map ROI Calulate ntal D y for x east 0: max_pxels East -1 //of the elevaton map ROI - Compute the offset of the startng pxel of eah blok Reset D y Calulate ntal D x Load DEM pxels for γ north 0:INTERPOLATION_FACTOR -1 Reset D x Calulate ntal D z for χ East 0: INTERPOLATION_FACTOR -1 Calulate Calulate j Compute the orrespondng ntensty value for pantng Inrement D x, D z endfor χ; Inrement D y endfor γ; endfor x; endfor y; r y 33 z IV Implementaton and Expermental results A. Mult threaded algorthm The teratve algorthm gven above was mult threaded targetng a CUDA envronment. Note that the nter loop data dependenes have been removed and the ode s parallelzable usng multple threads. The omputatons for eah DEM loaton an be parallelzed for a square wndow of INTERPOLATION_FACTOR sze on eah sde (wdth and heght). Ths an be used as the blok sze to spawn that many threads n the CUDA envronment. The resoures on the GPU do not support a very large INTERPOLATION_FACTOR f t s used to generate the number of threads. A onstant multpler fator was onsdered. Ths was also not attratve as the INTERPOLATION_FACTOR was a varable and ould be any value upto 255. Therefore, (INTERPOLATION_FACTOR x1) threads were used n the CUDA envronment, nstead of the possble (INTERPOLATION_FACTOR x INTERPOLATION_FACTOR). Ths results n a loop n the kernel, whh exeutes INTERPOLATION_FACTOR number of loops. Note that the mamum number of thread on a proessor has been nreased to from 768, for GTX 6xx seres (Kepler) GPUs and later. However, to ensure ompatblty wth older ards ths ondton has been mantaned n the present mplementaton. The grd sze was set to the sze of the ROI of the elevaton map n the East and North dreton respetvely. The nput mage, and the elevaton map were oped to the GPU (global) and memory was alloated to return the resultng mage. Some other requred parameters, suh as, the amera model, the GPS oordnates of the amera, were also passed to the GPU. The strategy that has been used to map the ode on the GPU s to use the shared memory and the loal memory, as muh as, possble. The 2x2 array for the loalzed DEM values, the D E are alloated n shared memory. D N s alloated an array of sze INTERPOLATION_FACTOR n shared memory. All the other varables are assgned to loal memory. Ths memory alloaton enables regstraton wth even larger nterpolaton fators to be omputed on the GPU. These varables are aessed by all the threads assgned to a MP. The pseudo ode of the kernel runnng on the GPU s gven next.

5 nt k= threadidx.x; //range 0:nterpolaton_fator-1 f(k==0) { Reset D y Calulate D x and save n shared memory Load DEM pxels and save n shared memory } synthread(); //Synhronze all the threads Reset D x Calulate ntal D z for χ East 0: INTERPOLATION_FACTOR -1 Calulate ; Calulate j; Compute the orrespondng ntensty value for pantng Inrement D x, D z ; endfor χ; The offset of the resultant mage where the panted value s store s omputed n the kernel. Ths offset s heked f t s wthn the borders of the resultant mage. If t s wthn, the panng value s stored. Sne every MP omputes dfferent offsets to the resultng mage there s no possblty of oalesed memory wrtes. Basally, there s not need for all the threads to be synhronzed before mage savng. The allng routne runnng on the CPU s as follow. Frst, the memory s alloated on the deve for the mage data, the elevaton map and the results. The elevaton map s oped to the deve only one n many regstraton mage alls. Seond, the mage and the elevaton map are oped to the deve. Thrd, the number of threads that must be ntalzed on the blok and the grd are set and the deve based regstraton funton s alled. Fourth on the return, the results are oped from the deve the host. Note that the sze of the memory of the mage, the elevaton map and the results are dfferent. They are not the same sze n any gve senaro. //Alloate memory for the mage, elevaton map //and the results and opy the arrays. udamallo((vod**) &d_mg_data, mg_mem_sze); udamallo((vod**) &d_terran_data,terran_mem_sze); udamallo((vod**) &d_results, results_mem_sze); udamempy(d_mg_data, mg_data, mg_mem_sze, udamempyhosttodeve); udamempy(d_terran_data, terran_data, terran_mem_sze, udamempyhosttodeve); dm3 blok_threads(nterpolaton_fator, 1); dm3 grd(max_pxels East, max_pxels North ); //Call the GPU routne regstraton_gpu<<<grd, blok_threads>>> (d_mg_data, d_terran_data, d_results, other_params); //Copy the results to host udamempy(gpu_results, d_results, results_mem_sze, udamempydevetohost); B. Expermental Results Varous tests were onduted usng dfferent nput mages and were regstered. In one test, an mage of sze 9000x9000 of haraters and a terran of sze 7202x4501 floats were used. Usng a ROI of 162x127 and an INTERPOLATION_FACTOR of 81, resulted n an mage of sze (162x81)x(127x81), whh s 13122x Ths mplementaton and the CPU based mplementaton were ompared for lateny. The lateny for the CPU verson was 4.15 seonds and the GTX 580 GPU verson, the lateny was 0.4 seonds. Ths gave an average speedup of 10.3x over the CPU verson runnng on a 3.4 GHz Quad ore Xeon based mahne. V Conluson Ths paper gves extensve detals of the proposed mplementaton. The tests onduted seem to prove that the GPU verson s well suted for robust aeral mage regstraton. The proposed GPU targeted mplementaton of the methodology of ths paper s effetve n redung the lateny for aeral mage regstraton funton. More testng s requred to asertan the robustness for varous nput mages, and another verson should be developed targetng the Kepler ard and n OpenCL as future works. REFERENCES [1] Douglass A. Alexander, Robert G. Deen, Paul M. Andres, Payam Zaman, Helen B. Mortensen, Amy C. Chen, Mhael K. Cayanan, Jeffrey R. Hall, Vadm S. Klohko, Oleg Parser, Carol L. Stanley, Charles K. Thompson, and Gary M. Yag. "Proessng of Mars Exploraton Rover magery for sene and operatons plannng," Journal of Geophysal Researh, vol. 111, [2] Emmanuel Chrstophe, Julen Mhel, and Jord Inglada. Remote Sensng Proessng: From Multore to GPU. IEEE Journal of Seleted Tops n Appled Earth Observatons and Remote Sensng, vol 4, ssue 3: , September [3] Kahang D and Rongng L. CAVHOR amera model and ts photogrammetr onverson for planetary applatons. Journal of Geophysal Researh, vol. 109, [4] A. Goshtasby. 2-D and 3-D Image Regstraton for Medal, Remote Sensng, and Industral Applatons, New York, Wley Press, [5] Veljko M. Jovanov, Mhael M. Smyth, Ja Zong, Robert Ando and Graham W. Bothwell, "MISR Photogrammetr data reduton for geophysal retrevals," IEEE Transatons on Geosene and Remote Sensng, vol. 36, ssue 4, pp , July [6]Rttavee Matungka, Yuan F. Zheng, Robert L. Ewng, "Image Regstraton Usng Adaptve Polar Transform," IEEE Transatons on Image Proessng, Vol. 18, no. 10, pp Ot [7] Nvda Corporaton. "CUDA C Programmng Gude." [May 12, 2013]. [8] Clark F. Olson, Adnan I. Ansar, Curts W. Padgett. "Robust Regstraton of Aeral Image Sequenes," n Advanes

6 n Vsual Computng Leture Notes n Computer Sene, Volume 5876, G Bebs et. al.(eds), Berln Hedeleberg: Sprnger, 2009, pp [9] Clark F. Olson. "Image regstraton by algnng entropes." n Proeedngs of the IEEE Computer Soety Conferene on Computer Vson and Pattern Reognton. Volume 2. pp , De [10] Clark F. Olson. "A general method for geometr feature mathng and model extraton." Internatonal Journal of Computer Vson. Volume 45, Sprnger, 39-54, [11] Jgnesh N. Sarvaya, Suprava Patnak, Kajal Kothar. Image Regstraton Usng Log Polar Transform and Phase Correlaton to Reover Hgher Sale, Journal of Pattern Reognton Researh 7, 2012, pp [12] Yaser Shekh, Sohab Kahn, Mubarak Shah. "Featurebased geo regstraton of aeral mages", Anthony Stefands, Slva Nttel (Eds), n Geosensor Networks, Boa Raton, Florda, USA, CRC Press, 2004, pp [13] Yaser Shekh, Sohab Kahn, Mubarak Shah, Rhard Cannata. Geodet Algnment of Aeral Vdeo Frames, Mubarak Shah Rakesh Kumar (Eds), n Vdeo Regstraton, Vdeo Computng Seres, Kluwer Aadem Publshers, pp [14] Juan Sh, Jnlng Wang, Yamng Xu, "Use of GPS/INS observatons for effent mathng of UAV mages", In Proeedngs of the Internatonal Global Navgaton Satellte Systems Soety IGNSS Symposum, [15] H. Tuo, L. Zhang, Y. Lu. Mult sensor aeral mage regstraton usng dret hstogram spefaton. n Proeedngs of the IEEE Internatonal Conferene on Networkng, Sensng and Control. 2004, pp [16] Y. Xong, F. Quek. Automat aeral mage regstraton wthout orrespondene. n Proeedngs of the 4th Internatonal Conferene on Computer Vson Systems [17] M. S. Yasen, P. Agathokls. "A robust, feature-based algorthm for aeral mage regstraton," n Proeedngs of the IEEE Internatonal Symposum on Industral Eletrons , [18] L. Zagorhev and A. Goshtasby, A omparatve study of transformaton funtons for nonrgd mage regstraton, IEEE Trans. Image Proessng, vol. 15, no. 3, pp , [19] Ztova, J. Flusser. "Image regstraton methods: A survey," n Image and Vson Computng, Volume 21, J. M. Frahm, M. Pant et. al.(eds), Elsver, 2003, pp [20] S. Zoka and G.Wolberg, "Image regstraton usng logpolar mappngs for reovery of large-sale smlarty and projetve transformatons," IEEE Trans. Image Proessng, vol. 14, no. 10, pp , Ot [21] S. Zoka and G.Wolberg, "Robust Image Regstraton Usng Log-polar Transform," In Proeedngs of the IEEE Internatonal Conferene on Image Proessng, September, vol. 1, pp , 2005.