Image Registration among UAV Image Sequence and Google Satellite Image Under Quality Mismatch

Transcription

1 0 th Intenational Confeence on ITS Telecommunications Image Registation among UAV Image Sequence and Google Satellite Image Unde Quality Mismatch Shih-Ming Huang and Ching-Chun Huang Depatment of Electical Engineeing National Kaohsiung Univesity of Applied Sciences Kaohsiung, Taiwan Cheng-Chuan Chou Compute & Communications Reseach Laboatoies, Industial Technology Reseach Institute Hsinchu, Taiwan Abstact In this pape, we developed an Unmanned Aeial Vehicle (UAV) image egistation system consisting of UAV image to UAV image egistation, UAV image to Google satellite image egistation, and egistation efinement with a nomalized vaiant of mutual infomation fo quality mismatch poblem We show the limitation of the conventional mutual infomation fo quality mismatch and then suggest using a nomalized vaiant of mutual infomation to efine egistation between UAV image and Google satellite image Expeiments caied out on the ealistic UAV image sequence and Google satellite image show that the developed system could povide bette UAV-to-satellite image egistation and could conque the quality mismatch poblem UAV Figue Illustation of UAV-based applications with compute and Google Eath Keywods-image egistation; mutual infomation; quality mismatch; I INTRODUCTION An Unmanned Aeial Vehicle (UAV) is simple emotely piloted aicafts, but can be contolled to cay out flight plans Nowadays, UAV has been inceasingly employed in vaious applications with compute and Google Eath/map, as shown in Fig In [], a numbe of Unmanned Aeial Vehicles ae developed and opeated fo diffeent applications, such as agicultual and civilian suveillance, militay mission, disaste obsevation, seaching and escue mission etc The main advantages of unmanned aeial vehicles ae twofold: a) UAV is easie and moe suitable because of a significantly less time needed to stat a mission; b) the mission can be pefomed on a lowe cost Moeove, a majo task of UAV is to captue the image sequence of video suveillance egions and geolocalize the objects of inteest To achieve this goal, the coe technique is an image egistation pocess between a video steam and an othophotogaph Image egistation is the pocess of finding the coesponding points between two images at the same scene, taken at diffeent times, fom diffeent views, and by diffeent cameas The difficulty of image egistation has been extensively discussed in the last two decades Although many methods have been poposed [-9], it is still an open poblem The est of this pape is oganized as follows Section eviews the elated woks Section 3 defines the poblem In Section 4, we pesent the poposed system Expeimental esults ae shown in Section 5 Section 6 daws a conclusion and futue wok Figue Illustation of quality mismatch between the (a) UAV image and (b) Google satellite image II (a) RELATED WORKS Numeous eseaches have been poposed to addess the image egistation poblem The scale invaiant featue tansfom (SIFT) [5, 6] based on the distinctive and nonepetitive local featues is a popula method fo image egistation In pactice, the numbe of detected featues used fo image egistation is significantly smalle than the numbe of pixels in the image In [6], Fan et al applied SIFT to find point-wise coespondences between slide images and video fames fo automatically matching and time-aligning electonic slides to videos of coesponding pesentations Moeove, some methods poposed identifying image desciptos of local intensity pattens to egiste successive images such as the Kanade-Lucas-Tomasi featue tacke (KLT) [7] and, the optical flow techniques [8] Hee, both KLT and optical flow have been used in many image egistation applications [9] unde simila quality Although, those image egistation methods pefom well between successive images, they may not be diectly applied to the mapping between a UAV image and the Google satellite image The easons come fom many (b) //$300 0 IEEE 3

2 aspects One majo eason is the image quality of the UAV image and the satellite image is highly mismatched In [0, ], Lin et al have poposed an UAV-based image egistation system SIFT featues ae used fo consecutive UAV image egistation Then, the mutual infomation is used fo efinement between the UAV image and the Google satellite image Howeve, fo a ealistic application, the quality mismatch as depicted in Figue would appea between the UAV image and the Google satellite image and geatly degade the pefomance of image egistation Fom the Figue, we could see that the quality mismatch is mainly aisen fom the sevee blu effect due to low esolution, stong illumination vaiations, and diffeent colo epesentations Thus, in this pape, the majo goal is to develop a obust image egistation system among the UAV image sequence and Google satellite image with the consideation of the quality mismatch poblem In this wok, we fist ty to develop an UAV image egistation system to egiste the UAV image onto the low esolution satellite image, povided by Google Eath/Map Also, to ovecome the quality mismatch poblem, we suggest using a nomalized vaiant of mutual infomation fo similaity measue between the UAV and Google satellite image while egisteing UAV images onto Google satellite image III PROBLEM DEFINITION Given an image sequence fom UAV, I 0, I,, I n, as shown in Fig 3(a), and a Google satellite image M, as shown in Fig 3(b) Thee ae two diffeent tansfomation matices needed in ou wok Fist, the tansfomation between two UAV images can be appoximately fomulated by a homogaphy matix Second, similaly, the tansfomation between an UAV image and the Google satellite image also can be descibed by anothe homogaphy matix Hee, the goal of ou eseach is to estimate the homogaphy matices that can pecisely egiste UAV images onto the Google satellite map with less distotion Howeve, since the geometic distotions might occu in the UAV image sequences, it is difficult to egiste the consecutive UAV images pecisely In addition, since the UAV images and the Google satellite image ae taken at diffeent times by diffeent cameas with diffeent views, the appeaance of the UAV image and the satellite image ae quite diffeent Moeove, the quality mismatch between the UAV images and the Google satellite image makes the egistation among UAV images and Google satellite image even moe difficult It may pefom image egistation between consecutive UAV images by using the scale invaiant featue tansfom (SIFT) techniques In contast, fo UAV-to-Google image egistation, it is a geat challenging task due to quality mismatch, as shown in Fig It should be noted that the quality mismatch addessed in this pape includes diffeent illumination conditions and qualities due to images fom diffeent sensos, and even low esolution in the Google satellite image Moeove, it is woth noting that since the goal of ou wok is to egiste the UAV image sequence onto the Google satellite image coectly, the citical poblem is how to estimate the homogaphy between each UAV image and Google satellite image Even though a good image egistation among consecutive UAV images can be achieved, a good egistation between each UAV image and Google satellite image still cannot be pomised because of quality mismatch (a) (b) Figue 3 Samples of the (a) UAV image sequence and (b) Google satellite image IV SYSTEM Fo system desciption, mathematically, H i, i denotes the homogaphy matix fom I i to I i, so H i, i I i = Ii Moeove, H i, M denotes the homogaphy matix fom I i to M, so H i, M Ii = M i, whee M i is the image that I i pojects onto M To futhe pefom efinement, H is estimated by finding coespondences in the ovelap egion between the UAV image and Google satellite image afte waping I i onto M oughly Finally, M i = H i, M I i, whee H i, M = H Hi, M and M i is the image afte efinement Algoithm shows the algoithmic desciption of the system step by step The following sections will explain the algoithmic desciption in details Algoithm : UAV-to-Satellite Image Registation UAV-to-Satellite Image Registation Algoithm Step 0: Initial condition Estimate H 0, M with human intevention Fo Each UAV Image Step : Consecutive UAV Image Registation Estimate H i, i Step : UAV to Satellite Image Registation Estimate H i, M = Hi, M Hi, i Step 3: Refinement Step 3: Estimate H Step 3: Estimate H i, M = H Hi, M Step 4: Waping I i onto M End 3

3 A Consecutive UAV image egistation Consecutive UAV image egistation is pefomed to estimate H i, i by using the scale invaiant featue tansfom (SIFT) featues, which can povide obust desciptions acoss vaiations in 3D viewpoint and ae invaiant to scale and otation The neaest neighbo citeion is used fo featue matching The RANdom SAmple Consensus (RANSAC) [3], [4] is adopted to emove outlies fo estimating H i, i, as pesented in the Step in the Algoithm This step is impotant since diectly estimating homogaphy matix coectly between the UAV image and Google satellite image is difficult due to geometic distotion and quality mismatch poblem So H i, i is used to achieving pojection oughly, then pefoming efinement by the following step B UAV to Google satellite image egistation To egiste the UAV image onto the Google satellite image, the H i, i and H i, M can be used to estimate H i, M = Hi, M Hi, i, as pesented in the Step in the Algoithm Howeve, as mentioned above, egisteing UAV image sequence onto the Google satellite image is a geat challenging task due to quality mismatch So this ough egistation using H i, i will poduce the accumulated eo due to geometic distotion Thus, a efinement is necessay fo the UAV to Google satellite image egistation On the othe hand, quality mismatch poblem also makes efinement difficult To fine-tune the homogaphy matix between the UAV image and the Google satellite image, the ovelap egion between the UAV image and Google satellite image egistation is used to estimate H, as pesented in the Step 3 in the Algoithm In ode to estimate H, we find coespondences in the ovelap egion by waping M i to highe esolution by H M, i = H i, M since H i, M is invetible By geneating many coespondences in the ovelap egion, the similaity of these coespondences is measued Then the RANSAC emoves the outlies to deive H Finally, the H i, M can be efined as H i, M = H Hi, M, as pesented in the Step 3 in the Algoithm to be an effective method compaed to othe measues such coss-coelation, intensity diffeence In [0, ], the mutual infomation is used, which equies estimating joint histogam of the two images and is computed as MI( X, Y ) = H ( X ) + H ( Y ) H ( X, Y ) () whee X and Y ae two images, H (X ) and H (Y ) ae the entopies of X and Y, and H ( X, Y ) is thei joint entopy The ealistic case addessed in this pape is the quality mismatch between the UAV image sequence and the Google satellite image Thus, we popose to use a nomalized vaiant of mutual infomation (NVMI) MI ( X, Y ) NVMI ( X, Y ) = (3) H ( X ) + H ( Y ) which attains a minimum value when X and Y ae independent In othe wods, when the UAV image and the Google satellite image ae independent, the NVMI yields a minimum value Theefoe, fo finding coespondences between the UAV image and Google satellite image, moe coespondences with high similaity can be given C Similaity Measue The geneated coespondences is measued by maximizing the similaity defined as * J = ag max SIM ( R, R ) () J whee R I denotes the egion centeed at the pixel I, and R I denotes the egion centeed at the pixel J A local seaching aound the pixel J is conducted to find the pixel that yields the maximal SIM Using mutual infomation as a citeion fo medical image egistation has been both theoetically and pactically poved I J Figue 4 Samples of the UAV image (top) and the coesponding satellite image (bottom) Hee we pesent an example of similaity computation by MI and NVMI espectively fo visually obsevation Fig 4 show a UAV image and the coesponding satellite image copped manually, in which the blocks ae divided unifomly The yellow block fom the satellite image is used to compute similaity ove the UAV image As show in Fig 5, we can see 33

4 that NVMI has moe shap similaity distibution than that MI In othe wods, MI has smoothe distibution Besides, NVMI can estimate coect position match between the UAV image and the coesponding satellite image as shown in Fig 4 but MI fails Five image pais of the UAV images and the coesponding Google satellite images copped manually wee used in the expeiment Each image was esized to pixels and divided into 5 image patches, as shown in Fig 4, which wee used to compute similaity fo estimating estimated_p As shown in Table, we can see that the NVMI can get lage the aveage CPE than MI Also, the NVMI pefoms bette than MI consistently So we can expect that the bette coespondences can be estimated by NVMI fo deiving H C Expeimental esults Two UAV image sequences wee used to evaluate the pefomance Each UAV sequence includes five UAV images Fo pefomance compaison, we also compaed ou system using a nomalized vaiant of mutual infomation with the most elated one using mutual infomation poposed in [0], [] Fig 6 shows the esults of the fist UAV image sequence to Google satellite image egistation We can see that ou system pefoms bette than that poposed in [0], [] In the fouth fame, ou system still poduces good egistation wheeas the othe system does not Besides, moe distotion due to accumulated eo appeas in the fifth fame by the othe system Figue 5 Example of similaity computation by MI (top) and NVMI (bottom) espectively fo visually obsevation The cicle denotes the maximum value in each similaity matix TABLE I SAPD COMPARISON OF MI AND NVMI CPE by MI CPE by NVMI 3 3 A Dataset V EXPERIMENTS The UAV image sequences used in the expeiment wee povided by the Industial Technology Reseach Institute (ITRI) [] and wee collected by an UAV, as shown in Fig 3(a) The UAV image is in size of pixels The Google satellite image in size of pixels is fom the Google Map Seve and is the same on the Google Eath/Map No pepocessing pefoms on the both images B Compaison of MI and NVMI To assess the effectiveness of the mutual infomation fo similaity measue unde quality mismatch, we popose a new measue to count coect position estimation (CPE) ove all position diffeence defined as CPE = f ( estimated _ p tue _ p) (4) whee the estimated_p denotes the estimated position, tue_p denotes the gound tue, and f ( ) = if = 0; othewise f ( ) =0 The lage CPE means the similaity measue can estimate moe coespondences pecisely Aveage 8 36 VI CONCLUSION AND FUTURE WORK In this pape, we have developed an UAV to Google satellite image egistation system Moeove, we fist conside the quality mismatch poblem between the UAV image and Google satellite image Fom the expeimental esults, we can see that the poposed system with a nomalized 34

5 vaiant of mutual infomation could obtain bette egistation esults Futhemoe, we have poposed a measue CPE to assess the effectiveness of the mutual infomation fo similaity measue unde quality mismatch The CPE also shows that the NVMI can pefom bette than the MI Futue woks will poceed to integate the poposed system with Google Eath/Map fo pactical application The Google Eath/Map [5] is the most advanced map sevice application and technology developed by Google Google Eath/Map povides abundant geogaphical infomation including map and satellite image and local business infomation including business locations, contact infomation etc Thus, fo instance, we can pefom the envionmental change detection by combining the UAV and Google Eath/Map On the othe hand, because of high computational complexity in mutual infomation computation, it is necessay to developing a fast algoithm fo mutual infomation computation to educe computational complexity fo eal-time application ACKNOWLEDGMENT This eseach is sponsoed by the Industial Technology Reseach Institute (ITRI), Taiwan, ROC and NSC 0-- E REFERENCES [] Wikipedia [Online] Avaliable: [] L G Bown, A Suvey Of Image Registation Techniques, ACM Computing Suvey, vol 4, no 4, pp , 99 [3] B Zitov and Jan Flusse, Image Registation Methods: A Suvey, Image and Vision Computing, vol, no, pp , 003 [4] J P Pluim, J B Maintz, and M A Viegeve, "Mutual-Infomation- Based Registation of Medical Images: A Suvey," IEEE Tansaction on Medical Imaging, vol, no 8, pp , Aug 003 [5] M Bown and D Lowe, Automatic Panoamic Image Stitching using Invaiant Featues, Intenational Jounal of Compute Vision, vol 74, no, pp 59-73, 007 [6] Quanfu Fan, Kobus Banad, Anon Ami, and Alon Efat, Robust Spatiotempoal Matching of Electonic Slides to Pesentation Videos, IEEE Tansactions on Image Pocessing, vol 0, no 8, pp 35-38, 0 [7] M Vivet, B Matinez, X Binefa, DLIG: Diect Local Indiect Global Alignment fo Video Mosaicing, IEEE Tansactions on Cicuits and Systems fo Video Technology, vol, no, pp , 0 [8] S Bake and I Matthews, Lucas-Kanade 0 Yeas On: A Unifying Famewok, Intenational Jounal of Compute Vision, vol 56, no 3, pp - 55, Mach, 004 [9] N Rebiee, M-F Auclai-Fotie, and F Deschenes, Image Mosaicing using Local Optical Flow Registation, 9 th Intenational Confeence on Patten Recognition (ICPR), pp -5, 008 [0] Yuping Lin and Géad Medioni, Map-Enhanced UAV Iimage Sequence Registation and Synchonization of Multiple Image Sequences, in IEEE Compute Vision and Patten Recognition (CVPR) Wokshop on Image Registation and Fusion, pp -7, 007 [] Yuping Lin, Qian Yu, and Géad Medioni, Map-Enhanced UAV Image Sequence Registation, in IEEE Wokshop on Applications of Compute Vision (WACV), 007 [] Industial Technology Reseach Institute (ITRI) [Online] Avaliable: [3] M A Fischle and R C Bolles, Random Sample Consensus: A Paadigm Fo Model Fitting With Applications To Image Analysis And Automated Catogaphy, Communication Of theacm, vol 4, no 6, pp , 98 [4] Sunglok Choi, Taemin Kim, and Wonpil Yu, Pefomance Evaluation of RANSAC Family, In Poceedings of the Bitish Machine Vision Confeence (BMVC), pp -, 009 [5] Google Eath [Online] Avaliable: Figue 6 Image egistation esult fame by fame (fom top to bottom) of the fist UAV image sequence onto the satellite image (a) system in [0, ] (b) ous 35